![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/blocks.png\")
What a fun competition! I enjoyed this competition! Thank you Kaggle and host.
\nBelow are 3 building blocks of deep learning. By creating different sequences of these building blocks, we can build powerful models! Each block takes an input shape of (batch_size, sequence_length, features) and outputs the same. The RNN \"block\" uses bidirectional (example code here). The CNN \"block\" uses sums and products of multiple dilated convolutions (from WaveNet paper here and example code here). And the Transformer blocks uses self attention (example code here). These 3 \"blocks\" can create features from signals without needing feature engineering!
Our data loader will randomly give us 24 hour windows that always contain both one onset and one wakeup. During each epoch, we will get different cuts. This serves as crop data augmentation.
\n ![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/dataloader.png\")
For the head of our model, we use an architecture that is not in any public discussion nor public notebook. We use a technique from NLP called QuestionAnswering (i.e. BertForQuestionAnswering). This is analogous to Computer Vision's object detection. All public discussions and notebooks use NLP NER (i.e. BertForTokenClassification) which is analogous to Computer Vision segmentation.
The way to implement QA is for our data loader to always include one onset and one wakeup in every train sequence. Then we apply 1440 multiclass softmax loss to pick the correct minute of target. Later we will use a separately trained classification model to remove false positives (since our QA model was trained on 100% positives and doesn't locate negatives well).
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/qa-head2.png\")
Approximately CV 0.800 LB 0.750
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/model2b.png\")
Approximately CV 0.805 LB 0.755
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/model1c.png\")
Approximately CV +0.050 LB 0.050 Wow!
\nA huge trick in this competition is to make multiple onset and wakeup predictions per night. First we make a first choice for every user every night. With 200 test users and approximately 30 nights per user and 2 targets, this is about 12000 predictions. Next we make a second guess (i.e. the second max probability) for each night and divide it by 2^1. This is 12000 more guesses. Next we made a 3rd guess (i.e. 3rd max prob) and divide it by 2^2. We continue this for 30 guesses per night where the last guesses are divided by 2^29. This boost CV by about +0.050 wow 😀!
This works because Average Precision metric is always improved when adding more predictions if the new predictions have a lower score than all previous predictions. (More explanation here)
\nApproximately CV +0.020 LB 0.020
\nFor the metric Average Precision, the score we assign to each prediction is very important. First we use the probability from our above models as the starting score. Next we use CatBoost with handmade convolution features (CatBoost achieves LB 0.705 by itself wow!), 1D-Unet, 2D-Unet, Mel Spectrograms, ResNet34, EfficientNetB5, and Audio WaveNet to rerank the scores. This is accomplished by predicting a new score for each prediction and then averaging all the scores.
Approximately CV +0.010 LB 0.010
\nMy CNN-Transformer-GRU and Deep-GRU are very diverse. First I apply NMS (non maximum supression) to each model individually. Whenever two predictions are within 5 minutes of each other, we keep only the prediction which largest probability. Afterward we ensemble the two single models with WBF (weighted box fusion). Whenever two predictions are within 5 minutes of each other (and are the same level 1 thru 30 explained above in \"extra scores trick\"), then we replace the two with the average step position.
I published the code for these two TensorFlow models here
", "messages": [ { "id": "2550266", "postDate": "12/06/2023 00:10:37", "content": "What a fun competition! I enjoyed this competition! Thank you Kaggle and host.
\nBelow are 3 building blocks of deep learning. By creating different sequences of these building blocks, we can build powerful models! Each block takes an input shape of (batch_size, sequence_length, features) and outputs the same. The RNN \"block\" uses bidirectional (example code here). The CNN \"block\" uses sums and products of multiple dilated convolutions (from WaveNet paper here and example code here). And the Transformer blocks uses self attention (example code here). These 3 \"blocks\" can create features from signals without needing feature engineering!
Our data loader will randomly give us 24 hour windows that always contain both one onset and one wakeup. During each epoch, we will get different cuts. This serves as crop data augmentation.
\n
For the head of our model, we use an architecture that is not in any public discussion nor public notebook. We use a technique from NLP called QuestionAnswering (i.e. BertForQuestionAnswering). This is analogous to Computer Vision's object detection. All public discussions and notebooks use NLP NER (i.e. BertForTokenClassification) which is analogous to Computer Vision segmentation.
The way to implement QA is for our data loader to always include one onset and one wakeup in every train sequence. Then we apply 1440 multiclass softmax loss to pick the correct minute of target. Later we will use a separately trained classification model to remove false positives (since our QA model was trained on 100% positives and doesn't locate negatives well).
\n
Approximately CV 0.800 LB 0.750
\n
Approximately CV 0.805 LB 0.755
\n
Approximately CV +0.050 LB 0.050 Wow!
\nA huge trick in this competition is to make multiple onset and wakeup predictions per night. First we make a first choice for every user every night. With 200 test users and approximately 30 nights per user and 2 targets, this is about 12000 predictions. Next we make a second guess (i.e. the second max probability) for each night and divide it by 2^1. This is 12000 more guesses. Next we made a 3rd guess (i.e. 3rd max prob) and divide it by 2^2. We continue this for 30 guesses per night where the last guesses are divided by 2^29. This boost CV by about +0.050 wow 😀!
This works because Average Precision metric is always improved when adding more predictions if the new predictions have a lower score than all previous predictions. (More explanation here)
\nApproximately CV +0.020 LB 0.020
\nFor the metric Average Precision, the score we assign to each prediction is very important. First we use the probability from our above models as the starting score. Next we use CatBoost with handmade convolution features (CatBoost achieves LB 0.705 by itself wow!), 1D-Unet, 2D-Unet, Mel Spectrograms, ResNet34, EfficientNetB5, and Audio WaveNet to rerank the scores. This is accomplished by predicting a new score for each prediction and then averaging all the scores.
Approximately CV +0.010 LB 0.010
\nMy CNN-Transformer-GRU and Deep-GRU are very diverse. First I apply NMS (non maximum supression) to each model individually. Whenever two predictions are within 5 minutes of each other, we keep only the prediction which largest probability. Afterward we ensemble the two single models with WBF (weighted box fusion). Whenever two predictions are within 5 minutes of each other (and are the same level 1 thru 30 explained above in \"extra scores trick\"), then we replace the two with the average step position.
I published the code for these two TensorFlow models here
", "rawMarkdown": "What a fun competition! I enjoyed this competition! Thank you Kaggle and host.\n\n# Building Blocks of GPU Deep Learning!\nBelow are 3 building blocks of deep learning. By creating different sequences of these building blocks, we can build powerful models! Each block takes an input shape of `(batch_size, sequence_length, features)` and outputs the same. The RNN \"block\" uses **bidirectional** (example code [here][6]). The CNN \"block\" uses sums and products of multiple **dilated convolutions** (from WaveNet paper [here][5] and example code [here][6]). And the Transformer blocks uses **self attention** (example code [here][6]). These 3 \"blocks\" can create features from signals without needing feature engineering!\n\n\n\n# Data Loader - Cut Data into Daily Chunks\nOur data loader will randomly give us 24 hour windows that **always contain both one onset and one wakeup**. During each epoch, we will get different cuts. This serves as crop data augmentation.\n \n\n# Model Head is NLP QA Softmax (not NLP NER)\nFor the head of our model, we use an architecture that is **not** in any public discussion **nor** public notebook. We use a technique from NLP called QuestionAnswering (i.e. `BertForQuestionAnswering`). This is analogous to Computer Vision's `object detection`. All public discussions and notebooks use NLP NER (i.e. `BertForTokenClassification`) which is analogous to Computer Vision `segmentation`. \n\nThe way to implement QA is for our data loader to always include one onset and one wakeup in every train sequence. Then we apply 1440 multiclass softmax loss to pick the correct minute of target. Later we will use a separately trained classification model to remove false positives (since our QA model was trained on 100% positives and doesn't locate negatives well).\n\n\n# My CNN-Transformer-GRU, 80 features\n**Approximately CV 0.800 LB 0.750**\n\n\n# My Deep-GRU, 2 features\n**Approximately CV 0.805 LB 0.755**\n\n\n# Extra Predictions Trick\n**Approximately CV +0.050 LB 0.050 Wow!**\nA huge trick in this competition is to make multiple onset and wakeup predictions per night. First we make a first choice for every user every night. With 200 test users and approximately 30 nights per user and 2 targets, this is about 12000 predictions. Next we make a second guess (i.e. the second max probability) for each night and divide it by `2^1`. This is 12000 more guesses. Next we made a 3rd guess (i.e. 3rd max prob) and divide it by `2^2`. We continue this for 30 guesses per night where the last guesses are divided by `2^29`. This boost CV by about +0.050 wow 😀! \n\nThis works because Average Precision metric is **always improved** when adding more predictions if the new predictions have a lower score than all previous predictions. (More explanation [here][1])\n\n# Rerank Scores\n**Approximately CV +0.020 LB 0.020**\nFor the metric Average Precision, the score we assign to each prediction is very important. First we use the probability from our above models as the starting score. Next we use CatBoost with handmade convolution features (**CatBoost achieves LB 0.705** by itself wow!), 1D-Unet, 2D-Unet, Mel Spectrograms, ResNet34, EfficientNetB5, and Audio WaveNet to rerank the scores. This is accomplished by predicting a new score for each prediction and then averaging all the scores.\n\n# Ensemble with NMS and WBF\n**Approximately CV +0.010 LB 0.010**\nMy CNN-Transformer-GRU and Deep-GRU are very diverse. First I apply NMS (non maximum supression) to each model individually. Whenever two predictions are within 5 minutes of each other, we keep only the prediction which largest probability. Afterward we ensemble the two single models with WBF (weighted box fusion). Whenever two predictions are within 5 minutes of each other (and are the same level 1 thru 30 explained above in \"extra scores trick\"), then we replace the two with the average step position.\n\n# Solution Code\nI published the code for these two TensorFlow models [here][6]\n\n[1]: https://www.kaggle.com/competitions/vinbigdata-chest-xray-abnormalities-detection/discussion/229637\n[2]: https://www.kaggle.com/code/ragnar123/wavenet-with-1-more-feature\n[3]: https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-112\n[4]: https://www.tensorflow.org/text/tutorials/transformer\n[5]: https://arxiv.org/abs/1609.03499\n[6]: https://www.kaggle.com/cdeotte/11th-place-gold-cv-835-public-lb-788", "votes": null }, { "id": "2550270", "postDate": "12/06/2023 00:19:53", "content": "Learning from you yet again, a pleasure to watch you work. Congrats Chris!
", "rawMarkdown": "Learning from you yet again, a pleasure to watch you work. Congrats Chris!", "votes": null }, { "id": "2550274", "postDate": "12/06/2023 00:22:31", "content": "Great solution as always Chris. Very well done!! You were in the top for a very long time and I was like, okay Chris is for sure doing something with the post-processing again 😅
\nAnd I was wondering which features did you generate for 80 features part?
Congratulations with one more solo gold. Was rooting for you all the time and so sad you got hit by shakeup.
\nWas a fun ride.
\nReally enjoyed the competition.
Ruby K_mat and you always pushed us to try harder.
\nOur solution is 100% result of the Francisco (Fnoa) hard work and he will write a review (my lgmb single model is 0.757 public / 0.82 private)
", "rawMarkdown": "Congratulations with one more solo gold. Was rooting for you all the time and so sad you got hit by shakeup.\n\nWas a fun ride. \nReally enjoyed the competition.\n\nRuby K_mat and you always pushed us to try harder.\n\nOur solution is 100% result of the Francisco (Fnoa) hard work and he will write a review (my lgmb single model is 0.757 public / 0.82 private)", "votes": null }, { "id": "2550278", "postDate": "12/06/2023 00:26:29", "content": "Wow that's an amazing LGBM model. I'm looking forward to your writeup.
\nI was always watching your team and trying to stay ahead of you haha 😀 . You pushed me to work harder 😀 We were tied with same public LB score on many days. Congratulations building a strong generalizing model and jumping upward on private LB!
", "rawMarkdown": "Wow that's an amazing LGBM model. I'm looking forward to your writeup.\n\nI was always watching your team and trying to stay ahead of you haha 😀 . You pushed me to work harder 😀 We were tied with same public LB score on many days. Congratulations building a strong generalizing model and jumping upward on private LB!", "votes": null }, { "id": "2550284", "postDate": "12/06/2023 00:30:14", "content": "https://www.kaggle.com/kyakovlev/cmi-baseline-lgbm-v201
\nlgbm - I'll clean private dataset and publish run with train also
Thanks Sinan. The Average Precision metric has lots of way to exploit. The main ones are \"making more predictions which have a lower score than every other previous prediction\" will always increase the Average Precision metric. You can read about is here That is the idea behind the 30 levels of predictions each night where each new set gets divided by 2. The second way to exploit AP is to rerank/modify the scores. We want all false positives to have lower score than true positives for boost in metric.
\nAnother PP trick that I used (and did not write about above) was to add or subtract 1 to any prediction which was exactly a multiple of 12. This gave a nice boost. (Because of the metric, we do not want to predict perfect multiples of 12).
", "rawMarkdown": "Thanks Sinan. The Average Precision metric has lots of way to exploit. The main ones are \"making more predictions which have a lower score than every other previous prediction\" will always increase the Average Precision metric. You can read about is [here][1] That is the idea behind the 30 levels of predictions each night where each new set gets divided by 2. The second way to exploit AP is to rerank/modify the scores. We want all false positives to have lower score than true positives for boost in metric.\n\nAnother PP trick that I used (and did not write about above) was to add or subtract 1 to any prediction which was exactly a multiple of 12. This gave a nice boost. (Because of the metric, we do not want to predict perfect multiples of 12). \n\n[1]: https://www.kaggle.com/competitions/vinbigdata-chest-xray-abnormalities-detection/discussion/229637", "votes": null }, { "id": "2550292", "postDate": "12/06/2023 00:41:35", "content": "Thank you so much Chris. These are all great gems to have in the arsenal. Will keep coming to this post probably multiple times again in the future!
", "rawMarkdown": "Thank you so much Chris. These are all great gems to have in the arsenal. Will keep coming to this post probably multiple times again in the future!", "votes": null }, { "id": "2550300", "postDate": "12/06/2023 00:52:06", "content": "Congratulations once again @cdeotte! It was a fun battle for days 😀
", "rawMarkdown": "Congratulations once again @cdeotte! It was a fun battle for days 😀", "votes": null }, { "id": "2550302", "postDate": "12/06/2023 00:57:50", "content": "Absolutely was fun! I knew your team was working hard every day and that encouraged me to work hard every day!
\nCongratulations on an awesome 3rd place cash gold finish!
", "rawMarkdown": "Absolutely was fun! I knew your team was working hard every day and that encouraged me to work hard every day!\n\nCongratulations on an awesome 3rd place cash gold finish!", "votes": null }, { "id": "2550311", "postDate": "12/06/2023 01:05:05", "content": "Congrats Chris on the gold!
\nAnd thanks for the clearly written writeup. Your solution is so sophisticated, yet easy to understand 🙂
\nI found this idea very clever:
\n\"Later we will use a separately trained classification model to remove false positives (since our QA model was trained on 100% positives and doesn't locate negatives well).\"
\nI have one question:
\nYou are dividing the second best prediction's probability by 2^2, the third best predictions's by 2^3, etc. How did you realise that this is the right way to 'adjust' the probabilities?
", "rawMarkdown": "Congrats Chris on the gold!\n\nAnd thanks for the clearly written writeup. Your solution is so sophisticated, yet easy to understand 🙂\n\nI found this idea very clever:\n\n\"Later we will use a separately trained classification model to remove false positives (since our QA model was trained on 100% positives and doesn't locate negatives well).\"\n\nI have one question:\n\nYou are dividing the second best prediction's probability by 2^2, the third best predictions's by 2^3, etc. How did you realise that this is the right way to 'adjust' the probabilities?", "votes": null }, { "id": "2550312", "postDate": "12/06/2023 01:08:59", "content": "Interesting solution! My solution is quite similar to yours - I used a window of 2 days and a custom defined self attention module instead of the usual transformer block.
\nI didn't do extra postprocessing for the scores, I guess reranking with GB methods was what I needed for better calibrated probabilities.
\nThe postprocessing I did is to align the predictions to the nearest .25 or .75 second, augmenting the predictions by +-1 second (+0.005 CV), and removing repeated signals by matrix profile (+0.001 CV and private LB)
", "rawMarkdown": "Interesting solution! My solution is quite similar to yours - I used a window of 2 days and a custom defined self attention module instead of the usual transformer block. \n\nI didn't do extra postprocessing for the scores, I guess reranking with GB methods was what I needed for better calibrated probabilities.\n\nThe postprocessing I did is to align the predictions to the nearest .25 or .75 second, augmenting the predictions by +-1 second (+0.005 CV), and removing repeated signals by matrix profile (+0.001 CV and private LB)", "votes": null }, { "id": "2550313", "postDate": "12/06/2023 01:09:20", "content": "I’m impressed to see GRU having such a high score! 🚀
", "rawMarkdown": "I’m impressed to see GRU having such a high score! 🚀", "votes": null }, { "id": "2550317", "postDate": "12/06/2023 01:13:22", "content": "\n\nYou are dividing the second best prediction's probability by 2^2, the third best predictions's by 2^3, etc. How did you realise that this is the right way to 'adjust' the probabilities?
\n
The first thing to realize is that we cannot start making second predictions for one night before we make first predictions for every user every night. (Because when we make second predictions then certainly either the first or second prediction will be a false positive and hurt metric). The Average Precision metric will rank every user every night all together and we want all false positives to have a lower score than true positives to optimize Average Precision. So the majority of our second predictions need to have a lower score than the majority of our first predictions.
\nThe exact division value is a hyperparameter that I determined from CV. First I used 10 because we want the second set of predictions to be \"an order of magnitude\" lower than the first set of predictions. The hyperparameter was exponent in DIVIDE_CONSTANT = 10**exponent. It turned out that exponent = 1/5 was best which makes DIVIDE CONSTANT approximately 2.
Amazing solution as always, Chris. Congrats on one more gold
", "rawMarkdown": "Amazing solution as always, Chris. Congrats on one more gold", "votes": null }, { "id": "2550332", "postDate": "12/06/2023 01:29:51", "content": "Awesome job! Congratulations @chauyh on solo gold finish.
\nI'm curious did you build features and/or post process information from the fact that sometimes the host make fake anglez enmo? For example, you could make a new feature \"has this anglez value at this daily timestep been seen before\" boolean feature. Or you could post process remove predictions from time windows that had fake data?
\nI'm curious if there was helpful information here. I think my models naturally removed this stuff.
", "rawMarkdown": "Awesome job! Congratulations @chauyh on solo gold finish.\n\nI'm curious did you build features and/or post process information from the fact that sometimes the host make fake anglez enmo? For example, you could make a new feature \"has this anglez value at this daily timestep been seen before\" boolean feature. Or you could post process remove predictions from time windows that had fake data?\n\nI'm curious if there was helpful information here. I think my models naturally removed this stuff.", "votes": null }, { "id": "2550343", "postDate": "12/06/2023 01:41:01", "content": "A lot of great ideas! If some of the top ranks on the LB combined their ideas, 0.88 might be possible!
", "rawMarkdown": "A lot of great ideas! If some of the top ranks on the LB combined their ideas, 0.88 might be possible!", "votes": null }, { "id": "2550349", "postDate": "12/06/2023 01:48:40", "content": "Thanks @fritzcremer . Congratulations on amazing cash solo gold !!
", "rawMarkdown": "Thanks @fritzcremer . Congratulations on amazing cash solo gold !!", "votes": null }, { "id": "2550437", "postDate": "12/06/2023 03:15:13", "content": "Yes it is impressive how well GRU can do with only input the two features anglez and enmo. Note I did post process the prediction logits for all minute % 15 which equaled one of the values that did not occur in the train data. (There were 4 rare values for onset and 4 rare values for wakeup). For these rare logits, I divided their logit prediction by 2 before performing softmax. This boosted CV about approximately +0.010 or +0.020 (if i remember correctly)
I also tried adding minute % 15 as a feature for the GRU. It boosted the CV score, but it was better to not include this feature and post process. This is probably because the model will overfit when the feature minute % 15 is present.
What a unique solution. Congratulations @cdeotte on just another brilliant performance. We all learn from you everytime
", "rawMarkdown": "What a unique solution. Congratulations @cdeotte on just another brilliant performance. We all learn from you everytime", "votes": null }, { "id": "2550503", "postDate": "12/06/2023 04:49:30", "content": "Thanks @cdeotte and congrats on solo gold too!
\nMy approach is to train a DL model end to end and use as little manual processing /postprocessing as possible. I only applied a box-cox transform to enmo, and used the raw anglez as input to the DL model.
\nAt the last stages of the competition (~1 week), I found out that postprocessing to calibrate probabilities is quite important. I focused on the missing labels problem but the gains were trivial (+0.0004 CV, compared to approach already with matrix profile pruning).
\nI tried very hard to find patterns leading to missing labels but I couldn't find any. The only consistent pattern was repeated data, which is could easily be pruned away with matrix profile. Interestingly, there are some labels in the training set for the repeated data.
\nLet me first try some of your postprocessing methods and I'll post a detailed writeup of my methods and findings during the weekend!
", "rawMarkdown": "Thanks @cdeotte and congrats on solo gold too!\n\nMy approach is to train a DL model end to end and use as little manual processing /postprocessing as possible. I only applied a box-cox transform to enmo, and used the raw anglez as input to the DL model.\n\nAt the last stages of the competition (~1 week), I found out that postprocessing to calibrate probabilities is quite important. I focused on the missing labels problem but the gains were trivial (+0.0004 CV, compared to approach already with matrix profile pruning).\n\nI tried very hard to find patterns leading to missing labels but I couldn't find any. The only consistent pattern was repeated data, which is could easily be pruned away with matrix profile. Interestingly, there are some labels in the training set for the repeated data.\n\nLet me first try some of your postprocessing methods and I'll post a detailed writeup of my methods and findings during the weekend!", "votes": null }, { "id": "2550628", "postDate": "12/06/2023 06:48:10", "content": "There is always something to learn from you @cdeotte. Thanks for the nice writeup. Congratulations on one more gold
", "rawMarkdown": "There is always something to learn from you @cdeotte. Thanks for the nice writeup. Congratulations on one more gold", "votes": null }, { "id": "2550695", "postDate": "12/06/2023 08:16:43", "content": "Amazing solution. Can you please share an inference code ?)
", "rawMarkdown": "Amazing solution. Can you please share an inference code ?)", "votes": null }, { "id": "2550761", "postDate": "12/06/2023 08:54:42", "content": "Despite not participating in the competition, I learned a lot about it just by reading this.
\nImpressive solution as always, congrats Chris.
", "rawMarkdown": "Despite not participating in the competition, I learned a lot about it just by reading this.\n\nImpressive solution as always, congrats Chris.", "votes": null }, { "id": "2550824", "postDate": "12/06/2023 10:01:22", "content": "Well done on the solo gold ! @cdeotte . we missed out on these metric specific tricks which turned out to be a huge deal, will make sure to study the metric more carefully next time. Really appreciate you sharing this!
", "rawMarkdown": "Well done on the solo gold ! @cdeotte . we missed out on these metric specific tricks which turned out to be a huge deal, will make sure to study the metric more carefully next time. Really appreciate you sharing this!", "votes": null }, { "id": "2550916", "postDate": "12/06/2023 11:54:44", "content": "Hi,Congrats Chris on the gold! I have a question I would like to ask. Is BertForTokenClassification much worse than BertForQuestionAnswering? Why did you choose BertForQuestionAnswering? I understand that BertForTokenClassification can also be used for subsequent post-processing processes.
", "rawMarkdown": "Hi,Congrats Chris on the gold! I have a question I would like to ask. Is BertForTokenClassification much worse than BertForQuestionAnswering? Why did you choose BertForQuestionAnswering? I understand that BertForTokenClassification can also be used for subsequent post-processing processes.", "votes": null }, { "id": "2551066", "postDate": "12/06/2023 13:42:20", "content": "Well Done Chris! Impressive solution.
", "rawMarkdown": "Well Done Chris! Impressive solution.", "votes": null }, { "id": "2551434", "postDate": "12/06/2023 17:37:51", "content": "Thanks Ertugrul
", "rawMarkdown": "Thanks Ertugrul", "votes": null }, { "id": "2551520", "postDate": "12/06/2023 18:16:42", "content": "Thanks, 3 more to go for Grandmaster 😁 Congrats to your 22nd Gold haha
", "rawMarkdown": "Thanks, 3 more to go for Grandmaster 😁 Congrats to your 22nd Gold haha", "votes": null }, { "id": "2551630", "postDate": "12/06/2023 20:41:04", "content": "Hi @sayoulala I think both methods work well. To me the task seemed like QA so I went QA from the beginning. Below are predictions from each technique. I think we need different post process for NER vs. QA. For my QA pipeline, I could simply find the top30 maxes to be my top30 predictions per night since QA makes peaks on single minute values. With NER, I think we would need to use a different way to find additional predictions.
\nFor diversity, i also trained NER models and used them to improve the score value for each QA prediction.
![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/qa.png\")
![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/ner.png\")
Fabulous sharing on GPU deep learning. Learnt a lot from u.
", "rawMarkdown": "Fabulous sharing on GPU deep learning. Learnt a lot from u.", "votes": null }, { "id": "2551816", "postDate": "12/07/2023 02:04:37", "content": "Do you mind sharing the pseudo code you used to automatically generate features?
", "rawMarkdown": "Do you mind sharing the pseudo code you used to automatically generate features?", "votes": null }, { "id": "2551940", "postDate": "12/07/2023 04:38:26", "content": "ResNet50 GPU deep learning
", "rawMarkdown": "ResNet50 GPU deep learning", "votes": null }, { "id": "2552021", "postDate": "12/07/2023 06:02:52", "content": "The observations on the evaluation metrics are most impressive to me so I will implement such technique in late submissions. Thank you!
", "rawMarkdown": "The observations on the evaluation metrics are most impressive to me so I will implement such technique in late submissions. Thank you!", "votes": null }, { "id": "2552201", "postDate": "12/07/2023 09:33:38", "content": "Congratulations. Thanks for sharing your solution details. Very interesting approach indeed.
", "rawMarkdown": "Congratulations. Thanks for sharing your solution details. Very interesting approach indeed.", "votes": null }, { "id": "2552957", "postDate": "12/07/2023 22:35:39", "content": "Thanks NikhilMishra. Congratulations to you and penguin46. Great job achieving cash gold !!
", "rawMarkdown": "Thanks NikhilMishra. Congratulations to you and penguin46. Great job achieving cash gold !!", "votes": null }, { "id": "2553212", "postDate": "12/08/2023 04:54:59", "content": "I have no word. It's a great post processing!
", "rawMarkdown": "I have no word. It's a great post processing!", "votes": null }, { "id": "2553215", "postDate": "12/08/2023 04:57:52", "content": "I have a problem to come up with what the right way to build input data is.
\nCould you explain why you selected the input data with both an onset and a wakeup?
This is a powerful technique to achieve top results in object detection and object segmentation Kaggle competitions!
\nWe want the model to locate where onset and wakeup is. So every train sample contains an onset and wakeup. (We cannot learn location if there is no onset wakeup). Therefore our dataloader, provides random 24 hour crops and every crop contains both onset and wakeup.
This model will beat other models and techniques for locating onset and wakeup. However this model can not determine existence. This model will think that every day contains an onset and wakeup. So we also need to build a second model (not described in my solution post). The second model job is a classifier that outputs 0 or 1 predicting when a 24 window has an onset and wakeup.
We train the second model with a different data loader. The data loader for the second model provides all types of random 24 hour crops. Some crops have onset wakeup, and some crops do not have onset wakeup. The second model is binary classifier that learns when onset wakeup is present or not.
\nThen our inference pipeline uses both models, our first model predicts onset wakeup for every night, i.e. 24 hours. And our second model removes false positives. Whenever our second model outputs prob < 0.2 for a night (i.e. 24 hour window), then we remove the first model prediction from submission.csv.
That's very logical approach.
\nAnd I really appreciate your long and detail answer. I'm so touched. :)
UPDATE I published the code for these two TensorFlow models here
", "rawMarkdown": "**UPDATE** I published the code for these two TensorFlow models [here][1]\n\n[1]: https://www.kaggle.com/cdeotte/11th-place-gold-cv-835-public-lb-788", "votes": null }, { "id": "2554642", "postDate": "12/09/2023 09:17:28", "content": "Excellent work! Really learnt a lot from this. I should have stuck with my keras framework rather than going to learn pytorch. Now I know…
", "rawMarkdown": "Excellent work! Really learnt a lot from this. I should have stuck with my keras framework rather than going to learn pytorch. Now I know...", "votes": null }, { "id": "2554711", "postDate": "12/09/2023 10:44:28", "content": "Thanks for Sharing
", "rawMarkdown": "Thanks for Sharing", "votes": null }, { "id": "2555388", "postDate": "12/09/2023 21:32:46", "content": "The RNN, CNN, and Transformer blocks generate features for us. Code is shown here. Then the model heads makes the prediction using features from the proceeding blocks.
", "rawMarkdown": "The RNN, CNN, and Transformer blocks generate features for us. Code is shown [here][1]. Then the model heads makes the prediction using features from the proceeding blocks.\n\n[1]: https://www.kaggle.com/code/cdeotte/11th-place-gold-cv-835-public-lb-788", "votes": null }, { "id": "2558156", "postDate": "12/12/2023 00:24:35", "content": "For reference, here are the 3 popular ways to create a target in this competition. All three performed well. From all three, we can compute a step and score for submission.csv:
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/targets2.png\")
For reference, here are two Mel Spectrograms with 1D prediction and 2D prediction and ground truth:
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/mel1.png\")
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/mel2.png\")
Congratulations! Such an excellent work. As a beginner I have really learned a lot from your solution. I am still a little bit confused about two details of your approach.
\nOne is that you set crops for 24 hours which is 17280 steps but your input for Transformer Model is 1440. So I'm wondering whether you are using the mean of 12 steps as one data point as input. May I also asked about the 80 features you used?
\nAnd the second one is about labels. It is a great idea to seperate \"onset\" and \"awake\" as two different labels and I noticed that some competitors applied a gaussian window around the event step. So did you use the same solution or just use original \"0\" and \"1\" labels?
\nThank you so much if you can reply to my comment.
", "rawMarkdown": "Congratulations! Such an excellent work. As a beginner I have really learned a lot from your solution. I am still a little bit confused about two details of your approach. \n\nOne is that you set crops for 24 hours which is 17280 steps but your input for Transformer Model is 1440. So I'm wondering whether you are using the mean of 12 steps as one data point as input. May I also asked about the 80 features you used?\n\nAnd the second one is about labels. It is a great idea to seperate \"onset\" and \"awake\" as two different labels and I noticed that some competitors applied a gaussian window around the event step. So did you use the same solution or just use original \"0\" and \"1\" labels?\n\nThank you so much if you can reply to my comment.", "votes": null }, { "id": "2559395", "postDate": "12/12/2023 19:32:07", "content": "Thank you
\n\n\nOne is that you set crops for 24 hours which is 17280 steps but your input for Transformer Model is 1440
\n
My CNN-Transformer-GRU uses an input of 1440. The data loader output just skips every 12th using the following Python indexing X_train[:,::12,:].
\n\nMay I also asked about the 80 features you used?
\n
I used the 79 features from this popular LGBM notebook here. And I added the additional feature minute%15 to learn the weird labeler bias. (I also built my own CatBoost with my own 200 features and my CatBoost scored LB 0.705 but my Transformer used those public notebook features).
\n\nSo did you use the same solution or just use original \"0\" and \"1\" labels?
\n
I used the original \"0\" and \"1\". Each of my two models here has output sequence length 1440 which is the number of minutes per day. The target for each day crop were 1439 zeros with 1 one where the actual target is. My loss is softmax multi-class crossentropy, so the model had no trouble learning the unique target \"1\" out of \"1440\" different classes. (I.e. the multiclass is the 1440 different minutes each day and the model finds the correct minute each day)
", "rawMarkdown": "Thank you\n\n>One is that you set crops for 24 hours which is 17280 steps but your input for Transformer Model is 1440\n\nMy `CNN-Transformer-GRU` uses an input of 1440. The data loader output just skips every 12th using the following Python indexing `X_train[:,::12,:]`.\n\n>May I also asked about the 80 features you used?\n\nI used the 79 features from this popular LGBM notebook [here][1]. And I added the additional feature `minute%15` to learn the weird labeler bias. (I also built my own CatBoost with my own 200 features and my CatBoost scored LB 0.705 but my Transformer used those public notebook features). \n\n>So did you use the same solution or just use original \"0\" and \"1\" labels?\n\nI used the original \"0\" and \"1\". Each of my two models here has output sequence length 1440 which is the number of minutes per day. The target for each day crop were 1439 zeros with 1 one where the actual target is. My loss is softmax multi-class crossentropy, so the model had no trouble learning the unique target \"1\" out of \"1440\" different classes. (I.e. the multiclass is the 1440 different minutes each day and the model finds the correct minute each day)\n\n[1]: https://www.kaggle.com/code/renatoreggiani/lightgbm-starter-cmi", "votes": null }, { "id": "2559403", "postDate": "12/12/2023 19:42:58", "content": "Got it. Thank you so much!
", "rawMarkdown": "Got it. Thank you so much!", "votes": null }, { "id": "2559919", "postDate": "12/13/2023 06:29:30", "content": "Congratulations! Always can learn a lot from your work. Especially the Deep GRU model that only use 2 original features. Really impressive!
\nI'm trying to train your Deep GRU model, but I got gradient vanishing problem. I use exactly your 'build_model_2()' code, but If I call the \"build_model_2()\" function 5 times, maybe got gradient vanishing for 3 or 4 times.
\nI guess I must miss something somewhere, such as some special initializer? or some kind of random seed setting? Or other stuffs.
\nSo, will you public all your code? Then I can check where I'm going wrong.
Again, thanks a lot for the sharing of your excellent work!
", "rawMarkdown": "Congratulations! Always can learn a lot from your work. Especially the Deep GRU model that only use 2 original features. Really impressive!\n\nI'm trying to train your Deep GRU model, but I got gradient vanishing problem. I use exactly your 'build_model_2()' code, but If I call the \"build_model_2()\" function 5 times, maybe got gradient vanishing for 3 or 4 times.\nI guess I must miss something somewhere, such as some special initializer? or some kind of random seed setting? Or other stuffs.\nSo, will you public all your code? Then I can check where I'm going wrong. \n\nAgain, thanks a lot for the sharing of your excellent work!", "votes": null }, { "id": "2560579", "postDate": "12/13/2023 19:41:21", "content": "Wanted to perform this data transformation but implementation and computational bottleneck was a problem. Had wanted to use the sleeping and waking data spectrograms as images to a 2D CNN but kept crashing. Was this ever feasible given the constraints of the competition? Still would like to implement it fully and compare.
", "rawMarkdown": "Wanted to perform this data transformation but implementation and computational bottleneck was a problem. Had wanted to use the sleeping and waking data spectrograms as images to a 2D CNN but kept crashing. Was this ever feasible given the constraints of the competition? Still would like to implement it fully and compare.", "votes": null }, { "id": "2560603", "postDate": "12/13/2023 20:13:27", "content": "Yes this was feasible. My final solution used this model to predict a score for each onset and wakeup and I added the score to the score from my Transformer and GRU to improve CV and LB. Also by itself, this model achieved good results.
An easy way to avoid memory error is to process data in chunks and save intermediate data to disk. We can do this both for train and infer.
\nFirst read Kaggle's train parquet in chunks and we iterate through every user and write each user to disk as a NumPy file. So for each user we have data shape (days, 17280, 4) where days is the number of 24 hour periods (per user), and 17280 is the number of steps in 24 hours. And 4 is anglez, enmo, and the two targets. (Also we begin each user at 3pm and end at 3pm and add padding to beginning and ending if necessary. That makes all the 24 hour windows the same).
Next we iterate through each of these NumPys. We load from disk then convert to image. Then save images to disk. The code below creates 2 channel images of size 256x1536 for each day for each user:
\nimport librosa\nCROP = 17_280; SRR = 17_280; HL = 12; HH = 256\n\n j,u enumerate(USERS):\n data = np.nan_to_num( np.load(f) )\n jj = []\n r range(len(data)):\n ii = []\n mel_spec = librosa.feature.melspectrogram(=data[r,:,0], =SRR, =2048, =HL, \n =HH, =0, =SRR//2, =2048//4)\n mel_spec_db = librosa.power_to_db(mel_spec, =np.max).astype(np.float32)\n ii.append(mel_spec_db)\n\n mel_spec = librosa.feature.melspectrogram(=data[r,:,1], =SRR, =2048, =HL, \n =HH, =0, =SRR//2, =2048//4)\n mel_spec_db = librosa.power_to_db(mel_spec, =np.max).astype(np.float32)\n ii.append(mel_spec_db)\n\n img2 = np.transpose( np.stack(ii), [1,2,0])\n jj.append(img2)\n\n jj = np.stack(jj)\n np.save(f,jj)\nAlso note that the highest scoring public notebook uses Mel Spectrograms (created by PyTorch) and provides code on GitHub. Discussion here
", "rawMarkdown": "Also note that the highest scoring public notebook uses Mel Spectrograms (created by PyTorch) and provides code on GitHub. Discussion [here][1]\n\n[1]: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940", "votes": null }, { "id": "2560620", "postDate": "12/13/2023 20:26:47", "content": "Thank you Chris. Wish I had had this earlier on. Gave up due to memory lags and crashes. I will work with this for future similar competitions. Once more, I really appreciate your hard work and taking the time to explain the secret dark arts of ML and DL in a succinct manner to us lost-in-the-woods folks.
", "rawMarkdown": "Thank you Chris. Wish I had had this earlier on. Gave up due to memory lags and crashes. I will work with this for future similar competitions. Once more, I really appreciate your hard work and taking the time to explain the secret dark arts of ML and DL in a succinct manner to us lost-in-the-woods folks.", "votes": null }, { "id": "2560622", "postDate": "12/13/2023 20:29:07", "content": "Ok Chris. Will check it out. I wonder if I doing this on TPUs would make that much difference in training and inference speed? Its the scoring time that gives me ulcers.
", "rawMarkdown": "Ok Chris. Will check it out. I wonder if I doing this on TPUs would make that much difference in training and inference speed? Its the scoring time that gives me ulcers.", "votes": null }, { "id": "2560626", "postDate": "12/13/2023 20:32:44", "content": "Oh I see. The librosa library makes it easier. Was trying to do it from scratch but never got a spectrogram plot good enough as yours. Must be some bug in my code.
", "rawMarkdown": "Oh I see. The librosa library makes it easier. Was trying to do it from scratch but never got a spectrogram plot good enough as yours. Must be some bug in my code.", "votes": null }, { "id": "2560778", "postDate": "12/14/2023 02:00:58", "content": "Congratulations! Very interesting approach indeed
", "rawMarkdown": "Congratulations! Very interesting approach indeed", "votes": null }, { "id": "2561855", "postDate": "12/14/2023 22:02:57", "content": "Do you mean that you get gradient vanishing during training? If so, does your data loader output the correct train data? Every train sample must have X_train with shape (batch, 17280, 2) where the 2 features are anglez and enmo standardized (i.e minus mean divide by std using the mean and std of the entire train data). And the target is list of length two [y_train_1, y_train_2] where each are shape (batch, 1440) and each have exactly 1439 zeros and 1 one? (i.e. there is exactly 1 target in every sample. We do not use 24 hour windows that are missing target).
Thanks for your reply.
\nYes, I get gradient vanishing when I train the Deep-GRU model.
\nI checked many times, it seems my data loader satisfies all the demands that you described. But gradient vanishing still happens from time to time.
\nToday, I set global seed using \"tf.random.set_seed()\", and set the operation seeds of initializers for the Conv1d, GRU, Dense layers in the model. so that the initial weights in the model will be the same every time the model is built.
\nLuckily, it works. Gradient vanishing won't happen.
\nMaybe there is still something wrong in my data loader, so that it demand some special initial weights in the model.
\nAnyway, I can avoid gradient vanishing for now. Hope the unknown bug doesn't affect the training result.
\n😅😅😅
Really interesting approach. Your solution was like a Random Forrest, implemented each ML-Code separatelly, instead decision trees.. Haha!
\nI'm completing this challenge (late submission), but in another way! Congrats bro!
Learning from you yet again, a pleasure to watch you work. Congrats Chris!
", "votes": null, "replies": [] }, { "id": 2550274, "author_name": "snnclsr", "author_url": "", "post_date": "12/06/2023 00:22:31", "content": "Great solution as always Chris. Very well done!! You were in the top for a very long time and I was like, okay Chris is for sure doing something with the post-processing again 😅
\nAnd I was wondering which features did you generate for 80 features part?
Thanks Sinan. The Average Precision metric has lots of way to exploit. The main ones are \"making more predictions which have a lower score than every other previous prediction\" will always increase the Average Precision metric. You can read about is here That is the idea behind the 30 levels of predictions each night where each new set gets divided by 2. The second way to exploit AP is to rerank/modify the scores. We want all false positives to have lower score than true positives for boost in metric.
\nAnother PP trick that I used (and did not write about above) was to add or subtract 1 to any prediction which was exactly a multiple of 12. This gave a nice boost. (Because of the metric, we do not want to predict perfect multiples of 12).
", "votes": null, "replies": [ { "id": 2550292, "author_name": "snnclsr", "author_url": "", "post_date": "12/06/2023 00:41:35", "content": "Thank you so much Chris. These are all great gems to have in the arsenal. Will keep coming to this post probably multiple times again in the future!
", "votes": null, "replies": [] } ] } ] }, { "id": 2550276, "author_name": "kyakovlev", "author_url": "", "post_date": "12/06/2023 00:23:19", "content": "Congratulations with one more solo gold. Was rooting for you all the time and so sad you got hit by shakeup.
\nWas a fun ride.
\nReally enjoyed the competition.
Ruby K_mat and you always pushed us to try harder.
\nOur solution is 100% result of the Francisco (Fnoa) hard work and he will write a review (my lgmb single model is 0.757 public / 0.82 private)
", "votes": null, "replies": [ { "id": 2550278, "author_name": "cdeotte", "author_url": "", "post_date": "12/06/2023 00:26:29", "content": "Wow that's an amazing LGBM model. I'm looking forward to your writeup.
\nI was always watching your team and trying to stay ahead of you haha 😀 . You pushed me to work harder 😀 We were tied with same public LB score on many days. Congratulations building a strong generalizing model and jumping upward on private LB!
", "votes": null, "replies": [ { "id": 2550284, "author_name": "kyakovlev", "author_url": "", "post_date": "12/06/2023 00:30:14", "content": "https://www.kaggle.com/kyakovlev/cmi-baseline-lgbm-v201
\nlgbm - I'll clean private dataset and publish run with train also
Congratulations once again @cdeotte! It was a fun battle for days 😀
", "votes": null, "replies": [ { "id": 2550302, "author_name": "cdeotte", "author_url": "", "post_date": "12/06/2023 00:57:50", "content": "Absolutely was fun! I knew your team was working hard every day and that encouraged me to work hard every day!
\nCongratulations on an awesome 3rd place cash gold finish!
", "votes": null, "replies": [] } ] }, { "id": 2550311, "author_name": "sambitmukherjee", "author_url": "", "post_date": "12/06/2023 01:05:05", "content": "Congrats Chris on the gold!
\nAnd thanks for the clearly written writeup. Your solution is so sophisticated, yet easy to understand 🙂
\nI found this idea very clever:
\n\"Later we will use a separately trained classification model to remove false positives (since our QA model was trained on 100% positives and doesn't locate negatives well).\"
\nI have one question:
\nYou are dividing the second best prediction's probability by 2^2, the third best predictions's by 2^3, etc. How did you realise that this is the right way to 'adjust' the probabilities?
", "votes": null, "replies": [ { "id": 2550317, "author_name": "cdeotte", "author_url": "", "post_date": "12/06/2023 01:13:22", "content": "\n\nYou are dividing the second best prediction's probability by 2^2, the third best predictions's by 2^3, etc. How did you realise that this is the right way to 'adjust' the probabilities?
\n
The first thing to realize is that we cannot start making second predictions for one night before we make first predictions for every user every night. (Because when we make second predictions then certainly either the first or second prediction will be a false positive and hurt metric). The Average Precision metric will rank every user every night all together and we want all false positives to have a lower score than true positives to optimize Average Precision. So the majority of our second predictions need to have a lower score than the majority of our first predictions.
\nThe exact division value is a hyperparameter that I determined from CV. First I used 10 because we want the second set of predictions to be \"an order of magnitude\" lower than the first set of predictions. The hyperparameter was exponent in DIVIDE_CONSTANT = 10**exponent. It turned out that exponent = 1/5 was best which makes DIVIDE CONSTANT approximately 2.
Interesting solution! My solution is quite similar to yours - I used a window of 2 days and a custom defined self attention module instead of the usual transformer block.
\nI didn't do extra postprocessing for the scores, I guess reranking with GB methods was what I needed for better calibrated probabilities.
\nThe postprocessing I did is to align the predictions to the nearest .25 or .75 second, augmenting the predictions by +-1 second (+0.005 CV), and removing repeated signals by matrix profile (+0.001 CV and private LB)
", "votes": null, "replies": [ { "id": 2550332, "author_name": "cdeotte", "author_url": "", "post_date": "12/06/2023 01:29:51", "content": "Awesome job! Congratulations @chauyh on solo gold finish.
\nI'm curious did you build features and/or post process information from the fact that sometimes the host make fake anglez enmo? For example, you could make a new feature \"has this anglez value at this daily timestep been seen before\" boolean feature. Or you could post process remove predictions from time windows that had fake data?
\nI'm curious if there was helpful information here. I think my models naturally removed this stuff.
", "votes": null, "replies": [ { "id": 2550503, "author_name": "chauyh", "author_url": "", "post_date": "12/06/2023 04:49:30", "content": "Thanks @cdeotte and congrats on solo gold too!
\nMy approach is to train a DL model end to end and use as little manual processing /postprocessing as possible. I only applied a box-cox transform to enmo, and used the raw anglez as input to the DL model.
\nAt the last stages of the competition (~1 week), I found out that postprocessing to calibrate probabilities is quite important. I focused on the missing labels problem but the gains were trivial (+0.0004 CV, compared to approach already with matrix profile pruning).
\nI tried very hard to find patterns leading to missing labels but I couldn't find any. The only consistent pattern was repeated data, which is could easily be pruned away with matrix profile. Interestingly, there are some labels in the training set for the repeated data.
\nLet me first try some of your postprocessing methods and I'll post a detailed writeup of my methods and findings during the weekend!
", "votes": null, "replies": [] } ] } ] }, { "id": 2550313, "author_name": "anthonytherrien", "author_url": "", "post_date": "12/06/2023 01:09:20", "content": "I’m impressed to see GRU having such a high score! 🚀
", "votes": null, "replies": [ { "id": 2550437, "author_name": "cdeotte", "author_url": "", "post_date": "12/06/2023 03:15:13", "content": "Yes it is impressive how well GRU can do with only input the two features anglez and enmo. Note I did post process the prediction logits for all minute % 15 which equaled one of the values that did not occur in the train data. (There were 4 rare values for onset and 4 rare values for wakeup). For these rare logits, I divided their logit prediction by 2 before performing softmax. This boosted CV about approximately +0.010 or +0.020 (if i remember correctly)
I also tried adding minute % 15 as a feature for the GRU. It boosted the CV score, but it was better to not include this feature and post process. This is probably because the model will overfit when the feature minute % 15 is present.
Amazing solution as always, Chris. Congrats on one more gold
", "votes": null, "replies": [] }, { "id": 2550343, "author_name": "fritzcremer", "author_url": "", "post_date": "12/06/2023 01:41:01", "content": "A lot of great ideas! If some of the top ranks on the LB combined their ideas, 0.88 might be possible!
", "votes": null, "replies": [ { "id": 2550349, "author_name": "cdeotte", "author_url": "", "post_date": "12/06/2023 01:48:40", "content": "Thanks @fritzcremer . Congratulations on amazing cash solo gold !!
", "votes": null, "replies": [ { "id": 2551520, "author_name": "fritzcremer", "author_url": "", "post_date": "12/06/2023 18:16:42", "content": "Thanks, 3 more to go for Grandmaster 😁 Congrats to your 22nd Gold haha
", "votes": null, "replies": [] } ] } ] }, { "id": 2550496, "author_name": "nikhilmishradev", "author_url": "", "post_date": "12/06/2023 04:42:19", "content": "What a unique solution. Congratulations @cdeotte on just another brilliant performance. We all learn from you everytime
", "votes": null, "replies": [ { "id": 2552957, "author_name": "cdeotte", "author_url": "", "post_date": "12/07/2023 22:35:39", "content": "Thanks NikhilMishra. Congratulations to you and penguin46. Great job achieving cash gold !!
", "votes": null, "replies": [] } ] }, { "id": 2550628, "author_name": "anilreddyvv", "author_url": "", "post_date": "12/06/2023 06:48:10", "content": "There is always something to learn from you @cdeotte. Thanks for the nice writeup. Congratulations on one more gold
", "votes": null, "replies": [] }, { "id": 2550695, "author_name": "nazarko99", "author_url": "", "post_date": "12/06/2023 08:16:43", "content": "Amazing solution. Can you please share an inference code ?)
", "votes": null, "replies": [] }, { "id": 2550761, "author_name": "datafan07", "author_url": "", "post_date": "12/06/2023 08:54:42", "content": "Despite not participating in the competition, I learned a lot about it just by reading this.
\nImpressive solution as always, congrats Chris.
", "votes": null, "replies": [ { "id": 2551434, "author_name": "cdeotte", "author_url": "", "post_date": "12/06/2023 17:37:51", "content": "Thanks Ertugrul
", "votes": null, "replies": [] } ] }, { "id": 2550824, "author_name": "fahmiayari", "author_url": "", "post_date": "12/06/2023 10:01:22", "content": "Well done on the solo gold ! @cdeotte . we missed out on these metric specific tricks which turned out to be a huge deal, will make sure to study the metric more carefully next time. Really appreciate you sharing this!
", "votes": null, "replies": [] }, { "id": 2550916, "author_name": "sayoulala", "author_url": "", "post_date": "12/06/2023 11:54:44", "content": "Hi,Congrats Chris on the gold! I have a question I would like to ask. Is BertForTokenClassification much worse than BertForQuestionAnswering? Why did you choose BertForQuestionAnswering? I understand that BertForTokenClassification can also be used for subsequent post-processing processes.
", "votes": null, "replies": [ { "id": 2551630, "author_name": "cdeotte", "author_url": "", "post_date": "12/06/2023 20:41:04", "content": "Hi @sayoulala I think both methods work well. To me the task seemed like QA so I went QA from the beginning. Below are predictions from each technique. I think we need different post process for NER vs. QA. For my QA pipeline, I could simply find the top30 maxes to be my top30 predictions per night since QA makes peaks on single minute values. With NER, I think we would need to use a different way to find additional predictions.
\nFor diversity, i also trained NER models and used them to improve the score value for each QA prediction.
Well Done Chris! Impressive solution.
", "votes": null, "replies": [] }, { "id": 2551698, "author_name": "lizzyshi2003", "author_url": "", "post_date": "12/06/2023 22:05:38", "content": "Fabulous sharing on GPU deep learning. Learnt a lot from u.
", "votes": null, "replies": [] }, { "id": 2551816, "author_name": "samu2505", "author_url": "", "post_date": "12/07/2023 02:04:37", "content": "Do you mind sharing the pseudo code you used to automatically generate features?
", "votes": null, "replies": [ { "id": 2555388, "author_name": "cdeotte", "author_url": "", "post_date": "12/09/2023 21:32:46", "content": "The RNN, CNN, and Transformer blocks generate features for us. Code is shown here. Then the model heads makes the prediction using features from the proceeding blocks.
", "votes": null, "replies": [] } ] }, { "id": 2551940, "author_name": "pytorchnuruzzaman", "author_url": "", "post_date": "12/07/2023 04:38:26", "content": "ResNet50 GPU deep learning
", "votes": null, "replies": [] }, { "id": 2552021, "author_name": "coyamax", "author_url": "", "post_date": "12/07/2023 06:02:52", "content": "The observations on the evaluation metrics are most impressive to me so I will implement such technique in late submissions. Thank you!
", "votes": null, "replies": [] }, { "id": 2552201, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "12/07/2023 09:33:38", "content": "Congratulations. Thanks for sharing your solution details. Very interesting approach indeed.
", "votes": null, "replies": [] }, { "id": 2553212, "author_name": "dongyk", "author_url": "", "post_date": "12/08/2023 04:54:59", "content": "I have no word. It's a great post processing!
", "votes": null, "replies": [ { "id": 2553215, "author_name": "dongyk", "author_url": "", "post_date": "12/08/2023 04:57:52", "content": "I have a problem to come up with what the right way to build input data is.
\nCould you explain why you selected the input data with both an onset and a wakeup?
This is a powerful technique to achieve top results in object detection and object segmentation Kaggle competitions!
\nWe want the model to locate where onset and wakeup is. So every train sample contains an onset and wakeup. (We cannot learn location if there is no onset wakeup). Therefore our dataloader, provides random 24 hour crops and every crop contains both onset and wakeup.
This model will beat other models and techniques for locating onset and wakeup. However this model can not determine existence. This model will think that every day contains an onset and wakeup. So we also need to build a second model (not described in my solution post). The second model job is a classifier that outputs 0 or 1 predicting when a 24 window has an onset and wakeup.
We train the second model with a different data loader. The data loader for the second model provides all types of random 24 hour crops. Some crops have onset wakeup, and some crops do not have onset wakeup. The second model is binary classifier that learns when onset wakeup is present or not.
\nThen our inference pipeline uses both models, our first model predicts onset wakeup for every night, i.e. 24 hours. And our second model removes false positives. Whenever our second model outputs prob < 0.2 for a night (i.e. 24 hour window), then we remove the first model prediction from submission.csv.
That's very logical approach.
\nAnd I really appreciate your long and detail answer. I'm so touched. :)
UPDATE I published the code for these two TensorFlow models here
", "votes": null, "replies": [] }, { "id": 2554642, "author_name": "aidlre001", "author_url": "", "post_date": "12/09/2023 09:17:28", "content": "Excellent work! Really learnt a lot from this. I should have stuck with my keras framework rather than going to learn pytorch. Now I know…
", "votes": null, "replies": [] }, { "id": 2554711, "author_name": "raihan150146", "author_url": "", "post_date": "12/09/2023 10:44:28", "content": "Thanks for Sharing
", "votes": null, "replies": [] }, { "id": 2558156, "author_name": "cdeotte", "author_url": "", "post_date": "12/12/2023 00:24:35", "content": "For reference, here are the 3 popular ways to create a target in this competition. All three performed well. From all three, we can compute a step and score for submission.csv:
\n
For reference, here are two Mel Spectrograms with 1D prediction and 2D prediction and ground truth:
\n
\n
Wanted to perform this data transformation but implementation and computational bottleneck was a problem. Had wanted to use the sleeping and waking data spectrograms as images to a 2D CNN but kept crashing. Was this ever feasible given the constraints of the competition? Still would like to implement it fully and compare.
", "votes": null, "replies": [ { "id": 2560603, "author_name": "cdeotte", "author_url": "", "post_date": "12/13/2023 20:13:27", "content": "Yes this was feasible. My final solution used this model to predict a score for each onset and wakeup and I added the score to the score from my Transformer and GRU to improve CV and LB. Also by itself, this model achieved good results.
An easy way to avoid memory error is to process data in chunks and save intermediate data to disk. We can do this both for train and infer.
\nFirst read Kaggle's train parquet in chunks and we iterate through every user and write each user to disk as a NumPy file. So for each user we have data shape (days, 17280, 4) where days is the number of 24 hour periods (per user), and 17280 is the number of steps in 24 hours. And 4 is anglez, enmo, and the two targets. (Also we begin each user at 3pm and end at 3pm and add padding to beginning and ending if necessary. That makes all the 24 hour windows the same).
Next we iterate through each of these NumPys. We load from disk then convert to image. Then save images to disk. The code below creates 2 channel images of size 256x1536 for each day for each user:
\nimport librosa\nCROP = 17_280; SRR = 17_280; HL = 12; HH = 256\n\n j,u enumerate(USERS):\n data = np.nan_to_num( np.load(f) )\n jj = []\n r range(len(data)):\n ii = []\n mel_spec = librosa.feature.melspectrogram(=data[r,:,0], =SRR, =2048, =HL, \n =HH, =0, =SRR//2, =2048//4)\n mel_spec_db = librosa.power_to_db(mel_spec, =np.max).astype(np.float32)\n ii.append(mel_spec_db)\n\n mel_spec = librosa.feature.melspectrogram(=data[r,:,1], =SRR, =2048, =HL, \n =HH, =0, =SRR//2, =2048//4)\n mel_spec_db = librosa.power_to_db(mel_spec, =np.max).astype(np.float32)\n ii.append(mel_spec_db)\n\n img2 = np.transpose( np.stack(ii), [1,2,0])\n jj.append(img2)\n\n jj = np.stack(jj)\n np.save(f,jj)\nThank you Chris. Wish I had had this earlier on. Gave up due to memory lags and crashes. I will work with this for future similar competitions. Once more, I really appreciate your hard work and taking the time to explain the secret dark arts of ML and DL in a succinct manner to us lost-in-the-woods folks.
", "votes": null, "replies": [] } ] }, { "id": 2560605, "author_name": "cdeotte", "author_url": "", "post_date": "12/13/2023 20:14:39", "content": "Also note that the highest scoring public notebook uses Mel Spectrograms (created by PyTorch) and provides code on GitHub. Discussion here
", "votes": null, "replies": [ { "id": 2560622, "author_name": "aidlre001", "author_url": "", "post_date": "12/13/2023 20:29:07", "content": "Ok Chris. Will check it out. I wonder if I doing this on TPUs would make that much difference in training and inference speed? Its the scoring time that gives me ulcers.
", "votes": null, "replies": [ { "id": 2560626, "author_name": "aidlre001", "author_url": "", "post_date": "12/13/2023 20:32:44", "content": "Oh I see. The librosa library makes it easier. Was trying to do it from scratch but never got a spectrogram plot good enough as yours. Must be some bug in my code.
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2559343, "author_name": "miraclejy", "author_url": "", "post_date": "12/12/2023 18:48:35", "content": "Congratulations! Such an excellent work. As a beginner I have really learned a lot from your solution. I am still a little bit confused about two details of your approach.
\nOne is that you set crops for 24 hours which is 17280 steps but your input for Transformer Model is 1440. So I'm wondering whether you are using the mean of 12 steps as one data point as input. May I also asked about the 80 features you used?
\nAnd the second one is about labels. It is a great idea to seperate \"onset\" and \"awake\" as two different labels and I noticed that some competitors applied a gaussian window around the event step. So did you use the same solution or just use original \"0\" and \"1\" labels?
\nThank you so much if you can reply to my comment.
", "votes": null, "replies": [ { "id": 2559395, "author_name": "cdeotte", "author_url": "", "post_date": "12/12/2023 19:32:07", "content": "Thank you
\n\n\nOne is that you set crops for 24 hours which is 17280 steps but your input for Transformer Model is 1440
\n
My CNN-Transformer-GRU uses an input of 1440. The data loader output just skips every 12th using the following Python indexing X_train[:,::12,:].
\n\nMay I also asked about the 80 features you used?
\n
I used the 79 features from this popular LGBM notebook here. And I added the additional feature minute%15 to learn the weird labeler bias. (I also built my own CatBoost with my own 200 features and my CatBoost scored LB 0.705 but my Transformer used those public notebook features).
\n\nSo did you use the same solution or just use original \"0\" and \"1\" labels?
\n
I used the original \"0\" and \"1\". Each of my two models here has output sequence length 1440 which is the number of minutes per day. The target for each day crop were 1439 zeros with 1 one where the actual target is. My loss is softmax multi-class crossentropy, so the model had no trouble learning the unique target \"1\" out of \"1440\" different classes. (I.e. the multiclass is the 1440 different minutes each day and the model finds the correct minute each day)
", "votes": null, "replies": [ { "id": 2559403, "author_name": "miraclejy", "author_url": "", "post_date": "12/12/2023 19:42:58", "content": "Got it. Thank you so much!
", "votes": null, "replies": [] } ] } ] }, { "id": 2559919, "author_name": "huoyeqianxun", "author_url": "", "post_date": "12/13/2023 06:29:30", "content": "Congratulations! Always can learn a lot from your work. Especially the Deep GRU model that only use 2 original features. Really impressive!
\nI'm trying to train your Deep GRU model, but I got gradient vanishing problem. I use exactly your 'build_model_2()' code, but If I call the \"build_model_2()\" function 5 times, maybe got gradient vanishing for 3 or 4 times.
\nI guess I must miss something somewhere, such as some special initializer? or some kind of random seed setting? Or other stuffs.
\nSo, will you public all your code? Then I can check where I'm going wrong.
Again, thanks a lot for the sharing of your excellent work!
", "votes": null, "replies": [ { "id": 2561855, "author_name": "cdeotte", "author_url": "", "post_date": "12/14/2023 22:02:57", "content": "Do you mean that you get gradient vanishing during training? If so, does your data loader output the correct train data? Every train sample must have X_train with shape (batch, 17280, 2) where the 2 features are anglez and enmo standardized (i.e minus mean divide by std using the mean and std of the entire train data). And the target is list of length two [y_train_1, y_train_2] where each are shape (batch, 1440) and each have exactly 1439 zeros and 1 one? (i.e. there is exactly 1 target in every sample. We do not use 24 hour windows that are missing target).
Thanks for your reply.
\nYes, I get gradient vanishing when I train the Deep-GRU model.
\nI checked many times, it seems my data loader satisfies all the demands that you described. But gradient vanishing still happens from time to time.
\nToday, I set global seed using \"tf.random.set_seed()\", and set the operation seeds of initializers for the Conv1d, GRU, Dense layers in the model. so that the initial weights in the model will be the same every time the model is built.
\nLuckily, it works. Gradient vanishing won't happen.
\nMaybe there is still something wrong in my data loader, so that it demand some special initial weights in the model.
\nAnyway, I can avoid gradient vanishing for now. Hope the unknown bug doesn't affect the training result.
\n😅😅😅
Congratulations! Very interesting approach indeed
", "votes": null, "replies": [] }, { "id": 2568995, "author_name": "valbercesar1", "author_url": "", "post_date": "12/21/2023 02:43:34", "content": "Really interesting approach. Your solution was like a Random Forrest, implemented each ML-Code separatelly, instead decision trees.. Haha!
\nI'm completing this challenge (late submission), but in another way! Congrats bro!