First of all, I would like to thank @tubotubo for sharing your high-quality code. I am going to share my solution.
\nThis is my first post on a solution, so I apologize if I'm being rude in any way. Although the content is poor, I hope it will be of help to you.
Cleaning features using FFT: Referring to cmi-sleep-detection-fast-fourier-transformation, data was cleaned using different thresholds (98.75, 99.0, 99.5, 99.75, 99.9) for enmo and anglez. This generated FFT-based features (e.g. fft_9875).
\nFor cross-validation, I use a 5-fold GroupKFold.
\nI built seven different models based on @tubotubo's code , each using a different feature set and network architecture (LSTM, GRU, UNet, Transformer, etc.).
| model | \nCV | \nPubic (5-fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| FeatureExtractor(LSTM(dim=64)+GRU(dim=64)) + UNet + UNet1DDecoder with no fft feature | \n0.74373 | \n0.746 | \n0.791 | \n
| FeatureExtractor(LSTM(dim=64)+GRU(dim=64)) + UNet + UNet1DDecoder with fft_9900, fft_9950, fft_9975, fft_9990 | \n0.75250 | \n0.727 | \n0.791 | \n
| FeatureExtractor(LSTM(dim=64)+GRU(dim=64)) + UNet + UNet1DDecoder with fft_9875 | \n0.76012 | \n0.747 | \n0.795 | \n
| FeatureExtractor(TransformerFeatureExtractor(dim=64)) + UNet + UNet1DDecoder with fft_9875 | \n0.74572 | \n\n | \n |
| FeatureExtractor(LSTM(dim=128)+GRU(dim=128)) + UNet + UNet1DDecoder with fft_9875 | \n0.76400 | \n0.739 | \n0.801 | \n
| FeatureExtractor(LSTM(dim=128)+GRU(dim=128)) + UNet + UNet1DDecoder with fft_9900 | \n0.76208 | \n0.735 | \n0.795 | \n
| FeatureExtractor(CNNSpectrogram(dim=128)) + UNet + UNet1DDecoder with fft_9875 | \n0.74933 | \n\n | \n |
Cleaning features using FFT had a big effect on CV, but when I opened the lid, it didn't seem to have a big effect on the private score.
\nBy increasing the number of dimensions during feature extraction from 64 to 128, the accuracy of CV and private score was significantly improved.
Two-step ensemble approach:
\nIn the first step, predictions from seven models were ensembled using three different methods: Optuna, Nelder-Mead, and Hill Climbing.
\nNext, in the second step, these three types of ensemble results were further ensembled using Optuna to obtain the final results.
\nThis approach yielded good results in both cross-validation and private scores. The first and second stages of Private were both 0.802, but the second stage was slightly better, but it may not have been necessary to go this far.
| Ensemble method | \nCV | \nPubic (5-fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| optuna | \n0.79038 | \n0.746 | \n0.802 | \n
| nelder-mead | \n0.78951 | \n\n | \n |
| optuna | \n0.78801 | \n\n | \n |
| Ensemble method | \nCV | \nPubic (5-fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| optuna | \n0.79049 | \n0.746 | \n0.802 | \n
Finally, thank you for organizing the competition. It was a very challenging competition and I learned a lot.
\nMoreover, this is my second silver medal. I want to continue working hard to become a kaggle master. Thank you everyone for letting me learn so much. I will continue to do my best.
", "messages": [ { "id": "2550568", "postDate": "12/06/2023 05:38:30", "content": "First of all, I would like to thank @tubotubo for sharing your high-quality code. I am going to share my solution.
\nThis is my first post on a solution, so I apologize if I'm being rude in any way. Although the content is poor, I hope it will be of help to you.
Cleaning features using FFT: Referring to cmi-sleep-detection-fast-fourier-transformation, data was cleaned using different thresholds (98.75, 99.0, 99.5, 99.75, 99.9) for enmo and anglez. This generated FFT-based features (e.g. fft_9875).
\nFor cross-validation, I use a 5-fold GroupKFold.
\nI built seven different models based on @tubotubo's code , each using a different feature set and network architecture (LSTM, GRU, UNet, Transformer, etc.).
| model | \nCV | \nPubic (5-fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| FeatureExtractor(LSTM(dim=64)+GRU(dim=64)) + UNet + UNet1DDecoder with no fft feature | \n0.74373 | \n0.746 | \n0.791 | \n
| FeatureExtractor(LSTM(dim=64)+GRU(dim=64)) + UNet + UNet1DDecoder with fft_9900, fft_9950, fft_9975, fft_9990 | \n0.75250 | \n0.727 | \n0.791 | \n
| FeatureExtractor(LSTM(dim=64)+GRU(dim=64)) + UNet + UNet1DDecoder with fft_9875 | \n0.76012 | \n0.747 | \n0.795 | \n
| FeatureExtractor(TransformerFeatureExtractor(dim=64)) + UNet + UNet1DDecoder with fft_9875 | \n0.74572 | \n\n | \n |
| FeatureExtractor(LSTM(dim=128)+GRU(dim=128)) + UNet + UNet1DDecoder with fft_9875 | \n0.76400 | \n0.739 | \n0.801 | \n
| FeatureExtractor(LSTM(dim=128)+GRU(dim=128)) + UNet + UNet1DDecoder with fft_9900 | \n0.76208 | \n0.735 | \n0.795 | \n
| FeatureExtractor(CNNSpectrogram(dim=128)) + UNet + UNet1DDecoder with fft_9875 | \n0.74933 | \n\n | \n |
Cleaning features using FFT had a big effect on CV, but when I opened the lid, it didn't seem to have a big effect on the private score.
\nBy increasing the number of dimensions during feature extraction from 64 to 128, the accuracy of CV and private score was significantly improved.
Two-step ensemble approach:
\nIn the first step, predictions from seven models were ensembled using three different methods: Optuna, Nelder-Mead, and Hill Climbing.
\nNext, in the second step, these three types of ensemble results were further ensembled using Optuna to obtain the final results.
\nThis approach yielded good results in both cross-validation and private scores. The first and second stages of Private were both 0.802, but the second stage was slightly better, but it may not have been necessary to go this far.
| Ensemble method | \nCV | \nPubic (5-fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| optuna | \n0.79038 | \n0.746 | \n0.802 | \n
| nelder-mead | \n0.78951 | \n\n | \n |
| optuna | \n0.78801 | \n\n | \n |
| Ensemble method | \nCV | \nPubic (5-fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| optuna | \n0.79049 | \n0.746 | \n0.802 | \n
Finally, thank you for organizing the competition. It was a very challenging competition and I learned a lot.
\nMoreover, this is my second silver medal. I want to continue working hard to become a kaggle master. Thank you everyone for letting me learn so much. I will continue to do my best.
", "rawMarkdown": "First of all, I would like to thank [@tubotubo](https://www.kaggle.com/tubotubo) for sharing your high-quality code. I am going to share my solution.\nThis is my first post on a solution, so I apologize if I'm being rude in any way. Although the content is poor, I hope it will be of help to you.\n\n### 1. Data preprocessing\n- Null Value Removal: Removed rows with null timestamp values in the dataset to prevent misinterpretation of the analysis and reduce uncertainty in model training.\n- Event count consistency check: Filtered out unmatched data at the beginning ('onset') and end ('wakeup') of events, increasing data integrity and analysis reliability.\n\n\n### 2. Feature Engineering\nCleaning features using FFT: Referring to [cmi-sleep-detection-fast-fourier-transformation](https://www.kaggle.com/code/jjinho/cmi-sleep-detection-fast-fourier-transformation), data was cleaned using different thresholds (98.75, 99.0, 99.5, 99.75, 99.9) for enmo and anglez. This generated FFT-based features (e.g. fft_9875).\n\n\n### 3. Model\nFor cross-validation, I use a 5-fold GroupKFold.\nI built seven different models based on [@tubotubo's code](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940) , each using a different feature set and network architecture (LSTM, GRU, UNet, Transformer, etc.).\n\n| model | CV | Pubic (5-fold) | Private (5-fold)|\n| --- | --- | --- | --- |\n| FeatureExtractor(LSTM(dim=64)+GRU(dim=64)) + UNet + UNet1DDecoder with no fft feature| 0.74373|0.746|0.791|\n| FeatureExtractor(LSTM(dim=64)+GRU(dim=64)) + UNet + UNet1DDecoder with fft_9900, fft_9950, fft_9975, fft_9990| **0.75250**|0.727|0.791|\n| FeatureExtractor(LSTM(dim=64)+GRU(dim=64)) + UNet + UNet1DDecoder with fft_9875| 0.76012|0.747|0.795|\n| FeatureExtractor(TransformerFeatureExtractor(dim=64)) + UNet + UNet1DDecoder with fft_9875| 0.74572| | |\n| FeatureExtractor(LSTM(dim=128)+GRU(dim=128)) + UNet + UNet1DDecoder with fft_9875| **0.76400**|0.739|0.801|\n| FeatureExtractor(LSTM(dim=128)+GRU(dim=128)) + UNet + UNet1DDecoder with fft_9900| 0.76208|0.735|0.795|\n| FeatureExtractor(CNNSpectrogram(dim=128)) + UNet + UNet1DDecoder with fft_9875| 0.74933| | |\n\n\n\nCleaning features using FFT had a big effect on CV, but when I opened the lid, it didn't seem to have a big effect on the private score.\nBy increasing the number of dimensions during feature extraction from 64 to 128, the accuracy of CV and private score was significantly improved.\n\n\n### 4. Model Ensemble\nTwo-step ensemble approach: \nIn the first step, predictions from seven models were ensembled using three different methods: Optuna, Nelder-Mead, and Hill Climbing.\nNext, in the second step, these three types of ensemble results were further ensembled using Optuna to obtain the final results.\nThis approach yielded good results in both cross-validation and private scores. The first and second stages of Private were both 0.802, but the second stage was slightly better, but it may not have been necessary to go this far.\n\n### first stage\n| Ensemble method | CV | Pubic (5-fold) | Private (5-fold)|\n| --- | --- | --- | --- |\n| optuna| 0.79038|0.746|0.802|\n| nelder-mead| 0.78951| | |\n| optuna| 0.78801 | | |\n\n### second stage\n| Ensemble method | CV | Pubic (5-fold) | Private (5-fold)|\n| --- | --- | --- | --- |\n| optuna| 0.79049|0.746|0.802|\n\n\n### 5. Approaches that didn't work\n- Applying Focal Loss: The cross-validation score improved when using Focal Loss, but an error occurred during submission.\n- Applying asymmetric Gaussian and exponentially decaying distributions: I applied asymmetrically Gaussian and exponentially decaying distributions to the labels, but these approaches did not contribute to improved performance.\n- Additional feature engineering: I tried additional features such as moving average, standard deviation, difference signal, cumulative sum, and autocorrelation, but these also did not improve performance.\n- Using DeBERTa-v3-small: I used DeBERTa-v3-small for feature extraction, but learning did not proceed well and I did not get the expected results.\n\n\n\nFinally, thank you for organizing the competition. It was a very challenging competition and I learned a lot.\n\nMoreover, this is my second silver medal. I want to continue working hard to become a kaggle master. Thank you everyone for letting me learn so much. I will continue to do my best.", "votes": null }, { "id": "2995316", "postDate": "09/22/2024 06:58:52", "content": "thanks for sharing
", "rawMarkdown": "thanks for sharing", "votes": null } ], "comments": [ { "id": 2995316, "author_name": "samanfatima7", "author_url": "", "post_date": "09/22/2024 06:58:52", "content": "thanks for sharing
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2550568": "First of all, I would like to thank [@tubotubo](https://www.kaggle.com/tubotubo) for sharing your high-quality code. I am going to share my solution.\nThis is my first post on a solution, so I apologize if I'm being rude in any way. Although the content is poor, I hope it will be of help to you.\n\n### 1. Data preprocessing\n- Null Value Removal: Removed rows with null timestamp values in the dataset to prevent misinterpretation of the analysis and reduce uncertainty in model training.\n- Event count consistency check: Filtered out unmatched data at the beginning ('onset') and end ('wakeup') of events, increasing data integrity and analysis reliability.\n\n\n### 2. Feature Engineering\nCleaning features using FFT: Referring to [cmi-sleep-detection-fast-fourier-transformation](https://www.kaggle.com/code/jjinho/cmi-sleep-detection-fast-fourier-transformation), data was cleaned using different thresholds (98.75, 99.0, 99.5, 99.75, 99.9) for enmo and anglez. This generated FFT-based features (e.g. fft_9875).\n\n\n### 3. Model\nFor cross-validation, I use a 5-fold GroupKFold.\nI built seven different models based on [@tubotubo's code](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/452940) , each using a different feature set and network architecture (LSTM, GRU, UNet, Transformer, etc.).\n\n| model | CV | Pubic (5-fold) | Private (5-fold)|\n| --- | --- | --- | --- |\n| FeatureExtractor(LSTM(dim=64)+GRU(dim=64)) + UNet + UNet1DDecoder with no fft feature| 0.74373|0.746|0.791|\n| FeatureExtractor(LSTM(dim=64)+GRU(dim=64)) + UNet + UNet1DDecoder with fft_9900, fft_9950, fft_9975, fft_9990| **0.75250**|0.727|0.791|\n| FeatureExtractor(LSTM(dim=64)+GRU(dim=64)) + UNet + UNet1DDecoder with fft_9875| 0.76012|0.747|0.795|\n| FeatureExtractor(TransformerFeatureExtractor(dim=64)) + UNet + UNet1DDecoder with fft_9875| 0.74572| | |\n| FeatureExtractor(LSTM(dim=128)+GRU(dim=128)) + UNet + UNet1DDecoder with fft_9875| **0.76400**|0.739|0.801|\n| FeatureExtractor(LSTM(dim=128)+GRU(dim=128)) + UNet + UNet1DDecoder with fft_9900| 0.76208|0.735|0.795|\n| FeatureExtractor(CNNSpectrogram(dim=128)) + UNet + UNet1DDecoder with fft_9875| 0.74933| | |\n\n\n\nCleaning features using FFT had a big effect on CV, but when I opened the lid, it didn't seem to have a big effect on the private score.\nBy increasing the number of dimensions during feature extraction from 64 to 128, the accuracy of CV and private score was significantly improved.\n\n\n### 4. Model Ensemble\nTwo-step ensemble approach: \nIn the first step, predictions from seven models were ensembled using three different methods: Optuna, Nelder-Mead, and Hill Climbing.\nNext, in the second step, these three types of ensemble results were further ensembled using Optuna to obtain the final results.\nThis approach yielded good results in both cross-validation and private scores. The first and second stages of Private were both 0.802, but the second stage was slightly better, but it may not have been necessary to go this far.\n\n### first stage\n| Ensemble method | CV | Pubic (5-fold) | Private (5-fold)|\n| --- | --- | --- | --- |\n| optuna| 0.79038|0.746|0.802|\n| nelder-mead| 0.78951| | |\n| optuna| 0.78801 | | |\n\n### second stage\n| Ensemble method | CV | Pubic (5-fold) | Private (5-fold)|\n| --- | --- | --- | --- |\n| optuna| 0.79049|0.746|0.802|\n\n\n### 5. Approaches that didn't work\n- Applying Focal Loss: The cross-validation score improved when using Focal Loss, but an error occurred during submission.\n- Applying asymmetric Gaussian and exponentially decaying distributions: I applied asymmetrically Gaussian and exponentially decaying distributions to the labels, but these approaches did not contribute to improved performance.\n- Additional feature engineering: I tried additional features such as moving average, standard deviation, difference signal, cumulative sum, and autocorrelation, but these also did not improve performance.\n- Using DeBERTa-v3-small: I used DeBERTa-v3-small for feature extraction, but learning did not proceed well and I did not get the expected results.\n\n\n\nFinally, thank you for organizing the competition. It was a very challenging competition and I learned a lot.\n\nMoreover, this is my second silver medal. I want to continue working hard to become a kaggle master. Thank you everyone for letting me learn so much. I will continue to do my best.", "2995316": "thanks for sharing" }, "source": "meta" }