![\"Model](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1265131%2Fad78d36028ed6dc7ab1d3e79631b0172%2FFireShot%20Capture%20019%20-%20Online%20FlowChart%20%20Diagrams%20Editor%20-%20Mermaid%20Live%20Editor%20-%20mermaid.live%20(1).png?generation=1701841137052567&alt=media\")
Firstly thank you to my teammate @ryotayoshinobu for the amazing performance. Please read his part of the solution.
\nI am really glad I teamed up with him and learnt a lot
A big thanks to the organizers as well for such an interesting competition. A really tough one.
\nYou can find the solution code here:
\nhttps://github.com/nikhilmishradevelop/kaggle-child-mind-institute-detect-sleep-states
\nValidation Strategy: GroupKFold on Series Ids
\nModel Inputs: 17280 x n_features length sequences as input (17280 = 12 steps_per_minute x 60_minutes * 24 hours)
\nModel Outputs: 17280 x 2 (one for onset and other for wakeup)
\nModel Type: Regression Model
\nLoss Type: Cubic Loss i.e. abs(y_true-y_pred)**3
The last remaining part of the sequences with length < 17280 were padded to make them equal to 17280
\nMy solution is a modified UNET (averaged over 4 models, 2 LSTMs and 2 GRUs) using normalized Gaussian targets similar to what @tolgadincer described. A big thanks to him for sharing a good method, early on in this competition.
\nGood Features help in faster convergence and much better scores, so adding good features were important (Read penguin's solution for some other good features)
\nMy final 4 model ensemble results:
\nCV: 0.828
\nPublic LB: 0.789
\nPrivate LB: 0.841
Since 17280 is a very long sequence length (makes training very slow and harder to train), we reduce the sequence length by patching it.
\nInput: 17280 * n_features (34)
\nPatch Size: (Used different patch sizes 3, 4, 5, or 6 in different models).
Modified Sequence Length: 17280 to 17280//Patch_Size sequence length
\nModified Feature Size: k * 4 * num_features (where k is a Dense layer output dim)
\nUNET Encoder -> Bottleneck -> Transformer -> GRU or LSTM -> Unet Decoder
\nEach layer of UNET encoder had concatenated connections to the 1st layer
\nInitial Output size: 17280//Patch_Size, 2 * Patch_size
\nReshaped Output Size: (17280, 2)
\nInitially I was doing just doing simple peak detection when I was competing solo.
\nI wanted to, but could not formulate a good way to rerank model predictions using lightgbm, but @kmat2019 is a great read for that.
\nThanks to @ryotayoshinobu , I started applying NMS, and finally experimented with a kind of WBF algorithm and made it work. This WBF algorithm gave a score of 0.79 to 0.793 on last day.
\nBut it was worse by 0.001 in our private LB lol, so WBF hurt.
\nWBF Working
\nInitialization and Convolution: The function starts by applying convolution with the specified convolution_kernel to smooth the data.
Peak Detection Loop: It iteratively searches for peaks in the data. The loop runs until the maximum count (max_count) is reached or the peak value falls below a threshold (max_thresh).
Adaptive Window and Weight Calculation:
\ncurr_max_power) and a distance parameter (k_dist).section_weight_method (logarithmic or linear), weights are calculated for each section of the data around the peak.Weighted Average and Score Calculation: For each detected peak, a weighted average is calculated to determine the score of the peak.
\nThis is done by considering the peak and k of its neighbours. This score is influenced by the method of weight calculation and other hyperparameters like log_base, log_scale, and weight_coeff.
Suppression and Updating Predictions:
\noverlap_coeff and preds_reduction_power.I will share the implementation of the WBF part code, you can see if tuning these hyperparameters help in your models too. This accepts predictions per series id.
\n(I found hyperparameters by a mix of manual and automated hyperparamter tuning approach)
\n ():\n k_dist = hyperparams[]\n log_base = hyperparams[]\n log_scale = hyperparams[]\n curr_max_power = hyperparams[]\n weight_coeff = hyperparams[]\n convolution_kernel = hyperparams[]\n section_weight_method = hyperparams[]\n preds_reduction_power = hyperparams[]\n overlap_coeff = hyperparams[]\n min_distance = hyperparams[]\n\n preds = preds_orig.copy()\n preds = np.convolve(preds, convolution_kernel, mode=)\n\n count = \n indices = []\n scores = []\n\n count < max_count:\n curr_max_idx = np.argmax(preds)\n curr_max = preds[curr_max_idx]\n\n curr_max < max_thresh:\n \n\n k = (k_dist - (min_distance, (curr_max**curr_max_power)))\n\n start_idx = (curr_max_idx - k, )\n end_idx = (curr_max_idx + k + , (preds))\n\n section = preds[start_idx:end_idx]\n\n \n distances = np.(np.arange((section)) - k)\n section_weight_method == :\n weights = / (log_base ** (distances / (k * log_scale)))\n section_weight_method == :\n weights = - (distances / k) * weight_coeff\n \n\n weighted_avg = np.(section * weights) / np.(weights)\n\n scores.append(weighted_avg)\n indices.append(curr_max_idx)\n\n preds[start_idx:end_idx] *= (( - weights * overlap_coeff))**preds_reduction_power\n\n count += \n\n indices, scores\nFinal ensemble was a blended weight of my and Penguin's subs regression postprocessed using WBF
\nFinal_Sub = WBF(Penguins_Predictions * 0.25 + Nikhil's Predictions*0.75)
\nSince our predictions had slightly different scales of regression targets, Penguin's Predictions were first scaled by using a simple power transform i.e Penguin's_Predictions ** 0.7
\nEnsemble CV: 0.835
\nEnsemble Public LB: 0.793
\nEnsemble Private LB: 0.845
P.S: This competition is the first time I did not use gradient boosting, even after I knew it worked so well, me being a tabular guy. I mistake I guess. I love lightgbm and will definitely use one in the next competition.
\nCongratulations to all the the top performers, it was a good fight, till it lasted :)
", "messages": [ { "id": "2550489", "postDate": "12/06/2023 04:20:15", "content": "Firstly thank you to my teammate @ryotayoshinobu for the amazing performance. Please read his part of the solution.
\nI am really glad I teamed up with him and learnt a lot
A big thanks to the organizers as well for such an interesting competition. A really tough one.
\nYou can find the solution code here:
\nhttps://github.com/nikhilmishradevelop/kaggle-child-mind-institute-detect-sleep-states
\nValidation Strategy: GroupKFold on Series Ids
\nModel Inputs: 17280 x n_features length sequences as input (17280 = 12 steps_per_minute x 60_minutes * 24 hours)
\nModel Outputs: 17280 x 2 (one for onset and other for wakeup)
\nModel Type: Regression Model
\nLoss Type: Cubic Loss i.e. abs(y_true-y_pred)**3
The last remaining part of the sequences with length < 17280 were padded to make them equal to 17280
\nMy solution is a modified UNET (averaged over 4 models, 2 LSTMs and 2 GRUs) using normalized Gaussian targets similar to what @tolgadincer described. A big thanks to him for sharing a good method, early on in this competition.
\nGood Features help in faster convergence and much better scores, so adding good features were important (Read penguin's solution for some other good features)
\nMy final 4 model ensemble results:
\nCV: 0.828
\nPublic LB: 0.789
\nPrivate LB: 0.841
Since 17280 is a very long sequence length (makes training very slow and harder to train), we reduce the sequence length by patching it.
\nInput: 17280 * n_features (34)
\nPatch Size: (Used different patch sizes 3, 4, 5, or 6 in different models).
Modified Sequence Length: 17280 to 17280//Patch_Size sequence length
\nModified Feature Size: k * 4 * num_features (where k is a Dense layer output dim)
\nUNET Encoder -> Bottleneck -> Transformer -> GRU or LSTM -> Unet Decoder
\nEach layer of UNET encoder had concatenated connections to the 1st layer
\nInitial Output size: 17280//Patch_Size, 2 * Patch_size
\nReshaped Output Size: (17280, 2)
\nInitially I was doing just doing simple peak detection when I was competing solo.
\nI wanted to, but could not formulate a good way to rerank model predictions using lightgbm, but @kmat2019 is a great read for that.
\nThanks to @ryotayoshinobu , I started applying NMS, and finally experimented with a kind of WBF algorithm and made it work. This WBF algorithm gave a score of 0.79 to 0.793 on last day.
\nBut it was worse by 0.001 in our private LB lol, so WBF hurt.
\nWBF Working
\nInitialization and Convolution: The function starts by applying convolution with the specified convolution_kernel to smooth the data.
Peak Detection Loop: It iteratively searches for peaks in the data. The loop runs until the maximum count (max_count) is reached or the peak value falls below a threshold (max_thresh).
Adaptive Window and Weight Calculation:
\ncurr_max_power) and a distance parameter (k_dist).section_weight_method (logarithmic or linear), weights are calculated for each section of the data around the peak.Weighted Average and Score Calculation: For each detected peak, a weighted average is calculated to determine the score of the peak.
\nThis is done by considering the peak and k of its neighbours. This score is influenced by the method of weight calculation and other hyperparameters like log_base, log_scale, and weight_coeff.
Suppression and Updating Predictions:
\noverlap_coeff and preds_reduction_power.I will share the implementation of the WBF part code, you can see if tuning these hyperparameters help in your models too. This accepts predictions per series id.
\n(I found hyperparameters by a mix of manual and automated hyperparamter tuning approach)
\n ():\n k_dist = hyperparams[]\n log_base = hyperparams[]\n log_scale = hyperparams[]\n curr_max_power = hyperparams[]\n weight_coeff = hyperparams[]\n convolution_kernel = hyperparams[]\n section_weight_method = hyperparams[]\n preds_reduction_power = hyperparams[]\n overlap_coeff = hyperparams[]\n min_distance = hyperparams[]\n\n preds = preds_orig.copy()\n preds = np.convolve(preds, convolution_kernel, mode=)\n\n count = \n indices = []\n scores = []\n\n count < max_count:\n curr_max_idx = np.argmax(preds)\n curr_max = preds[curr_max_idx]\n\n curr_max < max_thresh:\n \n\n k = (k_dist - (min_distance, (curr_max**curr_max_power)))\n\n start_idx = (curr_max_idx - k, )\n end_idx = (curr_max_idx + k + , (preds))\n\n section = preds[start_idx:end_idx]\n\n \n distances = np.(np.arange((section)) - k)\n section_weight_method == :\n weights = / (log_base ** (distances / (k * log_scale)))\n section_weight_method == :\n weights = - (distances / k) * weight_coeff\n \n\n weighted_avg = np.(section * weights) / np.(weights)\n\n scores.append(weighted_avg)\n indices.append(curr_max_idx)\n\n preds[start_idx:end_idx] *= (( - weights * overlap_coeff))**preds_reduction_power\n\n count += \n\n indices, scores\nFinal ensemble was a blended weight of my and Penguin's subs regression postprocessed using WBF
\nFinal_Sub = WBF(Penguins_Predictions * 0.25 + Nikhil's Predictions*0.75)
\nSince our predictions had slightly different scales of regression targets, Penguin's Predictions were first scaled by using a simple power transform i.e Penguin's_Predictions ** 0.7
\nEnsemble CV: 0.835
\nEnsemble Public LB: 0.793
\nEnsemble Private LB: 0.845
P.S: This competition is the first time I did not use gradient boosting, even after I knew it worked so well, me being a tabular guy. I mistake I guess. I love lightgbm and will definitely use one in the next competition.
\nCongratulations to all the the top performers, it was a good fight, till it lasted :)
", "rawMarkdown": "Firstly thank you to my teammate @ryotayoshinobu for the amazing performance. Please read his part of the [solution](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/459597).\nI am really glad I teamed up with him and learnt a lot\n\nA big thanks to the organizers as well for such an interesting competition. A really tough one.\n\nYou can find the solution code here:\n\nhttps://github.com/nikhilmishradevelop/kaggle-child-mind-institute-detect-sleep-states\n\n# Solution Summary\n\n**Validation Strategy**: GroupKFold on Series Ids\n\n**Model Inputs**: 17280 x n_features length sequences as input (17280 = 12 steps_per_minute x 60_minutes * 24 hours)\n**Model Outputs**: 17280 x 2 (one for onset and other for wakeup)\n**Model Type**: Regression Model\n**Loss Type**: Cubic Loss i.e. abs(y_true-y_pred)**3\n\nThe last remaining part of the sequences with length < 17280 were padded to make them equal to 17280\n\nMy solution is a modified UNET (averaged over 4 models, 2 LSTMs and 2 GRUs) using normalized Gaussian targets similar to what @tolgadincer described. A big thanks to him for sharing a good method, early on in this competition.\n\n## Features:\n\n1. Original Sequence Features: Enmo, Anglez\n2. TimeStamp Features: Hour and Weekday\n3. Derived Sequence Features: Anglez difference, Enmo Difference, HDCZA features etc)\n\nGood Features help in faster convergence and much better scores, so adding good features were important (Read penguin's solution for some other good features)\n\n**My final 4 model ensemble results**: \n\n**CV**: 0.828\n**Public LB**: 0.789\n**Private LB**: 0.841\n\n\n# **Model Architecture**\n\n\n.png?generation=1701841137052567&alt=media)\n\n\n##Patching to reduce sequence length\n\nSince 17280 is a very long sequence length (makes training very slow and harder to train), we reduce the sequence length by patching it.\n\n**Input:** 17280 * n_features (34)\n**Patch Size:** (Used different patch sizes 3, 4, 5, or 6 in different models).\n\n**Modified Sequence Length:** 17280 to 17280//Patch_Size sequence length\n\n**Modified Feature Size:** k * 4 * num_features (where k is a Dense layer output dim)\n\n## Modified Unet Part\n\n***UNET Encoder -> Bottleneck -> Transformer -> GRU or LSTM -> Unet Decoder***\n\nEach layer of UNET encoder had concatenated connections to the 1st layer\n\n**Initial Output size:** 17280//Patch_Size, 2 * Patch_size\n\n**Reshaped Output Size:** (17280, 2)\n\n\n\n\n# Post Processing\n\nInitially I was doing just doing simple peak detection when I was competing solo.\n\nI wanted to, but could not formulate a good way to rerank model predictions using lightgbm, but @kmat2019 is a great read for that.\n\nThanks to @ryotayoshinobu , I started applying NMS, and finally experimented with a kind of WBF algorithm and made it work. This WBF algorithm gave a score of 0.79 to 0.793 on last day.\n\nBut it was worse by 0.001 in our private LB lol, so WBF hurt.\n\nWBF Working\n\n1. **Initialization and Convolution**: The function starts by applying convolution with the specified `convolution_kernel` to smooth the data.\n\n2. **Peak Detection Loop**: It iteratively searches for peaks in the data. The loop runs until the maximum count (`max_count`) is reached or the peak value falls below a threshold (`max_thresh`).\n\n3. **Adaptive Window and Weight Calculation**:\n - A dynamic window size around each peak is determined based on the current maximum value's power (`curr_max_power`) and a distance parameter (`k_dist`).\n - Depending on the `section_weight_method` (logarithmic or linear), weights are calculated for each section of the data around the peak.\n\n4. **Weighted Average and Score Calculation**: For each detected peak, a weighted average is calculated to determine the score of the peak. \nThis is done by considering the peak and k of its neighbours. This score is influenced by the method of weight calculation and other hyperparameters like `log_base`, `log_scale`, and `weight_coeff`.\n\n5. **Suppression and Updating Predictions**:\n - After each peak is detected, the function suppresses the neighboring values to avoid detecting the same peak multiple times. This is controlled by `overlap_coeff` and `preds_reduction_power`.\n - The indices and scores of detected peaks are stored and returned after the loop completes.\n\n\n\nI will share the implementation of the WBF part code, you can see if tuning these hyperparameters help in your models too. This accepts predictions per series id.\n\n(I found hyperparameters by a mix of manual and automated hyperparamter tuning approach)\n\n```\ndef wbf_nikhil(preds_orig, max_thresh=0.1, max_count=700, hyperparams=None):\n k_dist = hyperparams['k_dist']\n log_base = hyperparams['log_base']\n log_scale = hyperparams['log_scale']\n curr_max_power = hyperparams['curr_max_power']\n weight_coeff = hyperparams['weight_coeff']\n convolution_kernel = hyperparams['convolution_kernel']\n section_weight_method = hyperparams['section_weight_method']\n preds_reduction_power = hyperparams['preds_reduction_power']\n overlap_coeff = hyperparams['overlap_coeff']\n min_distance = hyperparams['min_distance']\n \n preds = preds_orig.copy()\n preds = np.convolve(preds, convolution_kernel, mode='same')\n\n count = 0\n indices = []\n scores = []\n\n while count < max_count:\n curr_max_idx = np.argmax(preds)\n curr_max = preds[curr_max_idx]\n\n if curr_max < max_thresh:\n break\n\n k = int(k_dist - max(min_distance, (curr_max**curr_max_power)))\n\n start_idx = max(curr_max_idx - k, 0)\n end_idx = min(curr_max_idx + k + 1, len(preds))\n\n section = preds[start_idx:end_idx]\n\n # Different weight calculation methods\n distances = np.abs(np.arange(len(section)) - k)\n if section_weight_method == 'logarithmic':\n weights = 1 / (log_base ** (distances / (k * log_scale)))\n elif section_weight_method == 'linear':\n weights = 1 - (distances / k) * weight_coeff\n # Add more methods as needed\n\n weighted_avg = np.sum(section * weights) / np.sum(weights)\n\n scores.append(weighted_avg)\n indices.append(curr_max_idx)\n\n preds[start_idx:end_idx] *= ((1 - weights * overlap_coeff))**preds_reduction_power\n\n count += 1\n\n return indices, scores\n```\n\n## Final Ensemble\n\n\nFinal ensemble was a blended weight of my and Penguin's subs regression postprocessed using WBF\n\nFinal_Sub = WBF(Penguins_Predictions * 0.25 + Nikhil's Predictions*0.75)\n\nSince our predictions had slightly different scales of regression targets, Penguin's Predictions were first scaled by using a simple power transform i.e Penguin's_Predictions ** 0.7\n\n**Ensemble CV:** 0.835\n**Ensemble Public LB:** 0.793\n**Ensemble Private LB:** 0.845\n\n\nP.S: This competition is the first time I did not use gradient boosting, even after I knew it worked so well, me being a tabular guy. I mistake I guess. I love lightgbm and will definitely use one in the next competition.\n\nCongratulations to all the the top performers, it was a good fight, till it lasted :)", "votes": null }, { "id": "2550672", "postDate": "12/06/2023 07:51:23", "content": "A big congratulations bhai
", "rawMarkdown": "A big congratulations bhai", "votes": null }, { "id": "2551529", "postDate": "12/06/2023 18:27:22", "content": "how u understand what layers should u use in your model?
", "rawMarkdown": "how u understand what layers should u use in your model?", "votes": null }, { "id": "2551748", "postDate": "12/06/2023 23:45:50", "content": "Great solution writeup @nikhilmishradev Thank you for sharing! Additional thanks for sharing the WBF implementation!
", "rawMarkdown": "Great solution writeup @nikhilmishradev Thank you for sharing! Additional thanks for sharing the WBF implementation!", "votes": null }, { "id": "2553913", "postDate": "12/08/2023 16:01:23", "content": "more of a combination of standard practices, reading papers to see what works best, and a bit of intuition with hit and trial. Usually you would want to start with a standard practice, like UNet for segmentation like tasks, and build upon it
", "rawMarkdown": "more of a combination of standard practices, reading papers to see what works best, and a bit of intuition with hit and trial. Usually you would want to start with a standard practice, like UNet for segmentation like tasks, and build upon it", "votes": null }, { "id": "2556803", "postDate": "12/11/2023 01:56:09", "content": "Congratulations @nikhilmishradev @ryotayoshinobu ! When you two teamed up, i knew you would make a formidable team because you two has the best CV LB single model scores in the CV LB discussion post.
\nCongratulations on strong finish achieving cash gold 4th place!
", "rawMarkdown": "Congratulations @nikhilmishradev @ryotayoshinobu ! When you two teamed up, i knew you would make a formidable team because you two has the best CV LB single model scores in the CV LB discussion post.\n\nCongratulations on strong finish achieving cash gold 4th place!", "votes": null }, { "id": "2557123", "postDate": "12/11/2023 08:10:20", "content": "Thank you @cdeotte , would definitely want to team up with a legend like you in some competition
", "rawMarkdown": "Thank you @cdeotte , would definitely want to team up with a legend like you in some competition", "votes": null } ], "comments": [ { "id": 2550672, "author_name": "nasere", "author_url": "", "post_date": "12/06/2023 07:51:23", "content": "A big congratulations bhai
", "votes": null, "replies": [] }, { "id": 2551529, "author_name": "yanlukianchik", "author_url": "", "post_date": "12/06/2023 18:27:22", "content": "how u understand what layers should u use in your model?
", "votes": null, "replies": [ { "id": 2553913, "author_name": "nikhilmishradev", "author_url": "", "post_date": "12/08/2023 16:01:23", "content": "more of a combination of standard practices, reading papers to see what works best, and a bit of intuition with hit and trial. Usually you would want to start with a standard practice, like UNet for segmentation like tasks, and build upon it
", "votes": null, "replies": [] } ] }, { "id": 2551748, "author_name": "snnclsr", "author_url": "", "post_date": "12/06/2023 23:45:50", "content": "Great solution writeup @nikhilmishradev Thank you for sharing! Additional thanks for sharing the WBF implementation!
", "votes": null, "replies": [] }, { "id": 2556803, "author_name": "cdeotte", "author_url": "", "post_date": "12/11/2023 01:56:09", "content": "Congratulations @nikhilmishradev @ryotayoshinobu ! When you two teamed up, i knew you would make a formidable team because you two has the best CV LB single model scores in the CV LB discussion post.
\nCongratulations on strong finish achieving cash gold 4th place!
", "votes": null, "replies": [ { "id": 2557123, "author_name": "nikhilmishradev", "author_url": "", "post_date": "12/11/2023 08:10:20", "content": "Thank you @cdeotte , would definitely want to team up with a legend like you in some competition
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2550489": "Firstly thank you to my teammate @ryotayoshinobu for the amazing performance. Please read his part of the [solution](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/459597).\nI am really glad I teamed up with him and learnt a lot\n\nA big thanks to the organizers as well for such an interesting competition. A really tough one.\n\nYou can find the solution code here:\n\nhttps://github.com/nikhilmishradevelop/kaggle-child-mind-institute-detect-sleep-states\n\n# Solution Summary\n\n**Validation Strategy**: GroupKFold on Series Ids\n\n**Model Inputs**: 17280 x n_features length sequences as input (17280 = 12 steps_per_minute x 60_minutes * 24 hours)\n**Model Outputs**: 17280 x 2 (one for onset and other for wakeup)\n**Model Type**: Regression Model\n**Loss Type**: Cubic Loss i.e. abs(y_true-y_pred)**3\n\nThe last remaining part of the sequences with length < 17280 were padded to make them equal to 17280\n\nMy solution is a modified UNET (averaged over 4 models, 2 LSTMs and 2 GRUs) using normalized Gaussian targets similar to what @tolgadincer described. A big thanks to him for sharing a good method, early on in this competition.\n\n## Features:\n\n1. Original Sequence Features: Enmo, Anglez\n2. TimeStamp Features: Hour and Weekday\n3. Derived Sequence Features: Anglez difference, Enmo Difference, HDCZA features etc)\n\nGood Features help in faster convergence and much better scores, so adding good features were important (Read penguin's solution for some other good features)\n\n**My final 4 model ensemble results**: \n\n**CV**: 0.828\n**Public LB**: 0.789\n**Private LB**: 0.841\n\n\n# **Model Architecture**\n\n\n.png?generation=1701841137052567&alt=media)\n\n\n##Patching to reduce sequence length\n\nSince 17280 is a very long sequence length (makes training very slow and harder to train), we reduce the sequence length by patching it.\n\n**Input:** 17280 * n_features (34)\n**Patch Size:** (Used different patch sizes 3, 4, 5, or 6 in different models).\n\n**Modified Sequence Length:** 17280 to 17280//Patch_Size sequence length\n\n**Modified Feature Size:** k * 4 * num_features (where k is a Dense layer output dim)\n\n## Modified Unet Part\n\n***UNET Encoder -> Bottleneck -> Transformer -> GRU or LSTM -> Unet Decoder***\n\nEach layer of UNET encoder had concatenated connections to the 1st layer\n\n**Initial Output size:** 17280//Patch_Size, 2 * Patch_size\n\n**Reshaped Output Size:** (17280, 2)\n\n\n\n\n# Post Processing\n\nInitially I was doing just doing simple peak detection when I was competing solo.\n\nI wanted to, but could not formulate a good way to rerank model predictions using lightgbm, but @kmat2019 is a great read for that.\n\nThanks to @ryotayoshinobu , I started applying NMS, and finally experimented with a kind of WBF algorithm and made it work. This WBF algorithm gave a score of 0.79 to 0.793 on last day.\n\nBut it was worse by 0.001 in our private LB lol, so WBF hurt.\n\nWBF Working\n\n1. **Initialization and Convolution**: The function starts by applying convolution with the specified `convolution_kernel` to smooth the data.\n\n2. **Peak Detection Loop**: It iteratively searches for peaks in the data. The loop runs until the maximum count (`max_count`) is reached or the peak value falls below a threshold (`max_thresh`).\n\n3. **Adaptive Window and Weight Calculation**:\n - A dynamic window size around each peak is determined based on the current maximum value's power (`curr_max_power`) and a distance parameter (`k_dist`).\n - Depending on the `section_weight_method` (logarithmic or linear), weights are calculated for each section of the data around the peak.\n\n4. **Weighted Average and Score Calculation**: For each detected peak, a weighted average is calculated to determine the score of the peak. \nThis is done by considering the peak and k of its neighbours. This score is influenced by the method of weight calculation and other hyperparameters like `log_base`, `log_scale`, and `weight_coeff`.\n\n5. **Suppression and Updating Predictions**:\n - After each peak is detected, the function suppresses the neighboring values to avoid detecting the same peak multiple times. This is controlled by `overlap_coeff` and `preds_reduction_power`.\n - The indices and scores of detected peaks are stored and returned after the loop completes.\n\n\n\nI will share the implementation of the WBF part code, you can see if tuning these hyperparameters help in your models too. This accepts predictions per series id.\n\n(I found hyperparameters by a mix of manual and automated hyperparamter tuning approach)\n\n```\ndef wbf_nikhil(preds_orig, max_thresh=0.1, max_count=700, hyperparams=None):\n k_dist = hyperparams['k_dist']\n log_base = hyperparams['log_base']\n log_scale = hyperparams['log_scale']\n curr_max_power = hyperparams['curr_max_power']\n weight_coeff = hyperparams['weight_coeff']\n convolution_kernel = hyperparams['convolution_kernel']\n section_weight_method = hyperparams['section_weight_method']\n preds_reduction_power = hyperparams['preds_reduction_power']\n overlap_coeff = hyperparams['overlap_coeff']\n min_distance = hyperparams['min_distance']\n \n preds = preds_orig.copy()\n preds = np.convolve(preds, convolution_kernel, mode='same')\n\n count = 0\n indices = []\n scores = []\n\n while count < max_count:\n curr_max_idx = np.argmax(preds)\n curr_max = preds[curr_max_idx]\n\n if curr_max < max_thresh:\n break\n\n k = int(k_dist - max(min_distance, (curr_max**curr_max_power)))\n\n start_idx = max(curr_max_idx - k, 0)\n end_idx = min(curr_max_idx + k + 1, len(preds))\n\n section = preds[start_idx:end_idx]\n\n # Different weight calculation methods\n distances = np.abs(np.arange(len(section)) - k)\n if section_weight_method == 'logarithmic':\n weights = 1 / (log_base ** (distances / (k * log_scale)))\n elif section_weight_method == 'linear':\n weights = 1 - (distances / k) * weight_coeff\n # Add more methods as needed\n\n weighted_avg = np.sum(section * weights) / np.sum(weights)\n\n scores.append(weighted_avg)\n indices.append(curr_max_idx)\n\n preds[start_idx:end_idx] *= ((1 - weights * overlap_coeff))**preds_reduction_power\n\n count += 1\n\n return indices, scores\n```\n\n## Final Ensemble\n\n\nFinal ensemble was a blended weight of my and Penguin's subs regression postprocessed using WBF\n\nFinal_Sub = WBF(Penguins_Predictions * 0.25 + Nikhil's Predictions*0.75)\n\nSince our predictions had slightly different scales of regression targets, Penguin's Predictions were first scaled by using a simple power transform i.e Penguin's_Predictions ** 0.7\n\n**Ensemble CV:** 0.835\n**Ensemble Public LB:** 0.793\n**Ensemble Private LB:** 0.845\n\n\nP.S: This competition is the first time I did not use gradient boosting, even after I knew it worked so well, me being a tabular guy. I mistake I guess. I love lightgbm and will definitely use one in the next competition.\n\nCongratulations to all the the top performers, it was a good fight, till it lasted :)", "2550672": "A big congratulations bhai", "2551529": "how u understand what layers should u use in your model?", "2551748": "Great solution writeup @nikhilmishradev Thank you for sharing! Additional thanks for sharing the WBF implementation!", "2553913": "more of a combination of standard practices, reading papers to see what works best, and a bit of intuition with hit and trial. Usually you would want to start with a standard practice, like UNet for segmentation like tasks, and build upon it", "2556803": "Congratulations @nikhilmishradev @ryotayoshinobu ! When you two teamed up, i knew you would make a formidable team because you two has the best CV LB single model scores in the CV LB discussion post.\n\nCongratulations on strong finish achieving cash gold 4th place!", "2557123": "Thank you @cdeotte , would definitely want to team up with a legend like you in some competition" }, "source": "meta" }