{
  "id": 448288,
  "title": "RNN Critical Point Baselines | How to Improve From Here",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/448288",
  "author_name": "werus23",
  "post_date": "2023-10-19T01:53:35.403000",
  "votes": 50,
  "comment_count": 22,
  "views": 0,
  "content": "<p>Since it seems like public notebooks are stuck on gradient boosting methods (and because I want to get medals), I am publicizing a crude version of my current NN pipeline, which seems to work.</p>\n<h1>Links:</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/werus23/sleep-critical-point-prepare-data\" target=\"_blank\">Data Prep notebook</a></li>\n<li><a href=\"https://www.kaggle.com/code/werus23/sleep-critical-point-train/\" target=\"_blank\">Training notebook</a></li>\n<li><a href=\"https://www.kaggle.com/code/werus23/sleep-critical-point-infer/\" target=\"_blank\">Inference Notebook</a></li>\n</ul>\n<h1>Method:</h1>\n<ul>\n<li>Do not separate data into days</li>\n<li>Downsample the series into minutes</li>\n<li>Fit a target gaussian distribution along onset and wakeup times</li>\n<li>Use the predictions to rank the predictions via the \"score\" column in the submission dataframe</li>\n</ul>\n<h1>Next steps:</h1>\n<ul>\n<li>Cross validation</li>\n<li>Optimize parameters</li>\n<li>Post-processing</li>\n<li>Pseudo-labelling?</li>\n<li>Transformer?</li>\n</ul>\n<p>Any other ideas on how to improve from here?</p>",
  "messages": [
    {
      "id": 2488059,
      "postDate": "2023-10-19T01:53:35.403Z",
      "content": "<p>Since it seems like public notebooks are stuck on gradient boosting methods (and because I want to get medals), I am publicizing a crude version of my current NN pipeline, which seems to work.</p>\n<h1>Links:</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/werus23/sleep-critical-point-prepare-data\" target=\"_blank\">Data Prep notebook</a></li>\n<li><a href=\"https://www.kaggle.com/code/werus23/sleep-critical-point-train/\" target=\"_blank\">Training notebook</a></li>\n<li><a href=\"https://www.kaggle.com/code/werus23/sleep-critical-point-infer/\" target=\"_blank\">Inference Notebook</a></li>\n</ul>\n<h1>Method:</h1>\n<ul>\n<li>Do not separate data into days</li>\n<li>Downsample the series into minutes</li>\n<li>Fit a target gaussian distribution along onset and wakeup times</li>\n<li>Use the predictions to rank the predictions via the \"score\" column in the submission dataframe</li>\n</ul>\n<h1>Next steps:</h1>\n<ul>\n<li>Cross validation</li>\n<li>Optimize parameters</li>\n<li>Post-processing</li>\n<li>Pseudo-labelling?</li>\n<li>Transformer?</li>\n</ul>\n<p>Any other ideas on how to improve from here?</p>",
      "rawMarkdown": "Since it seems like public notebooks are stuck on gradient boosting methods (and because I want to get medals), I am publicizing a crude version of my current NN pipeline, which seems to work.\n \n# Links:\n - [Data Prep notebook](https://www.kaggle.com/code/werus23/sleep-critical-point-prepare-data)\n - [Training notebook](https://www.kaggle.com/code/werus23/sleep-critical-point-train/)\n - [Inference Notebook](https://www.kaggle.com/code/werus23/sleep-critical-point-infer/)\n\n\n# Method:\n - Do not separate data into days\n - Downsample the series into minutes\n - Fit a target gaussian distribution along onset and wakeup times\n - Use the predictions to rank the predictions via the \"score\" column in the submission dataframe\n\n\n# Next steps:\n - Cross validation\n - Optimize parameters\n - Post-processing\n - Pseudo-labelling?\n - Transformer?\n\nAny other ideas on how to improve from here?",
      "votes": 50
    },
    {
      "id": 2491705,
      "postDate": "2023-10-21T23:07:08.837Z",
      "content": "<p>Thanks for your great notebooks, have you tried to increase the <code>num_layers</code> or <code>hidden_size</code> ? I find it easy to encounter OOM problem when testing on the hidden dataset.</p>",
      "rawMarkdown": "Thanks for your great notebooks, have you tried to increase the ``num_layers`` or ``hidden_size`` ? I find it easy to encounter OOM problem when testing on the hidden dataset.",
      "votes": 3,
      "replies": [
        {
          "id": 2491762,
          "postDate": "2023-10-22T02:51:34.130Z",
          "content": "<p>Yes, you may need to run the inference on GPU with half-precision if you are running out of memory on CPU.</p>",
          "rawMarkdown": "Yes, you may need to run the inference on GPU with half-precision if you are running out of memory on CPU.",
          "votes": 3,
          "replies": [
            {
              "id": 2491772,
              "postDate": "2023-10-22T03:13:00.677Z",
              "content": "<p>Yeah, I think so. Thank you very much !</p>",
              "rawMarkdown": "Yeah, I think so. Thank you very much !",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2489000,
      "postDate": "2023-10-19T16:13:26.977Z",
      "content": "<p>It might be useful to use the KLDivLoss instead of the MSELoss, since this is a probability distribution you are looking for with the gaussian distribution.</p>",
      "rawMarkdown": "It might be useful to use the KLDivLoss instead of the MSELoss, since this is a probability distribution you are looking for with the gaussian distribution.",
      "votes": 4,
      "replies": [
        {
          "id": 2489004,
          "postDate": "2023-10-19T16:16:47.197Z",
          "content": "<p>Sounds great, I’ll check this out, and record the result here, thank you! <a href=\"https://www.kaggle.com/danielphalen\" target=\"_blank\">@danielphalen</a> </p>",
          "rawMarkdown": "Sounds great, I’ll check this out, and record the result here, thank you! @danielphalen ",
          "votes": 2
        },
        {
          "id": 2509203,
          "postDate": "2023-11-02T08:54:06.927Z",
          "content": "<p>Thank you for sharing great info!!! <a href=\"https://www.kaggle.com/WERUS23\" target=\"_blank\">@WERUS23</a> <a href=\"https://www.kaggle.com/danielphalen\" target=\"_blank\">@danielphalen</a><br>\ni tried the KLDivLoss for few days, and it work but Training loss is too large. do you have any tipps?</p>",
          "rawMarkdown": "Thank you for sharing great info!!! @WERUS23 @danielphalen\ni tried the KLDivLoss for few days, and it work but Training loss is too large. do you have any tipps?",
          "replies": [
            {
              "id": 2509646,
              "postDate": "2023-11-02T13:41:02.163Z",
              "content": "<p>I'll get a couple of notebooks together and post them soon!</p>",
              "rawMarkdown": "I'll get a couple of notebooks together and post them soon!",
              "votes": 2
            },
            {
              "id": 2510165,
              "postDate": "2023-11-02T18:54:56.723Z",
              "content": "<p>Hey, they are up and running!</p>\n<p><a href=\"https://www.kaggle.com/code/danielphalen/cmss-grunet-infer\" target=\"_blank\">https://www.kaggle.com/code/danielphalen/cmss-grunet-infer</a></p>",
              "rawMarkdown": "Hey, they are up and running!\n\n[https://www.kaggle.com/code/danielphalen/cmss-grunet-infer](https://www.kaggle.com/code/danielphalen/cmss-grunet-infer)",
              "votes": 3
            },
            {
              "id": 2510489,
              "postDate": "2023-11-03T03:34:24.070Z",
              "content": "<p>Im excited to learn a lot from the notebook! i appreciate all your help <a href=\"https://www.kaggle.com/danielphalen\" target=\"_blank\">@danielphalen</a></p>",
              "rawMarkdown": "Im excited to learn a lot from the notebook! i appreciate all your help @danielphalen"
            }
          ]
        }
      ]
    },
    {
      "id": 2510208,
      "postDate": "2023-11-02T19:34:31.977Z",
      "content": "<blockquote>\n  <p>Fit a target gaussian distribution along onset and wakeup times</p>\n</blockquote>\n<p>I didn't fully understood the idea. Should we put centers of distributions exactly at the points \"sleep\" and \"wakeup\" or should we put the centroid at the middle point between the \"sleep\" and \"wakeup\"?</p>",
      "rawMarkdown": ">Fit a target gaussian distribution along onset and wakeup times\n\nI didn't fully understood the idea. Should we put centers of distributions exactly at the points \"sleep\" and \"wakeup\" or should we put the centroid at the middle point between the \"sleep\" and \"wakeup\"?",
      "votes": 1,
      "replies": [
        {
          "id": 2510245,
          "postDate": "2023-11-02T20:25:23.787Z",
          "content": "<p>The centers exactly at the points</p>",
          "rawMarkdown": "The centers exactly at the points",
          "votes": 1
        }
      ]
    },
    {
      "id": 2488193,
      "postDate": "2023-10-19T05:43:14.003Z",
      "content": "<p>It makes sense, appreciate it <a href=\"https://www.kaggle.com/werus23\" target=\"_blank\">@werus23</a>. I’m just wondering why using unnormalized pred to rank, what’s the motivation?</p>",
      "rawMarkdown": "It makes sense, appreciate it @werus23. I’m just wondering why using unnormalized pred to rank, what’s the motivation?",
      "votes": 2,
      "replies": [
        {
          "id": 2488214,
          "postDate": "2023-10-19T06:06:43.477Z",
          "content": "<p>Well in this case normalizing the prediction does nothing because it doesn't change the order of the predictions. I guess there was no need in specifying that. </p>\n<p>Thanks for pointing that out!</p>",
          "rawMarkdown": "Well in this case normalizing the prediction does nothing because it doesn't change the order of the predictions. I guess there was no need in specifying that. \n\nThanks for pointing that out!",
          "votes": 3,
          "replies": [
            {
              "id": 2488476,
              "postDate": "2023-10-19T09:23:26.150Z",
              "rawMarkdown": "",
              "isDeleted": true
            }
          ]
        }
      ]
    },
    {
      "id": 2493529,
      "postDate": "2023-10-23T13:14:42.060Z",
      "content": "<p>Would you like to ask which parameters may be adjusted to improve it?</p>",
      "rawMarkdown": "Would you like to ask which parameters may be adjusted to improve it?"
    },
    {
      "id": 2492755,
      "postDate": "2023-10-22T21:08:29.890Z",
      "content": "<p>Tried the KLDivLoss option. Gives NaNs after a few iterations so changed it back.</p>",
      "rawMarkdown": "Tried the KLDivLoss option. Gives NaNs after a few iterations so changed it back.",
      "replies": [
        {
          "id": 2492847,
          "postDate": "2023-10-23T00:17:49.087Z",
          "content": "<p>This might be because you need to ensure it is a proper distribution.  Take your target and make sure they normalize to 1, and you might need to apply torch.LogSoftmax to the output of your distribution.  I have gotten some decent results.</p>",
          "rawMarkdown": "This might be because you need to ensure it is a proper distribution.  Take your target and make sure they normalize to 1, and you might need to apply torch.LogSoftmax to the output of your distribution.  I have gotten some decent results.",
          "votes": 1,
          "replies": [
            {
              "id": 2493304,
              "postDate": "2023-10-23T09:54:21.187Z",
              "content": "<p>Ok Daniel Phalen. Will do. Thanks.</p>",
              "rawMarkdown": "Ok Daniel Phalen. Will do. Thanks."
            },
            {
              "id": 2493441,
              "postDate": "2023-10-23T12:17:09.793Z",
              "content": "<p>Also, you need to eliminate series_ids that have no events.  That will also cause NaNs</p>",
              "rawMarkdown": "Also, you need to eliminate series_ids that have no events.  That will also cause NaNs",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2492674,
      "postDate": "2023-10-22T18:19:08.570Z",
      "content": "<p><a href=\"https://www.kaggle.com/werus23\" target=\"_blank\">@werus23</a>, </p>\n<p>Great and excellent work by the way. I completely agree about the ML methods. Also, based on my experiences, they are not as intuitive as the DL methods especially for this kind of problem (again, at least for me). Also, the GPU acceleration makes all the difference and avoids the ulcer inducing slow times of the ML schemes only to give you a minute change for novices like myself.</p>\n<p>Increase training set size and number of epochs. You will get some initial jumps and later modest increases. Trying to see if I can augment the dataset and increase number of training entries. Is anyone heading in that direction.</p>",
      "rawMarkdown": "@werus23, \n\nGreat and excellent work by the way. I completely agree about the ML methods. Also, based on my experiences, they are not as intuitive as the DL methods especially for this kind of problem (again, at least for me). Also, the GPU acceleration makes all the difference and avoids the ulcer inducing slow times of the ML schemes only to give you a minute change for novices like myself.\n\nIncrease training set size and number of epochs. You will get some initial jumps and later modest increases. Trying to see if I can augment the dataset and increase number of training entries. Is anyone heading in that direction.",
      "replies": [
        {
          "id": 2492679,
          "postDate": "2023-10-22T18:26:38.533Z",
          "content": "<p>Also pondering the Transformer option too.</p>",
          "rawMarkdown": "Also pondering the Transformer option too."
        }
      ]
    },
    {
      "id": 2488064,
      "postDate": "2023-10-19T02:18:32.740Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 2491705,
      "author_name": "Zhecheng Li",
      "author_url": "",
      "post_date": "2023-10-21T23:07:08.837000",
      "content": "<p>Thanks for your great notebooks, have you tried to increase the <code>num_layers</code> or <code>hidden_size</code> ? I find it easy to encounter OOM problem when testing on the hidden dataset.</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2491762,
          "author_name": "werus23",
          "author_url": "",
          "post_date": "2023-10-22T02:51:34.130000",
          "content": "<p>Yes, you may need to run the inference on GPU with half-precision if you are running out of memory on CPU.</p>",
          "votes": 3,
          "replies": [
            {
              "id": 2491772,
              "author_name": "Zhecheng Li",
              "author_url": "",
              "post_date": "2023-10-22T03:13:00.677000",
              "content": "<p>Yeah, I think so. Thank you very much !</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2489000,
      "author_name": "Daniel Phalen",
      "author_url": "",
      "post_date": "2023-10-19T16:13:26.977000",
      "content": "<p>It might be useful to use the KLDivLoss instead of the MSELoss, since this is a probability distribution you are looking for with the gaussian distribution.</p>",
      "votes": 4,
      "replies": [
        {
          "id": 2489004,
          "author_name": "Lamal",
          "author_url": "",
          "post_date": "2023-10-19T16:16:47.197000",
          "content": "<p>Sounds great, I’ll check this out, and record the result here, thank you! <a href=\"https://www.kaggle.com/danielphalen\" target=\"_blank\">@danielphalen</a> </p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 2509203,
          "author_name": "footrio",
          "author_url": "",
          "post_date": "2023-11-02T08:54:06.927000",
          "content": "<p>Thank you for sharing great info!!! <a href=\"https://www.kaggle.com/WERUS23\" target=\"_blank\">@WERUS23</a> <a href=\"https://www.kaggle.com/danielphalen\" target=\"_blank\">@danielphalen</a><br>\ni tried the KLDivLoss for few days, and it work but Training loss is too large. do you have any tipps?</p>",
          "votes": 0,
          "replies": [
            {
              "id": 2509646,
              "author_name": "Daniel Phalen",
              "author_url": "",
              "post_date": "2023-11-02T13:41:02.163000",
              "content": "<p>I'll get a couple of notebooks together and post them soon!</p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 2510165,
              "author_name": "Daniel Phalen",
              "author_url": "",
              "post_date": "2023-11-02T18:54:56.723000",
              "content": "<p>Hey, they are up and running!</p>\n<p><a href=\"https://www.kaggle.com/code/danielphalen/cmss-grunet-infer\" target=\"_blank\">https://www.kaggle.com/code/danielphalen/cmss-grunet-infer</a></p>",
              "votes": 3,
              "replies": []
            },
            {
              "id": 2510489,
              "author_name": "footrio",
              "author_url": "",
              "post_date": "2023-11-03T03:34:24.070000",
              "content": "<p>Im excited to learn a lot from the notebook! i appreciate all your help <a href=\"https://www.kaggle.com/danielphalen\" target=\"_blank\">@danielphalen</a></p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2510208,
      "author_name": "Man of the year",
      "author_url": "",
      "post_date": "2023-11-02T19:34:31.977000",
      "content": "<blockquote>\n  <p>Fit a target gaussian distribution along onset and wakeup times</p>\n</blockquote>\n<p>I didn't fully understood the idea. Should we put centers of distributions exactly at the points \"sleep\" and \"wakeup\" or should we put the centroid at the middle point between the \"sleep\" and \"wakeup\"?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2510245,
          "author_name": "werus23",
          "author_url": "",
          "post_date": "2023-11-02T20:25:23.787000",
          "content": "<p>The centers exactly at the points</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2488193,
      "author_name": "Lamal",
      "author_url": "",
      "post_date": "2023-10-19T05:43:14.003000",
      "content": "<p>It makes sense, appreciate it <a href=\"https://www.kaggle.com/werus23\" target=\"_blank\">@werus23</a>. I’m just wondering why using unnormalized pred to rank, what’s the motivation?</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2488214,
          "author_name": "werus23",
          "author_url": "",
          "post_date": "2023-10-19T06:06:43.477000",
          "content": "<p>Well in this case normalizing the prediction does nothing because it doesn't change the order of the predictions. I guess there was no need in specifying that. </p>\n<p>Thanks for pointing that out!</p>",
          "votes": 3,
          "replies": [
            {
              "id": 2488476,
              "author_name": "",
              "author_url": "",
              "post_date": "2023-10-19T09:23:26.150000",
              "content": "",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2493529,
      "author_name": "SEVEN",
      "author_url": "",
      "post_date": "2023-10-23T13:14:42.060000",
      "content": "<p>Would you like to ask which parameters may be adjusted to improve it?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2492755,
      "author_name": "AIDLRE001",
      "author_url": "",
      "post_date": "2023-10-22T21:08:29.890000",
      "content": "<p>Tried the KLDivLoss option. Gives NaNs after a few iterations so changed it back.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2492847,
          "author_name": "Daniel Phalen",
          "author_url": "",
          "post_date": "2023-10-23T00:17:49.087000",
          "content": "<p>This might be because you need to ensure it is a proper distribution.  Take your target and make sure they normalize to 1, and you might need to apply torch.LogSoftmax to the output of your distribution.  I have gotten some decent results.</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2493304,
              "author_name": "AIDLRE001",
              "author_url": "",
              "post_date": "2023-10-23T09:54:21.187000",
              "content": "<p>Ok Daniel Phalen. Will do. Thanks.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2493441,
              "author_name": "Daniel Phalen",
              "author_url": "",
              "post_date": "2023-10-23T12:17:09.793000",
              "content": "<p>Also, you need to eliminate series_ids that have no events.  That will also cause NaNs</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2492674,
      "author_name": "AIDLRE001",
      "author_url": "",
      "post_date": "2023-10-22T18:19:08.570000",
      "content": "<p><a href=\"https://www.kaggle.com/werus23\" target=\"_blank\">@werus23</a>, </p>\n<p>Great and excellent work by the way. I completely agree about the ML methods. Also, based on my experiences, they are not as intuitive as the DL methods especially for this kind of problem (again, at least for me). Also, the GPU acceleration makes all the difference and avoids the ulcer inducing slow times of the ML schemes only to give you a minute change for novices like myself.</p>\n<p>Increase training set size and number of epochs. You will get some initial jumps and later modest increases. Trying to see if I can augment the dataset and increase number of training entries. Is anyone heading in that direction.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2492679,
          "author_name": "AIDLRE001",
          "author_url": "",
          "post_date": "2023-10-22T18:26:38.533000",
          "content": "<p>Also pondering the Transformer option too.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2488064,
      "author_name": "",
      "author_url": "",
      "post_date": "2023-10-19T02:18:32.740000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2488059": "Since it seems like public notebooks are stuck on gradient boosting methods (and because I want to get medals), I am publicizing a crude version of my current NN pipeline, which seems to work.\n \n# Links:\n - [Data Prep notebook](https://www.kaggle.com/code/werus23/sleep-critical-point-prepare-data)\n - [Training notebook](https://www.kaggle.com/code/werus23/sleep-critical-point-train/)\n - [Inference Notebook](https://www.kaggle.com/code/werus23/sleep-critical-point-infer/)\n\n\n# Method:\n - Do not separate data into days\n - Downsample the series into minutes\n - Fit a target gaussian distribution along onset and wakeup times\n - Use the predictions to rank the predictions via the \"score\" column in the submission dataframe\n\n\n# Next steps:\n - Cross validation\n - Optimize parameters\n - Post-processing\n - Pseudo-labelling?\n - Transformer?\n\nAny other ideas on how to improve from here?",
    "2491705": "Thanks for your great notebooks, have you tried to increase the ``num_layers`` or ``hidden_size`` ? I find it easy to encounter OOM problem when testing on the hidden dataset.",
    "2489000": "It might be useful to use the KLDivLoss instead of the MSELoss, since this is a probability distribution you are looking for with the gaussian distribution.",
    "2510208": ">Fit a target gaussian distribution along onset and wakeup times\n\nI didn't fully understood the idea. Should we put centers of distributions exactly at the points \"sleep\" and \"wakeup\" or should we put the centroid at the middle point between the \"sleep\" and \"wakeup\"?",
    "2488193": "It makes sense, appreciate it @werus23. I’m just wondering why using unnormalized pred to rank, what’s the motivation?",
    "2493529": "Would you like to ask which parameters may be adjusted to improve it?",
    "2492755": "Tried the KLDivLoss option. Gives NaNs after a few iterations so changed it back.",
    "2492674": "@werus23, \n\nGreat and excellent work by the way. I completely agree about the ML methods. Also, based on my experiences, they are not as intuitive as the DL methods especially for this kind of problem (again, at least for me). Also, the GPU acceleration makes all the difference and avoids the ulcer inducing slow times of the ML schemes only to give you a minute change for novices like myself.\n\nIncrease training set size and number of epochs. You will get some initial jumps and later modest increases. Trying to see if I can augment the dataset and increase number of training entries. Is anyone heading in that direction.",
    "2488064": ""
  }
}