{
  "id": 444152,
  "title": "Child Mind Institute - Notebook Out of Memory",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/444152",
  "author_name": "",
  "post_date": "2023-09-30T13:44:44.997765800Z",
  "votes": 2,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hi Kaggle Family.</p>\n<p>I need your help. Maybe I´m confused. I get error when I try to submit my submission file to this competition. I always get messages error: \"Notebook Out of Memory\", \"Your Notebook requested more memory (RAM)\". </p>\n<p>I dont know why, because when I run my Notebook, I check line by line the RAM memory amount used, and this RAM amount does not exceed 7GB, when total RAM size is 30GB.</p>\n<p>Can anyone get the same issue?</p>\n<p>Although my submissions are failed, It counts in daily submission attemps.</p>\n<p>Thanks to all. <br>\nBest Regards!</p>",
  "messages": [
    {
      "id": "2462635",
      "postDate": "09/30/2023 13:44:44",
      "content": "<p>Hi Kaggle Family.</p>\n<p>I need your help. Maybe I´m confused. I get error when I try to submit my submission file to this competition. I always get messages error: \"Notebook Out of Memory\", \"Your Notebook requested more memory (RAM)\". </p>\n<p>I dont know why, because when I run my Notebook, I check line by line the RAM memory amount used, and this RAM amount does not exceed 7GB, when total RAM size is 30GB.</p>\n<p>Can anyone get the same issue?</p>\n<p>Although my submissions are failed, It counts in daily submission attemps.</p>\n<p>Thanks to all. <br>\nBest Regards!</p>",
      "rawMarkdown": "Hi Kaggle Family.\n\nI need your help. Maybe I´m confused. I get error when I try to submit my submission file to this competition. I always get messages error: \"Notebook Out of Memory\", \"Your Notebook requested more memory (RAM)\". \n\nI dont know why, because when I run my Notebook, I check line by line the RAM memory amount used, and this RAM amount does not exceed 7GB, when total RAM size is 30GB.\n\nCan anyone get the same issue?\n\nAlthough my submissions are failed, It counts in daily submission attemps.\n\nThanks to all. \nBest Regards!",
      "votes": null
    },
    {
      "id": "2462807",
      "postDate": "09/30/2023 16:44:50",
      "content": "<p>Hi, the test set is much bigger than the test set. Hence, when you load the test set into memory and create additional features on it and then load the model to make the predictions, you take up a lot of memory. This is much more than what you used for training. You can do a few things here:<br>\n1) Compress the data further into int16 or smaller data yoes to preserve memory. <br>\n2) Load the test series iteratively. You can load only one series id and then create feature, make your predictions and then write they to a csv fie. You can then combine these csv files. <a href=\"https://www.kaggle.com/code/anilreddyvv/tackling-out-of-memory-issue-on-test-set\" target=\"_blank\">This</a> is a greate notebook that shows that </p>",
      "rawMarkdown": "Hi, the test set is much bigger than the test set. Hence, when you load the test set into memory and create additional features on it and then load the model to make the predictions, you take up a lot of memory. This is much more than what you used for training. You can do a few things here:\n1) Compress the data further into int16 or smaller data yoes to preserve memory. \n2) Load the test series iteratively. You can load only one series id and then create feature, make your predictions and then write they to a csv fie. You can then combine these csv files. [This](https://www.kaggle.com/code/anilreddyvv/tackling-out-of-memory-issue-on-test-set) is a greate notebook that shows that",
      "votes": null
    },
    {
      "id": "2462837",
      "postDate": "09/30/2023 17:31:48",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/abhranta\" target=\"_blank\">@abhranta</a> .</p>\n<p>First of all, thank you for your fast anwser.</p>\n<p>OK, I´m going to follow your two steps, and give you feedback of results.</p>\n<p>Thank you so much! Beste Regards!</p>",
      "rawMarkdown": "Hi @abhranta .\n\nFirst of all, thank you for your fast anwser.\n\nOK, I´m going to follow your two steps, and give you feedback of results.\n\nThank you so much! Beste Regards!",
      "votes": null
    },
    {
      "id": "2463872",
      "postDate": "10/01/2023 16:24:54",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/abhranta\" target=\"_blank\">@abhranta</a> .</p>\n<p>Your second point (load and predict test series iteratively)resolve my problem! Thank you so much!!</p>\n<p>Best Regards!!!</p>",
      "rawMarkdown": "Hi @abhranta .\n\nYour second point (load and predict test series iteratively)resolve my problem! Thank you so much!!\n\nBest Regards!!!",
      "votes": null
    },
    {
      "id": "2488500",
      "postDate": "10/19/2023 09:41:52",
      "content": "<p>Hello! When uploading the solution to submission, can we use data from other sets that I prepared in advance in another program on kaggle? If you can do this, can you please teach me how to do it, I can't understand.</p>",
      "rawMarkdown": "Hello! When uploading the solution to submission, can we use data from other sets that I prepared in advance in another program on kaggle? If you can do this, can you please teach me how to do it, I can't understand.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2462807,
      "author_name": "abhranta",
      "author_url": "",
      "post_date": "09/30/2023 16:44:50",
      "content": "<p>Hi, the test set is much bigger than the test set. Hence, when you load the test set into memory and create additional features on it and then load the model to make the predictions, you take up a lot of memory. This is much more than what you used for training. You can do a few things here:<br>\n1) Compress the data further into int16 or smaller data yoes to preserve memory. <br>\n2) Load the test series iteratively. You can load only one series id and then create feature, make your predictions and then write they to a csv fie. You can then combine these csv files. <a href=\"https://www.kaggle.com/code/anilreddyvv/tackling-out-of-memory-issue-on-test-set\" target=\"_blank\">This</a> is a greate notebook that shows that </p>",
      "votes": null,
      "replies": [
        {
          "id": 2462837,
          "author_name": "lvarochapresto",
          "author_url": "",
          "post_date": "09/30/2023 17:31:48",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/abhranta\" target=\"_blank\">@abhranta</a> .</p>\n<p>First of all, thank you for your fast anwser.</p>\n<p>OK, I´m going to follow your two steps, and give you feedback of results.</p>\n<p>Thank you so much! Beste Regards!</p>",
          "votes": null,
          "replies": [
            {
              "id": 2463872,
              "author_name": "lvarochapresto",
              "author_url": "",
              "post_date": "10/01/2023 16:24:54",
              "content": "<p>Hi <a href=\"https://www.kaggle.com/abhranta\" target=\"_blank\">@abhranta</a> .</p>\n<p>Your second point (load and predict test series iteratively)resolve my problem! Thank you so much!!</p>\n<p>Best Regards!!!</p>",
              "votes": null,
              "replies": []
            }
          ]
        },
        {
          "id": 2488500,
          "author_name": "rinatbayanov",
          "author_url": "",
          "post_date": "10/19/2023 09:41:52",
          "content": "<p>Hello! When uploading the solution to submission, can we use data from other sets that I prepared in advance in another program on kaggle? If you can do this, can you please teach me how to do it, I can't understand.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2462635": "Hi Kaggle Family.\n\nI need your help. Maybe I´m confused. I get error when I try to submit my submission file to this competition. I always get messages error: \"Notebook Out of Memory\", \"Your Notebook requested more memory (RAM)\". \n\nI dont know why, because when I run my Notebook, I check line by line the RAM memory amount used, and this RAM amount does not exceed 7GB, when total RAM size is 30GB.\n\nCan anyone get the same issue?\n\nAlthough my submissions are failed, It counts in daily submission attemps.\n\nThanks to all. \nBest Regards!",
    "2462807": "Hi, the test set is much bigger than the test set. Hence, when you load the test set into memory and create additional features on it and then load the model to make the predictions, you take up a lot of memory. This is much more than what you used for training. You can do a few things here:\n1) Compress the data further into int16 or smaller data yoes to preserve memory. \n2) Load the test series iteratively. You can load only one series id and then create feature, make your predictions and then write they to a csv fie. You can then combine these csv files. [This](https://www.kaggle.com/code/anilreddyvv/tackling-out-of-memory-issue-on-test-set) is a greate notebook that shows that",
    "2462837": "Hi @abhranta .\n\nFirst of all, thank you for your fast anwser.\n\nOK, I´m going to follow your two steps, and give you feedback of results.\n\nThank you so much! Beste Regards!",
    "2463872": "Hi @abhranta .\n\nYour second point (load and predict test series iteratively)resolve my problem! Thank you so much!!\n\nBest Regards!!!",
    "2488500": "Hello! When uploading the solution to submission, can we use data from other sets that I prepared in advance in another program on kaggle? If you can do this, can you please teach me how to do it, I can't understand."
  },
  "source": "meta"
}