{
  "id": 411919,
  "title": "Memory Allocation Error",
  "url": "/competitions/cafa-5-protein-function-prediction/discussion/411919",
  "author_name": "",
  "post_date": "2023-05-21T15:50:10.853745900Z",
  "votes": 3,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hey! I've been trying to use kfold cross validation and every time I'm getting this error: \"Your notebook tried to allocate too much memory and could not be recovered. You will need to start a new session\". Did anybody face the same issue? It never happened before. <br>\nThank you in advance. </p>",
  "messages": [
    {
      "id": "2268274",
      "postDate": "05/21/2023 15:50:10",
      "content": "<p>Hey! I've been trying to use kfold cross validation and every time I'm getting this error: \"Your notebook tried to allocate too much memory and could not be recovered. You will need to start a new session\". Did anybody face the same issue? It never happened before. <br>\nThank you in advance. </p>",
      "rawMarkdown": "Hey! I've been trying to use kfold cross validation and every time I'm getting this error: \"Your notebook tried to allocate too much memory and could not be recovered. You will need to start a new session\". Did anybody face the same issue? It never happened before. \nThank you in advance.",
      "votes": null
    },
    {
      "id": "2268279",
      "postDate": "05/21/2023 15:56:46",
      "content": "<p>Hello 👋<br>\nMany reasons can trigger this error, the most common being:</p>\n<ul>\n<li>you loaded a lot of data into memory and your RAM is full. Have you tried doing:</li>\n</ul>\n<pre><code> unused_frame\ngc.collect()\n</code></pre>\n<ul>\n<li><p>you are fitting some heavy model whose computation is killing your RAM. Have you tried smaller models or easier data processing steps?</p></li>\n<li><p>Are you loading big CSVs? You might fill your RAM quickly</p></li>\n</ul>\n<p>In general, memory issues are caused by large Pandas DataFrames (and their copies) being stored in memory. </p>",
      "rawMarkdown": "Hello 👋\nMany reasons can trigger this error, the most common being:\n- you loaded a lot of data into memory and your RAM is full. Have you tried doing:\n```python\ndel unused_frame\ngc.collect()\n```\n- you are fitting some heavy model whose computation is killing your RAM. Have you tried smaller models or easier data processing steps?\n\n- Are you loading big CSVs? You might fill your RAM quickly\n\nIn general, memory issues are caused by large Pandas DataFrames (and their copies) being stored in memory.",
      "votes": null
    },
    {
      "id": "2268288",
      "postDate": "05/21/2023 16:00:30",
      "content": "<p>Oh yes! I just removed this in my new version of notebook. Makes sense. I need to delete the unused data frames. Thank you so much for pointing it out! :)</p>",
      "rawMarkdown": "Oh yes! I just removed this in my new version of notebook. Makes sense. I need to delete the unused data frames. Thank you so much for pointing it out! :)",
      "votes": null
    },
    {
      "id": "2269570",
      "postDate": "05/22/2023 14:54:12",
      "content": "<p>You have only a certain amount of ram. Probably you needed to much memory (= RAM). Maybe you can run your model only of a part of the data? </p>",
      "rawMarkdown": "You have only a certain amount of ram. Probably you needed to much memory (= RAM). Maybe you can run your model only of a part of the data?",
      "votes": null
    },
    {
      "id": "2272554",
      "postDate": "05/24/2023 15:14:01",
      "content": "<p>Yeah, I had been storing a lot of data in the memory which was raising the issue. Finally, removed the unused data and it worked! Thank you for your insight! :)</p>",
      "rawMarkdown": "Yeah, I had been storing a lot of data in the memory which was raising the issue. Finally, removed the unused data and it worked! Thank you for your insight! :)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2268279,
      "author_name": "wadgek",
      "author_url": "",
      "post_date": "05/21/2023 15:56:46",
      "content": "<p>Hello 👋<br>\nMany reasons can trigger this error, the most common being:</p>\n<ul>\n<li>you loaded a lot of data into memory and your RAM is full. Have you tried doing:</li>\n</ul>\n<pre><code> unused_frame\ngc.collect()\n</code></pre>\n<ul>\n<li><p>you are fitting some heavy model whose computation is killing your RAM. Have you tried smaller models or easier data processing steps?</p></li>\n<li><p>Are you loading big CSVs? You might fill your RAM quickly</p></li>\n</ul>\n<p>In general, memory issues are caused by large Pandas DataFrames (and their copies) being stored in memory. </p>",
      "votes": null,
      "replies": [
        {
          "id": 2268288,
          "author_name": "ahsuna123",
          "author_url": "",
          "post_date": "05/21/2023 16:00:30",
          "content": "<p>Oh yes! I just removed this in my new version of notebook. Makes sense. I need to delete the unused data frames. Thank you so much for pointing it out! :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2269570,
      "author_name": "konhugo",
      "author_url": "",
      "post_date": "05/22/2023 14:54:12",
      "content": "<p>You have only a certain amount of ram. Probably you needed to much memory (= RAM). Maybe you can run your model only of a part of the data? </p>",
      "votes": null,
      "replies": [
        {
          "id": 2272554,
          "author_name": "ahsuna123",
          "author_url": "",
          "post_date": "05/24/2023 15:14:01",
          "content": "<p>Yeah, I had been storing a lot of data in the memory which was raising the issue. Finally, removed the unused data and it worked! Thank you for your insight! :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2268274": "Hey! I've been trying to use kfold cross validation and every time I'm getting this error: \"Your notebook tried to allocate too much memory and could not be recovered. You will need to start a new session\". Did anybody face the same issue? It never happened before. \nThank you in advance.",
    "2268279": "Hello 👋\nMany reasons can trigger this error, the most common being:\n- you loaded a lot of data into memory and your RAM is full. Have you tried doing:\n```python\ndel unused_frame\ngc.collect()\n```\n- you are fitting some heavy model whose computation is killing your RAM. Have you tried smaller models or easier data processing steps?\n\n- Are you loading big CSVs? You might fill your RAM quickly\n\nIn general, memory issues are caused by large Pandas DataFrames (and their copies) being stored in memory.",
    "2268288": "Oh yes! I just removed this in my new version of notebook. Makes sense. I need to delete the unused data frames. Thank you so much for pointing it out! :)",
    "2269570": "You have only a certain amount of ram. Probably you needed to much memory (= RAM). Maybe you can run your model only of a part of the data?",
    "2272554": "Yeah, I had been storing a lot of data in the memory which was raising the issue. Finally, removed the unused data and it worked! Thank you for your insight! :)"
  },
  "source": "meta"
}