{
  "id": 295416,
  "title": "Memory errors, large datas",
  "url": "/competitions/g-research-crypto-forecasting/discussion/295416",
  "author_name": "",
  "post_date": "2021-12-16T01:06:05.064586Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi guys, I am new to ML and kaggle competitons ?</p>\n<p>My questions are;</p>\n<p>1-  When I use this large data set on Kaggle, always I got allocate memory errors<br>\n2- When I use on my local, my CPU wasn't enough at all</p>\n<p>I already created my model with piece of data but I couldn't use with all data set because of above reason.</p>\n<p>I would be very grateful if someone could guide me.</p>",
  "messages": [
    {
      "id": "1619461",
      "postDate": "12/16/2021 01:06:05",
      "content": "<p>Hi guys, I am new to ML and kaggle competitons ?</p>\n<p>My questions are;</p>\n<p>1-  When I use this large data set on Kaggle, always I got allocate memory errors<br>\n2- When I use on my local, my CPU wasn't enough at all</p>\n<p>I already created my model with piece of data but I couldn't use with all data set because of above reason.</p>\n<p>I would be very grateful if someone could guide me.</p>",
      "rawMarkdown": "Hi guys, I am new to ML and kaggle competitons ?\n\nMy questions are;\n\n1-  When I use this large data set on Kaggle, always I got allocate memory errors\n2- When I use on my local, my CPU wasn't enough at all\n\nI already created my model with piece of data but I couldn't use with all data set because of above reason.\n\nI would be very grateful if someone could guide me.",
      "votes": null
    },
    {
      "id": "1619528",
      "postDate": "12/16/2021 02:41:11",
      "content": "<p>Hello <a href=\"https://www.kaggle.com/atilaysamiloglu\" target=\"_blank\">@atilaysamiloglu</a>, have you tried the memory optimization function that has been around for a while?<br>\n\"reduce_mem_usage\"<br>\nHere is my Notebook with implementation, for another competition but the concept is the same.<br>\n<a href=\"https://www.kaggle.com/cv13j0/tps-dec21-beginner-code\" target=\"_blank\">Example Notebook</a></p>",
      "rawMarkdown": "Hello @atilaysamiloglu, have you tried the memory optimization function that has been around for a while?\n\"reduce_mem_usage\"\nHere is my Notebook with implementation, for another competition but the concept is the same.\n[Example Notebook](https://www.kaggle.com/cv13j0/tps-dec21-beginner-code)",
      "votes": null
    },
    {
      "id": "1619917",
      "postDate": "12/16/2021 09:14:22",
      "content": "<p>Hi thanks for answering</p>\n<p>Yes I just tried that notebook but still I got same error ;<br>\n'Your notebook tried to allocate more memory than is available. It has restarted.'</p>",
      "rawMarkdown": "Hi thanks for answering\n\nYes I just tried that notebook but still I got same error ;\n'Your notebook tried to allocate more memory than is available. It has restarted.'",
      "votes": null
    },
    {
      "id": "1620188",
      "postDate": "12/16/2021 14:14:36",
      "content": "<p>Interesting, I never had any issues on my runs after I implemented the \"reduce_mem_usage\"</p>",
      "rawMarkdown": "Interesting, I never had any issues on my runs after I implemented the \"reduce_mem_usage\"",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1619528,
      "author_name": "cv13j0",
      "author_url": "",
      "post_date": "12/16/2021 02:41:11",
      "content": "<p>Hello <a href=\"https://www.kaggle.com/atilaysamiloglu\" target=\"_blank\">@atilaysamiloglu</a>, have you tried the memory optimization function that has been around for a while?<br>\n\"reduce_mem_usage\"<br>\nHere is my Notebook with implementation, for another competition but the concept is the same.<br>\n<a href=\"https://www.kaggle.com/cv13j0/tps-dec21-beginner-code\" target=\"_blank\">Example Notebook</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 1619917,
          "author_name": "atilaysamiloglu",
          "author_url": "",
          "post_date": "12/16/2021 09:14:22",
          "content": "<p>Hi thanks for answering</p>\n<p>Yes I just tried that notebook but still I got same error ;<br>\n'Your notebook tried to allocate more memory than is available. It has restarted.'</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1620188,
          "author_name": "cv13j0",
          "author_url": "",
          "post_date": "12/16/2021 14:14:36",
          "content": "<p>Interesting, I never had any issues on my runs after I implemented the \"reduce_mem_usage\"</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1619461": "Hi guys, I am new to ML and kaggle competitons ?\n\nMy questions are;\n\n1-  When I use this large data set on Kaggle, always I got allocate memory errors\n2- When I use on my local, my CPU wasn't enough at all\n\nI already created my model with piece of data but I couldn't use with all data set because of above reason.\n\nI would be very grateful if someone could guide me.",
    "1619528": "Hello @atilaysamiloglu, have you tried the memory optimization function that has been around for a while?\n\"reduce_mem_usage\"\nHere is my Notebook with implementation, for another competition but the concept is the same.\n[Example Notebook](https://www.kaggle.com/cv13j0/tps-dec21-beginner-code)",
    "1619917": "Hi thanks for answering\n\nYes I just tried that notebook but still I got same error ;\n'Your notebook tried to allocate more memory than is available. It has restarted.'",
    "1620188": "Interesting, I never had any issues on my runs after I implemented the \"reduce_mem_usage\""
  },
  "source": "meta"
}