{
  "id": 120591,
  "title": "The kernel keeps died due to out-of-memory problem, how to solve?",
  "url": "/competitions/ashrae-energy-prediction/discussion/120591",
  "author_name": "",
  "post_date": "2019-12-07T07:55:53.249964100Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>The dataset for this competition is large, and the 16GB RAM kaggle kernel provides easily crushed during training and testing. I've applied reduce_mem_usage method:</p>\n\n<p><a href=\"https://www.kaggle.com/laurencelin/asharepredictor/edit\">my kernel</a></p>\n\n<p>But it's not enough, during the preprocessing of test data every time the kernel failed before I do the prediction. \nI've seen some others on kaggle recommend to run on gcloud, it seems appealing, but I'm not familiar to gcloud platform, and there are lots of service that I don't know which one is the simplest solution of my problem. </p>\n\n<p>Should I create a jupyter notebook on google cloud, and upload the dataset to it, then train model and all the computation stuff on cloud machine? However, when I create a new jupyter notebook on gcloud, it have only 15GB RAM... I'm not sure if it's because I'm using free gcloud service. How could I leverage google cloud to work on kaggle competition in the most simple way?</p>\n\n<p>Thank you for any help!</p>",
  "messages": [
    {
      "id": "689637",
      "postDate": "12/07/2019 07:55:53",
      "content": "<p>The dataset for this competition is large, and the 16GB RAM kaggle kernel provides easily crushed during training and testing. I've applied reduce_mem_usage method:</p>\n\n<p><a href=\"https://www.kaggle.com/laurencelin/asharepredictor/edit\">my kernel</a></p>\n\n<p>But it's not enough, during the preprocessing of test data every time the kernel failed before I do the prediction. \nI've seen some others on kaggle recommend to run on gcloud, it seems appealing, but I'm not familiar to gcloud platform, and there are lots of service that I don't know which one is the simplest solution of my problem. </p>\n\n<p>Should I create a jupyter notebook on google cloud, and upload the dataset to it, then train model and all the computation stuff on cloud machine? However, when I create a new jupyter notebook on gcloud, it have only 15GB RAM... I'm not sure if it's because I'm using free gcloud service. How could I leverage google cloud to work on kaggle competition in the most simple way?</p>\n\n<p>Thank you for any help!</p>",
      "rawMarkdown": "The dataset for this competition is large, and the 16GB RAM kaggle kernel provides easily crushed during training and testing. I've applied reduce_mem_usage method:\n\n [my kernel](https://www.kaggle.com/laurencelin/asharepredictor/edit)\n\nBut it's not enough, during the preprocessing of test data every time the kernel failed before I do the prediction. \nI've seen some others on kaggle recommend to run on gcloud, it seems appealing, but I'm not familiar to gcloud platform, and there are lots of service that I don't know which one is the simplest solution of my problem. \n\nShould I create a jupyter notebook on google cloud, and upload the dataset to it, then train model and all the computation stuff on cloud machine? However, when I create a new jupyter notebook on gcloud, it have only 15GB RAM... I'm not sure if it's because I'm using free gcloud service. How could I leverage google cloud to work on kaggle competition in the most simple way?\n\nThank you for any help!",
      "votes": null
    },
    {
      "id": "689644",
      "postDate": "12/07/2019 08:06:07",
      "content": "<p>can't open the kernel.</p>\n\n<p>btw if you want to run on kaggle:\n- reduce memory of every dataset before join.\n- preprocess in another kernel train and test and save as hdf, pickle\n- load train before, fit, delete train , load test and predict.\n- delete all unnecessary variable\n- gc.collect every time\n- select a maximum of 26-29 column to train if you use lgbm/xgb otherwise it will crash.</p>\n\n<p>For Gcloud , when it will run out of memory it will ask you if you want more ram and you will have 25gb</p>",
      "rawMarkdown": "can't open the kernel.\n\nbtw if you want to run on kaggle:\n- reduce memory of every dataset before join.\n- preprocess in another kernel train and test and save as hdf, pickle\n- load train before, fit, delete train , load test and predict.\n- delete all unnecessary variable\n- gc.collect every time\n- select a maximum of 26-29 column to train if you use lgbm/xgb otherwise it will crash.\n\nFor Gcloud , when it will run out of memory it will ask you if you want more ram and you will have 25gb",
      "votes": null
    },
    {
      "id": "689941",
      "postDate": "12/07/2019 17:48:19",
      "content": "<p>Reasonable use of reduing memory function can help a lot</p>",
      "rawMarkdown": "Reasonable use of reduing memory function can help a lot",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 689644,
      "author_name": "stenford23",
      "author_url": "",
      "post_date": "12/07/2019 08:06:07",
      "content": "<p>can't open the kernel.</p>\n\n<p>btw if you want to run on kaggle:\n- reduce memory of every dataset before join.\n- preprocess in another kernel train and test and save as hdf, pickle\n- load train before, fit, delete train , load test and predict.\n- delete all unnecessary variable\n- gc.collect every time\n- select a maximum of 26-29 column to train if you use lgbm/xgb otherwise it will crash.</p>\n\n<p>For Gcloud , when it will run out of memory it will ask you if you want more ram and you will have 25gb</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 689941,
      "author_name": "dreamvill",
      "author_url": "",
      "post_date": "12/07/2019 17:48:19",
      "content": "<p>Reasonable use of reduing memory function can help a lot</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "689637": "The dataset for this competition is large, and the 16GB RAM kaggle kernel provides easily crushed during training and testing. I've applied reduce_mem_usage method:\n\n [my kernel](https://www.kaggle.com/laurencelin/asharepredictor/edit)\n\nBut it's not enough, during the preprocessing of test data every time the kernel failed before I do the prediction. \nI've seen some others on kaggle recommend to run on gcloud, it seems appealing, but I'm not familiar to gcloud platform, and there are lots of service that I don't know which one is the simplest solution of my problem. \n\nShould I create a jupyter notebook on google cloud, and upload the dataset to it, then train model and all the computation stuff on cloud machine? However, when I create a new jupyter notebook on gcloud, it have only 15GB RAM... I'm not sure if it's because I'm using free gcloud service. How could I leverage google cloud to work on kaggle competition in the most simple way?\n\nThank you for any help!",
    "689644": "can't open the kernel.\n\nbtw if you want to run on kaggle:\n- reduce memory of every dataset before join.\n- preprocess in another kernel train and test and save as hdf, pickle\n- load train before, fit, delete train , load test and predict.\n- delete all unnecessary variable\n- gc.collect every time\n- select a maximum of 26-29 column to train if you use lgbm/xgb otherwise it will crash.\n\nFor Gcloud , when it will run out of memory it will ask you if you want more ram and you will have 25gb",
    "689941": "Reasonable use of reduing memory function can help a lot"
  },
  "source": "meta"
}