{
  "id": 113581,
  "title": "Tips for dealing with memory limitations",
  "url": "/competitions/ashrae-energy-prediction/discussion/113581",
  "author_name": "",
  "post_date": "2019-10-20T14:50:02.154274700Z",
  "votes": 26,
  "comment_count": 6,
  "views": 0,
  "content": "<p>1) Always experiment with a deterministic sample from the train set:\n<code>train_df = train_df.sample(10**6, random_state=0)</code></p>\n\n<p>2) When it comes to submission, you can use this code to make chunk by chunk predictions:\n```\ndef load_test(chunk_size):\n    for df in pd.read_csv('data/test.csv', chunksize=chunk_size):\n        yield df</p>\n\n<p>test_ids = np.array([], dtype=np.int32)\npreds = np.array([], dtype=np.float32)</p>\n\n<p>for test_df in tqdm_notebook(load_test(CHUNK_SIZE)):\n    test_df = get_features(test_df)</p>\n\n<pre><code>chunk_preds = models[0].predict(test_df[features])\nfor model in models[1:]:\n    chunk_preds += model.predict(test_df[features])\nchunk_preds /= len(models)\n\npreds = np.append(preds, chunk_preds)\ntest_ids = np.append(test_ids, test_df[\"row_id\"].values)\n</code></pre>\n\n<p>submission = pd.DataFrame({\n    \"row_id\": test_ids,\n    \"meter_reading\": np.expm1(preds),\n})\n```</p>",
  "messages": [
    {
      "id": "653498",
      "postDate": "10/20/2019 14:50:02",
      "content": "<p>1) Always experiment with a deterministic sample from the train set:\n<code>train_df = train_df.sample(10**6, random_state=0)</code></p>\n\n<p>2) When it comes to submission, you can use this code to make chunk by chunk predictions:\n```\ndef load_test(chunk_size):\n    for df in pd.read_csv('data/test.csv', chunksize=chunk_size):\n        yield df</p>\n\n<p>test_ids = np.array([], dtype=np.int32)\npreds = np.array([], dtype=np.float32)</p>\n\n<p>for test_df in tqdm_notebook(load_test(CHUNK_SIZE)):\n    test_df = get_features(test_df)</p>\n\n<pre><code>chunk_preds = models[0].predict(test_df[features])\nfor model in models[1:]:\n    chunk_preds += model.predict(test_df[features])\nchunk_preds /= len(models)\n\npreds = np.append(preds, chunk_preds)\ntest_ids = np.append(test_ids, test_df[\"row_id\"].values)\n</code></pre>\n\n<p>submission = pd.DataFrame({\n    \"row_id\": test_ids,\n    \"meter_reading\": np.expm1(preds),\n})\n```</p>",
      "rawMarkdown": "1) Always experiment with a deterministic sample from the train set:\n`train_df = train_df.sample(10**6, random_state=0)`\n\n2) When it comes to submission, you can use this code to make chunk by chunk predictions:\n```\ndef load_test(chunk_size):\n    for df in pd.read_csv('data/test.csv', chunksize=chunk_size):\n        yield df\n\ntest_ids = np.array([], dtype=np.int32)\npreds = np.array([], dtype=np.float32)\n\nfor test_df in tqdm_notebook(load_test(CHUNK_SIZE)):\n    test_df = get_features(test_df)\n\n    chunk_preds = models[0].predict(test_df[features])\n    for model in models[1:]:\n        chunk_preds += model.predict(test_df[features])\n    chunk_preds /= len(models)\n\n    preds = np.append(preds, chunk_preds)\n    test_ids = np.append(test_ids, test_df[\"row_id\"].values)\n\nsubmission = pd.DataFrame({\n    \"row_id\": test_ids,\n    \"meter_reading\": np.expm1(preds),\n})\n```",
      "votes": null
    },
    {
      "id": "653530",
      "postDate": "10/20/2019 15:39:29",
      "content": "<p>Thanks for this post! I can add another tip:</p>\n\n<ul>\n<li>Using Colab with 25 GB of RAM :smile:</li>\n</ul>",
      "rawMarkdown": "Thanks for this post! I can add another tip:\n\n- Using Colab with 25 GB of RAM :smile:",
      "votes": null
    },
    {
      "id": "653534",
      "postDate": "10/20/2019 15:42:55",
      "content": "<p>Nice! What are the specs and run time limit of colab? Is it better than Kaggle kernels only in terms of RAM?</p>",
      "rawMarkdown": "Nice! What are the specs and run time limit of colab? Is it better than Kaggle kernels only in terms of RAM?",
      "votes": null
    },
    {
      "id": "653539",
      "postDate": "10/20/2019 15:49:37",
      "content": "<p>Very Helpful...\nThanks <a href=\"/divrikwicky\">@divrikwicky</a> </p>",
      "rawMarkdown": "Very Helpful...\nThanks @divrikwicky",
      "votes": null
    },
    {
      "id": "653543",
      "postDate": "10/20/2019 16:03:03",
      "content": "<p>The runtime limit should be 9h, so the same as Kaggle kernels. The cool thing is that there's no GPU per week limitation at the moment.</p>\n\n<p>It's also pretty simple to work with Colab and Drive to read and write files. It requires less time rather than connect through Kaggle's API.</p>",
      "rawMarkdown": "The runtime limit should be 9h, so the same as Kaggle kernels. The cool thing is that there's no GPU per week limitation at the moment.\n\nIt's also pretty simple to work with Colab and Drive to read and write files. It requires less time rather than connect through Kaggle's API.",
      "votes": null
    },
    {
      "id": "654399",
      "postDate": "10/21/2019 20:21:11",
      "content": "<p>zram enabled working with 8 GB physical memory</p>",
      "rawMarkdown": "zram enabled working with 8 GB physical memory",
      "votes": null
    },
    {
      "id": "655341",
      "postDate": "10/23/2019 00:23:41",
      "content": "<p>Thanks for the tip!!!</p>\n\n<p>Some re-factoring ( maybe not 100% correct  depeding on how you are processing with the models array and how predcition works (assuming on raw meter readings for now):</p>\n\n<p><code>\nfor test_df in pd.read_csv('data/test.csv', chunksize=int(41697601/10)):\n    test_df = get_features(test_df)\n    for model in models:\n        test_df['meter_reading']= test_df.apply(model.predict)\n    with open('submission.csv', 'a') as f:\n                test_df[['row_id','meter_reading']].to_csv(f, header=False)\n</code></p>",
      "rawMarkdown": "Thanks for the tip!!!\n\nSome re-factoring ( maybe not 100% correct  depeding on how you are processing with the models array and how predcition works (assuming on raw meter readings for now):\n\n\n```\nfor test_df in pd.read_csv('data/test.csv', chunksize=int(41697601/10)):\n    test_df = get_features(test_df)\n    for model in models:\n        test_df['meter_reading']= test_df.apply(model.predict)\n    with open('submission.csv', 'a') as f:\n                test_df[['row_id','meter_reading']].to_csv(f, header=False)\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 653530,
      "author_name": "raimonds1993",
      "author_url": "",
      "post_date": "10/20/2019 15:39:29",
      "content": "<p>Thanks for this post! I can add another tip:</p>\n\n<ul>\n<li>Using Colab with 25 GB of RAM :smile:</li>\n</ul>",
      "votes": null,
      "replies": [
        {
          "id": 653534,
          "author_name": "aerdem4",
          "author_url": "",
          "post_date": "10/20/2019 15:42:55",
          "content": "<p>Nice! What are the specs and run time limit of colab? Is it better than Kaggle kernels only in terms of RAM?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 653543,
          "author_name": "raimonds1993",
          "author_url": "",
          "post_date": "10/20/2019 16:03:03",
          "content": "<p>The runtime limit should be 9h, so the same as Kaggle kernels. The cool thing is that there's no GPU per week limitation at the moment.</p>\n\n<p>It's also pretty simple to work with Colab and Drive to read and write files. It requires less time rather than connect through Kaggle's API.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 653539,
      "author_name": "veeralakrishna",
      "author_url": "",
      "post_date": "10/20/2019 15:49:37",
      "content": "<p>Very Helpful...\nThanks <a href=\"/divrikwicky\">@divrikwicky</a> </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 654399,
      "author_name": "michalpilch",
      "author_url": "",
      "post_date": "10/21/2019 20:21:11",
      "content": "<p>zram enabled working with 8 GB physical memory</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 655341,
      "author_name": "ptolemy",
      "author_url": "",
      "post_date": "10/23/2019 00:23:41",
      "content": "<p>Thanks for the tip!!!</p>\n\n<p>Some re-factoring ( maybe not 100% correct  depeding on how you are processing with the models array and how predcition works (assuming on raw meter readings for now):</p>\n\n<p><code>\nfor test_df in pd.read_csv('data/test.csv', chunksize=int(41697601/10)):\n    test_df = get_features(test_df)\n    for model in models:\n        test_df['meter_reading']= test_df.apply(model.predict)\n    with open('submission.csv', 'a') as f:\n                test_df[['row_id','meter_reading']].to_csv(f, header=False)\n</code></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "653498": "1) Always experiment with a deterministic sample from the train set:\n`train_df = train_df.sample(10**6, random_state=0)`\n\n2) When it comes to submission, you can use this code to make chunk by chunk predictions:\n```\ndef load_test(chunk_size):\n    for df in pd.read_csv('data/test.csv', chunksize=chunk_size):\n        yield df\n\ntest_ids = np.array([], dtype=np.int32)\npreds = np.array([], dtype=np.float32)\n\nfor test_df in tqdm_notebook(load_test(CHUNK_SIZE)):\n    test_df = get_features(test_df)\n\n    chunk_preds = models[0].predict(test_df[features])\n    for model in models[1:]:\n        chunk_preds += model.predict(test_df[features])\n    chunk_preds /= len(models)\n\n    preds = np.append(preds, chunk_preds)\n    test_ids = np.append(test_ids, test_df[\"row_id\"].values)\n\nsubmission = pd.DataFrame({\n    \"row_id\": test_ids,\n    \"meter_reading\": np.expm1(preds),\n})\n```",
    "653530": "Thanks for this post! I can add another tip:\n\n- Using Colab with 25 GB of RAM :smile:",
    "653534": "Nice! What are the specs and run time limit of colab? Is it better than Kaggle kernels only in terms of RAM?",
    "653539": "Very Helpful...\nThanks @divrikwicky",
    "653543": "The runtime limit should be 9h, so the same as Kaggle kernels. The cool thing is that there's no GPU per week limitation at the moment.\n\nIt's also pretty simple to work with Colab and Drive to read and write files. It requires less time rather than connect through Kaggle's API.",
    "654399": "zram enabled working with 8 GB physical memory",
    "655341": "Thanks for the tip!!!\n\nSome re-factoring ( maybe not 100% correct  depeding on how you are processing with the models array and how predcition works (assuming on raw meter readings for now):\n\n\n```\nfor test_df in pd.read_csv('data/test.csv', chunksize=int(41697601/10)):\n    test_df = get_features(test_df)\n    for model in models:\n        test_df['meter_reading']= test_df.apply(model.predict)\n    with open('submission.csv', 'a') as f:\n                test_df[['row_id','meter_reading']].to_csv(f, header=False)\n```"
  },
  "source": "meta"
}