{
  "id": 459511,
  "title": "I couldn't rectify notebook out of memory. Everytime error occurs it accounted as submission. My model accuracy is 92.536. thanks.",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/459511",
  "author_name": "",
  "post_date": "2023-12-05T14:17:46.781222400Z",
  "votes": -5,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Thanks. It is very interesting competition.</p>",
  "messages": [
    {
      "id": "2549773",
      "postDate": "12/05/2023 14:17:46",
      "content": "<p>Thanks. It is very interesting competition.</p>",
      "rawMarkdown": "Thanks. It is very interesting competition.",
      "votes": null
    },
    {
      "id": "2549789",
      "postDate": "12/05/2023 14:23:51",
      "content": "<p>When you have large datasets like this, it's a good idea to do things per batch to avoid OOM (Out of Memory) errors. </p>\n<p>In this competition you could do everything per series_id. i.e.</p>\n<p><strong>Data Preprocessing</strong> -&gt; <strong>Feature_Engineering</strong> -&gt; <strong>Generating_Predictions.</strong></p>\n<p>Since there are around 200 series ids in your test data, your memory requirement is roughly reduced by 200 times.</p>",
      "rawMarkdown": "When you have large datasets like this, it's a good idea to do things per batch to avoid OOM (Out of Memory) errors. \n\nIn this competition you could do everything per series_id. i.e.\n\n**Data Preprocessing** -> **Feature_Engineering** -> **Generating_Predictions.**\n\nSince there are around 200 series ids in your test data, your memory requirement is roughly reduced by 200 times.",
      "votes": null
    },
    {
      "id": "2550142",
      "postDate": "12/05/2023 20:22:04",
      "content": "<p>You can try like this code next time if meet same problem:</p>\n<pre><code> ():\n    test = pd.read_parquet(, filters=[(,,idx)])\n    testX = create_features(test)\n    testy = model.predict(testX)\n     testy\n\nseries_id  = pd.read_parquet(, columns=[])\nseries_id = series_id.series_id.unique()\n\ntests = []\n idx  series_id:\n    tests.append(predict_test(idx))\n</code></pre>",
      "rawMarkdown": "You can try like this code next time if meet same problem:\n\n```Python\ndef predict_test(idx):\n    test = pd.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet', filters=[('series_id','=',idx)])\n    testX = create_features(test)\n    testy = model.predict(testX)\n    return testy\n\nseries_id  = pd.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet', columns=['series_id'])\nseries_id = series_id.series_id.unique()\n\ntests = []\nfor idx in series_id:\n    tests.append(predict_test(idx))\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2549789,
      "author_name": "nikhilmishradev",
      "author_url": "",
      "post_date": "12/05/2023 14:23:51",
      "content": "<p>When you have large datasets like this, it's a good idea to do things per batch to avoid OOM (Out of Memory) errors. </p>\n<p>In this competition you could do everything per series_id. i.e.</p>\n<p><strong>Data Preprocessing</strong> -&gt; <strong>Feature_Engineering</strong> -&gt; <strong>Generating_Predictions.</strong></p>\n<p>Since there are around 200 series ids in your test data, your memory requirement is roughly reduced by 200 times.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2550142,
      "author_name": "kerrysun",
      "author_url": "",
      "post_date": "12/05/2023 20:22:04",
      "content": "<p>You can try like this code next time if meet same problem:</p>\n<pre><code> ():\n    test = pd.read_parquet(, filters=[(,,idx)])\n    testX = create_features(test)\n    testy = model.predict(testX)\n     testy\n\nseries_id  = pd.read_parquet(, columns=[])\nseries_id = series_id.series_id.unique()\n\ntests = []\n idx  series_id:\n    tests.append(predict_test(idx))\n</code></pre>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2549773": "Thanks. It is very interesting competition.",
    "2549789": "When you have large datasets like this, it's a good idea to do things per batch to avoid OOM (Out of Memory) errors. \n\nIn this competition you could do everything per series_id. i.e.\n\n**Data Preprocessing** -> **Feature_Engineering** -> **Generating_Predictions.**\n\nSince there are around 200 series ids in your test data, your memory requirement is roughly reduced by 200 times.",
    "2550142": "You can try like this code next time if meet same problem:\n\n```Python\ndef predict_test(idx):\n    test = pd.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet', filters=[('series_id','=',idx)])\n    testX = create_features(test)\n    testy = model.predict(testX)\n    return testy\n\nseries_id  = pd.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet', columns=['series_id'])\nseries_id = series_id.series_id.unique()\n\ntests = []\nfor idx in series_id:\n    tests.append(predict_test(idx))\n```"
  },
  "source": "meta"
}