{
  "id": 437600,
  "title": "How do I avoid Error: \"Notebook out of Memory\"?",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/437600",
  "author_name": "JoJolove",
  "post_date": "2023-09-07T11:58:25.424000",
  "votes": 6,
  "comment_count": 8,
  "views": 0,
  "content": "<p>Hello Kagglers,</p>\n<p>I had already several submissions failed because my Notebook run into this error: \"Notebook Out of Memory<br>\nYour notebook requested more memory (RAM) than is available.\"</p>\n<p>Even when running very simple code and basically just loading in the test_series.parquet data.<br>\nFor example this was on of my submissions that failed:</p>\n<p>`import pandas as pd<br>\nimport numpy as np<br>\nimport seaborn as sns<br>\nimport matplotlib.pyplot as plt<br>\nimport os</p>\n<p>BASE = \"data/\" if \"data\" in os.listdir(\"./\") else \"/kaggle/input/child-mind-institute-detect-sleep-states/\"</p>\n<p>test = pd.read_parquet(BASE+\"/test_series.parquet\")</p>\n<p>test['date'] = test['timestamp'].str.split('T', expand=True)[0]<br>\ntest['time'] = test['timestamp'].str.split('T', expand=True)[1].str.split('-', expand=True)[0]<br>\ntest['timestamp'] = pd.to_datetime(test['date']+' '+test['time'])<br>\ntest['hour'] = test['timestamp'].dt.hour</p>\n<p>onset = test[test[\"hour\"].between(22,24)].set_index(\"series_id\")[\"step\"]<br>\nwakeup = test[test[\"hour\"].between(7,9)].set_index(\"series_id\")[\"step\"]</p>\n<p>sub = pd.concat([<br>\n    onset.reset_index().assign(event=\"onset\"),<br>\n    wakeup.reset_index().assign(event=\"wakeup\"),<br>\n])<br>\nsub[\"score\"] = 0.33<br>\nsub.sort_values([\"series_id\",\"event\"], ascending=[0,1], inplace=True)<br>\nsub.reset_index(drop=True, inplace=True)<br>\nsub[\"row_id\"] = sub.index<br>\nsub = sub[[\"row_id\",\"series_id\",\"step\",\"event\",\"score\"]]<br>\nsub.to_csv(\"submission.csv\", index=False)<br>\nsub<br>\n`</p>",
  "messages": [
    {
      "id": 2427960,
      "postDate": "2023-09-07T15:02:59.607Z",
      "content": "<p>This block looks like the issue:</p>\n<pre><code>test = pd(BASE+) \ntest = test(, expand=True)\ntest = test(, expand=True)(, expand=True)\ntest = pd(test++test)\ntest = test\n</code></pre>\n<p>The following alternative is faster and requires less memory.  Note that the resulting dataframe won't have <code>timestamp</code>, <code>date</code>, or <code>time</code> columns, but you could modify the code if you want those.  I chose to omit them since it looked like your ultimate goal was the <code>hour</code> column.</p>\n<pre><code>import polars as pl\ntest = (pl(BASE+)\n          (\n              (pl()\n                 (pl, )\n                 ()\n                 ()\n              )\n          )\n          ()\n          ()\n          ()\n)\n</code></pre>",
      "rawMarkdown": "This block looks like the issue:\n\n```\ntest = pd.read_parquet(BASE+\"/test_series.parquet\") \ntest['date'] = test['timestamp'].str.split('T', expand=True)[0]\ntest['time'] = test['timestamp'].str.split('T', expand=True)[1].str.split('-', expand=True)[0]\ntest['timestamp'] = pd.to_datetime(test['date']+' '+test['time'])\ntest['hour'] = test['timestamp'].dt.hour\n```\n\nThe following alternative is faster and requires less memory.  Note that the resulting dataframe won't have `timestamp`, `date`, or `time` columns, but you could modify the code if you want those.  I chose to omit them since it looked like your ultimate goal was the `hour` column.\n```\nimport polars as pl\ntest = (pl.scan_parquet(BASE+\"/test_series.parquet\")\n          .with_columns(\n              (pl.col(\"timestamp\")\n                 .str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\")\n                 .dt.hour()\n                 .alias(\"hour\")\n              )\n          )\n          .drop(\"timestamp\")\n          .collect()\n          .to_pandas()\n)\n```",
      "votes": 7,
      "replies": [
        {
          "id": 2428001,
          "postDate": "2023-09-07T15:37:50.893Z",
          "content": "<p>Thank you Quillen!<br>\nThat looks like a highly memory-optimized code! </p>",
          "rawMarkdown": "Thank you Quillen!\nThat looks like a highly memory-optimized code! ",
          "votes": 1
        },
        {
          "id": 2428012,
          "postDate": "2023-09-07T15:46:43.003Z",
          "rawMarkdown": "",
          "votes": 2,
          "isDeleted": true
        }
      ]
    },
    {
      "id": 2427682,
      "postDate": "2023-09-07T11:58:25.423Z",
      "content": "<p>Hello Kagglers,</p>\n<p>I had already several submissions failed because my Notebook run into this error: \"Notebook Out of Memory<br>\nYour notebook requested more memory (RAM) than is available.\"</p>\n<p>Even when running very simple code and basically just loading in the test_series.parquet data.<br>\nFor example this was on of my submissions that failed:</p>\n<p>`import pandas as pd<br>\nimport numpy as np<br>\nimport seaborn as sns<br>\nimport matplotlib.pyplot as plt<br>\nimport os</p>\n<p>BASE = \"data/\" if \"data\" in os.listdir(\"./\") else \"/kaggle/input/child-mind-institute-detect-sleep-states/\"</p>\n<p>test = pd.read_parquet(BASE+\"/test_series.parquet\")</p>\n<p>test['date'] = test['timestamp'].str.split('T', expand=True)[0]<br>\ntest['time'] = test['timestamp'].str.split('T', expand=True)[1].str.split('-', expand=True)[0]<br>\ntest['timestamp'] = pd.to_datetime(test['date']+' '+test['time'])<br>\ntest['hour'] = test['timestamp'].dt.hour</p>\n<p>onset = test[test[\"hour\"].between(22,24)].set_index(\"series_id\")[\"step\"]<br>\nwakeup = test[test[\"hour\"].between(7,9)].set_index(\"series_id\")[\"step\"]</p>\n<p>sub = pd.concat([<br>\n    onset.reset_index().assign(event=\"onset\"),<br>\n    wakeup.reset_index().assign(event=\"wakeup\"),<br>\n])<br>\nsub[\"score\"] = 0.33<br>\nsub.sort_values([\"series_id\",\"event\"], ascending=[0,1], inplace=True)<br>\nsub.reset_index(drop=True, inplace=True)<br>\nsub[\"row_id\"] = sub.index<br>\nsub = sub[[\"row_id\",\"series_id\",\"step\",\"event\",\"score\"]]<br>\nsub.to_csv(\"submission.csv\", index=False)<br>\nsub<br>\n`</p>",
      "rawMarkdown": "Hello Kagglers,\n\nI had already several submissions failed because my Notebook run into this error: \"Notebook Out of Memory\nYour notebook requested more memory (RAM) than is available.\"\n\nEven when running very simple code and basically just loading in the test_series.parquet data.\nFor example this was on of my submissions that failed:\n\n`import pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport os\n\nBASE = \"data/\" if \"data\" in os.listdir(\"./\") else \"/kaggle/input/child-mind-institute-detect-sleep-states/\"\n\ntest = pd.read_parquet(BASE+\"/test_series.parquet\")\n\ntest['date'] = test['timestamp'].str.split('T', expand=True)[0]\ntest['time'] = test['timestamp'].str.split('T', expand=True)[1].str.split('-', expand=True)[0]\ntest['timestamp'] = pd.to_datetime(test['date']+' '+test['time'])\ntest['hour'] = test['timestamp'].dt.hour\n\nonset = test[test[\"hour\"].between(22,24)].set_index(\"series_id\")[\"step\"]\nwakeup = test[test[\"hour\"].between(7,9)].set_index(\"series_id\")[\"step\"]\n\nsub = pd.concat([\n    onset.reset_index().assign(event=\"onset\"),\n    wakeup.reset_index().assign(event=\"wakeup\"),\n])\nsub[\"score\"] = 0.33\nsub.sort_values([\"series_id\",\"event\"], ascending=[0,1], inplace=True)\nsub.reset_index(drop=True, inplace=True)\nsub[\"row_id\"] = sub.index\nsub = sub[[\"row_id\",\"series_id\",\"step\",\"event\",\"score\"]]\nsub.to_csv(\"submission.csv\", index=False)\nsub\n`",
      "votes": 5
    },
    {
      "id": 2428484,
      "postDate": "2023-09-07T22:59:11.063Z",
      "content": "<p>I had the same problem but solved it using dask.</p>\n<p>Codes with problems:</p>\n<pre><code>import pandas  pd\n\n\nte = pd.read\nte.timestamp = pd..dt.hour\n</code></pre>\n<p>Solution:</p>\n<pre><code>import dask\nimport dask as da\nimport dask as \n\n\nte = (BASE+)\nte = (te, format=)\n</code></pre>",
      "rawMarkdown": "I had the same problem but solved it using dask.\n\nCodes with problems:\n```\nimport pandas as pd\n\n\nte = pd.read_parquet(BASE+\"/test_series.parquet\")\nte.timestamp = pd.to_datetime(te.timestamp, format='%Y-%m-%dT%H:%M:%S-%f').dt.hour\n```\n\n\nSolution:\n```\nimport dask\nimport dask.array as da\nimport dask.dataframe as dd\n\n\nte = dd.read_parquet(BASE+\"/test_series.parquet\")\nte['timestamp'] = dd.to_datetime(te['timestamp'], format='%Y-%m-%dT%H:%M:%S-%f').dt.hour\n```",
      "votes": 6
    },
    {
      "id": 2427710,
      "postDate": "2023-09-07T12:24:18.380Z",
      "content": "<p>I get the same error too</p>",
      "rawMarkdown": "I get the same error too",
      "votes": 1
    },
    {
      "id": 2478634,
      "postDate": "2023-10-12T05:53:09.627Z",
      "content": "<p>u can also look here <a href=\"https://www.kaggle.com/code/andradaolteanu/zzz-good-night-sleep-with-80-memory-reduction\" target=\"_blank\">https://www.kaggle.com/code/andradaolteanu/zzz-good-night-sleep-with-80-memory-reduction</a></p>",
      "rawMarkdown": "u can also look here https://www.kaggle.com/code/andradaolteanu/zzz-good-night-sleep-with-80-memory-reduction"
    },
    {
      "id": 2472577,
      "postDate": "2023-10-07T12:27:06.213Z",
      "content": "<p>I'm a novice here, but for anyone using R instead of python, the \"data.table\" library is a lifesaver from a memory standpoint because it manipulates large tables in-place, instead of creating copies.</p>",
      "rawMarkdown": "I'm a novice here, but for anyone using R instead of python, the \"data.table\" library is a lifesaver from a memory standpoint because it manipulates large tables in-place, instead of creating copies."
    },
    {
      "id": 2428549,
      "postDate": "2023-09-08T02:11:03.600Z",
      "content": "<p>For the first time, tried <code>LazyLoading</code> with <a href=\"https://www.pola.rs/\" target=\"_blank\">Pola.rs</a> today. Loved it and it's fast ⚡</p>\n<pre><code>df_counts = (pl.scan_parquet()\n    .group_by(, maintain_order=)\n    .count()\n    .collect())\ndf_counts.head()\n</code></pre>",
      "rawMarkdown": "For the first time, tried `LazyLoading` with [Pola.rs](https://www.pola.rs/) today. Loved it and it's fast ⚡\n\n```python\ndf_counts = (pl.scan_parquet(f\"{data_directory}/train_series.parquet\")\n    .group_by(\"series_id\", maintain_order=True)\n    .count()\n    .collect())\ndf_counts.head()\n```",
      "votes": 3,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 2427960,
      "author_name": "Quillen",
      "author_url": "",
      "post_date": "2023-09-07T15:02:59.607000",
      "content": "<p>This block looks like the issue:</p>\n<pre><code>test = pd(BASE+) \ntest = test(, expand=True)\ntest = test(, expand=True)(, expand=True)\ntest = pd(test++test)\ntest = test\n</code></pre>\n<p>The following alternative is faster and requires less memory.  Note that the resulting dataframe won't have <code>timestamp</code>, <code>date</code>, or <code>time</code> columns, but you could modify the code if you want those.  I chose to omit them since it looked like your ultimate goal was the <code>hour</code> column.</p>\n<pre><code>import polars as pl\ntest = (pl(BASE+)\n          (\n              (pl()\n                 (pl, )\n                 ()\n                 ()\n              )\n          )\n          ()\n          ()\n          ()\n)\n</code></pre>",
      "votes": 7,
      "replies": [
        {
          "id": 2428001,
          "author_name": "JoJolove",
          "author_url": "",
          "post_date": "2023-09-07T15:37:50.893000",
          "content": "<p>Thank you Quillen!<br>\nThat looks like a highly memory-optimized code! </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2428012,
          "author_name": "",
          "author_url": "",
          "post_date": "2023-09-07T15:46:43.003000",
          "content": "",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2428484,
      "author_name": "Funmatu Funmatu",
      "author_url": "",
      "post_date": "2023-09-07T22:59:11.063000",
      "content": "<p>I had the same problem but solved it using dask.</p>\n<p>Codes with problems:</p>\n<pre><code>import pandas  pd\n\n\nte = pd.read\nte.timestamp = pd..dt.hour\n</code></pre>\n<p>Solution:</p>\n<pre><code>import dask\nimport dask as da\nimport dask as \n\n\nte = (BASE+)\nte = (te, format=)\n</code></pre>",
      "votes": 6,
      "replies": []
    },
    {
      "id": 2427710,
      "author_name": "Renato Reggiani",
      "author_url": "",
      "post_date": "2023-09-07T12:24:18.380000",
      "content": "<p>I get the same error too</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2478634,
      "author_name": "Roman Emelianov",
      "author_url": "",
      "post_date": "2023-10-12T05:53:09.627000",
      "content": "<p>u can also look here <a href=\"https://www.kaggle.com/code/andradaolteanu/zzz-good-night-sleep-with-80-memory-reduction\" target=\"_blank\">https://www.kaggle.com/code/andradaolteanu/zzz-good-night-sleep-with-80-memory-reduction</a></p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2472577,
      "author_name": "Sam Kirby",
      "author_url": "",
      "post_date": "2023-10-07T12:27:06.213000",
      "content": "<p>I'm a novice here, but for anyone using R instead of python, the \"data.table\" library is a lifesaver from a memory standpoint because it manipulates large tables in-place, instead of creating copies.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2428549,
      "author_name": "Bhargav Kowshik",
      "author_url": "",
      "post_date": "2023-09-08T02:11:03.600000",
      "content": "<p>For the first time, tried <code>LazyLoading</code> with <a href=\"https://www.pola.rs/\" target=\"_blank\">Pola.rs</a> today. Loved it and it's fast ⚡</p>\n<pre><code>df_counts = (pl.scan_parquet()\n    .group_by(, maintain_order=)\n    .count()\n    .collect())\ndf_counts.head()\n</code></pre>",
      "votes": 3,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2427960": "This block looks like the issue:\n\n```\ntest = pd.read_parquet(BASE+\"/test_series.parquet\") \ntest['date'] = test['timestamp'].str.split('T', expand=True)[0]\ntest['time'] = test['timestamp'].str.split('T', expand=True)[1].str.split('-', expand=True)[0]\ntest['timestamp'] = pd.to_datetime(test['date']+' '+test['time'])\ntest['hour'] = test['timestamp'].dt.hour\n```\n\nThe following alternative is faster and requires less memory.  Note that the resulting dataframe won't have `timestamp`, `date`, or `time` columns, but you could modify the code if you want those.  I chose to omit them since it looked like your ultimate goal was the `hour` column.\n```\nimport polars as pl\ntest = (pl.scan_parquet(BASE+\"/test_series.parquet\")\n          .with_columns(\n              (pl.col(\"timestamp\")\n                 .str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\")\n                 .dt.hour()\n                 .alias(\"hour\")\n              )\n          )\n          .drop(\"timestamp\")\n          .collect()\n          .to_pandas()\n)\n```",
    "2427682": "Hello Kagglers,\n\nI had already several submissions failed because my Notebook run into this error: \"Notebook Out of Memory\nYour notebook requested more memory (RAM) than is available.\"\n\nEven when running very simple code and basically just loading in the test_series.parquet data.\nFor example this was on of my submissions that failed:\n\n`import pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport os\n\nBASE = \"data/\" if \"data\" in os.listdir(\"./\") else \"/kaggle/input/child-mind-institute-detect-sleep-states/\"\n\ntest = pd.read_parquet(BASE+\"/test_series.parquet\")\n\ntest['date'] = test['timestamp'].str.split('T', expand=True)[0]\ntest['time'] = test['timestamp'].str.split('T', expand=True)[1].str.split('-', expand=True)[0]\ntest['timestamp'] = pd.to_datetime(test['date']+' '+test['time'])\ntest['hour'] = test['timestamp'].dt.hour\n\nonset = test[test[\"hour\"].between(22,24)].set_index(\"series_id\")[\"step\"]\nwakeup = test[test[\"hour\"].between(7,9)].set_index(\"series_id\")[\"step\"]\n\nsub = pd.concat([\n    onset.reset_index().assign(event=\"onset\"),\n    wakeup.reset_index().assign(event=\"wakeup\"),\n])\nsub[\"score\"] = 0.33\nsub.sort_values([\"series_id\",\"event\"], ascending=[0,1], inplace=True)\nsub.reset_index(drop=True, inplace=True)\nsub[\"row_id\"] = sub.index\nsub = sub[[\"row_id\",\"series_id\",\"step\",\"event\",\"score\"]]\nsub.to_csv(\"submission.csv\", index=False)\nsub\n`",
    "2428484": "I had the same problem but solved it using dask.\n\nCodes with problems:\n```\nimport pandas as pd\n\n\nte = pd.read_parquet(BASE+\"/test_series.parquet\")\nte.timestamp = pd.to_datetime(te.timestamp, format='%Y-%m-%dT%H:%M:%S-%f').dt.hour\n```\n\n\nSolution:\n```\nimport dask\nimport dask.array as da\nimport dask.dataframe as dd\n\n\nte = dd.read_parquet(BASE+\"/test_series.parquet\")\nte['timestamp'] = dd.to_datetime(te['timestamp'], format='%Y-%m-%dT%H:%M:%S-%f').dt.hour\n```",
    "2427710": "I get the same error too",
    "2478634": "u can also look here https://www.kaggle.com/code/andradaolteanu/zzz-good-night-sleep-with-80-memory-reduction",
    "2472577": "I'm a novice here, but for anyone using R instead of python, the \"data.table\" library is a lifesaver from a memory standpoint because it manipulates large tables in-place, instead of creating copies.",
    "2428549": "For the first time, tried `LazyLoading` with [Pola.rs](https://www.pola.rs/) today. Loved it and it's fast ⚡\n\n```python\ndf_counts = (pl.scan_parquet(f\"{data_directory}/train_series.parquet\")\n    .group_by(\"series_id\", maintain_order=True)\n    .count()\n    .collect())\ndf_counts.head()\n```"
  }
}