{
  "id": 440213,
  "title": "Notebook Out of Memory",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/440213",
  "author_name": "",
  "post_date": "2023-09-14T07:15:38.018191300Z",
  "votes": 2,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I am getting \"Notebook Out of Memory\" everytime after submission. Is it possible to let us know atleast the size of the test dataset, so that I may test it and try to debug the issue before submitting?</p>",
  "messages": [
    {
      "id": "2438281",
      "postDate": "09/14/2023 07:15:38",
      "content": "<p>I am getting \"Notebook Out of Memory\" everytime after submission. Is it possible to let us know atleast the size of the test dataset, so that I may test it and try to debug the issue before submitting?</p>",
      "rawMarkdown": "I am getting \"Notebook Out of Memory\" everytime after submission. Is it possible to let us know atleast the size of the test dataset, so that I may test it and try to debug the issue before submitting?",
      "votes": null
    },
    {
      "id": "2438315",
      "postDate": "09/14/2023 07:40:24",
      "content": "<p>Dear <a href=\"https://www.kaggle.com/digantakalita\" target=\"_blank\">@digantakalita</a> </p>\n<p>Note that we are told that \"<em>The full test set contains about 200 series.</em>\", so I estimate it contains around 92_000_000 rows. With this in mind we could create a mock full size test dataset from the training data via</p>\n<pre><code>test  = pd.read_parquet().head(n_rows)\n</code></pre>\n<p>simply to simulate a submission to see if it fits in memory.</p>\n<p>All the best,<br>\ncarl</p>",
      "rawMarkdown": "Dear @digantakalita \n\nNote that we are told that \"*The full test set contains about 200 series.*\", so I estimate it contains around 92_000_000 rows. With this in mind we could create a mock full size test dataset from the training data via\n\n```python\ntest  = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\").head(n_rows)\n```\n\nsimply to simulate a submission to see if it fits in memory.\n\nAll the best,\ncarl",
      "votes": null
    },
    {
      "id": "2438505",
      "postDate": "09/14/2023 10:12:14",
      "content": "<p>The test_series data is apparently much larger than train_series, I created a loop to filter the test and run it in parts.<br>\nYou can see this idea in this <a href=\"https://www.kaggle.com/code/renatoreggiani/feature-eng-ideas-and-lightgbm-cmi\" target=\"_blank\">notebook</a>.<br>\nIt is important to take this care because data from private tests is greater than that from public tests.</p>",
      "rawMarkdown": "The test_series data is apparently much larger than train_series, I created a loop to filter the test and run it in parts.\nYou can see this idea in this [notebook](https://www.kaggle.com/code/renatoreggiani/feature-eng-ideas-and-lightgbm-cmi).\nIt is important to take this care because data from private tests is greater than that from public tests.",
      "votes": null
    },
    {
      "id": "2438545",
      "postDate": "09/14/2023 10:51:49",
      "content": "<p>Yes this would be very helpful, as a lot of time is lost in blind experimentation to see if notebook will be oom, especially when creating new features</p>",
      "rawMarkdown": "Yes this would be very helpful, as a lot of time is lost in blind experimentation to see if notebook will be oom, especially when creating new features",
      "votes": null
    },
    {
      "id": "2438592",
      "postDate": "09/14/2023 11:21:13",
      "content": "<p>I see. You are predicting for each series_id on the test. Thats a really great idea. Thank you.</p>",
      "rawMarkdown": "I see. You are predicting for each series_id on the test. Thats a really great idea. Thank you.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2438315,
      "author_name": "carlmcbrideellis",
      "author_url": "",
      "post_date": "09/14/2023 07:40:24",
      "content": "<p>Dear <a href=\"https://www.kaggle.com/digantakalita\" target=\"_blank\">@digantakalita</a> </p>\n<p>Note that we are told that \"<em>The full test set contains about 200 series.</em>\", so I estimate it contains around 92_000_000 rows. With this in mind we could create a mock full size test dataset from the training data via</p>\n<pre><code>test  = pd.read_parquet().head(n_rows)\n</code></pre>\n<p>simply to simulate a submission to see if it fits in memory.</p>\n<p>All the best,<br>\ncarl</p>",
      "votes": null,
      "replies": [
        {
          "id": 2438545,
          "author_name": "josipvrdoljak",
          "author_url": "",
          "post_date": "09/14/2023 10:51:49",
          "content": "<p>Yes this would be very helpful, as a lot of time is lost in blind experimentation to see if notebook will be oom, especially when creating new features</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2438505,
      "author_name": "renatoreggiani",
      "author_url": "",
      "post_date": "09/14/2023 10:12:14",
      "content": "<p>The test_series data is apparently much larger than train_series, I created a loop to filter the test and run it in parts.<br>\nYou can see this idea in this <a href=\"https://www.kaggle.com/code/renatoreggiani/feature-eng-ideas-and-lightgbm-cmi\" target=\"_blank\">notebook</a>.<br>\nIt is important to take this care because data from private tests is greater than that from public tests.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2438592,
          "author_name": "digantakalita",
          "author_url": "",
          "post_date": "09/14/2023 11:21:13",
          "content": "<p>I see. You are predicting for each series_id on the test. Thats a really great idea. Thank you.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2438281": "I am getting \"Notebook Out of Memory\" everytime after submission. Is it possible to let us know atleast the size of the test dataset, so that I may test it and try to debug the issue before submitting?",
    "2438315": "Dear @digantakalita \n\nNote that we are told that \"*The full test set contains about 200 series.*\", so I estimate it contains around 92_000_000 rows. With this in mind we could create a mock full size test dataset from the training data via\n\n```python\ntest  = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\").head(n_rows)\n```\n\nsimply to simulate a submission to see if it fits in memory.\n\nAll the best,\ncarl",
    "2438505": "The test_series data is apparently much larger than train_series, I created a loop to filter the test and run it in parts.\nYou can see this idea in this [notebook](https://www.kaggle.com/code/renatoreggiani/feature-eng-ideas-and-lightgbm-cmi).\nIt is important to take this care because data from private tests is greater than that from public tests.",
    "2438545": "Yes this would be very helpful, as a lot of time is lost in blind experimentation to see if notebook will be oom, especially when creating new features",
    "2438592": "I see. You are predicting for each series_id on the test. Thats a really great idea. Thank you."
  },
  "source": "meta"
}