{
  "id": 165429,
  "title": "Unable to load the data into memory",
  "url": "/competitions/global-wheat-detection/discussion/165429",
  "author_name": "",
  "post_date": "2020-07-09T18:25:52.425652300Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I was trying to load the images into the Python environment but the kernel crashed with error - \"Your notebook tried to allocate more memory than is available. It has restarted.\".</p>\n\n<p>So, how can I work with this dataset on Kaggle?</p>",
  "messages": [
    {
      "id": "922011",
      "postDate": "07/09/2020 18:25:52",
      "content": "<p>I was trying to load the images into the Python environment but the kernel crashed with error - \"Your notebook tried to allocate more memory than is available. It has restarted.\".</p>\n\n<p>So, how can I work with this dataset on Kaggle?</p>",
      "rawMarkdown": "I was trying to load the images into the Python environment but the kernel crashed with error - \"Your notebook tried to allocate more memory than is available. It has restarted.\".\n\nSo, how can I work with this dataset on Kaggle?",
      "votes": null
    },
    {
      "id": "922024",
      "postDate": "07/09/2020 18:37:57",
      "content": "<p>Something to do with your batch size, model size, image size maybe? Try changing those</p>",
      "rawMarkdown": "Something to do with your batch size, model size, image size maybe? Try changing those",
      "votes": null
    },
    {
      "id": "922032",
      "postDate": "07/09/2020 18:41:26",
      "content": "<p>No that is not the case. I am just simply loading the images.</p>\n\n<p><code>\nimport cv2\nimages = []\nfor idx, row in img_df.iterrows():\n   images.append(csv.imread(os.path.join(data_dir, \"%.jpg\"%row['image_id'])))\n</code></p>",
      "rawMarkdown": "No that is not the case. I am just simply loading the images.\n\n```\nimport cv2\nimages = []\nfor idx, row in img_df.iterrows():\n   images.append(csv.imread(os.path.join(data_dir, \"%.jpg\"%row['image_id'])))\n```",
      "votes": null
    },
    {
      "id": "922053",
      "postDate": "07/09/2020 19:01:48",
      "content": "<p>You're trying to load thousands of 1024^2 images into RAM. There's simply not enough memory to do that on Kaggle notebooks</p>",
      "rawMarkdown": "You're trying to load thousands of 1024^2 images into RAM. There's simply not enough memory to do that on Kaggle notebooks",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 922024,
      "author_name": "jonykarki",
      "author_url": "",
      "post_date": "07/09/2020 18:37:57",
      "content": "<p>Something to do with your batch size, model size, image size maybe? Try changing those</p>",
      "votes": null,
      "replies": [
        {
          "id": 922032,
          "author_name": "pikkupr",
          "author_url": "",
          "post_date": "07/09/2020 18:41:26",
          "content": "<p>No that is not the case. I am just simply loading the images.</p>\n\n<p><code>\nimport cv2\nimages = []\nfor idx, row in img_df.iterrows():\n   images.append(csv.imread(os.path.join(data_dir, \"%.jpg\"%row['image_id'])))\n</code></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 922053,
          "author_name": "matthewmasters",
          "author_url": "",
          "post_date": "07/09/2020 19:01:48",
          "content": "<p>You're trying to load thousands of 1024^2 images into RAM. There's simply not enough memory to do that on Kaggle notebooks</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "922011": "I was trying to load the images into the Python environment but the kernel crashed with error - \"Your notebook tried to allocate more memory than is available. It has restarted.\".\n\nSo, how can I work with this dataset on Kaggle?",
    "922024": "Something to do with your batch size, model size, image size maybe? Try changing those",
    "922032": "No that is not the case. I am just simply loading the images.\n\n```\nimport cv2\nimages = []\nfor idx, row in img_df.iterrows():\n   images.append(csv.imread(os.path.join(data_dir, \"%.jpg\"%row['image_id'])))\n```",
    "922053": "You're trying to load thousands of 1024^2 images into RAM. There's simply not enough memory to do that on Kaggle notebooks"
  },
  "source": "meta"
}