{
  "id": 23730,
  "title": "Dataset Size",
  "url": "/competitions/bosch-production-line-performance/discussion/23730",
  "author_name": "",
  "post_date": "2016-09-17T18:42:35.400Z",
  "votes": null,
  "comment_count": 2,
  "views": 1026,
  "content": "<p>Hi, I've been looking through the forums and have seen several references to how 'large' the dataset is and how it's difficult to fit into memory. Though when I look at the data available, it only seems to be ~400 MB. I'm probably missing something, but what makes this dataset so large?</p>",
  "messages": [
    {
      "id": "135866",
      "postDate": "09/17/2016 18:42:35",
      "content": "<p>Hi, I've been looking through the forums and have seen several references to how 'large' the dataset is and how it's difficult to fit into memory. Though when I look at the data available, it only seems to be ~400 MB. I'm probably missing something, but what makes this dataset so large?</p>",
      "rawMarkdown": "Hi, I've been looking through the forums and have seen several references to how 'large' the dataset is and how it's difficult to fit into memory. Though when I look at the data available, it only seems to be ~400 MB. I'm probably missing something, but what makes this dataset so large?",
      "votes": null
    },
    {
      "id": "135882",
      "postDate": "09/17/2016 22:19:42",
      "content": "<p>1) Unzip it</p>\n\n<p>2) Merge data</p>\n\n<p>3) Fit it in memory</p>",
      "rawMarkdown": "1) Unzip it\r\n\r\n2) Merge data\r\n\r\n3) Fit it in memory",
      "votes": null
    },
    {
      "id": "136987",
      "postDate": "09/27/2016 06:48:44",
      "content": "<p>The size you see is only what of the zip file. After unzipping it, taking the train_numeric as an example (whose size is smallest as shown in my computer, 2.14GB), it's about 1000 columns and 1,000,000 rows, that is 1 billion values and blanks. Gigantic compared with former competitions' datasets.</p>",
      "rawMarkdown": "The size you see is only what of the zip file. After unzipping it, taking the train_numeric as an example (whose size is smallest as shown in my computer, 2.14GB), it's about 1000 columns and 1,000,000 rows, that is 1 billion values and blanks. Gigantic compared with former competitions' datasets.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 135882,
      "author_name": "carloshuertas",
      "author_url": "",
      "post_date": "09/17/2016 22:19:42",
      "content": "<p>1) Unzip it</p>\n\n<p>2) Merge data</p>\n\n<p>3) Fit it in memory</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 136987,
      "author_name": "hknmtt",
      "author_url": "",
      "post_date": "09/27/2016 06:48:44",
      "content": "<p>The size you see is only what of the zip file. After unzipping it, taking the train_numeric as an example (whose size is smallest as shown in my computer, 2.14GB), it's about 1000 columns and 1,000,000 rows, that is 1 billion values and blanks. Gigantic compared with former competitions' datasets.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "135866": "Hi, I've been looking through the forums and have seen several references to how 'large' the dataset is and how it's difficult to fit into memory. Though when I look at the data available, it only seems to be ~400 MB. I'm probably missing something, but what makes this dataset so large?",
    "135882": "1) Unzip it\r\n\r\n2) Merge data\r\n\r\n3) Fit it in memory",
    "136987": "The size you see is only what of the zip file. After unzipping it, taking the train_numeric as an example (whose size is smallest as shown in my computer, 2.14GB), it's about 1000 columns and 1,000,000 rows, that is 1 billion values and blanks. Gigantic compared with former competitions' datasets."
  },
  "source": "meta"
}