{
  "id": 23258,
  "title": "reading test_categorical.csv ",
  "url": "/competitions/bosch-production-line-performance/discussion/23258",
  "author_name": "",
  "post_date": "2016-09-01T14:54:15.120Z",
  "votes": 1,
  "comment_count": 1,
  "views": 433,
  "content": "<p>Hi All,\nI am trying to read test_categorical.csv  . I have defined the specific dtypes while reading but it still takes around 25Gb of RAM, any ideas of how to handle it. Also what can be used to store this data, so that it can be loaded efficiently afterwards.\nThanks </p>",
  "messages": [
    {
      "id": "133329",
      "postDate": "09/01/2016 14:54:15",
      "content": "<p>Hi All,\nI am trying to read test_categorical.csv  . I have defined the specific dtypes while reading but it still takes around 25Gb of RAM, any ideas of how to handle it. Also what can be used to store this data, so that it can be loaded efficiently afterwards.\nThanks </p>",
      "rawMarkdown": "Hi All,\r\nI am trying to read test_categorical.csv  . I have defined the specific dtypes while reading but it still takes around 25Gb of RAM, any ideas of how to handle it. Also what can be used to store this data, so that it can be loaded efficiently afterwards.\r\nThanks",
      "votes": null
    },
    {
      "id": "133336",
      "postDate": "09/01/2016 15:20:19",
      "content": "<p>Hi,</p>\n\n<p>You can try reading and processing it in chunks.\nSome ideas:</p>\n\n<p>Line by line:\nwith open(&quot;my_file.txt&quot;) as f:\n    for line in f:\n        do_some_work(line)</p>\n\n<p>In chunks:\nfor i in range(0,12):\n    dataset= genfromtxt(open('my_file.txt'),skip_header=1+100000*i,max_rows =100000,  delimiter=',', dtype='f8')   </p>",
      "rawMarkdown": "Hi,\r\n\r\nYou can try reading and processing it in chunks.\r\nSome ideas:\r\n\r\nLine by line:\r\nwith open(\"my_file.txt\") as f:\r\n    for line in f:\r\n        do_some_work(line)\r\n\r\n\r\nIn chunks:\r\nfor i in range(0,12):\r\n    dataset= genfromtxt(open('my_file.txt'),skip_header=1+100000*i,max_rows =100000,  delimiter=',', dtype='f8')",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 133336,
      "author_name": "umenopro",
      "author_url": "",
      "post_date": "09/01/2016 15:20:19",
      "content": "<p>Hi,</p>\n\n<p>You can try reading and processing it in chunks.\nSome ideas:</p>\n\n<p>Line by line:\nwith open(&quot;my_file.txt&quot;) as f:\n    for line in f:\n        do_some_work(line)</p>\n\n<p>In chunks:\nfor i in range(0,12):\n    dataset= genfromtxt(open('my_file.txt'),skip_header=1+100000*i,max_rows =100000,  delimiter=',', dtype='f8')   </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "133329": "Hi All,\r\nI am trying to read test_categorical.csv  . I have defined the specific dtypes while reading but it still takes around 25Gb of RAM, any ideas of how to handle it. Also what can be used to store this data, so that it can be loaded efficiently afterwards.\r\nThanks",
    "133336": "Hi,\r\n\r\nYou can try reading and processing it in chunks.\r\nSome ideas:\r\n\r\nLine by line:\r\nwith open(\"my_file.txt\") as f:\r\n    for line in f:\r\n        do_some_work(line)\r\n\r\n\r\nIn chunks:\r\nfor i in range(0,12):\r\n    dataset= genfromtxt(open('my_file.txt'),skip_header=1+100000*i,max_rows =100000,  delimiter=',', dtype='f8')"
  },
  "source": "meta"
}