{
  "id": 21930,
  "title": "Anaconda Python 2.7.x 64bit :overcoming out of momory error.",
  "url": "/competitions/grupo-bimbo-inventory-demand/discussion/21930",
  "author_name": "",
  "post_date": "2016-06-28T13:30:28.867Z",
  "votes": null,
  "comment_count": 3,
  "views": 997,
  "content": "<p>Hi all ,</p>\n\n<p>my laptop specifications are as follows.\nwindows 7 64 bit ultimate os\n4 gb ram.\nrunning python :\nAnaconda 2.7.x 64 bit: \nwhile iam using data=pandas.read_csv()  I am getting out of memory error.\nIs there a way to work with this data set. train.csv is appox 3.gb after uncompressing.</p>\n\n<p>any insight will be highly appriciated.</p>",
  "messages": [
    {
      "id": "125298",
      "postDate": "06/28/2016 13:30:28",
      "content": "<p>Hi all ,</p>\n\n<p>my laptop specifications are as follows.\nwindows 7 64 bit ultimate os\n4 gb ram.\nrunning python :\nAnaconda 2.7.x 64 bit: \nwhile iam using data=pandas.read_csv()  I am getting out of memory error.\nIs there a way to work with this data set. train.csv is appox 3.gb after uncompressing.</p>\n\n<p>any insight will be highly appriciated.</p>",
      "rawMarkdown": "Hi all ,\r\n\r\n my laptop specifications are as follows.\r\nwindows 7 64 bit ultimate os\r\n4 gb ram.\r\nrunning python :\r\nAnaconda 2.7.x 64 bit: \r\nwhile iam using data=pandas.read_csv()  I am getting out of memory error.\r\nIs there a way to work with this data set. train.csv is appox 3.gb after uncompressing.\r\n\r\nany insight will be highly appriciated.",
      "votes": null
    },
    {
      "id": "125325",
      "postDate": "06/28/2016 16:46:38",
      "content": "<p>You can read with limited columns with &#8220;usecols&quot; option in read_csv. With 4GB Ram you should also limit number of rows to read with &#8220;nrows&#8221; option.  (I would try to use AWS EC2 or add more RAM) Each week is about 10 million rows.</p>\n\n<pre><code>train =  pd.read_csv(&quot;train.csv&quot;,nrows=500000,usecols=[&quot;Semana&quot;,&quot;Canal_ID&quot;,&quot;Ruta_SAK&quot;,&quot;Cliente_ID&quot;,&quot;Producto_ID&quot;,&quot;Demanda_uni_equil&quot;,&quot;Agencia_ID&quot;])\n</code></pre>",
      "rawMarkdown": "You can read with limited columns with “usecols\" option in read_csv. With 4GB Ram you should also limit number of rows to read with “nrows” option.  (I would try to use AWS EC2 or add more RAM) Each week is about 10 million rows.\r\n\r\n    train =  pd.read_csv(\"train.csv\",nrows=500000,usecols=[\"Semana\",\"Canal_ID\",\"Ruta_SAK\",\"Cliente_ID\",\"Producto_ID\",\"Demanda_uni_equil\",\"Agencia_ID\"])",
      "votes": null
    },
    {
      "id": "125401",
      "postDate": "06/29/2016 06:10:38",
      "content": "<p>Try memap from numpy. See the link below. particularly look at the post from @Gerard Toonstra.</p>\n\n<p><a href=\"https://www.kaggle.com/c/state-farm-distracted-driver-detection/forums/t/20664/data-can-t-fit-in-memory/121941\">https://www.kaggle.com/c/state-farm-distracted-driver-detection/forums/t/20664/data-can-t-fit-in-memory/121941</a></p>\n\n<p>It gives you an example of how to  use memap and how to use generators to access the data in chunks.</p>\n\n<p>I have had success in using this for StateFarm, and was able to process 12GB of data on a machine with 8GB of RAM (actual available RAM is around 5 GB after Windows consumes close to 3GB).</p>\n\n<p>Its slow though as disk access takes time. But if you are desperate to compete, this helps. At least you can work with the whole data.</p>\n\n<p>Regards\nSayantan</p>",
      "rawMarkdown": "Try memap from numpy. See the link below. particularly look at the post from @Gerard Toonstra.\r\n\r\n<https://www.kaggle.com/c/state-farm-distracted-driver-detection/forums/t/20664/data-can-t-fit-in-memory/121941>\r\n\r\nIt gives you an example of how to  use memap and how to use generators to access the data in chunks.\r\n\r\nI have had success in using this for StateFarm, and was able to process 12GB of data on a machine with 8GB of RAM (actual available RAM is around 5 GB after Windows consumes close to 3GB).\r\n\r\nIts slow though as disk access takes time. But if you are desperate to compete, this helps. At least you can work with the whole data.\r\n\r\nRegards\r\nSayantan",
      "votes": null
    },
    {
      "id": "125402",
      "postDate": "06/29/2016 06:17:59",
      "content": "<p>thanks for the inputs</p>",
      "rawMarkdown": "thanks for the inputs",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 125325,
      "author_name": "rashmibanthia",
      "author_url": "",
      "post_date": "06/28/2016 16:46:38",
      "content": "<p>You can read with limited columns with &#8220;usecols&quot; option in read_csv. With 4GB Ram you should also limit number of rows to read with &#8220;nrows&#8221; option.  (I would try to use AWS EC2 or add more RAM) Each week is about 10 million rows.</p>\n\n<pre><code>train =  pd.read_csv(&quot;train.csv&quot;,nrows=500000,usecols=[&quot;Semana&quot;,&quot;Canal_ID&quot;,&quot;Ruta_SAK&quot;,&quot;Cliente_ID&quot;,&quot;Producto_ID&quot;,&quot;Demanda_uni_equil&quot;,&quot;Agencia_ID&quot;])\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125401,
      "author_name": "sayantan",
      "author_url": "",
      "post_date": "06/29/2016 06:10:38",
      "content": "<p>Try memap from numpy. See the link below. particularly look at the post from @Gerard Toonstra.</p>\n\n<p><a href=\"https://www.kaggle.com/c/state-farm-distracted-driver-detection/forums/t/20664/data-can-t-fit-in-memory/121941\">https://www.kaggle.com/c/state-farm-distracted-driver-detection/forums/t/20664/data-can-t-fit-in-memory/121941</a></p>\n\n<p>It gives you an example of how to  use memap and how to use generators to access the data in chunks.</p>\n\n<p>I have had success in using this for StateFarm, and was able to process 12GB of data on a machine with 8GB of RAM (actual available RAM is around 5 GB after Windows consumes close to 3GB).</p>\n\n<p>Its slow though as disk access takes time. But if you are desperate to compete, this helps. At least you can work with the whole data.</p>\n\n<p>Regards\nSayantan</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125402,
      "author_name": "bindukumar",
      "author_url": "",
      "post_date": "06/29/2016 06:17:59",
      "content": "<p>thanks for the inputs</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "125298": "Hi all ,\r\n\r\n my laptop specifications are as follows.\r\nwindows 7 64 bit ultimate os\r\n4 gb ram.\r\nrunning python :\r\nAnaconda 2.7.x 64 bit: \r\nwhile iam using data=pandas.read_csv()  I am getting out of memory error.\r\nIs there a way to work with this data set. train.csv is appox 3.gb after uncompressing.\r\n\r\nany insight will be highly appriciated.",
    "125325": "You can read with limited columns with “usecols\" option in read_csv. With 4GB Ram you should also limit number of rows to read with “nrows” option.  (I would try to use AWS EC2 or add more RAM) Each week is about 10 million rows.\r\n\r\n    train =  pd.read_csv(\"train.csv\",nrows=500000,usecols=[\"Semana\",\"Canal_ID\",\"Ruta_SAK\",\"Cliente_ID\",\"Producto_ID\",\"Demanda_uni_equil\",\"Agencia_ID\"])",
    "125401": "Try memap from numpy. See the link below. particularly look at the post from @Gerard Toonstra.\r\n\r\n<https://www.kaggle.com/c/state-farm-distracted-driver-detection/forums/t/20664/data-can-t-fit-in-memory/121941>\r\n\r\nIt gives you an example of how to  use memap and how to use generators to access the data in chunks.\r\n\r\nI have had success in using this for StateFarm, and was able to process 12GB of data on a machine with 8GB of RAM (actual available RAM is around 5 GB after Windows consumes close to 3GB).\r\n\r\nIts slow though as disk access takes time. But if you are desperate to compete, this helps. At least you can work with the whole data.\r\n\r\nRegards\r\nSayantan",
    "125402": "thanks for the inputs"
  },
  "source": "meta"
}