{
  "id": 20918,
  "title": "Memory error when reading training files using pandas",
  "url": "/competitions/facebook-v-predicting-check-ins/discussion/20918",
  "author_name": "",
  "post_date": "2016-05-12T21:08:02.437Z",
  "votes": null,
  "comment_count": 3,
  "views": 2283,
  "content": "<p>I'm quite a beginner at Kaggle. This question may be common and kind of silly, but could someone help me with this?</p>\n\n<p>When I tried to read the training file &quot;train.csv&quot; using pandas function read_csv(), I met the Memory error. My operation system is Windows in 64 bits, and my RAM has 8GB. I think the RAM is big enough to store the data. Did I do something wrong?</p>\n\n<p>How do you guys handle the big data? Thanks!</p>\n\n<p>EDIT: Thanks for all the help! The problem has been solved. I used python of a 32-bit version on my 64-bit system, which got me into trouble.</p>",
  "messages": [
    {
      "id": "119814",
      "postDate": "05/12/2016 21:08:02",
      "content": "<p>I'm quite a beginner at Kaggle. This question may be common and kind of silly, but could someone help me with this?</p>\n\n<p>When I tried to read the training file &quot;train.csv&quot; using pandas function read_csv(), I met the Memory error. My operation system is Windows in 64 bits, and my RAM has 8GB. I think the RAM is big enough to store the data. Did I do something wrong?</p>\n\n<p>How do you guys handle the big data? Thanks!</p>\n\n<p>EDIT: Thanks for all the help! The problem has been solved. I used python of a 32-bit version on my 64-bit system, which got me into trouble.</p>",
      "rawMarkdown": "I'm quite a beginner at Kaggle. This question may be common and kind of silly, but could someone help me with this?\r\n\r\nWhen I tried to read the training file \"train.csv\" using pandas function read_csv(), I met the Memory error. My operation system is Windows in 64 bits, and my RAM has 8GB. I think the RAM is big enough to store the data. Did I do something wrong?\r\n\r\nHow do you guys handle the big data? Thanks!\r\n\r\nEDIT: Thanks for all the help! The problem has been solved. I used python of a 32-bit version on my 64-bit system, which got me into trouble.",
      "votes": null
    },
    {
      "id": "119821",
      "postDate": "05/12/2016 21:34:25",
      "content": "<p><strong>1. Using chunks</strong></p>\n\n<pre><code>train_chunk = pd.read_csv('../input/train.csv', chunksize=100000)\n\nfor chunk in train_chunk:\n\n      train_df = pd.concat( [ train, chunk ] )\n</code></pre>\n\n<p><strong>2. Load only few columns</strong> </p>\n\n<pre><code>pd.read_csv('../input/train.csv', usecols=['some_column'])\n</code></pre>\n\n<p><strong>3. Load only subset of rows</strong></p>\n\n<pre><code>pd.read_csv(&quot;../input/train.csv&quot;, nrows=10000)\n</code></pre>",
      "rawMarkdown": "**1. Using chunks**\r\n    \r\n    train_chunk = pd.read_csv('../input/train.csv', chunksize=100000)\r\n    \r\n    for chunk in train_chunk:\r\n    \r\n          train_df = pd.concat( [ train, chunk ] )\r\n\r\n**2. Load only few columns** \r\n\r\n    pd.read_csv('../input/train.csv', usecols=['some_column'])\r\n\r\n**3. Load only subset of rows**\r\n\r\n    pd.read_csv(\"../input/train.csv\", nrows=10000)",
      "votes": null
    },
    {
      "id": "119826",
      "postDate": "05/12/2016 23:20:21",
      "content": "<p>8gb of RAM should be more than enougth for this dataset. Still, if you are running into problems, you could try to read the data in 32 bits, instead of the default 64 for your system.\nI just runed this on a 6gb laptop and worked without any problems:</p>\n\n<p><code>train = pd.read_csv(&quot;path/to/traincsv&quot;,  dtype=dict(row_id=np.int32, x=np.float32, y=np.float32, accuracy=np.int32, time=np.int32, place_id=np.int64))</code></p>\n\n<p>EDIT: place_id has to be 64 bits to avoid overflow</p>",
      "rawMarkdown": "8gb of RAM should be more than enougth for this dataset. Still, if you are running into problems, you could try to read the data in 32 bits, instead of the default 64 for your system.\r\nI just runed this on a 6gb laptop and worked without any problems:\r\n\r\n`train = pd.read_csv(\"path/to/traincsv\",  dtype=dict(row_id=np.int32, x=np.float32, y=np.float32, accuracy=np.int32, time=np.int32, place_id=np.int64))`\r\n\r\nEDIT: place_id has to be 64 bits to avoid overflow",
      "votes": null
    },
    {
      "id": "119924",
      "postDate": "05/13/2016 19:19:08",
      "content": "<p>As @Nickel mentioned, if you specify the types to pandas there shouldn't be any problem, Following will only take 1.3GB of RAM:</p>\n\n<pre><code>types = {'row_id': np.dtype(int),\n         'x': np.dtype(float),\n         'y' : np.dtype(float),\n         'accuracy': np.dtype(int),\n         'place_id': np.dtype(int) }\n\ntrain = pd.read_csv('~/kaggledata/facebook/train.csv', dtype=types, index_col=0)\n</code></pre>\n\n<p>After above if you run: <code>train.info()</code> you should get:</p>\n\n<pre><code>&lt;class 'pandas.core.frame.DataFrame'&gt;\nInt64Index: 29118021 entries, 0 to 29118020\nData columns (total 5 columns):\nx           float64\ny           float64\naccuracy    int64\ntime        int64\nplace_id    int64\ndtypes: float64(2), int64(3)\nmemory usage: 1.3 GB\n</code></pre>",
      "rawMarkdown": "As @Nickel mentioned, if you specify the types to pandas there shouldn't be any problem, Following will only take 1.3GB of RAM:\r\n\r\n    types = {'row_id': np.dtype(int),\r\n             'x': np.dtype(float),\r\n             'y' : np.dtype(float),\r\n             'accuracy': np.dtype(int),\r\n             'place_id': np.dtype(int) }\r\n             \r\n    train = pd.read_csv('~/kaggledata/facebook/train.csv', dtype=types, index_col=0)\r\n\r\nAfter above if you run: `train.info()` you should get:\r\n\r\n    <class 'pandas.core.frame.DataFrame'>\r\n    Int64Index: 29118021 entries, 0 to 29118020\r\n    Data columns (total 5 columns):\r\n    x           float64\r\n    y           float64\r\n    accuracy    int64\r\n    time        int64\r\n    place_id    int64\r\n    dtypes: float64(2), int64(3)\r\n    memory usage: 1.3 GB",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 119821,
      "author_name": "omarelgabry",
      "author_url": "",
      "post_date": "05/12/2016 21:34:25",
      "content": "<p><strong>1. Using chunks</strong></p>\n\n<pre><code>train_chunk = pd.read_csv('../input/train.csv', chunksize=100000)\n\nfor chunk in train_chunk:\n\n      train_df = pd.concat( [ train, chunk ] )\n</code></pre>\n\n<p><strong>2. Load only few columns</strong> </p>\n\n<pre><code>pd.read_csv('../input/train.csv', usecols=['some_column'])\n</code></pre>\n\n<p><strong>3. Load only subset of rows</strong></p>\n\n<pre><code>pd.read_csv(&quot;../input/train.csv&quot;, nrows=10000)\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 119826,
      "author_name": "nickel",
      "author_url": "",
      "post_date": "05/12/2016 23:20:21",
      "content": "<p>8gb of RAM should be more than enougth for this dataset. Still, if you are running into problems, you could try to read the data in 32 bits, instead of the default 64 for your system.\nI just runed this on a 6gb laptop and worked without any problems:</p>\n\n<p><code>train = pd.read_csv(&quot;path/to/traincsv&quot;,  dtype=dict(row_id=np.int32, x=np.float32, y=np.float32, accuracy=np.int32, time=np.int32, place_id=np.int64))</code></p>\n\n<p>EDIT: place_id has to be 64 bits to avoid overflow</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 119924,
      "author_name": "arman2",
      "author_url": "",
      "post_date": "05/13/2016 19:19:08",
      "content": "<p>As @Nickel mentioned, if you specify the types to pandas there shouldn't be any problem, Following will only take 1.3GB of RAM:</p>\n\n<pre><code>types = {'row_id': np.dtype(int),\n         'x': np.dtype(float),\n         'y' : np.dtype(float),\n         'accuracy': np.dtype(int),\n         'place_id': np.dtype(int) }\n\ntrain = pd.read_csv('~/kaggledata/facebook/train.csv', dtype=types, index_col=0)\n</code></pre>\n\n<p>After above if you run: <code>train.info()</code> you should get:</p>\n\n<pre><code>&lt;class 'pandas.core.frame.DataFrame'&gt;\nInt64Index: 29118021 entries, 0 to 29118020\nData columns (total 5 columns):\nx           float64\ny           float64\naccuracy    int64\ntime        int64\nplace_id    int64\ndtypes: float64(2), int64(3)\nmemory usage: 1.3 GB\n</code></pre>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "119814": "I'm quite a beginner at Kaggle. This question may be common and kind of silly, but could someone help me with this?\r\n\r\nWhen I tried to read the training file \"train.csv\" using pandas function read_csv(), I met the Memory error. My operation system is Windows in 64 bits, and my RAM has 8GB. I think the RAM is big enough to store the data. Did I do something wrong?\r\n\r\nHow do you guys handle the big data? Thanks!\r\n\r\nEDIT: Thanks for all the help! The problem has been solved. I used python of a 32-bit version on my 64-bit system, which got me into trouble.",
    "119821": "**1. Using chunks**\r\n    \r\n    train_chunk = pd.read_csv('../input/train.csv', chunksize=100000)\r\n    \r\n    for chunk in train_chunk:\r\n    \r\n          train_df = pd.concat( [ train, chunk ] )\r\n\r\n**2. Load only few columns** \r\n\r\n    pd.read_csv('../input/train.csv', usecols=['some_column'])\r\n\r\n**3. Load only subset of rows**\r\n\r\n    pd.read_csv(\"../input/train.csv\", nrows=10000)",
    "119826": "8gb of RAM should be more than enougth for this dataset. Still, if you are running into problems, you could try to read the data in 32 bits, instead of the default 64 for your system.\r\nI just runed this on a 6gb laptop and worked without any problems:\r\n\r\n`train = pd.read_csv(\"path/to/traincsv\",  dtype=dict(row_id=np.int32, x=np.float32, y=np.float32, accuracy=np.int32, time=np.int32, place_id=np.int64))`\r\n\r\nEDIT: place_id has to be 64 bits to avoid overflow",
    "119924": "As @Nickel mentioned, if you specify the types to pandas there shouldn't be any problem, Following will only take 1.3GB of RAM:\r\n\r\n    types = {'row_id': np.dtype(int),\r\n             'x': np.dtype(float),\r\n             'y' : np.dtype(float),\r\n             'accuracy': np.dtype(int),\r\n             'place_id': np.dtype(int) }\r\n             \r\n    train = pd.read_csv('~/kaggledata/facebook/train.csv', dtype=types, index_col=0)\r\n\r\nAfter above if you run: `train.info()` you should get:\r\n\r\n    <class 'pandas.core.frame.DataFrame'>\r\n    Int64Index: 29118021 entries, 0 to 29118020\r\n    Data columns (total 5 columns):\r\n    x           float64\r\n    y           float64\r\n    accuracy    int64\r\n    time        int64\r\n    place_id    int64\r\n    dtypes: float64(2), int64(3)\r\n    memory usage: 1.3 GB"
  },
  "source": "meta"
}