{
  "id": 23095,
  "title": "Using Python to store data",
  "url": "/competitions/bosch-production-line-performance/discussion/23095",
  "author_name": "",
  "post_date": "2016-08-25T15:19:26.043Z",
  "votes": 3,
  "comment_count": 1,
  "views": 1007,
  "content": "<p>Hello all,</p>\n\n<p>I have never dealt with such a large data set.  So I am starting with the train_numeric.csv file. I have been able to load the file as a Panda data frame in the form of chunks</p>\n\n<blockquote>\n  <p>chunk =\n  pd.read_csv(&quot;C:/Users/FTS.fts-gnosis/Documents/kaggle/Bosch_kaggle/train_numeric.csv&quot;,\n  dtype = np.float32, iterator=True, chunksize=1000)\n          xtrain = pd.concat(chunk)</p>\n</blockquote>\n\n<p>but now the other problem is to store that dataframe so that I do not have to load it again and again. </p>\n\n<p>I have tried using  1.Pickle and 2 hdf but may be the dataframe is about 8.6GB so it does not work. </p>\n\n<p>Any other recommendations will be greatly appreciated.</p>\n\n<p>Thanks</p>",
  "messages": [
    {
      "id": "132441",
      "postDate": "08/25/2016 15:19:26",
      "content": "<p>Hello all,</p>\n\n<p>I have never dealt with such a large data set.  So I am starting with the train_numeric.csv file. I have been able to load the file as a Panda data frame in the form of chunks</p>\n\n<blockquote>\n  <p>chunk =\n  pd.read_csv(&quot;C:/Users/FTS.fts-gnosis/Documents/kaggle/Bosch_kaggle/train_numeric.csv&quot;,\n  dtype = np.float32, iterator=True, chunksize=1000)\n          xtrain = pd.concat(chunk)</p>\n</blockquote>\n\n<p>but now the other problem is to store that dataframe so that I do not have to load it again and again. </p>\n\n<p>I have tried using  1.Pickle and 2 hdf but may be the dataframe is about 8.6GB so it does not work. </p>\n\n<p>Any other recommendations will be greatly appreciated.</p>\n\n<p>Thanks</p>",
      "rawMarkdown": "Hello all,\r\n\r\nI have never dealt with such a large data set.  So I am starting with the train_numeric.csv file. I have been able to load the file as a Panda data frame in the form of chunks\r\n\r\n   \r\n\r\n>  chunk =\r\n> pd.read_csv(\"C:/Users/FTS.fts-gnosis/Documents/kaggle/Bosch_kaggle/train_numeric.csv\",\r\n> dtype = np.float32, iterator=True, chunksize=1000)\r\n>         xtrain = pd.concat(chunk)\r\n\r\n but now the other problem is to store that dataframe so that I do not have to load it again and again. \r\n\r\nI have tried using  1.Pickle and 2 hdf but may be the dataframe is about 8.6GB so it does not work. \r\n\r\nAny other recommendations will be greatly appreciated.\r\n\r\nThanks",
      "votes": null
    },
    {
      "id": "132466",
      "postDate": "08/25/2016 19:25:36",
      "content": "<pre><code>import numpy as np\nimport h5py\nimport pandas as pd\n\nh5f = h5py.File('bosch.h5', 'w')\n\ndims = 968\nchunksize = 1024 * 10\n\ntraining_data = pd.DataFrame()\ntesting_data = pd.DataFrame()\n\ntraining_reader = pd.read_csv('train_numeric.csv', chunksize=chunksize)\ntesting_reader = pd.read_csv('test_numeric.csv', chunksize=chunksize)\n\n\ntrain_data = np.empty((0,dims)).astype(np.float16)\ntargets = np.empty((0,1)).astype(np.int8)\ntrain_ids = np.empty((0,1)).astype(np.int32)\nfor chunk in training_reader:\n    d = np.asarray(chunk.ix[:, 1:-1]).astype(np.float16)\n    t = np.asarray(chunk.Response).reshape(-1, 1).astype(np.int8)\n    i = np.asarray(chunk.Id).reshape(-1, 1).astype(np.int32) \n    train_data = np.vstack((train_data, d))\n    targets = np.vstack((targets, t))\n    train_ids = np.vstack((train_ids, i))\n\nh5f.create_dataset('train_data', data=train_data, compression=&quot;gzip&quot;)\nh5f.create_dataset('train_targets', data=targets, compression=&quot;gzip&quot;)\nh5f.create_dataset('train_ids', data=train_ids, compression=&quot;gzip&quot;)\n\ndel train_data, targets, train_ids\n\ntest_data = np.empty((0,dims)).astype(np.float16)\ntest_ids = np.empty((0,1)).astype(np.int32)\nfor chunk in testing_reader:\n    d = np.asarray(chunk.ix[:, 1:]).astype(np.float16)\n    i = np.asarray(chunk.Id).reshape(-1, 1).astype(np.int32) \n    test_data = np.vstack((test_data, d))\n    test_ids = np.vstack((test_ids, i))\n\nh5f.create_dataset('test_data', data=test_data, compression=&quot;gzip&quot;)\nh5f.create_dataset('test_ids', data=test_ids, compression=&quot;gzip&quot;)\n\ndel test_data, test_ids\n\nh5f.close()\n</code></pre>",
      "rawMarkdown": "import numpy as np\r\n    import h5py\r\n    import pandas as pd\r\n    \r\n    h5f = h5py.File('bosch.h5', 'w')\r\n    \r\n    dims = 968\r\n    chunksize = 1024 * 10\r\n    \r\n    training_data = pd.DataFrame()\r\n    testing_data = pd.DataFrame()\r\n    \r\n    training_reader = pd.read_csv('train_numeric.csv', chunksize=chunksize)\r\n    testing_reader = pd.read_csv('test_numeric.csv', chunksize=chunksize)\r\n    \r\n    \r\n    train_data = np.empty((0,dims)).astype(np.float16)\r\n    targets = np.empty((0,1)).astype(np.int8)\r\n    train_ids = np.empty((0,1)).astype(np.int32)\r\n    for chunk in training_reader:\r\n        d = np.asarray(chunk.ix[:, 1:-1]).astype(np.float16)\r\n        t = np.asarray(chunk.Response).reshape(-1, 1).astype(np.int8)\r\n        i = np.asarray(chunk.Id).reshape(-1, 1).astype(np.int32) \r\n        train_data = np.vstack((train_data, d))\r\n        targets = np.vstack((targets, t))\r\n        train_ids = np.vstack((train_ids, i))\r\n    \r\n    h5f.create_dataset('train_data', data=train_data, compression=\"gzip\")\r\n    h5f.create_dataset('train_targets', data=targets, compression=\"gzip\")\r\n    h5f.create_dataset('train_ids', data=train_ids, compression=\"gzip\")\r\n    \r\n    del train_data, targets, train_ids\r\n    \r\n    test_data = np.empty((0,dims)).astype(np.float16)\r\n    test_ids = np.empty((0,1)).astype(np.int32)\r\n    for chunk in testing_reader:\r\n        d = np.asarray(chunk.ix[:, 1:]).astype(np.float16)\r\n        i = np.asarray(chunk.Id).reshape(-1, 1).astype(np.int32) \r\n        test_data = np.vstack((test_data, d))\r\n        test_ids = np.vstack((test_ids, i))\r\n    \r\n    h5f.create_dataset('test_data', data=test_data, compression=\"gzip\")\r\n    h5f.create_dataset('test_ids', data=test_ids, compression=\"gzip\")\r\n    \r\n    del test_data, test_ids\r\n    \r\n    h5f.close()",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 132466,
      "author_name": "mike459",
      "author_url": "",
      "post_date": "08/25/2016 19:25:36",
      "content": "<pre><code>import numpy as np\nimport h5py\nimport pandas as pd\n\nh5f = h5py.File('bosch.h5', 'w')\n\ndims = 968\nchunksize = 1024 * 10\n\ntraining_data = pd.DataFrame()\ntesting_data = pd.DataFrame()\n\ntraining_reader = pd.read_csv('train_numeric.csv', chunksize=chunksize)\ntesting_reader = pd.read_csv('test_numeric.csv', chunksize=chunksize)\n\n\ntrain_data = np.empty((0,dims)).astype(np.float16)\ntargets = np.empty((0,1)).astype(np.int8)\ntrain_ids = np.empty((0,1)).astype(np.int32)\nfor chunk in training_reader:\n    d = np.asarray(chunk.ix[:, 1:-1]).astype(np.float16)\n    t = np.asarray(chunk.Response).reshape(-1, 1).astype(np.int8)\n    i = np.asarray(chunk.Id).reshape(-1, 1).astype(np.int32) \n    train_data = np.vstack((train_data, d))\n    targets = np.vstack((targets, t))\n    train_ids = np.vstack((train_ids, i))\n\nh5f.create_dataset('train_data', data=train_data, compression=&quot;gzip&quot;)\nh5f.create_dataset('train_targets', data=targets, compression=&quot;gzip&quot;)\nh5f.create_dataset('train_ids', data=train_ids, compression=&quot;gzip&quot;)\n\ndel train_data, targets, train_ids\n\ntest_data = np.empty((0,dims)).astype(np.float16)\ntest_ids = np.empty((0,1)).astype(np.int32)\nfor chunk in testing_reader:\n    d = np.asarray(chunk.ix[:, 1:]).astype(np.float16)\n    i = np.asarray(chunk.Id).reshape(-1, 1).astype(np.int32) \n    test_data = np.vstack((test_data, d))\n    test_ids = np.vstack((test_ids, i))\n\nh5f.create_dataset('test_data', data=test_data, compression=&quot;gzip&quot;)\nh5f.create_dataset('test_ids', data=test_ids, compression=&quot;gzip&quot;)\n\ndel test_data, test_ids\n\nh5f.close()\n</code></pre>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "132441": "Hello all,\r\n\r\nI have never dealt with such a large data set.  So I am starting with the train_numeric.csv file. I have been able to load the file as a Panda data frame in the form of chunks\r\n\r\n   \r\n\r\n>  chunk =\r\n> pd.read_csv(\"C:/Users/FTS.fts-gnosis/Documents/kaggle/Bosch_kaggle/train_numeric.csv\",\r\n> dtype = np.float32, iterator=True, chunksize=1000)\r\n>         xtrain = pd.concat(chunk)\r\n\r\n but now the other problem is to store that dataframe so that I do not have to load it again and again. \r\n\r\nI have tried using  1.Pickle and 2 hdf but may be the dataframe is about 8.6GB so it does not work. \r\n\r\nAny other recommendations will be greatly appreciated.\r\n\r\nThanks",
    "132466": "import numpy as np\r\n    import h5py\r\n    import pandas as pd\r\n    \r\n    h5f = h5py.File('bosch.h5', 'w')\r\n    \r\n    dims = 968\r\n    chunksize = 1024 * 10\r\n    \r\n    training_data = pd.DataFrame()\r\n    testing_data = pd.DataFrame()\r\n    \r\n    training_reader = pd.read_csv('train_numeric.csv', chunksize=chunksize)\r\n    testing_reader = pd.read_csv('test_numeric.csv', chunksize=chunksize)\r\n    \r\n    \r\n    train_data = np.empty((0,dims)).astype(np.float16)\r\n    targets = np.empty((0,1)).astype(np.int8)\r\n    train_ids = np.empty((0,1)).astype(np.int32)\r\n    for chunk in training_reader:\r\n        d = np.asarray(chunk.ix[:, 1:-1]).astype(np.float16)\r\n        t = np.asarray(chunk.Response).reshape(-1, 1).astype(np.int8)\r\n        i = np.asarray(chunk.Id).reshape(-1, 1).astype(np.int32) \r\n        train_data = np.vstack((train_data, d))\r\n        targets = np.vstack((targets, t))\r\n        train_ids = np.vstack((train_ids, i))\r\n    \r\n    h5f.create_dataset('train_data', data=train_data, compression=\"gzip\")\r\n    h5f.create_dataset('train_targets', data=targets, compression=\"gzip\")\r\n    h5f.create_dataset('train_ids', data=train_ids, compression=\"gzip\")\r\n    \r\n    del train_data, targets, train_ids\r\n    \r\n    test_data = np.empty((0,dims)).astype(np.float16)\r\n    test_ids = np.empty((0,1)).astype(np.int32)\r\n    for chunk in testing_reader:\r\n        d = np.asarray(chunk.ix[:, 1:]).astype(np.float16)\r\n        i = np.asarray(chunk.Id).reshape(-1, 1).astype(np.int32) \r\n        test_data = np.vstack((test_data, d))\r\n        test_ids = np.vstack((test_ids, i))\r\n    \r\n    h5f.create_dataset('test_data', data=test_data, compression=\"gzip\")\r\n    h5f.create_dataset('test_ids', data=test_ids, compression=\"gzip\")\r\n    \r\n    del test_data, test_ids\r\n    \r\n    h5f.close()"
  },
  "source": "meta"
}