{
  "id": 23287,
  "title": "Reading by chunks in Python",
  "url": "/competitions/bosch-production-line-performance/discussion/23287",
  "author_name": "",
  "post_date": "2016-09-02T13:16:40.240Z",
  "votes": 1,
  "comment_count": 11,
  "views": 1575,
  "content": "<p>Here is a Pyhton 3.5 code snippet that shows how to read data in manageable chunks.  The example below extracts all positive instances (rows with Response equal to 1).</p>\n\n<p>If you want to run it on Kaggle then you must edit the paths and replace <code>Data</code> with <code>../input</code> I think.</p>\n\n<p>You can easily adapt it to any other need.  I am not sure it wors in Python 2.7.x because iterators aren't lazy there.</p>\n\n<pre><code>def get_positive(chunksize):\n    reader_numeric = pd.read_csv('Data/train_numeric.csv', chunksize=chunksize)\n    reader_categorical = pd.read_csv('Data/train_categorical.csv', chunksize=chunksize)\n    reader_date = pd.read_csv('Data/train_date.csv', chunksize=chunksize)\n    reader = zip(reader_numeric, reader_categorical, reader_date)\n    first = True\n    for numeric, categorical, date in reader:\n        categorical.drop('Id', axis=1, inplace=True)\n        date.drop('Id', axis=1, inplace=True)\n        data = pd.concat([numeric, categorical, date], axis=1)\n        positive_data = data[data.Response == 1]\n        if first:\n            positive = positive_data.copy()\n            first = False\n        else:\n            positive = pd.concat([positive, positive_data])\n        print(positive_data.shape, positive.shape)\n    return positive\n\npositive = get_positive(10000)\n</code></pre>\n\n<p>Edited to get rid of multiple Id columns</p>",
  "messages": [
    {
      "id": "133472",
      "postDate": "09/02/2016 13:16:40",
      "content": "<p>Here is a Pyhton 3.5 code snippet that shows how to read data in manageable chunks.  The example below extracts all positive instances (rows with Response equal to 1).</p>\n\n<p>If you want to run it on Kaggle then you must edit the paths and replace <code>Data</code> with <code>../input</code> I think.</p>\n\n<p>You can easily adapt it to any other need.  I am not sure it wors in Python 2.7.x because iterators aren't lazy there.</p>\n\n<pre><code>def get_positive(chunksize):\n    reader_numeric = pd.read_csv('Data/train_numeric.csv', chunksize=chunksize)\n    reader_categorical = pd.read_csv('Data/train_categorical.csv', chunksize=chunksize)\n    reader_date = pd.read_csv('Data/train_date.csv', chunksize=chunksize)\n    reader = zip(reader_numeric, reader_categorical, reader_date)\n    first = True\n    for numeric, categorical, date in reader:\n        categorical.drop('Id', axis=1, inplace=True)\n        date.drop('Id', axis=1, inplace=True)\n        data = pd.concat([numeric, categorical, date], axis=1)\n        positive_data = data[data.Response == 1]\n        if first:\n            positive = positive_data.copy()\n            first = False\n        else:\n            positive = pd.concat([positive, positive_data])\n        print(positive_data.shape, positive.shape)\n    return positive\n\npositive = get_positive(10000)\n</code></pre>\n\n<p>Edited to get rid of multiple Id columns</p>",
      "rawMarkdown": "Here is a Pyhton 3.5 code snippet that shows how to read data in manageable chunks.  The example below extracts all positive instances (rows with Response equal to 1).\r\n\r\n\r\nIf you want to run it on Kaggle then you must edit the paths and replace `Data` with `../input` I think.\r\n\r\nYou can easily adapt it to any other need.  I am not sure it wors in Python 2.7.x because iterators aren't lazy there.\r\n\r\n    def get_positive(chunksize):\r\n        reader_numeric = pd.read_csv('Data/train_numeric.csv', chunksize=chunksize)\r\n        reader_categorical = pd.read_csv('Data/train_categorical.csv', chunksize=chunksize)\r\n        reader_date = pd.read_csv('Data/train_date.csv', chunksize=chunksize)\r\n        reader = zip(reader_numeric, reader_categorical, reader_date)\r\n        first = True\r\n        for numeric, categorical, date in reader:\r\n            categorical.drop('Id', axis=1, inplace=True)\r\n            date.drop('Id', axis=1, inplace=True)\r\n            data = pd.concat([numeric, categorical, date], axis=1)\r\n            positive_data = data[data.Response == 1]\r\n            if first:\r\n                positive = positive_data.copy()\r\n                first = False\r\n            else:\r\n                positive = pd.concat([positive, positive_data])\r\n            print(positive_data.shape, positive.shape)\r\n        return positive\r\n    \r\n    positive = get_positive(10000)\r\n\r\nEdited to get rid of multiple Id columns",
      "votes": null
    },
    {
      "id": "134121",
      "postDate": "09/05/2016 07:30:06",
      "content": "<p>Hi,\nI tried your script to basically concat all the tables, without any condition check. It gives the following error: </p>\n\n<pre><code>for i in range(0, total_len, chunk_len):\nOverflowError: Python int too large to convert to C long\n</code></pre>\n\n<p>Can you please help me?</p>",
      "rawMarkdown": "Hi,\r\nI tried your script to basically concat all the tables, without any condition check. It gives the following error: \r\n\r\n    for i in range(0, total_len, chunk_len):\r\n    OverflowError: Python int too large to convert to C long\r\n\r\nCan you please help me?",
      "votes": null
    },
    {
      "id": "134155",
      "postDate": "09/05/2016 13:00:17",
      "content": "<p>Are you showing all your code?\nwhat are the values of total_len and chunk_len?\nwhich Python version?</p>\n\n<p>For basic Python debug you should ask stackoverflow.</p>",
      "rawMarkdown": "Are you showing all your code?\r\nwhat are the values of total_len and chunk_len?\r\nwhich Python version?\r\n\r\nFor basic Python debug you should ask stackoverflow.",
      "votes": null
    },
    {
      "id": "134156",
      "postDate": "09/05/2016 13:07:28",
      "content": "<p>Please find the script below. On running the script I get the error that I mentioned in  my previous. It is a trace back.  </p>\n\n<pre><code>def get_testing(chunksize):\n\n    reader_numeric = pd.read_csv('/data/test_numeric.csv', chunksize=chunksize,dtype = dtype_numeric)\n    reader_categorical = pd.read_csv('/data/test_categorical.csv', chunksize=chunksize, dtype = dtype_categorical)\n    reader_date = pd.read_csv('/data/test_date.csv', chunksize=chunksize, dtype = dtype_date)\n    reader = zip(reader_numeric, reader_categorical, reader_date)\n    first = True\n    for numeric, categorical, date in reader:\n        data = pd.concat([numeric, categorical, date], axis=1)\n        if first:\n            data1 = data.copy()\n            first = False\n        else:\n            data1 = pd.concat([data1,data])\n    return data1\n\ntrain_data = get_training(10000)\n</code></pre>",
      "rawMarkdown": "Please find the script below. On running the script I get the error that I mentioned in  my previous. It is a trace back.  \r\n\r\n    def get_testing(chunksize):\r\n\r\n        reader_numeric = pd.read_csv('/data/test_numeric.csv', chunksize=chunksize,dtype = dtype_numeric)\r\n        reader_categorical = pd.read_csv('/data/test_categorical.csv', chunksize=chunksize, dtype = dtype_categorical)\r\n        reader_date = pd.read_csv('/data/test_date.csv', chunksize=chunksize, dtype = dtype_date)\r\n        reader = zip(reader_numeric, reader_categorical, reader_date)\r\n        first = True\r\n        for numeric, categorical, date in reader:\r\n            data = pd.concat([numeric, categorical, date], axis=1)\r\n            if first:\r\n                data1 = data.copy()\r\n                first = False\r\n            else:\r\n                data1 = pd.concat([data1,data])\r\n        return data1\r\n\r\n    train_data = get_training(10000)",
      "votes": null
    },
    {
      "id": "134157",
      "postDate": "09/05/2016 13:14:25",
      "content": "<p>what are the values of your dtypes?</p>\n\n<p>aare you using the orginal data or transformed data (eg turning categorical into numerics)? if the latter, then I guess you are trying to read a large integer into an int32.  </p>\n\n<p>Try without any dtype argument.</p>\n\n<p>edit: I think you are using dtype='int' when reading csv files with NaNs.  You should read files with NaNs with a float type, for instance dtype='float32'</p>",
      "rawMarkdown": "what are the values of your dtypes?\r\n\r\naare you using the orginal data or transformed data (eg turning categorical into numerics)? if the latter, then I guess you are trying to read a large integer into an int32.  \r\n\r\nTry without any dtype argument.\r\n\r\nedit: I think you are using dtype='int' when reading csv files with NaNs.  You should read files with NaNs with a float type, for instance dtype='float32'",
      "votes": null
    },
    {
      "id": "134160",
      "postDate": "09/05/2016 13:21:19",
      "content": "<p>For numeric the dtype is np.float16 and the categorical the dtype is float64 as there are strings involved. The data is original.\n If I turn categorical to numerical will the random forests work well. I am using One hot encoding right now, which I think is recommended for Random Forests.</p>",
      "rawMarkdown": "For numeric the dtype is np.float16 and the categorical the dtype is float64 as there are strings involved. The data is original.\r\n If I turn categorical to numerical will the random forests work well. I am using One hot encoding right now, which I think is recommended for Random Forests.",
      "votes": null
    },
    {
      "id": "134169",
      "postDate": "09/05/2016 15:36:27",
      "content": "<p>try dtype='float32' as I am not sure np.float16 can handle NaN.</p>",
      "rawMarkdown": "try dtype='float32' as I am not sure np.float16 can handle NaN.",
      "votes": null
    },
    {
      "id": "134172",
      "postDate": "09/05/2016 15:54:25",
      "content": "<p>Ok. I will try but just a question. will it not give an error if np.float16 is not able to load the numeric file? Right now it is not giving an error .</p>",
      "rawMarkdown": "Ok. I will try but just a question. will it not give an error if np.float16 is not able to load the numeric file? Right now it is not giving an error .",
      "votes": null
    },
    {
      "id": "134211",
      "postDate": "09/05/2016 19:46:16",
      "content": "<p>I think you are using an integer dtype somewhere.  If you do then you cannot read a file that contains NaN (missing values).</p>\n\n<p>Please confirm (or not).</p>",
      "rawMarkdown": "I think you are using an integer dtype somewhere.  If you do then you cannot read a file that contains NaN (missing values).\r\n\r\nPlease confirm (or not).",
      "votes": null
    },
    {
      "id": "134220",
      "postDate": "09/05/2016 21:35:57",
      "content": "<p>No . I am using int for Response and index. the issue is not with the reading of the files but concatenation is an issue(when the size of the dataframe increases).</p>",
      "rawMarkdown": "No . I am using int for Response and index. the issue is not with the reading of the files but concatenation is an issue(when the size of the dataframe increases).",
      "votes": null
    },
    {
      "id": "134823",
      "postDate": "09/08/2016 18:10:27",
      "content": "<p>Hi Zohair, are you using Python2 or Python3? I wonder if your issue is due to the fact that the zip() function behaves differently in Python2 vs Python3.</p>\n\n<p>In Python3 (CPMP's code), zip() returns an iterator, but in Python2 zip() returns a list. In Python 2, izip() returns an iterator like Python3's zip().</p>\n\n<p>If switching to Python3, or izip() in Python2 doesn't solve your problem, could you post your full stack trace of your error?</p>",
      "rawMarkdown": "Hi Zohair, are you using Python2 or Python3? I wonder if your issue is due to the fact that the zip() function behaves differently in Python2 vs Python3.\r\n\r\nIn Python3 (CPMP's code), zip() returns an iterator, but in Python2 zip() returns a list. In Python 2, izip() returns an iterator like Python3's zip().\r\n\r\nIf switching to Python3, or izip() in Python2 doesn't solve your problem, could you post your full stack trace of your error?",
      "votes": null
    },
    {
      "id": "134824",
      "postDate": "09/08/2016 18:27:09",
      "content": "<p>Hi Max, \nThe issue is with panda concatenation. I think it can not handle such a huge data. I am trying to do it for the whole data. Well I figured out a way for that as well. thanks</p>",
      "rawMarkdown": "Hi Max, \r\nThe issue is with panda concatenation. I think it can not handle such a huge data. I am trying to do it for the whole data. Well I figured out a way for that as well. thanks",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 134121,
      "author_name": "zohairzohair4",
      "author_url": "",
      "post_date": "09/05/2016 07:30:06",
      "content": "<p>Hi,\nI tried your script to basically concat all the tables, without any condition check. It gives the following error: </p>\n\n<pre><code>for i in range(0, total_len, chunk_len):\nOverflowError: Python int too large to convert to C long\n</code></pre>\n\n<p>Can you please help me?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134155,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "09/05/2016 13:00:17",
      "content": "<p>Are you showing all your code?\nwhat are the values of total_len and chunk_len?\nwhich Python version?</p>\n\n<p>For basic Python debug you should ask stackoverflow.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134156,
      "author_name": "zohairzohair4",
      "author_url": "",
      "post_date": "09/05/2016 13:07:28",
      "content": "<p>Please find the script below. On running the script I get the error that I mentioned in  my previous. It is a trace back.  </p>\n\n<pre><code>def get_testing(chunksize):\n\n    reader_numeric = pd.read_csv('/data/test_numeric.csv', chunksize=chunksize,dtype = dtype_numeric)\n    reader_categorical = pd.read_csv('/data/test_categorical.csv', chunksize=chunksize, dtype = dtype_categorical)\n    reader_date = pd.read_csv('/data/test_date.csv', chunksize=chunksize, dtype = dtype_date)\n    reader = zip(reader_numeric, reader_categorical, reader_date)\n    first = True\n    for numeric, categorical, date in reader:\n        data = pd.concat([numeric, categorical, date], axis=1)\n        if first:\n            data1 = data.copy()\n            first = False\n        else:\n            data1 = pd.concat([data1,data])\n    return data1\n\ntrain_data = get_training(10000)\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134157,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "09/05/2016 13:14:25",
      "content": "<p>what are the values of your dtypes?</p>\n\n<p>aare you using the orginal data or transformed data (eg turning categorical into numerics)? if the latter, then I guess you are trying to read a large integer into an int32.  </p>\n\n<p>Try without any dtype argument.</p>\n\n<p>edit: I think you are using dtype='int' when reading csv files with NaNs.  You should read files with NaNs with a float type, for instance dtype='float32'</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134160,
      "author_name": "zohairzohair4",
      "author_url": "",
      "post_date": "09/05/2016 13:21:19",
      "content": "<p>For numeric the dtype is np.float16 and the categorical the dtype is float64 as there are strings involved. The data is original.\n If I turn categorical to numerical will the random forests work well. I am using One hot encoding right now, which I think is recommended for Random Forests.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134169,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "09/05/2016 15:36:27",
      "content": "<p>try dtype='float32' as I am not sure np.float16 can handle NaN.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134172,
      "author_name": "zohairzohair4",
      "author_url": "",
      "post_date": "09/05/2016 15:54:25",
      "content": "<p>Ok. I will try but just a question. will it not give an error if np.float16 is not able to load the numeric file? Right now it is not giving an error .</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134211,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "09/05/2016 19:46:16",
      "content": "<p>I think you are using an integer dtype somewhere.  If you do then you cannot read a file that contains NaN (missing values).</p>\n\n<p>Please confirm (or not).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134220,
      "author_name": "zohairzohair4",
      "author_url": "",
      "post_date": "09/05/2016 21:35:57",
      "content": "<p>No . I am using int for Response and index. the issue is not with the reading of the files but concatenation is an issue(when the size of the dataframe increases).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134823,
      "author_name": "maxpowerwastaken",
      "author_url": "",
      "post_date": "09/08/2016 18:10:27",
      "content": "<p>Hi Zohair, are you using Python2 or Python3? I wonder if your issue is due to the fact that the zip() function behaves differently in Python2 vs Python3.</p>\n\n<p>In Python3 (CPMP's code), zip() returns an iterator, but in Python2 zip() returns a list. In Python 2, izip() returns an iterator like Python3's zip().</p>\n\n<p>If switching to Python3, or izip() in Python2 doesn't solve your problem, could you post your full stack trace of your error?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134824,
      "author_name": "zohairzohair4",
      "author_url": "",
      "post_date": "09/08/2016 18:27:09",
      "content": "<p>Hi Max, \nThe issue is with panda concatenation. I think it can not handle such a huge data. I am trying to do it for the whole data. Well I figured out a way for that as well. thanks</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "133472": "Here is a Pyhton 3.5 code snippet that shows how to read data in manageable chunks.  The example below extracts all positive instances (rows with Response equal to 1).\r\n\r\n\r\nIf you want to run it on Kaggle then you must edit the paths and replace `Data` with `../input` I think.\r\n\r\nYou can easily adapt it to any other need.  I am not sure it wors in Python 2.7.x because iterators aren't lazy there.\r\n\r\n    def get_positive(chunksize):\r\n        reader_numeric = pd.read_csv('Data/train_numeric.csv', chunksize=chunksize)\r\n        reader_categorical = pd.read_csv('Data/train_categorical.csv', chunksize=chunksize)\r\n        reader_date = pd.read_csv('Data/train_date.csv', chunksize=chunksize)\r\n        reader = zip(reader_numeric, reader_categorical, reader_date)\r\n        first = True\r\n        for numeric, categorical, date in reader:\r\n            categorical.drop('Id', axis=1, inplace=True)\r\n            date.drop('Id', axis=1, inplace=True)\r\n            data = pd.concat([numeric, categorical, date], axis=1)\r\n            positive_data = data[data.Response == 1]\r\n            if first:\r\n                positive = positive_data.copy()\r\n                first = False\r\n            else:\r\n                positive = pd.concat([positive, positive_data])\r\n            print(positive_data.shape, positive.shape)\r\n        return positive\r\n    \r\n    positive = get_positive(10000)\r\n\r\nEdited to get rid of multiple Id columns",
    "134121": "Hi,\r\nI tried your script to basically concat all the tables, without any condition check. It gives the following error: \r\n\r\n    for i in range(0, total_len, chunk_len):\r\n    OverflowError: Python int too large to convert to C long\r\n\r\nCan you please help me?",
    "134155": "Are you showing all your code?\r\nwhat are the values of total_len and chunk_len?\r\nwhich Python version?\r\n\r\nFor basic Python debug you should ask stackoverflow.",
    "134156": "Please find the script below. On running the script I get the error that I mentioned in  my previous. It is a trace back.  \r\n\r\n    def get_testing(chunksize):\r\n\r\n        reader_numeric = pd.read_csv('/data/test_numeric.csv', chunksize=chunksize,dtype = dtype_numeric)\r\n        reader_categorical = pd.read_csv('/data/test_categorical.csv', chunksize=chunksize, dtype = dtype_categorical)\r\n        reader_date = pd.read_csv('/data/test_date.csv', chunksize=chunksize, dtype = dtype_date)\r\n        reader = zip(reader_numeric, reader_categorical, reader_date)\r\n        first = True\r\n        for numeric, categorical, date in reader:\r\n            data = pd.concat([numeric, categorical, date], axis=1)\r\n            if first:\r\n                data1 = data.copy()\r\n                first = False\r\n            else:\r\n                data1 = pd.concat([data1,data])\r\n        return data1\r\n\r\n    train_data = get_training(10000)",
    "134157": "what are the values of your dtypes?\r\n\r\naare you using the orginal data or transformed data (eg turning categorical into numerics)? if the latter, then I guess you are trying to read a large integer into an int32.  \r\n\r\nTry without any dtype argument.\r\n\r\nedit: I think you are using dtype='int' when reading csv files with NaNs.  You should read files with NaNs with a float type, for instance dtype='float32'",
    "134160": "For numeric the dtype is np.float16 and the categorical the dtype is float64 as there are strings involved. The data is original.\r\n If I turn categorical to numerical will the random forests work well. I am using One hot encoding right now, which I think is recommended for Random Forests.",
    "134169": "try dtype='float32' as I am not sure np.float16 can handle NaN.",
    "134172": "Ok. I will try but just a question. will it not give an error if np.float16 is not able to load the numeric file? Right now it is not giving an error .",
    "134211": "I think you are using an integer dtype somewhere.  If you do then you cannot read a file that contains NaN (missing values).\r\n\r\nPlease confirm (or not).",
    "134220": "No . I am using int for Response and index. the issue is not with the reading of the files but concatenation is an issue(when the size of the dataframe increases).",
    "134823": "Hi Zohair, are you using Python2 or Python3? I wonder if your issue is due to the fact that the zip() function behaves differently in Python2 vs Python3.\r\n\r\nIn Python3 (CPMP's code), zip() returns an iterator, but in Python2 zip() returns a list. In Python 2, izip() returns an iterator like Python3's zip().\r\n\r\nIf switching to Python3, or izip() in Python2 doesn't solve your problem, could you post your full stack trace of your error?",
    "134824": "Hi Max, \r\nThe issue is with panda concatenation. I think it can not handle such a huge data. I am trying to do it for the whole data. Well I figured out a way for that as well. thanks"
  },
  "source": "meta"
}