{
  "id": 73726,
  "title": "Reading historical_transactions.csv file",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/73726",
  "author_name": "",
  "post_date": "2018-12-05T06:50:06.260501700Z",
  "votes": 1,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Hi All,\ni am trying to read the historical_transactions.csv file in jupyter notebook as well as kaggle kernel \nbut i am not able to read it because of it large volume of data. Can you any one please guide how to sort out this problem to analyse this file</p>",
  "messages": [
    {
      "id": "433532",
      "postDate": "12/05/2018 06:50:06",
      "content": "<p>Hi All,\ni am trying to read the historical_transactions.csv file in jupyter notebook as well as kaggle kernel \nbut i am not able to read it because of it large volume of data. Can you any one please guide how to sort out this problem to analyse this file</p>",
      "rawMarkdown": "Hi All,\ni am trying to read the historical_transactions.csv file in jupyter notebook as well as kaggle kernel \nbut i am not able to read it because of it large volume of data. Can you any one please guide how to sort out this problem to analyse this file",
      "votes": null
    },
    {
      "id": "433588",
      "postDate": "12/05/2018 07:53:06",
      "content": "<p>You can check you PC memory size. If it is smaller than the size of  historical_transactions.csv, then your computer cannot read it.</p>",
      "rawMarkdown": "You can check you PC memory size. If it is smaller than the size of  historical_transactions.csv, then your computer cannot read it.",
      "votes": null
    },
    {
      "id": "433994",
      "postDate": "12/05/2018 18:39:39",
      "content": "<p>My private kaggle kernel (Rmd) loads historical_transactions.csv and creates aggregations without any problems.</p>\n\n<p>If you PC doesn't have enough memory you can at least try to load only the first rows - pandas.read_csv(...nmax=10000) or read_csv(...n_max = 10000).</p>",
      "rawMarkdown": "My private kaggle kernel (Rmd) loads historical_transactions.csv and creates aggregations without any problems.\n\nIf you PC doesn't have enough memory you can at least try to load only the first rows - pandas.read\\_csv(...nmax=10000) or read\\_csv(...n_max = 10000).",
      "votes": null
    },
    {
      "id": "434370",
      "postDate": "12/06/2018 09:26:00",
      "content": "<p>i was able to read historical_transactions.csv on kaggle kernel with pandas as usual. try it again.</p>",
      "rawMarkdown": "i was able to read historical_transactions.csv on kaggle kernel with pandas as usual. try it again.",
      "votes": null
    },
    {
      "id": "435336",
      "postDate": "12/07/2018 21:51:43",
      "content": "<p>Memory should be 2-3 times bigger than the file size.</p>\n\n<p>Just one another option but slower of course, there is a chance to use hard disk. I dont remember how, I will try to find it and write it here</p>",
      "rawMarkdown": "Memory should be 2-3 times bigger than the file size.\n\nJust one another option but slower of course, there is a chance to use hard disk. I dont remember how, I will try to find it and write it here",
      "votes": null
    },
    {
      "id": "437177",
      "postDate": "12/11/2018 13:45:44",
      "content": "<p>delete unused items, that will help you to reduce RAM usage. You can also first read data, then make aggregations and save it to different csv. good luck</p>",
      "rawMarkdown": "delete unused items, that will help you to reduce RAM usage. You can also first read data, then make aggregations and save it to different csv. good luck",
      "votes": null
    },
    {
      "id": "459576",
      "postDate": "01/22/2019 02:49:40",
      "content": "<p>You can read it splitting it in chunks if your memory isn´t enough.</p>\n\n<p>Using pandas:</p>\n\n<p>reader = pd.read_csv(\"file.csv\",  ..., chunksize=[number of lines in each chunk], iterator=True)</p>\n\n<p>for chunk in reader:\n   process chunk ...</p>\n\n<p>Or you can try it in an environment with memory large enough, kaggle kernels may do the job.</p>",
      "rawMarkdown": "You can read it splitting it in chunks if your memory isn´t enough.\n\nUsing pandas:\n\nreader = pd.read_csv(\"file.csv\",  ..., chunksize=[number of lines in each chunk], iterator=True)\n\nfor chunk in reader:\n   process chunk ...\n\nOr you can try it in an environment with memory large enough, kaggle kernels may do the job.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 433588,
      "author_name": "jinjunding",
      "author_url": "",
      "post_date": "12/05/2018 07:53:06",
      "content": "<p>You can check you PC memory size. If it is smaller than the size of  historical_transactions.csv, then your computer cannot read it.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 433994,
      "author_name": "kailex",
      "author_url": "",
      "post_date": "12/05/2018 18:39:39",
      "content": "<p>My private kaggle kernel (Rmd) loads historical_transactions.csv and creates aggregations without any problems.</p>\n\n<p>If you PC doesn't have enough memory you can at least try to load only the first rows - pandas.read_csv(...nmax=10000) or read_csv(...n_max = 10000).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 434370,
      "author_name": "hakymulla",
      "author_url": "",
      "post_date": "12/06/2018 09:26:00",
      "content": "<p>i was able to read historical_transactions.csv on kaggle kernel with pandas as usual. try it again.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 435336,
      "author_name": "hozler",
      "author_url": "",
      "post_date": "12/07/2018 21:51:43",
      "content": "<p>Memory should be 2-3 times bigger than the file size.</p>\n\n<p>Just one another option but slower of course, there is a chance to use hard disk. I dont remember how, I will try to find it and write it here</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 437177,
      "author_name": "bcakal15",
      "author_url": "",
      "post_date": "12/11/2018 13:45:44",
      "content": "<p>delete unused items, that will help you to reduce RAM usage. You can also first read data, then make aggregations and save it to different csv. good luck</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 459576,
      "author_name": "jplamesa86",
      "author_url": "",
      "post_date": "01/22/2019 02:49:40",
      "content": "<p>You can read it splitting it in chunks if your memory isn´t enough.</p>\n\n<p>Using pandas:</p>\n\n<p>reader = pd.read_csv(\"file.csv\",  ..., chunksize=[number of lines in each chunk], iterator=True)</p>\n\n<p>for chunk in reader:\n   process chunk ...</p>\n\n<p>Or you can try it in an environment with memory large enough, kaggle kernels may do the job.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "433532": "Hi All,\ni am trying to read the historical_transactions.csv file in jupyter notebook as well as kaggle kernel \nbut i am not able to read it because of it large volume of data. Can you any one please guide how to sort out this problem to analyse this file",
    "433588": "You can check you PC memory size. If it is smaller than the size of  historical_transactions.csv, then your computer cannot read it.",
    "433994": "My private kaggle kernel (Rmd) loads historical_transactions.csv and creates aggregations without any problems.\n\nIf you PC doesn't have enough memory you can at least try to load only the first rows - pandas.read\\_csv(...nmax=10000) or read\\_csv(...n_max = 10000).",
    "434370": "i was able to read historical_transactions.csv on kaggle kernel with pandas as usual. try it again.",
    "435336": "Memory should be 2-3 times bigger than the file size.\n\nJust one another option but slower of course, there is a chance to use hard disk. I dont remember how, I will try to find it and write it here",
    "437177": "delete unused items, that will help you to reduce RAM usage. You can also first read data, then make aggregations and save it to different csv. good luck",
    "459576": "You can read it splitting it in chunks if your memory isn´t enough.\n\nUsing pandas:\n\nreader = pd.read_csv(\"file.csv\",  ..., chunksize=[number of lines in each chunk], iterator=True)\n\nfor chunk in reader:\n   process chunk ...\n\nOr you can try it in an environment with memory large enough, kaggle kernels may do the job."
  },
  "source": "meta"
}