{
  "id": 24056,
  "title": "File is too large to open, how do I get first look at the data?",
  "url": "/competitions/bosch-production-line-performance/discussion/24056",
  "author_name": "",
  "post_date": "2016-10-03T18:08:05.663Z",
  "votes": null,
  "comment_count": 2,
  "views": 580,
  "content": "<p>Hi,</p>\n\n<p>So the individual csv files are about 2.5 GB each. I can't open it in Excel, Notepad, or Notepad++.</p>\n\n<p>I tried loading it in python with pandas.open_csv and printing data.head()  and it ran until my computer ran out of resources and I had to end it.</p>\n\n<p>Any tips on where to start with looking at this data? or ways to slice it to a manageable size before I try to open it?</p>\n\n<p>Thanks</p>",
  "messages": [
    {
      "id": "137656",
      "postDate": "10/03/2016 18:08:05",
      "content": "<p>Hi,</p>\n\n<p>So the individual csv files are about 2.5 GB each. I can't open it in Excel, Notepad, or Notepad++.</p>\n\n<p>I tried loading it in python with pandas.open_csv and printing data.head()  and it ran until my computer ran out of resources and I had to end it.</p>\n\n<p>Any tips on where to start with looking at this data? or ways to slice it to a manageable size before I try to open it?</p>\n\n<p>Thanks</p>",
      "rawMarkdown": "Hi,\r\n\r\nSo the individual csv files are about 2.5 GB each. I can't open it in Excel, Notepad, or Notepad++.\r\n\r\nI tried loading it in python with pandas.open_csv and printing data.head()  and it ran until my computer ran out of resources and I had to end it.\r\n\r\nAny tips on where to start with looking at this data? or ways to slice it to a manageable size before I try to open it?\r\n\r\nThanks",
      "votes": null
    },
    {
      "id": "137662",
      "postDate": "10/03/2016 19:08:36",
      "content": "<p>I suggest you go to the <a href=\"https://www.kaggle.com/c/bosch-production-line-performance/kernels\"><strong>kernels</strong></a> page and start reading. There are few scripts that explicitly advertise low memory solution in their titles, but just about all script contain tricks for handling memory problems.</p>\n\n<p>Two general groups of solutions are: 1) reading only a select group of columns rather than the whole file;  2) reading data in manageable chunks and processing them in step-wise fashion.</p>\n\n<p>A bit of advice: you have to approach these problems as something that is not unique to you, and has therefore been addressed since this competition is almost 50 days old. If you browse the available pages, you will find out lots of answers and good pointers for future exploration.</p>",
      "rawMarkdown": "I suggest you go to the [__kernels__](https://www.kaggle.com/c/bosch-production-line-performance/kernels) page and start reading. There are few scripts that explicitly advertise low memory solution in their titles, but just about all script contain tricks for handling memory problems.\r\n\r\nTwo general groups of solutions are: 1) reading only a select group of columns rather than the whole file;  2) reading data in manageable chunks and processing them in step-wise fashion.\r\n\r\nA bit of advice: you have to approach these problems as something that is not unique to you, and has therefore been addressed since this competition is almost 50 days old. If you browse the available pages, you will find out lots of answers and good pointers for future exploration.",
      "votes": null
    },
    {
      "id": "137664",
      "postDate": "10/03/2016 19:55:17",
      "content": "<p>Glogg is a good option if you want to browse very large txt files...</p>\n\n<p><a href=\"http://glogg.bonnefon.org/\">http://glogg.bonnefon.org/</a></p>\n\n<p>That's how I observed sequential duplicates as mentioned here <a href=\"https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23553/sequential-duplicates-in-test-files/135083\">https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23553/sequential-duplicates-in-test-files/135083</a></p>\n\n<p>Browsing in glogg isn't going to get you close to a submission. So, as Tilii says, have a look at the kernels for examples of how to read, explore and make submissions in low memory.</p>\n\n<p>Good luck!</p>",
      "rawMarkdown": "Glogg is a good option if you want to browse very large txt files...\r\n\r\nhttp://glogg.bonnefon.org/\r\n\r\nThat's how I observed sequential duplicates as mentioned here https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23553/sequential-duplicates-in-test-files/135083\r\n\r\nBrowsing in glogg isn't going to get you close to a submission. So, as Tilii says, have a look at the kernels for examples of how to read, explore and make submissions in low memory.\r\n\r\nGood luck!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 137662,
      "author_name": "tilii7",
      "author_url": "",
      "post_date": "10/03/2016 19:08:36",
      "content": "<p>I suggest you go to the <a href=\"https://www.kaggle.com/c/bosch-production-line-performance/kernels\"><strong>kernels</strong></a> page and start reading. There are few scripts that explicitly advertise low memory solution in their titles, but just about all script contain tricks for handling memory problems.</p>\n\n<p>Two general groups of solutions are: 1) reading only a select group of columns rather than the whole file;  2) reading data in manageable chunks and processing them in step-wise fashion.</p>\n\n<p>A bit of advice: you have to approach these problems as something that is not unique to you, and has therefore been addressed since this competition is almost 50 days old. If you browse the available pages, you will find out lots of answers and good pointers for future exploration.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137664,
      "author_name": "nigelcarpenter",
      "author_url": "",
      "post_date": "10/03/2016 19:55:17",
      "content": "<p>Glogg is a good option if you want to browse very large txt files...</p>\n\n<p><a href=\"http://glogg.bonnefon.org/\">http://glogg.bonnefon.org/</a></p>\n\n<p>That's how I observed sequential duplicates as mentioned here <a href=\"https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23553/sequential-duplicates-in-test-files/135083\">https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23553/sequential-duplicates-in-test-files/135083</a></p>\n\n<p>Browsing in glogg isn't going to get you close to a submission. So, as Tilii says, have a look at the kernels for examples of how to read, explore and make submissions in low memory.</p>\n\n<p>Good luck!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "137656": "Hi,\r\n\r\nSo the individual csv files are about 2.5 GB each. I can't open it in Excel, Notepad, or Notepad++.\r\n\r\nI tried loading it in python with pandas.open_csv and printing data.head()  and it ran until my computer ran out of resources and I had to end it.\r\n\r\nAny tips on where to start with looking at this data? or ways to slice it to a manageable size before I try to open it?\r\n\r\nThanks",
    "137662": "I suggest you go to the [__kernels__](https://www.kaggle.com/c/bosch-production-line-performance/kernels) page and start reading. There are few scripts that explicitly advertise low memory solution in their titles, but just about all script contain tricks for handling memory problems.\r\n\r\nTwo general groups of solutions are: 1) reading only a select group of columns rather than the whole file;  2) reading data in manageable chunks and processing them in step-wise fashion.\r\n\r\nA bit of advice: you have to approach these problems as something that is not unique to you, and has therefore been addressed since this competition is almost 50 days old. If you browse the available pages, you will find out lots of answers and good pointers for future exploration.",
    "137664": "Glogg is a good option if you want to browse very large txt files...\r\n\r\nhttp://glogg.bonnefon.org/\r\n\r\nThat's how I observed sequential duplicates as mentioned here https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23553/sequential-duplicates-in-test-files/135083\r\n\r\nBrowsing in glogg isn't going to get you close to a submission. So, as Tilii says, have a look at the kernels for examples of how to read, explore and make submissions in low memory.\r\n\r\nGood luck!"
  },
  "source": "meta"
}