{
  "id": 45424,
  "title": "Partially reading the Train Data set ",
  "url": "/competitions/favorita-grocery-sales-forecasting/discussion/45424",
  "author_name": "",
  "post_date": "2017-12-11T09:51:59.974699900Z",
  "votes": 1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Is there a way i can partially read the train file without completly loading the Train dataset.  I use only 16gb ram. I tried loading the complete data set it threw me an error </p>",
  "messages": [
    {
      "id": "256145",
      "postDate": "12/11/2017 09:51:59",
      "content": "<p>Is there a way i can partially read the train file without completly loading the Train dataset.  I use only 16gb ram. I tried loading the complete data set it threw me an error </p>",
      "rawMarkdown": "Is there a way i can partially read the train file without completly loading the Train dataset.  I use only 16gb ram. I tried loading the complete data set it threw me an error",
      "votes": null
    },
    {
      "id": "256152",
      "postDate": "12/11/2017 10:40:30",
      "content": "<p>You can try using <a href=\"https://www.sublimetext.com/3\">SublimeText</a></p>",
      "rawMarkdown": "You can try using [SublimeText][1]\n\n\n  [1]: https://www.sublimetext.com/3",
      "votes": null
    },
    {
      "id": "256480",
      "postDate": "12/12/2017 03:16:43",
      "content": "<p>There are several kernels which can be helpful for you. Such as <a href=\"https://www.kaggle.com/ceshine/lgbm-starter\">https://www.kaggle.com/ceshine/lgbm-starter</a></p>",
      "rawMarkdown": "There are several kernels which can be helpful for you. Such as https://www.kaggle.com/ceshine/lgbm-starter",
      "votes": null
    },
    {
      "id": "258914",
      "postDate": "12/17/2017 09:58:45",
      "content": "<p>Hi,\nIf you're using pandas you can load the data in chunks and apply a filter for every chunk, then concatenate the filtered chunks to get a much smaller dataframe. In the following example I only keep august 2016 from the train data:</p>\n\n<pre><code>train_list = []\nchunk_iter = pd.read_csv(r\"Data\\train.csv\", parse_dates=[\"date\"], chunksize=1000000)\nfor chunk in chunk_iter:\n    chunk_kept = chunk[(chunk[\"date\"].dt.year == 2016) &amp; (chunk[\"date\"].dt.month == 8)]\n    train_list.append(chunk_kept)\ndf_train = pd.concat(train_list)\n</code></pre>\n\n<p>Hope this helps</p>",
      "rawMarkdown": "Hi,\nIf you're using pandas you can load the data in chunks and apply a filter for every chunk, then concatenate the filtered chunks to get a much smaller dataframe. In the following example I only keep august 2016 from the train data:\n\n    train_list = []\n    chunk_iter = pd.read_csv(r\"Data\\train.csv\", parse_dates=[\"date\"], chunksize=1000000)\n    for chunk in chunk_iter:\n        chunk_kept = chunk[(chunk[\"date\"].dt.year == 2016) &amp; (chunk[\"date\"].dt.month == 8)]\n        train_list.append(chunk_kept)\n    df_train = pd.concat(train_list)\n\nHope this helps",
      "votes": null
    },
    {
      "id": "259129",
      "postDate": "12/17/2017 18:51:43",
      "content": "<p>If you use Python, a convenient way for loading part of the data is by using skiprows as argument of read_csv, e.g:\npandas.read_csv(<em>file</em>,skiprows=range(a,b))</p>",
      "rawMarkdown": "If you use Python, a convenient way for loading part of the data is by using skiprows as argument of read_csv, e.g:\npandas.read_csv(*file*,skiprows=range(a,b))",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 256152,
      "author_name": "mrbeer",
      "author_url": "",
      "post_date": "12/11/2017 10:40:30",
      "content": "<p>You can try using <a href=\"https://www.sublimetext.com/3\">SublimeText</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 256480,
      "author_name": "kelexu",
      "author_url": "",
      "post_date": "12/12/2017 03:16:43",
      "content": "<p>There are several kernels which can be helpful for you. Such as <a href=\"https://www.kaggle.com/ceshine/lgbm-starter\">https://www.kaggle.com/ceshine/lgbm-starter</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 258914,
      "author_name": "medkar",
      "author_url": "",
      "post_date": "12/17/2017 09:58:45",
      "content": "<p>Hi,\nIf you're using pandas you can load the data in chunks and apply a filter for every chunk, then concatenate the filtered chunks to get a much smaller dataframe. In the following example I only keep august 2016 from the train data:</p>\n\n<pre><code>train_list = []\nchunk_iter = pd.read_csv(r\"Data\\train.csv\", parse_dates=[\"date\"], chunksize=1000000)\nfor chunk in chunk_iter:\n    chunk_kept = chunk[(chunk[\"date\"].dt.year == 2016) &amp; (chunk[\"date\"].dt.month == 8)]\n    train_list.append(chunk_kept)\ndf_train = pd.concat(train_list)\n</code></pre>\n\n<p>Hope this helps</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 259129,
      "author_name": "abrial",
      "author_url": "",
      "post_date": "12/17/2017 18:51:43",
      "content": "<p>If you use Python, a convenient way for loading part of the data is by using skiprows as argument of read_csv, e.g:\npandas.read_csv(<em>file</em>,skiprows=range(a,b))</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "256145": "Is there a way i can partially read the train file without completly loading the Train dataset.  I use only 16gb ram. I tried loading the complete data set it threw me an error",
    "256152": "You can try using [SublimeText][1]\n\n\n  [1]: https://www.sublimetext.com/3",
    "256480": "There are several kernels which can be helpful for you. Such as https://www.kaggle.com/ceshine/lgbm-starter",
    "258914": "Hi,\nIf you're using pandas you can load the data in chunks and apply a filter for every chunk, then concatenate the filtered chunks to get a much smaller dataframe. In the following example I only keep august 2016 from the train data:\n\n    train_list = []\n    chunk_iter = pd.read_csv(r\"Data\\train.csv\", parse_dates=[\"date\"], chunksize=1000000)\n    for chunk in chunk_iter:\n        chunk_kept = chunk[(chunk[\"date\"].dt.year == 2016) &amp; (chunk[\"date\"].dt.month == 8)]\n        train_list.append(chunk_kept)\n    df_train = pd.concat(train_list)\n\nHope this helps",
    "259129": "If you use Python, a convenient way for loading part of the data is by using skiprows as argument of read_csv, e.g:\npandas.read_csv(*file*,skiprows=range(a,b))"
  },
  "source": "meta"
}