{
  "id": 21719,
  "title": "Python script for sampling train.csv",
  "url": "/competitions/grupo-bimbo-inventory-demand/discussion/21719",
  "author_name": "",
  "post_date": "2016-06-16T08:29:05.060Z",
  "votes": 3,
  "comment_count": 2,
  "views": 788,
  "content": "<p>Here's a script I used to sample the large train.csv file. It is slow, but it works. Any better suggestions on how to handle such large files in Python are welcome.</p>\n\n<pre><code>print('Importing libraries...')\nimport random\nimport pandas as pd\n\noriginal_file = 'train.csv'\ntarget_file = 'train_sample.csv'\ntarget_lines = 500000\n\nprint('Counting original file rows...')\nf = open(original_file,'r')\nn_lines = sum(1 for l in f)\nf.close()\nprint('Total number of rows: ', str(n_lines))\n\n\nprint('Creating skip indexes...')\nskip = random.sample(range(1, n_lines), n_lines - target_lines)\n\nprint('Creating dataframe...')\ndata = pd.read_csv(original_file, header=0, skiprows=skip)\nskip = None\n\nprint('Saving sampled CSV...')\ndata.to_csv(target_file, index=False)\nprint('CSV saved &gt; ', target_file)\n</code></pre>",
  "messages": [
    {
      "id": "124207",
      "postDate": "06/16/2016 08:29:05",
      "content": "<p>Here's a script I used to sample the large train.csv file. It is slow, but it works. Any better suggestions on how to handle such large files in Python are welcome.</p>\n\n<pre><code>print('Importing libraries...')\nimport random\nimport pandas as pd\n\noriginal_file = 'train.csv'\ntarget_file = 'train_sample.csv'\ntarget_lines = 500000\n\nprint('Counting original file rows...')\nf = open(original_file,'r')\nn_lines = sum(1 for l in f)\nf.close()\nprint('Total number of rows: ', str(n_lines))\n\n\nprint('Creating skip indexes...')\nskip = random.sample(range(1, n_lines), n_lines - target_lines)\n\nprint('Creating dataframe...')\ndata = pd.read_csv(original_file, header=0, skiprows=skip)\nskip = None\n\nprint('Saving sampled CSV...')\ndata.to_csv(target_file, index=False)\nprint('CSV saved &gt; ', target_file)\n</code></pre>",
      "rawMarkdown": "Here's a script I used to sample the large train.csv file. It is slow, but it works. Any better suggestions on how to handle such large files in Python are welcome.\r\n\r\n    print('Importing libraries...')\r\n    import random\r\n    import pandas as pd\r\n    \r\n    original_file = 'train.csv'\r\n    target_file = 'train_sample.csv'\r\n    target_lines = 500000\r\n    \r\n    print('Counting original file rows...')\r\n    f = open(original_file,'r')\r\n    n_lines = sum(1 for l in f)\r\n    f.close()\r\n    print('Total number of rows: ', str(n_lines))\r\n    \r\n    \r\n    print('Creating skip indexes...')\r\n    skip = random.sample(range(1, n_lines), n_lines - target_lines)\r\n    \r\n    print('Creating dataframe...')\r\n    data = pd.read_csv(original_file, header=0, skiprows=skip)\r\n    skip = None\r\n    \r\n    print('Saving sampled CSV...')\r\n    data.to_csv(target_file, index=False)\r\n    print('CSV saved > ', target_file)",
      "votes": null
    },
    {
      "id": "124243",
      "postDate": "06/16/2016 12:20:31",
      "content": "<p>For everything like this I would suggest the excellent phraug scripts\n<a href=\"https://github.com/zygmuntz/phraug\">https://github.com/zygmuntz/phraug</a></p>",
      "rawMarkdown": "For everything like this I would suggest the excellent phraug scripts\r\n[https://github.com/zygmuntz/phraug][1]\r\n\r\n\r\n  [1]: https://github.com/zygmuntz/phraug",
      "votes": null
    },
    {
      "id": "124284",
      "postDate": "06/16/2016 19:26:13",
      "content": "<p>Thank you for this reference.</p>",
      "rawMarkdown": "Thank you for this reference.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 124243,
      "author_name": "sterby",
      "author_url": "",
      "post_date": "06/16/2016 12:20:31",
      "content": "<p>For everything like this I would suggest the excellent phraug scripts\n<a href=\"https://github.com/zygmuntz/phraug\">https://github.com/zygmuntz/phraug</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124284,
      "author_name": "gcmartinelli",
      "author_url": "",
      "post_date": "06/16/2016 19:26:13",
      "content": "<p>Thank you for this reference.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "124207": "Here's a script I used to sample the large train.csv file. It is slow, but it works. Any better suggestions on how to handle such large files in Python are welcome.\r\n\r\n    print('Importing libraries...')\r\n    import random\r\n    import pandas as pd\r\n    \r\n    original_file = 'train.csv'\r\n    target_file = 'train_sample.csv'\r\n    target_lines = 500000\r\n    \r\n    print('Counting original file rows...')\r\n    f = open(original_file,'r')\r\n    n_lines = sum(1 for l in f)\r\n    f.close()\r\n    print('Total number of rows: ', str(n_lines))\r\n    \r\n    \r\n    print('Creating skip indexes...')\r\n    skip = random.sample(range(1, n_lines), n_lines - target_lines)\r\n    \r\n    print('Creating dataframe...')\r\n    data = pd.read_csv(original_file, header=0, skiprows=skip)\r\n    skip = None\r\n    \r\n    print('Saving sampled CSV...')\r\n    data.to_csv(target_file, index=False)\r\n    print('CSV saved > ', target_file)",
    "124243": "For everything like this I would suggest the excellent phraug scripts\r\n[https://github.com/zygmuntz/phraug][1]\r\n\r\n\r\n  [1]: https://github.com/zygmuntz/phraug",
    "124284": "Thank you for this reference."
  },
  "source": "meta"
}