{
  "id": 11089,
  "title": "Shuffling the data in pure Python",
  "url": "/competitions/avazu-ctr-prediction/discussion/11089",
  "author_name": "",
  "post_date": "2014-11-30T00:20:37.387Z",
  "votes": 9,
  "comment_count": 2,
  "views": 1315,
  "content": "<p>I've written a simple pure Python script for shuffling large files with a memory buffer. In case you want to keep your codebase free of any C++ utils, this is the way to go : )</p>\n<p>It shuffles the train data in ~5 minutes on my late 2013 Macbook Pro, with a buffer of 10M lines (that's about 1.5GB).&nbsp;</p>\n<p>EDIT:&nbsp;Added&nbsp;a version with header support. Personally I use headerless files, but I guess some of you do use headers.</p>",
  "messages": [
    {
      "id": "59248",
      "postDate": "11/30/2014 00:20:37",
      "content": "<p>I've written a simple pure Python script for shuffling large files with a memory buffer. In case you want to keep your codebase free of any C++ utils, this is the way to go : )</p>\n<p>It shuffles the train data in ~5 minutes on my late 2013 Macbook Pro, with a buffer of 10M lines (that's about 1.5GB).&nbsp;</p>\n<p>EDIT:&nbsp;Added&nbsp;a version with header support. Personally I use headerless files, but I guess some of you do use headers.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59254",
      "postDate": "11/30/2014 03:31:25",
      "content": "<p>Are you getting noticeable improvements with shuffled files? I saw debate about that in another thread. On my end it's hurting the logloss in CV :-(</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59257",
      "postDate": "11/30/2014 05:50:31",
      "content": "<p>In first approach, it doesn't seem to hurt. Conceptually, it sounds a lot more rigorous to be using shuffled data, unless you are deliberately trying to exploit temporal continuity.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 59254,
      "author_name": "alexistaugeron",
      "author_url": "",
      "post_date": "11/30/2014 03:31:25",
      "content": "<p>Are you getting noticeable improvements with shuffled files? I saw debate about that in another thread. On my end it's hurting the logloss in CV :-(</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59257,
      "author_name": "fchollet",
      "author_url": "",
      "post_date": "11/30/2014 05:50:31",
      "content": "<p>In first approach, it doesn't seem to hurt. Conceptually, it sounds a lot more rigorous to be using shuffled data, unless you are deliberately trying to exploit temporal continuity.&nbsp;</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "59248": "",
    "59254": "",
    "59257": ""
  },
  "source": "meta"
}