{
  "id": 10872,
  "title": "Handling the training data at scale in Python",
  "url": "/competitions/avazu-ctr-prediction/discussion/10872",
  "author_name": "",
  "post_date": "2014-11-08T19:55:23.847Z",
  "votes": null,
  "comment_count": 3,
  "views": 1991,
  "content": "<p>Hey everyone,</p>\n<p>Kaggle newcomer here. The training dataset here is larger than what I'm used to dealing with, i.e. bigger than my machine limits. I'm used to using the Python/SKLearn toolchain, and I'm wondering if that is still a valid approach in this situation.</p>\n<p>Conceptually, here's what I think I have to do:<br>1) Split the training csv into two files using a random 70/30 split, ideally a different split for each training run<br>2) Use a generator to yield the lines in the resulting files for one-by-one encoding<br>3) Use another generator to feed the output of the previous generator into sklearn for training / xvalidation.<br>4) Use another dual generator setup on the test data set to write out predictions</p>\n<p>Is there a Pythonic way to handle this problem at scale? Does this put me outside the realm of sklearn and into hand rolled model implementations, e.g. <a href=\"https://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10745/beating-the-benchmark\">Abhishek's benchmark beater</a>? Would love any advice that I could get for this problem.</p>\n<p>Thanks in advance!</p>",
  "messages": [
    {
      "id": "57530",
      "postDate": "11/08/2014 19:55:23",
      "content": "<p>Hey everyone,</p>\n<p>Kaggle newcomer here. The training dataset here is larger than what I'm used to dealing with, i.e. bigger than my machine limits. I'm used to using the Python/SKLearn toolchain, and I'm wondering if that is still a valid approach in this situation.</p>\n<p>Conceptually, here's what I think I have to do:<br>1) Split the training csv into two files using a random 70/30 split, ideally a different split for each training run<br>2) Use a generator to yield the lines in the resulting files for one-by-one encoding<br>3) Use another generator to feed the output of the previous generator into sklearn for training / xvalidation.<br>4) Use another dual generator setup on the test data set to write out predictions</p>\n<p>Is there a Pythonic way to handle this problem at scale? Does this put me outside the realm of sklearn and into hand rolled model implementations, e.g. <a href=\"https://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10745/beating-the-benchmark\">Abhishek's benchmark beater</a>? Would love any advice that I could get for this problem.</p>\n<p>Thanks in advance!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57536",
      "postDate": "11/08/2014 21:16:30",
      "content": "<p>It's possible to get a reasonable score (around 0.40) with a sample-based approach. I've been using R rather than Python but a training sample of 2 million cases worked ok. Then again, I'm not expecting to compete at the top of the leaderboard - not sure if that can be done with a small sample.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57539",
      "postDate": "11/09/2014 04:00:05",
      "content": "<p>You can still use sklearn classifiers as long as they implement the partial_fit method, for instance SGDClassifier.</p>\n<p>You can then use pandas' read_csv with iterator=True to read the samples and pass them into your partial_fit function in chunks</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57561",
      "postDate": "11/09/2014 17:48:18",
      "content": "<p>Cross posted this on <a href=\"http://www.reddit.com/r/MachineLearning/comments/2lp6mm/splitting_8gb_dataset_into_traintest_split_in/\">Reddit</a>&nbsp;and someone suggested using HDF5 and linked to a <a href=\"http://www.h5py.org/\">Python interface</a>&nbsp;for it. This appears to&nbsp;let you&nbsp;treat massive binary stores as Numpy arrays, which is pretty cool.</p>\n<p>I will also check out the SGDClassifier's partial_fit as Leo suggested.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 57536,
      "author_name": "cartographic",
      "author_url": "",
      "post_date": "11/08/2014 21:16:30",
      "content": "<p>It's possible to get a reasonable score (around 0.40) with a sample-based approach. I've been using R rather than Python but a training sample of 2 million cases worked ok. Then again, I'm not expecting to compete at the top of the leaderboard - not sure if that can be done with a small sample.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57539,
      "author_name": "leovannierop",
      "author_url": "",
      "post_date": "11/09/2014 04:00:05",
      "content": "<p>You can still use sklearn classifiers as long as they implement the partial_fit method, for instance SGDClassifier.</p>\n<p>You can then use pandas' read_csv with iterator=True to read the samples and pass them into your partial_fit function in chunks</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57561,
      "author_name": "acpigeon",
      "author_url": "",
      "post_date": "11/09/2014 17:48:18",
      "content": "<p>Cross posted this on <a href=\"http://www.reddit.com/r/MachineLearning/comments/2lp6mm/splitting_8gb_dataset_into_traintest_split_in/\">Reddit</a>&nbsp;and someone suggested using HDF5 and linked to a <a href=\"http://www.h5py.org/\">Python interface</a>&nbsp;for it. This appears to&nbsp;let you&nbsp;treat massive binary stores as Numpy arrays, which is pretty cool.</p>\n<p>I will also check out the SGDClassifier's partial_fit as Leo suggested.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "57530": "",
    "57536": "",
    "57539": "",
    "57561": ""
  },
  "source": "meta"
}