{
  "id": 5710,
  "title": "How to run benchmark in low memory machines?",
  "url": "/competitions/expedia-personalized-sort/discussion/5710",
  "author_name": "",
  "post_date": "2013-09-10T07:10:49.143Z",
  "votes": null,
  "comment_count": 6,
  "views": 2698,
  "content": "<p>Getting &quot;MemoryError&quot;, While running the python benchmark code on Core2Duo 2.93GHz, 4GB Win 7 machine. Any help is appreciated. Thanks.</p>",
  "messages": [
    {
      "id": "30634",
      "postDate": "09/10/2013 07:10:49",
      "content": "<p>Getting &quot;MemoryError&quot;, While running the python benchmark code on Core2Duo 2.93GHz, 4GB Win 7 machine. Any help is appreciated. Thanks.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30645",
      "postDate": "09/10/2013 12:46:09",
      "content": "<p>I had the same problem, so I split train.csv into 5 files of approx 2 000 000 rows each and test.csv into 4 files approx 1 700 000 rows in each (I'm using Linux, so I simply use the split command). </p>\n<p>Some issues:</p>\n<p>1) You have add the header to each of the new files (or manage it in some way).</p>\n<p>2) You should ensure that srch_id isn't split across files (i.e., each block of srch_id should be completely contained in exactly one of the files).</p>\n<p>Once you've split the files, modify the SETTINGS.json file so that it references the new files (you'll need to do this each time you run the benchmark on one of the new smaller files).</p>\n<p>Training the classifier with smaller training sets will no doubt give weaker results, but if you look at the current train.py code, they only use 100 000 lines of train data anyways (their benchmark code is just example code).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "31294",
      "postDate": "09/21/2013 07:05:52",
      "content": "<p>try mysql :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "31330",
      "postDate": "09/22/2013 05:26:31",
      "content": "<p>Thanks Bob. Your idea of splitting the training file in Linux really helped me. Didn't think about it in the first place.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "31405",
      "postDate": "09/23/2013 14:51:59",
      "content": "<p>I have the same issue. Actually I did the same as you did.</p>\n<p>My machine is 4GB win7 2.5GHz. Can you run with 500Mb data? It seems it's still a problem for me. Is it because of my setting of memory limit?</p>\n<p>And when I do analysis of the data columns, I need exhausted use 'cut' in linux to split the file into each column. Do you have any better way to handle this problem?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "31410",
      "postDate": "09/23/2013 15:55:23",
      "content": "<p>dseven4evr: You are welcome.</p>\n<p>Arandom: I can only load in approx 2 000 000 rows at a time (my machine specs are similar to yours). I don't split the files on hard-disk by columns (only by rows). Once a subset of rows are loaded into memory you can filter out whichever subset of columns you like (I'm working mainly in python). This way you can experiment with all columns to determine importance and correlations. If you determine some columns insignificant you can choose not to load them in memory using the 'usecols' param of the read_csv method (there's many params for this method, another of which that might be of interest is 'nrows').</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "31653",
      "postDate": "09/27/2013 11:10:39",
      "content": "<p>I have 8gb ram with 4 core. When I am loading the training dataset into ipython, my system hangs.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 30645,
      "author_name": "robreynoldson",
      "author_url": "",
      "post_date": "09/10/2013 12:46:09",
      "content": "<p>I had the same problem, so I split train.csv into 5 files of approx 2 000 000 rows each and test.csv into 4 files approx 1 700 000 rows in each (I'm using Linux, so I simply use the split command). </p>\n<p>Some issues:</p>\n<p>1) You have add the header to each of the new files (or manage it in some way).</p>\n<p>2) You should ensure that srch_id isn't split across files (i.e., each block of srch_id should be completely contained in exactly one of the files).</p>\n<p>Once you've split the files, modify the SETTINGS.json file so that it references the new files (you'll need to do this each time you run the benchmark on one of the new smaller files).</p>\n<p>Training the classifier with smaller training sets will no doubt give weaker results, but if you look at the current train.py code, they only use 100 000 lines of train data anyways (their benchmark code is just example code).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 31294,
      "author_name": "lavizhao",
      "author_url": "",
      "post_date": "09/21/2013 07:05:52",
      "content": "<p>try mysql :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 31330,
      "author_name": "dr4g0nsl4y3r",
      "author_url": "",
      "post_date": "09/22/2013 05:26:31",
      "content": "<p>Thanks Bob. Your idea of splitting the training file in Linux really helped me. Didn't think about it in the first place.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 31405,
      "author_name": "arandom",
      "author_url": "",
      "post_date": "09/23/2013 14:51:59",
      "content": "<p>I have the same issue. Actually I did the same as you did.</p>\n<p>My machine is 4GB win7 2.5GHz. Can you run with 500Mb data? It seems it's still a problem for me. Is it because of my setting of memory limit?</p>\n<p>And when I do analysis of the data columns, I need exhausted use 'cut' in linux to split the file into each column. Do you have any better way to handle this problem?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 31410,
      "author_name": "robreynoldson",
      "author_url": "",
      "post_date": "09/23/2013 15:55:23",
      "content": "<p>dseven4evr: You are welcome.</p>\n<p>Arandom: I can only load in approx 2 000 000 rows at a time (my machine specs are similar to yours). I don't split the files on hard-disk by columns (only by rows). Once a subset of rows are loaded into memory you can filter out whichever subset of columns you like (I'm working mainly in python). This way you can experiment with all columns to determine importance and correlations. If you determine some columns insignificant you can choose not to load them in memory using the 'usecols' param of the read_csv method (there's many params for this method, another of which that might be of interest is 'nrows').</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 31653,
      "author_name": "chitrasen",
      "author_url": "",
      "post_date": "09/27/2013 11:10:39",
      "content": "<p>I have 8gb ram with 4 core. When I am loading the training dataset into ipython, my system hangs.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "30634": "",
    "30645": "",
    "31294": "",
    "31330": "",
    "31405": "",
    "31410": "",
    "31653": ""
  },
  "source": "meta"
}