{
  "id": 5559,
  "title": "Where are you running your algorithm?",
  "url": "/competitions/accelerometer-biometric-competition/discussion/5559",
  "author_name": "",
  "post_date": "2013-08-27T09:55:01.417Z",
  "votes": null,
  "comment_count": 7,
  "views": 2793,
  "content": "<p>I wrote alghoritm in Matlab, run it, and it took 97% of my 4 GB RAM memory and 10 GB on HDD. Nothing else is possible to doing on my machine in run time of procces. I was not suprised about that: data is huge. And God knows how long it will take to end..</p>\n<p>Where are you running your alghorims? Any variants?&nbsp;</p>",
  "messages": [
    {
      "id": "29552",
      "postDate": "08/27/2013 09:55:01",
      "content": "<p>I wrote alghoritm in Matlab, run it, and it took 97% of my 4 GB RAM memory and 10 GB on HDD. Nothing else is possible to doing on my machine in run time of procces. I was not suprised about that: data is huge. And God knows how long it will take to end..</p>\n<p>Where are you running your alghorims? Any variants?&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29610",
      "postDate": "08/28/2013 01:20:19",
      "content": "<p>i7-2600k and 16gb running python on linux. &nbsp;</p>\n<p>it takes about 40 minutes to process the train.csv</p>\n<p>and almost two hours to process the test.csv.</p>\n<p>I am still working on generating the submission file.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29616",
      "postDate": "08/28/2013 03:54:54",
      "content": "<p>when you say processing, did you mean reading or making any kind of transformations? I'm asking as it took about a minute to read the train.csv using pandas (read_csv) using python in win7 on mine. Which has much worse specs than what you've got!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29625",
      "postDate": "08/28/2013 05:05:22",
      "content": "<p>yes, reading it in and making transformations.&nbsp;</p>\n<p>Just reading it in takes about a minute.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29641",
      "postDate": "08/28/2013 13:33:26",
      "content": "<p><a title=\"View Il'ya Zhenin's profile\" href=\"http://www.kaggle.com/users/113094/il-ya-zhenin\">Il'ya Zhenin</a>,&nbsp; you can easily preprocess your data using sqlite/mysql/postgres. I use R + postgres(only for storing train and test sets) on my laptop with 8gb RAM.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30021",
      "postDate": "09/02/2013 22:38:12",
      "content": "<p>I'm using SQL, SSIS for data prep.</p>\n<p>Running algorithm in R on my labtop with 6GB RAM with quad core on window.</p>\n<p>I tried to optimize R by implementing file base dataset (not loading whole data to RAM) and multiple core (4 cores) usage. And it's still crunching......</p>\n<p>Also, running stock algorithms in Microsoft data mining.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30439",
      "postDate": "09/07/2013 18:28:53",
      "content": "<p>I run Java code, and I'm setting the max heap size to 2.5G, which is quite a bit more than what I usually need for this type of thing.</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30443",
      "postDate": "09/07/2013 18:59:29",
      "content": "<p>Hi everybody, an update..&nbsp;</p>\n<p>My python/pandas feature generation was taking up to 4-hours on test.csv so I went looking for a better solution.</p>\n<p>I tried multi-processing with 7 threads but found that it was thrashing due to passing data back and forth between the threads.&nbsp;&nbsp; (I never tried using shared memory, this might have been a good solution)</p>\n<p>I tried loading train.csv into sqlite3 and after running all night I hit break and had only loaded 188,000 rows.&nbsp; </p>\n<p>So, I converted all my code to PYPY (by removing all use of pandas, numpy and sklearn).&nbsp; Just straight python.&nbsp; It runs REALLY fast.&nbsp; However, the base CSV reader is crummy compared to the Pandas csv reader. &nbsp;</p>\n<p>So my current solution includes various flavors of python.&nbsp; Some really fast PYPY and some slower python/pandas/sklearn code.&nbsp;</p>\n<p>As an aside,&nbsp; one of the steps that I offloaded was sorting.&nbsp; Although the data appeared sorted, I wanted to be absolutely sure it was sorted by 'Device', 'T' for my feature generation.&nbsp; So I sorted the files with ipython notebook and pandas.&nbsp; Then saved the sorted file out train_sort.csv and test_sort.csv.&nbsp; These sorted files are the inputs to the feature generation step.</p>\n<p>Summary:&nbsp;</p>\n<p>Sorting - ipython notebook, pandas.&nbsp; About 10 minutes for both files.</p>\n<p>Feature generation: PYPY - about 2-4 minutes for each file</p>\n<p>Machine Learning - Python/Pandas/Sklearn</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 29610,
      "author_name": "geringer",
      "author_url": "",
      "post_date": "08/28/2013 01:20:19",
      "content": "<p>i7-2600k and 16gb running python on linux. &nbsp;</p>\n<p>it takes about 40 minutes to process the train.csv</p>\n<p>and almost two hours to process the test.csv.</p>\n<p>I am still working on generating the submission file.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29616,
      "author_name": "alzmcr",
      "author_url": "",
      "post_date": "08/28/2013 03:54:54",
      "content": "<p>when you say processing, did you mean reading or making any kind of transformations? I'm asking as it took about a minute to read the train.csv using pandas (read_csv) using python in win7 on mine. Which has much worse specs than what you've got!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29625,
      "author_name": "geringer",
      "author_url": "",
      "post_date": "08/28/2013 05:05:22",
      "content": "<p>yes, reading it in and making transformations.&nbsp;</p>\n<p>Just reading it in takes about a minute.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29641,
      "author_name": "dselivanov",
      "author_url": "",
      "post_date": "08/28/2013 13:33:26",
      "content": "<p><a title=\"View Il'ya Zhenin's profile\" href=\"http://www.kaggle.com/users/113094/il-ya-zhenin\">Il'ya Zhenin</a>,&nbsp; you can easily preprocess your data using sqlite/mysql/postgres. I use R + postgres(only for storing train and test sets) on my laptop with 8gb RAM.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 30021,
      "author_name": "tedkkim",
      "author_url": "",
      "post_date": "09/02/2013 22:38:12",
      "content": "<p>I'm using SQL, SSIS for data prep.</p>\n<p>Running algorithm in R on my labtop with 6GB RAM with quad core on window.</p>\n<p>I tried to optimize R by implementing file base dataset (not loading whole data to RAM) and multiple core (4 cores) usage. And it's still crunching......</p>\n<p>Also, running stock algorithms in Microsoft data mining.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 30439,
      "author_name": "solorzano",
      "author_url": "",
      "post_date": "09/07/2013 18:28:53",
      "content": "<p>I run Java code, and I'm setting the max heap size to 2.5G, which is quite a bit more than what I usually need for this type of thing.</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 30443,
      "author_name": "geringer",
      "author_url": "",
      "post_date": "09/07/2013 18:59:29",
      "content": "<p>Hi everybody, an update..&nbsp;</p>\n<p>My python/pandas feature generation was taking up to 4-hours on test.csv so I went looking for a better solution.</p>\n<p>I tried multi-processing with 7 threads but found that it was thrashing due to passing data back and forth between the threads.&nbsp;&nbsp; (I never tried using shared memory, this might have been a good solution)</p>\n<p>I tried loading train.csv into sqlite3 and after running all night I hit break and had only loaded 188,000 rows.&nbsp; </p>\n<p>So, I converted all my code to PYPY (by removing all use of pandas, numpy and sklearn).&nbsp; Just straight python.&nbsp; It runs REALLY fast.&nbsp; However, the base CSV reader is crummy compared to the Pandas csv reader. &nbsp;</p>\n<p>So my current solution includes various flavors of python.&nbsp; Some really fast PYPY and some slower python/pandas/sklearn code.&nbsp;</p>\n<p>As an aside,&nbsp; one of the steps that I offloaded was sorting.&nbsp; Although the data appeared sorted, I wanted to be absolutely sure it was sorted by 'Device', 'T' for my feature generation.&nbsp; So I sorted the files with ipython notebook and pandas.&nbsp; Then saved the sorted file out train_sort.csv and test_sort.csv.&nbsp; These sorted files are the inputs to the feature generation step.</p>\n<p>Summary:&nbsp;</p>\n<p>Sorting - ipython notebook, pandas.&nbsp; About 10 minutes for both files.</p>\n<p>Feature generation: PYPY - about 2-4 minutes for each file</p>\n<p>Machine Learning - Python/Pandas/Sklearn</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "29552": "",
    "29610": "",
    "29616": "",
    "29625": "",
    "29641": "",
    "30021": "",
    "30439": "",
    "30443": ""
  },
  "source": "meta"
}