{
  "id": 11246,
  "title": "Beat the benchmark with scikit-learn (low memory, ~.403)",
  "url": "/competitions/avazu-ctr-prediction/discussion/11246",
  "author_name": "",
  "post_date": "2014-12-15T17:55:56.350Z",
  "votes": 24,
  "comment_count": 27,
  "views": 10800,
  "content": "<p>Hi all,</p>\n<p>Here is my github repo for a beat-the-benchmark model that is has a lot of room for adaptation (especially regarding feature engineering). It should run on anyone's beat-up 15-inch macbook pro ;)&nbsp;</p>\n<p>https://github.com/mkneierV/kaggle_avazu_benchmark</p>\n<p>The README outlines simple steps to use the model. The model works as follows:</p>\n<p>1) generator to read in the dataset</p>\n<p>2) subsample the negatives</p>\n<p>3) hash the features</p>\n<p>4) acumulate train samples into batches</p>\n<p>5) fit a SGD logistic regression</p>\n<p>6) correct the intercept to account for the subsampling</p>\n\n<p>Tweaking the negative subampling rate, dataset size, and n_iter can all drive more performance. Also, go wild with the SGD model's parameters.</p>",
  "messages": [
    {
      "id": "60245",
      "postDate": "12/15/2014 17:55:56",
      "content": "<p>Hi all,</p>\n<p>Here is my github repo for a beat-the-benchmark model that is has a lot of room for adaptation (especially regarding feature engineering). It should run on anyone's beat-up 15-inch macbook pro ;)&nbsp;</p>\n<p>https://github.com/mkneierV/kaggle_avazu_benchmark</p>\n<p>The README outlines simple steps to use the model. The model works as follows:</p>\n<p>1) generator to read in the dataset</p>\n<p>2) subsample the negatives</p>\n<p>3) hash the features</p>\n<p>4) acumulate train samples into batches</p>\n<p>5) fit a SGD logistic regression</p>\n<p>6) correct the intercept to account for the subsampling</p>\n\n<p>Tweaking the negative subampling rate, dataset size, and n_iter can all drive more performance. Also, go wild with the SGD model's parameters.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60276",
      "postDate": "12/15/2014 20:50:30",
      "content": "<p>Hi Mikek,</p>\n<p>How do you install the lib.ml or lib.preprocessing modules? I have an Anaconda installation of python 3.4.1. I tried <strong>pip install lib.lm</strong> and here's the error I'm getting <strong>ImportError: No module named 'lib'</strong></p>\n<p>Thanks!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60284",
      "postDate": "12/15/2014 21:33:54",
      "content": "<p>Hi Max,</p>\n<p>ml.py and preprocessing.py are local scripts that reside in the lib module of the repository. If you fork the repository and run the model from that directory, everything should work fine :).&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60286",
      "postDate": "12/15/2014 21:34:38",
      "content": "<p>To be precise, this should score&nbsp;0.4021441 on the leaderboard.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60313",
      "postDate": "12/16/2014 02:22:30",
      "content": "<p>What is the total time required to run the script? What is the memory requirement? Thanks</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60374",
      "postDate": "12/16/2014 17:24:54",
      "content": "<p>Thanks Mikek. I was able to move pass that hurdle, but now I'm getting&nbsp;<strong>ValueError: blocks must be 2-D&nbsp;</strong>from the ml.py file at line 51, in the function partial_fit. Any idea? Thanks again!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60376",
      "postDate": "12/16/2014 17:38:46",
      "content": "<p>Hey Max,</p>\n<p>This would be an error in how the records are being processed. You should see the train_errors.log being populated with errors.</p>\n<p>Have you made sure that you:</p>\n<p>1) Have the most recent data</p>\n<p>2) Have the package dependencies specified in requirements.txt</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60378",
      "postDate": "12/16/2014 17:43:13",
      "content": "<p>I believe I have all dependencies installed, but I will double check. My train_errors log is indeed being populated. It doesn't matter if you add the '.csv' to the files, does it? I usually use R, I'm just trying to see what I can get out of python with this competition.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60387",
      "postDate": "12/16/2014 19:46:36",
      "content": "<p>Hey Max,</p>\n<p>I just commited some changes to the logging setup, which should help us identify the problem. Most likely, the problem you are facing has something to do with the the train dataset's names or format.&nbsp;</p>\n<p>Please try the following:</p>\n<p>1) Sync your code with the master branch on github</p>\n<p>2) Remove train_errors.log</p>\n<p>3) generate sub_train10 as specifed in the README</p>\n<p>4) Run the model, as specified in the README</p>\n<p>If it fails again, take a look at new train_errors.txt which will be generated. This should tell you exactly what is going wrong, and we can fix it from there.</p>\n<p>Let me know what you find out.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60393",
      "postDate": "12/16/2014 20:37:35",
      "content": "<p>Hi Mikek,</p>\n<p>Thanks again for your patience and willingness to help out. Please find attached my train_errors.log file. I'm going through it myself right now.</p>\n<p>Best,</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60395",
      "postDate": "12/16/2014 20:51:11",
      "content": "<p>I think I got it working now. Will let you know if I get the same results you posted or not. Still running as I speak. Any idea on how to speed up or palatalized this? Again, thanks a lot, Mikek.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "61767",
      "postDate": "12/23/2014 17:37:54",
      "content": "<p>Hi MikeK,</p>\n\n<p>This is really cool, I was working on something similar using scikit's built in partial_fit class methods, but this looks much more dynamic. Nice work!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63227",
      "postDate": "01/27/2015 07:15:33",
      "content": "<p>Hey,&nbsp;MikeK</p>\n<p>Thanks for posting this - very interesting! Can you explain why negative subsampling is needed here? Any good docs on this trick?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63232",
      "postDate": "01/27/2015 12:02:49",
      "content": "<p>[quote=Artem Yankov;63227]</p>\n<p>Hey,&nbsp;MikeK</p>\n<p>Thanks for posting this - very interesting! Can you explain why negative subsampling is needed here? Any good docs on this trick?</p>\n<p>[/quote]</p>\n<p>I'm curious about this too. Since both the train and test had the same subsampling, why is a correction needed?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63244",
      "postDate": "01/27/2015 18:43:34",
      "content": "<p>[quote=inversion;63232]</p>\n<p>[quote=Artem Yankov;63227]</p>\n<p>Hey,&nbsp;MikeK</p>\n<p>Thanks for posting this - very interesting! Can you explain why negative subsampling is needed here? Any good docs on this trick?</p>\n<p>[/quote]</p>\n<p>I'm curious about this too. Since both the train and test had the same subsampling, why is a correction needed?</p>\n<p>[/quote]</p>\n\n<p>Hi there.</p>\n<p>Negative subsampling isn't needed, but it is speeds up training and I found no significant negative impact on performance. To be clear, this algorithm performs subsampling in addition to the sampling the competition admin performed.&nbsp;</p>\n<p>Here is the paper that inspired me:</p>\n<p>http://tist.acm.org/papers/ACM-TIST-V5N4-TIST-2012-12-0197.R1.pdf</p>\n<p>Here is an interesting blog post on the topic:</p>\n<p>http://www.machinedlearnings.com/2010/11/on-unimportance-of-zeroes.html</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63245",
      "postDate": "01/27/2015 18:47:02",
      "content": "<p>[quote=MikeK;63244]</p>\n<p>Negative subsampling isn't needed, but it is speeds up training and I found no significant negative impact on performance. To be clear, this algorithm performs subsampling in addition to the sampling the competition admin performed.&nbsp;</p>\n<p>[/quote]</p>\n<p>Ah, that makes sense. Thanks for clarifying.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63247",
      "postDate": "01/27/2015 19:35:11",
      "content": "<p>[quote=MikeK;63244]</p>\n<p>Negative subsampling isn't needed, but it is speeds up training and I found no significant negative impact on performance. To be clear, this algorithm performs subsampling in addition to the sampling the competition admin performed.&nbsp;</p>\n<p>Here is the paper that inspired me:</p>\n<p>http://tist.acm.org/papers/ACM-TIST-V5N4-TIST-2012-12-0197.R1.pdf</p>\n<p>Here is an interesting blog post on the topic:</p>\n<p>http://www.machinedlearnings.com/2010/11/on-unimportance-of-zeroes.html</p>\n<p>[/quote]</p>\n<p>Gotcha, thank you! And about feature hashing, how did you choose the number of the features to be 2 ** 20?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63248",
      "postDate": "01/27/2015 20:15:12",
      "content": "<p>[quote=Artem Yankov;63247]</p>\n<p>Gotcha, thank you! And about feature hashing, how did you choose the number of the features to be 2 ** 20?</p>\n<p>[/quote]</p>\n<p>2**20 is a good starting point.</p>\n<p>Go with as large as you can go with your system memory, and then scale down to see if you get improved scores.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63259",
      "postDate": "01/28/2015 06:32:47",
      "content": "<p>[quote=inversion;63248]</p>\n<p>[quote=Artem Yankov;63247]</p>\n<p>Gotcha, thank you! And about feature hashing, how did you choose the number of the features to be 2 ** 20?</p>\n<p>[/quote]</p>\n<p>2**20 is a good starting point.</p>\n<p>Go with as large as you can go with your system memory, and then scale down to see if you get improved scores.</p>\n<p>[/quote]</p>\n\n<p>Thanks, inversion. Is there any particular reason to use hashing trick vs. one hot encoding here? I tried hashing trick on a subset and got worse results, but I keep playing with it.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63276",
      "postDate": "01/28/2015 13:40:51",
      "content": "<p>[quote=Artem Yankov;63259]</p>\n<p>Thanks, inversion. Is there any particular reason to use hashing trick vs. one hot encoding here? I tried hashing trick on a subset and got worse results, but I keep playing with it.&nbsp;</p>\n<p>[/quote]</p>\n<p>With hashing, you can read and process the file one line at a time (i.e., true online leaning). With &nbsp;one hot encoding, you have to have visibility to the entire dataset.</p>\n<p>I personally am using one hot encoding.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63321",
      "postDate": "01/29/2015 18:53:32",
      "content": "<p>[quote=inversion;63276]</p>\n<p>[quote=Artem Yankov;63259]</p>\n<p>Thanks, inversion. Is there any particular reason to use hashing trick vs. one hot encoding here? I tried hashing trick on a subset and got worse results, but I keep playing with it.&nbsp;</p>\n<p>[/quote]</p>\n<p>With hashing, you can read and process the file one line at a time (i.e., true online leaning). With &nbsp;one hot encoding, you have to have visibility to the entire dataset.</p>\n<p>I personally am using one hot encoding.</p>\n<p>[/quote]</p>\n\n<p>Gotcha, thanks! I created short and simplified versions without subsampling or anything fancy. When using onehotencoder I get 0.414. When using hashing trick all predictions turn out to be 0. You guys can tell if I'm doing anything stupid here?</p>\n<p>Version with the hashing trick&nbsp;</p>\n<p>https://gist.github.com/yankov/1ab961c108613cf2d63a</p>\n<p>Version with onehotencoder (works)</p>\n<p>https://gist.github.com/yankov/c2ec3864c963da3b59f6</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63495",
      "postDate": "02/03/2015 19:27:26",
      "content": "<p>Hi Artem:</p>\n<p>&nbsp; I cannot find OneHotEncoderPartial in your folders.</p>\n<p>Could you upload OneHotEncoderPartial file?</p>\n<p>Thanks</p>\n<p>[quote=Artem Yankov;63321]</p>\n<p>[quote=inversion;63276]</p>\n<p>[quote=Artem Yankov;63259]</p>\n<p>Thanks, inversion. Is there any particular reason to use hashing trick vs. one hot encoding here? I tried hashing trick on a subset and got worse results, but I keep playing with it.&nbsp;</p>\n<p>[/quote]</p>\n<p>With hashing, you can read and process the file one line at a time (i.e., true online leaning). With &nbsp;one hot encoding, you have to have visibility to the entire dataset.</p>\n<p>I personally am using one hot encoding.</p>\n<p>[/quote]</p>\n<p>Gotcha, thanks! I created short and simplified versions without subsampling or anything fancy. When using onehotencoder I get 0.414. When using hashing trick all predictions turn out to be 0. You guys can tell if I'm doing anything stupid here?</p>\n<p>Version with the hashing trick&nbsp;</p>\n<p>https://gist.github.com/yankov/1ab961c108613cf2d63a</p>\n<p>Version with onehotencoder (works)</p>\n<p>https://gist.github.com/yankov/c2ec3864c963da3b59f6</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63496",
      "postDate": "02/03/2015 19:39:40",
      "content": "<p>get 200gigs RAM, use SGD from sklearn n u will get below 0.40 easily ;)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63498",
      "postDate": "02/03/2015 19:48:05",
      "content": "<p>[quote=Zhuhongtu;63495]</p>\n<p>Hi Artem:</p>\n<p>&nbsp; I cannot find OneHotEncoderPartial in your folders.</p>\n<p>Could you upload OneHotEncoderPartial file?</p>\n<p>Thanks</p>\n<p>[quote=Artem Yankov;63321]</p>\n<p>[quote=inversion;63276]</p>\n<p>[quote=Artem Yankov;63259]</p>\n<p>Thanks, inversion. Is there any particular reason to use hashing trick vs. one hot encoding here? I tried hashing trick on a subset and got worse results, but I keep playing with it.&nbsp;</p>\n<p>[/quote]</p>\n<p>With hashing, you can read and process the file one line at a time (i.e., true online leaning). With &nbsp;one hot encoding, you have to have visibility to the entire dataset.</p>\n<p>I personally am using one hot encoding.</p>\n<p>[/quote]</p>\n<p>Gotcha, thanks! I created short and simplified versions without subsampling or anything fancy. When using onehotencoder I get 0.414. When using hashing trick all predictions turn out to be 0. You guys can tell if I'm doing anything stupid here?</p>\n<p>Version with the hashing trick&nbsp;</p>\n<p>https://gist.github.com/yankov/1ab961c108613cf2d63a</p>\n<p>Version with onehotencoder (works)</p>\n<p>https://gist.github.com/yankov/c2ec3864c963da3b59f6</p>\n<p>[/quote]</p>\n<p>[/quote]</p>\n<p>Here is it</p>\n<p>https://gist.github.com/yankov/a86d44bf0e6009944c21</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63499",
      "postDate": "02/03/2015 19:49:43",
      "content": "<p>[quote=Abhishek;63496]</p>\n<p>get 200gigs RAM, use SGD from sklearn n u will get below 0.40 easily ;)</p>\n<p>[/quote]</p>\n<p>Haha, yeah, I did that and spent ~$50 already on AWS. Still got to find the right parameters :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63534",
      "postDate": "02/04/2015 01:33:38",
      "content": "<p>[quote=Abhishek;63496]</p>\n<p>get 200gigs RAM, use SGD from sklearn n u will get below 0.40 easily ;)</p>\n<p>[/quote]</p>\n<p>Meh.</p>\n<p>See slide 33:</p>\n<p><strong>Hardware: 4x Intel CPU, 64 cores, 1TB RAM, $35K </strong></p>\n<p>http://stanford.edu/~rezab/nips2014workshop/slides/jure.pdf</p>\n<p>Slide 41 also has some good advice:</p>\n<p><em>Bottom Line: Get your own 1TB RAM server</em></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63738",
      "postDate": "02/07/2015 02:34:10",
      "content": "<p>Fixed that feature hashing btw, had to use input_type = &quot;string&quot;. Final short version</p>\n<p>https://gist.github.com/yankov/1ab961c108613cf2d63a</p>\n<p>Doesn't give a good score (.41). But maybe someone new to scikit-learn will find it useful.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "312577",
      "postDate": "04/12/2018 02:00:52",
      "content": "<p>this model running almost four hours</p>",
      "rawMarkdown": "this model running almost four hours",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 60276,
      "author_name": "maxsop",
      "author_url": "",
      "post_date": "12/15/2014 20:50:30",
      "content": "<p>Hi Mikek,</p>\n<p>How do you install the lib.ml or lib.preprocessing modules? I have an Anaconda installation of python 3.4.1. I tried <strong>pip install lib.lm</strong> and here's the error I'm getting <strong>ImportError: No module named 'lib'</strong></p>\n<p>Thanks!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60284,
      "author_name": "michaelkneier",
      "author_url": "",
      "post_date": "12/15/2014 21:33:54",
      "content": "<p>Hi Max,</p>\n<p>ml.py and preprocessing.py are local scripts that reside in the lib module of the repository. If you fork the repository and run the model from that directory, everything should work fine :).&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60286,
      "author_name": "michaelkneier",
      "author_url": "",
      "post_date": "12/15/2014 21:34:38",
      "content": "<p>To be precise, this should score&nbsp;0.4021441 on the leaderboard.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60313,
      "author_name": "jialunhe",
      "author_url": "",
      "post_date": "12/16/2014 02:22:30",
      "content": "<p>What is the total time required to run the script? What is the memory requirement? Thanks</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60374,
      "author_name": "maxsop",
      "author_url": "",
      "post_date": "12/16/2014 17:24:54",
      "content": "<p>Thanks Mikek. I was able to move pass that hurdle, but now I'm getting&nbsp;<strong>ValueError: blocks must be 2-D&nbsp;</strong>from the ml.py file at line 51, in the function partial_fit. Any idea? Thanks again!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60376,
      "author_name": "michaelkneier",
      "author_url": "",
      "post_date": "12/16/2014 17:38:46",
      "content": "<p>Hey Max,</p>\n<p>This would be an error in how the records are being processed. You should see the train_errors.log being populated with errors.</p>\n<p>Have you made sure that you:</p>\n<p>1) Have the most recent data</p>\n<p>2) Have the package dependencies specified in requirements.txt</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60378,
      "author_name": "maxsop",
      "author_url": "",
      "post_date": "12/16/2014 17:43:13",
      "content": "<p>I believe I have all dependencies installed, but I will double check. My train_errors log is indeed being populated. It doesn't matter if you add the '.csv' to the files, does it? I usually use R, I'm just trying to see what I can get out of python with this competition.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60387,
      "author_name": "michaelkneier",
      "author_url": "",
      "post_date": "12/16/2014 19:46:36",
      "content": "<p>Hey Max,</p>\n<p>I just commited some changes to the logging setup, which should help us identify the problem. Most likely, the problem you are facing has something to do with the the train dataset's names or format.&nbsp;</p>\n<p>Please try the following:</p>\n<p>1) Sync your code with the master branch on github</p>\n<p>2) Remove train_errors.log</p>\n<p>3) generate sub_train10 as specifed in the README</p>\n<p>4) Run the model, as specified in the README</p>\n<p>If it fails again, take a look at new train_errors.txt which will be generated. This should tell you exactly what is going wrong, and we can fix it from there.</p>\n<p>Let me know what you find out.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60393,
      "author_name": "maxsop",
      "author_url": "",
      "post_date": "12/16/2014 20:37:35",
      "content": "<p>Hi Mikek,</p>\n<p>Thanks again for your patience and willingness to help out. Please find attached my train_errors.log file. I'm going through it myself right now.</p>\n<p>Best,</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60395,
      "author_name": "maxsop",
      "author_url": "",
      "post_date": "12/16/2014 20:51:11",
      "content": "<p>I think I got it working now. Will let you know if I get the same results you posted or not. Still running as I speak. Any idea on how to speed up or palatalized this? Again, thanks a lot, Mikek.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 61767,
      "author_name": "davidvondollen",
      "author_url": "",
      "post_date": "12/23/2014 17:37:54",
      "content": "<p>Hi MikeK,</p>\n\n<p>This is really cool, I was working on something similar using scikit's built in partial_fit class methods, but this looks much more dynamic. Nice work!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63227,
      "author_name": "yankov",
      "author_url": "",
      "post_date": "01/27/2015 07:15:33",
      "content": "<p>Hey,&nbsp;MikeK</p>\n<p>Thanks for posting this - very interesting! Can you explain why negative subsampling is needed here? Any good docs on this trick?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63232,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "01/27/2015 12:02:49",
      "content": "<p>[quote=Artem Yankov;63227]</p>\n<p>Hey,&nbsp;MikeK</p>\n<p>Thanks for posting this - very interesting! Can you explain why negative subsampling is needed here? Any good docs on this trick?</p>\n<p>[/quote]</p>\n<p>I'm curious about this too. Since both the train and test had the same subsampling, why is a correction needed?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63244,
      "author_name": "michaelkneier",
      "author_url": "",
      "post_date": "01/27/2015 18:43:34",
      "content": "<p>[quote=inversion;63232]</p>\n<p>[quote=Artem Yankov;63227]</p>\n<p>Hey,&nbsp;MikeK</p>\n<p>Thanks for posting this - very interesting! Can you explain why negative subsampling is needed here? Any good docs on this trick?</p>\n<p>[/quote]</p>\n<p>I'm curious about this too. Since both the train and test had the same subsampling, why is a correction needed?</p>\n<p>[/quote]</p>\n\n<p>Hi there.</p>\n<p>Negative subsampling isn't needed, but it is speeds up training and I found no significant negative impact on performance. To be clear, this algorithm performs subsampling in addition to the sampling the competition admin performed.&nbsp;</p>\n<p>Here is the paper that inspired me:</p>\n<p>http://tist.acm.org/papers/ACM-TIST-V5N4-TIST-2012-12-0197.R1.pdf</p>\n<p>Here is an interesting blog post on the topic:</p>\n<p>http://www.machinedlearnings.com/2010/11/on-unimportance-of-zeroes.html</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63245,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "01/27/2015 18:47:02",
      "content": "<p>[quote=MikeK;63244]</p>\n<p>Negative subsampling isn't needed, but it is speeds up training and I found no significant negative impact on performance. To be clear, this algorithm performs subsampling in addition to the sampling the competition admin performed.&nbsp;</p>\n<p>[/quote]</p>\n<p>Ah, that makes sense. Thanks for clarifying.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63247,
      "author_name": "yankov",
      "author_url": "",
      "post_date": "01/27/2015 19:35:11",
      "content": "<p>[quote=MikeK;63244]</p>\n<p>Negative subsampling isn't needed, but it is speeds up training and I found no significant negative impact on performance. To be clear, this algorithm performs subsampling in addition to the sampling the competition admin performed.&nbsp;</p>\n<p>Here is the paper that inspired me:</p>\n<p>http://tist.acm.org/papers/ACM-TIST-V5N4-TIST-2012-12-0197.R1.pdf</p>\n<p>Here is an interesting blog post on the topic:</p>\n<p>http://www.machinedlearnings.com/2010/11/on-unimportance-of-zeroes.html</p>\n<p>[/quote]</p>\n<p>Gotcha, thank you! And about feature hashing, how did you choose the number of the features to be 2 ** 20?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63248,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "01/27/2015 20:15:12",
      "content": "<p>[quote=Artem Yankov;63247]</p>\n<p>Gotcha, thank you! And about feature hashing, how did you choose the number of the features to be 2 ** 20?</p>\n<p>[/quote]</p>\n<p>2**20 is a good starting point.</p>\n<p>Go with as large as you can go with your system memory, and then scale down to see if you get improved scores.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63259,
      "author_name": "yankov",
      "author_url": "",
      "post_date": "01/28/2015 06:32:47",
      "content": "<p>[quote=inversion;63248]</p>\n<p>[quote=Artem Yankov;63247]</p>\n<p>Gotcha, thank you! And about feature hashing, how did you choose the number of the features to be 2 ** 20?</p>\n<p>[/quote]</p>\n<p>2**20 is a good starting point.</p>\n<p>Go with as large as you can go with your system memory, and then scale down to see if you get improved scores.</p>\n<p>[/quote]</p>\n\n<p>Thanks, inversion. Is there any particular reason to use hashing trick vs. one hot encoding here? I tried hashing trick on a subset and got worse results, but I keep playing with it.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63276,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "01/28/2015 13:40:51",
      "content": "<p>[quote=Artem Yankov;63259]</p>\n<p>Thanks, inversion. Is there any particular reason to use hashing trick vs. one hot encoding here? I tried hashing trick on a subset and got worse results, but I keep playing with it.&nbsp;</p>\n<p>[/quote]</p>\n<p>With hashing, you can read and process the file one line at a time (i.e., true online leaning). With &nbsp;one hot encoding, you have to have visibility to the entire dataset.</p>\n<p>I personally am using one hot encoding.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63321,
      "author_name": "yankov",
      "author_url": "",
      "post_date": "01/29/2015 18:53:32",
      "content": "<p>[quote=inversion;63276]</p>\n<p>[quote=Artem Yankov;63259]</p>\n<p>Thanks, inversion. Is there any particular reason to use hashing trick vs. one hot encoding here? I tried hashing trick on a subset and got worse results, but I keep playing with it.&nbsp;</p>\n<p>[/quote]</p>\n<p>With hashing, you can read and process the file one line at a time (i.e., true online leaning). With &nbsp;one hot encoding, you have to have visibility to the entire dataset.</p>\n<p>I personally am using one hot encoding.</p>\n<p>[/quote]</p>\n\n<p>Gotcha, thanks! I created short and simplified versions without subsampling or anything fancy. When using onehotencoder I get 0.414. When using hashing trick all predictions turn out to be 0. You guys can tell if I'm doing anything stupid here?</p>\n<p>Version with the hashing trick&nbsp;</p>\n<p>https://gist.github.com/yankov/1ab961c108613cf2d63a</p>\n<p>Version with onehotencoder (works)</p>\n<p>https://gist.github.com/yankov/c2ec3864c963da3b59f6</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63495,
      "author_name": "zhuhongtu",
      "author_url": "",
      "post_date": "02/03/2015 19:27:26",
      "content": "<p>Hi Artem:</p>\n<p>&nbsp; I cannot find OneHotEncoderPartial in your folders.</p>\n<p>Could you upload OneHotEncoderPartial file?</p>\n<p>Thanks</p>\n<p>[quote=Artem Yankov;63321]</p>\n<p>[quote=inversion;63276]</p>\n<p>[quote=Artem Yankov;63259]</p>\n<p>Thanks, inversion. Is there any particular reason to use hashing trick vs. one hot encoding here? I tried hashing trick on a subset and got worse results, but I keep playing with it.&nbsp;</p>\n<p>[/quote]</p>\n<p>With hashing, you can read and process the file one line at a time (i.e., true online leaning). With &nbsp;one hot encoding, you have to have visibility to the entire dataset.</p>\n<p>I personally am using one hot encoding.</p>\n<p>[/quote]</p>\n<p>Gotcha, thanks! I created short and simplified versions without subsampling or anything fancy. When using onehotencoder I get 0.414. When using hashing trick all predictions turn out to be 0. You guys can tell if I'm doing anything stupid here?</p>\n<p>Version with the hashing trick&nbsp;</p>\n<p>https://gist.github.com/yankov/1ab961c108613cf2d63a</p>\n<p>Version with onehotencoder (works)</p>\n<p>https://gist.github.com/yankov/c2ec3864c963da3b59f6</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63496,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "02/03/2015 19:39:40",
      "content": "<p>get 200gigs RAM, use SGD from sklearn n u will get below 0.40 easily ;)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63498,
      "author_name": "yankov",
      "author_url": "",
      "post_date": "02/03/2015 19:48:05",
      "content": "<p>[quote=Zhuhongtu;63495]</p>\n<p>Hi Artem:</p>\n<p>&nbsp; I cannot find OneHotEncoderPartial in your folders.</p>\n<p>Could you upload OneHotEncoderPartial file?</p>\n<p>Thanks</p>\n<p>[quote=Artem Yankov;63321]</p>\n<p>[quote=inversion;63276]</p>\n<p>[quote=Artem Yankov;63259]</p>\n<p>Thanks, inversion. Is there any particular reason to use hashing trick vs. one hot encoding here? I tried hashing trick on a subset and got worse results, but I keep playing with it.&nbsp;</p>\n<p>[/quote]</p>\n<p>With hashing, you can read and process the file one line at a time (i.e., true online leaning). With &nbsp;one hot encoding, you have to have visibility to the entire dataset.</p>\n<p>I personally am using one hot encoding.</p>\n<p>[/quote]</p>\n<p>Gotcha, thanks! I created short and simplified versions without subsampling or anything fancy. When using onehotencoder I get 0.414. When using hashing trick all predictions turn out to be 0. You guys can tell if I'm doing anything stupid here?</p>\n<p>Version with the hashing trick&nbsp;</p>\n<p>https://gist.github.com/yankov/1ab961c108613cf2d63a</p>\n<p>Version with onehotencoder (works)</p>\n<p>https://gist.github.com/yankov/c2ec3864c963da3b59f6</p>\n<p>[/quote]</p>\n<p>[/quote]</p>\n<p>Here is it</p>\n<p>https://gist.github.com/yankov/a86d44bf0e6009944c21</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63499,
      "author_name": "yankov",
      "author_url": "",
      "post_date": "02/03/2015 19:49:43",
      "content": "<p>[quote=Abhishek;63496]</p>\n<p>get 200gigs RAM, use SGD from sklearn n u will get below 0.40 easily ;)</p>\n<p>[/quote]</p>\n<p>Haha, yeah, I did that and spent ~$50 already on AWS. Still got to find the right parameters :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63534,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "02/04/2015 01:33:38",
      "content": "<p>[quote=Abhishek;63496]</p>\n<p>get 200gigs RAM, use SGD from sklearn n u will get below 0.40 easily ;)</p>\n<p>[/quote]</p>\n<p>Meh.</p>\n<p>See slide 33:</p>\n<p><strong>Hardware: 4x Intel CPU, 64 cores, 1TB RAM, $35K </strong></p>\n<p>http://stanford.edu/~rezab/nips2014workshop/slides/jure.pdf</p>\n<p>Slide 41 also has some good advice:</p>\n<p><em>Bottom Line: Get your own 1TB RAM server</em></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63738,
      "author_name": "yankov",
      "author_url": "",
      "post_date": "02/07/2015 02:34:10",
      "content": "<p>Fixed that feature hashing btw, had to use input_type = &quot;string&quot;. Final short version</p>\n<p>https://gist.github.com/yankov/1ab961c108613cf2d63a</p>\n<p>Doesn't give a good score (.41). But maybe someone new to scikit-learn will find it useful.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 312577,
      "author_name": "zhegeliang2",
      "author_url": "",
      "post_date": "04/12/2018 02:00:52",
      "content": "<p>this model running almost four hours</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "60245": "",
    "60276": "",
    "60284": "",
    "60286": "",
    "60313": "",
    "60374": "",
    "60376": "",
    "60378": "",
    "60387": "",
    "60393": "",
    "60395": "",
    "61767": "",
    "63227": "",
    "63232": "",
    "63244": "",
    "63245": "",
    "63247": "",
    "63248": "",
    "63259": "",
    "63276": "",
    "63321": "",
    "63495": "",
    "63496": "",
    "63498": "",
    "63499": "",
    "63534": "",
    "63738": "",
    "312577": "this model running almost four hours"
  },
  "source": "meta"
}