{
  "id": 12471,
  "title": "What was your best single model?",
  "url": "/competitions/avazu-ctr-prediction/discussion/12471",
  "author_name": "",
  "post_date": "2015-02-10T19:50:08.163Z",
  "votes": 7,
  "comment_count": 20,
  "views": 6275,
  "content": "<p>Hey guys,</p>\n<p>My best single model scored 0.3917 on the public leaderboard, and was the tinrtgu code with:</p>\n<p>- Manually engineered interactions. I had more success with interactions based on 5+ features.</p>\n<p>- Coding features that appeared less than 5 times in the training set as &quot;rare&quot;.</p>\n<p>&nbsp;- Removing single features that decreased performance based on a day 30 validation.</p>\n<p>Thanks tinrtgu for the amazing code, and congratulations to the winners.</p>",
  "messages": [
    {
      "id": "63958",
      "postDate": "02/10/2015 19:50:08",
      "content": "<p>Hey guys,</p>\n<p>My best single model scored 0.3917 on the public leaderboard, and was the tinrtgu code with:</p>\n<p>- Manually engineered interactions. I had more success with interactions based on 5+ features.</p>\n<p>- Coding features that appeared less than 5 times in the training set as &quot;rare&quot;.</p>\n<p>&nbsp;- Removing single features that decreased performance based on a day 30 validation.</p>\n<p>Thanks tinrtgu for the amazing code, and congratulations to the winners.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63960",
      "postDate": "02/10/2015 20:08:53",
      "content": "<p>This model should score 0.3901240 public.</p>\n<p>Please test and confirm.</p>\n<p>Thanks tinrtgu!</p>\n<p>Best. KaggleCode. Ever.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63961",
      "postDate": "02/10/2015 20:18:30",
      "content": "<p>I used vowpal wabbit until the end. My best single model was pretty simple and not that high-scoring, though it does clear the 0.4 hurdle that many others using vw were struggling to beat. I split date and time, added 'site_*' and 'app_*' interactions, and used L2 regularization. Tuned with vw-hypersearch using the last day's data for cross-validation and scored 0.3952868 (Edit: public 0.3952868, private&nbsp;0.3934794).</p>\n<p>The attached file munges the data to the format I used (thanks due to Triskelion, from whose code it was adapted), after which the flag '-q sa' creates the interactions.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63962",
      "postDate": "02/10/2015 20:23:18",
      "content": "<p>[quote=jwjohnson314;63961]</p>\n<p>I used vowpal wabbit until the end. My best single model was pretty simple and not that high-scoring, though it does clear the 0.4 hurdle that many others using vw were struggling to beat. I split date and time, added 'site_*' and 'app_*' interactions, and used L2 regularization. Tuned with vw-hypersearch using the last day's data for cross-validation and scored 0.3952868.</p>\n<p>[/quote]</p>\n<p>Can you actually post the command line code for vw you used?</p>\n<p>For instance, one part of my ensemble was likely something like this:<br>vw click.train_original_data.vw -f click.model.c.nn.vw --loss_function logistic --link logistic -l 1.7 -b 26 --power_t 0.58<br>There are many others part of the ensemble which individually may not give much but together are fairly powerful.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63964",
      "postDate": "02/10/2015 20:25:56",
      "content": "<p>Sure, it was:</p>\n<p>vw -d train.vw -b 22 -l 0.158789 --l2 4.7e-14 -q sa -f model.vw --loss_function logistic</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63971",
      "postDate": "02/10/2015 21:19:57",
      "content": "<p>I tinkered with Vowpal to take a single model down to something close to 0.3915:</p>\n<p>- made each feature a separate namespace</p>\n<p>- manually explored quadratic and cubic combinations that offered improvements</p>\n<p>In the end, I reached a similar score with tinrtgu's fantastic code and just took a simple average of the two.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63976",
      "postDate": "02/10/2015 22:28:19",
      "content": "<p>[quote=Mike Kim;63960]</p>\n<p>This model should score 0.3901240 public.</p>\n<p>Please test and confirm.</p>\n<p>Thanks tinrtgu!</p>\n<p>Best. KaggleCode. Ever.</p>\n<p>[/quote]</p>\n<p>Hi Mike, thanks for sharing.</p>\n<p>Please, could you explain the variables 'asite' and 'asite0'? Aren't they just duplicates of app_id?</p>\n<p>The same with 'app_site', 'site_app' and 'as2'?</p>\n<p>Thanks!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63981",
      "postDate": "02/11/2015 00:25:54",
      "content": "<p>It increased public leaderboard (greedy), so I included it. I'm not exactly sure why it made it better. It's purely multi collinear yes, but perhaps necessary given people split on the combo of site_id + app_id.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63996",
      "postDate": "02/11/2015 03:49:03",
      "content": "<p>My individual models scored (public LB):</p>\n<p>libFM (MCMC, dim=1,1,10, 8 million data points per cut, 2000 iterations, bagged over 6 random cuts of the data):&nbsp;0.3926204</p>\n<p>The 'fast_solution_plus' FTRL code from Sam Hocevar and Yannick Martel from the beat-the-benchmark thread&nbsp;with parameters (L1=2, L2=0.1,alpha=0.006,beta=1,bits=24,epochs=3,dropout=1, interactions and device counters on):&nbsp;0.3924675</p>\n<p>My final submission was a blend of those two, plus a version of the FTRL model with no interactions or device counters and a slight variant of the libFM parameters, for a public score of 0.3903096</p>\n<p>Also tried random forests and SVM, but they didn't keep up or really help the blend score.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63998",
      "postDate": "02/11/2015 04:48:44",
      "content": "<p>[quote=Nicholas Guttenberg;63996]</p>\n<p>...&nbsp;libFM (MCMC, dim=1,1,10, 8 million data points per cut, 2000 iterations, bagged over 6 random cuts of the data):&nbsp;0.3926204...</p>\n<p>[/quote]</p>\n<p>How long did that MCMC run take?&nbsp;I put LibFM to work on the Amazon dataset, which is around 1/1000th the size of this and it took 20-30 minutes for 3000 iterations if I recall correctly. It did produce the best results out of the available solvers though.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63999",
      "postDate": "02/11/2015 05:00:33",
      "content": "<p>I mentioned it in the other thread, but my best single model scored&nbsp;0.3894276 (private: 0.3876412).&nbsp;It was tinrtgu's FTRL-proximal, quite a bit like Mike's with some carefully chosen&nbsp;specific interactions.</p>\n<p>One big difference is that I used a giant alpha and L1/L2 parameters, by comparison. I started at 0.11 with L1/L2 of 8 each. And after each epoch, I decreased the rate by 15%.</p>\n<p>Most of my interactions were just 2-d. I merged device and app columns to condense and simplify and replaced C15 &amp; C16 with the combination.</p>\n<p>Also useful as&nbsp;1-d features and interactions&nbsp;were measuring counts by device_ip, and&nbsp;&nbsp;app/site id. Knowing how many records a user generated wasn't that useful, nor was the number of unique sites/apps; but the rate of those two features was useful. A few similar&nbsp;features were used extensively.</p>\n<p>The real reason I tried&nbsp;that sort of thing was because the 2nd version of the data set had a single IP that generated a lot of clicks at about 70% CTR or so...but it disappeared in the test set. And there were several like it. So I was curious if that pattern could be found without clicks. However, in this&nbsp;3rd version of the data, there wasn't much evidence of that behavior. So those counting features were more for sizing up all new users, rather than just bot-like&nbsp;users.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64000",
      "postDate": "02/11/2015 05:26:49",
      "content": "<p>[quote=David Thaler;63998]</p>\n<p>[quote=Nicholas Guttenberg;63996]</p>\n<p>...&nbsp;libFM (MCMC, dim=1,1,10, 8 million data points per cut, 2000 iterations, bagged over 6 random cuts of the data):&nbsp;0.3926204...</p>\n<p>[/quote]</p>\n<p>How long did that MCMC run take?&nbsp;I put LibFM to work on the Amazon dataset, which is around 1/1000th the size of this and it took 20-30 minutes for 3000 iterations if I recall correctly. It did produce the best results out of the available solvers though.</p>\n<p>[/quote]</p>\n<p>Something like 3 days, but I could run about 3 of them in parallel with each-other before running out of RAM.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64002",
      "postDate": "02/11/2015 05:53:00",
      "content": "<p>[quote=Mike Kim;63960]</p>\n<p>This model should score 0.3901240 public.</p>\n<p>[/quote]</p>\n\n<p>Mike Kim, I've never seen someone code duplicates of a feature before! You had site_id + app_id twice, app_id+site_id as well, and app_id three times. What made&nbsp;you think of this? I guess effectively you were weighting these features...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64003",
      "postDate": "02/11/2015 06:52:20",
      "content": "<p>[quote=Synthient;64002]</p>\n<p>...I've never seen someone code duplicates of a feature before! You had site_id + app_id twice, app_id+site_id as well, and app_id three times. What made&nbsp;you think of this? I guess effectively you were weighting these features...</p>\n<p>[/quote]</p>\n<p>This is probably part of the explanation: &nbsp;&nbsp;</p>\n<p><code>L2 = 610#330#1. # L2 regularization, larger value means more regularized &nbsp;&nbsp;</code></p>\n<p>That's a really high L2 regularization parameter.&nbsp;Probably that value was good for device_ip and its interactions, but sub-optimally high for app/site_id, which had relatively few values and a good bit of influence. The weights associated with those features (app/site_id) might have been overly constrained by that high L2 parameter. So having copies of the variable would allow the total weight on those variables to grow a bit higher.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64004",
      "postDate": "02/11/2015 07:50:31",
      "content": "<p>As others mentioned, you can split the dataset into site and app partitions but you lose power (compared to a large single model) that way. I couldn't get as high of a score using two models (one site, one app), but the above code seemed to work on the public LB. I guess I had some of the intuition David mentioned.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64005",
      "postDate": "02/11/2015 07:53:25",
      "content": "<p>Private LB: 0.3878</p>\n<p>I split app and site and replaced categorical values occurring less than 10 times with a dummy. And used Tinrtglu's FTRL:</p>\n<p>APP:<br>alpha=0.080000 beta=0.500000<br>L1=1.000000 L2=1.000000<br>D=2147483648 <br>epochs=3 <br>interactions:<br>app_id + device_model<br>app_id + device_id<br>delete the following fields:<br>hour C1 banner_pos site_id site_domain site_category app_domain device_type C17 C19 C21</p>\n<p>SITE:<br>alpha=0.075000 beta=0.100000<br>L1=1.300000 L2=14.000000<br>D=2147483648<br>epochs=1<br>interactions=<br>site_id + device_model<br>site_domain + C20<br>delete the following fields:<br>C17 C19</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64012",
      "postDate": "02/11/2015 09:43:44",
      "content": "<p>My best single model (actually I had no enough time and HW to try ensembles, so it's my only model) gain 0.3893116 on public LB and 0.3871975 on private LB. It was a forked VW with some heuristics implemented inside in it. I gave here a link on details: www.kaggle.com/c/avazu-ctr-prediction/forums/t/12476/vowpal-wabbit-lessons/64010#post64010</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64019",
      "postDate": "02/11/2015 12:28:44",
      "content": "<p>0.3919 (0.38999), just &nbsp;by</p>\n<p><code>vw ./train.vw -b 26 -q up -q pd --loss_function logistic --ftrl --ftrl_alpha 0.009</code></p>\n<p>where u &nbsp;- namespace for &nbsp;features <strong>C*</strong> , p - for <strong>site_*\\app_*</strong> and d - for<strong> device_*</strong>.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64061",
      "postDate": "02/11/2015 20:01:08",
      "content": "<p>My best single model scored 0.3905 public/0.3887 private. It was a VW model that used all of the original data except 'hour'. It had quadratic interactions over all of the fields&nbsp;from the original data except device_ip/id, which have high cardinality. I also added count-based features (not interacted):</p>\n<ul>\n<li>log count&nbsp;of web/app events by device_ip</li>\n<li>log count&nbsp;of unique web/app events by device_ip</li>\n<li>log count of visits to this site/app by this device_ip</li>\n<li>log count of web/app/all traffic and the log ratio of web/app events grouped by hour</li>\n</ul>\n<p>That set of count features is not carefully constructed. I just added that stuff in the last couple of days. Without those features, the model got 0.3919/0.3901, similar to what Mikhail Trofimov reports above.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64079",
      "postDate": "02/12/2015 00:12:56",
      "content": "<p>My best single model scored 0.3889111/3869869 (public/private). My model based on tinrtgu's code (thanks&nbsp;tinrtgu!!) &nbsp;and I made following modifications:</p>\n<p>-&gt; Dropout is used to regularize the model (estimated improvement = ~0.001-0.002)</p>\n<p>-&gt; Predict the ctr at current, previous and next hour, and time averaging them by using &lt;pred&gt;=0.1*&lt;pre_pred&gt; + 0.8*&lt;cur_pred&gt; + 0.1*&lt;nex_pred&gt; (estimated improvement = ~0.0001-0.0003)</p>\n<p>-&gt; Add&nbsp;crc32 hashing method to reduce the collision</p>\n<p>Features:</p>\n<p>-&gt; Similar to laserwolf,&nbsp;I replaced device_ip&nbsp;values occurring less than 5 times with a dummy.</p>\n<p>-&gt; Similar to David Thaler, I calculate the inverse factorial count of all features (daily, hourly and whole dataset).</p>\n<p>-&gt; Estimate the trend of each features by comparing the features' frequency with their frequency in the previous hour (+/- is less than 10%, consider the trend are the 'same', else set the trend to 'high' or 'low').&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64182",
      "postDate": "02/13/2015 19:00:55",
      "content": "<p>My best single model is a custom implementation of Microsoft Research AdPredictor algorithm, using feature hashing and a few others tricks:</p>\n<ul>\n<li>all feature interactions</li>\n<li>keeping only hour from datetime, and hour binning</li>\n<li>removing device_id / device_ip appearing less than 5 times</li>\n<li>adding a count of times since beginning of day a particular device_id/app_id or site_id combination has already been seen (capped to 8), and the same with device_ip</li>\n<li>a single pass</li>\n</ul>\n<p>Pretty simple, indeed, and going down to 0.3900968 public / 0.3881285 private.</p>\n<p>I have found AdPredictor quite fun to implement, reasonably efficient even in Python / pypy and easy to tune: only one parameter to tune, all others proved to be unproductive...</p>\n<p>With my partner, we merged it with other models.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 63960,
      "author_name": "mikeskim",
      "author_url": "",
      "post_date": "02/10/2015 20:08:53",
      "content": "<p>This model should score 0.3901240 public.</p>\n<p>Please test and confirm.</p>\n<p>Thanks tinrtgu!</p>\n<p>Best. KaggleCode. Ever.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63961,
      "author_name": "jwjohnson314",
      "author_url": "",
      "post_date": "02/10/2015 20:18:30",
      "content": "<p>I used vowpal wabbit until the end. My best single model was pretty simple and not that high-scoring, though it does clear the 0.4 hurdle that many others using vw were struggling to beat. I split date and time, added 'site_*' and 'app_*' interactions, and used L2 regularization. Tuned with vw-hypersearch using the last day's data for cross-validation and scored 0.3952868 (Edit: public 0.3952868, private&nbsp;0.3934794).</p>\n<p>The attached file munges the data to the format I used (thanks due to Triskelion, from whose code it was adapted), after which the flag '-q sa' creates the interactions.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63962,
      "author_name": "mikeskim",
      "author_url": "",
      "post_date": "02/10/2015 20:23:18",
      "content": "<p>[quote=jwjohnson314;63961]</p>\n<p>I used vowpal wabbit until the end. My best single model was pretty simple and not that high-scoring, though it does clear the 0.4 hurdle that many others using vw were struggling to beat. I split date and time, added 'site_*' and 'app_*' interactions, and used L2 regularization. Tuned with vw-hypersearch using the last day's data for cross-validation and scored 0.3952868.</p>\n<p>[/quote]</p>\n<p>Can you actually post the command line code for vw you used?</p>\n<p>For instance, one part of my ensemble was likely something like this:<br>vw click.train_original_data.vw -f click.model.c.nn.vw --loss_function logistic --link logistic -l 1.7 -b 26 --power_t 0.58<br>There are many others part of the ensemble which individually may not give much but together are fairly powerful.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63964,
      "author_name": "jwjohnson314",
      "author_url": "",
      "post_date": "02/10/2015 20:25:56",
      "content": "<p>Sure, it was:</p>\n<p>vw -d train.vw -b 22 -l 0.158789 --l2 4.7e-14 -q sa -f model.vw --loss_function logistic</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63971,
      "author_name": "cartographic",
      "author_url": "",
      "post_date": "02/10/2015 21:19:57",
      "content": "<p>I tinkered with Vowpal to take a single model down to something close to 0.3915:</p>\n<p>- made each feature a separate namespace</p>\n<p>- manually explored quadratic and cubic combinations that offered improvements</p>\n<p>In the end, I reached a similar score with tinrtgu's fantastic code and just took a simple average of the two.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63976,
      "author_name": "mariofilho",
      "author_url": "",
      "post_date": "02/10/2015 22:28:19",
      "content": "<p>[quote=Mike Kim;63960]</p>\n<p>This model should score 0.3901240 public.</p>\n<p>Please test and confirm.</p>\n<p>Thanks tinrtgu!</p>\n<p>Best. KaggleCode. Ever.</p>\n<p>[/quote]</p>\n<p>Hi Mike, thanks for sharing.</p>\n<p>Please, could you explain the variables 'asite' and 'asite0'? Aren't they just duplicates of app_id?</p>\n<p>The same with 'app_site', 'site_app' and 'as2'?</p>\n<p>Thanks!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63981,
      "author_name": "mikeskim",
      "author_url": "",
      "post_date": "02/11/2015 00:25:54",
      "content": "<p>It increased public leaderboard (greedy), so I included it. I'm not exactly sure why it made it better. It's purely multi collinear yes, but perhaps necessary given people split on the combo of site_id + app_id.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63996,
      "author_name": "ngutten",
      "author_url": "",
      "post_date": "02/11/2015 03:49:03",
      "content": "<p>My individual models scored (public LB):</p>\n<p>libFM (MCMC, dim=1,1,10, 8 million data points per cut, 2000 iterations, bagged over 6 random cuts of the data):&nbsp;0.3926204</p>\n<p>The 'fast_solution_plus' FTRL code from Sam Hocevar and Yannick Martel from the beat-the-benchmark thread&nbsp;with parameters (L1=2, L2=0.1,alpha=0.006,beta=1,bits=24,epochs=3,dropout=1, interactions and device counters on):&nbsp;0.3924675</p>\n<p>My final submission was a blend of those two, plus a version of the FTRL model with no interactions or device counters and a slight variant of the libFM parameters, for a public score of 0.3903096</p>\n<p>Also tried random forests and SVM, but they didn't keep up or really help the blend score.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63998,
      "author_name": "davidthaler",
      "author_url": "",
      "post_date": "02/11/2015 04:48:44",
      "content": "<p>[quote=Nicholas Guttenberg;63996]</p>\n<p>...&nbsp;libFM (MCMC, dim=1,1,10, 8 million data points per cut, 2000 iterations, bagged over 6 random cuts of the data):&nbsp;0.3926204...</p>\n<p>[/quote]</p>\n<p>How long did that MCMC run take?&nbsp;I put LibFM to work on the Amazon dataset, which is around 1/1000th the size of this and it took 20-30 minutes for 3000 iterations if I recall correctly. It did produce the best results out of the available solvers though.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63999,
      "author_name": "mlandry",
      "author_url": "",
      "post_date": "02/11/2015 05:00:33",
      "content": "<p>I mentioned it in the other thread, but my best single model scored&nbsp;0.3894276 (private: 0.3876412).&nbsp;It was tinrtgu's FTRL-proximal, quite a bit like Mike's with some carefully chosen&nbsp;specific interactions.</p>\n<p>One big difference is that I used a giant alpha and L1/L2 parameters, by comparison. I started at 0.11 with L1/L2 of 8 each. And after each epoch, I decreased the rate by 15%.</p>\n<p>Most of my interactions were just 2-d. I merged device and app columns to condense and simplify and replaced C15 &amp; C16 with the combination.</p>\n<p>Also useful as&nbsp;1-d features and interactions&nbsp;were measuring counts by device_ip, and&nbsp;&nbsp;app/site id. Knowing how many records a user generated wasn't that useful, nor was the number of unique sites/apps; but the rate of those two features was useful. A few similar&nbsp;features were used extensively.</p>\n<p>The real reason I tried&nbsp;that sort of thing was because the 2nd version of the data set had a single IP that generated a lot of clicks at about 70% CTR or so...but it disappeared in the test set. And there were several like it. So I was curious if that pattern could be found without clicks. However, in this&nbsp;3rd version of the data, there wasn't much evidence of that behavior. So those counting features were more for sizing up all new users, rather than just bot-like&nbsp;users.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64000,
      "author_name": "ngutten",
      "author_url": "",
      "post_date": "02/11/2015 05:26:49",
      "content": "<p>[quote=David Thaler;63998]</p>\n<p>[quote=Nicholas Guttenberg;63996]</p>\n<p>...&nbsp;libFM (MCMC, dim=1,1,10, 8 million data points per cut, 2000 iterations, bagged over 6 random cuts of the data):&nbsp;0.3926204...</p>\n<p>[/quote]</p>\n<p>How long did that MCMC run take?&nbsp;I put LibFM to work on the Amazon dataset, which is around 1/1000th the size of this and it took 20-30 minutes for 3000 iterations if I recall correctly. It did produce the best results out of the available solvers though.</p>\n<p>[/quote]</p>\n<p>Something like 3 days, but I could run about 3 of them in parallel with each-other before running out of RAM.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64002,
      "author_name": "brundage",
      "author_url": "",
      "post_date": "02/11/2015 05:53:00",
      "content": "<p>[quote=Mike Kim;63960]</p>\n<p>This model should score 0.3901240 public.</p>\n<p>[/quote]</p>\n\n<p>Mike Kim, I've never seen someone code duplicates of a feature before! You had site_id + app_id twice, app_id+site_id as well, and app_id three times. What made&nbsp;you think of this? I guess effectively you were weighting these features...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64003,
      "author_name": "davidthaler",
      "author_url": "",
      "post_date": "02/11/2015 06:52:20",
      "content": "<p>[quote=Synthient;64002]</p>\n<p>...I've never seen someone code duplicates of a feature before! You had site_id + app_id twice, app_id+site_id as well, and app_id three times. What made&nbsp;you think of this? I guess effectively you were weighting these features...</p>\n<p>[/quote]</p>\n<p>This is probably part of the explanation: &nbsp;&nbsp;</p>\n<p><code>L2 = 610#330#1. # L2 regularization, larger value means more regularized &nbsp;&nbsp;</code></p>\n<p>That's a really high L2 regularization parameter.&nbsp;Probably that value was good for device_ip and its interactions, but sub-optimally high for app/site_id, which had relatively few values and a good bit of influence. The weights associated with those features (app/site_id) might have been overly constrained by that high L2 parameter. So having copies of the variable would allow the total weight on those variables to grow a bit higher.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64004,
      "author_name": "mikeskim",
      "author_url": "",
      "post_date": "02/11/2015 07:50:31",
      "content": "<p>As others mentioned, you can split the dataset into site and app partitions but you lose power (compared to a large single model) that way. I couldn't get as high of a score using two models (one site, one app), but the above code seemed to work on the public LB. I guess I had some of the intuition David mentioned.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64005,
      "author_name": "vvarkey",
      "author_url": "",
      "post_date": "02/11/2015 07:53:25",
      "content": "<p>Private LB: 0.3878</p>\n<p>I split app and site and replaced categorical values occurring less than 10 times with a dummy. And used Tinrtglu's FTRL:</p>\n<p>APP:<br>alpha=0.080000 beta=0.500000<br>L1=1.000000 L2=1.000000<br>D=2147483648 <br>epochs=3 <br>interactions:<br>app_id + device_model<br>app_id + device_id<br>delete the following fields:<br>hour C1 banner_pos site_id site_domain site_category app_domain device_type C17 C19 C21</p>\n<p>SITE:<br>alpha=0.075000 beta=0.100000<br>L1=1.300000 L2=14.000000<br>D=2147483648<br>epochs=1<br>interactions=<br>site_id + device_model<br>site_domain + C20<br>delete the following fields:<br>C17 C19</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64012,
      "author_name": "trufanov",
      "author_url": "",
      "post_date": "02/11/2015 09:43:44",
      "content": "<p>My best single model (actually I had no enough time and HW to try ensembles, so it's my only model) gain 0.3893116 on public LB and 0.3871975 on private LB. It was a forked VW with some heuristics implemented inside in it. I gave here a link on details: www.kaggle.com/c/avazu-ctr-prediction/forums/t/12476/vowpal-wabbit-lessons/64010#post64010</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64019,
      "author_name": "mikhailtrofimov",
      "author_url": "",
      "post_date": "02/11/2015 12:28:44",
      "content": "<p>0.3919 (0.38999), just &nbsp;by</p>\n<p><code>vw ./train.vw -b 26 -q up -q pd --loss_function logistic --ftrl --ftrl_alpha 0.009</code></p>\n<p>where u &nbsp;- namespace for &nbsp;features <strong>C*</strong> , p - for <strong>site_*\\app_*</strong> and d - for<strong> device_*</strong>.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64061,
      "author_name": "davidthaler",
      "author_url": "",
      "post_date": "02/11/2015 20:01:08",
      "content": "<p>My best single model scored 0.3905 public/0.3887 private. It was a VW model that used all of the original data except 'hour'. It had quadratic interactions over all of the fields&nbsp;from the original data except device_ip/id, which have high cardinality. I also added count-based features (not interacted):</p>\n<ul>\n<li>log count&nbsp;of web/app events by device_ip</li>\n<li>log count&nbsp;of unique web/app events by device_ip</li>\n<li>log count of visits to this site/app by this device_ip</li>\n<li>log count of web/app/all traffic and the log ratio of web/app events grouped by hour</li>\n</ul>\n<p>That set of count features is not carefully constructed. I just added that stuff in the last couple of days. Without those features, the model got 0.3919/0.3901, similar to what Mikhail Trofimov reports above.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64079,
      "author_name": "senbong",
      "author_url": "",
      "post_date": "02/12/2015 00:12:56",
      "content": "<p>My best single model scored 0.3889111/3869869 (public/private). My model based on tinrtgu's code (thanks&nbsp;tinrtgu!!) &nbsp;and I made following modifications:</p>\n<p>-&gt; Dropout is used to regularize the model (estimated improvement = ~0.001-0.002)</p>\n<p>-&gt; Predict the ctr at current, previous and next hour, and time averaging them by using &lt;pred&gt;=0.1*&lt;pre_pred&gt; + 0.8*&lt;cur_pred&gt; + 0.1*&lt;nex_pred&gt; (estimated improvement = ~0.0001-0.0003)</p>\n<p>-&gt; Add&nbsp;crc32 hashing method to reduce the collision</p>\n<p>Features:</p>\n<p>-&gt; Similar to laserwolf,&nbsp;I replaced device_ip&nbsp;values occurring less than 5 times with a dummy.</p>\n<p>-&gt; Similar to David Thaler, I calculate the inverse factorial count of all features (daily, hourly and whole dataset).</p>\n<p>-&gt; Estimate the trend of each features by comparing the features' frequency with their frequency in the previous hour (+/- is less than 10%, consider the trend are the 'same', else set the trend to 'high' or 'low').&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64182,
      "author_name": "martely",
      "author_url": "",
      "post_date": "02/13/2015 19:00:55",
      "content": "<p>My best single model is a custom implementation of Microsoft Research AdPredictor algorithm, using feature hashing and a few others tricks:</p>\n<ul>\n<li>all feature interactions</li>\n<li>keeping only hour from datetime, and hour binning</li>\n<li>removing device_id / device_ip appearing less than 5 times</li>\n<li>adding a count of times since beginning of day a particular device_id/app_id or site_id combination has already been seen (capped to 8), and the same with device_ip</li>\n<li>a single pass</li>\n</ul>\n<p>Pretty simple, indeed, and going down to 0.3900968 public / 0.3881285 private.</p>\n<p>I have found AdPredictor quite fun to implement, reasonably efficient even in Python / pypy and easy to tune: only one parameter to tune, all others proved to be unproductive...</p>\n<p>With my partner, we merged it with other models.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "63958": "",
    "63960": "",
    "63961": "",
    "63962": "",
    "63964": "",
    "63971": "",
    "63976": "",
    "63981": "",
    "63996": "",
    "63998": "",
    "63999": "",
    "64000": "",
    "64002": "",
    "64003": "",
    "64004": "",
    "64005": "",
    "64012": "",
    "64019": "",
    "64061": "",
    "64079": "",
    "64182": ""
  },
  "source": "meta"
}