{
  "id": 11033,
  "title": "Is there still substantial room for improvement?",
  "url": "/competitions/avazu-ctr-prediction/discussion/11033",
  "author_name": "",
  "post_date": "2014-11-24T15:52:53.683Z",
  "votes": 6,
  "comment_count": 36,
  "views": 11371,
  "content": "<p>Ok, I admit, I haven't looked into this data thoroughly yet, but I have already tried&nbsp;throwing at it a couple of &quot;classics&quot; and some more &quot;Kaggle tricks&quot;. A quick and dirty ensemble between XBG (LB .40) and a linear model&nbsp;(.396) gives pretty much no improvement, which, for me, is really surprising. Thinking about <a href=\"http://www.kaggle.com/c/avazu-ctr-prediction/forums/t/11016/0-39-more-to-the-dataset-than-meets-the-eye/58680#post58680\">M's post</a>, and her smart insights on&nbsp;variable selection, the most likely way to explain what I'm seeing is that, as she said, there are really only a handful of features that matter, and both a linear an tree-based models are picking those relationships up and modelling them similarly. Too similarly...</p>\n<p>Every competition I participated in has a couple of glass ceilings&nbsp;participants eventually end up shattering. I wonder if this might be a Kaggle first, where we reached a plateau so soon in the competition (well, not THAT soon, considering all the false starts...).</p>",
  "messages": [
    {
      "id": "58777",
      "postDate": "11/24/2014 15:52:53",
      "content": "<p>Ok, I admit, I haven't looked into this data thoroughly yet, but I have already tried&nbsp;throwing at it a couple of &quot;classics&quot; and some more &quot;Kaggle tricks&quot;. A quick and dirty ensemble between XBG (LB .40) and a linear model&nbsp;(.396) gives pretty much no improvement, which, for me, is really surprising. Thinking about <a href=\"http://www.kaggle.com/c/avazu-ctr-prediction/forums/t/11016/0-39-more-to-the-dataset-than-meets-the-eye/58680#post58680\">M's post</a>, and her smart insights on&nbsp;variable selection, the most likely way to explain what I'm seeing is that, as she said, there are really only a handful of features that matter, and both a linear an tree-based models are picking those relationships up and modelling them similarly. Too similarly...</p>\n<p>Every competition I participated in has a couple of glass ceilings&nbsp;participants eventually end up shattering. I wonder if this might be a Kaggle first, where we reached a plateau so soon in the competition (well, not THAT soon, considering all the false starts...).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58779",
      "postDate": "11/24/2014 16:16:23",
      "content": "<p>Nah, there's definitely more room. I'm going out on a limb and predicting the LB will get to at least 0.37.</p>\n<p>I'm still in &quot;dinking-around mode&quot; and have been able to get to 0.393.</p>\n<p>I'm expecting improvements once I bring out the power tools.</p>\n<p>Also . . . with 200+ LB entries left between now and the contest end, there's still plenty of opportunity to overfit. &nbsp;:-)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58780",
      "postDate": "11/24/2014 16:17:54",
      "content": "<p>I think there's room for improvement but haven't figured it out yet. I'm not sure if the sampling strategy has introduced oddities. I totally changed data - only used prior probabilities and still got .39 on validation - didn't even use any original features. I'm going to manually go through some of the data. I reckon there's a &quot;golden feature&quot; somewhere</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58781",
      "postDate": "11/24/2014 16:25:08",
      "content": "<p>Just out of curiosity, what's XBG? Or do you mean XGB?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58782",
      "postDate": "11/24/2014 16:29:06",
      "content": "<p>[quote=lewis ml;58781]</p>\n<p>Just out of curiosity, what's XBG? Or do you mean XGB?</p>\n<p>[/quote]</p>\n<p>xgboost.&nbsp;https://github.com/tqchen/xgboost</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58783",
      "postDate": "11/24/2014 16:30:42",
      "content": "<p>More than half of the LB is based on same benchmark code. If people try new ideas, the score should improve....</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58785",
      "postDate": "11/24/2014 16:33:29",
      "content": "<p>[quote=lewis ml;58781]</p>\n<p>Just out of curiosity, what's XBG? Or do you mean XGB?</p>\n<p>[/quote]</p>\n<p>XGB. My bad...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58786",
      "postDate": "11/24/2014 16:33:54",
      "content": "<p>[quote=Abhishek;58783]</p>\n<p>More than half of the LB is based on same benchmark code. If people try new ideas, the score should improve....</p>\n<p>[/quote]</p>\n<p>Giulio has used XGB - not in benchmark code. I've tried several different things. I only ran the &quot;remove variable&quot; code using the benchmark and that was only because it was quick</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58787",
      "postDate": "11/24/2014 16:44:01",
      "content": "<p>I also tried VW and &nbsp;ensembled it with XGB and benchmark.</p>\n<p>weighted average of VW&nbsp;and&nbsp;XGB -&gt; 0.394xxx.</p>\n<p>weighted average of VW,XGB and benchmark -&gt; 0.394xxx</p>\n<p>So, the benchmark adds nothing on top of VW+XGB. Which might indicate that the benchmark and VW /XGB are not modelling the data differently enough.</p>\n<p>I'll change gear and will start looking for golden features or &quot;golden rows&quot;.</p>\n<p>After that I'll just sit and wait for someone to post a beat the benchmark &lt;0.390 code :-)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58788",
      "postDate": "11/24/2014 16:49:21",
      "content": "<p>[quote=Giulio;58785]</p>\n<p>[quote=lewis ml;58781]</p>\n<p>Just out of curiosity, what's XBG? Or do you mean XGB?</p>\n<p>[/quote]</p>\n<p>XGB. My bad...</p>\n<p>[/quote]</p>\n<p>No, I make the same mistake. Not the easiest acronym. Just wanted to check my understanding.</p>\n<p>A more general question: of the all options on the table, which approach do you think is the most convenient for this sort of problem for someone working with R, if you were just thinking of getting a 'pretty good' score rather than top of the leaderboard and if further gains will require a lot of effort?</p>\n<p>I ask because tools like Random Forest are so convenient and user-friendly and I'm curious to know which technique people might consider to be equivalent for this sort of problem.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58790",
      "postDate": "11/24/2014 17:17:46",
      "content": "<p>[quote=Giulio;58777]</p>\n<p>&nbsp;A quick and dirty ensemble between XBG (LB .40) and a linear model&nbsp;(.396) gives pretty much no improvement, which, for me, is really surprising.&nbsp;</p>\n<p>[/quote]</p>\n<p>Same here, nothing blends well. We blend a FM with the benchmark, only tiny improvement! But I believe there is room. So could anyone tell me what's different from version 2? People made progress really fast last time. Simpy because of cracking things?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58791",
      "postDate": "11/24/2014 17:26:53",
      "content": "<p>It's been less than a week since the contest started. What's with the pessimism? Has there ever been a contest in the history of Kaggle where the 7 day best public is even within the top 10% of final private leaderboard?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58792",
      "postDate": "11/24/2014 17:34:13",
      "content": "<p>I blend two logistic model with different learning rate and got 0.0003 LB improvement.</p>\n<p>Is this tiny? ;)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58794",
      "postDate": "11/24/2014 17:39:35",
      "content": "<p>[quote=Mike Kim;58791]</p>\n<p>It's been less than a week since the contest started. What's with the pessimism? Has there ever been a contest in the history of Kaggle where the 7 day best public is even within the top 10% of final private leaderboard?</p>\n<p>[/quote]</p>\n<p>lol. I totally agree with your argument.</p>\n<p>Yet,&nbsp;somehow this competition does give that 'feeling' of 'not much scope for improvement'. I'm not sure what or why is it so, but this competition is just weird.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58796",
      "postDate": "11/24/2014 17:45:42",
      "content": "<p>[quote=Rohan Rao;58794]</p>\n<p>Yet,&nbsp;somehow this competition does give that 'feeling' of 'not much scope for improvement'. I'm not sure what or why is it so, but this competition is just weird.</p>\n<p>[/quote]</p>\n<p>It does, doesn't it? :-)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58799",
      "postDate": "11/24/2014 18:07:37",
      "content": "<p>I can tell that there's no more room to improvement!!! &nbsp; But if I do that I'll be lying...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58801",
      "postDate": "11/24/2014 18:23:01",
      "content": "<p>[quote=rcarson;58790]</p>\n<p>Same here, nothing blends well. We blend a FM with the benchmark, only tiny improvement! But I believe there is room. So could anyone tell me what's different from version 2? People made progress really fast last time. Simpy because of cracking things?</p>\n<p>[/quote]</p>\n\n<p>Are you using libFM out of curiosity?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58802",
      "postDate": "11/24/2014 18:27:39",
      "content": "<p>One thing I have concern is 'fake' click. Did anyone hear about get paid to click ads?</p>\n<p>In the model it will show high click rate for certain device ID and&nbsp;those ID is very active. This will skew the model results. M's post shows the dimension of the data is low.&nbsp;</p>\n<p>Maybe a simple EDA will tell...&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58803",
      "postDate": "11/24/2014 18:45:01",
      "content": "<p>[quote=Inspector;58801]</p>\n<p>Are you using libFM out of curiosity?</p>\n<p>[/quote]</p>\n<p>no, we use the fm of 3 idiots' winning solution of the last CTR contest. On the shoulders of a giant &nbsp;:D</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58804",
      "postDate": "11/24/2014 18:47:11",
      "content": "<p>Just out of curiosity, while getting &lt;0.4 on LB what is your score on CV ? Cause I'm getting as low as 0.27 on CV (with&nbsp;different holdout strategies), but my best LB score is, well,&nbsp;crap.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58805",
      "postDate": "11/24/2014 19:00:22",
      "content": "<p>[quote=rcarson;58803]</p>\n<p>[quote=Inspector;58801]</p>\n<p>Are you using libFM out of curiosity?</p>\n<p>[/quote]</p>\n<p>no, we use the fm of 3 idiots' winning solution of the last CTR contest. On the shoulders of a giant &nbsp;:D</p>\n<p>[/quote]</p>\n<p>You move fast... :-)</p>\n<p>That's another reason why I asked the initial question. This competition is not incredibly different from Criteo. Not the same, but it is not unconcievable that some of the things that worked in Criteo would work here. So, I had/have the feeling many folks are already trying fairly advanced models/ensembles, yet the improvement is still low. It is true that this competition is not truly a week old, but at this time of the competition it is common to see much more &quot;spacing&quot; in the top. Instead top ranks are already separated only by 3rd decimals, despite evidence of many folks throwing many things at this problem.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58806",
      "postDate": "11/24/2014 19:05:05",
      "content": "<p>[quote=Giulio;58777]&nbsp;ensemble between XBG (LB .40) and a linear model&nbsp;(.396)&nbsp;[/quote]</p>\n<p>You mean gradient boosting applied to feature histograms? In validation, I've had down to 0.397 with that technique, and I haven't explored 1% of what's possible. As for linear models on sparse features, I assume they can easily go down to 0.390 (lowest I've had so far was 0.392).&nbsp;</p>\n<p>Clearly you aren't trying hard enough.</p>\n<p>Ensembling doesn't work? That's because your models aren't different enough. FYI gradient boosting on histograms and linear SGD over sparse features isn't actually much different (since the coefficients of the linear model will converge towards the histograms).</p>\n<p>I think this contest will end up around 0.37-0.38. There is plenty of room for improvement. Maybe some innovation will even necessary, though, rather than merely microwaving the algorithms that did well in the Criteo challenge.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58807",
      "postDate": "11/24/2014 19:21:03",
      "content": "<p>[quote=fchollet;58806]</p>\n<p>[quote=Giulio;58777]&nbsp;ensemble between XBG (LB .40) and a linear model&nbsp;(.396)&nbsp;[/quote]</p>\n<p>You mean gradient boosting applied to feature histograms?&nbsp;</p>\n<p>[/quote]</p>\n<p>Hi, what is the&nbsp;feature histograms actually? I have seen it for a couple of times. Thank you!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58808",
      "postDate": "11/24/2014 19:22:08",
      "content": "<p>[quote=fchollet;58806]</p>\n<p>Clearly you aren't trying hard enough.</p>\n<p>[/quote]</p>\n<p>LOL. I'm definitely not. That's kind of the first thing I said in my original post :-)</p>\n<p>[quote=fchollet;58806]</p>\n<p>Ensembling doesn't work? That's because your models aren't different enough. FYI gradient boosting on histograms and linear SGD over sparse features isn't actually much different (since the coefficients of the linear model will converge towards the histograms).</p>\n<p>[/quote]</p>\n<p>I'm using XGB on the encoded features. No histograms.&nbsp;<br>Can you elaborate on the convergence thing? I believe you but it's not intuitive for me...</p>\n<p>[quote=fchollet;58806]</p>\n<p>I think this contest will end up around 0.37-0.38. There is plenty of room for improvement. Maybe some innovation will even necessary, though, rather than merely microwaving the algorithms that did well in the Criteo challenge.</p>\n<p>[/quote]</p>\n<p>I hope it does. I'm confident it will. Good luck to you! ;-)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58809",
      "postDate": "11/24/2014 19:32:41",
      "content": "<p>[quote=rcarson;58807]</p>\n<p>Hi, what is the&nbsp;feature histograms actually? I have seen it for a couple of times. Thank you!</p>\n<p>[/quote]</p>\n<p>You compute the clicks &amp; impressions histograms of feature values, then you encode each feature value as its average CTR (or some variation of it) over the training data. You lose information in the process, but it can make your model generalize better --maybe.</p>\n<p>[quote]</p>\n<p>Can you elaborate on the convergence thing? I believe you but it's not intuitive for me...</p>\n<p>[/quote]</p>\n<p>In a sparse linear model, you encode the occurrence of a feature value as a binary indicator in a specific dimension. The coefficient for that dimension in your model will converge towards the average CTR for that feature value, times some feature-specific weight. And if you add regularization it will converge towards a &quot;smoothed CTR&quot; value, eg. (some factor+clicks)/(impressions+some other factor).</p>\n<p>Roughly.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58823",
      "postDate": "11/24/2014 22:24:39",
      "content": "<p>[quote=fchollet;58809]</p>\n<p>You compute the clicks &amp; impressions histograms of feature values, then you encode each feature value as its average CTR (or some variation of it) over the training data. You lose information in the process, but it can make your model generalize better --maybe.</p>\n<p>[/quote]</p>\n\n<p>A histogram here then simply means average CTR per feature? So if you do this for, say site_id, then each site_id gets replaced with its marginal CTR - where marginal refers to the fact we would be aggregating over all other variables...and we could of course consider interactions between multiple variables (replacing a given tuple with the empirical CTR)?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58839",
      "postDate": "11/25/2014 01:02:59",
      "content": "<p>[quote=Inspector;58823]</p>\n<p>A histogram here then simply means average CTR per feature? So if you do this for, say site_id, then each site_id gets replaced with its marginal CTR - where marginal refers to the fact we would be aggregating over all other variables...and we could of course consider interactions between multiple variables (replacing a given tuple with the empirical CTR)?</p>\n<p>[/quote]</p>\n<p>Yes, although you have to keep in mind that your encoding get increasingly less reliable as you apply it to histograms that contain fewer events. That's why a smoothed out version might be preferable.</p>\n<p>In its simplest form it would be a simple average CTR, yes. In the general case it could be any encoding that is a function of the histogram of a feature value: e = f(impressions(value), clicks(value)) in R**n</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58840",
      "postDate": "11/25/2014 01:33:19",
      "content": "<p>[quote=fchollet;58839]</p>\n<p>Yes, although you have to keep in mind that your encoding get increasingly less reliable as you apply it to histograms that contain fewer events. That's why a smoothed out version might be preferable.</p>\n<p>In its simplest form it would be a simple average CTR, yes. In the general case it could be any encoding that is a function of the histogram of a feature value: e = f(impressions(value), clicks(value)) in R**n</p>\n<p>[/quote]</p>\n\n<p>Cool, thanks. I have never used the technique myself, but I suspect that using a shrinkage estimator (shrinking towards the grand mean, ala hierarchical mixed models in statistics) might be a nice solution.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59009",
      "postDate": "11/26/2014 14:19:55",
      "content": "<p>[quote=rcarson;58803]</p>\n<p>[quote=Inspector;58801]</p>\n<p>Are you using libFM out of curiosity?</p>\n<p>[/quote]</p>\n<p>no, we use the fm of 3 idiots' winning solution of the last CTR contest. On the shoulders of a giant &nbsp;:D</p>\n<p>[/quote]</p>\n<p>have you used it with&nbsp;gbdt features?</p>\n<p>When I've tried to use it I haven't added the&nbsp;gbdt feature and I've got much worse score compared to the benchmark.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59010",
      "postDate": "11/26/2014 14:55:14",
      "content": "<p>[quote=clustifier;59009]</p>\n<p>[quote=rcarson;58803]</p>\n<p>[quote=Inspector;58801]</p>\n<p>Are you using libFM out of curiosity?</p>\n<p>[/quote]</p>\n<p>no, we use the fm of 3 idiots' winning solution of the last CTR contest. On the shoulders of a giant &nbsp;:D</p>\n<p>[/quote]</p>\n<p>have you used it with&nbsp;gbdt features?</p>\n<p>When I've tried to use it I haven't added the&nbsp;gbdt feature and I've got much worse score compared to the benchmark.</p>\n<p>[/quote]</p>\n\n<p>Correct me if I'm wrong, but intuitively decision tree features could add something to the model if there would be any numerical features, as it was in criteo (because they can learn non-linear data from them). Here, if you have no numerical features and one-hot encoder for all categorical ones, additional features from trees shouldn't add much. What do you think?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59016",
      "postDate": "11/26/2014 16:34:06",
      "content": "<p>[quote=clustifier;59009]</p>\n<p>have you used it with&nbsp;gbdt features?</p>\n<p>When I've tried to use it I haven't added the&nbsp;gbdt feature and I've got much worse score compared to the benchmark.</p>\n<p>[/quote]</p>\n<p>I haven't used the gbdt yet. I just did random stuff and it looks good. Here is what I did and I have no idea which one helps or hurts</p>\n<p>1) all the features are treated as categorical features.</p>\n<p>2) frequent feature and infrequent features are not differentiated. I'm lazy.</p>\n<p>3) split &quot;hour&quot; to date+hour like what the benchmark did.</p>\n<p>And some parameter tuning I'm not ready to share now :D</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59017",
      "postDate": "11/26/2014 17:06:24",
      "content": "<p>I tried very similar to what rcarson&nbsp;did and used gbdt for training, but get very lame results (~ 0.51xxxx). Some thing must be wrong......</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59018",
      "postDate": "11/26/2014 17:16:02",
      "content": "<p>[quote=Ivan Lobov;59010]</p>\n<p>[quote=clustifier;59009]</p>\n<p>[quote=rcarson;58803]</p>\n<p>[quote=Inspector;58801]</p>\n<p>Are you using libFM out of curiosity?</p>\n<p>[/quote]</p>\n<p>no, we use the fm of 3 idiots' winning solution of the last CTR contest. On the shoulders of a giant &nbsp;:D</p>\n<p>[/quote]</p>\n<p>have you used it with&nbsp;gbdt features?</p>\n<p>When I've tried to use it I haven't added the&nbsp;gbdt feature and I've got much worse score compared to the benchmark.</p>\n<p>[/quote]</p>\n<p>Correct me if I'm wrong, but intuitively decision tree features could add something to the model if there would be any numerical features, as it was in criteo (because they can learn non-linear data from them). Here, if you have no numerical features and one-hot encoder for all categorical ones, additional features from trees shouldn't add much. What do you think?</p>\n<p>[/quote]</p>\n<p>Shouldn't a decision tree be able to capture feature interaction better, even for categorical data, compared to a linear model? For instance, suppose you had three relevant features after one-hotting, A, B, and C, and wanted a positive result for A&amp;B&amp;C but a negative result else. The tree can capture that perfectly, but the linear model will return w_A + w_B + w_C + offset. The difference in the linear model between A&amp;B&amp;C and A&amp;B&amp;~C is only w_C&lt;1, but the difference between those in the tree can be 1, it can perfectly distinguish.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59020",
      "postDate": "11/26/2014 17:25:59",
      "content": "<p>Tree feature has nothing to do with numerical feature. They just add cross effect of most frequent features. Right?</p>\n<p>[quote=Ivan Lobov;59010]</p>\n<p>Correct me if I'm wrong, but intuitively decision tree features could add something to the model if there would be any numerical features, as it was in criteo (because they can learn non-linear data from them). Here, if you have no numerical features and one-hot encoder for all categorical ones, additional features from trees shouldn't add much. What do you think?</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59021",
      "postDate": "11/26/2014 17:30:49",
      "content": "<p>[quote=superfan123;59020]</p>\n<p>Tree feature has nothing to do with numerical feature. They just add cross effect of most frequent features. Right?</p>\n<p>[/quote]</p>\n<p>I wouldn't quite say that. Even on a single numeric feature, a tree is fundamentally a different response (a heaviside function) from, say, a linear model (a straight line with some slope and intercept.) But yeah, they definitely can capture cross effects!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59024",
      "postDate": "11/26/2014 17:42:32",
      "content": "<p>[quote=LiamHuber;59018]</p>\n<p>Shouldn't a decision tree be able to capture feature interaction better, even for categorical data, compared to a linear model? For instance, suppose you had three relevant features after one-hotting, A, B, and C, and wanted a positive result for A&amp;B&amp;C but a negative result else. The tree can capture that perfectly, but the linear model will return w_A + w_B + w_C + offset. The difference in the linear model between A&amp;B&amp;C and A&amp;B&amp;~C is only w_C&lt;1, but the difference between those in the tree can be 1, it can perfectly distinguish.</p>\n<p>[/quote]</p>\n<p>You're very right! Linear model can capture the same things, but with feature interactions being used. Which is kind of slow if we're talking about 3-4 features dependecies. So using decision trees is probably a good idea :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59068",
      "postDate": "11/27/2014 02:21:35",
      "content": "<p>[quote=rcarson;58803]</p>\n<p>no, we use the fm of 3 idiots' winning solution of the last CTR contest. On the shoulders of a giant &nbsp;:D</p>\n<p>[/quote]</p>\n\n<p>Hey Guys,</p>\n<p>I am curious if anyone has used LibFM successfully in any Kaggle Competition? Also, has anyone used the low rank interaction feature of VW - it sounds like it should replicate what LibFM does...</p>\n<p>https://github.com/JohnLangford/vowpal_wabbit/tree/master/demo/movielens</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 58779,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "11/24/2014 16:16:23",
      "content": "<p>Nah, there's definitely more room. I'm going out on a limb and predicting the LB will get to at least 0.37.</p>\n<p>I'm still in &quot;dinking-around mode&quot; and have been able to get to 0.393.</p>\n<p>I'm expecting improvements once I bring out the power tools.</p>\n<p>Also . . . with 200+ LB entries left between now and the contest end, there's still plenty of opportunity to overfit. &nbsp;:-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58780,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "11/24/2014 16:17:54",
      "content": "<p>I think there's room for improvement but haven't figured it out yet. I'm not sure if the sampling strategy has introduced oddities. I totally changed data - only used prior probabilities and still got .39 on validation - didn't even use any original features. I'm going to manually go through some of the data. I reckon there's a &quot;golden feature&quot; somewhere</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58781,
      "author_name": "cartographic",
      "author_url": "",
      "post_date": "11/24/2014 16:25:08",
      "content": "<p>Just out of curiosity, what's XBG? Or do you mean XGB?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58782,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "11/24/2014 16:29:06",
      "content": "<p>[quote=lewis ml;58781]</p>\n<p>Just out of curiosity, what's XBG? Or do you mean XGB?</p>\n<p>[/quote]</p>\n<p>xgboost.&nbsp;https://github.com/tqchen/xgboost</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58783,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "11/24/2014 16:30:42",
      "content": "<p>More than half of the LB is based on same benchmark code. If people try new ideas, the score should improve....</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58785,
      "author_name": "adjgiulio",
      "author_url": "",
      "post_date": "11/24/2014 16:33:29",
      "content": "<p>[quote=lewis ml;58781]</p>\n<p>Just out of curiosity, what's XBG? Or do you mean XGB?</p>\n<p>[/quote]</p>\n<p>XGB. My bad...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58786,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "11/24/2014 16:33:54",
      "content": "<p>[quote=Abhishek;58783]</p>\n<p>More than half of the LB is based on same benchmark code. If people try new ideas, the score should improve....</p>\n<p>[/quote]</p>\n<p>Giulio has used XGB - not in benchmark code. I've tried several different things. I only ran the &quot;remove variable&quot; code using the benchmark and that was only because it was quick</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58787,
      "author_name": "adjgiulio",
      "author_url": "",
      "post_date": "11/24/2014 16:44:01",
      "content": "<p>I also tried VW and &nbsp;ensembled it with XGB and benchmark.</p>\n<p>weighted average of VW&nbsp;and&nbsp;XGB -&gt; 0.394xxx.</p>\n<p>weighted average of VW,XGB and benchmark -&gt; 0.394xxx</p>\n<p>So, the benchmark adds nothing on top of VW+XGB. Which might indicate that the benchmark and VW /XGB are not modelling the data differently enough.</p>\n<p>I'll change gear and will start looking for golden features or &quot;golden rows&quot;.</p>\n<p>After that I'll just sit and wait for someone to post a beat the benchmark &lt;0.390 code :-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58788,
      "author_name": "cartographic",
      "author_url": "",
      "post_date": "11/24/2014 16:49:21",
      "content": "<p>[quote=Giulio;58785]</p>\n<p>[quote=lewis ml;58781]</p>\n<p>Just out of curiosity, what's XBG? Or do you mean XGB?</p>\n<p>[/quote]</p>\n<p>XGB. My bad...</p>\n<p>[/quote]</p>\n<p>No, I make the same mistake. Not the easiest acronym. Just wanted to check my understanding.</p>\n<p>A more general question: of the all options on the table, which approach do you think is the most convenient for this sort of problem for someone working with R, if you were just thinking of getting a 'pretty good' score rather than top of the leaderboard and if further gains will require a lot of effort?</p>\n<p>I ask because tools like Random Forest are so convenient and user-friendly and I'm curious to know which technique people might consider to be equivalent for this sort of problem.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58790,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "11/24/2014 17:17:46",
      "content": "<p>[quote=Giulio;58777]</p>\n<p>&nbsp;A quick and dirty ensemble between XBG (LB .40) and a linear model&nbsp;(.396) gives pretty much no improvement, which, for me, is really surprising.&nbsp;</p>\n<p>[/quote]</p>\n<p>Same here, nothing blends well. We blend a FM with the benchmark, only tiny improvement! But I believe there is room. So could anyone tell me what's different from version 2? People made progress really fast last time. Simpy because of cracking things?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58791,
      "author_name": "mikeskim",
      "author_url": "",
      "post_date": "11/24/2014 17:26:53",
      "content": "<p>It's been less than a week since the contest started. What's with the pessimism? Has there ever been a contest in the history of Kaggle where the 7 day best public is even within the top 10% of final private leaderboard?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58792,
      "author_name": "superfan123",
      "author_url": "",
      "post_date": "11/24/2014 17:34:13",
      "content": "<p>I blend two logistic model with different learning rate and got 0.0003 LB improvement.</p>\n<p>Is this tiny? ;)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58794,
      "author_name": "rohanrao",
      "author_url": "",
      "post_date": "11/24/2014 17:39:35",
      "content": "<p>[quote=Mike Kim;58791]</p>\n<p>It's been less than a week since the contest started. What's with the pessimism? Has there ever been a contest in the history of Kaggle where the 7 day best public is even within the top 10% of final private leaderboard?</p>\n<p>[/quote]</p>\n<p>lol. I totally agree with your argument.</p>\n<p>Yet,&nbsp;somehow this competition does give that 'feeling' of 'not much scope for improvement'. I'm not sure what or why is it so, but this competition is just weird.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58796,
      "author_name": "adjgiulio",
      "author_url": "",
      "post_date": "11/24/2014 17:45:42",
      "content": "<p>[quote=Rohan Rao;58794]</p>\n<p>Yet,&nbsp;somehow this competition does give that 'feeling' of 'not much scope for improvement'. I'm not sure what or why is it so, but this competition is just weird.</p>\n<p>[/quote]</p>\n<p>It does, doesn't it? :-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58799,
      "author_name": "titericz",
      "author_url": "",
      "post_date": "11/24/2014 18:07:37",
      "content": "<p>I can tell that there's no more room to improvement!!! &nbsp; But if I do that I'll be lying...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58801,
      "author_name": "inspector",
      "author_url": "",
      "post_date": "11/24/2014 18:23:01",
      "content": "<p>[quote=rcarson;58790]</p>\n<p>Same here, nothing blends well. We blend a FM with the benchmark, only tiny improvement! But I believe there is room. So could anyone tell me what's different from version 2? People made progress really fast last time. Simpy because of cracking things?</p>\n<p>[/quote]</p>\n\n<p>Are you using libFM out of curiosity?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58802,
      "author_name": "superfan123",
      "author_url": "",
      "post_date": "11/24/2014 18:27:39",
      "content": "<p>One thing I have concern is 'fake' click. Did anyone hear about get paid to click ads?</p>\n<p>In the model it will show high click rate for certain device ID and&nbsp;those ID is very active. This will skew the model results. M's post shows the dimension of the data is low.&nbsp;</p>\n<p>Maybe a simple EDA will tell...&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58803,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "11/24/2014 18:45:01",
      "content": "<p>[quote=Inspector;58801]</p>\n<p>Are you using libFM out of curiosity?</p>\n<p>[/quote]</p>\n<p>no, we use the fm of 3 idiots' winning solution of the last CTR contest. On the shoulders of a giant &nbsp;:D</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58804,
      "author_name": "dmitrynozhnin",
      "author_url": "",
      "post_date": "11/24/2014 18:47:11",
      "content": "<p>Just out of curiosity, while getting &lt;0.4 on LB what is your score on CV ? Cause I'm getting as low as 0.27 on CV (with&nbsp;different holdout strategies), but my best LB score is, well,&nbsp;crap.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58805,
      "author_name": "adjgiulio",
      "author_url": "",
      "post_date": "11/24/2014 19:00:22",
      "content": "<p>[quote=rcarson;58803]</p>\n<p>[quote=Inspector;58801]</p>\n<p>Are you using libFM out of curiosity?</p>\n<p>[/quote]</p>\n<p>no, we use the fm of 3 idiots' winning solution of the last CTR contest. On the shoulders of a giant &nbsp;:D</p>\n<p>[/quote]</p>\n<p>You move fast... :-)</p>\n<p>That's another reason why I asked the initial question. This competition is not incredibly different from Criteo. Not the same, but it is not unconcievable that some of the things that worked in Criteo would work here. So, I had/have the feeling many folks are already trying fairly advanced models/ensembles, yet the improvement is still low. It is true that this competition is not truly a week old, but at this time of the competition it is common to see much more &quot;spacing&quot; in the top. Instead top ranks are already separated only by 3rd decimals, despite evidence of many folks throwing many things at this problem.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58806,
      "author_name": "fchollet",
      "author_url": "",
      "post_date": "11/24/2014 19:05:05",
      "content": "<p>[quote=Giulio;58777]&nbsp;ensemble between XBG (LB .40) and a linear model&nbsp;(.396)&nbsp;[/quote]</p>\n<p>You mean gradient boosting applied to feature histograms? In validation, I've had down to 0.397 with that technique, and I haven't explored 1% of what's possible. As for linear models on sparse features, I assume they can easily go down to 0.390 (lowest I've had so far was 0.392).&nbsp;</p>\n<p>Clearly you aren't trying hard enough.</p>\n<p>Ensembling doesn't work? That's because your models aren't different enough. FYI gradient boosting on histograms and linear SGD over sparse features isn't actually much different (since the coefficients of the linear model will converge towards the histograms).</p>\n<p>I think this contest will end up around 0.37-0.38. There is plenty of room for improvement. Maybe some innovation will even necessary, though, rather than merely microwaving the algorithms that did well in the Criteo challenge.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58807,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "11/24/2014 19:21:03",
      "content": "<p>[quote=fchollet;58806]</p>\n<p>[quote=Giulio;58777]&nbsp;ensemble between XBG (LB .40) and a linear model&nbsp;(.396)&nbsp;[/quote]</p>\n<p>You mean gradient boosting applied to feature histograms?&nbsp;</p>\n<p>[/quote]</p>\n<p>Hi, what is the&nbsp;feature histograms actually? I have seen it for a couple of times. Thank you!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58808,
      "author_name": "adjgiulio",
      "author_url": "",
      "post_date": "11/24/2014 19:22:08",
      "content": "<p>[quote=fchollet;58806]</p>\n<p>Clearly you aren't trying hard enough.</p>\n<p>[/quote]</p>\n<p>LOL. I'm definitely not. That's kind of the first thing I said in my original post :-)</p>\n<p>[quote=fchollet;58806]</p>\n<p>Ensembling doesn't work? That's because your models aren't different enough. FYI gradient boosting on histograms and linear SGD over sparse features isn't actually much different (since the coefficients of the linear model will converge towards the histograms).</p>\n<p>[/quote]</p>\n<p>I'm using XGB on the encoded features. No histograms.&nbsp;<br>Can you elaborate on the convergence thing? I believe you but it's not intuitive for me...</p>\n<p>[quote=fchollet;58806]</p>\n<p>I think this contest will end up around 0.37-0.38. There is plenty of room for improvement. Maybe some innovation will even necessary, though, rather than merely microwaving the algorithms that did well in the Criteo challenge.</p>\n<p>[/quote]</p>\n<p>I hope it does. I'm confident it will. Good luck to you! ;-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58809,
      "author_name": "fchollet",
      "author_url": "",
      "post_date": "11/24/2014 19:32:41",
      "content": "<p>[quote=rcarson;58807]</p>\n<p>Hi, what is the&nbsp;feature histograms actually? I have seen it for a couple of times. Thank you!</p>\n<p>[/quote]</p>\n<p>You compute the clicks &amp; impressions histograms of feature values, then you encode each feature value as its average CTR (or some variation of it) over the training data. You lose information in the process, but it can make your model generalize better --maybe.</p>\n<p>[quote]</p>\n<p>Can you elaborate on the convergence thing? I believe you but it's not intuitive for me...</p>\n<p>[/quote]</p>\n<p>In a sparse linear model, you encode the occurrence of a feature value as a binary indicator in a specific dimension. The coefficient for that dimension in your model will converge towards the average CTR for that feature value, times some feature-specific weight. And if you add regularization it will converge towards a &quot;smoothed CTR&quot; value, eg. (some factor+clicks)/(impressions+some other factor).</p>\n<p>Roughly.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58823,
      "author_name": "inspector",
      "author_url": "",
      "post_date": "11/24/2014 22:24:39",
      "content": "<p>[quote=fchollet;58809]</p>\n<p>You compute the clicks &amp; impressions histograms of feature values, then you encode each feature value as its average CTR (or some variation of it) over the training data. You lose information in the process, but it can make your model generalize better --maybe.</p>\n<p>[/quote]</p>\n\n<p>A histogram here then simply means average CTR per feature? So if you do this for, say site_id, then each site_id gets replaced with its marginal CTR - where marginal refers to the fact we would be aggregating over all other variables...and we could of course consider interactions between multiple variables (replacing a given tuple with the empirical CTR)?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58839,
      "author_name": "fchollet",
      "author_url": "",
      "post_date": "11/25/2014 01:02:59",
      "content": "<p>[quote=Inspector;58823]</p>\n<p>A histogram here then simply means average CTR per feature? So if you do this for, say site_id, then each site_id gets replaced with its marginal CTR - where marginal refers to the fact we would be aggregating over all other variables...and we could of course consider interactions between multiple variables (replacing a given tuple with the empirical CTR)?</p>\n<p>[/quote]</p>\n<p>Yes, although you have to keep in mind that your encoding get increasingly less reliable as you apply it to histograms that contain fewer events. That's why a smoothed out version might be preferable.</p>\n<p>In its simplest form it would be a simple average CTR, yes. In the general case it could be any encoding that is a function of the histogram of a feature value: e = f(impressions(value), clicks(value)) in R**n</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58840,
      "author_name": "inspector",
      "author_url": "",
      "post_date": "11/25/2014 01:33:19",
      "content": "<p>[quote=fchollet;58839]</p>\n<p>Yes, although you have to keep in mind that your encoding get increasingly less reliable as you apply it to histograms that contain fewer events. That's why a smoothed out version might be preferable.</p>\n<p>In its simplest form it would be a simple average CTR, yes. In the general case it could be any encoding that is a function of the histogram of a feature value: e = f(impressions(value), clicks(value)) in R**n</p>\n<p>[/quote]</p>\n\n<p>Cool, thanks. I have never used the technique myself, but I suspect that using a shrinkage estimator (shrinking towards the grand mean, ala hierarchical mixed models in statistics) might be a nice solution.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59009,
      "author_name": "clustifier",
      "author_url": "",
      "post_date": "11/26/2014 14:19:55",
      "content": "<p>[quote=rcarson;58803]</p>\n<p>[quote=Inspector;58801]</p>\n<p>Are you using libFM out of curiosity?</p>\n<p>[/quote]</p>\n<p>no, we use the fm of 3 idiots' winning solution of the last CTR contest. On the shoulders of a giant &nbsp;:D</p>\n<p>[/quote]</p>\n<p>have you used it with&nbsp;gbdt features?</p>\n<p>When I've tried to use it I haven't added the&nbsp;gbdt feature and I've got much worse score compared to the benchmark.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59010,
      "author_name": "ivanlobov",
      "author_url": "",
      "post_date": "11/26/2014 14:55:14",
      "content": "<p>[quote=clustifier;59009]</p>\n<p>[quote=rcarson;58803]</p>\n<p>[quote=Inspector;58801]</p>\n<p>Are you using libFM out of curiosity?</p>\n<p>[/quote]</p>\n<p>no, we use the fm of 3 idiots' winning solution of the last CTR contest. On the shoulders of a giant &nbsp;:D</p>\n<p>[/quote]</p>\n<p>have you used it with&nbsp;gbdt features?</p>\n<p>When I've tried to use it I haven't added the&nbsp;gbdt feature and I've got much worse score compared to the benchmark.</p>\n<p>[/quote]</p>\n\n<p>Correct me if I'm wrong, but intuitively decision tree features could add something to the model if there would be any numerical features, as it was in criteo (because they can learn non-linear data from them). Here, if you have no numerical features and one-hot encoder for all categorical ones, additional features from trees shouldn't add much. What do you think?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59016,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "11/26/2014 16:34:06",
      "content": "<p>[quote=clustifier;59009]</p>\n<p>have you used it with&nbsp;gbdt features?</p>\n<p>When I've tried to use it I haven't added the&nbsp;gbdt feature and I've got much worse score compared to the benchmark.</p>\n<p>[/quote]</p>\n<p>I haven't used the gbdt yet. I just did random stuff and it looks good. Here is what I did and I have no idea which one helps or hurts</p>\n<p>1) all the features are treated as categorical features.</p>\n<p>2) frequent feature and infrequent features are not differentiated. I'm lazy.</p>\n<p>3) split &quot;hour&quot; to date+hour like what the benchmark did.</p>\n<p>And some parameter tuning I'm not ready to share now :D</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59017,
      "author_name": "simengy82",
      "author_url": "",
      "post_date": "11/26/2014 17:06:24",
      "content": "<p>I tried very similar to what rcarson&nbsp;did and used gbdt for training, but get very lame results (~ 0.51xxxx). Some thing must be wrong......</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59018,
      "author_name": "liamhuber",
      "author_url": "",
      "post_date": "11/26/2014 17:16:02",
      "content": "<p>[quote=Ivan Lobov;59010]</p>\n<p>[quote=clustifier;59009]</p>\n<p>[quote=rcarson;58803]</p>\n<p>[quote=Inspector;58801]</p>\n<p>Are you using libFM out of curiosity?</p>\n<p>[/quote]</p>\n<p>no, we use the fm of 3 idiots' winning solution of the last CTR contest. On the shoulders of a giant &nbsp;:D</p>\n<p>[/quote]</p>\n<p>have you used it with&nbsp;gbdt features?</p>\n<p>When I've tried to use it I haven't added the&nbsp;gbdt feature and I've got much worse score compared to the benchmark.</p>\n<p>[/quote]</p>\n<p>Correct me if I'm wrong, but intuitively decision tree features could add something to the model if there would be any numerical features, as it was in criteo (because they can learn non-linear data from them). Here, if you have no numerical features and one-hot encoder for all categorical ones, additional features from trees shouldn't add much. What do you think?</p>\n<p>[/quote]</p>\n<p>Shouldn't a decision tree be able to capture feature interaction better, even for categorical data, compared to a linear model? For instance, suppose you had three relevant features after one-hotting, A, B, and C, and wanted a positive result for A&amp;B&amp;C but a negative result else. The tree can capture that perfectly, but the linear model will return w_A + w_B + w_C + offset. The difference in the linear model between A&amp;B&amp;C and A&amp;B&amp;~C is only w_C&lt;1, but the difference between those in the tree can be 1, it can perfectly distinguish.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59020,
      "author_name": "superfan123",
      "author_url": "",
      "post_date": "11/26/2014 17:25:59",
      "content": "<p>Tree feature has nothing to do with numerical feature. They just add cross effect of most frequent features. Right?</p>\n<p>[quote=Ivan Lobov;59010]</p>\n<p>Correct me if I'm wrong, but intuitively decision tree features could add something to the model if there would be any numerical features, as it was in criteo (because they can learn non-linear data from them). Here, if you have no numerical features and one-hot encoder for all categorical ones, additional features from trees shouldn't add much. What do you think?</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59021,
      "author_name": "liamhuber",
      "author_url": "",
      "post_date": "11/26/2014 17:30:49",
      "content": "<p>[quote=superfan123;59020]</p>\n<p>Tree feature has nothing to do with numerical feature. They just add cross effect of most frequent features. Right?</p>\n<p>[/quote]</p>\n<p>I wouldn't quite say that. Even on a single numeric feature, a tree is fundamentally a different response (a heaviside function) from, say, a linear model (a straight line with some slope and intercept.) But yeah, they definitely can capture cross effects!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59024,
      "author_name": "ivanlobov",
      "author_url": "",
      "post_date": "11/26/2014 17:42:32",
      "content": "<p>[quote=LiamHuber;59018]</p>\n<p>Shouldn't a decision tree be able to capture feature interaction better, even for categorical data, compared to a linear model? For instance, suppose you had three relevant features after one-hotting, A, B, and C, and wanted a positive result for A&amp;B&amp;C but a negative result else. The tree can capture that perfectly, but the linear model will return w_A + w_B + w_C + offset. The difference in the linear model between A&amp;B&amp;C and A&amp;B&amp;~C is only w_C&lt;1, but the difference between those in the tree can be 1, it can perfectly distinguish.</p>\n<p>[/quote]</p>\n<p>You're very right! Linear model can capture the same things, but with feature interactions being used. Which is kind of slow if we're talking about 3-4 features dependecies. So using decision trees is probably a good idea :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59068,
      "author_name": "inspector",
      "author_url": "",
      "post_date": "11/27/2014 02:21:35",
      "content": "<p>[quote=rcarson;58803]</p>\n<p>no, we use the fm of 3 idiots' winning solution of the last CTR contest. On the shoulders of a giant &nbsp;:D</p>\n<p>[/quote]</p>\n\n<p>Hey Guys,</p>\n<p>I am curious if anyone has used LibFM successfully in any Kaggle Competition? Also, has anyone used the low rank interaction feature of VW - it sounds like it should replicate what LibFM does...</p>\n<p>https://github.com/JohnLangford/vowpal_wabbit/tree/master/demo/movielens</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "58777": "",
    "58779": "",
    "58780": "",
    "58781": "",
    "58782": "",
    "58783": "",
    "58785": "",
    "58786": "",
    "58787": "",
    "58788": "",
    "58790": "",
    "58791": "",
    "58792": "",
    "58794": "",
    "58796": "",
    "58799": "",
    "58801": "",
    "58802": "",
    "58803": "",
    "58804": "",
    "58805": "",
    "58806": "",
    "58807": "",
    "58808": "",
    "58809": "",
    "58823": "",
    "58839": "",
    "58840": "",
    "59009": "",
    "59010": "",
    "59016": "",
    "59017": "",
    "59018": "",
    "59020": "",
    "59021": "",
    "59024": "",
    "59068": ""
  },
  "source": "meta"
}