{
  "id": 1416,
  "title": "use of collaborative filtering methods ?",
  "url": "/competitions/WhatDoYouKnow/discussion/1416",
  "author_name": "",
  "post_date": "2012-03-01T09:10:44.647Z",
  "votes": null,
  "comment_count": 11,
  "views": 11186,
  "content": "<p>I wonder whether anybody has tried a <strong>collaborative filtering</strong> method (&quot;students who liked this question also like these other questions&quot;) based on singular value decomposition and got a better score than the Rasch model benchmark ?</p>\r\n<p>Personally, I tried using pyrsvd with some modifications (e.g. added a logistic function transformation) with different numbers of latent variables but I did not get an improvement over the benchmark. I have to admit though that I did\r\n<strong>not</strong> do a systematic determination of the learning rate, regularization parameters or the number of latent variables due to limited time available, I did however split these by track name as the benchmark model does.</p>\r\n<p>&nbsp;</p>",
  "messages": [
    {
      "id": "8775",
      "postDate": "03/01/2012 09:10:44",
      "content": "<p>I wonder whether anybody has tried a <strong>collaborative filtering</strong> method (&quot;students who liked this question also like these other questions&quot;) based on singular value decomposition and got a better score than the Rasch model benchmark ?</p>\r\n<p>Personally, I tried using pyrsvd with some modifications (e.g. added a logistic function transformation) with different numbers of latent variables but I did not get an improvement over the benchmark. I have to admit though that I did\r\n<strong>not</strong> do a systematic determination of the learning rate, regularization parameters or the number of latent variables due to limited time available, I did however split these by track name as the benchmark model does.</p>\r\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "8783",
      "postDate": "03/01/2012 16:33:01",
      "content": "<p>I implemented an SVD/collaborative filtering variant that scored better than the benchmark.&nbsp; Rather than segment the data by &quot;track&quot; or other criteria my implementation used bias corrections for tracks and a few other things.&nbsp; Like you I used a logistic\r\n filter.</p>\r\n<p>My best score for this method alone was: 0.25275</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "8788",
      "postDate": "03/01/2012 20:16:55",
      "content": "<p>Almost all of my submissions were based on matrix factorization (collaborative filtering) using the (user,question,outcome) triplets. I was seeing BCD's close to that of Yetiman's.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "8856",
      "postDate": "03/02/2012 13:36:11",
      "content": "<p>So, actually none of our submissions were based on collaborative filtering. Neil tried a couple times but never really spent enough time on it. Now we're going to have to go back and learn it appears. I don't feel so bad with 7th since we weren't even using\r\n the &quot;correct&quot; technique. We just used standard ol' lme4, gbm and glmnet.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "8888",
      "postDate": "03/02/2012 18:18:28",
      "content": "<p>I tried matrix factorization with user and question biases, optimized for logistic loss.<br>\r\nIt is implemented in the open source package MyMediaLite:</p>\r\n<p>https://github.com/zenogantner/MyMediaLite/blob/master/src/MyMediaLite/RatingPrediction/BiasedMatrixFactorization.cs</p>\r\n<p>I did not do a lot of tuning, and did not perform a lot task-specific tweaks to the input data.<br>\r\nThe best result was 0.25597, from an unweighted blend of several matrix factorizations.</p>\r\n<p>If there is some interest, I can prepare a blog post about the details ...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "8894",
      "postDate": "03/02/2012 20:12:20",
      "content": "<p>The lme4 3Pl model prediction equation looks very similar to the matrix factorization model with latent factors of size 3.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "8919",
      "postDate": "03/03/2012 09:13:22",
      "content": "<p>All my submissions are made with a Factorization Machine, which is a generic factorization model. My best scores 0.24597 (private) and 0.24515 (public) are a single model and use no ensembles. I will publish the software (libFM, version 1.3) soon on http://www.libfm.org/.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "8920",
      "postDate": "03/03/2012 10:34:38",
      "content": "<p>We used a Restricted Boltzmann Machine for collaborative filtering on the dataset. (what we did was an implenetation of this paper:&nbsp;http://www.cs.utoronto.ca/~hinton/absps/netflix.pdf</p>\r\n<p>It didn't work that well though, our final scores were 0.25380 public and 0.25464 private. We did this as a final project for our machine learning lecture, if anyone is interested we can publish our small writeup &#43; code.\r\n</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "8936",
      "postDate": "03/03/2012 22:42:09",
      "content": "<p>@mathso: I think that would be interesting.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "8971",
      "postDate": "03/05/2012 01:58:58",
      "content": "<p>Well, looks like you proved the appropriateness of your methods Steffen. Congrats!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "8975",
      "postDate": "03/05/2012 03:08:17",
      "content": "<p>@Steffan - I haven't looked at your factorization machine source code, but I just finished reading your papers.&nbsp; Nice work!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "9221",
      "postDate": "03/11/2012 10:30:53",
      "content": "<p>Thanks for all the replies !</p>\r\n<p>So I conclude that the idea of using a collaborative filtering method wasn't too bad and I could have beaten the benchmark by investigating more (which I still can do).\r\n</p>\r\n<p>Incidentally, I also came across the restrictive Boltzmann machine paper mentioned by mathso (but this being a spare time activity here I did not have the time to implement it let alone test the implementation...).\r\n</p>\r\n<p>And I will certainly have a look at the papers supporting libFM, congratulations Steffen !</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 8783,
      "author_name": "yetiman",
      "author_url": "",
      "post_date": "03/01/2012 16:33:01",
      "content": "<p>I implemented an SVD/collaborative filtering variant that scored better than the benchmark.&nbsp; Rather than segment the data by &quot;track&quot; or other criteria my implementation used bias corrections for tracks and a few other things.&nbsp; Like you I used a logistic\r\n filter.</p>\r\n<p>My best score for this method alone was: 0.25275</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 8788,
      "author_name": "tritonsd",
      "author_url": "",
      "post_date": "03/01/2012 20:16:55",
      "content": "<p>Almost all of my submissions were based on matrix factorization (collaborative filtering) using the (user,question,outcome) triplets. I was seeing BCD's close to that of Yetiman's.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 8856,
      "author_name": "biasvariance",
      "author_url": "",
      "post_date": "03/02/2012 13:36:11",
      "content": "<p>So, actually none of our submissions were based on collaborative filtering. Neil tried a couple times but never really spent enough time on it. Now we're going to have to go back and learn it appears. I don't feel so bad with 7th since we weren't even using\r\n the &quot;correct&quot; technique. We just used standard ol' lme4, gbm and glmnet.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 8888,
      "author_name": "zenog15462",
      "author_url": "",
      "post_date": "03/02/2012 18:18:28",
      "content": "<p>I tried matrix factorization with user and question biases, optimized for logistic loss.<br>\r\nIt is implemented in the open source package MyMediaLite:</p>\r\n<p>https://github.com/zenogantner/MyMediaLite/blob/master/src/MyMediaLite/RatingPrediction/BiasedMatrixFactorization.cs</p>\r\n<p>I did not do a lot of tuning, and did not perform a lot task-specific tweaks to the input data.<br>\r\nThe best result was 0.25597, from an unweighted blend of several matrix factorizations.</p>\r\n<p>If there is some interest, I can prepare a blog post about the details ...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 8894,
      "author_name": "tritonsd",
      "author_url": "",
      "post_date": "03/02/2012 20:12:20",
      "content": "<p>The lme4 3Pl model prediction equation looks very similar to the matrix factorization model with latent factors of size 3.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 8919,
      "author_name": "steffenrendle",
      "author_url": "",
      "post_date": "03/03/2012 09:13:22",
      "content": "<p>All my submissions are made with a Factorization Machine, which is a generic factorization model. My best scores 0.24597 (private) and 0.24515 (public) are a single model and use no ensembles. I will publish the software (libFM, version 1.3) soon on http://www.libfm.org/.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 8920,
      "author_name": "mathso",
      "author_url": "",
      "post_date": "03/03/2012 10:34:38",
      "content": "<p>We used a Restricted Boltzmann Machine for collaborative filtering on the dataset. (what we did was an implenetation of this paper:&nbsp;http://www.cs.utoronto.ca/~hinton/absps/netflix.pdf</p>\r\n<p>It didn't work that well though, our final scores were 0.25380 public and 0.25464 private. We did this as a final project for our machine learning lecture, if anyone is interested we can publish our small writeup &#43; code.\r\n</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 8936,
      "author_name": "zenog15462",
      "author_url": "",
      "post_date": "03/03/2012 22:42:09",
      "content": "<p>@mathso: I think that would be interesting.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 8971,
      "author_name": "biasvariance",
      "author_url": "",
      "post_date": "03/05/2012 01:58:58",
      "content": "<p>Well, looks like you proved the appropriateness of your methods Steffen. Congrats!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 8975,
      "author_name": "yetiman",
      "author_url": "",
      "post_date": "03/05/2012 03:08:17",
      "content": "<p>@Steffan - I haven't looked at your factorization machine source code, but I just finished reading your papers.&nbsp; Nice work!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 9221,
      "author_name": "holzner",
      "author_url": "",
      "post_date": "03/11/2012 10:30:53",
      "content": "<p>Thanks for all the replies !</p>\r\n<p>So I conclude that the idea of using a collaborative filtering method wasn't too bad and I could have beaten the benchmark by investigating more (which I still can do).\r\n</p>\r\n<p>Incidentally, I also came across the restrictive Boltzmann machine paper mentioned by mathso (but this being a spare time activity here I did not have the time to implement it let alone test the implementation...).\r\n</p>\r\n<p>And I will certainly have a look at the papers supporting libFM, congratulations Steffen !</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "8775": "",
    "8783": "",
    "8788": "",
    "8856": "",
    "8888": "",
    "8894": "",
    "8919": "",
    "8920": "",
    "8936": "",
    "8971": "",
    "8975": "",
    "9221": ""
  },
  "source": "meta"
}