{
  "id": 897,
  "title": "Is this one too easy ?",
  "url": "/competitions/SemiSupervisedFeatureLearning/discussion/897",
  "author_name": "",
  "post_date": "2011-09-28T22:14:41.490Z",
  "votes": null,
  "comment_count": 2,
  "views": 1984,
  "content": "<p>So the benchmark score is .980, and someone already got to .993 on the leaderboard now. I'm thinking any serious effort will get you above .995 (or even .999), and any difference after that won't be&nbsp;statistically significant any more.</p>\r\n<p>Anyway, what's the benchmark algorithm ?</p>",
  "messages": [
    {
      "id": "5724",
      "postDate": "09/28/2011 22:14:41",
      "content": "<p>So the benchmark score is .980, and someone already got to .993 on the leaderboard now. I'm thinking any serious effort will get you above .995 (or even .999), and any difference after that won't be&nbsp;statistically significant any more.</p>\r\n<p>Anyway, what's the benchmark algorithm ?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "5725",
      "postDate": "09/28/2011 22:22:56",
      "content": "<p>How are you calculating statistical significance? At AUC=0.995; n=50000, you can think of this (roughly) as 250 incorrectly classified cases. That still has a very low probability of 2 people ordering them the same way by chance.</p>\r\n<p>The benchmark algorithm is described in the comp pages as being based on similarity to a set of clusters found using a fast K-means algorithm.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "5726",
      "postDate": "09/28/2011 22:39:51",
      "content": "<p>As mentioned on another thread, we won't be surprised to see AUC values above 0.999x or higher by the end of the competition. &nbsp;One useful way to think about this is to look at the &quot;AUC Loss&quot;, or (1-AUC), which shows the probability of making a pairwise comparison\r\n error. &nbsp;Moving from from 0.993 to 0.999, for example, would be a large reduction in pairwise comparison error rate.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 5725,
      "author_name": "jhoward",
      "author_url": "",
      "post_date": "09/28/2011 22:22:56",
      "content": "<p>How are you calculating statistical significance? At AUC=0.995; n=50000, you can think of this (roughly) as 250 incorrectly classified cases. That still has a very low probability of 2 people ordering them the same way by chance.</p>\r\n<p>The benchmark algorithm is described in the comp pages as being based on similarity to a set of clusters found using a fast K-means algorithm.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 5726,
      "author_name": "argv",
      "author_url": "",
      "post_date": "09/28/2011 22:39:51",
      "content": "<p>As mentioned on another thread, we won't be surprised to see AUC values above 0.999x or higher by the end of the competition. &nbsp;One useful way to think about this is to look at the &quot;AUC Loss&quot;, or (1-AUC), which shows the probability of making a pairwise comparison\r\n error. &nbsp;Moving from from 0.993 to 0.999, for example, would be a large reduction in pairwise comparison error rate.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "5724": "",
    "5725": "",
    "5726": ""
  },
  "source": "meta"
}