{
  "id": 6992,
  "title": "leaderboard evaluation details",
  "url": "/competitions/galaxy-zoo-the-galaxy-challenge/discussion/6992",
  "author_name": "",
  "post_date": "2014-02-03T01:08:24.727Z",
  "votes": null,
  "comment_count": 8,
  "views": 1759,
  "content": "<p>Should contestants be aiming to produce evaluations that match the distribution of the &quot;citizen scientists&quot; responses for each level in the decision tree or simply trying to pick the most probable classification?</p>\n<p>ie.</p>\n<p>Lets say for a given image, puny humans would have decided according the following distribution:</p>\n<p>Class1.1,Class1.2,Class1.3</p>\n<p>.5, .25,.25</p>\n<p>Is it preferred that my robot tried to predict that distribution or more simply what it thought was the likely answer, say:</p>\n<p>Class1.1,Class1.2,Class1.3</p>\n<p>1., 0., 0.</p>\n<p>Is the goal of this competition to predict how many people would have got it wrong as well as what the most popular classification was?</p>\n<p>&nbsp;</p>",
  "messages": [
    {
      "id": "38289",
      "postDate": "02/03/2014 01:08:24",
      "content": "<p>Should contestants be aiming to produce evaluations that match the distribution of the &quot;citizen scientists&quot; responses for each level in the decision tree or simply trying to pick the most probable classification?</p>\n<p>ie.</p>\n<p>Lets say for a given image, puny humans would have decided according the following distribution:</p>\n<p>Class1.1,Class1.2,Class1.3</p>\n<p>.5, .25,.25</p>\n<p>Is it preferred that my robot tried to predict that distribution or more simply what it thought was the likely answer, say:</p>\n<p>Class1.1,Class1.2,Class1.3</p>\n<p>1., 0., 0.</p>\n<p>Is the goal of this competition to predict how many people would have got it wrong as well as what the most popular classification was?</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "38294",
      "postDate": "02/03/2014 06:56:23",
      "content": "<p>Almost the exact same question was posed and answered here:&nbsp;<a href=\"https://www.kaggle.com/c/galaxy-zoo-the-galaxy-challenge/forums/t/6939/goal-of-competition\">https://www.kaggle.com/c/galaxy-zoo-the-galaxy-challenge/forums/t/6939/goal-of-competition</a></p>\n<p>The task is to predict the same scores as from the human classifiers, since this is the available data.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "38309",
      "postDate": "02/03/2014 16:55:58",
      "content": "<p>No that isn't the same question. Yes I understand the goal is to predict how the human classifiers would likely have classified the image. The question is do I need to also predict the distribution of drunk / wrong / clumsy humans (for a better score)?</p>\n<p>[edit]</p>\n<p>I guess I could do a quick test against the leaderboard and see which method scores higher...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "38312",
      "postDate": "02/03/2014 17:56:22",
      "content": "<p>You have to predict probabilities, thus RMSE as the evaluation metric. If you think that 30% of the humans answered A to question 1, that 30% answered B and that 40% answered C to the first question, then your first three outputs should be: 0.3, 0.3, 0.4</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "38314",
      "postDate": "02/03/2014 18:02:11",
      "content": "<p>Oh I see, that's a little more complicated then. Thank you though.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "38315",
      "postDate": "02/03/2014 18:26:41",
      "content": "<p>Well, I think that actually depends on your approach to the problem. With some approaches (my initial approach) this actually makes the problem a bit simpler.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "38317",
      "postDate": "02/03/2014 18:44:55",
      "content": "<p>You don't happen to know a method to tease out a prediction confidence metric from a Neural Network do you? :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "38318",
      "postDate": "02/03/2014 18:46:48",
      "content": "<p>Well that very much depends on the activation functions in your network. If you play a bit around with softmax-layers and sigmoid activation functions, you might get exactly what you are looking for.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "38323",
      "postDate": "02/03/2014 20:27:02",
      "content": "<p>Yes I see a way that could work. hmm....</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 38294,
      "author_name": "utdiscant",
      "author_url": "",
      "post_date": "02/03/2014 06:56:23",
      "content": "<p>Almost the exact same question was posed and answered here:&nbsp;<a href=\"https://www.kaggle.com/c/galaxy-zoo-the-galaxy-challenge/forums/t/6939/goal-of-competition\">https://www.kaggle.com/c/galaxy-zoo-the-galaxy-challenge/forums/t/6939/goal-of-competition</a></p>\n<p>The task is to predict the same scores as from the human classifiers, since this is the available data.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 38309,
      "author_name": "tanstaafl",
      "author_url": "",
      "post_date": "02/03/2014 16:55:58",
      "content": "<p>No that isn't the same question. Yes I understand the goal is to predict how the human classifiers would likely have classified the image. The question is do I need to also predict the distribution of drunk / wrong / clumsy humans (for a better score)?</p>\n<p>[edit]</p>\n<p>I guess I could do a quick test against the leaderboard and see which method scores higher...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 38312,
      "author_name": "utdiscant",
      "author_url": "",
      "post_date": "02/03/2014 17:56:22",
      "content": "<p>You have to predict probabilities, thus RMSE as the evaluation metric. If you think that 30% of the humans answered A to question 1, that 30% answered B and that 40% answered C to the first question, then your first three outputs should be: 0.3, 0.3, 0.4</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 38314,
      "author_name": "tanstaafl",
      "author_url": "",
      "post_date": "02/03/2014 18:02:11",
      "content": "<p>Oh I see, that's a little more complicated then. Thank you though.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 38315,
      "author_name": "utdiscant",
      "author_url": "",
      "post_date": "02/03/2014 18:26:41",
      "content": "<p>Well, I think that actually depends on your approach to the problem. With some approaches (my initial approach) this actually makes the problem a bit simpler.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 38317,
      "author_name": "tanstaafl",
      "author_url": "",
      "post_date": "02/03/2014 18:44:55",
      "content": "<p>You don't happen to know a method to tease out a prediction confidence metric from a Neural Network do you? :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 38318,
      "author_name": "utdiscant",
      "author_url": "",
      "post_date": "02/03/2014 18:46:48",
      "content": "<p>Well that very much depends on the activation functions in your network. If you play a bit around with softmax-layers and sigmoid activation functions, you might get exactly what you are looking for.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 38323,
      "author_name": "tanstaafl",
      "author_url": "",
      "post_date": "02/03/2014 20:27:02",
      "content": "<p>Yes I see a way that could work. hmm....</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "38289": "",
    "38294": "",
    "38309": "",
    "38312": "",
    "38314": "",
    "38315": "",
    "38317": "",
    "38318": "",
    "38323": ""
  },
  "source": "meta"
}