{
  "id": 315,
  "title": "What is the evaluation method?",
  "url": "/competitions/WIC2011/discussion/315",
  "author_name": "",
  "post_date": "2011-03-02T02:10:32Z",
  "votes": null,
  "comment_count": 15,
  "views": 8449,
  "content": "What is the evaluation method?&nbsp; The other contests describe that.<br mce_bogus=\"1\">",
  "messages": [
    {
      "id": "1846",
      "postDate": "03/02/2011 02:10:32",
      "content": "What is the evaluation method?&nbsp; The other contests describe that.<br mce_bogus=\"1\">",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1847",
      "postDate": "03/02/2011 02:10:32",
      "content": "Mean Absolute Error (MAE) is used as an evaluation metric.<br>I thought when an evaluation metric is chosen, a default page describing it will be automatically added but it does not seem to be the case.<br mce_bogus=\"1\">",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1857",
      "postDate": "03/02/2011 02:10:32",
      "content": "Mean absolute error of what? Of the true class posterior (a 0-1 vector) and the predicted class posterior?<div><br></div><div>The evaluation measure appears to have a strange bug. I just uploaded an extremely simple baseline (all-zero posterior matrix), and got an MAE of 0?! I was expecting to get an MAE of 1/55.</div>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1858",
      "postDate": "03/02/2011 02:10:32",
      "content": "The leaderboard now changed. I'm still on top of the list, but my MAE has gone up to 3860?! And my submission date is 1 Jan 1970?!<div><br></div><div>Looks like something's significantly messed up...</div>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1859",
      "postDate": "03/02/2011 02:10:32",
      "content": "Anthony will correct me if I am wrong but MAE is the average absolute difference compuited cell by cell between the solution and the table you have sent.<br>Sending an all zero table will obviously give you a good score because test images are written by just one writer !<br>I am the competition host, but there is nothing I can do to correct the bug.<br>I will mail Anthony now, and hopefully, he will get back to us tomorrow.<br mce_bogus=\"1\">",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1869",
      "postDate": "03/02/2011 02:10:32",
      "content": "Ok, Anthony fixed the bug. The leaderboard is working fine again.",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1871",
      "postDate": "03/02/2011 02:10:32",
      "content": "The bug is not fixed.  I can assure you I did not get a 0.  There was something wrong with the way my entry was submitted.<br mce_bogus=\"1\">",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1883",
      "postDate": "03/02/2011 02:10:32",
      "content": "i will echo the concerns above. i submitted a constant (c=0.1) matrix, and since each doc has exactly one writer, i could compute the MAE i should get.<div><br></div><div>with 53 test cases and 54 authors, the error should be something like ((1/54)*53*53+(53/54)*53*1)/(54*53)=0.03635117.</div><div><br></div><div>however, kaggle reports 0.0693122.</div><div><br></div><div>i suspect that this is a distinct issue from the bug that was affecting kaggle.</div><div><div><br></div><div>hopefully when the data is augmented we can also get a firm verified description of the penalty. i also think it would make more sense to go with the mean over the MAE for each test case (as van der Maaten described), as opposed to the MAE of the entire grid, but whatever. as long as it's defined and consistent...</div></div>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1884",
      "postDate": "03/02/2011 02:10:32",
      "content": "Kaggle displays the <u>public </u>leaderboard which is computed on just a part of the test set !<br>Getting even 0 on this partial test does not mean you have 0 on the whole test set.<br mce_bogus=\"1\">",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1888",
      "postDate": "03/02/2011 02:10:32",
      "content": "The Public MAE given on the Submissions page for my 2nd submission is still \"Scoring...\" instead of an actual number like 0 which is reported on the Leaderboard page.  For my first submission 0.238571 is reported as was also given on the leaderboard so that is correct.  At least in my case I contend something did go wrong in the scoring process.  Perhaps I did get a 0 and the submission page just needs to reflect this.",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1891",
      "postDate": "03/02/2011 02:10:32",
      "content": "even on the public leaderboard, the MAE for any positive constant matrix, if computed as claimed, should be 0.03635117=<meta charset=\"utf-8\"><span class=\"Apple-style-span\" style=\"border-collapse: collapse; color: rgb(51, 51, 51); font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: 12px; line-height: 16px; \">(1/54)*53*J+(53/54)*1*J)/(54*J), which isn't what i get. also, the MAE for my first submission increased by a factor of 2.07 overnight, which is a little bit odd.</span>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1892",
      "postDate": "03/02/2011 02:10:32",
      "content": "Entries made before we fixed the leaderboard were scored incorrectly. I have now re-scored the relevant entries. The error was the fault of Kaggle and not the competition organizers.&nbsp;<div><div><br></div><div>Apologies!</div></div><div><br></div><div>Anthony</div>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1901",
      "postDate": "03/02/2011 02:10:32",
      "content": "Just to be sure I understood it correctly: the true matrix is a zero-one matrix in which each row has just a single one?",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1902",
      "postDate": "03/02/2011 02:10:32",
      "content": "Yes.<br mce_bogus=\"1\">",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1903",
      "postDate": "03/02/2011 02:10:32",
      "content": "Are you scoring the cells for the writers that aren't the author of the document? It seems strange that someone could get a great score just by uploading a file with nothing but zeros.",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "1904",
      "postDate": "03/02/2011 02:10:32",
      "content": "All the cells are taken into account in the evaluation.<br>We used for this edition the evaluation metric that is available in Kaggle.<br>Hopefully, there will be other editions of the contest and we will come up with other evaluation metrics.<br>For now, please do not consider any score as great if it is not better than the benchmark.<br><br mce_bogus=\"1\">",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1847,
      "author_name": "ahassaine",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "Mean Absolute Error (MAE) is used as an evaluation metric.<br>I thought when an evaluation metric is chosen, a default page describing it will be automatically added but it does not seem to be the case.<br mce_bogus=\"1\">",
      "votes": null,
      "replies": []
    },
    {
      "id": 1857,
      "author_name": "lvdmaaten",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "Mean absolute error of what? Of the true class posterior (a 0-1 vector) and the predicted class posterior?<div><br></div><div>The evaluation measure appears to have a strange bug. I just uploaded an extremely simple baseline (all-zero posterior matrix), and got an MAE of 0?! I was expecting to get an MAE of 1/55.</div>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1858,
      "author_name": "lvdmaaten",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "The leaderboard now changed. I'm still on top of the list, but my MAE has gone up to 3860?! And my submission date is 1 Jan 1970?!<div><br></div><div>Looks like something's significantly messed up...</div>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1859,
      "author_name": "ahassaine",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "Anthony will correct me if I am wrong but MAE is the average absolute difference compuited cell by cell between the solution and the table you have sent.<br>Sending an all zero table will obviously give you a good score because test images are written by just one writer !<br>I am the competition host, but there is nothing I can do to correct the bug.<br>I will mail Anthony now, and hopefully, he will get back to us tomorrow.<br mce_bogus=\"1\">",
      "votes": null,
      "replies": []
    },
    {
      "id": 1869,
      "author_name": "lvdmaaten",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "Ok, Anthony fixed the bug. The leaderboard is working fine again.",
      "votes": null,
      "replies": []
    },
    {
      "id": 1871,
      "author_name": "gregwerner",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "The bug is not fixed.  I can assure you I did not get a 0.  There was something wrong with the way my entry was submitted.<br mce_bogus=\"1\">",
      "votes": null,
      "replies": []
    },
    {
      "id": 1883,
      "author_name": "tteravai",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "i will echo the concerns above. i submitted a constant (c=0.1) matrix, and since each doc has exactly one writer, i could compute the MAE i should get.<div><br></div><div>with 53 test cases and 54 authors, the error should be something like ((1/54)*53*53+(53/54)*53*1)/(54*53)=0.03635117.</div><div><br></div><div>however, kaggle reports 0.0693122.</div><div><br></div><div>i suspect that this is a distinct issue from the bug that was affecting kaggle.</div><div><div><br></div><div>hopefully when the data is augmented we can also get a firm verified description of the penalty. i also think it would make more sense to go with the mean over the MAE for each test case (as van der Maaten described), as opposed to the MAE of the entire grid, but whatever. as long as it's defined and consistent...</div></div>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1884,
      "author_name": "ahassaine",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "Kaggle displays the <u>public </u>leaderboard which is computed on just a part of the test set !<br>Getting even 0 on this partial test does not mean you have 0 on the whole test set.<br mce_bogus=\"1\">",
      "votes": null,
      "replies": []
    },
    {
      "id": 1888,
      "author_name": "gregwerner",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "The Public MAE given on the Submissions page for my 2nd submission is still \"Scoring...\" instead of an actual number like 0 which is reported on the Leaderboard page.  For my first submission 0.238571 is reported as was also given on the leaderboard so that is correct.  At least in my case I contend something did go wrong in the scoring process.  Perhaps I did get a 0 and the submission page just needs to reflect this.",
      "votes": null,
      "replies": []
    },
    {
      "id": 1891,
      "author_name": "tteravai",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "even on the public leaderboard, the MAE for any positive constant matrix, if computed as claimed, should be 0.03635117=<meta charset=\"utf-8\"><span class=\"Apple-style-span\" style=\"border-collapse: collapse; color: rgb(51, 51, 51); font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: 12px; line-height: 16px; \">(1/54)*53*J+(53/54)*1*J)/(54*J), which isn't what i get. also, the MAE for my first submission increased by a factor of 2.07 overnight, which is a little bit odd.</span>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1892,
      "author_name": "antgoldbloom",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "Entries made before we fixed the leaderboard were scored incorrectly. I have now re-scored the relevant entries. The error was the fault of Kaggle and not the competition organizers.&nbsp;<div><div><br></div><div>Apologies!</div></div><div><br></div><div>Anthony</div>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1901,
      "author_name": "lvdmaaten",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "Just to be sure I understood it correctly: the true matrix is a zero-one matrix in which each row has just a single one?",
      "votes": null,
      "replies": []
    },
    {
      "id": 1902,
      "author_name": "ahassaine",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "Yes.<br mce_bogus=\"1\">",
      "votes": null,
      "replies": []
    },
    {
      "id": 1903,
      "author_name": "cerin5111",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "Are you scoring the cells for the writers that aren't the author of the document? It seems strange that someone could get a great score just by uploading a file with nothing but zeros.",
      "votes": null,
      "replies": []
    },
    {
      "id": 1904,
      "author_name": "ahassaine",
      "author_url": "",
      "post_date": "03/02/2011 02:10:32",
      "content": "All the cells are taken into account in the evaluation.<br>We used for this edition the evaluation metric that is available in Kaggle.<br>Hopefully, there will be other editions of the contest and we will come up with other evaluation metrics.<br>For now, please do not consider any score as great if it is not better than the benchmark.<br><br mce_bogus=\"1\">",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1846": "",
    "1847": "",
    "1857": "",
    "1858": "",
    "1859": "",
    "1869": "",
    "1871": "",
    "1883": "",
    "1884": "",
    "1888": "",
    "1891": "",
    "1892": "",
    "1901": "",
    "1902": "",
    "1903": "",
    "1904": ""
  },
  "source": "meta"
}