{ "id": 4293, "title": "Welcome!", "url": "/competitions/challenges-in-representation-learning-multi-modal-learning/discussion/4293", "author_name": "", "post_date": "2013-04-12T23:33:33.047Z", "votes": null, "comment_count": 7, "views": 2760, "content": "

Welcome to the challenge from the workshop organizers. I will be watching the forum if you have any questions. Yoshua, Dumitru, and I will also respond to e-mail at icml2013WREPL@gmail.com . We hope you have fun and get impressive results!

", "messages": [ { "id": "22726", "postDate": "04/12/2013 23:33:33", "content": "

Welcome to the challenge from the workshop organizers. I will be watching the forum if you have any questions. Yoshua, Dumitru, and I will also respond to e-mail at icml2013WREPL@gmail.com . We hope you have fun and get impressive results!

", "rawMarkdown": "", "votes": null }, { "id": "22954", "postDate": "04/17/2013 11:18:52", "content": "

I was considering having a go at this, but it looks like the problem has essentially been solved since two teams have already achieved an AUC of 1.0. I don't want to diminish the efforts of the competitors, but I'm finding these results a bit hard to believe\r\n :) Is there something wrong with the scoring mechanism?

", "rawMarkdown": "", "votes": null }, { "id": "22958", "postDate": "04/17/2013 12:26:50", "content": "

No, there's nothing wrong with the scoring mechanism. I've checked the submitted .csv files by hand and they do indeed label the test data correctly. Everyone, including RBM, is a bit surprised at the results but they don't seem to be a bug.

", "rawMarkdown": "", "votes": null }, { "id": "22961", "postDate": "04/17/2013 13:08:36", "content": "

This is surprising to me too, as I recall RBM and I had a similar score for a short time beforehand. I'm tempted to also try and get an AUC of 1 but at this point it seems that effort would be better spent on the other challenges.

", "rawMarkdown": "", "votes": null }, { "id": "22963", "postDate": "04/17/2013 13:18:27", "content": "

ryank, I feel the same way.

\r\n

Of course, one unpleasant possibility is that these submissions were hand labeled. There are only 1000 testing example to label (if I understand correctly), so it's not a big deal for a human to hand label\r\n them and submit a perfect score. 

\r\n

I am not suggesting that the Kagglers at the top of the leaderboard did this, but only remarking that if that WERE the case, it would be a shame that no one else is bothering to solve the problem since it seems\r\n like it has been solved already.

", "rawMarkdown": "", "votes": null }, { "id": "22981", "postDate": "04/17/2013 15:39:06", "content": "

We haven't yet fully verified the submissions but I think they are legitimate. I can't elaborate on why right now because I don't want to disclose details of how other teams' submissions work. I'll announce more when we've verified the submissions and decided\r\n what to do.

", "rawMarkdown": "", "votes": null }, { "id": "22990", "postDate": "04/17/2013 17:49:09", "content": "

Fair enough, kudos to them for figuring it out then! In that case I'm a bit surprised by the poor performance of the baselines :)

", "rawMarkdown": "", "votes": null }, { "id": "22991", "postDate": "04/17/2013 17:53:43", "content": "

Follow up: only one of them is an actual entry in the contest. The other is hand labels that the user didn't intend to have entered in the final contest. Since there are two prizes, it's still worth seeing if you can figure out how to get 100%.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 22954, "author_name": "sedielem", "author_url": "", "post_date": "04/17/2013 11:18:52", "content": "

I was considering having a go at this, but it looks like the problem has essentially been solved since two teams have already achieved an AUC of 1.0. I don't want to diminish the efforts of the competitors, but I'm finding these results a bit hard to believe\r\n :) Is there something wrong with the scoring mechanism?

", "votes": null, "replies": [] }, { "id": 22958, "author_name": "iangoodfellow", "author_url": "", "post_date": "04/17/2013 12:26:50", "content": "

No, there's nothing wrong with the scoring mechanism. I've checked the submitted .csv files by hand and they do indeed label the test data correctly. Everyone, including RBM, is a bit surprised at the results but they don't seem to be a bug.

", "votes": null, "replies": [] }, { "id": 22961, "author_name": "ryankiros", "author_url": "", "post_date": "04/17/2013 13:08:36", "content": "

This is surprising to me too, as I recall RBM and I had a similar score for a short time beforehand. I'm tempted to also try and get an AUC of 1 but at this point it seems that effort would be better spent on the other challenges.

", "votes": null, "replies": [] }, { "id": 22963, "author_name": "nitaidean", "author_url": "", "post_date": "04/17/2013 13:18:27", "content": "

ryank, I feel the same way.

\r\n

Of course, one unpleasant possibility is that these submissions were hand labeled. There are only 1000 testing example to label (if I understand correctly), so it's not a big deal for a human to hand label\r\n them and submit a perfect score. 

\r\n

I am not suggesting that the Kagglers at the top of the leaderboard did this, but only remarking that if that WERE the case, it would be a shame that no one else is bothering to solve the problem since it seems\r\n like it has been solved already.

", "votes": null, "replies": [] }, { "id": 22981, "author_name": "iangoodfellow", "author_url": "", "post_date": "04/17/2013 15:39:06", "content": "

We haven't yet fully verified the submissions but I think they are legitimate. I can't elaborate on why right now because I don't want to disclose details of how other teams' submissions work. I'll announce more when we've verified the submissions and decided\r\n what to do.

", "votes": null, "replies": [] }, { "id": 22990, "author_name": "sedielem", "author_url": "", "post_date": "04/17/2013 17:49:09", "content": "

Fair enough, kudos to them for figuring it out then! In that case I'm a bit surprised by the poor performance of the baselines :)

", "votes": null, "replies": [] }, { "id": 22991, "author_name": "iangoodfellow", "author_url": "", "post_date": "04/17/2013 17:53:43", "content": "

Follow up: only one of them is an actual entry in the contest. The other is hand labels that the user didn't intend to have entered in the final contest. Since there are two prizes, it's still worth seeing if you can figure out how to get 100%.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "22726": "", "22954": "", "22958": "", "22961": "", "22963": "", "22981": "", "22990": "", "22991": "" }, "source": "meta" }