{ "id": 12410, "title": "rules clarification please - \"human prediction\"?", "url": "/competitions/avazu-ctr-prediction/discussion/12410", "author_name": "", "post_date": "2015-02-04T21:38:25.263Z", "votes": null, "comment_count": 5, "views": 1205, "content": "

Under the heading "Submissions", the rules state:

\n

Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.

\n

What constitutes human prediction? If, for instance, I create a formula composed of a certain number of factors and decide to exclude one or more of them entirely or in part for various reasons (relatively small correlation with clicks in the training set, small number of records falling into one or more of the values for the factor, I just don't think certain factors would help, etc.), would that be a violation? What would clearly violate of this rule?

\n

Thanks,

\n

Ted

", "messages": [ { "id": "63611", "postDate": "02/04/2015 21:38:25", "content": "

Under the heading "Submissions", the rules state:

\n

Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.

\n

What constitutes human prediction? If, for instance, I create a formula composed of a certain number of factors and decide to exclude one or more of them entirely or in part for various reasons (relatively small correlation with clicks in the training set, small number of records falling into one or more of the values for the factor, I just don't think certain factors would help, etc.), would that be a violation? What would clearly violate of this rule?

\n

Thanks,

\n

Ted

", "rawMarkdown": "", "votes": null }, { "id": "63612", "postDate": "02/04/2015 22:01:34", "content": "

The idea behind a good contest is that it produces models and approaches that generalize well to new data. This rule helps with that. A clear violation of this rule would be hand labeling a sentiment analysis test set or image recognition task: these are relatively easy for humans to label, and require more advanced algo's for computers to automatically classify. Those datasets are usually labeled by humans in the first place, so you will just replicate that (if file is "test14.jpg" then prediction is "horse"). In short human experts labeling is not machine learning. Writing formulas with correlations is.

\n

For this challenge this rule is a bit moot. It would be neigh impossible to hand label (I don't know any expert capable of that) and there are far too many samples (even Mechanical Turk would be expensive). If your method performs on unseen data then it is usually ok, in regards to this specific rule.

", "rawMarkdown": "", "votes": null }, { "id": "63614", "postDate": "02/04/2015 22:13:04", "content": "

Thanks Triskelion, that helps a lot!

", "rawMarkdown": "", "votes": null }, { "id": "63616", "postDate": "02/04/2015 22:17:24", "content": "

A small aside: this rule also covers hand labeling _some_ test samples and adding these labeled samples to the train set AKA fully supervised learning.

", "rawMarkdown": "", "votes": null }, { "id": "63657", "postDate": "02/05/2015 20:38:17", "content": "

>>A small aside: this rule also covers hand labeling _some_ test samples and adding these labeled samples to the train set AKA fully supervised learning.<<

\n

Thanks again. I'm not sure why anyone would want to that, however, even if it were allowed as:

\n

(1) The test data does not include clicks

\n

and

\n

(2) The final judging will be on different rows than the ones currently determining the Leaderboard rankings.

\n

But perhaps I am not missing something.

", "rawMarkdown": "", "votes": null }, { "id": "63662", "postDate": "02/05/2015 22:12:43", "content": "

[quote=ComputerGuyTed;63657]

\n

>>A small aside: this rule also covers hand labeling _some_ test samples and adding these labeled samples to the train set AKA fully supervised learning.<<

\n

Thanks again. I'm not sure why anyone would want to that, however, even if it were allowed as:

\n

(1) The test data does not include clicks

\n

and

\n

(2) The final judging will be on different rows than the ones currently determining the Leaderboard rankings.

\n

But perhaps I am not missing something.

\n

[/quote]

\n

Why would anyone want to do that? Well, because it can give you a significant boost in your leaderboard score. But it's a form of cheating -- training and testing on (some of) the same data doesn't make sense conceptually as your algorithm would be memorizing, not generalizing. It would do well on the test set but perform poorly on unseen data.

\n

As Triskelion said, hand-labeling is not practical in this competition. But there are others where it's feasible. For instance, there was a competition where the goal was to distinguish pictures of cats from pictures of dogs. Hand-labeling the test set would be easy for a human in that case.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 63612, "author_name": "triskelion", "author_url": "", "post_date": "02/04/2015 22:01:34", "content": "

The idea behind a good contest is that it produces models and approaches that generalize well to new data. This rule helps with that. A clear violation of this rule would be hand labeling a sentiment analysis test set or image recognition task: these are relatively easy for humans to label, and require more advanced algo's for computers to automatically classify. Those datasets are usually labeled by humans in the first place, so you will just replicate that (if file is "test14.jpg" then prediction is "horse"). In short human experts labeling is not machine learning. Writing formulas with correlations is.

\n

For this challenge this rule is a bit moot. It would be neigh impossible to hand label (I don't know any expert capable of that) and there are far too many samples (even Mechanical Turk would be expensive). If your method performs on unseen data then it is usually ok, in regards to this specific rule.

", "votes": null, "replies": [] }, { "id": 63614, "author_name": "computerguyted", "author_url": "", "post_date": "02/04/2015 22:13:04", "content": "

Thanks Triskelion, that helps a lot!

", "votes": null, "replies": [] }, { "id": 63616, "author_name": "triskelion", "author_url": "", "post_date": "02/04/2015 22:17:24", "content": "

A small aside: this rule also covers hand labeling _some_ test samples and adding these labeled samples to the train set AKA fully supervised learning.

", "votes": null, "replies": [] }, { "id": 63657, "author_name": "computerguyted", "author_url": "", "post_date": "02/05/2015 20:38:17", "content": "

>>A small aside: this rule also covers hand labeling _some_ test samples and adding these labeled samples to the train set AKA fully supervised learning.<<

\n

Thanks again. I'm not sure why anyone would want to that, however, even if it were allowed as:

\n

(1) The test data does not include clicks

\n

and

\n

(2) The final judging will be on different rows than the ones currently determining the Leaderboard rankings.

\n

But perhaps I am not missing something.

", "votes": null, "replies": [] }, { "id": 63662, "author_name": "davidtran", "author_url": "", "post_date": "02/05/2015 22:12:43", "content": "

[quote=ComputerGuyTed;63657]

\n

>>A small aside: this rule also covers hand labeling _some_ test samples and adding these labeled samples to the train set AKA fully supervised learning.<<

\n

Thanks again. I'm not sure why anyone would want to that, however, even if it were allowed as:

\n

(1) The test data does not include clicks

\n

and

\n

(2) The final judging will be on different rows than the ones currently determining the Leaderboard rankings.

\n

But perhaps I am not missing something.

\n

[/quote]

\n

Why would anyone want to do that? Well, because it can give you a significant boost in your leaderboard score. But it's a form of cheating -- training and testing on (some of) the same data doesn't make sense conceptually as your algorithm would be memorizing, not generalizing. It would do well on the test set but perform poorly on unseen data.

\n

As Triskelion said, hand-labeling is not practical in this competition. But there are others where it's feasible. For instance, there was a competition where the goal was to distinguish pictures of cats from pictures of dogs. Hand-labeling the test set would be easy for a human in that case.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "63611": "", "63612": "", "63614": "", "63616": "", "63657": "", "63662": "" }, "source": "meta" }