{
"id": 316,
"title": "discrepancy?",
"url": "/competitions/WIC2011/discussion/316",
"author_name": "",
"post_date": "2011-03-02T05:11:41Z",
"votes": null,
"comment_count": 10,
"views": 6255,
"content": "maybe i'm missing something totally obvious, but the training set includes 54 authors (2 paragfs each for a total of 108 training vectors). adding one for the unknown, this makes 55 probabilities per line. however the instructions and sample_train.csv have only space for 54 (including the unknown). specifically, the sample_train.csv seems to omit author \"040\" who does appear in train.csv. what's up with this?
i'm currently working only with the data in train.csv, but there is an author \"040\" in the images data too.
",
"messages": [
{
"id": "1848",
"postDate": "03/02/2011 05:11:41",
"content": "maybe i'm missing something totally obvious, but the training set includes 54 authors (2 paragfs each for a total of 108 training vectors). adding one for the unknown, this makes 55 probabilities per line. however the instructions and sample_train.csv have only space for 54 (including the unknown). specifically, the sample_train.csv seems to omit author \"040\" who does appear in train.csv. what's up with this?
i'm currently working only with the data in train.csv, but there is an author \"040\" in the images data too.
",
"rawMarkdown": "",
"votes": null
},
{
"id": "1849",
"postDate": "03/02/2011 05:11:41",
"content": "i meant \"sample_entry.csv\" of course. some php error is preventing me from editing the post.",
"rawMarkdown": "",
"votes": null
},
{
"id": "1851",
"postDate": "03/02/2011 05:11:41",
"content": "Author \"040\" is indeed only in the training set. Because in the test set, there is no 3rd paragraph corresponding to this author.
If your method outputs that author 040 is the most probable author for a certain image in the test set, then something must be wrong with your algorithm.
",
"rawMarkdown": "",
"votes": null
},
{
"id": "1852",
"postDate": "03/02/2011 05:11:41",
"content": "great. just sanity checking; thanks.
will we have any more side information? e.g. is each author represented at most once in the test set, &c.?
",
"rawMarkdown": "",
"votes": null
},
{
"id": "1853",
"postDate": "03/02/2011 05:11:41",
"content": "The number of writers which are not represented at most once and the number of unknown writers will not be communicated before the end of the competition.
",
"rawMarkdown": "",
"votes": null
},
{
"id": "2019",
"postDate": "03/02/2011 05:11:41",
"content": "So to be 100% clear about entry submissions, the file should have 54 lines, the first of which is a header. Each line should have 55 entries, which aside from the header line, should be composed of (1) the identifying writer (e.g., 'BC'), (2) 53 entries associated with the writers of the training data and *excluding an entry for writer '40'* and (3) one entry for 'unknown'. Are the headers interpreted by your scoring routine or must everything be in the exact same order as in the sample_entry.csv file? Thanks.
",
"rawMarkdown": "",
"votes": null
},
{
"id": "2020",
"postDate": "03/02/2011 05:11:41",
"content": "I don't know if the headers are interpreted by the scoring routine. Only Kaggle developers can answer this question. For now, please consider submitting in the exact same order as sample_entry.csv
The writer 40 is already excluded in the sample_entry.csv, Otherwise, your statements are correct.
However, if your method outputs that it is 50% probable that image ZZ is written by XX and 50% probable that it is written by an unknown writer, then it might be more interesting to have 0.5 in column XX and 0.5 in column 'unknown'.
",
"rawMarkdown": "",
"votes": null
},
{
"id": "2022",
"postDate": "03/02/2011 05:11:41",
"content": "Can you clarify what you mean by that last statement: \"However,\r\n if your method outputs that it is 50% probable that image ZZ is \r\nwritten by XX and 50% probable that it is written by an unknown writer,\r\n then it might be more interesting to have 0.5 in column XX and 0.5 in \r\ncolumn 'unknown'.\"??
More interesting than what? Than setting one column to \"1\" and the other to \"0\"?
",
"rawMarkdown": "",
"votes": null
},
{
"id": "2023",
"postDate": "03/02/2011 05:11:41",
"content": "Yes.
",
"rawMarkdown": "",
"votes": null
},
{
"id": "2028",
"postDate": "03/02/2011 05:11:41",
"content": "I don't think that's true.
If your belief is truly that the two are equally probably, than the expected loss of predicting 0-1 or 1-0 is the same as the expected loss of predicting 0.5-0.5 (or any other pair of values that adds up to one). So you might just as well randomly pick one of the two, and predict a 1 there...
A 0.5-0.5 prediction would only be favorable if the benchmark had used, e.g., a squared loss.
",
"rawMarkdown": "",
"votes": null
},
{
"id": "2029",
"postDate": "03/02/2011 05:11:41",
"content": "You are right. For MAE both are the same.
However, we might ask you by the end of the competition to provide us with custom submissions with probabilities instead of just 0 and 1. This will allows us to perform further evaluations (like identification rate for TOP-N writers).
",
"rawMarkdown": "",
"votes": null
}
],
"comments": [
{
"id": 1849,
"author_name": "tteravai",
"author_url": "",
"post_date": "03/02/2011 05:11:41",
"content": "i meant \"sample_entry.csv\" of course. some php error is preventing me from editing the post.",
"votes": null,
"replies": []
},
{
"id": 1851,
"author_name": "ahassaine",
"author_url": "",
"post_date": "03/02/2011 05:11:41",
"content": "Author \"040\" is indeed only in the training set. Because in the test set, there is no 3rd paragraph corresponding to this author.
If your method outputs that author 040 is the most probable author for a certain image in the test set, then something must be wrong with your algorithm.
",
"votes": null,
"replies": []
},
{
"id": 1852,
"author_name": "tteravai",
"author_url": "",
"post_date": "03/02/2011 05:11:41",
"content": "great. just sanity checking; thanks.
will we have any more side information? e.g. is each author represented at most once in the test set, &c.?
",
"votes": null,
"replies": []
},
{
"id": 1853,
"author_name": "ahassaine",
"author_url": "",
"post_date": "03/02/2011 05:11:41",
"content": "The number of writers which are not represented at most once and the number of unknown writers will not be communicated before the end of the competition.
",
"votes": null,
"replies": []
},
{
"id": 2019,
"author_name": "matthewroos",
"author_url": "",
"post_date": "03/02/2011 05:11:41",
"content": "So to be 100% clear about entry submissions, the file should have 54 lines, the first of which is a header. Each line should have 55 entries, which aside from the header line, should be composed of (1) the identifying writer (e.g., 'BC'), (2) 53 entries associated with the writers of the training data and *excluding an entry for writer '40'* and (3) one entry for 'unknown'. Are the headers interpreted by your scoring routine or must everything be in the exact same order as in the sample_entry.csv file? Thanks.
",
"votes": null,
"replies": []
},
{
"id": 2020,
"author_name": "ahassaine",
"author_url": "",
"post_date": "03/02/2011 05:11:41",
"content": "I don't know if the headers are interpreted by the scoring routine. Only Kaggle developers can answer this question. For now, please consider submitting in the exact same order as sample_entry.csv
The writer 40 is already excluded in the sample_entry.csv, Otherwise, your statements are correct.
However, if your method outputs that it is 50% probable that image ZZ is written by XX and 50% probable that it is written by an unknown writer, then it might be more interesting to have 0.5 in column XX and 0.5 in column 'unknown'.
",
"votes": null,
"replies": []
},
{
"id": 2022,
"author_name": "matthewroos",
"author_url": "",
"post_date": "03/02/2011 05:11:41",
"content": "Can you clarify what you mean by that last statement: \"However,\r\n if your method outputs that it is 50% probable that image ZZ is \r\nwritten by XX and 50% probable that it is written by an unknown writer,\r\n then it might be more interesting to have 0.5 in column XX and 0.5 in \r\ncolumn 'unknown'.\"??
More interesting than what? Than setting one column to \"1\" and the other to \"0\"?
",
"votes": null,
"replies": []
},
{
"id": 2023,
"author_name": "ahassaine",
"author_url": "",
"post_date": "03/02/2011 05:11:41",
"content": "Yes.
",
"votes": null,
"replies": []
},
{
"id": 2028,
"author_name": "lvdmaaten",
"author_url": "",
"post_date": "03/02/2011 05:11:41",
"content": "I don't think that's true.
If your belief is truly that the two are equally probably, than the expected loss of predicting 0-1 or 1-0 is the same as the expected loss of predicting 0.5-0.5 (or any other pair of values that adds up to one). So you might just as well randomly pick one of the two, and predict a 1 there...
A 0.5-0.5 prediction would only be favorable if the benchmark had used, e.g., a squared loss.
",
"votes": null,
"replies": []
},
{
"id": 2029,
"author_name": "ahassaine",
"author_url": "",
"post_date": "03/02/2011 05:11:41",
"content": "You are right. For MAE both are the same.
However, we might ask you by the end of the competition to provide us with custom submissions with probabilities instead of just 0 and 1. This will allows us to perform further evaluations (like identification rate for TOP-N writers).
",
"votes": null,
"replies": []
}
],
"raw_markdown_by_id": {
"1848": "",
"1849": "",
"1851": "",
"1852": "",
"1853": "",
"2019": "",
"2020": "",
"2022": "",
"2023": "",
"2028": "",
"2029": ""
},
"source": "meta"
}