Since i get some counter-intuitive results I just want to clarify how you calculate the public leaderboard score.
\n@philculliton So my question is boiling down to the following:
\nIn the data used for calculating the public leaderboard, can we assume that for a given Id (publication) you have all mentioned datasets listed?
\nOr could it be that once I would consider the full test data, that for a given 'Id' there might be more datasets, compared to the ones I observe when looking only at the subset data used for the public leaderboard calculation?
", "messages": [ { "id": "1270143", "postDate": "04/11/2021 10:45:10", "content": "Since i get some counter-intuitive results I just want to clarify how you calculate the public leaderboard score.
\n@philculliton So my question is boiling down to the following:
\nIn the data used for calculating the public leaderboard, can we assume that for a given Id (publication) you have all mentioned datasets listed?
\nOr could it be that once I would consider the full test data, that for a given 'Id' there might be more datasets, compared to the ones I observe when looking only at the subset data used for the public leaderboard calculation?
", "rawMarkdown": "Since i get some counter-intuitive results I just want to clarify how you calculate the public leaderboard score. \n\n@philculliton So my question is boiling down to the following:\n\nIn the data used for calculating the public leaderboard, can we assume that for a given Id (publication) you have all mentioned datasets listed? \n\nOr could it be that once I would consider the full test data, that for a given 'Id' there might be more datasets, compared to the ones I observe when looking only at the subset data used for the public leaderboard calculation?", "votes": null }, { "id": "1270185", "postDate": "04/11/2021 11:46:10", "content": "Hi - thanks for the question. I'm sure @philculliton will respond on Monday with the details, but in the full test data there are more datasets than the subset data in the public leaderboard. Only 12% of the test data are used for the public leaderboard calculation.
\nThanks so very much for your engagement and interest.
@julialane thanks for your prompt reply, but I think I have to better explain my question.
\nMy question is the following:
\nlet us look at a specific example publication which you use for the public leaderboard evaluation:
\nfor concreteness let us give it 'Id' =1
\nIf I look at the full test data, it mentions 3 datasets: a, b, and c.
\nThe question I have is whether we can be sure that in the data used for the public leaderboard we see all those 3 datasets, or is the split happening in such a way that i see only a subset of those 3 datasets.
\nThank you very much in advance for your reply.
", "rawMarkdown": "julialane thanks for your prompt reply, but I think I have to better explain my question.\n\nMy question is the following:\n\nlet us look at a specific example publication which you use for the public leaderboard evaluation:\n\nfor concreteness let us give it 'Id' =1\n\nIf I look at the full test data, it mentions 3 datasets: a, b, and c.\n\nThe question I have is whether we can be sure that in the data used for the public leaderboard we see all those 3 datasets, or is the split happening in such a way that i see only a subset of those 3 datasets.\n\nThank you very much in advance for your reply.", "votes": null }, { "id": "1270203", "postDate": "04/11/2021 12:24:54", "content": "Got it. The split is done by publications. So publication \"1\" cannot be in both the public and private leaderboard. Datasets a, b and c might, however, be in both - albeit in different publications. Also, datasets x, y and z might be in the private leaderboard but not in the public. Again thanks for your interest!
", "rawMarkdown": "Got it. The split is done by publications. So publication \"1\" cannot be in both the public and private leaderboard. Datasets a, b and c might, however, be in both - albeit in different publications. Also, datasets x, y and z might be in the private leaderboard but not in the public. Again thanks for your interest!", "votes": null }, { "id": "1270222", "postDate": "04/11/2021 12:43:13", "content": "Questions about Evaluation are still arising. Other recent posts:
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230457
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230091#1269093
In my opinion, verbal descriptions of the Evaluation are insufficient given its nature. I think it would be in the best interest of the competition for Kaggle to provide its evaluation code, a mathematical description, or at least multiple examples that cover the expected types of cases.
", "rawMarkdown": "Questions about Evaluation are still arising. Other recent posts: \nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230457\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230091#1269093\n\nIn my opinion, verbal descriptions of the Evaluation are insufficient given its nature. I think it would be in the best interest of the competition for Kaggle to provide its evaluation code, a mathematical description, or at least multiple examples that cover the expected types of cases.", "votes": null }, { "id": "1271457", "postDate": "04/12/2021 16:12:07", "content": "@robrichter - good question. The training documents that are also in public test have labels that are not included in the training labels. For performance on the public leaderboard you'll need to identify datasets not included in the training data. Re: the split, the publications in private are entirely disjoint from public - so any publications in the public set will be used to determine the public score only.
\n@trentb - thanks! We'll provide an example. Our internal evaluation code is optimized / distributed C# and much more likely to be confusing rather than helpful.
", "rawMarkdown": "robrichter - good question. The training documents that are also in public test have labels that are *not* included in the training labels. For performance on the public leaderboard you'll need to identify datasets not included in the training data. Re: the split, the publications in private are entirely disjoint from public - so any publications in the public set will be used to determine the public score only.\n\n@trentb - thanks! We'll provide an example. Our internal evaluation code is optimized / distributed C# and much more likely to be confusing rather than helpful.", "votes": null }, { "id": "1271768", "postDate": "04/12/2021 22:12:42", "content": "\nJust to be clear, the publications are disjoint but are the labels disjoint as well (between private test and public test)? In other words, would people be rewarded for trawling for additional labels in the training data?
\nAlso, could you monitor the public/private LB gap and consider switching the LB so both public and private use a split of the 88% disjoint publications? The current metric and setup (penalizing FN and use of train data w/ add. labels) is an incentive to trawl for additional labels in the publications we have. That's not necessarily the same thing as developing an approach that can extract labels on unseen data.
", "rawMarkdown": "philculliton \n\nJust to be clear, the publications are disjoint but are the labels disjoint as well (between private test and public test)? In other words, would people be rewarded for trawling for additional labels in the training data? \n\nAlso, could you monitor the public/private LB gap and consider switching the LB so both public and private use a split of the 88% disjoint publications? The current metric and setup (penalizing FN and use of train data w/ add. labels) is an incentive to trawl for additional labels in the publications we have. That's not necessarily the same thing as developing an approach that can extract labels on unseen data.", "votes": null } ], "comments": [ { "id": 1270185, "author_name": "julialane", "author_url": "", "post_date": "04/11/2021 11:46:10", "content": "Hi - thanks for the question. I'm sure @philculliton will respond on Monday with the details, but in the full test data there are more datasets than the subset data in the public leaderboard. Only 12% of the test data are used for the public leaderboard calculation.
\nThanks so very much for your engagement and interest.
@julialane thanks for your prompt reply, but I think I have to better explain my question.
\nMy question is the following:
\nlet us look at a specific example publication which you use for the public leaderboard evaluation:
\nfor concreteness let us give it 'Id' =1
\nIf I look at the full test data, it mentions 3 datasets: a, b, and c.
\nThe question I have is whether we can be sure that in the data used for the public leaderboard we see all those 3 datasets, or is the split happening in such a way that i see only a subset of those 3 datasets.
\nThank you very much in advance for your reply.
", "votes": null, "replies": [ { "id": 1270222, "author_name": "trentb", "author_url": "", "post_date": "04/11/2021 12:43:13", "content": "Questions about Evaluation are still arising. Other recent posts:
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230457
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230091#1269093
In my opinion, verbal descriptions of the Evaluation are insufficient given its nature. I think it would be in the best interest of the competition for Kaggle to provide its evaluation code, a mathematical description, or at least multiple examples that cover the expected types of cases.
", "votes": null, "replies": [] }, { "id": 1271457, "author_name": "philculliton", "author_url": "", "post_date": "04/12/2021 16:12:07", "content": "@robrichter - good question. The training documents that are also in public test have labels that are not included in the training labels. For performance on the public leaderboard you'll need to identify datasets not included in the training data. Re: the split, the publications in private are entirely disjoint from public - so any publications in the public set will be used to determine the public score only.
\n@trentb - thanks! We'll provide an example. Our internal evaluation code is optimized / distributed C# and much more likely to be confusing rather than helpful.
", "votes": null, "replies": [] }, { "id": 1271768, "author_name": "leecming", "author_url": "", "post_date": "04/12/2021 22:12:42", "content": "\nJust to be clear, the publications are disjoint but are the labels disjoint as well (between private test and public test)? In other words, would people be rewarded for trawling for additional labels in the training data?
\nAlso, could you monitor the public/private LB gap and consider switching the LB so both public and private use a split of the 88% disjoint publications? The current metric and setup (penalizing FN and use of train data w/ add. labels) is an incentive to trawl for additional labels in the publications we have. That's not necessarily the same thing as developing an approach that can extract labels on unseen data.
", "votes": null, "replies": [] } ] }, { "id": 1270203, "author_name": "julialane", "author_url": "", "post_date": "04/11/2021 12:24:54", "content": "Got it. The split is done by publications. So publication \"1\" cannot be in both the public and private leaderboard. Datasets a, b and c might, however, be in both - albeit in different publications. Also, datasets x, y and z might be in the private leaderboard but not in the public. Again thanks for your interest!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1270143": "Since i get some counter-intuitive results I just want to clarify how you calculate the public leaderboard score. \n\n@philculliton So my question is boiling down to the following:\n\nIn the data used for calculating the public leaderboard, can we assume that for a given Id (publication) you have all mentioned datasets listed? \n\nOr could it be that once I would consider the full test data, that for a given 'Id' there might be more datasets, compared to the ones I observe when looking only at the subset data used for the public leaderboard calculation?", "1270185": "Hi - thanks for the question. I'm sure @philculliton will respond on Monday with the details, but in the full test data there are more datasets than the subset data in the public leaderboard. Only 12% of the test data are used for the public leaderboard calculation. \nThanks so very much for your engagement and interest.", "1270189": "julialane thanks for your prompt reply, but I think I have to better explain my question.\n\nMy question is the following:\n\nlet us look at a specific example publication which you use for the public leaderboard evaluation:\n\nfor concreteness let us give it 'Id' =1\n\nIf I look at the full test data, it mentions 3 datasets: a, b, and c.\n\nThe question I have is whether we can be sure that in the data used for the public leaderboard we see all those 3 datasets, or is the split happening in such a way that i see only a subset of those 3 datasets.\n\nThank you very much in advance for your reply.", "1270203": "Got it. The split is done by publications. So publication \"1\" cannot be in both the public and private leaderboard. Datasets a, b and c might, however, be in both - albeit in different publications. Also, datasets x, y and z might be in the private leaderboard but not in the public. Again thanks for your interest!", "1270222": "Questions about Evaluation are still arising. Other recent posts: \nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230457\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230091#1269093\n\nIn my opinion, verbal descriptions of the Evaluation are insufficient given its nature. I think it would be in the best interest of the competition for Kaggle to provide its evaluation code, a mathematical description, or at least multiple examples that cover the expected types of cases.", "1271457": "robrichter - good question. The training documents that are also in public test have labels that are *not* included in the training labels. For performance on the public leaderboard you'll need to identify datasets not included in the training data. Re: the split, the publications in private are entirely disjoint from public - so any publications in the public set will be used to determine the public score only.\n\n@trentb - thanks! We'll provide an example. Our internal evaluation code is optimized / distributed C# and much more likely to be confusing rather than helpful.", "1271768": "philculliton \n\nJust to be clear, the publications are disjoint but are the labels disjoint as well (between private test and public test)? In other words, would people be rewarded for trawling for additional labels in the training data? \n\nAlso, could you monitor the public/private LB gap and consider switching the LB so both public and private use a split of the 88% disjoint publications? The current metric and setup (penalizing FN and use of train data w/ add. labels) is an incentive to trawl for additional labels in the publications we have. That's not necessarily the same thing as developing an approach that can extract labels on unseen data." }, "source": "meta" }