Hi all!
\nThanks everyone for all of your feedback and patience. The metric has been updated to Jaccard FBeta 0.5, which will account for false negatives while still weighting precision. All existing submissions have been rescored. If your submission was invalid given the updated solution (i.e. could not run after 2 retries) it was invalidated - a small number of submissions were impacted. Submissions have been reenabled.
\nWe'd also like to note that - as you've noticed - a percentage of the public test samples ARE drawn from train. This is deliberate. As mentioned in the dataset description, the training set is not exhaustively labeled. The public test samples that come from train have additional labels that are not included in the training CSV. It's our hope that these will serve as guides for the difficult task of labeling the private test set. A large number of public test samples are NOT crossover from train, so you will still need to solve the problem to get a high public score.
\nThere is no crossover between train and the private test set, so string matching won't be an effective strategy there. It's my mistake that this all wasn't made clear in the dataset description - we originally described the dataset this way, and changed it during our final review.
\nThanks again, and good luck on the competition.
", "messages": [ { "id": "1251460", "postDate": "03/24/2021 19:34:06", "content": "Hi all!
\nThanks everyone for all of your feedback and patience. The metric has been updated to Jaccard FBeta 0.5, which will account for false negatives while still weighting precision. All existing submissions have been rescored. If your submission was invalid given the updated solution (i.e. could not run after 2 retries) it was invalidated - a small number of submissions were impacted. Submissions have been reenabled.
\nWe'd also like to note that - as you've noticed - a percentage of the public test samples ARE drawn from train. This is deliberate. As mentioned in the dataset description, the training set is not exhaustively labeled. The public test samples that come from train have additional labels that are not included in the training CSV. It's our hope that these will serve as guides for the difficult task of labeling the private test set. A large number of public test samples are NOT crossover from train, so you will still need to solve the problem to get a high public score.
\nThere is no crossover between train and the private test set, so string matching won't be an effective strategy there. It's my mistake that this all wasn't made clear in the dataset description - we originally described the dataset this way, and changed it during our final review.
\nThanks again, and good luck on the competition.
", "rawMarkdown": "Hi all!\n\nThanks everyone for all of your feedback and patience. The metric has been updated to Jaccard FBeta 0.5, which will account for false negatives while still weighting precision. All existing submissions have been rescored. If your submission was invalid given the updated solution (i.e. could not run after 2 retries) it was invalidated - a small number of submissions were impacted. Submissions have been reenabled.\n\nWe'd also like to note that - as you've noticed - a percentage of the public test samples ARE drawn from train. This is deliberate. As mentioned in the dataset description, the training set is not exhaustively labeled. The public test samples that come from train have additional labels that are not included in the training CSV. It's our hope that these will serve as guides for the difficult task of labeling the private test set. A large number of public test samples are NOT crossover from train, so you will still need to solve the problem to get a high public score.\n\nThere is no crossover between train and the private test set, so string matching won't be an effective strategy there. It's my mistake that this all wasn't made clear in the dataset description - we originally described the dataset this way, and changed it during our final review.\n\nThanks again, and good luck on the competition.", "votes": null }, { "id": "1251478", "postDate": "03/24/2021 20:15:37", "content": "After the recent re-scoring, my string literal notebook scores are:
\n0.357 - v5 | input dataset_label + dataset_title + pub_title | output cleaned_label
\n\n0.464 - v9 | input dataset_title | output dataset_title
\n\nWould it be possible to get some clarification as to what the PredictionString is being compared to, and which field exactly are we trying to predict?
\nAlso are there labels contained in Test that are not mentioned in Train?
", "rawMarkdown": "After the recent re-scoring, my string literal notebook scores are:\n\n0.357 - v5 | input dataset_label + dataset_title + pub_title | output cleaned_label\n- https://www.kaggle.com/jamesmcguigan/coleridge-string-literals?scriptVersionId=57586922\n\n0.464 - v9 | input dataset_title | output dataset_title\n- https://www.kaggle.com/jamesmcguigan/coleridge-string-literals?scriptVersionId=57628162\n\nWould it be possible to get some clarification as to what the PredictionString is being compared to, and which field exactly are we trying to predict?\n\nAlso are there labels contained in Test that are not mentioned in Train?", "votes": null }, { "id": "1251484", "postDate": "03/24/2021 20:26:21", "content": "Hi @jamesmcguigan - thanks!
\nIn this competition, predictions are drawn from the documents. PredictionString is being compared against extracts from publications that indicate a dataset was mentioned in that publication. The strings that predictions are being compared against have been cleaned, as indicated in the Evaluation page.
\nYes! There are a huge number of labels in test that are not mentioned in train at all. So the task is less to match strings, and more to create an NLP solution for generalizing the \"can I identify datasets mentioned in this document\" problem.
", "rawMarkdown": "Hi @jamesmcguigan - thanks!\n\nIn this competition, predictions are drawn from the documents. PredictionString is being compared against extracts from publications that indicate a dataset was mentioned in that publication. The strings that predictions are being compared against have been cleaned, as indicated in the Evaluation page.\n\nYes! There are a huge number of labels in test that are not mentioned in train at all. So the task is less to match strings, and more to create an NLP solution for generalizing the \"can I identify datasets mentioned in this document\" problem.", "votes": null }, { "id": "1251495", "postDate": "03/24/2021 20:47:25", "content": "And for additonal clarification, does cleaned_label in train.csv represent the examples of extracted text we are attempting to predict for the training data?
Correct. cleaned_label represents examples of (cleaned) extracted text we're attempting to predict for the training data.
\n\nas you've noticed - a percentage of the public test samples ARE drawn from train
\n
I hadn't noticed so far, LOL.
", "rawMarkdown": "> as you've noticed - a percentage of the public test samples ARE drawn from train\n\nI hadn't noticed so far, LOL.", "votes": null }, { "id": "1251834", "postDate": "03/25/2021 07:14:05", "content": "hey @philculliton ,
\nCan you also explain what exactly would constitute for a FN? It is mentioned.
\n\nAny ground truths with no nearest predictions are counted as false negatives (FN).
\n
Does it mean a ground truth for which there are no TP(no matched prediction above 0.5)?
\nOr it hints at the case when there is no prediction at all ?
Hi @philculliton,
\nMay I ask about the evaluation metric, is it possible for one predicted string to match more than one ground truth?
This is my notebook https://www.kaggle.com/tthien/meow-baseline
\nIt have status of \"Submission Scoring Error\".
\nI believe I output correct format.
Can you take a look.
", "rawMarkdown": "This is my notebook https://www.kaggle.com/tthien/meow-baseline \nIt have status of \"Submission Scoring Error\".\nI believe I output correct format. \n\nCan you take a look.", "votes": null }, { "id": "1254169", "postDate": "03/27/2021 11:48:43", "content": "As if local splitting wasn't tricky enough, they made the public LB mostly useless too 😒
", "rawMarkdown": "As if local splitting wasn't tricky enough, they made the public LB mostly useless too 😒", "votes": null }, { "id": "1266254", "postDate": "04/07/2021 15:46:31", "content": "Also was wondering on that.
\nThink, it is possible, as you need to hit 0.5 threshold which is absolutely achievable since some datasets names are VERY close (compare \"national education longitudinal study\" and \"education longitudinal study\" :))
Hello @philculliton,
\nthank you for your commitment. I believe that this competition is really valuable for us to learn. But still, the evaluation metric is somehow vague, which makes people confused.
\nSo could you show any dummy table (like 10 rows) which includes two columns with ground truth and prediction, and FBeta score for it?
Hi @dtmanager1979 - thanks! Yep, we plan to add some examples in the next couple days, that should help clear things up.
", "rawMarkdown": "Hi @dtmanager1979 - thanks! Yep, we plan to add some examples in the next couple days, that should help clear things up.", "votes": null }, { "id": "1279114", "postDate": "04/20/2021 15:41:11", "content": "Hi @philculliton,
\nThanx, for all the hard work!
\nDo you have any idea as to when these examples will be added?
\nIf these examples don't adress this question
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230457
\ncould you answer this also please?
\nThanx!
@philculliton if bringing your comment again,
\n\n\nThere are a huge number of labels in test that are not mentioned in train at all.
\n
Can I summarize also the number of the cleaned label in the test dataset != the number of the cleaned label in the train dataset?
\nActually, there are 130 unique cleaned labels in the train dataset.
\nBut can I imagine that there might be 150 unique cleaned labels in the test dataset?
\nAm I wrong?
What can we say about datasets? It is clear to me that for each text we can find several extracts mentioning ONE specific dataset. But are the datasets ONLY those that appear in the train? Or are there others, perhaps to be found in the other datasets made available by Coleridge such as the 'additional-gov-datasets'? Thank you in advance for your reply.
", "rawMarkdown": "What can we say about datasets? It is clear to me that for each text we can find several extracts mentioning ONE specific dataset. But are the datasets ONLY those that appear in the train? Or are there others, perhaps to be found in the other datasets made available by Coleridge such as the 'additional-gov-datasets'? Thank you in advance for your reply.", "votes": null }, { "id": "1355968", "postDate": "06/18/2021 16:41:05", "content": "I'm not sure why you want to remove the train labels.
\nThe original quote from host is:
\nThere is no crossover between train and the private test set, so string matching won't be an effective strategy there.
I think this statement means that there is no crossover in the train text and private test text, not that there is no crossover in the train and private test label. so I think that private test text may still contain the train label.
\nI would like to hear your explanation
", "rawMarkdown": "I'm not sure why you want to remove the train labels.\n\nThe original quote from host is: \n`There is no crossover between train and the private test set, so string matching won't be an effective strategy there.`\n\nI think this statement means that there is no crossover in the train text and private test text, not that there is no crossover in the train and private test label. so I think that private test text may still contain the train label. \n\nI would like to hear your explanation", "votes": null } ], "comments": [ { "id": 1251478, "author_name": "jamesmcguigan", "author_url": "", "post_date": "03/24/2021 20:15:37", "content": "After the recent re-scoring, my string literal notebook scores are:
\n0.357 - v5 | input dataset_label + dataset_title + pub_title | output cleaned_label
\n\n0.464 - v9 | input dataset_title | output dataset_title
\n\nWould it be possible to get some clarification as to what the PredictionString is being compared to, and which field exactly are we trying to predict?
\nAlso are there labels contained in Test that are not mentioned in Train?
", "votes": null, "replies": [ { "id": 1251484, "author_name": "philculliton", "author_url": "", "post_date": "03/24/2021 20:26:21", "content": "Hi @jamesmcguigan - thanks!
\nIn this competition, predictions are drawn from the documents. PredictionString is being compared against extracts from publications that indicate a dataset was mentioned in that publication. The strings that predictions are being compared against have been cleaned, as indicated in the Evaluation page.
\nYes! There are a huge number of labels in test that are not mentioned in train at all. So the task is less to match strings, and more to create an NLP solution for generalizing the \"can I identify datasets mentioned in this document\" problem.
", "votes": null, "replies": [] }, { "id": 1251495, "author_name": "jamesmcguigan", "author_url": "", "post_date": "03/24/2021 20:47:25", "content": "And for additonal clarification, does cleaned_label in train.csv represent the examples of extracted text we are attempting to predict for the training data?
Correct. cleaned_label represents examples of (cleaned) extracted text we're attempting to predict for the training data.
@philculliton if bringing your comment again,
\n\n\nThere are a huge number of labels in test that are not mentioned in train at all.
\n
Can I summarize also the number of the cleaned label in the test dataset != the number of the cleaned label in the train dataset?
\nActually, there are 130 unique cleaned labels in the train dataset.
\nBut can I imagine that there might be 150 unique cleaned labels in the test dataset?
\nAm I wrong?
\n\nas you've noticed - a percentage of the public test samples ARE drawn from train
\n
I hadn't noticed so far, LOL.
", "votes": null, "replies": [ { "id": 1254169, "author_name": "suicaokhoailang", "author_url": "", "post_date": "03/27/2021 11:48:43", "content": "As if local splitting wasn't tricky enough, they made the public LB mostly useless too 😒
", "votes": null, "replies": [] } ] }, { "id": 1251834, "author_name": "ibtesama", "author_url": "", "post_date": "03/25/2021 07:14:05", "content": "hey @philculliton ,
\nCan you also explain what exactly would constitute for a FN? It is mentioned.
\n\nAny ground truths with no nearest predictions are counted as false negatives (FN).
\n
Does it mean a ground truth for which there are no TP(no matched prediction above 0.5)?
\nOr it hints at the case when there is no prediction at all ?
Hi @philculliton,
\nMay I ask about the evaluation metric, is it possible for one predicted string to match more than one ground truth?
Also was wondering on that.
\nThink, it is possible, as you need to hit 0.5 threshold which is absolutely achievable since some datasets names are VERY close (compare \"national education longitudinal study\" and \"education longitudinal study\" :))
This is my notebook https://www.kaggle.com/tthien/meow-baseline
\nIt have status of \"Submission Scoring Error\".
\nI believe I output correct format.
Can you take a look.
", "votes": null, "replies": [] }, { "id": 1271575, "author_name": "dtmanager1979", "author_url": "", "post_date": "04/12/2021 17:45:59", "content": "Hello @philculliton,
\nthank you for your commitment. I believe that this competition is really valuable for us to learn. But still, the evaluation metric is somehow vague, which makes people confused.
\nSo could you show any dummy table (like 10 rows) which includes two columns with ground truth and prediction, and FBeta score for it?
Hi @dtmanager1979 - thanks! Yep, we plan to add some examples in the next couple days, that should help clear things up.
", "votes": null, "replies": [] }, { "id": 1279114, "author_name": "julesvanligtenberg", "author_url": "", "post_date": "04/20/2021 15:41:11", "content": "Hi @philculliton,
\nThanx, for all the hard work!
\nDo you have any idea as to when these examples will be added?
\nIf these examples don't adress this question
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230457
\ncould you answer this also please?
\nThanx!
What can we say about datasets? It is clear to me that for each text we can find several extracts mentioning ONE specific dataset. But are the datasets ONLY those that appear in the train? Or are there others, perhaps to be found in the other datasets made available by Coleridge such as the 'additional-gov-datasets'? Thank you in advance for your reply.
", "votes": null, "replies": [] }, { "id": 1355968, "author_name": "syxuming", "author_url": "", "post_date": "06/18/2021 16:41:05", "content": "I'm not sure why you want to remove the train labels.
\nThe original quote from host is:
\nThere is no crossover between train and the private test set, so string matching won't be an effective strategy there.
I think this statement means that there is no crossover in the train text and private test text, not that there is no crossover in the train and private test label. so I think that private test text may still contain the train label.
\nI would like to hear your explanation
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1251460": "Hi all!\n\nThanks everyone for all of your feedback and patience. The metric has been updated to Jaccard FBeta 0.5, which will account for false negatives while still weighting precision. All existing submissions have been rescored. If your submission was invalid given the updated solution (i.e. could not run after 2 retries) it was invalidated - a small number of submissions were impacted. Submissions have been reenabled.\n\nWe'd also like to note that - as you've noticed - a percentage of the public test samples ARE drawn from train. This is deliberate. As mentioned in the dataset description, the training set is not exhaustively labeled. The public test samples that come from train have additional labels that are not included in the training CSV. It's our hope that these will serve as guides for the difficult task of labeling the private test set. A large number of public test samples are NOT crossover from train, so you will still need to solve the problem to get a high public score.\n\nThere is no crossover between train and the private test set, so string matching won't be an effective strategy there. It's my mistake that this all wasn't made clear in the dataset description - we originally described the dataset this way, and changed it during our final review.\n\nThanks again, and good luck on the competition.", "1251478": "After the recent re-scoring, my string literal notebook scores are:\n\n0.357 - v5 | input dataset_label + dataset_title + pub_title | output cleaned_label\n- https://www.kaggle.com/jamesmcguigan/coleridge-string-literals?scriptVersionId=57586922\n\n0.464 - v9 | input dataset_title | output dataset_title\n- https://www.kaggle.com/jamesmcguigan/coleridge-string-literals?scriptVersionId=57628162\n\nWould it be possible to get some clarification as to what the PredictionString is being compared to, and which field exactly are we trying to predict?\n\nAlso are there labels contained in Test that are not mentioned in Train?", "1251484": "Hi @jamesmcguigan - thanks!\n\nIn this competition, predictions are drawn from the documents. PredictionString is being compared against extracts from publications that indicate a dataset was mentioned in that publication. The strings that predictions are being compared against have been cleaned, as indicated in the Evaluation page.\n\nYes! There are a huge number of labels in test that are not mentioned in train at all. So the task is less to match strings, and more to create an NLP solution for generalizing the \"can I identify datasets mentioned in this document\" problem.", "1251495": "And for additonal clarification, does `cleaned_label` in `train.csv` represent the examples of extracted text we are attempting to predict for the training data?", "1251503": "Correct. `cleaned_label` represents examples of (cleaned) extracted text we're attempting to predict for the training data.", "1251547": "> as you've noticed - a percentage of the public test samples ARE drawn from train\n\nI hadn't noticed so far, LOL.", "1251834": "hey @philculliton ,\nCan you also explain what exactly would constitute for a FN? It is mentioned.\n\n> Any ground truths with no nearest predictions are counted as false negatives (FN).\n\nDoes it mean a ground truth for which there are no TP(no matched prediction above 0.5)?\nOr it hints at the case when there is no prediction at all ?", "1254117": "Hi @philculliton,\nMay I ask about the evaluation metric, is it possible for one predicted string to match more than one ground truth?", "1254141": "This is my notebook https://www.kaggle.com/tthien/meow-baseline \nIt have status of \"Submission Scoring Error\".\nI believe I output correct format. \n\nCan you take a look.", "1254169": "As if local splitting wasn't tricky enough, they made the public LB mostly useless too 😒", "1266254": "Also was wondering on that. \nThink, it is possible, as you need to hit 0.5 threshold which is absolutely achievable since some datasets names are VERY close (compare \"national education longitudinal study\" and \"education longitudinal study\" :))", "1271575": "Hello @philculliton,\nthank you for your commitment. I believe that this competition is really valuable for us to learn. But still, the evaluation metric is somehow vague, which makes people confused. \nSo could you show any dummy table (like 10 rows) which includes two columns with ground truth and prediction, and FBeta score for it?", "1271581": "Hi @dtmanager1979 - thanks! Yep, we plan to add some examples in the next couple days, that should help clear things up.", "1279114": "Hi @philculliton,\n\nThanx, for all the hard work!\nDo you have any idea as to when these examples will be added?\nIf these examples don't adress this question\n[https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230457](url)\ncould you answer this also please?\nThanx!", "1279332": "philculliton if bringing your comment again, \n> There are a huge number of labels **in test that are not mentioned in train** at all.\n\nCan I summarize also the number of the cleaned label in the test dataset != the number of the cleaned label in the train dataset? \n\nActually, there are 130 unique cleaned labels in the train dataset. \n\nBut can I imagine that there might be 150 unique cleaned labels in the test dataset? \nAm I wrong?", "1343495": "What can we say about datasets? It is clear to me that for each text we can find several extracts mentioning ONE specific dataset. But are the datasets ONLY those that appear in the train? Or are there others, perhaps to be found in the other datasets made available by Coleridge such as the 'additional-gov-datasets'? Thank you in advance for your reply.", "1355968": "I'm not sure why you want to remove the train labels.\n\nThe original quote from host is: \n`There is no crossover between train and the private test set, so string matching won't be an effective strategy there.`\n\nI think this statement means that there is no crossover in the train text and private test text, not that there is no crossover in the train and private test label. so I think that private test text may still contain the train label. \n\nI would like to hear your explanation" }, "source": "meta" }