Thanks to the hosts for the recent revision to annotations! Although not much details have been given, it seems the revision has been done mainly to improve the coverage of the weakly labeled test set. This is great and will help to score higher models that generalize better.
\nUnfortunately, it seems the revision has not taken into account the ambiguity in the prediction string. Many discussions are going on around this (see e.g. here, here, and here).
\nThis discussion is to explicitly ask the hosts to standardize the prediction strings so that models can be evaluated more reliably. Please upvote to increase the chances it gets done!
I suggest a simple standardization: only ONE label for EACH mention. The label should be the longest possible piece of text that makes sense to the hosts but only ONE for ONE mention.
\nBelow I report a few cases where multiple labels are used in the train set even if the publication contains only one single mention. There are also some few cases where the labels in train are NOT referenced as datasets. I ran tests that makes me think the same ambiguity is present in the test set, and this has a large randomizing effect on the model evaluation and on the leaderboard.
\nID: 002cbf56-5158-4ec7-83fd-51fa7829bb13
\nProblem: Two labels but only One mention: baltimore longitudinal study of aging blsa and baltimore longitudinal study of aging
\n\nSchrack et al. showed an increase in REE with the number of chronic diseases in a study with \"healthy\" volunteers from the Baltimore Longitudinal Study of Aging (BLSA) cohort [36] .
\n
ID: 0035a1ba-6d1e-487b-bc3d-d49c5d64a3e9
\nProblem: Two labels but only One mention: baltimore longitudinal study of aging blsa and baltimore longitudinal study of aging
\n\nThe baseline images are selected from the Baltimore Longitudinal Study of Aging (BLSA) [48] data set.
\n
ID: 0085fd27-7924-41cb-b268-574356c3d6f7
\nProblem: Two labels but only One mention: national education longitudinal study and education longitudinal study
\n\nOther empirical results from the National Education Longitudinal Study show that same-gender teaching increases the test scores of eighth-grade children (Dee, 2005b) .
\n
ID: 009a8e9d-f33c-4319-9f58-fc3e2e71e02b
\nProblem: Two labels but only One mention: adni and alzheimer s disease neuroimaging initiative adni
\n\nAlzheimer's Disease Neuroimaging Initiative (ADNI) and post-mortem studies demonstrated evidence for the power of FDG PET as a biomarker for AD.
\n
ID: 00243b98-f868-45e4-9b83-0c346c7ecad5
\nProblem: Incomplete label: early childhood longitudinal study
\n\nThe investigators used data from the Early Childhood Longitudinal Study-Kindergarten Cohort (ECLS-K) to estimate whether and to what extent the timing and persistence of mathematics difficulties (MD) in kindergarten predicted children's first through fifth grade math growth trajectories.
\n
ID: 000efc17-13d8-433d-8f62-a3932fe4f3b8
\nProblem: Missing label: train contains adni and alzheimer s disease neuroimaging initiative adni. The following label should also be in train but it misses: alzheimer s disease neuroimaging initiative
\n\nTo address this, we used multimodal data from the National Alzheimer's Coordinating Center and Alzheimer's Disease Neuroimaging Initiative data sets.
\n
ID: 0010357a-6365-4e5f-b982-582e6d32c3ee
\nPorblem: Wrong label: genome sequence of covid 19 is not referenced as a dataset
\n\n", "messages": [ { "id": "1295539", "postDate": "05/06/2021 14:35:06", "content": "The whole genome sequence of COVID -19 has about 50% similarity to that of MERS-CoV.
\n
Thanks to the hosts for the recent revision to annotations! Although not much details have been given, it seems the revision has been done mainly to improve the coverage of the weakly labeled test set. This is great and will help to score higher models that generalize better.
\nUnfortunately, it seems the revision has not taken into account the ambiguity in the prediction string. Many discussions are going on around this (see e.g. here, here, and here).
\nThis discussion is to explicitly ask the hosts to standardize the prediction strings so that models can be evaluated more reliably. Please upvote to increase the chances it gets done!
I suggest a simple standardization: only ONE label for EACH mention. The label should be the longest possible piece of text that makes sense to the hosts but only ONE for ONE mention.
\nBelow I report a few cases where multiple labels are used in the train set even if the publication contains only one single mention. There are also some few cases where the labels in train are NOT referenced as datasets. I ran tests that makes me think the same ambiguity is present in the test set, and this has a large randomizing effect on the model evaluation and on the leaderboard.
\nID: 002cbf56-5158-4ec7-83fd-51fa7829bb13
\nProblem: Two labels but only One mention: baltimore longitudinal study of aging blsa and baltimore longitudinal study of aging
\n\nSchrack et al. showed an increase in REE with the number of chronic diseases in a study with \"healthy\" volunteers from the Baltimore Longitudinal Study of Aging (BLSA) cohort [36] .
\n
ID: 0035a1ba-6d1e-487b-bc3d-d49c5d64a3e9
\nProblem: Two labels but only One mention: baltimore longitudinal study of aging blsa and baltimore longitudinal study of aging
\n\nThe baseline images are selected from the Baltimore Longitudinal Study of Aging (BLSA) [48] data set.
\n
ID: 0085fd27-7924-41cb-b268-574356c3d6f7
\nProblem: Two labels but only One mention: national education longitudinal study and education longitudinal study
\n\nOther empirical results from the National Education Longitudinal Study show that same-gender teaching increases the test scores of eighth-grade children (Dee, 2005b) .
\n
ID: 009a8e9d-f33c-4319-9f58-fc3e2e71e02b
\nProblem: Two labels but only One mention: adni and alzheimer s disease neuroimaging initiative adni
\n\nAlzheimer's Disease Neuroimaging Initiative (ADNI) and post-mortem studies demonstrated evidence for the power of FDG PET as a biomarker for AD.
\n
ID: 00243b98-f868-45e4-9b83-0c346c7ecad5
\nProblem: Incomplete label: early childhood longitudinal study
\n\nThe investigators used data from the Early Childhood Longitudinal Study-Kindergarten Cohort (ECLS-K) to estimate whether and to what extent the timing and persistence of mathematics difficulties (MD) in kindergarten predicted children's first through fifth grade math growth trajectories.
\n
ID: 000efc17-13d8-433d-8f62-a3932fe4f3b8
\nProblem: Missing label: train contains adni and alzheimer s disease neuroimaging initiative adni. The following label should also be in train but it misses: alzheimer s disease neuroimaging initiative
\n\nTo address this, we used multimodal data from the National Alzheimer's Coordinating Center and Alzheimer's Disease Neuroimaging Initiative data sets.
\n
ID: 0010357a-6365-4e5f-b982-582e6d32c3ee
\nPorblem: Wrong label: genome sequence of covid 19 is not referenced as a dataset
\n\n", "rawMarkdown": "Thanks to the hosts for the recent [revision to annotations](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236508)! Although not much details have been given, it seems the revision has been done mainly to improve the coverage of the [weakly labeled test set](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/233170). This is great and will help to score higher models that generalize better.\n\nUnfortunately, it seems the revision has not taken into account the ambiguity in the prediction string. Many discussions are going on around this (see e.g. [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/235822), [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/235752), and [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/231893)). \n**This discussion is to explicitly ask the hosts to standardize the prediction strings so that models can be evaluated more reliably. Please upvote to increase the chances it gets done!**\n\n*I suggest a simple standardization: only ONE label for EACH mention. The label should be the longest possible piece of text that makes sense to the hosts but only ONE for ONE mention.*\n\nBelow I report a few cases where multiple labels are used in the train set even if the publication contains only one single mention. There are also some few cases where the labels in train are NOT referenced as datasets. I ran tests that makes me think the same ambiguity is present in the test set, and this has a large randomizing effect on the model evaluation and on the leaderboard.\n\n_______________________________________________\n\n**ID**: 002cbf56-5158-4ec7-83fd-51fa7829bb13\n**Problem**: Two labels but only One mention: `baltimore longitudinal study of aging blsa` and `baltimore longitudinal study of aging`\n> Schrack et al. showed an increase in REE with the number of chronic diseases in a study with \"healthy\" volunteers from the Baltimore Longitudinal Study of Aging (BLSA) cohort [36] .\n\n**ID**: 0035a1ba-6d1e-487b-bc3d-d49c5d64a3e9\n**Problem**: Two labels but only One mention: `baltimore longitudinal study of aging blsa` and `baltimore longitudinal study of aging`\n> The baseline images are selected from the Baltimore Longitudinal Study of Aging (BLSA) [48] data set. \n\n**ID**: 0085fd27-7924-41cb-b268-574356c3d6f7\n**Problem**: Two labels but only One mention: `national education longitudinal study` and `education longitudinal study`\n> Other empirical results from the National Education Longitudinal Study show that same-gender teaching increases the test scores of eighth-grade children (Dee, 2005b) .\n\n**ID**: 009a8e9d-f33c-4319-9f58-fc3e2e71e02b\n**Problem**: Two labels but only One mention: `adni` and `alzheimer s disease neuroimaging initiative adni`\n> Alzheimer's Disease Neuroimaging Initiative (ADNI) and post-mortem studies demonstrated evidence for the power of FDG PET as a biomarker for AD.\n\n**ID**: 00243b98-f868-45e4-9b83-0c346c7ecad5\n**Problem**: Incomplete label: `early childhood longitudinal study`\n> The investigators used data from the Early Childhood Longitudinal Study-Kindergarten Cohort (ECLS-K) to estimate whether and to what extent the timing and persistence of mathematics difficulties (MD) in kindergarten predicted children's first through fifth grade math growth trajectories.\n\n**ID**: 000efc17-13d8-433d-8f62-a3932fe4f3b8\n**Problem**: Missing label: train contains `adni` and `alzheimer s disease neuroimaging initiative adni`. The following label should also be in train but it misses: `alzheimer s disease neuroimaging initiative`\n> To address this, we used multimodal data from the National Alzheimer's Coordinating Center and Alzheimer's Disease Neuroimaging Initiative data sets. \n\n**ID**: 0010357a-6365-4e5f-b982-582e6d32c3ee\n**Porblem**: Wrong label: `genome sequence of covid 19` is not referenced as a dataset\n> The whole genome sequence of COVID -19 has about 50% similarity to that of MERS-CoV.\n\n@julialane @philculliton", "votes": null }, { "id": "1295606", "postDate": "05/06/2021 15:26:30", "content": "The whole genome sequence of COVID -19 has about 50% similarity to that of MERS-CoV.
\n
This kind of problem will greatly affect the enthusiasm of participants in the competition, and I am also a little frustrated because the text understanding model does not work well. 😨
", "rawMarkdown": "This kind of problem will greatly affect the enthusiasm of participants in the competition, and I am also a little frustrated because the text understanding model does not work well. 😨", "votes": null }, { "id": "1295654", "postDate": "05/06/2021 15:51:02", "content": "Asking for another round of re-annotations doesn't seem practical/realistic given that we're approaching the mid-point of the competition timeline and the current re-scoring isn't even complete. Annotating texts like these isn't trivial and I'm sure the organisers have been working like crazy to deliver the current changes.
\nA more practical change is to shift the metric back to f-score beta=0 so that False Negatives aren't penalised (so your concerns listed above are negated) and switching to using a split of the current private test-set for the public LB (with the cons noted as being outweighed by the positives).
", "rawMarkdown": "Asking for another round of re-annotations doesn't seem practical/realistic given that we're approaching the mid-point of the competition timeline and the current re-scoring isn't even complete. Annotating texts like these isn't trivial and I'm sure the organisers have been working like crazy to deliver the current changes.\n\nA more practical change is to shift the metric back to f-score beta=0 so that False Negatives aren't penalised (so your concerns listed above are negated) and switching to using a split of the current private test-set for the public LB (with the cons noted as being outweighed by the positives).", "votes": null }, { "id": "1295673", "postDate": "05/06/2021 16:07:35", "content": "I share your frustration.
", "rawMarkdown": "I share your frustration.", "votes": null }, { "id": "1295685", "postDate": "05/06/2021 16:21:43", "content": "If False Negatives aren't penalised, then submitting an empy file with a single correct prediction would achieve the top score (??)
\nI guess the labels were annoted that way because of the old metric. The metric was not penalizing False Negatives and the hosts wanted to give some extra chances to match the mentions. But changing the metric without standardizing the labels now makes the evaluation (partially) random. I see they are investing $90.000 in this task. It may deserve some extra work to get better models in the end.
", "rawMarkdown": "If False Negatives aren't penalised, then submitting an empy file with a single correct prediction would achieve the top score (??)\n\nI guess the labels were annoted that way because of the old metric. The metric was not penalizing False Negatives and the hosts wanted to give some extra chances to match the mentions. But changing the metric without standardizing the labels now makes the evaluation (partially) random. I see they are investing $90.000 in this task. It may deserve some extra work to get better models in the end.", "votes": null }, { "id": "1295966", "postDate": "05/06/2021 20:45:15", "content": "You are forced to make a prediction for every single ID so that's incorrect for the hidden test set. So the problem is more with the split since people are string matching for the training set.
", "rawMarkdown": "You are forced to make a prediction for every single ID so that's incorrect for the hidden test set. So the problem is more with the split since people are string matching for the training set.", "votes": null }, { "id": "1295977", "postDate": "05/06/2021 21:02:02", "content": "You can predict an empty string for each id (this is what I mean with empty file). That would achieve a score of 1.000. See also here. Fortunately this is fixed with the new evaluation metric. The problem is the inconsistency of the labels. Checking the training set manually, the inconsistency is self-evident (see above for just a few examples).
", "rawMarkdown": "You can predict an empty string for each id (this is what I mean with empty file). That would achieve a score of 1.000. See also [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/229812). Fortunately this is fixed with the new evaluation metric. The problem is the inconsistency of the labels. Checking the training set manually, the inconsistency is self-evident (see above for just a few examples).", "votes": null }, { "id": "1295981", "postDate": "05/06/2021 21:08:13", "content": "I don't think that's right. The link you're pointing to just says that people were able to submit correct submissions for every ID (not empty submissions) because of string matching.
\nFrom the evaluation description: \"For each publication Id in the test set, you must predict excerpts\".
", "rawMarkdown": "I don't think that's right. The link you're pointing to just says that people were able to submit correct submissions for every ID (not empty submissions) because of string matching.\n\nFrom the evaluation description: \"For each publication Id in the test set, you must predict excerpts\".", "votes": null }, { "id": "1295993", "postDate": "05/06/2021 21:27:25", "content": "It's correct. You can predict empty strings for each id. This would all be False Negatives. If not penalized for False Negatives, it's enough to predict only one single True Positive and you get a score of 1.000. That's pretty much what already happened and there is no way to get back to the old metric.
", "rawMarkdown": "It's correct. You can predict empty strings for each id. This would all be False Negatives. If not penalized for False Negatives, it's enough to predict only one single True Positive and you get a score of 1.000. That's pretty much what already happened and there is no way to get back to the old metric.", "votes": null }, { "id": "1296580", "postDate": "05/07/2021 10:58:13", "content": "Maybe that would be the solution: Counting empty stings as FP so that we would need to predict at least one correct label per publication!?
", "rawMarkdown": "Maybe that would be the solution: Counting empty stings as FP so that we would need to predict at least one correct label per publication!?", "votes": null }, { "id": "1296591", "postDate": "05/07/2021 11:07:09", "content": "Not all publications have at least one dataset in it
", "rawMarkdown": "Not all publications have at least one dataset in it", "votes": null }, { "id": "1296608", "postDate": "05/07/2021 11:21:48", "content": "In the test set? How do you know that?
\nIn this particular case a empty string should not count as a FP.
", "rawMarkdown": "In the test set? How do you know that?\n\nIn this particular case a empty string should not count as a FP.", "votes": null }, { "id": "1296629", "postDate": "05/07/2021 11:43:47", "content": "We can check for train. For test, I can imagine not all publication use at least one dataset.
\nIn any case, it seems to me that this solution implies that the optimal strategy would be to predict only the most likely dataset for each publication. Not really in line with the description page, isn't it?
", "rawMarkdown": "We can check for train. For test, I can imagine not all publication use at least one dataset.\n\nIn any case, it seems to me that this solution implies that the optimal strategy would be to predict only the most likely dataset for each publication. Not really in line with the description page, isn't it?", "votes": null }, { "id": "1296655", "postDate": "05/07/2021 12:07:49", "content": "Yes it is for sure not optimal, but better than what we have now IMO.
\nIt also seems to me that another annotation round is rather unrealistic, annotating 8000 publications is a huge undertaking.
", "rawMarkdown": "Yes it is for sure not optimal, but better than what we have now IMO.\n\nIt also seems to me that another annotation round is rather unrealistic, annotating 8000 publications is a huge undertaking.", "votes": null } ], "comments": [ { "id": 1295606, "author_name": "vanle73", "author_url": "", "post_date": "05/06/2021 15:26:30", "content": "This kind of problem will greatly affect the enthusiasm of participants in the competition, and I am also a little frustrated because the text understanding model does not work well. 😨
", "votes": null, "replies": [ { "id": 1295673, "author_name": "eguidotti", "author_url": "", "post_date": "05/06/2021 16:07:35", "content": "I share your frustration.
", "votes": null, "replies": [] } ] }, { "id": 1295654, "author_name": "leecming", "author_url": "", "post_date": "05/06/2021 15:51:02", "content": "Asking for another round of re-annotations doesn't seem practical/realistic given that we're approaching the mid-point of the competition timeline and the current re-scoring isn't even complete. Annotating texts like these isn't trivial and I'm sure the organisers have been working like crazy to deliver the current changes.
\nA more practical change is to shift the metric back to f-score beta=0 so that False Negatives aren't penalised (so your concerns listed above are negated) and switching to using a split of the current private test-set for the public LB (with the cons noted as being outweighed by the positives).
", "votes": null, "replies": [ { "id": 1295685, "author_name": "eguidotti", "author_url": "", "post_date": "05/06/2021 16:21:43", "content": "If False Negatives aren't penalised, then submitting an empy file with a single correct prediction would achieve the top score (??)
\nI guess the labels were annoted that way because of the old metric. The metric was not penalizing False Negatives and the hosts wanted to give some extra chances to match the mentions. But changing the metric without standardizing the labels now makes the evaluation (partially) random. I see they are investing $90.000 in this task. It may deserve some extra work to get better models in the end.
", "votes": null, "replies": [] }, { "id": 1295966, "author_name": "leecming", "author_url": "", "post_date": "05/06/2021 20:45:15", "content": "You are forced to make a prediction for every single ID so that's incorrect for the hidden test set. So the problem is more with the split since people are string matching for the training set.
", "votes": null, "replies": [] }, { "id": 1295977, "author_name": "eguidotti", "author_url": "", "post_date": "05/06/2021 21:02:02", "content": "You can predict an empty string for each id (this is what I mean with empty file). That would achieve a score of 1.000. See also here. Fortunately this is fixed with the new evaluation metric. The problem is the inconsistency of the labels. Checking the training set manually, the inconsistency is self-evident (see above for just a few examples).
", "votes": null, "replies": [] }, { "id": 1295981, "author_name": "leecming", "author_url": "", "post_date": "05/06/2021 21:08:13", "content": "I don't think that's right. The link you're pointing to just says that people were able to submit correct submissions for every ID (not empty submissions) because of string matching.
\nFrom the evaluation description: \"For each publication Id in the test set, you must predict excerpts\".
", "votes": null, "replies": [] }, { "id": 1295993, "author_name": "eguidotti", "author_url": "", "post_date": "05/06/2021 21:27:25", "content": "It's correct. You can predict empty strings for each id. This would all be False Negatives. If not penalized for False Negatives, it's enough to predict only one single True Positive and you get a score of 1.000. That's pretty much what already happened and there is no way to get back to the old metric.
", "votes": null, "replies": [] }, { "id": 1296580, "author_name": "hannes82", "author_url": "", "post_date": "05/07/2021 10:58:13", "content": "Maybe that would be the solution: Counting empty stings as FP so that we would need to predict at least one correct label per publication!?
", "votes": null, "replies": [] }, { "id": 1296591, "author_name": "eguidotti", "author_url": "", "post_date": "05/07/2021 11:07:09", "content": "Not all publications have at least one dataset in it
", "votes": null, "replies": [] }, { "id": 1296608, "author_name": "hannes82", "author_url": "", "post_date": "05/07/2021 11:21:48", "content": "In the test set? How do you know that?
\nIn this particular case a empty string should not count as a FP.
", "votes": null, "replies": [] }, { "id": 1296629, "author_name": "eguidotti", "author_url": "", "post_date": "05/07/2021 11:43:47", "content": "We can check for train. For test, I can imagine not all publication use at least one dataset.
\nIn any case, it seems to me that this solution implies that the optimal strategy would be to predict only the most likely dataset for each publication. Not really in line with the description page, isn't it?
", "votes": null, "replies": [] }, { "id": 1296655, "author_name": "hannes82", "author_url": "", "post_date": "05/07/2021 12:07:49", "content": "Yes it is for sure not optimal, but better than what we have now IMO.
\nIt also seems to me that another annotation round is rather unrealistic, annotating 8000 publications is a huge undertaking.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1295539": "Thanks to the hosts for the recent [revision to annotations](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236508)! Although not much details have been given, it seems the revision has been done mainly to improve the coverage of the [weakly labeled test set](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/233170). This is great and will help to score higher models that generalize better.\n\nUnfortunately, it seems the revision has not taken into account the ambiguity in the prediction string. Many discussions are going on around this (see e.g. [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/235822), [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/235752), and [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/231893)). \n**This discussion is to explicitly ask the hosts to standardize the prediction strings so that models can be evaluated more reliably. Please upvote to increase the chances it gets done!**\n\n*I suggest a simple standardization: only ONE label for EACH mention. The label should be the longest possible piece of text that makes sense to the hosts but only ONE for ONE mention.*\n\nBelow I report a few cases where multiple labels are used in the train set even if the publication contains only one single mention. There are also some few cases where the labels in train are NOT referenced as datasets. I ran tests that makes me think the same ambiguity is present in the test set, and this has a large randomizing effect on the model evaluation and on the leaderboard.\n\n_______________________________________________\n\n**ID**: 002cbf56-5158-4ec7-83fd-51fa7829bb13\n**Problem**: Two labels but only One mention: `baltimore longitudinal study of aging blsa` and `baltimore longitudinal study of aging`\n> Schrack et al. showed an increase in REE with the number of chronic diseases in a study with \"healthy\" volunteers from the Baltimore Longitudinal Study of Aging (BLSA) cohort [36] .\n\n**ID**: 0035a1ba-6d1e-487b-bc3d-d49c5d64a3e9\n**Problem**: Two labels but only One mention: `baltimore longitudinal study of aging blsa` and `baltimore longitudinal study of aging`\n> The baseline images are selected from the Baltimore Longitudinal Study of Aging (BLSA) [48] data set. \n\n**ID**: 0085fd27-7924-41cb-b268-574356c3d6f7\n**Problem**: Two labels but only One mention: `national education longitudinal study` and `education longitudinal study`\n> Other empirical results from the National Education Longitudinal Study show that same-gender teaching increases the test scores of eighth-grade children (Dee, 2005b) .\n\n**ID**: 009a8e9d-f33c-4319-9f58-fc3e2e71e02b\n**Problem**: Two labels but only One mention: `adni` and `alzheimer s disease neuroimaging initiative adni`\n> Alzheimer's Disease Neuroimaging Initiative (ADNI) and post-mortem studies demonstrated evidence for the power of FDG PET as a biomarker for AD.\n\n**ID**: 00243b98-f868-45e4-9b83-0c346c7ecad5\n**Problem**: Incomplete label: `early childhood longitudinal study`\n> The investigators used data from the Early Childhood Longitudinal Study-Kindergarten Cohort (ECLS-K) to estimate whether and to what extent the timing and persistence of mathematics difficulties (MD) in kindergarten predicted children's first through fifth grade math growth trajectories.\n\n**ID**: 000efc17-13d8-433d-8f62-a3932fe4f3b8\n**Problem**: Missing label: train contains `adni` and `alzheimer s disease neuroimaging initiative adni`. The following label should also be in train but it misses: `alzheimer s disease neuroimaging initiative`\n> To address this, we used multimodal data from the National Alzheimer's Coordinating Center and Alzheimer's Disease Neuroimaging Initiative data sets. \n\n**ID**: 0010357a-6365-4e5f-b982-582e6d32c3ee\n**Porblem**: Wrong label: `genome sequence of covid 19` is not referenced as a dataset\n> The whole genome sequence of COVID -19 has about 50% similarity to that of MERS-CoV.\n\n@julialane @philculliton", "1295606": "This kind of problem will greatly affect the enthusiasm of participants in the competition, and I am also a little frustrated because the text understanding model does not work well. 😨", "1295654": "Asking for another round of re-annotations doesn't seem practical/realistic given that we're approaching the mid-point of the competition timeline and the current re-scoring isn't even complete. Annotating texts like these isn't trivial and I'm sure the organisers have been working like crazy to deliver the current changes.\n\nA more practical change is to shift the metric back to f-score beta=0 so that False Negatives aren't penalised (so your concerns listed above are negated) and switching to using a split of the current private test-set for the public LB (with the cons noted as being outweighed by the positives).", "1295673": "I share your frustration.", "1295685": "If False Negatives aren't penalised, then submitting an empy file with a single correct prediction would achieve the top score (??)\n\nI guess the labels were annoted that way because of the old metric. The metric was not penalizing False Negatives and the hosts wanted to give some extra chances to match the mentions. But changing the metric without standardizing the labels now makes the evaluation (partially) random. I see they are investing $90.000 in this task. It may deserve some extra work to get better models in the end.", "1295966": "You are forced to make a prediction for every single ID so that's incorrect for the hidden test set. So the problem is more with the split since people are string matching for the training set.", "1295977": "You can predict an empty string for each id (this is what I mean with empty file). That would achieve a score of 1.000. See also [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/229812). Fortunately this is fixed with the new evaluation metric. The problem is the inconsistency of the labels. Checking the training set manually, the inconsistency is self-evident (see above for just a few examples).", "1295981": "I don't think that's right. The link you're pointing to just says that people were able to submit correct submissions for every ID (not empty submissions) because of string matching.\n\nFrom the evaluation description: \"For each publication Id in the test set, you must predict excerpts\".", "1295993": "It's correct. You can predict empty strings for each id. This would all be False Negatives. If not penalized for False Negatives, it's enough to predict only one single True Positive and you get a score of 1.000. That's pretty much what already happened and there is no way to get back to the old metric.", "1296580": "Maybe that would be the solution: Counting empty stings as FP so that we would need to predict at least one correct label per publication!?", "1296591": "Not all publications have at least one dataset in it", "1296608": "In the test set? How do you know that?\n\nIn this particular case a empty string should not count as a FP.", "1296629": "We can check for train. For test, I can imagine not all publication use at least one dataset.\n\nIn any case, it seems to me that this solution implies that the optimal strategy would be to predict only the most likely dataset for each publication. Not really in line with the description page, isn't it?", "1296655": "Yes it is for sure not optimal, but better than what we have now IMO.\n\nIt also seems to me that another annotation round is rather unrealistic, annotating 8000 publications is a huge undertaking." }, "source": "meta" }