There are only 130 'cleaned_label's / only 45 unique 'dataset_title's for the 14,316 training data files.
\n1st Problem: Even these 130 are not really unique, and contain many duplicates which are spelled differently.
\nFor example:
2nd Problem: Same file has multiple labels (multiple rows).
\n[Edit - not a problem - see CPMP comment and my update below]
\nFor example:
[Update] I realized that one file can have multiple labels as per evaluation page which mentioned that \"Multiple predictions are delineated with a pipe (|) character in the submission file\" and gives below example:
\n01e4e08c-ffea-45a7-adde-6a0c0ad755fc,space location data|national space objects|national space dataset3rd Problem: Inconsistent naming where some labels contain shortnames instead of full name such as \"xx yy zz dataset\".
\n4th Problem: Whether to include word 'data' / 'dataset' in the label or not? (in cases where those words are present in the corresponding text)
\nFor example:
5th Problem: Ambiguous and unexpected labels are used.
\nFor example:
\nFile d4b155e4-e351-4bea-a81e-f558d1f2882a contains several phrases such as:
Another example file 3bd0f678-7a16-4c6b-8692-3a57f2d7d0d1 specifies the label as \"advanced national seismic system comprehensive catalog\", whereas the paper actually has many more references to a different dataset as the primary source of data \"U.S. Geological Survey (USGS) \"Did You Feel It?\" (DYFI) data\". Why was DYFI data not included in the label? Here also, even if we correctly identify \"DYFI data\", we will still get a \"0\" score for this file.
\n@julialane @maggiemd Could you please let us know about the steps used for generating train data 'cleaned_label's? I was really expecting a \"single\" correct label per dataset (and a single prediction per file when only one dataset is present).
\nPS: In cases where the same file/paper references multiple datasets, then the \"pipe dilineation\" is appropriate (still using only one label for each single dataset).
There are only 130 'cleaned_label's / only 45 unique 'dataset_title's for the 14,316 training data files.
\n1st Problem: Even these 130 are not really unique, and contain many duplicates which are spelled differently.
\nFor example:
2nd Problem: Same file has multiple labels (multiple rows).
\n[Edit - not a problem - see CPMP comment and my update below]
\nFor example:
[Update] I realized that one file can have multiple labels as per evaluation page which mentioned that \"Multiple predictions are delineated with a pipe (|) character in the submission file\" and gives below example:
\n01e4e08c-ffea-45a7-adde-6a0c0ad755fc,space location data|national space objects|national space dataset3rd Problem: Inconsistent naming where some labels contain shortnames instead of full name such as \"xx yy zz dataset\".
\n4th Problem: Whether to include word 'data' / 'dataset' in the label or not? (in cases where those words are present in the corresponding text)
\nFor example:
5th Problem: Ambiguous and unexpected labels are used.
\nFor example:
\nFile d4b155e4-e351-4bea-a81e-f558d1f2882a contains several phrases such as:
Another example file 3bd0f678-7a16-4c6b-8692-3a57f2d7d0d1 specifies the label as \"advanced national seismic system comprehensive catalog\", whereas the paper actually has many more references to a different dataset as the primary source of data \"U.S. Geological Survey (USGS) \"Did You Feel It?\" (DYFI) data\". Why was DYFI data not included in the label? Here also, even if we correctly identify \"DYFI data\", we will still get a \"0\" score for this file.
\n@julialane @maggiemd Could you please let us know about the steps used for generating train data 'cleaned_label's? I was really expecting a \"single\" correct label per dataset (and a single prediction per file when only one dataset is present).
\nPS: In cases where the same file/paper references multiple datasets, then the \"pipe dilineation\" is appropriate (still using only one label for each single dataset).
2nd is not a problem, it is a feature. A paper can refer to multiple datasets.
", "rawMarkdown": "2nd is not a problem, it is a feature. A paper can refer to multiple datasets.", "votes": null }, { "id": "1250718", "postDate": "03/24/2021 08:22:51", "content": "Also, there is an additional difficulty (it could be considered to be a labeling error, I suppose):
\nThere are overlapping entities. For example in text f8b03c87-9d1a-4f20-b76b-cb6c69d447b2 There is an entity national education longitudinal study and inside it is an entity education longitudinal study. Thus, common approaches to NER can't work because they assume that one token can belong only to one entity.
Exactly, was gonna open an issue regarding this
", "rawMarkdown": "Exactly, was gonna open an issue regarding this", "votes": null }, { "id": "1251136", "postDate": "03/24/2021 14:10:31", "content": "Good questions! There are many potential labels for a given dataset, as each label is literally the context of the dataset in a document. So a given dataset may be referred to in many different ways by many documents. A given dataset may also be referred to multiple times within a document.
\nThe test data contains a fairly huge number of unseen datasets and dataset labels. The idea is to generalize to them. Train deliberately is a) not exhaustively labeled (i.e. there are some labels that have not been identified) and b) contains enough to work on the generalization problem but not just string-match against test.
\nRe: your 4th problem, the idea of using Jaccard with a threshold of 0.5 to determine true positives is that sometimes it's ambiguous as to which context words define a dataset. Leaving out or including \"data\" in any of those examples would still match against the context.
\nRe: the 5th problem, we're hoping to find uniquely identifiable mentions within each publication. Your five top examples are extremely ambiguous as to which dataset they might refer to.
\nThanks for your questions! Hopefully that helps a bit!
", "rawMarkdown": "Good questions! There are many potential labels for a given dataset, as each label is literally the context of the dataset in a document. So a given dataset may be referred to in many different ways by many documents. A given dataset may also be referred to multiple times within a document.\n\nThe test data contains a fairly huge number of unseen datasets and dataset labels. The idea is to generalize to them. Train deliberately is a) not exhaustively labeled (i.e. there are some labels that have not been identified) and b) contains enough to work on the generalization problem but not just string-match against test.\n\nRe: your 4th problem, the idea of using Jaccard with a threshold of 0.5 to determine true positives is that sometimes it's ambiguous as to which context words define a dataset. Leaving out or including \"data\" in any of those examples would still match against the context.\n\nRe: the 5th problem, we're hoping to find uniquely identifiable mentions within each publication. Your five top examples are extremely ambiguous as to which dataset they might refer to.\n\nThanks for your questions! Hopefully that helps a bit!", "votes": null }, { "id": "1251693", "postDate": "03/25/2021 03:15:41", "content": "@philculliton @julialane could you please clarify problems 3 and 5 further:
\nProblem 3: In some cases shorter names are used, whereas in other cases longer names are used (by prepending with the source institution name) - so even if we extract the dataset name correctly, we will get a \"0\" score for that item.
\nProblem 5: I have added another example - could you please reply why \"DYFI data\", which appears to be more important in that paper compared to \"comprehensive catalog\" is not included in the label? Here also, even if we correctly identify \"DYFI data\", we will still get a \"0\" score for this file.
\nHi @sirishks - thanks for the follow-up!
\nRe: problem 3 - the training labels are noisy and, as mentioned below, not exhaustive. They're meant as a guide for modeling semantic context. Discovering the unlabeled or noisily-labeled datasets in train will help you in this task. You are not intended to find these same datasets in test using these precise labels. Test, on the other hand, is exhaustively labeled and the labels have been cleaned up.
\nRe: problem 5 - the training data is not exhaustively labeled. There are potentially many dataset mentions (and datasets) in train that are not labeled at all.
", "rawMarkdown": "Hi @sirishks - thanks for the follow-up!\n\nRe: problem 3 - the training labels are noisy and, as mentioned below, not exhaustive. They're meant as a guide for modeling semantic context. Discovering the unlabeled or noisily-labeled datasets in train will help you in this task. You are not intended to find these same datasets in test using these precise labels. Test, on the other hand, is exhaustively labeled and the labels have been cleaned up.\n\nRe: problem 5 - the training data is not exhaustively labeled. There are potentially many dataset mentions (and datasets) in train that are not labeled at all.", "votes": null }, { "id": "1253341", "postDate": "03/26/2021 16:21:04", "content": "@philculliton @julialane but this will result in a very unreliable public leaderboard (because good models can have low public lb scores but high private lb scores), and our scores and ranks will be useless in estimating the final private lb rankings, and will lead to a large shake up in Kaggle terminology, which is not a desired result ..
", "rawMarkdown": "philculliton @julialane but this will result in a **very unreliable** public leaderboard (because good models can have low public lb scores but high private lb scores), and our scores and ranks will be useless in estimating the final private lb rankings, and will lead to a **large shake up** in Kaggle terminology, which is **not a desired** result ..", "votes": null }, { "id": "1253539", "postDate": "03/26/2021 20:59:45", "content": "Hi @sirishks -
\nThanks for the feedback! Given the structure of the public leaderboard I expect good scores to indicate better models. I suspect a good model will be a) solving the private-test-like cases in public (and doing better than the baseline because of it) and therefore b) solving the private test well.
\nThis is an uncommon approach to competitions, but for the problem - generalizing information extraction for NLP - I think it's a reasonable way to do it. It's not enough to simply model train and do well on test, but if you can sort out the problems in train and apply that to test it should help you.
", "rawMarkdown": "Hi @sirishks -\n\nThanks for the feedback! Given the structure of the public leaderboard I expect good scores to indicate better models. I suspect a good model will be a) solving the private-test-like cases in public (and doing better than the baseline because of it) and therefore b) solving the private test well.\n\nThis is an uncommon approach to competitions, but for the problem - generalizing information extraction for NLP - I think it's a reasonable way to do it. It's not enough to simply model train and do well on test, but if you can sort out the problems in train and apply that to test it should help you.", "votes": null }, { "id": "1254122", "postDate": "03/27/2021 11:13:10", "content": "Crazy thing is, by matching that 45 datasets and their variances one could get to ~0.7. I sincerely hope they're messing with the public test set on purpose, if that distribution holds with the private set it will be ridiculous. In fact, the score returned for that method should be 0.0.
\nThat was 45 samples vs the whole world, Jesus.
\nDefinitely not having a good feeling about this.
@philculliton @julialane you said
\n> Test, on the other hand, is exhaustively labeled and the labels have been cleaned up.
\nI can confirm that even this is not correct - (i) labels are still ambiguous (random substrings are being used), (ii) also a single Test Id (public LB) still uses multiple labels for the SAME dataset for the same file.
I ran an experiment where I got a higher score when using multiple labels for the same dataset within the same PredictionString:
\n(A similar example is also present in train/6d2ff020-c87b-4abc-a757-0b7ba25a87fa.json where even though the file actually contains the full label \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\", but train.csv contains multiple labels i.e 2 rows for the SAME dataset for the same Id/file. The substring \"adni\" is provided as a valid training label, which is ambiguous.)
\nThe above example is similar to the training label mentioned in CPMP's topic, where he was referring to presence of both 'Aging Integrated Database (AGID)' as well as 'Aging Integrated Database' in the same file.
\nMy findings:
\nI accept your comments about handling noisy real world data, but there should not be ambiguity in expected output. So, I strongly disagree with your previous replies, as I have found that test labels in Leaderboard evaluation are still ambiguous (not cleaned up).
My recommendation:
\nFor each dataset,
\n(i) use the \"full dataset name\" i.e. \"longest possible label\" which is actually present in the corresponding file, and remove all substrings from expected PredictionString and
\n(ii) ensure that we have a single longest possible label per dataset per file (no duplicates / multiple labels for same dataset in same file).
\nThis will help remove ambiguity in LB predictions, and we will be able to build models which are actually useful!
@philculliton @julialane Please feel free to repeat my experiment and confirm my findings yourselves.
", "rawMarkdown": "philculliton @julialane you said\n`> Test, on the other hand, is exhaustively labeled and the labels have been cleaned up.`\nI can confirm that even **this is not correct** - (i) labels are still ambiguous (random substrings are being used), (ii) also a single Test Id (public LB) still uses **multiple labels for the SAME dataset** for the same file.\n\nI ran an experiment where I got a *higher score* when using **multiple labels for the same dataset within the same PredictionString**:\n- **Experiment 1**: only submitted the longest label (ignoring substrings), present in the file, i.e. single label for a single dataset\n PredictionString=*'alzheimer s disease neuroimaging initiative adni'*\n- **Experiment 2**: submitted multiple/all possible labels for each dataset (after ensuring that these multiple labels/substrings are actually present in the file)\n PredictionString=*'alzheimer s disease neuroimaging initiative|adni|alzheimer s disease neuroimaging initiative adni'*\n(public test file 2100032a-7c33-4bff-97ef-690822c43466)\n\n(A similar example is also present in train/6d2ff020-c87b-4abc-a757-0b7ba25a87fa.json where even though the file actually contains the full label \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\", but **train.csv contains multiple labels i.e 2 rows for the SAME dataset for the same Id/file**. The substring \"adni\" is provided as a valid training label, which **is ambiguous**.) \n\nThe above example is similar to the training label mentioned in [CPMP's topic](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228337), where he was referring to presence of both 'Aging Integrated Database (AGID)' as well as 'Aging Integrated Database' in the same file.\n\n**My findings:**\nI accept your comments about handling noisy real world data, but there should not be ambiguity in expected output. So, I strongly disagree with your previous replies, as I have found that **test labels in Leaderboard evaluation are still ambiguous (not cleaned up)**.\n\n**My recommendation:**\nFor each dataset,\n(i) use the \"full dataset name\" i.e. \"longest possible label\" which is actually present in the corresponding file, and **remove all substrings** from expected PredictionString and \n(ii) ensure that we have a **single** longest possible label per dataset per file (no duplicates / multiple labels for same dataset in same file).\nThis will help remove ambiguity in LB predictions, and we will be able to build models which are actually useful!\n\n@philculliton @julialane Please feel free to repeat my experiment and confirm my findings yourselves.", "votes": null }, { "id": "1254870", "postDate": "03/28/2021 07:31:45", "content": "I sincerely hope they're messing with the public test set on purpose
\nThey are also messing with the private Test set / LB results - see findings of my experiments below.
There are indeed 45 dataset titles in the training datasets, but 130 dataset labels because authors in publications refer to the datasets in different ways. We accept multiple different possibilities. For example, we accept both ADNI and Alzheimer's Disease Neuroimaging Initiative (ADNI) – we are not penalizing for predicting only ADNI.
\nAs @philculliton has already pointed out, the private leaderboard does not contain any labels from the training set. So constructing a lookup table of the training datasets will not be useful. If you look at the intent of the challenge, it is to find dataset references from the semantic context. I expect @philculliton will respond in more depth on Monday.
\nI hope this helps, and we are thrilled by the degree of interest and the thoughtfulness of your questions.
Hi Julia, I still feel quite unclear about this:
\n\n\nwe accept both ADNI and Alzheimer's Disease Neuroimaging Initiative (ADNI) – we are not penalizing for predicting only ADNI.
\n
According to the evaluation page, if I predict both \"ADNI\" and \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\" for a given sample, and the label is just \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\", the \"ADNI\" that I predicted will be added to the FP pool, thus lowering my score.
\n\n", "rawMarkdown": "Hi Julia, I still feel quite unclear about this:\n> we accept both ADNI and Alzheimer's Disease Neuroimaging Initiative (ADNI) – we are not penalizing for predicting only ADNI.\n\nAccording to the evaluation page, if I predict both \"ADNI\" and \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\" for a given sample, and the label is just \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\", the \"ADNI\" that I predicted will be added to the FP pool, thus lowering my score.\n\n> Any matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are counted as true positives (TP), the remainder as false positives (FP).", "votes": null }, { "id": "1255150", "postDate": "03/28/2021 13:56:03", "content": "Any matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are counted as true positives (TP), the remainder as false positives (FP).
\n
@julialane - thank you for your reply and we hope that @philculliton will address the actual score discrepancies observed in my experiments soon.
\nI want to emphasize my recommendation again viz that even if labels differ from file to file, within a specific file / Id, there should be only a single (longest possible) label for each dataset (and penalize substring labels or multiple labels for same dataset).
\nAnother thought is about your reply that you
\naccept both ADNI and Alzheimer's Disease Neuroimaging Initiative (ADNI) as correct,
\nwhich still results in some ambiguity because other labels are not so simple.
\nPerhaps you can instead remove the Jaccard threshold of 0.5 so that someone who predicts just ADNI will still get partial credit for the answer.
@sirishks @suicaokhoailang - just want to note that the private set is essentially disjoint and there's no label-matching crossover between it at train. That 0.7 score is literally getting the training labels right in public test. No such effect exists in private test.
\nTrain (and to a degree the public test) is intended as a guide / workset for the difficult task of labeling the private set. String matching isn't part of that solution.
", "rawMarkdown": "sirishks @suicaokhoailang - just want to note that the private set is essentially disjoint and there's no label-matching crossover between it at train. That 0.7 score is literally getting the training labels right in public test. No such effect exists in private test.\n\nTrain (and to a degree the public test) is intended as a guide / workset for the difficult task of labeling the private set. String matching isn't part of that solution.", "votes": null }, { "id": "1256084", "postDate": "03/29/2021 14:30:42", "content": "@philculliton It is great that private test set is disjoint.
\nHowever, my question is why we are only getting 0.7 score and not a perfect 1.0 score on public LB?
\nI believe that it is because you are (i) allowing multiple labels for same dataset for the same file and (ii) allowing substrings.
There is definitely a bug in your LB score (jaccard) calculation because I expected a higher score when submitting the \"longest label\" correctly, but instead noticed that \"multiple substring labels\" gave a higher score.
\nPlease see my experiments below and my recommendations for making this competition better.
\nPerhaps you can reply using an actual example, say public test file 2100032a-7c33-4bff-97ef-690822c43466, and show (i) what is the correct expected PredictionString ('alzheimer s disease neuroimaging initiative adni') for this file, and (ii) how you are incorporating Julia's answer (accepting different strings as correct) while computing jaccard score?, along with (iii) example combinations of substrings such as 'alzheimer s disease neuroimaging initiative' or 'adni' or 'alzheimer s disease neuroimaging initiative|adni' and show how much score will be given to each of these partially correct answers (some with incorrect numerator and others with incorrect denominator in the score formula).
", "rawMarkdown": "philculliton It is great that private test set is disjoint.\n\nHowever, my question is why we are only getting 0.7 score and not a perfect 1.0 score on public LB?\nI believe that it is because you are (i) allowing multiple labels for same dataset **for the same file** and (ii) allowing substrings.\n\nThere is definitely a **bug in your LB score (jaccard) calculation** because I expected a higher score when submitting the \"longest label\" correctly, but instead noticed that \"multiple substring labels\" gave a higher score. \n\nPlease see my experiments below and my recommendations for making this competition better.\n\nPerhaps you can reply using an actual example, say public test file 2100032a-7c33-4bff-97ef-690822c43466, and show (i) what is the correct expected PredictionString ('alzheimer s disease neuroimaging initiative adni') for this file, and (ii) how you are incorporating Julia's answer (accepting different strings as correct) while computing jaccard score?, along with (iii) example combinations of substrings such as 'alzheimer s disease neuroimaging initiative' or 'adni' or 'alzheimer s disease neuroimaging initiative|adni' and show how much score will be given to each of these partially correct answers (some with incorrect numerator and others with incorrect denominator in the score formula).", "votes": null }, { "id": "1256089", "postDate": "03/29/2021 14:35:25", "content": "I had the same concern which is why I am so picky on differing labels for same dataset.
\n\n\nAccording to the evaluation page, if I predict both \"ADNI\" and \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\" for a given sample, and the label is just \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\", the \"ADNI\" that I predicted will be added to the FP pool, thus lowering my score.
\n
Any matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are counted as true positives (TP), the remainder as false positives (FP).\n@sirishks - the primary reason you're getting 0.7 and not 1.0 on the public leaderboard is because there are a large number of labels in public test that are not included in train. Every public test sample has at least one label that has not been identified.
\nI am looking into your recommendations and experiments, and discussing with the host, but the expectation that you'll get 1.0 simply by getting all training labels correct is invalid. That's not the task.
", "rawMarkdown": "sirishks - the primary reason you're getting 0.7 and not 1.0 on the public leaderboard is because there are a large number of labels in public test that are not included in train. Every public test sample has at least one label that has not been identified.\n\nI am looking into your recommendations and experiments, and discussing with the host, but the expectation that you'll get 1.0 simply by getting all training labels correct is invalid. That's not the task.", "votes": null }, { "id": "1268155", "postDate": "04/09/2021 07:08:00", "content": "I think this answer of your's should be made a seperate thread , because it clears the confusion all are having about datasets labels etc , our task is not string matching , but some understanding of semantic context for dataset labels usage , that's why straightforward matching gets 0.7 but not 1.0.
", "rawMarkdown": "I think this answer of your's should be made a seperate thread , because it clears the confusion all are having about datasets labels etc , our task is not string matching , but some understanding of semantic context for dataset labels usage , that's why straightforward matching gets 0.7 but not 1.0.", "votes": null }, { "id": "1268513", "postDate": "04/09/2021 13:28:31", "content": "@philculliton Could you elaborate a little more on what you mean by:
\n \"Test, on the other hand, is exhaustively labeled and the labels have been cleaned up.\"\nDoes 'exhaustively labeled' mean:
\nIf #1, how do you choose between different references for the same dataset (first, longest, arbitrary)? If #2, why does the LB score get worse when we include alternative forms of the same dataset?
\nAnd what does 'cleaned up' mean?
", "rawMarkdown": "philculliton Could you elaborate a little more on what you mean by:\n\n \"Test, on the other hand, is exhaustively labeled and the labels have been cleaned up.\"\n\nDoes 'exhaustively labeled' mean:\n\n1. Each unique dataset that is mentioned in a record is labeled.\n2. All unique references of each dataset (including different references for the same dataset) are labeled.\n\nIf #1, how do you choose between different references for the same dataset (first, longest, arbitrary)? If #2, why does the LB score get worse when we include alternative forms of the same dataset?\n\nAnd what does 'cleaned up' mean?", "votes": null }, { "id": "1268831", "postDate": "04/09/2021 20:06:18", "content": "In addition, I am curious if only datasets that were used by the authors to produce their publication should be labeled. For example, if the authors reference a dataset that they did not use in the present publication (maybe when discussing previous work) should this dataset be labeled?
", "rawMarkdown": "In addition, I am curious if only datasets that were used by the authors to produce their publication should be labeled. For example, if the authors reference a dataset that they did not use in the present publication (maybe when discussing previous work) should this dataset be labeled?", "votes": null }, { "id": "1270869", "postDate": "04/12/2021 05:40:35", "content": "@julialane @philculliton It's been 2-weeks since you said
\nI am looking into your recommendations and experiments, and discussing with the host
Did you make a decision regarding my recommendations viz,
\nWhether you adopt my recommendations or not, below question still requires to be clarified:
\nExample scenario: Let's use the longest single label as ground truth = 'alzheimer s disease neuroimaging initiative adni'
Aiming for semantic understanding of dataset mentions is pretty ambitious. Is the competition (public LB etc) currently set up to encourage that?
\nI’m currently ranked pretty high on the public LB and I’m skeptical that a top public LB submission translates (generalizes) to the private LB setting i.e., unseen publications and labels.
\nI’d suggest reconfiguring the LB so both public and private LB are based on the disjoint publications.
\nThe last thing both the organizers and competitors want is a bunch of solutions that are “gamed” for the public LB and don’t generalize well leading to a massive shakeup.
", "rawMarkdown": "julialane @philculliton \n\nAiming for semantic understanding of dataset mentions is pretty ambitious. Is the competition (public LB etc) currently set up to encourage that?\n\nI’m currently ranked pretty high on the public LB and I’m skeptical that a top public LB submission translates (generalizes) to the private LB setting i.e., unseen publications and labels.\n\nI’d suggest reconfiguring the LB so both public and private LB are based on the disjoint publications.\n\nThe last thing both the organizers and competitors want is a bunch of solutions that are “gamed” for the public LB and don’t generalize well leading to a massive shakeup.", "votes": null }, { "id": "1273400", "postDate": "04/14/2021 10:07:26", "content": "I second this. I still can't understand the decision to use a part of the training data as public test, LB score is practically useless atm.
", "rawMarkdown": "I second this. I still can't understand the decision to use a part of the training data as public test, LB score is practically useless atm.", "votes": null }, { "id": "1273863", "postDate": "04/14/2021 17:51:54", "content": "Hi @leecming - great question, thanks! The public test set is currently set up to try and reward generalization. A significant percentage of the public test set is not crossover with the training set (and involves a large number of datasets / labels not seen in train). Better scores on the LB will be solving for those examples.
", "rawMarkdown": "Hi @leecming - great question, thanks! The public test set is currently set up to try and reward generalization. A significant percentage of the public test set is not crossover with the training set (and involves a large number of datasets / labels not seen in train). Better scores on the LB will be solving for those examples.", "votes": null }, { "id": "1274052", "postDate": "04/15/2021 00:03:50", "content": "\nThanks for the explanation (and I don't think downvoting a constructive response is helpful).
\nUltimately, it boils down to - does the public LB correlate well to private LB?
\nAll I'd ask is for the organizers to keep an eye on this and consider updating the split if necessary (and earlier rather than later in the competition).
\nThe task is difficult enough that we don't need an additional layer of obfuscation/complexity in trying to translate public LB to private LB performance from our end.
", "rawMarkdown": "philculliton \n\nThanks for the explanation (and I don't think downvoting a constructive response is helpful).\n\nUltimately, it boils down to - does the public LB correlate well to private LB?\n\nAll I'd ask is for the organizers to keep an eye on this and consider updating the split if necessary (and earlier rather than later in the competition). \n\nThe task is difficult enough that we don't need an additional layer of obfuscation/complexity in trying to translate public LB to private LB performance from our end.", "votes": null }, { "id": "1274142", "postDate": "04/15/2021 03:21:41", "content": "\n\n\nUltimately, it boils down to - does the public LB correlate well to private LB?
\n
Is this desirable? LB probing is a plague and maybe the only issue with Kaggle competition.
\nWhat is desirable is a private LB that measures generalization. For instance, in time series competitions, having public LB the next period, say next week, and private LB the following period, i.e. second week, is a much better way to assess model performance than a public LB based on a random sampling of next two periods. Then you need to set a local validation that evaluates the model ability to predict well 2 weeks ahead.
\nHere it means that we need to set a local validation and rely on it rather than LB probing. To your point this may be a little too ambitious for a Kaggle competition as we have only very weak labels.
\nI have been thinking about how to do this and will re enter only if I find a way (and if the ground truth issues that keep popping up in the forum are clarified via some examples).
", "rawMarkdown": "leecming\n\n> Ultimately, it boils down to - does the public LB correlate well to private LB?\n\nIs this desirable? LB probing is a plague and maybe the only issue with Kaggle competition.\n\nWhat is desirable is a private LB that measures generalization. For instance, in time series competitions, having public LB the next period, say next week, and private LB the following period, i.e. second week, is a much better way to assess model performance than a public LB based on a random sampling of next two periods. Then you need to set a local validation that evaluates the model ability to predict well 2 weeks ahead.\n\nHere it means that we need to set a local validation and rely on it rather than LB probing. To your point this may be a little too ambitious for a Kaggle competition as we have only very weak labels.\n\nI have been thinking about how to do this and will re enter only if I find a way (and if the ground truth issues that keep popping up in the forum are clarified via some examples).", "votes": null }, { "id": "1274146", "postDate": "04/15/2021 03:39:53", "content": "Allowing for an open-ended number of text predictions per document and setting up a disjointed hidden test set means that it's much harder to probe (versus a vanilla classification/regression task).
\nI get the tension between having a meaningful public LB and preventing probing but imo, this competition is on the wrong end of the continuum.
", "rawMarkdown": "Allowing for an open-ended number of text predictions per document and setting up a disjointed hidden test set means that it's much harder to probe (versus a vanilla classification/regression task). \n\nI get the tension between having a meaningful public LB and preventing probing but imo, this competition is on the wrong end of the continuum.", "votes": null }, { "id": "1274147", "postDate": "04/15/2021 03:39:57", "content": "@philculliton
\nHere's my two cents. It's a terrible idea to blindly trust the LB, however the LB can still show you a rough idea of where you're standing compared to other competitors. As I said in my thread, the LB here is dead due to how the test data is set up, it's impossible to know how others are doing, what the real baseline may look like or whether our current model can be even considered working. In the last couple of weeks we tried really hard to make a model that we * think* may generalize quite well to unseen data, and yet it hasn't even beaten the string matching method.
\nI agree with @leecming that validation in this competition is hard enough, we don't need this.
\nMy suggestion is to make the public LB totally disjoint from train and reduce the submission limit to 2 per day.
Another way of thinking about it is this -
\nfor it to be possible to conduct meaningful cross validation locally, we should have gotten the full label set for train (and presumably a different public test LB). How do you do proper CV if your hold-out set isn't complete.
\n\nI’d suggest reconfiguring the LB so both public and private LB are based on the disjoint publications.
\n
This is a bad idea in general since the best public LB probers will win on private LB.
\nBut I agree on all the other comments mentioned here.
There may not be an easy solution for this competition setup but I just don't get the logic of how the current setup is useful. It breaks local validation and it breaks public LB usefulness.
\nThe bare minimum to ensure model generalization I would think would be to have train, public LB and private LB, each independently complete but pairwise disjoint (on publications). Having labels also disjoint might be infeasible from host's perspective but if that is also possible, nothing like it.
\nOr maybe we need a detailed explanation from hosts about why the current setup is ideal to build a solution that makes business sense. Can't seem to find a way to convince myself.
", "rawMarkdown": "> I’d suggest reconfiguring the LB so both public and private LB are based on the disjoint publications.\n\nThis is a bad idea in general since the best public LB probers will win on private LB.\nBut I agree on all the other comments mentioned here.\n\nThere may not be an easy solution for this competition setup but I just don't get the logic of how the current setup is useful. It breaks local validation and it breaks public LB usefulness.\n\nThe bare minimum to ensure model generalization I would think would be to have train, public LB and private LB, each independently complete but pairwise disjoint (on publications). Having labels also disjoint might be infeasible from host's perspective but if that is also possible, nothing like it.\n\nOr maybe we need a detailed explanation from hosts about why the current setup is ideal to build a solution that makes business sense. Can't seem to find a way to convince myself.", "votes": null }, { "id": "1274172", "postDate": "04/15/2021 04:21:53", "content": "My biggest concern is that I doubt the hidden test data is exhaustively labeled. This is why non-string matching models, which generalize, are doing so poorly. They are capturing legitimate dataset mentions that are not labeled. Good generalization, what the host obviously wants, is actually being discouraged.
\nWould it be possible for Kaggle staff to take a close look at the hidden test data to see if my suspicion about unlabeled dataset mentions is valid? If it's valid, then I don't see how this is anything other than a string matching competition.
", "rawMarkdown": "My biggest concern is that I doubt the hidden test data is exhaustively labeled. This is why non-string matching models, which generalize, are doing so poorly. They are capturing legitimate dataset mentions that are not labeled. Good generalization, what the host obviously wants, is actually being discouraged.\n\nWould it be possible for Kaggle staff to take a close look at the hidden test data to see if my suspicion about unlabeled dataset mentions is valid? If it's valid, then I don't see how this is anything other than a string matching competition.", "votes": null }, { "id": "1274407", "postDate": "04/15/2021 09:25:40", "content": "\n\nHow do you do proper CV if your hold-out set isn't complete.
\n
More and more Kaggle competitions are like this, BirdCLEF and Human Protein Atlas are weakly labelled too for instance.
\nThe days of supervised learning are over, state of the art is moving to semi supervised learning and self supervised learning. So is Kaggle.
", "rawMarkdown": "> How do you do proper CV if your hold-out set isn't complete. \n\nMore and more Kaggle competitions are like this, BirdCLEF and Human Protein Atlas are weakly labelled too for instance. \n\nThe days of supervised learning are over, state of the art is moving to semi supervised learning and self supervised learning. So is Kaggle.", "votes": null }, { "id": "1274409", "postDate": "04/15/2021 09:26:07", "content": "\n\nI doubt the hidden test data is exhaustively labeled
\n
On what ground?
", "rawMarkdown": "> I doubt the hidden test data is exhaustively labeled\n\nOn what ground?", "votes": null }, { "id": "1274561", "postDate": "04/15/2021 12:12:03", "content": "I am doubting the hidden test data is exhaustively labeled based on the discussion topic here. Since I created that topic, I've discovered other datasets that do not appear to be labeled in the hidden test data.
", "rawMarkdown": "I am doubting the hidden test data is exhaustively labeled based on the discussion topic [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/231893). Since I created that topic, I've discovered other datasets that do not appear to be labeled in the hidden test data.", "votes": null }, { "id": "1274572", "postDate": "04/15/2021 12:28:44", "content": "Human annotations for NLP tasks are inherently noisy and I expect there to be a fair number of labelling errors/misses even with a rigorous multi-annotator methodology. Just eyeball some of the longer training docs with a lot of formatting errors and you'd be hard-pressed to pick out all the labels.
\nThe original metric of using an f-measure with 0 beta would have mitigated that by allowing us to pick a single, high confidence prediction for each ID. Switching to 0.5 beta was a kludge so that the LB wouldn't be filled with perfect scores but the unintended consequence was to make the actual hidden test set much harder to solve.
\nI guess this is a long-winded way to say \"good luck to everyone, we'll need it\"
", "rawMarkdown": "Human annotations for NLP tasks are inherently noisy and I expect there to be a fair number of labelling errors/misses even with a rigorous multi-annotator methodology. Just eyeball some of the longer training docs with a lot of formatting errors and you'd be hard-pressed to pick out all the labels. \n\nThe original metric of using an f-measure with 0 beta would have mitigated that by allowing us to pick a single, high confidence prediction for each ID. Switching to 0.5 beta was a kludge so that the LB wouldn't be filled with perfect scores but the unintended consequence was to make the actual hidden test set much harder to solve. \n\nI guess this is a long-winded way to say \"good luck to everyone, we'll need it\"", "votes": null }, { "id": "1274596", "postDate": "04/15/2021 12:56:44", "content": "\n\nI've discovered other datasets that do not appear to be labeled in the hidden test data.
\n
How can you see the hidden dataset?
", "rawMarkdown": "> I've discovered other datasets that do not appear to be labeled in the hidden test data.\n\nHow can you see the hidden dataset?", "votes": null }, { "id": "1274600", "postDate": "04/15/2021 12:58:10", "content": "\n\nSwitching to 0.5 beta was a kludge so that the LB wouldn't be filled with perfect scores but the unintended consequence was to make the actual hidden test set much harder to solve.
\n
You nailed it. All questions asked about ground truth are indeed due to this change. Because it is unclear how redundant predictions are handled now.
", "rawMarkdown": "> Switching to 0.5 beta was a kludge so that the LB wouldn't be filled with perfect scores but the unintended consequence was to make the actual hidden test set much harder to solve. \n\nYou nailed it. All questions asked about ground truth are indeed due to this change. Because it is unclear how redundant predictions are handled now.", "votes": null }, { "id": "1274639", "postDate": "04/15/2021 13:41:15", "content": "CPMP, I cannot see the hidden dataset. The hidden dataset is not definitively missing labels. My observations are demonstrating that there is reason to doubt that the hidden dataset is exhaustively labeled. It is unknown just how noisy labeling is. My concern is that it is noisy enough to discourage models that generalize.
", "rawMarkdown": "CPMP, I cannot see the hidden dataset. The hidden dataset is not definitively missing labels. My observations are demonstrating that there is reason to doubt that the hidden dataset is exhaustively labeled. It is unknown just how noisy labeling is. My concern is that it is noisy enough to discourage models that generalize.", "votes": null }, { "id": "1276645", "postDate": "04/17/2021 19:26:18", "content": "@philculliton test data do not seem to be fully labeled. Would you mind having a look here? Thanks
", "rawMarkdown": "philculliton test data do not seem to be fully labeled. Would you mind having a look [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/233170)? Thanks", "votes": null } ], "comments": [ { "id": 1250704, "author_name": "cpmpml", "author_url": "", "post_date": "03/24/2021 08:08:14", "content": "2nd is not a problem, it is a feature. A paper can refer to multiple datasets.
", "votes": null, "replies": [] }, { "id": 1250718, "author_name": "artgor", "author_url": "", "post_date": "03/24/2021 08:22:51", "content": "Also, there is an additional difficulty (it could be considered to be a labeling error, I suppose):
\nThere are overlapping entities. For example in text f8b03c87-9d1a-4f20-b76b-cb6c69d447b2 There is an entity national education longitudinal study and inside it is an entity education longitudinal study. Thus, common approaches to NER can't work because they assume that one token can belong only to one entity.
Exactly, was gonna open an issue regarding this
", "votes": null, "replies": [] } ] }, { "id": 1251136, "author_name": "sirishks", "author_url": "", "post_date": "03/24/2021 14:10:31", "content": "Good questions! There are many potential labels for a given dataset, as each label is literally the context of the dataset in a document. So a given dataset may be referred to in many different ways by many documents. A given dataset may also be referred to multiple times within a document.
\nThe test data contains a fairly huge number of unseen datasets and dataset labels. The idea is to generalize to them. Train deliberately is a) not exhaustively labeled (i.e. there are some labels that have not been identified) and b) contains enough to work on the generalization problem but not just string-match against test.
\nRe: your 4th problem, the idea of using Jaccard with a threshold of 0.5 to determine true positives is that sometimes it's ambiguous as to which context words define a dataset. Leaving out or including \"data\" in any of those examples would still match against the context.
\nRe: the 5th problem, we're hoping to find uniquely identifiable mentions within each publication. Your five top examples are extremely ambiguous as to which dataset they might refer to.
\nThanks for your questions! Hopefully that helps a bit!
", "votes": null, "replies": [ { "id": 1251693, "author_name": "sirishks", "author_url": "", "post_date": "03/25/2021 03:15:41", "content": "@philculliton @julialane could you please clarify problems 3 and 5 further:
\nProblem 3: In some cases shorter names are used, whereas in other cases longer names are used (by prepending with the source institution name) - so even if we extract the dataset name correctly, we will get a \"0\" score for that item.
\nProblem 5: I have added another example - could you please reply why \"DYFI data\", which appears to be more important in that paper compared to \"comprehensive catalog\" is not included in the label? Here also, even if we correctly identify \"DYFI data\", we will still get a \"0\" score for this file.
\nHi @sirishks - thanks for the follow-up!
\nRe: problem 3 - the training labels are noisy and, as mentioned below, not exhaustive. They're meant as a guide for modeling semantic context. Discovering the unlabeled or noisily-labeled datasets in train will help you in this task. You are not intended to find these same datasets in test using these precise labels. Test, on the other hand, is exhaustively labeled and the labels have been cleaned up.
\nRe: problem 5 - the training data is not exhaustively labeled. There are potentially many dataset mentions (and datasets) in train that are not labeled at all.
", "votes": null, "replies": [] }, { "id": 1253341, "author_name": "sirishks", "author_url": "", "post_date": "03/26/2021 16:21:04", "content": "@philculliton @julialane but this will result in a very unreliable public leaderboard (because good models can have low public lb scores but high private lb scores), and our scores and ranks will be useless in estimating the final private lb rankings, and will lead to a large shake up in Kaggle terminology, which is not a desired result ..
", "votes": null, "replies": [] }, { "id": 1253539, "author_name": "philculliton", "author_url": "", "post_date": "03/26/2021 20:59:45", "content": "Hi @sirishks -
\nThanks for the feedback! Given the structure of the public leaderboard I expect good scores to indicate better models. I suspect a good model will be a) solving the private-test-like cases in public (and doing better than the baseline because of it) and therefore b) solving the private test well.
\nThis is an uncommon approach to competitions, but for the problem - generalizing information extraction for NLP - I think it's a reasonable way to do it. It's not enough to simply model train and do well on test, but if you can sort out the problems in train and apply that to test it should help you.
", "votes": null, "replies": [] }, { "id": 1268513, "author_name": "psilogram", "author_url": "", "post_date": "04/09/2021 13:28:31", "content": "@philculliton Could you elaborate a little more on what you mean by:
\n \"Test, on the other hand, is exhaustively labeled and the labels have been cleaned up.\"\nDoes 'exhaustively labeled' mean:
\nIf #1, how do you choose between different references for the same dataset (first, longest, arbitrary)? If #2, why does the LB score get worse when we include alternative forms of the same dataset?
\nAnd what does 'cleaned up' mean?
", "votes": null, "replies": [] }, { "id": 1268831, "author_name": "trentb", "author_url": "", "post_date": "04/09/2021 20:06:18", "content": "In addition, I am curious if only datasets that were used by the authors to produce their publication should be labeled. For example, if the authors reference a dataset that they did not use in the present publication (maybe when discussing previous work) should this dataset be labeled?
", "votes": null, "replies": [] }, { "id": 1276645, "author_name": "eguidotti", "author_url": "", "post_date": "04/17/2021 19:26:18", "content": "@philculliton test data do not seem to be fully labeled. Would you mind having a look here? Thanks
", "votes": null, "replies": [] } ] }, { "id": 1254122, "author_name": "suicaokhoailang", "author_url": "", "post_date": "03/27/2021 11:13:10", "content": "Crazy thing is, by matching that 45 datasets and their variances one could get to ~0.7. I sincerely hope they're messing with the public test set on purpose, if that distribution holds with the private set it will be ridiculous. In fact, the score returned for that method should be 0.0.
\nThat was 45 samples vs the whole world, Jesus.
\nDefinitely not having a good feeling about this.
I sincerely hope they're messing with the public test set on purpose
\nThey are also messing with the private Test set / LB results - see findings of my experiments below.
@sirishks @suicaokhoailang - just want to note that the private set is essentially disjoint and there's no label-matching crossover between it at train. That 0.7 score is literally getting the training labels right in public test. No such effect exists in private test.
\nTrain (and to a degree the public test) is intended as a guide / workset for the difficult task of labeling the private set. String matching isn't part of that solution.
", "votes": null, "replies": [] }, { "id": 1256084, "author_name": "sirishks", "author_url": "", "post_date": "03/29/2021 14:30:42", "content": "@philculliton It is great that private test set is disjoint.
\nHowever, my question is why we are only getting 0.7 score and not a perfect 1.0 score on public LB?
\nI believe that it is because you are (i) allowing multiple labels for same dataset for the same file and (ii) allowing substrings.
There is definitely a bug in your LB score (jaccard) calculation because I expected a higher score when submitting the \"longest label\" correctly, but instead noticed that \"multiple substring labels\" gave a higher score.
\nPlease see my experiments below and my recommendations for making this competition better.
\nPerhaps you can reply using an actual example, say public test file 2100032a-7c33-4bff-97ef-690822c43466, and show (i) what is the correct expected PredictionString ('alzheimer s disease neuroimaging initiative adni') for this file, and (ii) how you are incorporating Julia's answer (accepting different strings as correct) while computing jaccard score?, along with (iii) example combinations of substrings such as 'alzheimer s disease neuroimaging initiative' or 'adni' or 'alzheimer s disease neuroimaging initiative|adni' and show how much score will be given to each of these partially correct answers (some with incorrect numerator and others with incorrect denominator in the score formula).
", "votes": null, "replies": [] }, { "id": 1256119, "author_name": "philculliton", "author_url": "", "post_date": "03/29/2021 15:00:53", "content": "@sirishks - the primary reason you're getting 0.7 and not 1.0 on the public leaderboard is because there are a large number of labels in public test that are not included in train. Every public test sample has at least one label that has not been identified.
\nI am looking into your recommendations and experiments, and discussing with the host, but the expectation that you'll get 1.0 simply by getting all training labels correct is invalid. That's not the task.
", "votes": null, "replies": [] }, { "id": 1268155, "author_name": "uahmedkhan", "author_url": "", "post_date": "04/09/2021 07:08:00", "content": "I think this answer of your's should be made a seperate thread , because it clears the confusion all are having about datasets labels etc , our task is not string matching , but some understanding of semantic context for dataset labels usage , that's why straightforward matching gets 0.7 but not 1.0.
", "votes": null, "replies": [] } ] }, { "id": 1254773, "author_name": "sirishks", "author_url": "", "post_date": "03/28/2021 05:45:20", "content": "@philculliton @julialane you said
\n> Test, on the other hand, is exhaustively labeled and the labels have been cleaned up.
\nI can confirm that even this is not correct - (i) labels are still ambiguous (random substrings are being used), (ii) also a single Test Id (public LB) still uses multiple labels for the SAME dataset for the same file.
I ran an experiment where I got a higher score when using multiple labels for the same dataset within the same PredictionString:
\n(A similar example is also present in train/6d2ff020-c87b-4abc-a757-0b7ba25a87fa.json where even though the file actually contains the full label \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\", but train.csv contains multiple labels i.e 2 rows for the SAME dataset for the same Id/file. The substring \"adni\" is provided as a valid training label, which is ambiguous.)
\nThe above example is similar to the training label mentioned in CPMP's topic, where he was referring to presence of both 'Aging Integrated Database (AGID)' as well as 'Aging Integrated Database' in the same file.
\nMy findings:
\nI accept your comments about handling noisy real world data, but there should not be ambiguity in expected output. So, I strongly disagree with your previous replies, as I have found that test labels in Leaderboard evaluation are still ambiguous (not cleaned up).
My recommendation:
\nFor each dataset,
\n(i) use the \"full dataset name\" i.e. \"longest possible label\" which is actually present in the corresponding file, and remove all substrings from expected PredictionString and
\n(ii) ensure that we have a single longest possible label per dataset per file (no duplicates / multiple labels for same dataset in same file).
\nThis will help remove ambiguity in LB predictions, and we will be able to build models which are actually useful!
@philculliton @julialane Please feel free to repeat my experiment and confirm my findings yourselves.
", "votes": null, "replies": [] }, { "id": 1255035, "author_name": "julialane", "author_url": "", "post_date": "03/28/2021 11:08:11", "content": "There are indeed 45 dataset titles in the training datasets, but 130 dataset labels because authors in publications refer to the datasets in different ways. We accept multiple different possibilities. For example, we accept both ADNI and Alzheimer's Disease Neuroimaging Initiative (ADNI) – we are not penalizing for predicting only ADNI.
\nAs @philculliton has already pointed out, the private leaderboard does not contain any labels from the training set. So constructing a lookup table of the training datasets will not be useful. If you look at the intent of the challenge, it is to find dataset references from the semantic context. I expect @philculliton will respond in more depth on Monday.
\nI hope this helps, and we are thrilled by the degree of interest and the thoughtfulness of your questions.
Hi Julia, I still feel quite unclear about this:
\n\n\nwe accept both ADNI and Alzheimer's Disease Neuroimaging Initiative (ADNI) – we are not penalizing for predicting only ADNI.
\n
According to the evaluation page, if I predict both \"ADNI\" and \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\" for a given sample, and the label is just \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\", the \"ADNI\" that I predicted will be added to the FP pool, thus lowering my score.
\n\n", "votes": null, "replies": [] }, { "id": 1255150, "author_name": "sirishks", "author_url": "", "post_date": "03/28/2021 13:56:03", "content": "Any matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are counted as true positives (TP), the remainder as false positives (FP).
\n
@julialane - thank you for your reply and we hope that @philculliton will address the actual score discrepancies observed in my experiments soon.
\nI want to emphasize my recommendation again viz that even if labels differ from file to file, within a specific file / Id, there should be only a single (longest possible) label for each dataset (and penalize substring labels or multiple labels for same dataset).
\nAnother thought is about your reply that you
\naccept both ADNI and Alzheimer's Disease Neuroimaging Initiative (ADNI) as correct,
\nwhich still results in some ambiguity because other labels are not so simple.
\nPerhaps you can instead remove the Jaccard threshold of 0.5 so that someone who predicts just ADNI will still get partial credit for the answer.
I had the same concern which is why I am so picky on differing labels for same dataset.
\n\n\nAccording to the evaluation page, if I predict both \"ADNI\" and \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\" for a given sample, and the label is just \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\", the \"ADNI\" that I predicted will be added to the FP pool, thus lowering my score.
\n
Any matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are counted as true positives (TP), the remainder as false positives (FP).\n@julialane @philculliton It's been 2-weeks since you said
\nI am looking into your recommendations and experiments, and discussing with the host
Did you make a decision regarding my recommendations viz,
\nWhether you adopt my recommendations or not, below question still requires to be clarified:
\nExample scenario: Let's use the longest single label as ground truth = 'alzheimer s disease neuroimaging initiative adni'
Aiming for semantic understanding of dataset mentions is pretty ambitious. Is the competition (public LB etc) currently set up to encourage that?
\nI’m currently ranked pretty high on the public LB and I’m skeptical that a top public LB submission translates (generalizes) to the private LB setting i.e., unseen publications and labels.
\nI’d suggest reconfiguring the LB so both public and private LB are based on the disjoint publications.
\nThe last thing both the organizers and competitors want is a bunch of solutions that are “gamed” for the public LB and don’t generalize well leading to a massive shakeup.
", "votes": null, "replies": [ { "id": 1273400, "author_name": "suicaokhoailang", "author_url": "", "post_date": "04/14/2021 10:07:26", "content": "I second this. I still can't understand the decision to use a part of the training data as public test, LB score is practically useless atm.
", "votes": null, "replies": [] }, { "id": 1273863, "author_name": "philculliton", "author_url": "", "post_date": "04/14/2021 17:51:54", "content": "Hi @leecming - great question, thanks! The public test set is currently set up to try and reward generalization. A significant percentage of the public test set is not crossover with the training set (and involves a large number of datasets / labels not seen in train). Better scores on the LB will be solving for those examples.
", "votes": null, "replies": [] }, { "id": 1274052, "author_name": "leecming", "author_url": "", "post_date": "04/15/2021 00:03:50", "content": "\nThanks for the explanation (and I don't think downvoting a constructive response is helpful).
\nUltimately, it boils down to - does the public LB correlate well to private LB?
\nAll I'd ask is for the organizers to keep an eye on this and consider updating the split if necessary (and earlier rather than later in the competition).
\nThe task is difficult enough that we don't need an additional layer of obfuscation/complexity in trying to translate public LB to private LB performance from our end.
", "votes": null, "replies": [] }, { "id": 1274142, "author_name": "cpmpml", "author_url": "", "post_date": "04/15/2021 03:21:41", "content": "\n\n\nUltimately, it boils down to - does the public LB correlate well to private LB?
\n
Is this desirable? LB probing is a plague and maybe the only issue with Kaggle competition.
\nWhat is desirable is a private LB that measures generalization. For instance, in time series competitions, having public LB the next period, say next week, and private LB the following period, i.e. second week, is a much better way to assess model performance than a public LB based on a random sampling of next two periods. Then you need to set a local validation that evaluates the model ability to predict well 2 weeks ahead.
\nHere it means that we need to set a local validation and rely on it rather than LB probing. To your point this may be a little too ambitious for a Kaggle competition as we have only very weak labels.
\nI have been thinking about how to do this and will re enter only if I find a way (and if the ground truth issues that keep popping up in the forum are clarified via some examples).
", "votes": null, "replies": [] }, { "id": 1274146, "author_name": "leecming", "author_url": "", "post_date": "04/15/2021 03:39:53", "content": "Allowing for an open-ended number of text predictions per document and setting up a disjointed hidden test set means that it's much harder to probe (versus a vanilla classification/regression task).
\nI get the tension between having a meaningful public LB and preventing probing but imo, this competition is on the wrong end of the continuum.
", "votes": null, "replies": [] }, { "id": 1274147, "author_name": "suicaokhoailang", "author_url": "", "post_date": "04/15/2021 03:39:57", "content": "@philculliton
\nHere's my two cents. It's a terrible idea to blindly trust the LB, however the LB can still show you a rough idea of where you're standing compared to other competitors. As I said in my thread, the LB here is dead due to how the test data is set up, it's impossible to know how others are doing, what the real baseline may look like or whether our current model can be even considered working. In the last couple of weeks we tried really hard to make a model that we * think* may generalize quite well to unseen data, and yet it hasn't even beaten the string matching method.
\nI agree with @leecming that validation in this competition is hard enough, we don't need this.
\nMy suggestion is to make the public LB totally disjoint from train and reduce the submission limit to 2 per day.
Another way of thinking about it is this -
\nfor it to be possible to conduct meaningful cross validation locally, we should have gotten the full label set for train (and presumably a different public test LB). How do you do proper CV if your hold-out set isn't complete.
\n\nI’d suggest reconfiguring the LB so both public and private LB are based on the disjoint publications.
\n
This is a bad idea in general since the best public LB probers will win on private LB.
\nBut I agree on all the other comments mentioned here.
There may not be an easy solution for this competition setup but I just don't get the logic of how the current setup is useful. It breaks local validation and it breaks public LB usefulness.
\nThe bare minimum to ensure model generalization I would think would be to have train, public LB and private LB, each independently complete but pairwise disjoint (on publications). Having labels also disjoint might be infeasible from host's perspective but if that is also possible, nothing like it.
\nOr maybe we need a detailed explanation from hosts about why the current setup is ideal to build a solution that makes business sense. Can't seem to find a way to convince myself.
", "votes": null, "replies": [] }, { "id": 1274172, "author_name": "trentb", "author_url": "", "post_date": "04/15/2021 04:21:53", "content": "My biggest concern is that I doubt the hidden test data is exhaustively labeled. This is why non-string matching models, which generalize, are doing so poorly. They are capturing legitimate dataset mentions that are not labeled. Good generalization, what the host obviously wants, is actually being discouraged.
\nWould it be possible for Kaggle staff to take a close look at the hidden test data to see if my suspicion about unlabeled dataset mentions is valid? If it's valid, then I don't see how this is anything other than a string matching competition.
", "votes": null, "replies": [] }, { "id": 1274407, "author_name": "cpmpml", "author_url": "", "post_date": "04/15/2021 09:25:40", "content": "\n\nHow do you do proper CV if your hold-out set isn't complete.
\n
More and more Kaggle competitions are like this, BirdCLEF and Human Protein Atlas are weakly labelled too for instance.
\nThe days of supervised learning are over, state of the art is moving to semi supervised learning and self supervised learning. So is Kaggle.
", "votes": null, "replies": [] }, { "id": 1274409, "author_name": "cpmpml", "author_url": "", "post_date": "04/15/2021 09:26:07", "content": "\n\nI doubt the hidden test data is exhaustively labeled
\n
On what ground?
", "votes": null, "replies": [] }, { "id": 1274561, "author_name": "trentb", "author_url": "", "post_date": "04/15/2021 12:12:03", "content": "I am doubting the hidden test data is exhaustively labeled based on the discussion topic here. Since I created that topic, I've discovered other datasets that do not appear to be labeled in the hidden test data.
", "votes": null, "replies": [] }, { "id": 1274572, "author_name": "leecming", "author_url": "", "post_date": "04/15/2021 12:28:44", "content": "Human annotations for NLP tasks are inherently noisy and I expect there to be a fair number of labelling errors/misses even with a rigorous multi-annotator methodology. Just eyeball some of the longer training docs with a lot of formatting errors and you'd be hard-pressed to pick out all the labels.
\nThe original metric of using an f-measure with 0 beta would have mitigated that by allowing us to pick a single, high confidence prediction for each ID. Switching to 0.5 beta was a kludge so that the LB wouldn't be filled with perfect scores but the unintended consequence was to make the actual hidden test set much harder to solve.
\nI guess this is a long-winded way to say \"good luck to everyone, we'll need it\"
", "votes": null, "replies": [] }, { "id": 1274596, "author_name": "cpmpml", "author_url": "", "post_date": "04/15/2021 12:56:44", "content": "\n\nI've discovered other datasets that do not appear to be labeled in the hidden test data.
\n
How can you see the hidden dataset?
", "votes": null, "replies": [] }, { "id": 1274600, "author_name": "cpmpml", "author_url": "", "post_date": "04/15/2021 12:58:10", "content": "\n\nSwitching to 0.5 beta was a kludge so that the LB wouldn't be filled with perfect scores but the unintended consequence was to make the actual hidden test set much harder to solve.
\n
You nailed it. All questions asked about ground truth are indeed due to this change. Because it is unclear how redundant predictions are handled now.
", "votes": null, "replies": [] }, { "id": 1274639, "author_name": "trentb", "author_url": "", "post_date": "04/15/2021 13:41:15", "content": "CPMP, I cannot see the hidden dataset. The hidden dataset is not definitively missing labels. My observations are demonstrating that there is reason to doubt that the hidden dataset is exhaustively labeled. It is unknown just how noisy labeling is. My concern is that it is noisy enough to discourage models that generalize.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1250607": "There are only 130 'cleaned_label's / only 45 unique 'dataset_title's for the 14,316 training data files.\n\n**1st Problem**: Even these 130 are not really unique, and contain many duplicates which are spelled differently.\nFor example:\n- 3 different labels - '2019 ncov complete genome sequences', '2019 ncov genome sequence', '2019 ncov genome sequences'\n- 2 labels - 'advanced national seismic system anss comprehensive catalog comcat ', 'advanced national seismic system comprehensive catalog'\n- 2 labels - 'agricultural resource management survey', 'agricultural resources management survey'\n- 2 lables - 'anss comprehensive catalog', 'anss comprehensive earthquake catalog'\n- ...\n\n**2nd Problem**: Same file has multiple labels (multiple rows).\n[Edit - not a problem - see CPMP comment and my update below] \nFor example:\n- 170113f9-399c-489e-ab53-2faf5c64c5bc file has 22 rows in train.csv with many different labels - 'beginning postsecondary students', 'beginning postsecondary students longitudinal study', 'early childhood longitudinal study', 'education longitudinal study', 'high school longitudinal study', 'higher education research and development survey', 'national center for science and engineering statistics higher education research and development survey', 'national center for science and engineering statistics survey of doctorate recipients', 'national center for science and engineering statistics survey of earned doctorates', 'national center for science and engineering statistics survey of graduate students and postdoctorates in science and engineering', 'national center for science and engineering statistics survey of industrial research and development', 'national center for science and engineering statistics survey of science and engineering research facilities', 'national education longitudinal study', 'national science foundation higher education research and development survey', 'national science foundation survey of earned doctorates', 'national science foundation survey of science and engineering research facilities', 'survey of doctorate recipients', 'survey of earned doctorates', 'survey of graduate students and postdoctorates in science and engineering', 'survey of industrial research and development', 'survey of science and engineering research facilities', 'trends in international mathematics and science study'\n- fd23e7e0-a5d2-4f98-992d-9209c85153bb has 2 rows each with a different label - 'cas covid 19 antiviral candidate compounds data set', 'cas covid 19 antiviral candidate compounds data'\n- ...\n\n\n[Update] I realized that one file can have multiple labels as per [evaluation page](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation) which mentioned that \"Multiple predictions are delineated with a pipe (|) character in the submission file\" and gives below example:\n- `01e4e08c-ffea-45a7-adde-6a0c0ad755fc,space location data|national space objects|national space dataset`\nHowever, this example incorrectly looks like all the (multiple) cleaned labels are referring to the same dataset viz \"national space objects location dataset\".\n\n**3rd Problem**: Inconsistent naming where some labels contain shortnames instead of full name such as \"xx yy zz dataset\".\n- For example, 024db495-59a3-44d0-9656-e65671b0824d has short label 'adni' instead of expected value \"adni 2 longitudinal mri dataset\". A correct answer (\"adni 2 longitudinal mri dataset\") will get a \"0\" score for this because 1/(1+5-1) <0.5 threshold.\n- For example, long label 'national center for science and engineering statistics survey of industrial research and development', where the parent organization 'national center for science and engineering statistics' is added in front of the dataset 'survey of industrial research and development'. Again a correct answer (\"survey of industrial research and development\") will get a \"0\" score for this because 6/(13+6-6) <0.5 threshold.\n\n**4th Problem**: Whether to include word 'data' / 'dataset' in the label or not? (in cases where those words are present in the corresponding text)\nFor example:\n- labels using word 'dataset': covid 19 open research dataset\n- labels using word 'data': covid 19 open research data\n- labels using word 'database': rsna international covid 19 open radiology database\n- labels not using word 'data' at all: covid 19 genome sequences\n\n**5th Problem**: Ambiguous and unexpected labels are used.\nFor example:\nFile d4b155e4-e351-4bea-a81e-f558d1f2882a contains several phrases such as:\n- clinical and autopsy data\n- Multicenter prospective cohort study data\n- data from prospective studies\n- clinical data from Wuhan\n- Pathology data from our autopsy series\n- COVID-19 genome sequences\nHowever, only 1 single label is used \"covid 19 genome sequences\" for this file.\nWhy are references to other data such as \"clinical and autopsy data\", \"prospective cohort study data\" not used as labels?\n\nAnother example file 3bd0f678-7a16-4c6b-8692-3a57f2d7d0d1 specifies the label as \"advanced national seismic system comprehensive catalog\", whereas the paper actually has many more references to a different dataset as the primary source of data \"U.S. Geological Survey (USGS) \"Did You Feel It?\" (DYFI) data\". Why was DYFI data not included in the label? Here also, even if we correctly identify \"DYFI data\", we will still get a \"0\" score for this file.\n\n\n@julialane @maggiemd Could you please let us know about the steps used for generating train data 'cleaned_label's? I was really expecting a \"single\" correct label per dataset (and a single prediction per file when only one dataset is present).\nPS: In cases where the same file/paper references multiple datasets, then the \"pipe dilineation\" is appropriate (still using only one label for each single dataset).", "1250704": "2nd is not a problem, it is a feature. A paper can refer to multiple datasets.", "1250718": "Also, there is an additional difficulty (it could be considered to be a labeling error, I suppose):\n\nThere are overlapping entities. For example in text `f8b03c87-9d1a-4f20-b76b-cb6c69d447b2` There is an entity `national education longitudinal study` and inside it is an entity `education longitudinal study`. Thus, common approaches to NER can't work because they assume that one token can belong only to one entity.", "1250777": "Exactly, was gonna open an issue regarding this", "1251136": "Submissions are disabled @ 2pm GMT 24-March, and re-enabled after 5.5 hrs\n- [Metric issue](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228393) has been fixed!", "1251493": "Good questions! There are many potential labels for a given dataset, as each label is literally the context of the dataset in a document. So a given dataset may be referred to in many different ways by many documents. A given dataset may also be referred to multiple times within a document.\n\nThe test data contains a fairly huge number of unseen datasets and dataset labels. The idea is to generalize to them. Train deliberately is a) not exhaustively labeled (i.e. there are some labels that have not been identified) and b) contains enough to work on the generalization problem but not just string-match against test.\n\nRe: your 4th problem, the idea of using Jaccard with a threshold of 0.5 to determine true positives is that sometimes it's ambiguous as to which context words define a dataset. Leaving out or including \"data\" in any of those examples would still match against the context.\n\nRe: the 5th problem, we're hoping to find uniquely identifiable mentions within each publication. Your five top examples are extremely ambiguous as to which dataset they might refer to.\n\nThanks for your questions! Hopefully that helps a bit!", "1251693": "philculliton @julialane could you please clarify problems 3 and 5 further:\n\n**Problem 3**: In some cases shorter names are used, whereas in other cases longer names are used (by prepending with the source institution name) - so even if we extract the dataset name correctly, we will get a \"0\" score for that item.\n- For example, short label 'adni' instead of expected value \"adni 2 longitudinal mri dataset\". A correct answer (\"adni 2 longitudinal mri dataset\") will get a \"0\" score for this because 1/(1+5-1) <0.5 threshold.\n- For example, long label 'national center for science and engineering statistics survey of industrial research and development', where the parent organization 'national center for science and engineering statistics' is added in front of the dataset 'survey of industrial research and development'. Again a correct answer (\"survey of industrial research and development\") will get a \"0\" score for this because 6/(13+6-6) <0.5 threshold.\n\n**Problem 5**: I have added another example - could you please reply why \"DYFI data\", which appears to be more important in that paper compared to \"comprehensive catalog\" is not included in the label? Here also, even if we correctly identify \"DYFI data\", we will still get a \"0\" score for this file.\n- [Another example file 3bd0f678-7a16-4c6b-8692-3a57f2d7d0d1 specifies the label as \"advanced national seismic system comprehensive catalog\", whereas the paper actually has many more references to a different dataset as the primary source of data \"U.S. Geological Survey (USGS) \"Did You Feel It?\" (DYFI) data\".]", "1253208": "Hi @sirishks - thanks for the follow-up!\n\nRe: problem 3 - the training labels are noisy and, as mentioned below, not exhaustive. They're meant as a guide for modeling semantic context. Discovering the unlabeled or noisily-labeled datasets in train will help you in this task. You are not intended to find these same datasets in test using these precise labels. Test, on the other hand, is exhaustively labeled and the labels have been cleaned up.\n\nRe: problem 5 - the training data is not exhaustively labeled. There are potentially many dataset mentions (and datasets) in train that are not labeled at all.", "1253341": "philculliton @julialane but this will result in a **very unreliable** public leaderboard (because good models can have low public lb scores but high private lb scores), and our scores and ranks will be useless in estimating the final private lb rankings, and will lead to a **large shake up** in Kaggle terminology, which is **not a desired** result ..", "1253539": "Hi @sirishks -\n\nThanks for the feedback! Given the structure of the public leaderboard I expect good scores to indicate better models. I suspect a good model will be a) solving the private-test-like cases in public (and doing better than the baseline because of it) and therefore b) solving the private test well.\n\nThis is an uncommon approach to competitions, but for the problem - generalizing information extraction for NLP - I think it's a reasonable way to do it. It's not enough to simply model train and do well on test, but if you can sort out the problems in train and apply that to test it should help you.", "1254122": "Crazy thing is, by matching that 45 datasets and their variances one could get to ~0.7. I sincerely hope they're messing with the public test set on purpose, if that distribution holds with the private set it will be ridiculous. In fact, the score returned for that method should be 0.0.\nThat was 45 samples vs the whole world, Jesus.\nDefinitely not having a good feeling about this.", "1254773": "philculliton @julialane you said\n`> Test, on the other hand, is exhaustively labeled and the labels have been cleaned up.`\nI can confirm that even **this is not correct** - (i) labels are still ambiguous (random substrings are being used), (ii) also a single Test Id (public LB) still uses **multiple labels for the SAME dataset** for the same file.\n\nI ran an experiment where I got a *higher score* when using **multiple labels for the same dataset within the same PredictionString**:\n- **Experiment 1**: only submitted the longest label (ignoring substrings), present in the file, i.e. single label for a single dataset\n PredictionString=*'alzheimer s disease neuroimaging initiative adni'*\n- **Experiment 2**: submitted multiple/all possible labels for each dataset (after ensuring that these multiple labels/substrings are actually present in the file)\n PredictionString=*'alzheimer s disease neuroimaging initiative|adni|alzheimer s disease neuroimaging initiative adni'*\n(public test file 2100032a-7c33-4bff-97ef-690822c43466)\n\n(A similar example is also present in train/6d2ff020-c87b-4abc-a757-0b7ba25a87fa.json where even though the file actually contains the full label \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\", but **train.csv contains multiple labels i.e 2 rows for the SAME dataset for the same Id/file**. The substring \"adni\" is provided as a valid training label, which **is ambiguous**.) \n\nThe above example is similar to the training label mentioned in [CPMP's topic](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228337), where he was referring to presence of both 'Aging Integrated Database (AGID)' as well as 'Aging Integrated Database' in the same file.\n\n**My findings:**\nI accept your comments about handling noisy real world data, but there should not be ambiguity in expected output. So, I strongly disagree with your previous replies, as I have found that **test labels in Leaderboard evaluation are still ambiguous (not cleaned up)**.\n\n**My recommendation:**\nFor each dataset,\n(i) use the \"full dataset name\" i.e. \"longest possible label\" which is actually present in the corresponding file, and **remove all substrings** from expected PredictionString and \n(ii) ensure that we have a **single** longest possible label per dataset per file (no duplicates / multiple labels for same dataset in same file).\nThis will help remove ambiguity in LB predictions, and we will be able to build models which are actually useful!\n\n@philculliton @julialane Please feel free to repeat my experiment and confirm my findings yourselves.", "1254870": "`I sincerely hope they're messing with the public test set on purpose`\nThey are also messing with the private Test set / LB results - see findings of my experiments below.", "1255035": "There are indeed 45 dataset titles in the training datasets, but 130 dataset labels because authors in publications refer to the datasets in different ways. We accept multiple different possibilities. For example, we accept both ADNI and Alzheimer's Disease Neuroimaging Initiative (ADNI) – we are not penalizing for predicting only ADNI.\n\nAs @philculliton has already pointed out, the private leaderboard does not contain any labels from the training set. So constructing a lookup table of the training datasets will not be useful. If you look at the intent of the challenge, it is to find dataset references from the semantic context. I expect @philculliton will respond in more depth on Monday. \nI hope this helps, and we are thrilled by the degree of interest and the thoughtfulness of your questions.", "1255055": "Hi Julia, I still feel quite unclear about this:\n> we accept both ADNI and Alzheimer's Disease Neuroimaging Initiative (ADNI) – we are not penalizing for predicting only ADNI.\n\nAccording to the evaluation page, if I predict both \"ADNI\" and \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\" for a given sample, and the label is just \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\", the \"ADNI\" that I predicted will be added to the FP pool, thus lowering my score.\n\n> Any matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are counted as true positives (TP), the remainder as false positives (FP).", "1255150": "julialane - thank you for your reply and we hope that @philculliton will address the actual score discrepancies observed in **my experiments** soon.\n\nI want to emphasize my recommendation again viz that even if labels differ from file to file, within a specific file / Id, there should be only a single (longest possible) label for each dataset (and penalize substring labels or multiple labels for same dataset).\n\nAnother thought is about your reply that you\n`accept both ADNI and Alzheimer's Disease Neuroimaging Initiative (ADNI)` as correct,\nwhich still results in some ambiguity because other labels are not so simple.\nPerhaps you can instead remove the Jaccard threshold of 0.5 so that someone who predicts just ADNI will still get partial credit for the answer.", "1256026": "sirishks @suicaokhoailang - just want to note that the private set is essentially disjoint and there's no label-matching crossover between it at train. That 0.7 score is literally getting the training labels right in public test. No such effect exists in private test.\n\nTrain (and to a degree the public test) is intended as a guide / workset for the difficult task of labeling the private set. String matching isn't part of that solution.", "1256084": "philculliton It is great that private test set is disjoint.\n\nHowever, my question is why we are only getting 0.7 score and not a perfect 1.0 score on public LB?\nI believe that it is because you are (i) allowing multiple labels for same dataset **for the same file** and (ii) allowing substrings.\n\nThere is definitely a **bug in your LB score (jaccard) calculation** because I expected a higher score when submitting the \"longest label\" correctly, but instead noticed that \"multiple substring labels\" gave a higher score. \n\nPlease see my experiments below and my recommendations for making this competition better.\n\nPerhaps you can reply using an actual example, say public test file 2100032a-7c33-4bff-97ef-690822c43466, and show (i) what is the correct expected PredictionString ('alzheimer s disease neuroimaging initiative adni') for this file, and (ii) how you are incorporating Julia's answer (accepting different strings as correct) while computing jaccard score?, along with (iii) example combinations of substrings such as 'alzheimer s disease neuroimaging initiative' or 'adni' or 'alzheimer s disease neuroimaging initiative|adni' and show how much score will be given to each of these partially correct answers (some with incorrect numerator and others with incorrect denominator in the score formula).", "1256089": "I had the same concern which is why I am so picky on differing labels for same dataset.\n\n> According to the evaluation page, if I predict both \"ADNI\" and \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\" for a given sample, and the label is just \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\", the \"ADNI\" that I predicted will be added to the FP pool, thus lowering my score.\n\n Any matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are counted as true positives (TP), the remainder as false positives (FP).", "1256119": "sirishks - the primary reason you're getting 0.7 and not 1.0 on the public leaderboard is because there are a large number of labels in public test that are not included in train. Every public test sample has at least one label that has not been identified.\n\nI am looking into your recommendations and experiments, and discussing with the host, but the expectation that you'll get 1.0 simply by getting all training labels correct is invalid. That's not the task.", "1268155": "I think this answer of your's should be made a seperate thread , because it clears the confusion all are having about datasets labels etc , our task is not string matching , but some understanding of semantic context for dataset labels usage , that's why straightforward matching gets 0.7 but not 1.0.", "1268513": "philculliton Could you elaborate a little more on what you mean by:\n\n \"Test, on the other hand, is exhaustively labeled and the labels have been cleaned up.\"\n\nDoes 'exhaustively labeled' mean:\n\n1. Each unique dataset that is mentioned in a record is labeled.\n2. All unique references of each dataset (including different references for the same dataset) are labeled.\n\nIf #1, how do you choose between different references for the same dataset (first, longest, arbitrary)? If #2, why does the LB score get worse when we include alternative forms of the same dataset?\n\nAnd what does 'cleaned up' mean?", "1268831": "In addition, I am curious if only datasets that were used by the authors to produce their publication should be labeled. For example, if the authors reference a dataset that they did not use in the present publication (maybe when discussing previous work) should this dataset be labeled?", "1270869": "julialane @philculliton It's been 2-weeks since you said\n`I am looking into your recommendations and experiments, and discussing with the host`\n\nDid you make a decision regarding my recommendations viz,\n- Within a file, each dataset should have only a **single** correct label\n- Within a file, the correct label should be the **longest** label (and not abbreviations such as ADNI)\n\nWhether you adopt my recommendations or not, below question still requires to be clarified:\nExample scenario: Let's use the longest single label as ground truth = 'alzheimer s disease neuroimaging initiative adni'\n1. Single prediction i.e. PredictionString='adni' (1 word instead of 6 words)\n Here 1 / [(6+1)-1] = 1/6 < 0.5, so score=0 (single **FP**)\n2. Single prediction i.e. PredictionString='alzheimer s disease neuroimaging initiative' (5 words instead of 6 words)\n Here 5 / [(6+5)-5] = 5/6 > 0.5, so score=1 (single **TP**)\n3. Multiple prediction i.e. PredictionString='alzheimer s disease neuroimaging initiative|adni'\n Here we have 1 TP and 1 FP, so score = 1.25*TP/(1.25*TP + 0.25*FN + FP) = 1.25/(1.25+1) = 0.55\nPlease **confirm above calculations** for this example scenario.", "1273387": "julialane @philculliton \n\nAiming for semantic understanding of dataset mentions is pretty ambitious. Is the competition (public LB etc) currently set up to encourage that?\n\nI’m currently ranked pretty high on the public LB and I’m skeptical that a top public LB submission translates (generalizes) to the private LB setting i.e., unseen publications and labels.\n\nI’d suggest reconfiguring the LB so both public and private LB are based on the disjoint publications.\n\nThe last thing both the organizers and competitors want is a bunch of solutions that are “gamed” for the public LB and don’t generalize well leading to a massive shakeup.", "1273400": "I second this. I still can't understand the decision to use a part of the training data as public test, LB score is practically useless atm.", "1273863": "Hi @leecming - great question, thanks! The public test set is currently set up to try and reward generalization. A significant percentage of the public test set is not crossover with the training set (and involves a large number of datasets / labels not seen in train). Better scores on the LB will be solving for those examples.", "1274052": "philculliton \n\nThanks for the explanation (and I don't think downvoting a constructive response is helpful).\n\nUltimately, it boils down to - does the public LB correlate well to private LB?\n\nAll I'd ask is for the organizers to keep an eye on this and consider updating the split if necessary (and earlier rather than later in the competition). \n\nThe task is difficult enough that we don't need an additional layer of obfuscation/complexity in trying to translate public LB to private LB performance from our end.", "1274142": "leecming\n\n> Ultimately, it boils down to - does the public LB correlate well to private LB?\n\nIs this desirable? LB probing is a plague and maybe the only issue with Kaggle competition.\n\nWhat is desirable is a private LB that measures generalization. For instance, in time series competitions, having public LB the next period, say next week, and private LB the following period, i.e. second week, is a much better way to assess model performance than a public LB based on a random sampling of next two periods. Then you need to set a local validation that evaluates the model ability to predict well 2 weeks ahead.\n\nHere it means that we need to set a local validation and rely on it rather than LB probing. To your point this may be a little too ambitious for a Kaggle competition as we have only very weak labels.\n\nI have been thinking about how to do this and will re enter only if I find a way (and if the ground truth issues that keep popping up in the forum are clarified via some examples).", "1274146": "Allowing for an open-ended number of text predictions per document and setting up a disjointed hidden test set means that it's much harder to probe (versus a vanilla classification/regression task). \n\nI get the tension between having a meaningful public LB and preventing probing but imo, this competition is on the wrong end of the continuum.", "1274147": "philculliton \nHere's my two cents. It's a terrible idea to blindly trust the LB, however the LB can still show you a rough idea of where you're standing compared to other competitors. As I said in my thread, the LB here is dead due to how the test data is set up, it's impossible to know how others are doing, what the real baseline may look like or whether our current model can be even considered working. In the last couple of weeks we tried really hard to make a model that we * think* may generalize quite well to unseen data, and yet it hasn't even beaten the string matching method. \nI agree with @leecming that validation in this competition is hard enough, we don't need this. \nMy suggestion is to make the public LB totally disjoint from train and reduce the submission limit to 2 per day.", "1274150": "Another way of thinking about it is this - \nfor it to be possible to conduct meaningful cross validation locally, we should have gotten the full label set for train (and presumably a different public test LB). How do you do proper CV if your hold-out set isn't complete.", "1274161": "> I’d suggest reconfiguring the LB so both public and private LB are based on the disjoint publications.\n\nThis is a bad idea in general since the best public LB probers will win on private LB.\nBut I agree on all the other comments mentioned here.\n\nThere may not be an easy solution for this competition setup but I just don't get the logic of how the current setup is useful. It breaks local validation and it breaks public LB usefulness.\n\nThe bare minimum to ensure model generalization I would think would be to have train, public LB and private LB, each independently complete but pairwise disjoint (on publications). Having labels also disjoint might be infeasible from host's perspective but if that is also possible, nothing like it.\n\nOr maybe we need a detailed explanation from hosts about why the current setup is ideal to build a solution that makes business sense. Can't seem to find a way to convince myself.", "1274172": "My biggest concern is that I doubt the hidden test data is exhaustively labeled. This is why non-string matching models, which generalize, are doing so poorly. They are capturing legitimate dataset mentions that are not labeled. Good generalization, what the host obviously wants, is actually being discouraged.\n\nWould it be possible for Kaggle staff to take a close look at the hidden test data to see if my suspicion about unlabeled dataset mentions is valid? If it's valid, then I don't see how this is anything other than a string matching competition.", "1274407": "> How do you do proper CV if your hold-out set isn't complete. \n\nMore and more Kaggle competitions are like this, BirdCLEF and Human Protein Atlas are weakly labelled too for instance. \n\nThe days of supervised learning are over, state of the art is moving to semi supervised learning and self supervised learning. So is Kaggle.", "1274409": "> I doubt the hidden test data is exhaustively labeled\n\nOn what ground?", "1274561": "I am doubting the hidden test data is exhaustively labeled based on the discussion topic [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/231893). Since I created that topic, I've discovered other datasets that do not appear to be labeled in the hidden test data.", "1274572": "Human annotations for NLP tasks are inherently noisy and I expect there to be a fair number of labelling errors/misses even with a rigorous multi-annotator methodology. Just eyeball some of the longer training docs with a lot of formatting errors and you'd be hard-pressed to pick out all the labels. \n\nThe original metric of using an f-measure with 0 beta would have mitigated that by allowing us to pick a single, high confidence prediction for each ID. Switching to 0.5 beta was a kludge so that the LB wouldn't be filled with perfect scores but the unintended consequence was to make the actual hidden test set much harder to solve. \n\nI guess this is a long-winded way to say \"good luck to everyone, we'll need it\"", "1274596": "> I've discovered other datasets that do not appear to be labeled in the hidden test data.\n\nHow can you see the hidden dataset?", "1274600": "> Switching to 0.5 beta was a kludge so that the LB wouldn't be filled with perfect scores but the unintended consequence was to make the actual hidden test set much harder to solve. \n\nYou nailed it. All questions asked about ground truth are indeed due to this change. Because it is unclear how redundant predictions are handled now.", "1274639": "CPMP, I cannot see the hidden dataset. The hidden dataset is not definitively missing labels. My observations are demonstrating that there is reason to doubt that the hidden dataset is exhaustively labeled. It is unknown just how noisy labeling is. My concern is that it is noisy enough to discourage models that generalize.", "1276645": "philculliton test data do not seem to be fully labeled. Would you mind having a look [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/233170)? Thanks" }, "source": "meta" }