The hosts have confirmed that the test data are exhaustively labelled. This is extremely important as otherwise models that generalize well and discover new datasets would be heavily penalized. In contrast with the mission of this competition from the description page.
\nThe problem is that the test data are NOT exhaustively labeled.
\nBelow I describe the experiments I performed that lead me to this conclusion.
Experiment 1 I submitted the simple baseline model based on string matching (I know it's not the goal of the competition) that achieves 0.700 on the LB, as most of the participants.
\nExperiment 2 I found a new dataset not mentioned in the training set, namely \"Assessment of Educational Progress\". From publication 9d75e205-3030-44c6-ae21-fc25be319189 I check that this refers to a proper dataset:
\n\nThe National Assessment of Educational Progress (NAEP) provides scores that are comparable across states, but these are only available for a sample of students within a small sample of U.S. schools. In order to make the state test scores more meaningful, we have recalibrated the percent passing scores as percentiles of school performance within the state (following the approach by Logan, Minca and Adar 2012).
\n
Cool! Let's add assessment of educational progress (after cleaning) to the list of strings to match in the baseline model. Note: assessment of educational progress should be a true positive also for the longer label national assessment of educational progress naep as the Jaccardi score is above 0.5.
If a match was found within a publication, then assessment of educational progress was only included once in the submitted cleaned labels for the publication, in the same way as the setup described here. My scored dropped to 0.679. What?? OK, no panic! The hosts want a rich semantic context, so maybe my string matching baseline is predicting assessment of educational progress even in those cases where it is not referenced as a dataset but it is interpreted as something else (semantics). But then if I predict a random string my score would drop even more. Let's try it!
Experiment 3 Same baseline model + if a match was found within a publication for assessment of educational progress, then ThisStringWillNeverMatch was only included once in the submitted cleaned labels for the publication. Almost same score 0.676. Ok, I give up… whether \"Assessment of Educational Progress\" is almost never referenced as a dataset (which is hardly believable) or it is very weakly labeled in the test data.
Extra I tried with different datasets and same outcome. You can find the replication code here
\nUpdate: Moreover, can anyone explain why naive matching like if dataset in text: achieves a better score than matching only if the dataset is not a substring of something else, e.g. if regex.search(f\"\\\\b{dataset}\\\\b\", text): ?? Do we really want to predict the dataset \"ADNI\" in the following cases??
ID a8997f97-ba55-46f8-aef7-c1edce2140ae:
\n\nBekhradnia and Sastry (2005) study the in-and outflows of academic researchers in the UK using
\n
ID 29e11717-f3b4-4acb-8d97-ef9b6b428971:
\n\nhave much to contribute to methodologies for the pursuit of accuracy in adninistrative records.
\n
From the leaderboard it seems so…
\nThis raises major concerns about this competition and the degree of randomness in the evaluation (both public and private). Models that generalize well and discover new datasets would be heavily penalized, and I can't understand why this outcome would be optimal for the hosts. I would very much appreciate if someone can say otherwise and explain why!
", "messages": [ { "id": 1276573, "postDate": "2021-04-17T17:48:30.363Z", "content": "The hosts have confirmed that the test data are exhaustively labelled. This is extremely important as otherwise models that generalize well and discover new datasets would be heavily penalized. In contrast with the mission of this competition from the description page.
\nThe problem is that the test data are NOT exhaustively labeled.
\nBelow I describe the experiments I performed that lead me to this conclusion.
Experiment 1 I submitted the simple baseline model based on string matching (I know it's not the goal of the competition) that achieves 0.700 on the LB, as most of the participants.
\nExperiment 2 I found a new dataset not mentioned in the training set, namely \"Assessment of Educational Progress\". From publication 9d75e205-3030-44c6-ae21-fc25be319189 I check that this refers to a proper dataset:
\n\nThe National Assessment of Educational Progress (NAEP) provides scores that are comparable across states, but these are only available for a sample of students within a small sample of U.S. schools. In order to make the state test scores more meaningful, we have recalibrated the percent passing scores as percentiles of school performance within the state (following the approach by Logan, Minca and Adar 2012).
\n
Cool! Let's add assessment of educational progress (after cleaning) to the list of strings to match in the baseline model. Note: assessment of educational progress should be a true positive also for the longer label national assessment of educational progress naep as the Jaccardi score is above 0.5.
If a match was found within a publication, then assessment of educational progress was only included once in the submitted cleaned labels for the publication, in the same way as the setup described here. My scored dropped to 0.679. What?? OK, no panic! The hosts want a rich semantic context, so maybe my string matching baseline is predicting assessment of educational progress even in those cases where it is not referenced as a dataset but it is interpreted as something else (semantics). But then if I predict a random string my score would drop even more. Let's try it!
Experiment 3 Same baseline model + if a match was found within a publication for assessment of educational progress, then ThisStringWillNeverMatch was only included once in the submitted cleaned labels for the publication. Almost same score 0.676. Ok, I give up… whether \"Assessment of Educational Progress\" is almost never referenced as a dataset (which is hardly believable) or it is very weakly labeled in the test data.
Extra I tried with different datasets and same outcome. You can find the replication code here
\nUpdate: Moreover, can anyone explain why naive matching like if dataset in text: achieves a better score than matching only if the dataset is not a substring of something else, e.g. if regex.search(f\"\\\\b{dataset}\\\\b\", text): ?? Do we really want to predict the dataset \"ADNI\" in the following cases??
ID a8997f97-ba55-46f8-aef7-c1edce2140ae:
\n\nBekhradnia and Sastry (2005) study the in-and outflows of academic researchers in the UK using
\n
ID 29e11717-f3b4-4acb-8d97-ef9b6b428971:
\n\nhave much to contribute to methodologies for the pursuit of accuracy in adninistrative records.
\n
From the leaderboard it seems so…
\nThis raises major concerns about this competition and the degree of randomness in the evaluation (both public and private). Models that generalize well and discover new datasets would be heavily penalized, and I can't understand why this outcome would be optimal for the hosts. I would very much appreciate if someone can say otherwise and explain why!
", "rawMarkdown": "The hosts have confirmed that the test data are [exhaustively labelled](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228322#1253208). This is extremely important as otherwise models that **generalize well** and discover new datasets would be **heavily penalized**. In contrast with the mission of this competition from the description page.\n\n**The problem is that the test data are NOT exhaustively labeled.** \nBelow I describe the experiments I performed that lead me to this conclusion.\n\n**Experiment 1** I submitted the simple baseline model based on string matching (I know it's not the goal of the competition) that achieves 0.700 on the LB, as most of the participants.\n\n**Experiment 2** I found a new dataset not mentioned in the training set, namely ~~\"Youth Risk Behavior Survey\"~~ \"Assessment of Educational Progress\". From publication ~~`34f6924a-a599-4566-9dc6-521249174721`~~ `9d75e205-3030-44c6-ae21-fc25be319189` I check that this refers to a proper dataset: \n> The National Assessment of Educational Progress (NAEP) provides scores that are comparable across states, but these are only available for a sample of students within a small sample of U.S. schools. In order to make the state test scores more meaningful, we have recalibrated the percent passing scores as percentiles of school performance within the state (following the approach by Logan, Minca and Adar 2012).\n\nCool! Let's add `assessment of educational progress` (after cleaning) to the list of strings to match in the baseline model. Note: `assessment of educational progress` should be a true positive also for the longer label `national assessment of educational progress naep` as the Jaccardi score is above 0.5. \n\nIf a match was found within a publication, then `assessment of educational progress` was only included once in the submitted cleaned labels for the publication, in the same way as the setup described [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/231893). My scored dropped to 0.679. What?? OK, no panic! The hosts want a rich semantic context, so maybe my string matching baseline is predicting `assessment of educational progress` even in those cases where it is not referenced as a dataset but it is interpreted as something else (semantics). But then if I predict a random string my score would drop even more. Let's try it!\n \n**Experiment 3** Same baseline model + if a match was found within a publication for `assessment of educational progress`, then `ThisStringWillNeverMatch` was only included once in the submitted cleaned labels for the publication. Almost same score 0.676. Ok, I give up... whether \"Assessment of Educational Progress\" is almost never referenced as a dataset (which is hardly believable) or it is very weakly labeled in the test data.\n\n**Extra** I tried with different datasets and same outcome. You can find the replication code [here](https://www.kaggle.com/eguidotti/test-data-are-not-fully-labeled)\n\n**Update**: Moreover, can anyone explain why naive matching like `if dataset in text:` achieves a better score than matching only if the dataset is not a substring of something else, e.g. `if regex.search(f\"\\\\b{dataset}\\\\b\", text):` ?? Do we really want to predict the dataset \"ADNI\" in the following cases??\n\nID `a8997f97-ba55-46f8-aef7-c1edce2140ae`:\n> Bekhr**adni**a and Sastry (2005) study the in-and outflows of academic researchers in the UK using\n\nID `29e11717-f3b4-4acb-8d97-ef9b6b428971`:\n> have much to contribute to methodologies for the pursuit of accuracy in **adni**nistrative records.\n\nFrom the leaderboard it seems so...\n\nThis raises major concerns about this competition and the degree of randomness in the evaluation (both public and private). Models that generalize well and discover new datasets would be heavily penalized, and I can't understand why this outcome would be optimal for the hosts. I would very much appreciate if someone can say otherwise and explain why!", "votes": 32 }, { "id": 1293388, "postDate": "2021-05-04T19:46:31.060Z", "content": "Hey have a look here, maybe this is going to be fixed by the hosts
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236508
Hallelujah ! Well that gave me a nice boost haha 😃
", "rawMarkdown": "Hallelujah ! Well that gave me a nice boost haha 😃" } ] }, { "id": 1277007, "postDate": "2021-04-18T09:06:50.960Z", "content": "I might be misreading your scenario but isn't the dataset in your example already labeled by the provided train label file as \"school survey on crime and safety\" for that specific ID? It might be a problem that \"school survey on crime and safety ssocs\" dings you but it's hard to tell if there's another error in your setup.
\nI don't disagree with the concern that there seem to be issues with how exhaustively the test set is labelled but this isn't the best example.
", "rawMarkdown": "I might be misreading your scenario but isn't the dataset in your example already labeled by the provided train label file as \"school survey on crime and safety\" for that specific ID? It might be a problem that \"school survey on crime and safety ssocs\" dings you but it's hard to tell if there's another error in your setup. \n\nI don't disagree with the concern that there seem to be issues with how exhaustively the test set is labelled but this isn't the best example. ", "votes": 3, "replies": [ { "id": 1277033, "postDate": "2021-04-18T10:16:00.277Z", "content": "Thanks @leecming! You are perfectly right. It seems I overlooked school survey on crime and safety in the training data. I apologize for that. So basically, matching also survey on crime and safety was adding duplicates that lead to a lower score. Makes sense.
However, I tried with other datasets and the problem is the same. I agree with you that my example was not the best one. So I updated the description above and used youth risk behavior survey, which I'm pretty sure it is not labeled at all in the training set. The score drops from 0.70 to 0.42, even worse than the previous example.
You can find the replication code here.
", "rawMarkdown": "Thanks @leecming! You are perfectly right. It seems I overlooked `school survey on crime and safety` in the training data. I apologize for that. So basically, matching also `survey on crime and safety` was adding duplicates that lead to a lower score. Makes sense. \n\nHowever, I tried with other datasets and the problem is the same. I agree with you that my example was not the best one. So I updated the description above and used `youth risk behavior survey`, which I'm pretty sure it is not labeled at all in the training set. The score drops from 0.70 to 0.42, even worse than the previous example. \n\nYou can find the replication code [here](https://www.kaggle.com/eguidotti/test-data-are-not-fully-labeled). ", "votes": 2 }, { "id": 1290366, "postDate": "2021-05-01T21:36:18.457Z", "content": "A bit late to the party, but could it be because:
\n... you'll identify data sets that the publications' authors used in their work. \nAnd in this case, the dataset is not used by the authors but simply referenced from another's work?
", "rawMarkdown": "A bit late to the party, but could it be because:\n```\n... you'll identify data sets that the publications' authors used in their work. \n```\n\nAnd in this case, the dataset is not used by the authors but simply referenced from another's work?\n\n\n", "votes": 1 }, { "id": 1290388, "postDate": "2021-05-01T22:24:39.433Z", "content": "Still not convinced…. given the score I get that would mean youth risk behavior survey is always referenced from another's work and never used by the authors. Also, this wouldn't explain why naive matching works best that matching only if the dataset is not a substring of something else.
I had a look at your example kernel and there is a bug?
\nYou have:
for dataset in datasets:\n if dataset in text:\n matches += [dataset]\n if 'youth risk behavior survey' in text:\n matches += ['ThisStringWillNeverMatch']\nBut you should probably have:
\nfor dataset in datasets:\n if dataset in text:\n matches += [dataset]\nif 'youth risk behavior survey' in text:\n matches += ['ThisStringWillNeverMatch']\nOtherwise, you are actually submitting ThisStringWillNeverMatch 130 times everytime youth risk behavior survey appears.
If you do it in the second way then the score goes from 0.701 to 0.697, instead of 0.701 to your 0.42.
\nI also tried doing:
\nfor dataset in datasets:\n if dataset in text:\n matches += [dataset]\nif 'youth risk behavior survey' in text:\n matches += ['youth risk behavior survey']\nIn which case the score did not change (.701)
\nSo for this particular case, for all the of the papers where 'youth risk behavior survey' appears, probably half have it as labels and half don't have it (assuming I would get a score boost if the 100% of the papers with 'youth risk behavior survey' also had that label).
I do think the split was done weirdly, it's mad that you are able to get a higher accuracy by naively matching string like adni. My guess here is because adni is still a relatively rarely occuring substring and there are more documents with adni that use the study adni than documents that simply contain the substring but not the study.
Let me know what you think
", "rawMarkdown": "I had a look at your example kernel and there is a bug? \nYou have:\n```python\nfor dataset in datasets:\n if dataset in text:\n matches += [dataset]\n if 'youth risk behavior survey' in text:\n matches += ['ThisStringWillNeverMatch']\n```\n\nBut you should probably have:\n```python\nfor dataset in datasets:\n if dataset in text:\n matches += [dataset]\nif 'youth risk behavior survey' in text:\n matches += ['ThisStringWillNeverMatch']\n```\n\nOtherwise, you are actually submitting `ThisStringWillNeverMatch` 130 times everytime `youth risk behavior survey` appears. \n\nIf you do it in the second way then the score goes from 0.701 to 0.697, instead of 0.701 to your 0.42.\n\nI also tried doing:\n\n```python\nfor dataset in datasets:\n if dataset in text:\n matches += [dataset]\nif 'youth risk behavior survey' in text:\n matches += ['youth risk behavior survey']\n```\n\nIn which case the score did not change (.701)\n\nSo for this particular case, for all the of the papers where `'youth risk behavior survey'` appears, probably half have it as labels and half don't have it (assuming I would get a score boost if the 100% of the papers with `'youth risk behavior survey'` also had that label).\n\nI do think the split was done weirdly, it's mad that you are able to get a higher accuracy by naively matching string like `adni`. My guess here is because `adni` is still a relatively rarely occuring substring and there are more documents with adni that use the study adni than documents that simply contain the substring but not the study. \n\nLet me know what you think", "votes": 2 }, { "id": 1292360, "postDate": "2021-05-03T21:22:27.343Z", "content": "Yes, that was definitely a bug. Thanks for letting me know! This explains the huge drop in the score (now fixed). But I've been trying with other datasets and the problem is always the same. The more datasets I add, the more the score drops. And in such a way that it is almost the same if I predict the dataset name or ThisStringWillNeverMatch. E.g. with assessment of educational progress the score drops from 0.700 to 0.679 and it is 0.676 when I predict ThisStringWillNeverMatch. See the updated description above.
I don't know. It may be the case that I'm still doing something wrong (and I hope so), but I've been running quite a few tests and they all seem to suggest that the test data are not fully labeled. Another option would be that the fraction of \"correct mentions\" (whatever it means) is very small compared to the number of times the dataset name appear in the text. But in this case we should have been given more info about what a correct mention is, or alternatively a fully labeled training set to learn from. But the train data have been confirmed not to be fully labeled, the test data seem to be weakly labeled, and the prediction labels are amiguous… this looks like random guess!
", "rawMarkdown": "Yes, that was definitely a bug. Thanks for letting me know! This explains the huge drop in the score (now fixed). But I've been trying with other datasets and the problem is always the same. The more datasets I add, the more the score drops. And in such a way that it is almost the same if I predict the dataset name or `ThisStringWillNeverMatch`. E.g. with `assessment of educational progress` the score drops from 0.700 to 0.679 and it is 0.676 when I predict `ThisStringWillNeverMatch`. See the updated description above.\n\nI don't know. It may be the case that I'm still doing something wrong (and I hope so), but I've been running quite a few tests and they all seem to suggest that the test data are not fully labeled. Another option would be that the fraction of \"correct mentions\" (whatever it means) is very small compared to the number of times the dataset name appear in the text. But in this case we should have been given more info about what a correct mention is, or alternatively a fully labeled training set to learn from. But the [train data have been confirmed not to be fully labeled](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228322), the test data seem to be weakly labeled, and the [prediction labels are amiguous](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/231893)... this looks like random guess!", "votes": 2 }, { "id": 1292733, "postDate": "2021-05-04T08:43:58.787Z", "content": "Yeah, I, unfortunately, have to agree, a lot of the labels are random…
\nI've experimented quite a bit as well with manually probing \"unlabeled\" datasets and nothing I tried helps.
Looking more deeply at some of the data, it makes no sense why some are even labelled with certain gold labels. Some studies introduce and use new samples of data, and only mention the main databases like National Education Longitudinal Study as a previous work reference. In which case, would why would the organizers want it labelled as gold (if the study/data is not actually being used)?
Also some of the reference \"datasets\" are not even datasets at all - the SLOSH model for instance is an algorithm/method, which makes no sense for us to try to identify in the same context as databases/datasets (https://www.nhc.noaa.gov/surge/slosh.php).
Add to this, the inconsistency at which certain acronyms are labelled and some are not, it doesn't really bode well that generalised solutions are harmed rather than rewarded.
\nLooks like you recently got a nice random guess though haha :) Or maybe a way out of this mess?
", "rawMarkdown": "Yeah, I, unfortunately, have to agree, a lot of the labels are random...\nI've experimented quite a bit as well with manually probing \"unlabeled\" datasets and nothing I tried helps. \n\nLooking more deeply at some of the data, it makes no sense why some are even labelled with certain gold labels. Some studies introduce and use new samples of data, and only mention the main databases like `National Education Longitudinal Study` as a previous work reference. In which case, would why would the organizers want it labelled as gold (if the study/data is not actually being used)?\n\nAlso some of the reference \"datasets\" are not even datasets at all - the `SLOSH model` for instance is an algorithm/method, which makes no sense for us to try to identify in the same context as databases/datasets (https://www.nhc.noaa.gov/surge/slosh.php). \n\nAdd to this, the inconsistency at which certain acronyms are labelled and some are not, it doesn't really bode well that generalised solutions are harmed rather than rewarded.\n\nLooks like you recently got a nice random guess though haha :) Or maybe a way out of this mess?", "votes": 2 }, { "id": 1292766, "postDate": "2021-05-04T09:11:47.603Z", "content": "Thanks for sharing your tests, I'm also wondering about your questions but wasn't able to find an answer.
\n\n\nLooks like you recently got a nice random guess though haha :)
\n
Ahah yes it's crazy I'm just doing some tests on string matching and got 10th on the leaderboard! That makes no sense at all. Hope @philculliton will be able to shed some light on this soon.
", "rawMarkdown": "Thanks for sharing your tests, I'm also wondering about your questions but wasn't able to find an answer.\n\n> Looks like you recently got a nice random guess though haha :)\n\nAhah yes it's crazy I'm just doing some tests on string matching and got 10th on the leaderboard! That makes no sense at all. Hope @philculliton will be able to shed some light on this soon.", "votes": 2 } ] }, { "id": 1290815, "postDate": "2021-05-02T12:51:28.207Z", "content": "I think the ground truths are not very accurate in the test set and the training set is definitely not exhaustively labeled IMO. I have found these datasets (hand checked many) that are Gov't data and are not in the labelled training data. When I add these datasets to my \"is in\" test, my score goes down from 70% to 67% - this indicates to me that my system is finding some of these in the test documents, but they are likely not in the ground truths. Obviously it could be some of these are not proper datasets, titles etc., but this demonstrates the point, how did they create the ground truths on all of these documents - seems like an exhaustive labeling system is what we are trying to build. So if they have that why do they need us? If they are doing the ground truths by hand, it is hard to think it is catching ALL the government datasets and variations in citations.
\nlocal food economic impact assessment
\ntvb model
\nmexican american study project
\nnational longitudinal study of adolescent health
\nnational health interview survey
\nproject talent longitudinal study
\nschools and staffing survey
\nprivate school universe survey
\nfast response survey system
\nnational center for education statistics schools and staffing survey
\nnational assessment of educational progress
\ncoinmon core of data
\nnational postsecondary student aid study
\npostsecondary student aid study
\nnational study of postsecondary faculty
\nsurvey on vocational programs in secondary schools
\ndistrict survey of alternative schools and programs
\nstate nonfiscal survey of public elementary secondary education
\nnational public education financial survey
\nannual survey of government finances school systems f 33 survey
\nintegrated postsecondary education data system
\nipeds
\nnsopf
\nbps data
\npostsecondary students longitudinal study
\npublic elementary secondary school universe survey
\nccd local education agency universe survey
\nccd national public education financial survey
\nnces national education longitudinal study of 1988
\nagricultural resource management survey
\nagro ecological zone
\naez model
\nusda corn yield data
\nargonne national laboratory s greet
\ngreet model
\nargonne national laboratory s cclub model
\nnational education longitudinal survey
\neducation longitudinal study of 2002
\nprogress in international reading literacy study
\nusgs national water quality assessment
\nipeds fall enrollment dataset
\nfall enrollment dataset
\ndelta cost project
\nannual survey of colleges standard research compilation
\nchesapeake bay watershed land cover data series
Thanks @mlconsult to confirm also your tests go in the same direction. Just make sure, do not add to the \"is in\" list substrings of datasets that are already in the list. For instance I see postsecondary students longitudinal study which is a substring of beginning postsecondary students longitudinal study that is already labelled in the train data. This may lead to duplicates in the prediction string. But I very much doubt that this is the reason of the drop in the score.
Thanks @eguidotti - to clarify, when I put my predictions in the list I do not include any duplicates.
", "rawMarkdown": "Thanks @eguidotti - to clarify, when I put my predictions in the list I do not include any duplicates." }, { "id": 1293905, "postDate": "2021-05-05T08:51:02.103Z", "content": "Hi @eguidotti can you explain this part to me please \"I see postsecondary students longitudinal study which is a substring of beginning postsecondary students longitudinal study that is already labelled in the train data. This may lead to duplicates in the prediction string. \"
\nI have hand made a list of all the dataset labels I can find in the training set https://www.kaggle.com/cascadenite/training-set-found-labels and many of them are substrings of each other eg:
\nAdult Changes in Thought
\nAdult Changes in Thought ACT
\nAdult Changes in Thought Study ACT
there are also a lot of spelling permutations. Should I only be taking some of them as labels even though they are mentioned in the text as misspelt or shorter versions?
", "rawMarkdown": "Hi @eguidotti can you explain this part to me please \"I see postsecondary students longitudinal study which is a substring of beginning postsecondary students longitudinal study that is already labelled in the train data. This may lead to duplicates in the prediction string. \"\n\nI have hand made a list of all the dataset labels I can find in the training set [https://www.kaggle.com/cascadenite/training-set-found-labels](url) and many of them are substrings of each other eg:\n\nAdult Changes in Thought\nAdult Changes in Thought ACT\nAdult Changes in Thought Study ACT\n\nthere are also a lot of spelling permutations. Should I only be taking some of them as labels even though they are mentioned in the text as misspelt or shorter versions?" }, { "id": 1294075, "postDate": "2021-05-05T11:54:43.210Z", "content": "Let's take Adult Changes in Thought and Adult Changes in Thought ACT as an example. If you add both of them to the \"is in\" list of strings to match and the publication contains something like:
\n\n[…] Adult Changes in Thought ACT […]
\n
then both strings will match the publication. You will end up predicting both Adult Changes in Thought and Adult Changes in Thought ACT even if the publication contains only a single mention for Adult Changes in Thought ACT. This is a duplicate that would be penalized.
From my understanding, we should predict the exact mention (even if misspelled) but it's still not clear to me how to define the boundaries of the mentions, sometimes the training set seems to be labelled only with the acronym, sometimes with tha dataset name, sometimes both. No clue.
", "rawMarkdown": "Let's take `Adult Changes in Thought` and `Adult Changes in Thought ACT` as an example. If you add both of them to the \"is in\" list of strings to match and the publication contains something like:\n> [...] Adult Changes in Thought ACT [...]\n\nthen both strings will match the publication. You will end up predicting both `Adult Changes in Thought` and `Adult Changes in Thought ACT` even if the publication contains only a single mention for `Adult Changes in Thought ACT`. This is a duplicate that would be penalized. \n\nFrom my understanding, we should predict the exact mention (even if misspelled) but it's still not clear to me how to define the boundaries of the mentions, sometimes the training set seems to be labelled only with the acronym, sometimes with tha dataset name, sometimes both. No clue.", "votes": 1 } ] }, { "id": 1290634, "postDate": "2021-05-02T08:00:32.880Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1293388, "author_name": "Emanuele Guidotti", "author_url": "", "post_date": "2021-05-04T19:46:31.060000", "content": "Hey have a look here, maybe this is going to be fixed by the hosts
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236508
Hallelujah ! Well that gave me a nice boost haha 😃
", "votes": 0, "replies": [] } ] }, { "id": 1277007, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2021-04-18T09:06:50.960000", "content": "I might be misreading your scenario but isn't the dataset in your example already labeled by the provided train label file as \"school survey on crime and safety\" for that specific ID? It might be a problem that \"school survey on crime and safety ssocs\" dings you but it's hard to tell if there's another error in your setup.
\nI don't disagree with the concern that there seem to be issues with how exhaustively the test set is labelled but this isn't the best example.
", "votes": 3, "replies": [ { "id": 1277033, "author_name": "Emanuele Guidotti", "author_url": "", "post_date": "2021-04-18T10:16:00.277000", "content": "Thanks @leecming! You are perfectly right. It seems I overlooked school survey on crime and safety in the training data. I apologize for that. So basically, matching also survey on crime and safety was adding duplicates that lead to a lower score. Makes sense.
However, I tried with other datasets and the problem is the same. I agree with you that my example was not the best one. So I updated the description above and used youth risk behavior survey, which I'm pretty sure it is not labeled at all in the training set. The score drops from 0.70 to 0.42, even worse than the previous example.
You can find the replication code here.
", "votes": 2, "replies": [] }, { "id": 1290366, "author_name": "Gautier", "author_url": "", "post_date": "2021-05-01T21:36:18.457000", "content": "A bit late to the party, but could it be because:
\n... you'll identify data sets that the publications' authors used in their work. \nAnd in this case, the dataset is not used by the authors but simply referenced from another's work?
", "votes": 1, "replies": [] }, { "id": 1290388, "author_name": "Emanuele Guidotti", "author_url": "", "post_date": "2021-05-01T22:24:39.433000", "content": "Still not convinced…. given the score I get that would mean youth risk behavior survey is always referenced from another's work and never used by the authors. Also, this wouldn't explain why naive matching works best that matching only if the dataset is not a substring of something else.
I had a look at your example kernel and there is a bug?
\nYou have:
for dataset in datasets:\n if dataset in text:\n matches += [dataset]\n if 'youth risk behavior survey' in text:\n matches += ['ThisStringWillNeverMatch']\nBut you should probably have:
\nfor dataset in datasets:\n if dataset in text:\n matches += [dataset]\nif 'youth risk behavior survey' in text:\n matches += ['ThisStringWillNeverMatch']\nOtherwise, you are actually submitting ThisStringWillNeverMatch 130 times everytime youth risk behavior survey appears.
If you do it in the second way then the score goes from 0.701 to 0.697, instead of 0.701 to your 0.42.
\nI also tried doing:
\nfor dataset in datasets:\n if dataset in text:\n matches += [dataset]\nif 'youth risk behavior survey' in text:\n matches += ['youth risk behavior survey']\nIn which case the score did not change (.701)
\nSo for this particular case, for all the of the papers where 'youth risk behavior survey' appears, probably half have it as labels and half don't have it (assuming I would get a score boost if the 100% of the papers with 'youth risk behavior survey' also had that label).
I do think the split was done weirdly, it's mad that you are able to get a higher accuracy by naively matching string like adni. My guess here is because adni is still a relatively rarely occuring substring and there are more documents with adni that use the study adni than documents that simply contain the substring but not the study.
Let me know what you think
", "votes": 2, "replies": [] }, { "id": 1292360, "author_name": "Emanuele Guidotti", "author_url": "", "post_date": "2021-05-03T21:22:27.343000", "content": "Yes, that was definitely a bug. Thanks for letting me know! This explains the huge drop in the score (now fixed). But I've been trying with other datasets and the problem is always the same. The more datasets I add, the more the score drops. And in such a way that it is almost the same if I predict the dataset name or ThisStringWillNeverMatch. E.g. with assessment of educational progress the score drops from 0.700 to 0.679 and it is 0.676 when I predict ThisStringWillNeverMatch. See the updated description above.
I don't know. It may be the case that I'm still doing something wrong (and I hope so), but I've been running quite a few tests and they all seem to suggest that the test data are not fully labeled. Another option would be that the fraction of \"correct mentions\" (whatever it means) is very small compared to the number of times the dataset name appear in the text. But in this case we should have been given more info about what a correct mention is, or alternatively a fully labeled training set to learn from. But the train data have been confirmed not to be fully labeled, the test data seem to be weakly labeled, and the prediction labels are amiguous… this looks like random guess!
", "votes": 2, "replies": [] }, { "id": 1292733, "author_name": "Gautier", "author_url": "", "post_date": "2021-05-04T08:43:58.787000", "content": "Yeah, I, unfortunately, have to agree, a lot of the labels are random…
\nI've experimented quite a bit as well with manually probing \"unlabeled\" datasets and nothing I tried helps.
Looking more deeply at some of the data, it makes no sense why some are even labelled with certain gold labels. Some studies introduce and use new samples of data, and only mention the main databases like National Education Longitudinal Study as a previous work reference. In which case, would why would the organizers want it labelled as gold (if the study/data is not actually being used)?
Also some of the reference \"datasets\" are not even datasets at all - the SLOSH model for instance is an algorithm/method, which makes no sense for us to try to identify in the same context as databases/datasets (https://www.nhc.noaa.gov/surge/slosh.php).
Add to this, the inconsistency at which certain acronyms are labelled and some are not, it doesn't really bode well that generalised solutions are harmed rather than rewarded.
\nLooks like you recently got a nice random guess though haha :) Or maybe a way out of this mess?
", "votes": 2, "replies": [] }, { "id": 1292766, "author_name": "Emanuele Guidotti", "author_url": "", "post_date": "2021-05-04T09:11:47.603000", "content": "Thanks for sharing your tests, I'm also wondering about your questions but wasn't able to find an answer.
\n\n\nLooks like you recently got a nice random guess though haha :)
\n
Ahah yes it's crazy I'm just doing some tests on string matching and got 10th on the leaderboard! That makes no sense at all. Hope @philculliton will be able to shed some light on this soon.
", "votes": 2, "replies": [] } ] }, { "id": 1290815, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-05-02T12:51:28.207000", "content": "I think the ground truths are not very accurate in the test set and the training set is definitely not exhaustively labeled IMO. I have found these datasets (hand checked many) that are Gov't data and are not in the labelled training data. When I add these datasets to my \"is in\" test, my score goes down from 70% to 67% - this indicates to me that my system is finding some of these in the test documents, but they are likely not in the ground truths. Obviously it could be some of these are not proper datasets, titles etc., but this demonstrates the point, how did they create the ground truths on all of these documents - seems like an exhaustive labeling system is what we are trying to build. So if they have that why do they need us? If they are doing the ground truths by hand, it is hard to think it is catching ALL the government datasets and variations in citations.
\nlocal food economic impact assessment
\ntvb model
\nmexican american study project
\nnational longitudinal study of adolescent health
\nnational health interview survey
\nproject talent longitudinal study
\nschools and staffing survey
\nprivate school universe survey
\nfast response survey system
\nnational center for education statistics schools and staffing survey
\nnational assessment of educational progress
\ncoinmon core of data
\nnational postsecondary student aid study
\npostsecondary student aid study
\nnational study of postsecondary faculty
\nsurvey on vocational programs in secondary schools
\ndistrict survey of alternative schools and programs
\nstate nonfiscal survey of public elementary secondary education
\nnational public education financial survey
\nannual survey of government finances school systems f 33 survey
\nintegrated postsecondary education data system
\nipeds
\nnsopf
\nbps data
\npostsecondary students longitudinal study
\npublic elementary secondary school universe survey
\nccd local education agency universe survey
\nccd national public education financial survey
\nnces national education longitudinal study of 1988
\nagricultural resource management survey
\nagro ecological zone
\naez model
\nusda corn yield data
\nargonne national laboratory s greet
\ngreet model
\nargonne national laboratory s cclub model
\nnational education longitudinal survey
\neducation longitudinal study of 2002
\nprogress in international reading literacy study
\nusgs national water quality assessment
\nipeds fall enrollment dataset
\nfall enrollment dataset
\ndelta cost project
\nannual survey of colleges standard research compilation
\nchesapeake bay watershed land cover data series
Thanks @mlconsult to confirm also your tests go in the same direction. Just make sure, do not add to the \"is in\" list substrings of datasets that are already in the list. For instance I see postsecondary students longitudinal study which is a substring of beginning postsecondary students longitudinal study that is already labelled in the train data. This may lead to duplicates in the prediction string. But I very much doubt that this is the reason of the drop in the score.
Thanks @eguidotti - to clarify, when I put my predictions in the list I do not include any duplicates.
", "votes": 0, "replies": [] }, { "id": 1293905, "author_name": "Mark P", "author_url": "", "post_date": "2021-05-05T08:51:02.103000", "content": "Hi @eguidotti can you explain this part to me please \"I see postsecondary students longitudinal study which is a substring of beginning postsecondary students longitudinal study that is already labelled in the train data. This may lead to duplicates in the prediction string. \"
\nI have hand made a list of all the dataset labels I can find in the training set https://www.kaggle.com/cascadenite/training-set-found-labels and many of them are substrings of each other eg:
\nAdult Changes in Thought
\nAdult Changes in Thought ACT
\nAdult Changes in Thought Study ACT
there are also a lot of spelling permutations. Should I only be taking some of them as labels even though they are mentioned in the text as misspelt or shorter versions?
", "votes": 0, "replies": [] }, { "id": 1294075, "author_name": "Emanuele Guidotti", "author_url": "", "post_date": "2021-05-05T11:54:43.210000", "content": "Let's take Adult Changes in Thought and Adult Changes in Thought ACT as an example. If you add both of them to the \"is in\" list of strings to match and the publication contains something like:
\n\n[…] Adult Changes in Thought ACT […]
\n
then both strings will match the publication. You will end up predicting both Adult Changes in Thought and Adult Changes in Thought ACT even if the publication contains only a single mention for Adult Changes in Thought ACT. This is a duplicate that would be penalized.
From my understanding, we should predict the exact mention (even if misspelled) but it's still not clear to me how to define the boundaries of the mentions, sometimes the training set seems to be labelled only with the acronym, sometimes with tha dataset name, sometimes both. No clue.
", "votes": 1, "replies": [] } ] }, { "id": 1290634, "author_name": "", "author_url": "", "post_date": "2021-05-02T08:00:32.880000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1276573": "The hosts have confirmed that the test data are [exhaustively labelled](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228322#1253208). This is extremely important as otherwise models that **generalize well** and discover new datasets would be **heavily penalized**. In contrast with the mission of this competition from the description page.\n\n**The problem is that the test data are NOT exhaustively labeled.** \nBelow I describe the experiments I performed that lead me to this conclusion.\n\n**Experiment 1** I submitted the simple baseline model based on string matching (I know it's not the goal of the competition) that achieves 0.700 on the LB, as most of the participants.\n\n**Experiment 2** I found a new dataset not mentioned in the training set, namely ~~\"Youth Risk Behavior Survey\"~~ \"Assessment of Educational Progress\". From publication ~~`34f6924a-a599-4566-9dc6-521249174721`~~ `9d75e205-3030-44c6-ae21-fc25be319189` I check that this refers to a proper dataset: \n> The National Assessment of Educational Progress (NAEP) provides scores that are comparable across states, but these are only available for a sample of students within a small sample of U.S. schools. In order to make the state test scores more meaningful, we have recalibrated the percent passing scores as percentiles of school performance within the state (following the approach by Logan, Minca and Adar 2012).\n\nCool! Let's add `assessment of educational progress` (after cleaning) to the list of strings to match in the baseline model. Note: `assessment of educational progress` should be a true positive also for the longer label `national assessment of educational progress naep` as the Jaccardi score is above 0.5. \n\nIf a match was found within a publication, then `assessment of educational progress` was only included once in the submitted cleaned labels for the publication, in the same way as the setup described [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/231893). My scored dropped to 0.679. What?? OK, no panic! The hosts want a rich semantic context, so maybe my string matching baseline is predicting `assessment of educational progress` even in those cases where it is not referenced as a dataset but it is interpreted as something else (semantics). But then if I predict a random string my score would drop even more. Let's try it!\n \n**Experiment 3** Same baseline model + if a match was found within a publication for `assessment of educational progress`, then `ThisStringWillNeverMatch` was only included once in the submitted cleaned labels for the publication. Almost same score 0.676. Ok, I give up... whether \"Assessment of Educational Progress\" is almost never referenced as a dataset (which is hardly believable) or it is very weakly labeled in the test data.\n\n**Extra** I tried with different datasets and same outcome. You can find the replication code [here](https://www.kaggle.com/eguidotti/test-data-are-not-fully-labeled)\n\n**Update**: Moreover, can anyone explain why naive matching like `if dataset in text:` achieves a better score than matching only if the dataset is not a substring of something else, e.g. `if regex.search(f\"\\\\b{dataset}\\\\b\", text):` ?? Do we really want to predict the dataset \"ADNI\" in the following cases??\n\nID `a8997f97-ba55-46f8-aef7-c1edce2140ae`:\n> Bekhr**adni**a and Sastry (2005) study the in-and outflows of academic researchers in the UK using\n\nID `29e11717-f3b4-4acb-8d97-ef9b6b428971`:\n> have much to contribute to methodologies for the pursuit of accuracy in **adni**nistrative records.\n\nFrom the leaderboard it seems so...\n\nThis raises major concerns about this competition and the degree of randomness in the evaluation (both public and private). Models that generalize well and discover new datasets would be heavily penalized, and I can't understand why this outcome would be optimal for the hosts. I would very much appreciate if someone can say otherwise and explain why!", "1293388": "Hey have a look here, maybe this is going to be fixed by the hosts \nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236508 ", "1277007": "I might be misreading your scenario but isn't the dataset in your example already labeled by the provided train label file as \"school survey on crime and safety\" for that specific ID? It might be a problem that \"school survey on crime and safety ssocs\" dings you but it's hard to tell if there's another error in your setup. \n\nI don't disagree with the concern that there seem to be issues with how exhaustively the test set is labelled but this isn't the best example. ", "1290815": "I think the ground truths are not very accurate in the test set and the training set is definitely not exhaustively labeled IMO. I have found these datasets (hand checked many) that are Gov't data and are not in the labelled training data. When I add these datasets to my \"is in\" test, my score goes down from 70% to 67% - this indicates to me that my system is finding some of these in the test documents, but they are likely not in the ground truths. Obviously it could be some of these are not proper datasets, titles etc., but this demonstrates the point, how did they create the ground truths on all of these documents - seems like an exhaustive labeling system is what we are trying to build. So if they have that why do they need us? If they are doing the ground truths by hand, it is hard to think it is catching ALL the government datasets and variations in citations.\n\nlocal food economic impact assessment\ntvb model\nmexican american study project\nnational longitudinal study of adolescent health\nnational health interview survey\nproject talent longitudinal study\nschools and staffing survey\nprivate school universe survey\nfast response survey system\nnational center for education statistics schools and staffing survey\nnational assessment of educational progress\ncoinmon core of data\nnational postsecondary student aid study\npostsecondary student aid study\nnational study of postsecondary faculty\nsurvey on vocational programs in secondary schools\ndistrict survey of alternative schools and programs\nstate nonfiscal survey of public elementary secondary education\nnational public education financial survey\nannual survey of government finances school systems f 33 survey\nintegrated postsecondary education data system\nipeds\nnsopf\nbps data\npostsecondary students longitudinal study\npublic elementary secondary school universe survey\nccd local education agency universe survey\nccd national public education financial survey\nnces national education longitudinal study of 1988\nagricultural resource management survey\nagro ecological zone\naez model\nusda corn yield data\nargonne national laboratory s greet\ngreet model\nargonne national laboratory s cclub model\nnational education longitudinal survey\neducation longitudinal study of 2002\nprogress in international reading literacy study\nusgs national water quality assessment\nipeds fall enrollment dataset\nfall enrollment dataset\ndelta cost project\nannual survey of colleges standard research compilation\nchesapeake bay watershed land cover data series", "1290634": "" } }