Going through the data I realized that papers have datasets that are not in the labels. Take for example the guid f15e636a-2a86-4e72-b971-6eda79426055.
The label for it is Trends in International Mathematics and Science Study.
Here's an excerpt however from the full text:
\n\n\nWhat iS it? Trends in International Mathematics and Science Study is an assessment of fourth-and eighth-grade math and science knowledge. 33 However, we only reviewed frameworks guiding the math portion in this report. It was created in 1995 by the International Association for the Evaluation of Educational Achievement (IEA), a sixty-eight-member (as of 2009) independent organization of national education research organizations and government education research agencies ; the IEA also oversees the Progress in International Reading Literacy Study (PIRLS). 34 The two tests are run by the TIMSS & PIRLS International Study Center at Boston College. The TIMSS assessment is given every four years.
\n
The question is simple: If I were to predict both or just the not included one, I would be penalized for it… right? Since my understanding from the discussion thread here is that false positives are punished and this, dare I say correct prediction, will sadly fall under the same.
\n@philculliton could you kindly shed some light on it? Thanks a lot!
", "messages": [ { "id": "1264096", "postDate": "04/05/2021 21:46:03", "content": "Going through the data I realized that papers have datasets that are not in the labels. Take for example the guid f15e636a-2a86-4e72-b971-6eda79426055.
The label for it is Trends in International Mathematics and Science Study.
Here's an excerpt however from the full text:
\n\n\nWhat iS it? Trends in International Mathematics and Science Study is an assessment of fourth-and eighth-grade math and science knowledge. 33 However, we only reviewed frameworks guiding the math portion in this report. It was created in 1995 by the International Association for the Evaluation of Educational Achievement (IEA), a sixty-eight-member (as of 2009) independent organization of national education research organizations and government education research agencies ; the IEA also oversees the Progress in International Reading Literacy Study (PIRLS). 34 The two tests are run by the TIMSS & PIRLS International Study Center at Boston College. The TIMSS assessment is given every four years.
\n
The question is simple: If I were to predict both or just the not included one, I would be penalized for it… right? Since my understanding from the discussion thread here is that false positives are punished and this, dare I say correct prediction, will sadly fall under the same.
\n@philculliton could you kindly shed some light on it? Thanks a lot!
", "rawMarkdown": "Going through the data I realized that papers have datasets that are **not** in the labels. Take for example the guid `f15e636a-2a86-4e72-b971-6eda79426055`.\n\nThe label for it is `Trends in International Mathematics and Science Study`.\n\nHere's an excerpt however from the full text:\n\n> What iS it? **Trends in International Mathematics and Science Study** is an assessment of fourth-and eighth-grade math and science knowledge. 33 However, we only reviewed frameworks guiding the math portion in this report. It was created in 1995 by the International Association for the Evaluation of Educational Achievement (IEA), a sixty-eight-member (as of 2009) independent organization of national education research organizations and government education research agencies ; the IEA also oversees the **Progress in International Reading Literacy Study** (PIRLS). 34 The two tests are run by the TIMSS & PIRLS International Study Center at Boston College. The TIMSS assessment is given every four years.\n\nThe question is simple: If I were to predict both or just the not included one, I would be penalized for it... right? Since my understanding from the discussion thread [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/229144) is that false positives are punished and this, dare I say *correct* prediction, will sadly fall under the same.\n\n@philculliton could you kindly shed some light on it? Thanks a lot!", "votes": null }, { "id": "1264228", "postDate": "04/06/2021 02:10:09", "content": "from data description page:
\n\"The goal in this competition is not just to match known dataset strings but to generalize to datasets that have never been seen before using NLP and statistical techniques. A percentage of the public test set publications are drawn from the training set - not all datasets have been identified in train, so these unidentified datasets have been used as a portion of the public test labels. These should serve as guides for the difficult task of labeling the private test set.\"
\nwhat to be predicted is not only labels that present in the train labels, the unseen labels also should be predicted like you mentioned above.
", "rawMarkdown": "from data description page:\n\n\"The goal in this competition is not just to match known dataset strings but to generalize to datasets that have never been seen before using NLP and statistical techniques. A percentage of the public test set publications are drawn from the training set - not all datasets have been identified in train, so these unidentified datasets have been used as a portion of the public test labels. These should serve as guides for the difficult task of labeling the private test set.\"\n\n\nwhat to be predicted is not only labels that present in the train labels, the unseen labels also should be predicted like you mentioned above.", "votes": null }, { "id": "1264518", "postDate": "04/06/2021 08:07:30", "content": "Ah okay. Thanks a lot!
", "rawMarkdown": "Ah okay. Thanks a lot!", "votes": null }, { "id": "1274319", "postDate": "04/15/2021 07:27:30", "content": "As replied by Uday Kumar Gurugubelli, you should predict any label that refers to a database.
\nHowever, databases in the hidden test data set seem to be under-labeled. This might explain why NER recognition approaches seem to underperform compared to simple string matching. I think the host is working on this issue.
", "rawMarkdown": "As replied by Uday Kumar Gurugubelli, you should predict any label that refers to a database.\n\nHowever, databases in the hidden test data set seem to be under-labeled. This might explain why NER recognition approaches seem to underperform compared to simple string matching. I think the host is working on this issue.", "votes": null } ], "comments": [ { "id": 1264228, "author_name": "udaygurugubelli", "author_url": "", "post_date": "04/06/2021 02:10:09", "content": "from data description page:
\n\"The goal in this competition is not just to match known dataset strings but to generalize to datasets that have never been seen before using NLP and statistical techniques. A percentage of the public test set publications are drawn from the training set - not all datasets have been identified in train, so these unidentified datasets have been used as a portion of the public test labels. These should serve as guides for the difficult task of labeling the private test set.\"
\nwhat to be predicted is not only labels that present in the train labels, the unseen labels also should be predicted like you mentioned above.
", "votes": null, "replies": [ { "id": 1264518, "author_name": "shiqbal", "author_url": "", "post_date": "04/06/2021 08:07:30", "content": "Ah okay. Thanks a lot!
", "votes": null, "replies": [] } ] }, { "id": 1274319, "author_name": "averywu", "author_url": "", "post_date": "04/15/2021 07:27:30", "content": "As replied by Uday Kumar Gurugubelli, you should predict any label that refers to a database.
\nHowever, databases in the hidden test data set seem to be under-labeled. This might explain why NER recognition approaches seem to underperform compared to simple string matching. I think the host is working on this issue.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1264096": "Going through the data I realized that papers have datasets that are **not** in the labels. Take for example the guid `f15e636a-2a86-4e72-b971-6eda79426055`.\n\nThe label for it is `Trends in International Mathematics and Science Study`.\n\nHere's an excerpt however from the full text:\n\n> What iS it? **Trends in International Mathematics and Science Study** is an assessment of fourth-and eighth-grade math and science knowledge. 33 However, we only reviewed frameworks guiding the math portion in this report. It was created in 1995 by the International Association for the Evaluation of Educational Achievement (IEA), a sixty-eight-member (as of 2009) independent organization of national education research organizations and government education research agencies ; the IEA also oversees the **Progress in International Reading Literacy Study** (PIRLS). 34 The two tests are run by the TIMSS & PIRLS International Study Center at Boston College. The TIMSS assessment is given every four years.\n\nThe question is simple: If I were to predict both or just the not included one, I would be penalized for it... right? Since my understanding from the discussion thread [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/229144) is that false positives are punished and this, dare I say *correct* prediction, will sadly fall under the same.\n\n@philculliton could you kindly shed some light on it? Thanks a lot!", "1264228": "from data description page:\n\n\"The goal in this competition is not just to match known dataset strings but to generalize to datasets that have never been seen before using NLP and statistical techniques. A percentage of the public test set publications are drawn from the training set - not all datasets have been identified in train, so these unidentified datasets have been used as a portion of the public test labels. These should serve as guides for the difficult task of labeling the private test set.\"\n\n\nwhat to be predicted is not only labels that present in the train labels, the unseen labels also should be predicted like you mentioned above.", "1264518": "Ah okay. Thanks a lot!", "1274319": "As replied by Uday Kumar Gurugubelli, you should predict any label that refers to a database.\n\nHowever, databases in the hidden test data set seem to be under-labeled. This might explain why NER recognition approaches seem to underperform compared to simple string matching. I think the host is working on this issue." }, "source": "meta" }