{ "id": 231893, "title": "Label Ambiguity Remains", "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/231893", "author_name": "Trent", "post_date": "2021-04-10T22:27:47.453000", "votes": 18, "comment_count": 15, "views": 0, "content": "

There still seems to be a fair bit of ambiguity surrounding what the labels look like for the hidden test data against which we are scored. I ran two tests that only muddied the waters for me. I'll describe the tests below and provide my interpretation. Please provide some clarity if you can.

\n

Through some analysis leveraging a pre-trained NER model, I found some datasets in the training files that are not in train.csv. I picked two of them to test, American Community Survey (ACS) and National Crime Victimization Survey (NCVS).

\n

Test 1: American Community Survey
\nI added american community survey to the list of strings to match. If a match was found within a publication, then american community survey was only included once in the submitted cleaned labels for the publication. My score dropped by 0.002.

\n

Test 2: National Crime Victimization Survey
\nI added national crime victimization survey to the list of strings to match. If a match was found within a publication, then national crime victimization survey was only included once in the submitted cleaned labels for the publication. My score dropped by 0.001. (Just to be clear, american community survey was removed from the list of strings to match for this test.)

\n

Interpretation

\n\n
    \n
  1. ACS and NCVS (and potentially other datasets) are not included in the labels of hidden test data set on which we are scored. This condition would lead to false positives. If it is the case that a fair number of datasets are not labeled in the hidden test data, then I do not immediately know how to proceed in this competition.
    2. \n
  2. Longer labels like american community survey <extra_word> <extra_word> <extra_word> <extra_word> and national crime victimization survey <extra_word> <extra_word> <extra_word> <extra_word> <extra_word> are more prevalent than I assumed. However, it seems to me that the number of cases with enough extra words to produce a Jaccard Similarity below 0.5 would be relatively infrequent.
    3. \n
\n

Edit:
\nAnother potential factor is that I am incorrectly interpreting the intent of the Evaluation, or the Evaluation is still not quite correct.

", "messages": [ { "id": 1269777, "postDate": "2021-04-10T22:27:47.453Z", "content": "

There still seems to be a fair bit of ambiguity surrounding what the labels look like for the hidden test data against which we are scored. I ran two tests that only muddied the waters for me. I'll describe the tests below and provide my interpretation. Please provide some clarity if you can.

\n

Through some analysis leveraging a pre-trained NER model, I found some datasets in the training files that are not in train.csv. I picked two of them to test, American Community Survey (ACS) and National Crime Victimization Survey (NCVS).

\n

Test 1: American Community Survey
\nI added american community survey to the list of strings to match. If a match was found within a publication, then american community survey was only included once in the submitted cleaned labels for the publication. My score dropped by 0.002.

\n

Test 2: National Crime Victimization Survey
\nI added national crime victimization survey to the list of strings to match. If a match was found within a publication, then national crime victimization survey was only included once in the submitted cleaned labels for the publication. My score dropped by 0.001. (Just to be clear, american community survey was removed from the list of strings to match for this test.)

\n

Interpretation

\n\n
    \n
  1. ACS and NCVS (and potentially other datasets) are not included in the labels of hidden test data set on which we are scored. This condition would lead to false positives. If it is the case that a fair number of datasets are not labeled in the hidden test data, then I do not immediately know how to proceed in this competition.
    2. \n
  2. Longer labels like american community survey <extra_word> <extra_word> <extra_word> <extra_word> and national crime victimization survey <extra_word> <extra_word> <extra_word> <extra_word> <extra_word> are more prevalent than I assumed. However, it seems to me that the number of cases with enough extra words to produce a Jaccard Similarity below 0.5 would be relatively infrequent.
    3. \n
\n

Edit:
\nAnother potential factor is that I am incorrectly interpreting the intent of the Evaluation, or the Evaluation is still not quite correct.

", "rawMarkdown": "There still seems to be a fair bit of ambiguity surrounding what the labels look like for the hidden test data against which we are scored. I ran two tests that only muddied the waters for me. I'll describe the tests below and provide my interpretation. *Please provide some clarity if you can.*\n\nThrough some analysis leveraging a pre-trained NER model, I found some datasets in the training files that are not in train.csv. I picked two of them to test, American Community Survey (ACS) and National Crime Victimization Survey (NCVS).\n\n**Test 1: American Community Survey**\nI added `american community survey` to the list of strings to match. If a match was found within a publication, then `american community survey` was only included once in the submitted cleaned labels for the publication. My score dropped by 0.002.\n\n**Test 2: National Crime Victimization Survey**\nI added `national crime victimization survey` to the list of strings to match. If a match was found within a publication, then `national crime victimization survey` was only included once in the submitted cleaned labels for the publication. My score dropped by 0.001. (Just to be clear, `american community survey` was removed from the list of strings to match for this test.)\n\n**Interpretation**\n- ACS and NCVS are referenced within the text of publications in the hidden test data. If they were not referenced, then my score would not have changed at all.\n- I think that including every reference to a match in the submitted cleaned labels for a publication would have worsened my score even more. I cannot think of a scenario where submitting `american community survey` or `national crime victimization survey` for every mention within a publication would have improved my score given that including the match once for a publication worsened my score.\n- The drop in my score could be due to one or two factors:\n1. ACS and NCVS (and potentially other datasets) are not included in the labels of hidden test data set on which we are scored. This condition would lead to false positives. If it is the case that a fair number of datasets are not labeled in the hidden test data, then I do not immediately know how to proceed in this competition.\n2. Longer labels like `american community survey ` and `national crime victimization survey ` are more prevalent than I assumed. However, it seems to me that the number of cases with enough extra words to produce a Jaccard Similarity below 0.5 would be relatively infrequent.\n\nEdit:\nAnother potential factor is that I am incorrectly interpreting the intent of the Evaluation, or the Evaluation is still not quite correct.", "votes": 18 }, { "id": 1270861, "postDate": "2021-04-12T05:22:58.523Z", "content": "

I'm leaning more and more toward thinking that there are many dataset mentions in the documents that are not labeled. I speculate there is a subset of labels that are associated with the government departments/agencies acknowledged that is missing many datasets mentions. This means that generalization gets penalized due to a high number of false positives. This is why non-string matching methods that generalize are returning low scores for many people including myself -- because they are generalizing outside of the relatively small subset of labels. I hope my speculation is wrong!

", "rawMarkdown": "I'm leaning more and more toward thinking that there are many dataset mentions in the documents that are not labeled. I speculate there is a subset of labels that are associated with the government departments/agencies acknowledged that is missing many datasets mentions. This means that generalization gets penalized due to a high number of false positives. This is why non-string matching methods that generalize are returning low scores for many people including myself -- because they are generalizing outside of the relatively small subset of labels. I hope my speculation is wrong!", "votes": 9 }, { "id": 1269798, "postDate": "2021-04-10T23:18:39.007Z", "content": "

One explanation I could see is that they are labeled as \"ACS\" and \"NCVS\"!?

\n

In one of my tests I changed predicted acronyms to full names and my score dropped.

", "rawMarkdown": "One explanation I could see is that they are labeled as \"ACS\" and \"NCVS\"!?\n\nIn one of my tests I changed predicted acronyms to full names and my score dropped.", "votes": 4, "replies": [ { "id": 1269805, "postDate": "2021-04-10T23:25:39.287Z", "content": "

Good point! That's a possibility. If that were to be the case, then that would mean that the labels are present in the hidden data but that they are bad/incomplete since they should include the full name of the datasets if they appear as full names in the text.

", "rawMarkdown": "Good point! That's a possibility. If that were to be the case, then that would mean that the labels are present in the hidden data but that they are bad/incomplete since they should include the full name of the datasets if they appear as full names in the text.", "votes": 4 } ] }, { "id": 1275022, "postDate": "2021-04-15T20:13:42.773Z", "content": "

I raised a similar point in my post too… Stil no clarity. Hmmm…

", "rawMarkdown": "I raised a similar point in my [post](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230854) too... Stil no clarity. Hmmm...", "votes": 1 }, { "id": 1271088, "postDate": "2021-04-12T09:43:21.497Z", "content": "

Thanks for sharing this. I did similar tests and had the same observations (but not convinced yet that its because of missing labels). Something is clearly off or else misinterpreted considering the hosts have confirmed that the test data is exhaustively labelled 🤔

", "rawMarkdown": "Thanks for sharing this. I did similar tests and had the same observations (but not convinced yet that its because of missing labels). Something is clearly off or else misinterpreted considering the hosts have confirmed that the test data is [exhaustively labelled](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228322#1253208) 🤔", "votes": 2 }, { "id": 1286998, "postDate": "2021-04-28T15:47:21.687Z", "content": "

Thoughts on what a potential winning score looks like - is it 1.0 or is I something like 500.0?

", "rawMarkdown": "Thoughts on what a potential winning score looks like - is it 1.0 or is I something like 500.0? " }, { "id": 1275665, "postDate": "2021-04-16T14:53:14.427Z", "content": "

Not only test data is a problem. Training data also has some issues. Here i tried to pick a good example of labeling i am very suspicious about.

", "rawMarkdown": "Not only test data is a problem. Training data also has some issues. [Here](https://www.kaggle.com/lukaszruczynski/poor-labeling-of-training-data) i tried to pick a good example of labeling i am very suspicious about.", "replies": [ { "id": 1275724, "postDate": "2021-04-16T16:01:09.847Z", "content": "

The training set is not fully labeled as stated in the data description. But I have no clue why they decided to go this way and there seem to be a lot of issues with training data. The following links may help:

\n", "rawMarkdown": "The training set is not fully labeled as stated in the data description. But I have no clue why they decided to go this way and there seem to be a lot of issues with training data. The following links may help: \n- https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228322\n- https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228337", "votes": 1 } ] }, { "id": 1275445, "postDate": "2021-04-16T09:55:31.800Z", "content": "

I couldn't agree more with that! I also run some tests in this direction that confirm the behaviour described above. I'm thinking of the following 2 possibilities:

\n
    \n
  • Longer labels like american community survey <extra_word> <extra_word> <extra_word> <extra_word> are more prevalent than assumed. This might be the case as, from my understanding, the evaluation metrics rewards precision more than recall of a factor of 4 to 1. So for each longer label that is scored as a false positive we need to identify 4 true positives with the shorter label just to have zero improvement in the metric. I hope I'm wrong as this makes little sense to me and at the very least we should have been provided with more instructions on how to format the label precisely (e.g. when to include the extra words? when to include acronyms? etc.).
    • \n
  • Are we sure the dataset has been labeled manually? Has this been confirmed somewhere? Otherwise we are just learning the model with which the data were labeled that makes even less sense to me and I would quit the competition immediately.
    • \n
\n

Please correct me if I'm wrong!

", "rawMarkdown": "I couldn't agree more with that! I also run some tests in this direction that confirm the behaviour described above. I'm thinking of the following 2 possibilities:\n- Longer labels like `american community survey ` are more prevalent than assumed. This might be the case as, from my understanding, the evaluation metrics rewards precision more than recall of a factor of 4 to 1. So for each longer label that is scored as a false positive we need to identify 4 true positives with the shorter label just to have zero improvement in the metric. I hope I'm wrong as this makes little sense to me and at the very least we should have been provided with more instructions on how to format the label precisely (e.g. when to include the extra words? when to include acronyms? etc.).\n- Are we sure the dataset has been labeled manually? Has this been confirmed somewhere? Otherwise we are just learning the model with which the data were labeled that makes even less sense to me and I would quit the competition immediately.\n\nPlease correct me if I'm wrong! " }, { "id": 1271725, "postDate": "2021-04-12T20:59:03.620Z", "content": "

I don't think those are the best examples, to be honest. I can randomly talk about some survey of American communities in a sentence anywhere and that would \"jaccardly\" match the first database you mentioned.

\n

I even think the two words American communities would pass as a match if I understand Jaccard correctly. That means every time an author talks about some random American community you're counting it as a match.

\n

That would explain your drop in scores.

", "rawMarkdown": "I don't think those are the best examples, to be honest. I can randomly talk about some *survey of American communities* in a sentence anywhere and that would \"jaccardly\" match the first database you mentioned. \n\nI even think the two words *American communities* would pass as a match if I understand Jaccard correctly. That means every time an author talks about some random *American community* you're counting it as a match.\n\nThat would explain your drop in scores.", "replies": [ { "id": 1271730, "postDate": "2021-04-12T21:15:43.653Z", "content": "

I'm still taking an exact string matching approach until I better understand the data. The phrases survey of american communities and american communities would not be matched by my string american community survey, so they would never be candidate labels fed to the Jaccard similarity scorer.

", "rawMarkdown": "I'm still taking an exact string matching approach until I better understand the data. The phrases *survey of american communities* and *american communities* would not be matched by my string *american community survey*, so they would never be candidate labels fed to the Jaccard similarity scorer." }, { "id": 1271760, "postDate": "2021-04-12T22:01:23.560Z", "content": "

That doesn't matter.

\n

I see you haven't read the description of the jaccard measure. You need to understand it or you will be very confused. Go to the overview of the competition under the evaluation tab. You will see that the strategy of matching against a list of databases has no hope of working.

\n

It turns out that any anagram of a database will match. Even if only 50% of the letters match.

", "rawMarkdown": "That doesn't matter. \n\nI see you haven't read the description of the jaccard measure. You need to understand it or you will be very confused. Go to the overview of the competition under the evaluation tab. You will see that the strategy of matching against a list of databases has no hope of working. \n\nIt turns out that any anagram of a database will match. Even if only 50% of the letters match.", "votes": -2 }, { "id": 1271785, "postDate": "2021-04-12T22:56:40.717Z", "content": "

The Evaluation tab says that the Jaccard score is \"token-based,\" so it operates at the word level, not the character level.

\n
def jaccard(str1, str2): \n    a = set(str1.lower().split())\n    print('str1:',a)\n    b = set(str2.lower().split())\n    print('str2:',b)\n    c = a.intersection(b)\n    print('intersection:',c)\n    return float(len(c)) / (len(a) + len(b) - len(c))\n\nstr1 = \"survey of american commmunities\"\nstr2 = \"american community survey\"\n\nprint('score:', jaccard(str1, str2))\n
\n

The code above produces the following output:

\n
str1: {'american', 'survey', 'of', 'commmunities'}\nstr2: {'survey', 'american', 'community'}\nintersection: {'survey', 'american'}\nscore: 0.4\n
", "rawMarkdown": "The Evaluation tab says that the Jaccard score is \"token-based,\" so it operates at the word level, not the character level.\n```\ndef jaccard(str1, str2): \n a = set(str1.lower().split())\n print('str1:',a)\n b = set(str2.lower().split())\n print('str2:',b)\n c = a.intersection(b)\n print('intersection:',c)\n return float(len(c)) / (len(a) + len(b) - len(c))\n\nstr1 = \"survey of american commmunities\"\nstr2 = \"american community survey\"\n\nprint('score:', jaccard(str1, str2))\n```\nThe code above produces the following output:\n```\nstr1: {'american', 'survey', 'of', 'commmunities'}\nstr2: {'survey', 'american', 'community'}\nintersection: {'survey', 'american'}\nscore: 0.4\n``` ", "votes": 3 }, { "id": 1271827, "postDate": "2021-04-13T00:17:24.583Z", "content": "

That's interesting! Thanks for clarifying that. I was under the impression that it worked at the character level.

", "rawMarkdown": "That's interesting! Thanks for clarifying that. I was under the impression that it worked at the character level." } ] }, { "id": 1270809, "postDate": "2021-04-12T03:42:09.990Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1270861, "author_name": "Trent", "author_url": "", "post_date": "2021-04-12T05:22:58.523000", "content": "

I'm leaning more and more toward thinking that there are many dataset mentions in the documents that are not labeled. I speculate there is a subset of labels that are associated with the government departments/agencies acknowledged that is missing many datasets mentions. This means that generalization gets penalized due to a high number of false positives. This is why non-string matching methods that generalize are returning low scores for many people including myself -- because they are generalizing outside of the relatively small subset of labels. I hope my speculation is wrong!

", "votes": 9, "replies": [] }, { "id": 1269798, "author_name": "Hannes Öhler", "author_url": "", "post_date": "2021-04-10T23:18:39.007000", "content": "

One explanation I could see is that they are labeled as \"ACS\" and \"NCVS\"!?

\n

In one of my tests I changed predicted acronyms to full names and my score dropped.

", "votes": 4, "replies": [ { "id": 1269805, "author_name": "Trent", "author_url": "", "post_date": "2021-04-10T23:25:39.287000", "content": "

Good point! That's a possibility. If that were to be the case, then that would mean that the labels are present in the hidden data but that they are bad/incomplete since they should include the full name of the datasets if they appear as full names in the text.

", "votes": 4, "replies": [] } ] }, { "id": 1275022, "author_name": "Shabie Iqbal", "author_url": "", "post_date": "2021-04-15T20:13:42.773000", "content": "

I raised a similar point in my post too… Stil no clarity. Hmmm…

", "votes": 1, "replies": [] }, { "id": 1271088, "author_name": "Vopani", "author_url": "", "post_date": "2021-04-12T09:43:21.497000", "content": "

Thanks for sharing this. I did similar tests and had the same observations (but not convinced yet that its because of missing labels). Something is clearly off or else misinterpreted considering the hosts have confirmed that the test data is exhaustively labelled 🤔

", "votes": 2, "replies": [] }, { "id": 1286998, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-04-28T15:47:21.687000", "content": "

Thoughts on what a potential winning score looks like - is it 1.0 or is I something like 500.0?

", "votes": 0, "replies": [] }, { "id": 1275665, "author_name": "ruczu ", "author_url": "", "post_date": "2021-04-16T14:53:14.427000", "content": "

Not only test data is a problem. Training data also has some issues. Here i tried to pick a good example of labeling i am very suspicious about.

", "votes": 0, "replies": [ { "id": 1275724, "author_name": "Emanuele Guidotti", "author_url": "", "post_date": "2021-04-16T16:01:09.847000", "content": "

The training set is not fully labeled as stated in the data description. But I have no clue why they decided to go this way and there seem to be a lot of issues with training data. The following links may help:

\n", "votes": 1, "replies": [] } ] }, { "id": 1275445, "author_name": "Emanuele Guidotti", "author_url": "", "post_date": "2021-04-16T09:55:31.800000", "content": "

I couldn't agree more with that! I also run some tests in this direction that confirm the behaviour described above. I'm thinking of the following 2 possibilities:

\n
    \n
  • Longer labels like american community survey <extra_word> <extra_word> <extra_word> <extra_word> are more prevalent than assumed. This might be the case as, from my understanding, the evaluation metrics rewards precision more than recall of a factor of 4 to 1. So for each longer label that is scored as a false positive we need to identify 4 true positives with the shorter label just to have zero improvement in the metric. I hope I'm wrong as this makes little sense to me and at the very least we should have been provided with more instructions on how to format the label precisely (e.g. when to include the extra words? when to include acronyms? etc.).
    • \n
  • Are we sure the dataset has been labeled manually? Has this been confirmed somewhere? Otherwise we are just learning the model with which the data were labeled that makes even less sense to me and I would quit the competition immediately.
    • \n
\n

Please correct me if I'm wrong!

", "votes": 0, "replies": [] }, { "id": 1271725, "author_name": "Wilmer E. Henao", "author_url": "", "post_date": "2021-04-12T20:59:03.620000", "content": "

I don't think those are the best examples, to be honest. I can randomly talk about some survey of American communities in a sentence anywhere and that would \"jaccardly\" match the first database you mentioned.

\n

I even think the two words American communities would pass as a match if I understand Jaccard correctly. That means every time an author talks about some random American community you're counting it as a match.

\n

That would explain your drop in scores.

", "votes": 0, "replies": [ { "id": 1271730, "author_name": "Trent", "author_url": "", "post_date": "2021-04-12T21:15:43.653000", "content": "

I'm still taking an exact string matching approach until I better understand the data. The phrases survey of american communities and american communities would not be matched by my string american community survey, so they would never be candidate labels fed to the Jaccard similarity scorer.

", "votes": 0, "replies": [] }, { "id": 1271760, "author_name": "Wilmer E. Henao", "author_url": "", "post_date": "2021-04-12T22:01:23.560000", "content": "

That doesn't matter.

\n

I see you haven't read the description of the jaccard measure. You need to understand it or you will be very confused. Go to the overview of the competition under the evaluation tab. You will see that the strategy of matching against a list of databases has no hope of working.

\n

It turns out that any anagram of a database will match. Even if only 50% of the letters match.

", "votes": -2, "replies": [] }, { "id": 1271785, "author_name": "Trent", "author_url": "", "post_date": "2021-04-12T22:56:40.717000", "content": "

The Evaluation tab says that the Jaccard score is \"token-based,\" so it operates at the word level, not the character level.

\n
def jaccard(str1, str2): \n    a = set(str1.lower().split())\n    print('str1:',a)\n    b = set(str2.lower().split())\n    print('str2:',b)\n    c = a.intersection(b)\n    print('intersection:',c)\n    return float(len(c)) / (len(a) + len(b) - len(c))\n\nstr1 = \"survey of american commmunities\"\nstr2 = \"american community survey\"\n\nprint('score:', jaccard(str1, str2))\n
\n

The code above produces the following output:

\n
str1: {'american', 'survey', 'of', 'commmunities'}\nstr2: {'survey', 'american', 'community'}\nintersection: {'survey', 'american'}\nscore: 0.4\n
", "votes": 3, "replies": [] }, { "id": 1271827, "author_name": "Wilmer E. Henao", "author_url": "", "post_date": "2021-04-13T00:17:24.583000", "content": "

That's interesting! Thanks for clarifying that. I was under the impression that it worked at the character level.

", "votes": 0, "replies": [] } ] }, { "id": 1270809, "author_name": "", "author_url": "", "post_date": "2021-04-12T03:42:09.990000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1269777": "There still seems to be a fair bit of ambiguity surrounding what the labels look like for the hidden test data against which we are scored. I ran two tests that only muddied the waters for me. I'll describe the tests below and provide my interpretation. *Please provide some clarity if you can.*\n\nThrough some analysis leveraging a pre-trained NER model, I found some datasets in the training files that are not in train.csv. I picked two of them to test, American Community Survey (ACS) and National Crime Victimization Survey (NCVS).\n\n**Test 1: American Community Survey**\nI added `american community survey` to the list of strings to match. If a match was found within a publication, then `american community survey` was only included once in the submitted cleaned labels for the publication. My score dropped by 0.002.\n\n**Test 2: National Crime Victimization Survey**\nI added `national crime victimization survey` to the list of strings to match. If a match was found within a publication, then `national crime victimization survey` was only included once in the submitted cleaned labels for the publication. My score dropped by 0.001. (Just to be clear, `american community survey` was removed from the list of strings to match for this test.)\n\n**Interpretation**\n- ACS and NCVS are referenced within the text of publications in the hidden test data. If they were not referenced, then my score would not have changed at all.\n- I think that including every reference to a match in the submitted cleaned labels for a publication would have worsened my score even more. I cannot think of a scenario where submitting `american community survey` or `national crime victimization survey` for every mention within a publication would have improved my score given that including the match once for a publication worsened my score.\n- The drop in my score could be due to one or two factors:\n1. ACS and NCVS (and potentially other datasets) are not included in the labels of hidden test data set on which we are scored. This condition would lead to false positives. If it is the case that a fair number of datasets are not labeled in the hidden test data, then I do not immediately know how to proceed in this competition.\n2. Longer labels like `american community survey ` and `national crime victimization survey ` are more prevalent than I assumed. However, it seems to me that the number of cases with enough extra words to produce a Jaccard Similarity below 0.5 would be relatively infrequent.\n\nEdit:\nAnother potential factor is that I am incorrectly interpreting the intent of the Evaluation, or the Evaluation is still not quite correct.", "1270861": "I'm leaning more and more toward thinking that there are many dataset mentions in the documents that are not labeled. I speculate there is a subset of labels that are associated with the government departments/agencies acknowledged that is missing many datasets mentions. This means that generalization gets penalized due to a high number of false positives. This is why non-string matching methods that generalize are returning low scores for many people including myself -- because they are generalizing outside of the relatively small subset of labels. I hope my speculation is wrong!", "1269798": "One explanation I could see is that they are labeled as \"ACS\" and \"NCVS\"!?\n\nIn one of my tests I changed predicted acronyms to full names and my score dropped.", "1275022": "I raised a similar point in my [post](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230854) too... Stil no clarity. Hmmm...", "1271088": "Thanks for sharing this. I did similar tests and had the same observations (but not convinced yet that its because of missing labels). Something is clearly off or else misinterpreted considering the hosts have confirmed that the test data is [exhaustively labelled](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228322#1253208) 🤔", "1286998": "Thoughts on what a potential winning score looks like - is it 1.0 or is I something like 500.0? ", "1275665": "Not only test data is a problem. Training data also has some issues. [Here](https://www.kaggle.com/lukaszruczynski/poor-labeling-of-training-data) i tried to pick a good example of labeling i am very suspicious about.", "1275445": "I couldn't agree more with that! I also run some tests in this direction that confirm the behaviour described above. I'm thinking of the following 2 possibilities:\n- Longer labels like `american community survey ` are more prevalent than assumed. This might be the case as, from my understanding, the evaluation metrics rewards precision more than recall of a factor of 4 to 1. So for each longer label that is scored as a false positive we need to identify 4 true positives with the shorter label just to have zero improvement in the metric. I hope I'm wrong as this makes little sense to me and at the very least we should have been provided with more instructions on how to format the label precisely (e.g. when to include the extra words? when to include acronyms? etc.).\n- Are we sure the dataset has been labeled manually? Has this been confirmed somewhere? Otherwise we are just learning the model with which the data were labeled that makes even less sense to me and I would quit the competition immediately.\n\nPlease correct me if I'm wrong! ", "1271725": "I don't think those are the best examples, to be honest. I can randomly talk about some *survey of American communities* in a sentence anywhere and that would \"jaccardly\" match the first database you mentioned. \n\nI even think the two words *American communities* would pass as a match if I understand Jaccard correctly. That means every time an author talks about some random *American community* you're counting it as a match.\n\nThat would explain your drop in scores.", "1270809": "" } }