According to the Overview Evaluation page, the metric for this competition is the Jaccard Similarity score
\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len(list(set(l1).intersection(l2)))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\nIf s1 is the PredictionString field in the submission.csv, which dataset field is s2 being compared to? pub_title, dataset_title, dataset_label, cleaned_label?
According to the Overview Evaluation page, the metric for this competition is the Jaccard Similarity score
\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len(list(set(l1).intersection(l2)))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\nIf s1 is the PredictionString field in the submission.csv, which dataset field is s2 being compared to? pub_title, dataset_title, dataset_label, cleaned_label?
According to the data description page, it is my understanding that cleaned_label is compared to PredictionString.
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/data
\n\ndataset_label - a portion of the text that indicates the dataset
\n
\n cleaned_label - the dataset_label, as passed through the clean_text function from the Evaluation page
The dataset_label preprocessed by the clean_text function corresponds to the cleaned_label
I have tested this both ways:
\n0.457 score for string literal searching (all fields), and outputting cleaned_label
0.998 score for string literal searching (dataset_title only), and outputting dataset_title
This may potentially be a bug with the Kaggle scoring system and/or test dataset
", "rawMarkdown": "I have tested this both ways:\n\n0.457 score for string literal searching (all fields), and outputting `cleaned_label`\n- https://www.kaggle.com/jamesmcguigan/coleridge-string-literals?scriptVersionId=57586338\n\n0.998 score for string literal searching (`dataset_title` only), and outputting `dataset_title`\n- https://www.kaggle.com/jamesmcguigan/coleridge-string-literals?scriptVersionId=57628162\n\nThis may potentially be a bug with the Kaggle scoring system and/or test dataset", "votes": null }, { "id": "1251068", "postDate": "03/24/2021 13:22:52", "content": "I have understood the intent of your question correctly. I also think that the LB score is strange as you pointed out. I guess we have to wait for the official Kaggle response.
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228216#1250983
\n\n", "rawMarkdown": "I have understood the intent of your question correctly. I also think that the LB score is strange as you pointed out. I guess we have to wait for the official Kaggle response.\n\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228216#1250983\n> There is no dataleak. There seems to be an error in the leaderboard. We are waiting for @philculliton to fix it.", "votes": null }, { "id": "1251609", "postDate": "03/25/2021 00:12:51", "content": "There is no dataleak. There seems to be an error in the leaderboard. We are waiting for @philculliton to fix it.
\n
I also asked a similar question here, and got the following responses:
\n\n\nIn this competition, predictions are drawn from the documents. PredictionString is being compared against extracts from publications that indicate a dataset was mentioned in that publication. The strings that predictions are being compared against have been cleaned, as indicated in the Evaluation page.
\nYes! There are a huge number of labels in test that are not mentioned in train at all. So the task is less to match strings, and more to create an NLP solution for generalizing the \"can I identify datasets mentioned in this document\" problem.
\nCorrect. cleaned_label represents examples of (cleaned) extracted text we're attempting to predict for the training data.
\n
https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228463
", "rawMarkdown": "I also asked a similar question here, and got the following responses:\n\n> In this competition, predictions are drawn from the documents. PredictionString is being compared against extracts from publications that indicate a dataset was mentioned in that publication. The strings that predictions are being compared against have been cleaned, as indicated in the Evaluation page.\n>\n> Yes! There are a huge number of labels in test that are not mentioned in train at all. So the task is less to match strings, and more to create an NLP solution for generalizing the \"can I identify datasets mentioned in this document\" problem.\n>\n> Correct. cleaned_label represents examples of (cleaned) extracted text we're attempting to predict for the training data.\n\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228463", "votes": null } ], "comments": [ { "id": 1250936, "author_name": "kanbehmw", "author_url": "", "post_date": "03/24/2021 11:21:25", "content": "According to the data description page, it is my understanding that cleaned_label is compared to PredictionString.
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/data
\n\ndataset_label - a portion of the text that indicates the dataset
\n
\n cleaned_label - the dataset_label, as passed through the clean_text function from the Evaluation page
The dataset_label preprocessed by the clean_text function corresponds to the cleaned_label
I have tested this both ways:
\n0.457 score for string literal searching (all fields), and outputting cleaned_label
0.998 score for string literal searching (dataset_title only), and outputting dataset_title
This may potentially be a bug with the Kaggle scoring system and/or test dataset
", "votes": null, "replies": [] }, { "id": 1251068, "author_name": "kanbehmw", "author_url": "", "post_date": "03/24/2021 13:22:52", "content": "I have understood the intent of your question correctly. I also think that the LB score is strange as you pointed out. I guess we have to wait for the official Kaggle response.
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228216#1250983
\n\n", "votes": null, "replies": [] } ] }, { "id": 1251609, "author_name": "jamesmcguigan", "author_url": "", "post_date": "03/25/2021 00:12:51", "content": "There is no dataleak. There seems to be an error in the leaderboard. We are waiting for @philculliton to fix it.
\n
I also asked a similar question here, and got the following responses:
\n\n\nIn this competition, predictions are drawn from the documents. PredictionString is being compared against extracts from publications that indicate a dataset was mentioned in that publication. The strings that predictions are being compared against have been cleaned, as indicated in the Evaluation page.
\nYes! There are a huge number of labels in test that are not mentioned in train at all. So the task is less to match strings, and more to create an NLP solution for generalizing the \"can I identify datasets mentioned in this document\" problem.
\nCorrect. cleaned_label represents examples of (cleaned) extracted text we're attempting to predict for the training data.
\n
https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228463
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1250921": "According to the Overview Evaluation page, the metric for this competition is the Jaccard Similarity score\n```\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len(list(set(l1).intersection(l2)))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\n```\n- https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation\n\nIf `s1` is the `PredictionString` field in the `submission.csv`, which dataset field is `s2` being compared to? `pub_title`, `dataset_title`, `dataset_label`, `cleaned_label`?", "1250936": "According to the data description page, it is my understanding that `cleaned_label` is compared to `PredictionString`.\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/data\n\n> dataset_label - a portion of the text that indicates the dataset\ncleaned_label - the dataset_label, as passed through the clean_text function from the Evaluation page\n\nThe `dataset_label` preprocessed by the `clean_text` function corresponds to the `cleaned_label`", "1250976": "I have tested this both ways:\n\n0.457 score for string literal searching (all fields), and outputting `cleaned_label`\n- https://www.kaggle.com/jamesmcguigan/coleridge-string-literals?scriptVersionId=57586338\n\n0.998 score for string literal searching (`dataset_title` only), and outputting `dataset_title`\n- https://www.kaggle.com/jamesmcguigan/coleridge-string-literals?scriptVersionId=57628162\n\nThis may potentially be a bug with the Kaggle scoring system and/or test dataset", "1251068": "I have understood the intent of your question correctly. I also think that the LB score is strange as you pointed out. I guess we have to wait for the official Kaggle response.\n\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228216#1250983\n> There is no dataleak. There seems to be an error in the leaderboard. We are waiting for @philculliton to fix it.", "1251609": "I also asked a similar question here, and got the following responses:\n\n> In this competition, predictions are drawn from the documents. PredictionString is being compared against extracts from publications that indicate a dataset was mentioned in that publication. The strings that predictions are being compared against have been cleaned, as indicated in the Evaluation page.\n>\n> Yes! There are a huge number of labels in test that are not mentioned in train at all. So the task is less to match strings, and more to create an NLP solution for generalizing the \"can I identify datasets mentioned in this document\" problem.\n>\n> Correct. cleaned_label represents examples of (cleaned) extracted text we're attempting to predict for the training data.\n\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228463" }, "source": "meta" }