Score 0.997 score in a day; either this will be
\ni) a very high and close scoring competition,
\nii) a very lucky submission or
\niii) there may be a data leakβ¦
EDIT [7pm India time, 24th march]:
\nThe organisers @julialane have confirmed \"There is no dataleak. There seems to be an error in the leaderboard.\" π [ LINK: https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228216]
\nKaggle will fix it soon! Thanks!
@josephassaker, @ammarali32, @jamesmcguigan and @sirishks have shared datasets and notebooks and also highlighted in discussion board; and made them public that brought the issue to light. Thanks all of you πππ SORRY IF I MISSED SOMEONE!
Score 0.997 score in a day; either this will be
\ni) a very high and close scoring competition,
\nii) a very lucky submission or
\niii) there may be a data leakβ¦
EDIT [7pm India time, 24th march]:
\nThe organisers @julialane have confirmed \"There is no dataleak. There seems to be an error in the leaderboard.\" π [ LINK: https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228216]
\nKaggle will fix it soon! Thanks!
@josephassaker, @ammarali32, @jamesmcguigan and @sirishks have shared datasets and notebooks and also highlighted in discussion board; and made them public that brought the issue to light. Thanks all of you πππ SORRY IF I MISSED SOMEONE!
It's because of very few labels issue, and possible use of external dataset \"names\".
\n[Edit] Confirmed by @ammarali32 (LB score 0.996) below.
\nSee dataset https://www.kaggle.com/ammarali32/coleridge-external-data
Then show us how to get 0.997
", "rawMarkdown": "Then show us how to get 0.997", "votes": null }, { "id": "1250768", "postDate": "03/24/2021 09:04:07", "content": "I'm wondering if it's because the f-metric is set to only weigh precision.
\nAnyway, it's going to be disappointing if this competition ends up being an exercise in string matching
", "rawMarkdown": "I'm wondering if it's because the f-metric is set to only weigh precision.\n\nAnyway, it's going to be disappointing if this competition ends up being an exercise in string matching", "votes": null }, { "id": "1250770", "postDate": "03/24/2021 09:10:00", "content": "Within 12 hours in fact π
", "rawMarkdown": "Within 12 hours in fact π", "votes": null }, { "id": "1250779", "postDate": "03/24/2021 09:13:44", "content": "I'm with you, FN don't matter here, which may make the problem easier.
", "rawMarkdown": "I'm with you, FN don't matter here, which may make the problem easier.", "votes": null }, { "id": "1250814", "postDate": "03/24/2021 09:42:05", "content": "Hi everyone when I get back home I will publish a notebook on how to get such a score, This competition needs new public data or metrics, or the leaderboard after one week will be all 1 and no one knows what about the private.
\nI already told you that using dataset_labels matching only will give you such a score on my topic.
thanks @ammarali32 ! look forward to your post
", "rawMarkdown": "thanks @ammarali32 ! look forward to your post", "votes": null }, { "id": "1250848", "postDate": "03/24/2021 10:09:30", "content": "https://www.kaggle.com/ammarali32/coleridge-dataset-labels-matching
\nWell it seems that dataset title will give you 1 XD https://www.kaggle.com/josephassaker/coleridge-initiative-eda-na-ve-submission
\nUPD: since the original notebook by @jamesmcguigan was updated I will delete mine you can see the results here.
\nhttps://www.kaggle.com/jamesmcguigan/coleridge-string-literals
thanks for highlighting! I am sure the organisers and the Kaggle team will take note and update the data/metric
", "rawMarkdown": "thanks for highlighting! I am sure the organisers and the Kaggle team will take note and update the data/metric", "votes": null }, { "id": "1250893", "postDate": "03/24/2021 10:43:39", "content": "It seems that the test dataset didn't have any new entities. Competition solvedβ¦
", "rawMarkdown": "It seems that the test dataset didn't have any new entities. Competition solved...", "votes": null }, { "id": "1250937", "postDate": "03/24/2021 11:21:53", "content": "imho the solution sits in the json files indeadβ¦
\nhttps://www.kaggle.com/plarmuseau/merge-multiple-json-fileswith-id
", "rawMarkdown": "imho the solution sits in the json files indead...\n\nhttps://www.kaggle.com/plarmuseau/merge-multiple-json-fileswith-id", "votes": null }, { "id": "1250940", "postDate": "03/24/2021 11:28:53", "content": "@cpmpml Check out my notebook to see how you can very easily achieve 1.0 score.
\nKeep in mind that the primary goal of my notebook is EDA, but I somehow stumbled across a 1.0 score from the first attempt π
", "rawMarkdown": "cpmpml Check out [my notebook](https://www.kaggle.com/josephassaker/coleridge-initiative-eda-na-ve-submission) to see how you can very easily achieve 1.0 score.\n\nKeep in mind that the primary goal of my notebook is EDA, but I somehow stumbled across a 1.0 score from the first attempt π ", "votes": null }, { "id": "1250942", "postDate": "03/24/2021 11:30:34", "content": "Now this is funny xD
", "rawMarkdown": "Now this is funny xD", "votes": null }, { "id": "1250952", "postDate": "03/24/2021 11:37:41", "content": "I think someone should open a meme's post. 90K $ for free. XD
", "rawMarkdown": "I think someone should open a meme's post. 90K $ for free. XD", "votes": null }, { "id": "1250956", "postDate": "03/24/2021 11:42:42", "content": "@ammarali32 I'm totally up for it! lol π
", "rawMarkdown": "ammarali32 I'm totally up for it! lol π ", "votes": null }, { "id": "1250981", "postDate": "03/24/2021 12:06:20", "content": "I posted a question in another thread asking to get clarification on the metric:
\n\nI have tested this both ways:
\n0.457 score for string literal searching (all fields), and outputting cleaned_label
\n\n0.998 score for string literal searching (dataset_title only), and outputting dataset_title
\n\nThis may potentially be a bug with the Kaggle scoring system and/or test dataset
", "rawMarkdown": "I posted a question in another thread asking to get clarification on the metric:\n- https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228366\n\nI have tested this both ways:\n\n0.457 score for string literal searching (all fields), and outputting cleaned_label\n- https://www.kaggle.com/jamesmcguigan/coleridge-string-literals?scriptVersionId=57586338\n\n0.998 score for string literal searching (dataset_title only), and outputting dataset_title\n- https://www.kaggle.com/jamesmcguigan/coleridge-string-literals?scriptVersionId=57628162\n\nThis may potentially be a bug with the Kaggle scoring system and/or test dataset", "votes": null }, { "id": "1251014", "postDate": "03/24/2021 12:37:23", "content": "I saw it. I confirms that FN aren't penalized. I bet we will have a metric update soon.
\nIt also confirms that using dataset title is better than using cleaned label. The latter is weird.
", "rawMarkdown": "I saw it. I confirms that FN aren't penalized. I bet we will have a metric update soon.\n\nIt also confirms that using dataset title is better than using cleaned label. The latter is weird.", "votes": null }, { "id": "1251611", "postDate": "03/25/2021 00:16:31", "content": "The use of the F0 score is why string literal matching on dataset title was generating 100% scores. This has now been fixed. We are now using the F0.5 score, which balances Precision and Recall, but is biased towards Precision.
\n\n\nYou mention FN in the evaluation page, but the metric F0 does not use it?
\nThe description will change, but to note that we are moving to a Beta = 0.5. FNs are important here, but precision moreso.
\n
https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228393#1251248
", "rawMarkdown": "The use of the F0 score is why string literal matching on dataset title was generating 100% scores. This has now been fixed. We are now using the F0.5 score, which balances Precision and Recall, but is biased towards Precision.\n\n> You mention FN in the evaluation page, but the metric F0 does not use it? \n>\n> The description will change, but to note that we are moving to a Beta = 0.5. FNs are important here, but precision moreso.\n\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228393#1251248", "votes": null } ], "comments": [ { "id": 1250754, "author_name": "sirishks", "author_url": "", "post_date": "03/24/2021 08:50:07", "content": "It's because of very few labels issue, and possible use of external dataset \"names\".
\n[Edit] Confirmed by @ammarali32 (LB score 0.996) below.
\nSee dataset https://www.kaggle.com/ammarali32/coleridge-external-data
Then show us how to get 0.997
", "votes": null, "replies": [] }, { "id": 1250768, "author_name": "leecming", "author_url": "", "post_date": "03/24/2021 09:04:07", "content": "I'm wondering if it's because the f-metric is set to only weigh precision.
\nAnyway, it's going to be disappointing if this competition ends up being an exercise in string matching
", "votes": null, "replies": [] }, { "id": 1250779, "author_name": "cpmpml", "author_url": "", "post_date": "03/24/2021 09:13:44", "content": "I'm with you, FN don't matter here, which may make the problem easier.
", "votes": null, "replies": [] }, { "id": 1250940, "author_name": "josephassaker", "author_url": "", "post_date": "03/24/2021 11:28:53", "content": "@cpmpml Check out my notebook to see how you can very easily achieve 1.0 score.
\nKeep in mind that the primary goal of my notebook is EDA, but I somehow stumbled across a 1.0 score from the first attempt π
", "votes": null, "replies": [] }, { "id": 1251014, "author_name": "cpmpml", "author_url": "", "post_date": "03/24/2021 12:37:23", "content": "I saw it. I confirms that FN aren't penalized. I bet we will have a metric update soon.
\nIt also confirms that using dataset title is better than using cleaned label. The latter is weird.
", "votes": null, "replies": [] }, { "id": 1251611, "author_name": "jamesmcguigan", "author_url": "", "post_date": "03/25/2021 00:16:31", "content": "The use of the F0 score is why string literal matching on dataset title was generating 100% scores. This has now been fixed. We are now using the F0.5 score, which balances Precision and Recall, but is biased towards Precision.
\n\n\nYou mention FN in the evaluation page, but the metric F0 does not use it?
\nThe description will change, but to note that we are moving to a Beta = 0.5. FNs are important here, but precision moreso.
\n
https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228393#1251248
", "votes": null, "replies": [] } ] }, { "id": 1250770, "author_name": "arka47", "author_url": "", "post_date": "03/24/2021 09:10:00", "content": "Within 12 hours in fact π
", "votes": null, "replies": [] }, { "id": 1250814, "author_name": "ammarali32", "author_url": "", "post_date": "03/24/2021 09:42:05", "content": "Hi everyone when I get back home I will publish a notebook on how to get such a score, This competition needs new public data or metrics, or the leaderboard after one week will be all 1 and no one knows what about the private.
\nI already told you that using dataset_labels matching only will give you such a score on my topic.
thanks @ammarali32 ! look forward to your post
", "votes": null, "replies": [] }, { "id": 1250848, "author_name": "ammarali32", "author_url": "", "post_date": "03/24/2021 10:09:30", "content": "https://www.kaggle.com/ammarali32/coleridge-dataset-labels-matching
\nWell it seems that dataset title will give you 1 XD https://www.kaggle.com/josephassaker/coleridge-initiative-eda-na-ve-submission
\nUPD: since the original notebook by @jamesmcguigan was updated I will delete mine you can see the results here.
\nhttps://www.kaggle.com/jamesmcguigan/coleridge-string-literals
thanks for highlighting! I am sure the organisers and the Kaggle team will take note and update the data/metric
", "votes": null, "replies": [] }, { "id": 1250942, "author_name": "arka47", "author_url": "", "post_date": "03/24/2021 11:30:34", "content": "Now this is funny xD
", "votes": null, "replies": [] }, { "id": 1250952, "author_name": "ammarali32", "author_url": "", "post_date": "03/24/2021 11:37:41", "content": "I think someone should open a meme's post. 90K $ for free. XD
", "votes": null, "replies": [] }, { "id": 1250956, "author_name": "josephassaker", "author_url": "", "post_date": "03/24/2021 11:42:42", "content": "@ammarali32 I'm totally up for it! lol π
", "votes": null, "replies": [] } ] }, { "id": 1250893, "author_name": "artgor", "author_url": "", "post_date": "03/24/2021 10:43:39", "content": "It seems that the test dataset didn't have any new entities. Competition solvedβ¦
", "votes": null, "replies": [] }, { "id": 1250937, "author_name": "plarmuseau", "author_url": "", "post_date": "03/24/2021 11:21:53", "content": "imho the solution sits in the json files indeadβ¦
\nhttps://www.kaggle.com/plarmuseau/merge-multiple-json-fileswith-id
", "votes": null, "replies": [] }, { "id": 1250981, "author_name": "jamesmcguigan", "author_url": "", "post_date": "03/24/2021 12:06:20", "content": "I posted a question in another thread asking to get clarification on the metric:
\n\nI have tested this both ways:
\n0.457 score for string literal searching (all fields), and outputting cleaned_label
\n\n0.998 score for string literal searching (dataset_title only), and outputting dataset_title
\n\nThis may potentially be a bug with the Kaggle scoring system and/or test dataset
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1250745": "Score 0.997 score in a day; either this will be\ni) a very high and close scoring competition, \nii) a very lucky submission or \niii) there may be a data leak...\n\n\nEDIT [7pm India time, 24th march]:\n* The organisers @julialane have confirmed \"There is no dataleak. There seems to be an error in the leaderboard.\" π [ LINK: https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228216]\nKaggle will fix it soon! Thanks!\n\n* @josephassaker, @ammarali32, @jamesmcguigan and @sirishks have shared datasets and notebooks and also highlighted in discussion board; and made them public that brought the issue to light. Thanks all of you πππ SORRY IF I MISSED SOMEONE!", "1250754": "It's because of [very few labels](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228322) issue, and possible use of [external dataset \"names\"](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228324).\n[Edit] Confirmed by @ammarali32 (LB score 0.996) below.\nSee dataset [https://www.kaggle.com/ammarali32/coleridge-external-data](https://www.kaggle.com/ammarali32/coleridge-external-data)", "1250757": "Then show us how to get 0.997", "1250768": "I'm wondering if it's because the f-metric is set to only weigh precision.\n\nAnyway, it's going to be disappointing if this competition ends up being an exercise in string matching", "1250770": "Within 12 hours in fact π", "1250779": "I'm with you, FN don't matter here, which may make the problem easier.", "1250814": "Hi everyone when I get back home I will publish a notebook on how to get such a score, This competition needs new public data or metrics, or the leaderboard after one week will be all 1 and no one knows what about the private. \nI already told you that using dataset_labels matching only will give you such a score on my topic.", "1250839": "thanks @ammarali32 ! look forward to your post", "1250848": "https://www.kaggle.com/ammarali32/coleridge-dataset-labels-matching\nWell it seems that dataset title will give you 1 XD https://www.kaggle.com/josephassaker/coleridge-initiative-eda-na-ve-submission\nUPD: since the original notebook by @jamesmcguigan was updated I will delete mine you can see the results here.\nhttps://www.kaggle.com/jamesmcguigan/coleridge-string-literals", "1250867": "thanks for highlighting! I am sure the organisers and the Kaggle team will take note and update the data/metric", "1250893": "It seems that the test dataset didn't have any new entities. Competition solved...", "1250937": "imho the solution sits in the json files indead...\n\nhttps://www.kaggle.com/plarmuseau/merge-multiple-json-fileswith-id", "1250940": "cpmpml Check out [my notebook](https://www.kaggle.com/josephassaker/coleridge-initiative-eda-na-ve-submission) to see how you can very easily achieve 1.0 score.\n\nKeep in mind that the primary goal of my notebook is EDA, but I somehow stumbled across a 1.0 score from the first attempt π ", "1250942": "Now this is funny xD", "1250952": "I think someone should open a meme's post. 90K $ for free. XD", "1250956": "ammarali32 I'm totally up for it! lol π ", "1250981": "I posted a question in another thread asking to get clarification on the metric:\n- https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228366\n\nI have tested this both ways:\n\n0.457 score for string literal searching (all fields), and outputting cleaned_label\n- https://www.kaggle.com/jamesmcguigan/coleridge-string-literals?scriptVersionId=57586338\n\n0.998 score for string literal searching (dataset_title only), and outputting dataset_title\n- https://www.kaggle.com/jamesmcguigan/coleridge-string-literals?scriptVersionId=57628162\n\nThis may potentially be a bug with the Kaggle scoring system and/or test dataset", "1251014": "I saw it. I confirms that FN aren't penalized. I bet we will have a metric update soon.\n\nIt also confirms that using dataset title is better than using cleaned label. The latter is weird.", "1251611": "The use of the F0 score is why string literal matching on dataset title was generating 100% scores. This has now been fixed. We are now using the F0.5 score, which balances Precision and Recall, but is biased towards Precision.\n\n> You mention FN in the evaluation page, but the metric F0 does not use it? \n>\n> The description will change, but to note that we are moving to a Beta = 0.5. FNs are important here, but precision moreso.\n\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228393#1251248" }, "source": "meta" }