I think there must be shakeup at this competition,
\ncuz current public LB is 10% of total dataset, and the current metric F0.5 score more punishes False Positive than False Negative, it means if current model generates TP/TN/FP/FN in constant ratio, relatively smaller total number of predictions will be make higher score on current LB,
\nbut it's just 10%, and last result is from rest 90%.
\nso if there're more unknown words in private test data, it must be dangerous,
\nhow do you think others?
\n(and I also wanna ask how score diversity of model predictions are, like same bert scores 0.6 & 0.575 at LB with same structure but slightly different training method for me)
I think there must be shakeup at this competition,
\ncuz current public LB is 10% of total dataset, and the current metric F0.5 score more punishes False Positive than False Negative, it means if current model generates TP/TN/FP/FN in constant ratio, relatively smaller total number of predictions will be make higher score on current LB,
\nbut it's just 10%, and last result is from rest 90%.
\nso if there're more unknown words in private test data, it must be dangerous,
\nhow do you think others?
\n(and I also wanna ask how score diversity of model predictions are, like same bert scores 0.6 & 0.575 at LB with same structure but slightly different training method for me)
At present, there is no good way to identify the effect of the model through the validation set of the existing training set, because the number of labels in the training set is too small, and the distribution is extremely unbalanced like most problems. I think that with training the progress model is more inclined to find the name of the data set in the text by string matching.
\nBy the way, is it convenient to answer your 0.6 & 0.575 scores as the result of non-string matching?
", "rawMarkdown": "At present, there is no good way to identify the effect of the model through the validation set of the existing training set, because the number of labels in the training set is too small, and the distribution is extremely unbalanced like most problems. I think that with training the progress model is more inclined to find the name of the data set in the text by string matching. \n\nBy the way, is it convenient to answer your 0.6 & 0.575 scores as the result of non-string matching?", "votes": null }, { "id": "1338688", "postDate": "06/06/2021 16:32:12", "content": "Ok I'll answer
\nmy score .6&.575 are from bert output + string matching of np.unique(train.cleaned_label), I think almost everyone at top are using string matching.
\nI also tried to use govt data but just applying the govt data made score mess, so I at first I'm trying to make model better haha..
Thanks for your reply. I have applied QA model to predicting, got 0.505 ,maybe a little better than before w/o string matching, but just a small piece of improvement merging with string method, so I guess the model we got from training tends to exe string matching pattern because of having not a appropriate cv set to ensure the point which maybe result in much overfitting.
", "rawMarkdown": "Thanks for your reply. I have applied QA model to predicting, got 0.505 ,maybe a little better than before w/o string matching, but just a small piece of improvement merging with string method, so I guess the model we got from training tends to exe string matching pattern because of having not a appropriate cv set to ensure the point which maybe result in much overfitting.", "votes": null }, { "id": "1338710", "postDate": "06/06/2021 16:48:23", "content": "Yes, I can agree the trends of the model too.
\nso I'm trying not to make overfit to current labels.
https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/238005
\nI expect a major shakeup.
\nThere’s no real way to address this so my suggestion is to treat this as a “practice” competition and to not stress out too much about getting a medal/ranking.
\nThere’ll be a lucky draw (宝くじ) for 7 lucky teams ^^
", "rawMarkdown": "https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/238005\n\nI expect a major shakeup.\n\nThere’s no real way to address this so my suggestion is to treat this as a “practice” competition and to not stress out too much about getting a medal/ranking.\n\nThere’ll be a lucky draw (宝くじ) for 7 lucky teams ^^", "votes": null }, { "id": "1338945", "postDate": "06/06/2021 21:56:42", "content": "Yes, but I also think there must be the key that make us keep high score,
\nDo you still think you have possibility to shake to 50th〜100th place at final LB(with such .68 score)?
You could try making your model(s) more conservative to reduce FP but you won't get any good feedback from the LB since the public LB \"rewards\" more predictions.
\nIt would not surprise me if the top LB teams (including myself) do particularly bad - fall out into the \"no medal\" zone on a bad private LB split. No good CV + > 100 submissions might suggest overfitting to public LB.
", "rawMarkdown": "You could try making your model(s) more conservative to reduce FP but you won't get any good feedback from the LB since the public LB \"rewards\" more predictions.\n\nIt would not surprise me if the top LB teams (including myself) do particularly bad - fall out into the \"no medal\" zone on a bad private LB split. No good CV + > 100 submissions might suggest overfitting to public LB.", "votes": null }, { "id": "1339027", "postDate": "06/07/2021 01:00:12", "content": "I see, I also wonder about overfit because almost all of teams have really slow improvement on LB,
\nanyway I'm gonna try to create more (likely to be) better model,
\nhope god bress both of us get good results Lol
Is it like a single model result, like model trained on the entire train set or is it CVFold model? Just curious.
", "rawMarkdown": "Is it like a single model result, like model trained on the entire train set or is it CVFold model? Just curious.", "votes": null }, { "id": "1339041", "postDate": "06/07/2021 01:40:46", "content": "yes, it is the result of a single QA model w/o string matching.
", "rawMarkdown": "yes, it is the result of a single QA model w/o string matching.", "votes": null } ], "comments": [ { "id": 1338676, "author_name": "vanle73", "author_url": "", "post_date": "06/06/2021 16:20:09", "content": "At present, there is no good way to identify the effect of the model through the validation set of the existing training set, because the number of labels in the training set is too small, and the distribution is extremely unbalanced like most problems. I think that with training the progress model is more inclined to find the name of the data set in the text by string matching.
\nBy the way, is it convenient to answer your 0.6 & 0.575 scores as the result of non-string matching?
", "votes": null, "replies": [ { "id": 1338688, "author_name": "ryunosukeishizaki", "author_url": "", "post_date": "06/06/2021 16:32:12", "content": "Ok I'll answer
\nmy score .6&.575 are from bert output + string matching of np.unique(train.cleaned_label), I think almost everyone at top are using string matching.
\nI also tried to use govt data but just applying the govt data made score mess, so I at first I'm trying to make model better haha..
Thanks for your reply. I have applied QA model to predicting, got 0.505 ,maybe a little better than before w/o string matching, but just a small piece of improvement merging with string method, so I guess the model we got from training tends to exe string matching pattern because of having not a appropriate cv set to ensure the point which maybe result in much overfitting.
", "votes": null, "replies": [] }, { "id": 1338710, "author_name": "ryunosukeishizaki", "author_url": "", "post_date": "06/06/2021 16:48:23", "content": "Yes, I can agree the trends of the model too.
\nso I'm trying not to make overfit to current labels.
Is it like a single model result, like model trained on the entire train set or is it CVFold model? Just curious.
", "votes": null, "replies": [] }, { "id": 1339041, "author_name": "vanle73", "author_url": "", "post_date": "06/07/2021 01:40:46", "content": "yes, it is the result of a single QA model w/o string matching.
", "votes": null, "replies": [] } ] }, { "id": 1338937, "author_name": "leecming", "author_url": "", "post_date": "06/06/2021 21:47:58", "content": "https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/238005
\nI expect a major shakeup.
\nThere’s no real way to address this so my suggestion is to treat this as a “practice” competition and to not stress out too much about getting a medal/ranking.
\nThere’ll be a lucky draw (宝くじ) for 7 lucky teams ^^
", "votes": null, "replies": [ { "id": 1338945, "author_name": "ryunosukeishizaki", "author_url": "", "post_date": "06/06/2021 21:56:42", "content": "Yes, but I also think there must be the key that make us keep high score,
\nDo you still think you have possibility to shake to 50th〜100th place at final LB(with such .68 score)?
You could try making your model(s) more conservative to reduce FP but you won't get any good feedback from the LB since the public LB \"rewards\" more predictions.
\nIt would not surprise me if the top LB teams (including myself) do particularly bad - fall out into the \"no medal\" zone on a bad private LB split. No good CV + > 100 submissions might suggest overfitting to public LB.
", "votes": null, "replies": [] }, { "id": 1339027, "author_name": "ryunosukeishizaki", "author_url": "", "post_date": "06/07/2021 01:00:12", "content": "I see, I also wonder about overfit because almost all of teams have really slow improvement on LB,
\nanyway I'm gonna try to create more (likely to be) better model,
\nhope god bress both of us get good results Lol