Currently I'm running a simple post processing method to assert my model's real performance, that is to remove every prediction that has a Jaccard similarity greater than 0.5 to any of the known labels in this list:
\ntemp_1 = [x.lower() for x in train_df['dataset_label'].unique()]\ntemp_2 = [x.lower() for x in train_df['dataset_title'].unique()]\ntemp_3 = [x.lower() for x in train_df['cleaned_label'].unique()]\nexisting_labels = set(temp_1 + temp_2 + temp_3)\nWith that, my best score is 0.414 using a single model, I'm curious how well others are doing since it's impossible to know looking at the leaderboard.
", "messages": [ { "id": 1275455, "postDate": "2021-04-16T10:08:40.417Z", "content": "Currently I'm running a simple post processing method to assert my model's real performance, that is to remove every prediction that has a Jaccard similarity greater than 0.5 to any of the known labels in this list:
\ntemp_1 = [x.lower() for x in train_df['dataset_label'].unique()]\ntemp_2 = [x.lower() for x in train_df['dataset_title'].unique()]\ntemp_3 = [x.lower() for x in train_df['cleaned_label'].unique()]\nexisting_labels = set(temp_1 + temp_2 + temp_3)\nWith that, my best score is 0.414 using a single model, I'm curious how well others are doing since it's impossible to know looking at the leaderboard.
", "rawMarkdown": "Currently I'm running a simple post processing method to assert my model's real performance, that is to remove every prediction that has a Jaccard similarity greater than 0.5 to any of the known labels in this list:\n```\ntemp_1 = [x.lower() for x in train_df['dataset_label'].unique()]\ntemp_2 = [x.lower() for x in train_df['dataset_title'].unique()]\ntemp_3 = [x.lower() for x in train_df['cleaned_label'].unique()]\nexisting_labels = set(temp_1 + temp_2 + temp_3)\n\n```\n\nWith that, my best score is **0.414** using a single model, I'm curious how well others are doing since it's impossible to know looking at the leaderboard.", "votes": 70 }, { "id": 1293912, "postDate": "2021-05-05T08:57:34.917Z", "content": "Please can everyone come back and update their scores on this when the rerun of the submissions are done on the LB?
\n(I've been really trusting this thread as a measure of performance so it'd be nice to see how it changes)
More update: 0.414 single model, combining the model prediction with known labels gave us the current LB score.
", "rawMarkdown": "More update: 0.414 single model, combining the model prediction with known labels gave us the current LB score.", "votes": 3, "replies": [ { "id": 1285199, "postDate": "2021-04-26T17:16:33.907Z", "content": "Hi @suicaokhoailang was just starting out , can you please tell me which model/approach are you using , is it NER or some other approach?
\nIf it is NER are you just using the sentences which contain the dataset mention or whole of the text?
", "rawMarkdown": "Hi @suicaokhoailang was just starting out , can you please tell me which model/approach are you using , is it NER or some other approach?\n\nIf it is NER are you just using the sentences which contain the dataset mention or whole of the text?", "votes": 1 }, { "id": 1285515, "postDate": "2021-04-27T03:28:04.977Z", "content": "We're using a novel method and are not ready to disclose just yet, if it works out in the end we're happy to share.
", "rawMarkdown": "We're using a novel method and are not ready to disclose just yet, if it works out in the end we're happy to share.", "votes": 7 } ] }, { "id": 1277820, "postDate": "2021-04-19T09:00:55.170Z", "content": "Update: Our best score is 0.35 with single model.
", "rawMarkdown": "Update: Our best score is 0.35 with single model.", "votes": 3, "replies": [ { "id": 1277833, "postDate": "2021-04-19T09:23:21.603Z", "content": "single model single fold :D
", "rawMarkdown": "single model single fold :D ", "votes": 1 }, { "id": 1277836, "postDate": "2021-04-19T09:28:59.327Z", "content": "Wow :-) If we give updates here, I think this can really be an alternative LB (IMO it is already more useful than the Public LB)
", "rawMarkdown": "Wow :-) If we give updates here, I think this can really be an alternative LB (IMO it is already more useful than the Public LB)", "votes": 2 } ] }, { "id": 1276597, "postDate": "2021-04-17T18:08:30.207Z", "content": "Periodically updated: 0.425 is my public LB score with predictions removed with a Jaccard similarity >= 0.5 with any existing_labels.
Good one, are you using any external data?
", "rawMarkdown": "Good one, are you using any external data?" }, { "id": 1276873, "postDate": "2021-04-18T05:31:39.577Z", "content": "@trentb did you use any hand-label for training? if yes, it will help your score higher because it is not in the original training label but in the public test set.
", "rawMarkdown": "@trentb did you use any hand-label for training? if yes, it will help your score higher because it is not in the original training label but in the public test set." }, { "id": 1277297, "postDate": "2021-04-18T15:42:57.590Z", "content": "I am not doing any training yet. I am using a popular pretrained model and cleaning/filtering the results with basic string operations. These string operations are not informed by the training set labels.
", "rawMarkdown": "I am not doing any training yet. I am using a popular pretrained model and cleaning/filtering the results with basic string operations. These string operations are not informed by the training set labels.", "votes": 2 }, { "id": 1279913, "postDate": "2021-04-21T11:05:41.063Z", "content": "I can hazard a guess on this approach :) My main concern with using fixed string operations and other rule-driven text normalization steps is that it's pretty brittle - if the private datasets don't conform to similar conventions, it can blow up.
", "rawMarkdown": "I can hazard a guess on this approach :) My main concern with using fixed string operations and other rule-driven text normalization steps is that it's pretty brittle - if the private datasets don't conform to similar conventions, it can blow up.", "votes": 1 }, { "id": 1280503, "postDate": "2021-04-22T03:42:59.867Z", "content": "I made a few changes to my model and updated my score above to 0.293.
", "rawMarkdown": "I made a few changes to my model and updated my score above to 0.293.", "votes": 1 }, { "id": 1294548, "postDate": "2021-05-05T18:12:20.420Z", "content": "I got a nice increase from the rescoring. I have updated my score above to 0.374.
", "rawMarkdown": "I got a nice increase from the rescoring. I have updated my score above to 0.374.", "votes": 1 }, { "id": 1295113, "postDate": "2021-05-06T08:01:05.707Z", "content": "@trentb nice :D, almost our score drops a bit.
", "rawMarkdown": "@trentb nice :D, almost our score drops a bit. " }, { "id": 1298551, "postDate": "2021-05-09T02:34:03.070Z", "content": "I updated my initial post to reflect my new score of 0.410.
", "rawMarkdown": "I updated my initial post to reflect my new score of 0.410." }, { "id": 1333788, "postDate": "2021-06-03T04:20:57.597Z", "content": "0.425 is my new high score for a new approach I'm trying. It only scores 0.590 if I don't remove existing labels.
Congratulations @trentb! Do you use any model in your approach? Seems you only use str matching with custom labels?
", "rawMarkdown": "Congratulations @trentb! Do you use any model in your approach? Seems you only use str matching with custom labels?" }, { "id": 1334369, "postDate": "2021-06-03T12:59:02.980Z", "content": "My current LB score of 0.648 is based purely on string matching of presumed dataset labels. In my new approach, I'm writing some ugly rule-based code without providing specific strings of existing/presumed dataset labels. Of course I often see how the code performs on the training data, so I'm not totally blind to the training data. There is still no training involved although I'm considering training a model with the labels my code produces just for the sake of saying I have an ML model.
", "rawMarkdown": "My current LB score of 0.648 is based purely on string matching of presumed dataset labels. In my new approach, I'm writing some ugly rule-based code without providing specific strings of existing/presumed dataset labels. Of course I often see how the code performs on the training data, so I'm not totally blind to the training data. There is still no training involved although I'm considering training a model with the labels my code produces just for the sake of saying I have an ML model.", "votes": 8 }, { "id": 1339590, "postDate": "2021-06-07T10:46:03.813Z", "content": "Any update?
", "rawMarkdown": "Any update?" } ] }, { "id": 1297289, "postDate": "2021-05-07T22:04:09.623Z", "content": "I used a very standard BERT model to solve the NER problem. The single model result is 0.454. Combined with string matching can boost up the performance. However, we know string matching 1) may not work well on private LB and 2) is not the purpose for this competition.
\nI think many of our non-string matching models are facing performance deterioriation issues: validation performance should be pretty good on set aside training data but public LB scores are much lower. I think performance deterioriation is caused by the difference between training data and public LB testing data. If tagging method can be more consistent among training and testing data, non-string matching models will have better performance…
", "rawMarkdown": "I used a very standard BERT model to solve the NER problem. The single model result is 0.454. Combined with string matching can boost up the performance. However, we know string matching 1) may not work well on private LB and 2) is not the purpose for this competition.\n\nI think many of our non-string matching models are facing performance deterioriation issues: validation performance should be pretty good on set aside training data but public LB scores are much lower. I think performance deterioriation is caused by the difference between training data and public LB testing data. If tagging method can be more consistent among training and testing data, non-string matching models will have better performance...", "votes": 4 }, { "id": 1361642, "postDate": "2021-06-23T01:38:00.690Z", "content": "@suicaokhoailang Congratulations, and thank you for this topic! I would have crashed and burned without it.
", "rawMarkdown": "@suicaokhoailang Congratulations, and thank you for this topic! I would have crashed and burned without it.", "votes": 1, "replies": [ { "id": 1361644, "postDate": "2021-06-23T01:40:43.107Z", "content": "You're welcome, glad to see that many people who shared their scores here got great results in the end.
", "rawMarkdown": "You're welcome, glad to see that many people who shared their scores here got great results in the end.", "votes": 1 } ] }, { "id": 1351593, "postDate": "2021-06-16T12:50:55.890Z", "content": "Currently sitting at 0.407 For a Bi-LSTM Model with GloVe embeddings and 0.424 for a BERT Model. Have not tried incorporating string matching to both of these yet. Since the current leaderboards seem so skewed to string matching, I'm really on the fence about picking my 2 NON string matching models or one which has it incorporated
", "rawMarkdown": "Currently sitting at **0.407** For a Bi-LSTM Model with GloVe embeddings and **0.424** for a BERT Model. Have not tried incorporating string matching to both of these yet. Since the current leaderboards seem so skewed to string matching, I'm really on the fence about picking my 2 **NON** string matching models or one which has it incorporated", "votes": 1 }, { "id": 1328898, "postDate": "2021-05-30T16:11:54.970Z", "content": "I reached 0.393 using https://www.kaggle.com/tungmphung/pytorch-bert-for-named-entity-recognition work and could reach 0.412 using https://www.kaggle.com/theoviel/bert-for-question-answering-baseline-training
", "rawMarkdown": "I reached 0.393 using https://www.kaggle.com/tungmphung/pytorch-bert-for-named-entity-recognition work and could reach 0.412 using https://www.kaggle.com/theoviel/bert-for-question-answering-baseline-training", "votes": 1 }, { "id": 1313063, "postDate": "2021-05-18T11:45:35.047Z", "content": "So what is the best public leader board scores with a single model no string matching without removing known label predictions that are jaccard similar?
", "rawMarkdown": "So what is the best public leader board scores with a single model no string matching without removing known label predictions that are jaccard similar?", "votes": 1, "replies": [ { "id": 1313228, "postDate": "2021-05-18T13:07:51.260Z", "content": "Maybe 0.4+, not many people reported their results here.
", "rawMarkdown": "Maybe 0.4+, not many people reported their results here.", "votes": 1 }, { "id": 1313249, "postDate": "2021-05-18T13:19:28.573Z", "content": "Ok, I must be missing something - if you are getting .41 removing all predictions that are Jaccard similar >.5 to the known labels, why aren't you scoring MUCH higher when not removing Jaccard similar predictions?
", "rawMarkdown": "Ok, I must be missing something - if you are getting .41 removing all predictions that are Jaccard similar >.5 to the known labels, why aren't you scoring MUCH higher when not removing Jaccard similar predictions?", "votes": 1 }, { "id": 1313272, "postDate": "2021-05-18T13:32:31.030Z", "content": "If you laid out the metrics you will understand why:
\nF0.5 = TP/(TP + 0.2FN + 0.8FP)
\nA rough explanation is that our models predicted a large amount of false positives which are punished much harder than false negatives. Removing training labels means we moved some of our true positive predictions and turned them into false negatives, which doesn't affect the score that much due to the low coefficient.
", "rawMarkdown": "If you laid out the metrics you will understand why:\n\nF0.5 = TP/(TP + 0.2*FN + 0.8*FP)\n\nA rough explanation is that our models predicted a large amount of false positives which are punished much harder than false negatives. Removing training labels means we moved some of our true positive predictions and turned them into false negatives, which doesn't affect the score that much due to the low coefficient.", "votes": 5 }, { "id": 1313279, "postDate": "2021-05-18T13:39:55.997Z", "content": "They said there wouldn't be any math…. : ) - thanks - .8 is harsh and my best model has its share of FPs for sure - my best single model scores .49 (public LB) without matching and without removing any known label predictions. So it is pretty close to baseline matching, but I think it is over fitting to the training data I am using and I am concerned it won't generalize well on the test set / private LB.
\nThanks again for sharing and clarifying this @suicaokhoailang
", "rawMarkdown": "They said there wouldn't be any math.... : ) - thanks - .8 is harsh and my best model has its share of FPs for sure - my best single model scores .49 (public LB) without matching and without removing any known label predictions. So it is pretty close to baseline matching, but I think it is over fitting to the training data I am using and I am concerned it won't generalize well on the test set / private LB.\n\nThanks again for sharing and clarifying this @suicaokhoailang ", "votes": 1 }, { "id": 1318608, "postDate": "2021-05-22T12:35:41.493Z", "content": "Why did you remove the train labels, because it is completely impossible for the train labels to appear in the test?
", "rawMarkdown": "Why did you remove the train labels, because it is completely impossible for the train labels to appear in the test?" } ] }, { "id": 1302495, "postDate": "2021-05-11T15:18:09.050Z", "content": "Updata: test score-0.393 from single model based on bert-base pretrained model using multi answer method.
", "rawMarkdown": "Updata: test score-**0.393** from single model based on bert-base pretrained model using multi answer method.", "votes": 1, "replies": [ { "id": 1309687, "postDate": "2021-05-16T07:58:01.353Z", "content": "five single-fold models based the same QA architecture ensembled:0.409
", "rawMarkdown": "five single-fold models based the same QA architecture ensembled:0.409", "votes": 1 }, { "id": 1321859, "postDate": "2021-05-25T03:43:45.370Z", "content": "@vanle73 how can you make the model predict multiple answer span ? A typical approach of a QA model is to classify which token in your input sequence is the start token (the same to end tokens). So this model is only able to predict 1 answer given an input.
", "rawMarkdown": "@vanle73 how can you make the model predict multiple answer span ? A typical approach of a QA model is to classify which token in your input sequence is the start token (the same to end tokens). So this model is only able to predict 1 answer given an input. " }, { "id": 1328381, "postDate": "2021-05-30T07:59:09.563Z", "content": "We can create our questions for training, and get the final result by some postprocessing. : )
", "rawMarkdown": "We can create our questions for training, and get the final result by some postprocessing. : )" } ] }, { "id": 1294479, "postDate": "2021-05-05T17:18:32.370Z", "content": "More update: 0.409 single model. Overall all submissions got a reduced score.
", "rawMarkdown": "More update: 0.409 single model. Overall all submissions got a reduced score.", "votes": 1, "replies": [ { "id": 1295488, "postDate": "2021-05-06T13:51:12.770Z", "content": "About mine. 80% of the data is used as the training set, and 20% of the data is used as the validation set. The score of the validation set is 0.848, but only 0.32 online!
\nI am very curious whether your final score is the fusion of multiple trained models, rather than the post-processing of some string matching, looking forward to your reply!😉
", "rawMarkdown": "About mine. 80% of the data is used as the training set, and 20% of the data is used as the validation set. The score of the validation set is **0.848**, but only **0.32** online! \n\n I am very curious whether your final score is the fusion of multiple trained models, rather than the post-processing of some string matching, looking forward to your reply!😉" } ] }, { "id": 1275742, "postDate": "2021-04-16T16:30:17.757Z", "content": "using 2.5% size of training data, lb : 0.234
\nusing 1.5% size of training data, lb : 0.203
\nNot string matching, just a simple model.
did you remove all predicted labels which has a jaccard >= 0.5 with any label in the training set?
", "rawMarkdown": "did you remove all predicted labels which has a jaccard >= 0.5 with any label in the training set? " } ] }, { "id": 1341392, "postDate": "2021-06-08T16:11:13.740Z", "content": "I got 0.455 and 0.429 LB without string match & without remove similar known labels for two single folds
", "rawMarkdown": "I got 0.455 and 0.429 LB without string match & without remove similar known labels for two single folds", "votes": 2, "replies": [ { "id": 1351578, "postDate": "2021-06-16T12:19:43.743Z", "content": "Update, got 0.509 LB without string match & without removing similar known labels
", "rawMarkdown": "Update, got 0.509 LB without string match & without removing similar known labels" }, { "id": 1355520, "postDate": "2021-06-18T10:33:15.153Z", "content": "greate job, i am so tired with string matching in this competition. ^^. Hopefully can learn how do you trained model after this competition.
", "rawMarkdown": "greate job, i am so tired with string matching in this competition. ^^. Hopefully can learn how do you trained model after this competition. " } ] }, { "id": 1280860, "postDate": "2021-04-22T12:38:10.700Z", "content": "Our best score is 0.023 with single model! 😨
\nWe use spaCy's English model.
Without removing predictions that has a Jaccard similarity greater than 0.5 to any of existing_labels, this model's score is 0.595…
\nFrom my perspective, it means our model is overfitting to existing labels.
I'll do my best with spaCy's English model a little more.
", "rawMarkdown": "Our best score is 0.023 with single model! 😨\nWe use spaCy's English model.\n\nWithout removing predictions that has a Jaccard similarity greater than 0.5 to any of `existing_labels`, this model's score is 0.595...\nFrom my perspective, it means our model is overfitting to existing labels.\n\nI'll do my best with spaCy's English model a little more.", "votes": 2, "replies": [ { "id": 1281198, "postDate": "2021-04-22T17:26:41.863Z", "content": "I got a similar score with my local CV. Then I included weights in the cross entropy loss (to balance class imbalance due to the scarse occurence of datasets in the texts), which dramatically improved my score. Maybe this also applies to you. Good luck!
", "rawMarkdown": "I got a similar score with my local CV. Then I included weights in the cross entropy loss (to balance class imbalance due to the scarse occurence of datasets in the texts), which dramatically improved my score. Maybe this also applies to you. Good luck!", "votes": 3 }, { "id": 1281457, "postDate": "2021-04-23T01:06:08.150Z", "content": "Thank you so much!
\nI'll try it. 🙌
【May 5 update】
\nThis submission's score has changed to 0.019.
Great posting! And nice to see that I may not be that far off as the LB suggests :-) My score: 0.277
", "rawMarkdown": "Great posting! And nice to see that I may not be that far off as the LB suggests :-) My score: **0.277**", "votes": 2, "replies": [ { "id": 1275753, "postDate": "2021-04-16T16:53:00.527Z", "content": "did you remove all predicted labels which has a jaccard >= 0.5 with any label in the training set?
", "rawMarkdown": "did you remove all predicted labels which has a jaccard >= 0.5 with any label in the training set? " }, { "id": 1275800, "postDate": "2021-04-16T18:02:55.463Z", "content": "Yes. I thought that's the point of this thread.
", "rawMarkdown": "Yes. I thought that's the point of this thread." }, { "id": 1290823, "postDate": "2021-05-02T13:04:03.563Z", "content": "This has been quite a struggle with bugs and submission errors.. but now I can also give an update: 0.277
", "rawMarkdown": "This has been quite a struggle with bugs and submission errors.. but now I can also give an update: **0.277**" } ] }, { "id": 1337098, "postDate": "2021-06-05T12:15:33.973Z", "content": "This thread is relatively so much useful than public LB. Sould be pinned at top lol
\nI got 0.451 non-matching LB, haven't tried removing known labels.
If I understand correctly, @suicaokhoailang you got 0.664 using model w/o matching w/o removing known labels? and 0.414 w/o matching but after the removing known labels right?
I am really stuck and my score isn't moving, so I shared this notebook in hopes it helps someone move the needle. It is a Spacy classifier and uses no matching scores .44 on the LB. I have a similar BERT model which scores about the same. I extract using capitalized word sequences and acronyms like (BLSA) extracting the words before the acronym to avoid having to try and predict on all sentences.
\nhttps://www.kaggle.com/mlconsult/44-3-spacy-classifier-no-matching
", "rawMarkdown": "I am really stuck and my score isn't moving, so I shared this notebook in hopes it helps someone move the needle. It is a Spacy classifier and uses no matching scores .44 on the LB. I have a similar BERT model which scores about the same. I extract using capitalized word sequences and acronyms like (BLSA) extracting the words before the acronym to avoid having to try and predict on all sentences.\n\nhttps://www.kaggle.com/mlconsult/44-3-spacy-classifier-no-matching" }, { "id": 1297840, "postDate": "2021-05-08T11:24:00.853Z", "content": "I think the reason that prediction models are not working very well, among several other things, is because the challenge calls for:
\n\"Your predictions will be short excerpts from the publications that appear to note a dataset.\"
\nSo when you use matching to determine if a string \"is in\" a text, if it is, then you return the actual words from the text as your prediction - because those words are actually in the text. If you are returning a label from the dataset, that is not going to receive as good of score, because it is not actually what is in the text.
", "rawMarkdown": "I think the reason that prediction models are not working very well, among several other things, is because the challenge calls for:\n\n\"Your predictions will be short excerpts from the publications that appear to note a dataset.\"\n\nSo when you use matching to determine if a string \"is in\" a text, if it is, then you return the actual words from the text as your prediction - because those words are actually in the text. If you are returning a label from the dataset, that is not going to receive as good of score, because it is not actually what is in the text." }, { "id": 1295660, "postDate": "2021-05-06T15:55:50.770Z", "content": "0.346 on LB with the rerun of the submissions. I am using a very basic NER LSTM with pos label and word embedding
", "rawMarkdown": "**0.346** on LB with the rerun of the submissions. I am using a very basic NER LSTM with pos label and word embedding" }, { "id": 1294540, "postDate": "2021-05-05T18:07:59.217Z", "content": "My update: 0.300 I was lucky and my score increased a bit :-)
", "rawMarkdown": "My update: **0.300** I was lucky and my score increased a bit :-)" }, { "id": 1286620, "postDate": "2021-04-28T08:02:44.887Z", "content": "Great topic!
\nMay I ask why are you removing \"every prediction that has a Jaccard similarity greater than 0.5 to any of the known labels in this list\"? This could be a case when your model is not doing string matching, however Jaccard similarity of its prediction is greater than 0.5 to know dataset titles.
Hey this is an interesting approach. I assume this number comes from the submission, isn't it?
", "rawMarkdown": "Hey this is an interesting approach. I assume this number comes from the submission, isn't it?" }, { "id": 1279993, "postDate": "2021-04-21T13:01:21.870Z", "content": "My BERT model is obviously quite off the mark - only 0.185 without string matching
", "rawMarkdown": "My BERT model is obviously quite off the mark - only 0.185 without string matching ", "replies": [ { "id": 1280771, "postDate": "2021-04-22T10:27:22.427Z", "content": "got it up to 0.259 - things are improving but still a long way to go…
", "rawMarkdown": "got it up to 0.259 - things are improving but still a long way to go...", "votes": 2 }, { "id": 1295569, "postDate": "2021-05-06T14:57:05.593Z", "content": "[5th May update]. after the rescore (and after re-trying old changes that had been ineffective on the old data) I've managed to get it up to 0.307
", "rawMarkdown": "[5th May update]. after the rescore (and after re-trying old changes that had been ineffective on the old data) I've managed to get it up to 0.307" }, { "id": 1295582, "postDate": "2021-05-06T15:04:37.830Z", "content": "hello I’m curious if your model is bert-ner？ My bert-QA model has a final result of 0.336 after the recent re-evaluation.😁
", "rawMarkdown": "hello I’m curious if your model is bert-ner？ My bert-QA model has a final result of 0.336 after the recent re-evaluation.😁" }, { "id": 1295598, "postDate": "2021-05-06T15:20:18.153Z", "content": "Hi @vanle73 no its a BERT-MLM. I tried a Roberta-Squad straight out of the box and got about 0.12 so stopped that approach- looks like you had more success :) Did you fine tune it on this dataset? If so, just the given labels or a wider range?
", "rawMarkdown": "Hi @vanle73 no its a BERT-MLM. I tried a Roberta-Squad straight out of the box and got about 0.12 so stopped that approach- looks like you had more success :) Did you fine tune it on this dataset? If so, just the given labels or a wider range?", "votes": 1 }, { "id": 1295652, "postDate": "2021-05-06T15:50:16.403Z", "content": "before , I have tried the single-ans QA model to build the solution, however, I failed, it scored less than 0.2, so I have changed the model method by predicting the ans's number, it has a not bad result. my 0.336 score came from the multi-answer QA model based on bert-based pretrained model.
\nIf possible, we can communicate together via email to reach a good final game result 😄
That's a really high score, I think that may be around the upper bound of what one may achieve after removing train labels.
\nWhat do you mean by \" x 130 labels\"? Does that mean you trained 130 models one for each train label? I wonder how LGBM can generalize so well to unseen text.
did you ignore all 130 training labels when inference ?, 60.9 is almost maximized score, and ur model combine with string match should get 0.9 right now :)
", "rawMarkdown": "did you ignore all 130 training labels when inference ?, 60.9 is almost maximized score, and ur model combine with string match should get 0.9 right now :) ", "votes": 1 }, { "id": 1279086, "postDate": "2021-04-20T15:14:54.370Z", "content": "@suicaokhoailang @dathudeptrai Sorry. I completely misunderstood this topic. My score above is just a prediction for the existing labels. My score without existing labels is 0.054 (not LGBM).
", "rawMarkdown": "@suicaokhoailang @dathudeptrai Sorry. I completely misunderstood this topic. My score above is just a prediction for the existing labels. My score without existing labels is 0.054 (not LGBM).", "votes": 1 } ] }, { "id": 1362730, "postDate": "2021-06-23T15:34:33.720Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 1304083, "postDate": "2021-05-12T12:13:23.590Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1301040, "postDate": "2021-05-10T23:19:53.720Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1312649, "postDate": "2021-05-18T07:02:32.260Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1312650, "postDate": "2021-05-18T07:04:30.633Z", "content": "You got 47.5 after removing predictions similar to training ones? Wow I think you may have gotten the rest of them.
", "rawMarkdown": "You got 47.5 after removing predictions similar to training ones? Wow I think you may have gotten the rest of them.", "votes": 2 }, { "id": 1313061, "postDate": "2021-05-18T11:44:00.870Z", "content": "No I misread - I did not use any matching, but did not remove any jaccard similar predictions same as the original labels.
", "rawMarkdown": "No I misread - I did not use any matching, but did not remove any jaccard similar predictions same as the original labels.", "votes": 1 } ] }, { "id": 1300351, "postDate": "2021-05-10T12:06:22.517Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1300427, "postDate": "2021-05-10T12:56:14.817Z", "content": "The point of the thread is to remove predictions that were similar to those in training data though, not the opposite.
", "rawMarkdown": "The point of the thread is to remove predictions that **were** similar to those in training data though, not the opposite." } ] }, { "id": 1295316, "postDate": "2021-05-06T11:20:02.197Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1291218, "postDate": "2021-05-02T19:44:39.643Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1280977, "postDate": "2021-04-22T14:39:37.257Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1293912, "author_name": "Mark P", "author_url": "", "post_date": "2021-05-05T08:57:34.917000", "content": "Please can everyone come back and update their scores on this when the rerun of the submissions are done on the LB?
\n(I've been really trusting this thread as a measure of performance so it'd be nice to see how it changes)
More update: 0.414 single model, combining the model prediction with known labels gave us the current LB score.
", "votes": 3, "replies": [ { "id": 1285199, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2021-04-26T17:16:33.907000", "content": "Hi @suicaokhoailang was just starting out , can you please tell me which model/approach are you using , is it NER or some other approach?
\nIf it is NER are you just using the sentences which contain the dataset mention or whole of the text?
", "votes": 1, "replies": [] }, { "id": 1285515, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2021-04-27T03:28:04.977000", "content": "We're using a novel method and are not ready to disclose just yet, if it works out in the end we're happy to share.
", "votes": 7, "replies": [] } ] }, { "id": 1277820, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2021-04-19T09:00:55.170000", "content": "Update: Our best score is 0.35 with single model.
", "votes": 3, "replies": [ { "id": 1277833, "author_name": "Nguyen Quan Anh Minh", "author_url": "", "post_date": "2021-04-19T09:23:21.603000", "content": "single model single fold :D
", "votes": 1, "replies": [] }, { "id": 1277836, "author_name": "Hannes Öhler", "author_url": "", "post_date": "2021-04-19T09:28:59.327000", "content": "Wow :-) If we give updates here, I think this can really be an alternative LB (IMO it is already more useful than the Public LB)
", "votes": 2, "replies": [] } ] }, { "id": 1276597, "author_name": "Trent", "author_url": "", "post_date": "2021-04-17T18:08:30.207000", "content": "Periodically updated: 0.425 is my public LB score with predictions removed with a Jaccard similarity >= 0.5 with any existing_labels.
Good one, are you using any external data?
", "votes": 0, "replies": [] }, { "id": 1276873, "author_name": "Nguyen Quan Anh Minh", "author_url": "", "post_date": "2021-04-18T05:31:39.577000", "content": "@trentb did you use any hand-label for training? if yes, it will help your score higher because it is not in the original training label but in the public test set.
", "votes": 0, "replies": [] }, { "id": 1277297, "author_name": "Trent", "author_url": "", "post_date": "2021-04-18T15:42:57.590000", "content": "I am not doing any training yet. I am using a popular pretrained model and cleaning/filtering the results with basic string operations. These string operations are not informed by the training set labels.
", "votes": 2, "replies": [] }, { "id": 1279913, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2021-04-21T11:05:41.063000", "content": "I can hazard a guess on this approach :) My main concern with using fixed string operations and other rule-driven text normalization steps is that it's pretty brittle - if the private datasets don't conform to similar conventions, it can blow up.
", "votes": 1, "replies": [] }, { "id": 1280503, "author_name": "Trent", "author_url": "", "post_date": "2021-04-22T03:42:59.867000", "content": "I made a few changes to my model and updated my score above to 0.293.
", "votes": 1, "replies": [] }, { "id": 1294548, "author_name": "Trent", "author_url": "", "post_date": "2021-05-05T18:12:20.420000", "content": "I got a nice increase from the rescoring. I have updated my score above to 0.374.
", "votes": 1, "replies": [] }, { "id": 1295113, "author_name": "Nguyen Quan Anh Minh", "author_url": "", "post_date": "2021-05-06T08:01:05.707000", "content": "@trentb nice :D, almost our score drops a bit.
", "votes": 0, "replies": [] }, { "id": 1298551, "author_name": "Trent", "author_url": "", "post_date": "2021-05-09T02:34:03.070000", "content": "I updated my initial post to reflect my new score of 0.410.
", "votes": 0, "replies": [] }, { "id": 1333788, "author_name": "Trent", "author_url": "", "post_date": "2021-06-03T04:20:57.597000", "content": "0.425 is my new high score for a new approach I'm trying. It only scores 0.590 if I don't remove existing labels.
Congratulations @trentb! Do you use any model in your approach? Seems you only use str matching with custom labels?
", "votes": 0, "replies": [] }, { "id": 1334369, "author_name": "Trent", "author_url": "", "post_date": "2021-06-03T12:59:02.980000", "content": "My current LB score of 0.648 is based purely on string matching of presumed dataset labels. In my new approach, I'm writing some ugly rule-based code without providing specific strings of existing/presumed dataset labels. Of course I often see how the code performs on the training data, so I'm not totally blind to the training data. There is still no training involved although I'm considering training a model with the labels my code produces just for the sake of saying I have an ML model.
", "votes": 8, "replies": [] }, { "id": 1339590, "author_name": "Chiau Sing Chi", "author_url": "", "post_date": "2021-06-07T10:46:03.813000", "content": "Any update?
", "votes": 0, "replies": [] } ] }, { "id": 1297289, "author_name": "Xiaonan Ji", "author_url": "", "post_date": "2021-05-07T22:04:09.623000", "content": "I used a very standard BERT model to solve the NER problem. The single model result is 0.454. Combined with string matching can boost up the performance. However, we know string matching 1) may not work well on private LB and 2) is not the purpose for this competition.
\nI think many of our non-string matching models are facing performance deterioriation issues: validation performance should be pretty good on set aside training data but public LB scores are much lower. I think performance deterioriation is caused by the difference between training data and public LB testing data. If tagging method can be more consistent among training and testing data, non-string matching models will have better performance…
", "votes": 4, "replies": [] }, { "id": 1361642, "author_name": "Trent", "author_url": "", "post_date": "2021-06-23T01:38:00.690000", "content": "@suicaokhoailang Congratulations, and thank you for this topic! I would have crashed and burned without it.
", "votes": 1, "replies": [ { "id": 1361644, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2021-06-23T01:40:43.107000", "content": "You're welcome, glad to see that many people who shared their scores here got great results in the end.
", "votes": 1, "replies": [] } ] }, { "id": 1351593, "author_name": "maze508", "author_url": "", "post_date": "2021-06-16T12:50:55.890000", "content": "Currently sitting at 0.407 For a Bi-LSTM Model with GloVe embeddings and 0.424 for a BERT Model. Have not tried incorporating string matching to both of these yet. Since the current leaderboards seem so skewed to string matching, I'm really on the fence about picking my 2 NON string matching models or one which has it incorporated
", "votes": 1, "replies": [] }, { "id": 1328898, "author_name": "omarsamir", "author_url": "", "post_date": "2021-05-30T16:11:54.970000", "content": "I reached 0.393 using https://www.kaggle.com/tungmphung/pytorch-bert-for-named-entity-recognition work and could reach 0.412 using https://www.kaggle.com/theoviel/bert-for-question-answering-baseline-training
", "votes": 1, "replies": [] }, { "id": 1313063, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-05-18T11:45:35.047000", "content": "So what is the best public leader board scores with a single model no string matching without removing known label predictions that are jaccard similar?
", "votes": 1, "replies": [ { "id": 1313228, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2021-05-18T13:07:51.260000", "content": "Maybe 0.4+, not many people reported their results here.
", "votes": 1, "replies": [] }, { "id": 1313249, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-05-18T13:19:28.573000", "content": "Ok, I must be missing something - if you are getting .41 removing all predictions that are Jaccard similar >.5 to the known labels, why aren't you scoring MUCH higher when not removing Jaccard similar predictions?
", "votes": 1, "replies": [] }, { "id": 1313272, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2021-05-18T13:32:31.030000", "content": "If you laid out the metrics you will understand why:
\nF0.5 = TP/(TP + 0.2FN + 0.8FP)
\nA rough explanation is that our models predicted a large amount of false positives which are punished much harder than false negatives. Removing training labels means we moved some of our true positive predictions and turned them into false negatives, which doesn't affect the score that much due to the low coefficient.
", "votes": 5, "replies": [] }, { "id": 1313279, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-05-18T13:39:55.997000", "content": "They said there wouldn't be any math…. : ) - thanks - .8 is harsh and my best model has its share of FPs for sure - my best single model scores .49 (public LB) without matching and without removing any known label predictions. So it is pretty close to baseline matching, but I think it is over fitting to the training data I am using and I am concerned it won't generalize well on the test set / private LB.
\nThanks again for sharing and clarifying this @suicaokhoailang
", "votes": 1, "replies": [] }, { "id": 1318608, "author_name": "AndyX", "author_url": "", "post_date": "2021-05-22T12:35:41.493000", "content": "Why did you remove the train labels, because it is completely impossible for the train labels to appear in the test?
", "votes": 0, "replies": [] } ] }, { "id": 1302495, "author_name": "van73", "author_url": "", "post_date": "2021-05-11T15:18:09.050000", "content": "Updata: test score-0.393 from single model based on bert-base pretrained model using multi answer method.
", "votes": 1, "replies": [ { "id": 1309687, "author_name": "van73", "author_url": "", "post_date": "2021-05-16T07:58:01.353000", "content": "five single-fold models based the same QA architecture ensembled:0.409
", "votes": 1, "replies": [] }, { "id": 1321859, "author_name": "Liam Nguyen", "author_url": "", "post_date": "2021-05-25T03:43:45.370000", "content": "@vanle73 how can you make the model predict multiple answer span ? A typical approach of a QA model is to classify which token in your input sequence is the start token (the same to end tokens). So this model is only able to predict 1 answer given an input.
", "votes": 0, "replies": [] }, { "id": 1328381, "author_name": "van73", "author_url": "", "post_date": "2021-05-30T07:59:09.563000", "content": "We can create our questions for training, and get the final result by some postprocessing. : )
", "votes": 0, "replies": [] } ] }, { "id": 1294479, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2021-05-05T17:18:32.370000", "content": "More update: 0.409 single model. Overall all submissions got a reduced score.
", "votes": 1, "replies": [ { "id": 1295488, "author_name": "van73", "author_url": "", "post_date": "2021-05-06T13:51:12.770000", "content": "About mine. 80% of the data is used as the training set, and 20% of the data is used as the validation set. The score of the validation set is 0.848, but only 0.32 online!
\nI am very curious whether your final score is the fusion of multiple trained models, rather than the post-processing of some string matching, looking forward to your reply!😉
", "votes": 0, "replies": [] } ] }, { "id": 1275742, "author_name": "pin-rui", "author_url": "", "post_date": "2021-04-16T16:30:17.757000", "content": "using 2.5% size of training data, lb : 0.234
\nusing 1.5% size of training data, lb : 0.203
\nNot string matching, just a simple model.
did you remove all predicted labels which has a jaccard >= 0.5 with any label in the training set?
", "votes": 0, "replies": [] } ] }, { "id": 1341392, "author_name": "Jie Wu", "author_url": "", "post_date": "2021-06-08T16:11:13.740000", "content": "I got 0.455 and 0.429 LB without string match & without remove similar known labels for two single folds
", "votes": 2, "replies": [ { "id": 1351578, "author_name": "Jie Wu", "author_url": "", "post_date": "2021-06-16T12:19:43.743000", "content": "Update, got 0.509 LB without string match & without removing similar known labels
", "votes": 0, "replies": [] }, { "id": 1355520, "author_name": "Thang Dinh ", "author_url": "", "post_date": "2021-06-18T10:33:15.153000", "content": "greate job, i am so tired with string matching in this competition. ^^. Hopefully can learn how do you trained model after this competition.
", "votes": 0, "replies": [] } ] }, { "id": 1280860, "author_name": "zaccey", "author_url": "", "post_date": "2021-04-22T12:38:10.700000", "content": "Our best score is 0.023 with single model! 😨
\nWe use spaCy's English model.
Without removing predictions that has a Jaccard similarity greater than 0.5 to any of existing_labels, this model's score is 0.595…
\nFrom my perspective, it means our model is overfitting to existing labels.
I'll do my best with spaCy's English model a little more.
", "votes": 2, "replies": [ { "id": 1281198, "author_name": "Hannes Öhler", "author_url": "", "post_date": "2021-04-22T17:26:41.863000", "content": "I got a similar score with my local CV. Then I included weights in the cross entropy loss (to balance class imbalance due to the scarse occurence of datasets in the texts), which dramatically improved my score. Maybe this also applies to you. Good luck!
", "votes": 3, "replies": [] }, { "id": 1281457, "author_name": "zaccey", "author_url": "", "post_date": "2021-04-23T01:06:08.150000", "content": "Thank you so much!
\nI'll try it. 🙌
【May 5 update】
\nThis submission's score has changed to 0.019.
Great posting! And nice to see that I may not be that far off as the LB suggests :-) My score: 0.277
", "votes": 2, "replies": [ { "id": 1275753, "author_name": "Nguyen Quan Anh Minh", "author_url": "", "post_date": "2021-04-16T16:53:00.527000", "content": "did you remove all predicted labels which has a jaccard >= 0.5 with any label in the training set?
", "votes": 0, "replies": [] }, { "id": 1275800, "author_name": "Hannes Öhler", "author_url": "", "post_date": "2021-04-16T18:02:55.463000", "content": "Yes. I thought that's the point of this thread.
", "votes": 0, "replies": [] }, { "id": 1290823, "author_name": "Hannes Öhler", "author_url": "", "post_date": "2021-05-02T13:04:03.563000", "content": "This has been quite a struggle with bugs and submission errors.. but now I can also give an update: 0.277
", "votes": 0, "replies": [] } ] }, { "id": 1337098, "author_name": "Chien-Hsiang Hung", "author_url": "", "post_date": "2021-06-05T12:15:33.973000", "content": "This thread is relatively so much useful than public LB. Sould be pinned at top lol
\nI got 0.451 non-matching LB, haven't tried removing known labels.
If I understand correctly, @suicaokhoailang you got 0.664 using model w/o matching w/o removing known labels? and 0.414 w/o matching but after the removing known labels right?
I am really stuck and my score isn't moving, so I shared this notebook in hopes it helps someone move the needle. It is a Spacy classifier and uses no matching scores .44 on the LB. I have a similar BERT model which scores about the same. I extract using capitalized word sequences and acronyms like (BLSA) extracting the words before the acronym to avoid having to try and predict on all sentences.
\nhttps://www.kaggle.com/mlconsult/44-3-spacy-classifier-no-matching
", "votes": 0, "replies": [] }, { "id": 1297840, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-05-08T11:24:00.853000", "content": "I think the reason that prediction models are not working very well, among several other things, is because the challenge calls for:
\n\"Your predictions will be short excerpts from the publications that appear to note a dataset.\"
\nSo when you use matching to determine if a string \"is in\" a text, if it is, then you return the actual words from the text as your prediction - because those words are actually in the text. If you are returning a label from the dataset, that is not going to receive as good of score, because it is not actually what is in the text.
", "votes": 0, "replies": [] }, { "id": 1295660, "author_name": "davide", "author_url": "", "post_date": "2021-05-06T15:55:50.770000", "content": "0.346 on LB with the rerun of the submissions. I am using a very basic NER LSTM with pos label and word embedding
", "votes": 0, "replies": [] }, { "id": 1294540, "author_name": "Hannes Öhler", "author_url": "", "post_date": "2021-05-05T18:07:59.217000", "content": "My update: 0.300 I was lucky and my score increased a bit :-)
", "votes": 0, "replies": [] }, { "id": 1286620, "author_name": "Dubovskiy Dmitry", "author_url": "", "post_date": "2021-04-28T08:02:44.887000", "content": "Great topic!
\nMay I ask why are you removing \"every prediction that has a Jaccard similarity greater than 0.5 to any of the known labels in this list\"? This could be a case when your model is not doing string matching, however Jaccard similarity of its prediction is greater than 0.5 to know dataset titles.
Hey this is an interesting approach. I assume this number comes from the submission, isn't it?
", "votes": 0, "replies": [] }, { "id": 1279993, "author_name": "Mark P", "author_url": "", "post_date": "2021-04-21T13:01:21.870000", "content": "My BERT model is obviously quite off the mark - only 0.185 without string matching
", "votes": 0, "replies": [ { "id": 1280771, "author_name": "Mark P", "author_url": "", "post_date": "2021-04-22T10:27:22.427000", "content": "got it up to 0.259 - things are improving but still a long way to go…
", "votes": 2, "replies": [] }, { "id": 1295569, "author_name": "Mark P", "author_url": "", "post_date": "2021-05-06T14:57:05.593000", "content": "[5th May update]. after the rescore (and after re-trying old changes that had been ineffective on the old data) I've managed to get it up to 0.307
", "votes": 0, "replies": [] }, { "id": 1295582, "author_name": "van73", "author_url": "", "post_date": "2021-05-06T15:04:37.830000", "content": "hello I’m curious if your model is bert-ner？ My bert-QA model has a final result of 0.336 after the recent re-evaluation.😁
", "votes": 0, "replies": [] }, { "id": 1295598, "author_name": "Mark P", "author_url": "", "post_date": "2021-05-06T15:20:18.153000", "content": "Hi @vanle73 no its a BERT-MLM. I tried a Roberta-Squad straight out of the box and got about 0.12 so stopped that approach- looks like you had more success :) Did you fine tune it on this dataset? If so, just the given labels or a wider range?
", "votes": 1, "replies": [] }, { "id": 1295652, "author_name": "van73", "author_url": "", "post_date": "2021-05-06T15:50:16.403000", "content": "before , I have tried the single-ans QA model to build the solution, however, I failed, it scored less than 0.2, so I have changed the model method by predicting the ans's number, it has a not bad result. my 0.336 score came from the multi-answer QA model based on bert-based pretrained model.
\nIf possible, we can communicate together via email to reach a good final game result 😄
That's a really high score, I think that may be around the upper bound of what one may achieve after removing train labels.
\nWhat do you mean by \" x 130 labels\"? Does that mean you trained 130 models one for each train label? I wonder how LGBM can generalize so well to unseen text.
did you ignore all 130 training labels when inference ?, 60.9 is almost maximized score, and ur model combine with string match should get 0.9 right now :)
", "votes": 1, "replies": [] }, { "id": 1279086, "author_name": "OsciiArt", "author_url": "", "post_date": "2021-04-20T15:14:54.370000", "content": "@suicaokhoailang @dathudeptrai Sorry. I completely misunderstood this topic. My score above is just a prediction for the existing labels. My score without existing labels is 0.054 (not LGBM).
", "votes": 1, "replies": [] } ] }, { "id": 1362730, "author_name": "", "author_url": "", "post_date": "2021-06-23T15:34:33.720000", "content": "", "votes": -1, "replies": [] }, { "id": 1304083, "author_name": "", "author_url": "", "post_date": "2021-05-12T12:13:23.590000", "content": "", "votes": 0, "replies": [] }, { "id": 1301040, "author_name": "", "author_url": "", "post_date": "2021-05-10T23:19:53.720000", "content": "", "votes": 0, "replies": [ { "id": 1312649, "author_name": "", "author_url": "", "post_date": "2021-05-18T07:02:32.260000", "content": "", "votes": 0, "replies": [] }, { "id": 1312650, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2021-05-18T07:04:30.633000", "content": "You got 47.5 after removing predictions similar to training ones? Wow I think you may have gotten the rest of them.
", "votes": 2, "replies": [] }, { "id": 1313061, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-05-18T11:44:00.870000", "content": "No I misread - I did not use any matching, but did not remove any jaccard similar predictions same as the original labels.
", "votes": 1, "replies": [] } ] }, { "id": 1300351, "author_name": "", "author_url": "", "post_date": "2021-05-10T12:06:22.517000", "content": "", "votes": 0, "replies": [ { "id": 1300427, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2021-05-10T12:56:14.817000", "content": "The point of the thread is to remove predictions that were similar to those in training data though, not the opposite.
", "votes": 0, "replies": [] } ] }, { "id": 1295316, "author_name": "", "author_url": "", "post_date": "2021-05-06T11:20:02.197000", "content": "", "votes": 0, "replies": [] }, { "id": 1291218, "author_name": "", "author_url": "", "post_date": "2021-05-02T19:44:39.643000", "content": "", "votes": 0, "replies": [] }, { "id": 1280977, "author_name": "", "author_url": "", "post_date": "2021-04-22T14:39:37.257000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1275455": "Currently I'm running a simple post processing method to assert my model's real performance, that is to remove every prediction that has a Jaccard similarity greater than 0.5 to any of the known labels in this list:\n```\ntemp_1 = [x.lower() for x in train_df['dataset_label'].unique()]\ntemp_2 = [x.lower() for x in train_df['dataset_title'].unique()]\ntemp_3 = [x.lower() for x in train_df['cleaned_label'].unique()]\nexisting_labels = set(temp_1 + temp_2 + temp_3)\n\n```\n\nWith that, my best score is **0.414** using a single model, I'm curious how well others are doing since it's impossible to know looking at the leaderboard.", "1293912": "Please can everyone come back and update their scores on this when the rerun of the submissions are done on the LB?\n(I've been really trusting this thread as a measure of performance so it'd be nice to see how it changes)", "1283903": "More update: 0.414 single model, combining the model prediction with known labels gave us the current LB score.", "1277820": "Update: Our best score is 0.35 with single model.", "1276597": "Periodically updated: **0.425** is my public LB score with predictions removed with a Jaccard similarity >= 0.5 with any `existing_labels`.", "1297289": "I used a very standard BERT model to solve the NER problem. The single model result is 0.454. Combined with string matching can boost up the performance. However, we know string matching 1) may not work well on private LB and 2) is not the purpose for this competition.\n\nI think many of our non-string matching models are facing performance deterioriation issues: validation performance should be pretty good on set aside training data but public LB scores are much lower. I think performance deterioriation is caused by the difference between training data and public LB testing data. If tagging method can be more consistent among training and testing data, non-string matching models will have better performance...", "1361642": "@suicaokhoailang Congratulations, and thank you for this topic! I would have crashed and burned without it.", "1351593": "Currently sitting at **0.407** For a Bi-LSTM Model with GloVe embeddings and **0.424** for a BERT Model. Have not tried incorporating string matching to both of these yet. Since the current leaderboards seem so skewed to string matching, I'm really on the fence about picking my 2 **NON** string matching models or one which has it incorporated", "1328898": "I reached 0.393 using https://www.kaggle.com/tungmphung/pytorch-bert-for-named-entity-recognition work and could reach 0.412 using https://www.kaggle.com/theoviel/bert-for-question-answering-baseline-training", "1313063": "So what is the best public leader board scores with a single model no string matching without removing known label predictions that are jaccard similar?", "1302495": "Updata: test score-**0.393** from single model based on bert-base pretrained model using multi answer method.", "1294479": "More update: 0.409 single model. Overall all submissions got a reduced score.", "1275742": "using 2.5% size of training data, lb : 0.234\nusing 1.5% size of training data, lb : 0.203\nNot string matching, just a simple model.", "1341392": "I got 0.455 and 0.429 LB without string match & without remove similar known labels for two single folds", "1280860": "Our best score is 0.023 with single model! 😨\nWe use spaCy's English model.\n\nWithout removing predictions that has a Jaccard similarity greater than 0.5 to any of `existing_labels`, this model's score is 0.595...\nFrom my perspective, it means our model is overfitting to existing labels.\n\nI'll do my best with spaCy's English model a little more.", "1275629": "Great posting! And nice to see that I may not be that far off as the LB suggests :-) My score: **0.277**", "1337098": "This thread is relatively so much useful than public LB. Sould be pinned at top lol\n\nI got **0.451** `non-matching` LB, haven't tried `removing known labels`. \n\nIf I understand correctly, @suicaokhoailang you got **0.664** using model `w/o matching` `w/o removing known labels`? and **0.414** `w/o matching` but after the `removing known labels` right?", "1323889": "I am really stuck and my score isn't moving, so I shared this notebook in hopes it helps someone move the needle. It is a Spacy classifier and uses no matching scores .44 on the LB. I have a similar BERT model which scores about the same. I extract using capitalized word sequences and acronyms like (BLSA) extracting the words before the acronym to avoid having to try and predict on all sentences.\n\nhttps://www.kaggle.com/mlconsult/44-3-spacy-classifier-no-matching", "1297840": "I think the reason that prediction models are not working very well, among several other things, is because the challenge calls for:\n\n\"Your predictions will be short excerpts from the publications that appear to note a dataset.\"\n\nSo when you use matching to determine if a string \"is in\" a text, if it is, then you return the actual words from the text as your prediction - because those words are actually in the text. If you are returning a label from the dataset, that is not going to receive as good of score, because it is not actually what is in the text.", "1295660": "**0.346** on LB with the rerun of the submissions. I am using a very basic NER LSTM with pos label and word embedding", "1294540": "My update: **0.300** I was lucky and my score increased a bit :-)", "1286620": "Great topic! \nMay I ask why are you removing \"every prediction that has a Jaccard similarity greater than 0.5 to any of the known labels in this list\"? This could be a case when your model is not doing string matching, however Jaccard similarity of its prediction is greater than 0.5 to know dataset titles.", "1286326": "Hey this is an interesting approach. I assume this number comes from the submission, isn't it?", "1279993": "My BERT model is obviously quite off the mark - only 0.185 without string matching ", "1278480": "~~Binary-classification LigthGMB models x 130 labels : 0.609~~", "1362730": "", "1304083": "", "1301040": "", "1300351": "", "1295316": "", "1291218": "", "1280977": "" } }