{ "id": 244505, "title": "Let's build a Non-Train-Matching, Train-Label-Removed LB", "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/244505", "author_name": "Chien-Hsiang Hung", "post_date": "2021-06-07T05:42:41.421000", "votes": 35, "comment_count": 24, "views": 0, "content": "

I got 0.453 Non-Train-Matching LB, haven't tried Train-Label-Removed. Below are some results I found in the Discussion.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
TeamNTM/TLR ScoreModel
@trentb0.590/0.425
@zashio @paseri @makinotakuya @lystahi0.595/0.023spaCy
@mlconsult0.490, 0.440classifier, spaCy
@vanle73 @yunliangli @yechens0.505, 0.367QA, NER
@sreejaej0.463/0.490translation
@xiaonanji0.454NER
@suicaokhoailang @dathudeptrai0.414
@osamir0.412, 0.393QA, NER
@davidemariani0.346NER
@cascadenite0.307MLM
@udaygurugubelli0.300QA
@latong0.299
@hannes820.277
@osciiart0.054
\n

Sources:

\n\n

This LB only works if we're all willing to share. Please come and share your result, I'll update it accordingly so that we have a more reliable LB to check.

\n

Thanks to everyone willing to contribute,
\nHsiang

", "messages": [ { "id": 1339217, "postDate": "2021-06-07T05:42:41.420Z", "content": "

I got 0.453 Non-Train-Matching LB, haven't tried Train-Label-Removed. Below are some results I found in the Discussion.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
TeamNTM/TLR ScoreModel
@trentb0.590/0.425
@zashio @paseri @makinotakuya @lystahi0.595/0.023spaCy
@mlconsult0.490, 0.440classifier, spaCy
@vanle73 @yunliangli @yechens0.505, 0.367QA, NER
@sreejaej0.463/0.490translation
@xiaonanji0.454NER
@suicaokhoailang @dathudeptrai0.414
@osamir0.412, 0.393QA, NER
@davidemariani0.346NER
@cascadenite0.307MLM
@udaygurugubelli0.300QA
@latong0.299
@hannes820.277
@osciiart0.054
\n

Sources:

\n\n

This LB only works if we're all willing to share. Please come and share your result, I'll update it accordingly so that we have a more reliable LB to check.

\n

Thanks to everyone willing to contribute,
\nHsiang

", "rawMarkdown": "I got 0.453 `Non-Train-Matching` LB, haven't tried `Train-Label-Removed`. Below are some results I found in the Discussion.\n\n| Team | NTM/TLR Score | Model |\n| --- | --- | --- |\n| @trentb | 0.590/0.425 |\n| @zashio @paseri @makinotakuya @lystahi | 0.595/0.023 | spaCy |\n| @mlconsult | 0.490, 0.440 | classifier, spaCy |\n| @vanle73 @yunliangli @yechens | 0.505, 0.367 | QA, NER |\n| @sreejaej | 0.463/0.490 | translation |\n|@xiaonanji | 0.454 | NER |\n| @suicaokhoailang @dathudeptrai | 0.414 |\n| @osamir | 0.412, 0.393 | QA, NER |\n| @davidemariani | 0.346 | NER |\n| @cascadenite | 0.307 | MLM |\n| @udaygurugubelli | 0.300 | QA |\n| @latong | 0.299 | |\n| @hannes82 | 0.277 | |\n| @osciiart | 0.054 | |\n\nSources:\n- [What is your best score without string matching?](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/232964)\n- [Single Model Performance](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/240948#1318370)\n- [Use multi-answer QA model competition records](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/238226#1302509)\n\nThis LB only works if we're all willing to share. Please come and share your result, I'll update it accordingly so that we have a more reliable LB to check.\n\nThanks to everyone willing to contribute,\nHsiang", "votes": 35 }, { "id": 1339795, "postDate": "2021-06-07T13:13:09.997Z", "content": "

@chienhsianghung Thanks for putting this together! I should probably clarify that my (@trentb) 0.590 score is a rule-based model, so not strictly string matching. My strictly string matching model scores 0.648.

", "rawMarkdown": "@chienhsianghung Thanks for putting this together! I should probably clarify that my (@trentb) 0.590 score is a rule-based model, so not strictly string matching. My strictly string matching model scores 0.648.", "votes": 5, "replies": [ { "id": 1339996, "postDate": "2021-06-07T15:26:14.693Z", "content": "

Thanks for your contribution!

", "rawMarkdown": "Thanks for your contribution!" }, { "id": 1343517, "postDate": "2021-06-10T08:47:40.250Z", "content": "

@trentb That's a great score. I actually think rule-based method will be robust in this competition, there are only that many ways you can introduce a dataset.

", "rawMarkdown": "@trentb That's a great score. I actually think rule-based method will be robust in this competition, there are only that many ways you can introduce a dataset.", "votes": 1 }, { "id": 1344373, "postDate": "2021-06-10T21:26:45.307Z", "content": "

Thank you. Without a local CV, it's hard to not optimize to the public LB score (even excluding the existing datasets), which could come back to haunt me. We'll find out soon enough.

", "rawMarkdown": "Thank you. Without a local CV, it's hard to not optimize to the public LB score (even excluding the existing datasets), which could come back to haunt me. We'll find out soon enough." } ] }, { "id": 1340169, "postDate": "2021-06-07T17:31:46.523Z", "content": "

Thanks for putting this together @chienhsianghung - my .490 model is a BERT classifier. My .44 model is a spacy model - neither use matching and they DO NOT remove the original training set labels from the submission. These scores are public LB scores.

\n

I have shared the spacy notebook here - https://www.kaggle.com/mlconsult/44-3-spacy-classifier-no-matching

", "rawMarkdown": "Thanks for putting this together @chienhsianghung - my .490 model is a BERT classifier. My .44 model is a spacy model - neither use matching and they DO NOT remove the original training set labels from the submission. These scores are public LB scores.\n\nI have shared the spacy notebook here - https://www.kaggle.com/mlconsult/44-3-spacy-classifier-no-matching", "votes": 3, "replies": [ { "id": 1340604, "postDate": "2021-06-08T06:13:05.640Z", "content": "

Updated, thank you for your sharing!!! @mlconsult

", "rawMarkdown": "Updated, thank you for your sharing!!! @mlconsult " } ] }, { "id": 1340011, "postDate": "2021-06-07T15:41:38.410Z", "content": "

@chienhsianghung Nice job for collection, I'd add a NER model result of 0.367 w/o string matching.

", "rawMarkdown": "@chienhsianghung Nice job for collection, I'd add a NER model result of 0.367 w/o string matching.", "votes": 3, "replies": [ { "id": 1340018, "postDate": "2021-06-07T15:47:03.447Z", "content": "

Thank you @vanle73. It's thrilling to see you continuously improved your muti-QA model along the way!

", "rawMarkdown": "Thank you @vanle73. It's thrilling to see you continuously improved your muti-QA model along the way!", "votes": 1 }, { "id": 1340450, "postDate": "2021-06-08T00:36:37.357Z", "content": "

How does Multi-QA work if Question Answering predicts only a and index?

", "rawMarkdown": "How does Multi-QA work if Question Answering predicts only a and index?" } ] }, { "id": 1341884, "postDate": "2021-06-09T04:26:09.617Z", "content": "

Hi,

\n

Thanks for putting this together.

\n

I got a score of 0.463 with using only the model (I added a check to see if the string is int he text to avoid false positives though. (. https://www.kaggle.com/sreejaej/colleridge-attention-inference )
\n(Looks like the score improves to 0.49 if we remove the known labels. But I need to double confirm this ).

\n

I approached the problem like that of 'translation problem'. Input a text and attempt to generate a set of words given the previous word. ( https://www.kaggle.com/sreejaej/attention-is-all-you-need-cv ).

\n

Regards,
\nSreeja

", "rawMarkdown": "Hi, \n\nThanks for putting this together. \n\nI got a score of 0.463 with using only the model (I added a check to see if the string is int he text to avoid false positives though. (. https://www.kaggle.com/sreejaej/colleridge-attention-inference ) \n(Looks like the score improves to 0.49 if we remove the known labels. But I need to double confirm this ). \n\nI approached the problem like that of 'translation problem'. Input a text and attempt to generate a set of words given the previous word. ( https://www.kaggle.com/sreejaej/attention-is-all-you-need-cv ). \n\nRegards,\nSreeja ", "votes": 4, "replies": [ { "id": 1343008, "postDate": "2021-06-09T23:51:01.710Z", "content": "

Wow, nice to see a new approach, this is pretty novel! Congratulations on getting a good result, and thank you for sharing. @sreejaej

", "rawMarkdown": "Wow, nice to see a new approach, this is pretty novel! Congratulations on getting a good result, and thank you for sharing. @sreejaej ", "votes": 1 } ] }, { "id": 1356124, "postDate": "2021-06-18T18:39:59.560Z", "content": "

I'm not sure why you want to remove the train labels.

\n

The original quote from host is:
\nThere is no crossover between train and the private test set, so string matching won't be an effective strategy there.

\n

I think this statement means that there is no crossover in the train text and private test text, not that there is no crossover in the train and private test label. so I think that private test text may still contain the train label.

\n

I would like to hear your explanation

", "rawMarkdown": "I'm not sure why you want to remove the train labels.\n\nThe original quote from host is:\n`There is no crossover between train and the private test set, so string matching won't be an effective strategy there.`\n\nI think this statement means that there is no crossover in the train text and private test text, not that there is no crossover in the train and private test label. so I think that private test text may still contain the train label.\n\nI would like to hear your explanation", "votes": 1, "replies": [ { "id": 1356387, "postDate": "2021-06-19T01:57:49.623Z", "content": "

The theory is your model is tested only on what it finds beyond those known labels. Essentially no leakage. Also, there could be datasets in the train set that weren't labelled, but may be in test as they say there won't be any crossover labels. I think the problem is, no one really knows and you have to have the best AI that can find govt datasets in a text…

\n

The best test, IMO, would be absolutely no datasets in the train that are in the test, but this is not clear by the directions.

", "rawMarkdown": "The theory is your model is tested only on what it finds beyond those known labels. Essentially no leakage. Also, there could be datasets in the train set that weren't labelled, but may be in test as they say there won't be any crossover labels. I think the problem is, no one really knows and you have to have the best AI that can find govt datasets in a text...\n\nThe best test, IMO, would be absolutely no datasets in the train that are in the test, but this is not clear by the directions.", "votes": 3 }, { "id": 1356446, "postDate": "2021-06-19T03:22:22.343Z", "content": "

I got it. thank you!😃

", "rawMarkdown": "I got it. thank you!😃" } ] }, { "id": 1361055, "postDate": "2021-06-22T14:21:53.877Z", "content": "

Good luck everyone, finally got a non-string-matching NER model scored 0.442 (a SciBERT+Bi-LSTM-CRF model), not sure how it goes with the blending of other model outputs, fingers crossed!

", "rawMarkdown": "Good luck everyone, finally got a non-string-matching NER model scored **0.442** (a SciBERT+Bi-LSTM-CRF model), not sure how it goes with the blending of other model outputs, fingers crossed!" }, { "id": 1359696, "postDate": "2021-06-21T13:12:51.950Z", "content": "

Hey guys. I really wanted to know how the ground truth labels are formulated. For instance, in the train set, even in case of multiple dataset occurences/citations, there is still only one label mentioned in the train.csv (For example, just ADNI)……I just wanted to clarify if that's done on purpose (to make public test harder for the model) and that all mentioned datasets in the publication is expected as the output ?

", "rawMarkdown": "Hey guys. I really wanted to know how the ground truth labels are formulated. For instance, in the train set, even in case of multiple dataset occurences/citations, there is still only one label mentioned in the train.csv (For example, just ADNI)......I just wanted to clarify if that's done on purpose (to make public test harder for the model) and that all mentioned datasets in the publication is expected as the output ?" }, { "id": 1358639, "postDate": "2021-06-20T16:41:58.147Z", "content": "

I got 0.400 in TLR using NER in bert-sklearn (no string matching). But I had labeled some training data by hand and I did not remove those labels. So a higher TLR score does not necessarily mean that my model is doing better in finding unseen dataset names in my case. But I expect it to do better overall.

", "rawMarkdown": "I got 0.400 in TLR using NER in bert-sklearn (no string matching). But I had labeled some training data by hand and I did not remove those labels. So a higher TLR score does not necessarily mean that my model is doing better in finding unseen dataset names in my case. But I expect it to do better overall." }, { "id": 1351797, "postDate": "2021-06-16T16:00:18.693Z", "content": "

This is a good idea and I hope string matching solutions will not win out in the end - but maybe that's wishful thinking.

\n

We are at 0.512/0.392 (NTM/TLR) -> using spacy mostly

\n

Sadly gains on these metrics have actually led to decrease on the LB when enabling matching - probably due to predicting somewhat similar labels twice instead of once.

", "rawMarkdown": "This is a good idea and I hope string matching solutions will not win out in the end - but maybe that's wishful thinking.\n\nWe are at **0.512**/**0.392** (NTM/TLR) -> using spacy mostly \n\nSadly gains on these metrics have actually led to decrease on the LB when enabling matching - probably due to predicting somewhat similar labels twice instead of once." }, { "id": 1351110, "postDate": "2021-06-16T04:42:11.740Z", "content": "

For removing the known labels, which one of the two logics do you use?

\n

a. Remove all strings which are exactly matching to one of the train labels
\nb. Remove all string whose jaccard score (or any other text similarity metric) with any of existing labels is greater than some threshold value.

\n

So, if your existing label is \"agricultural resources management survey\" and your prediction string is \"agricultural resources management survey arms\", will you remove the prediction string?

", "rawMarkdown": "For removing the known labels, which one of the two logics do you use?\n\na. Remove all strings which are exactly matching to one of the train labels\nb. Remove all string whose jaccard score (or any other text similarity metric) with any of existing labels is greater than some threshold value.\n\nSo, if your existing label is \"agricultural resources management survey\" and your prediction string is \"agricultural resources management survey arms\", will you remove the prediction string?", "replies": [ { "id": 1351536, "postDate": "2021-06-16T11:36:59.923Z", "content": "

jaccard similarity .5

", "rawMarkdown": "jaccard similarity .5", "votes": 1 } ] }, { "id": 1350615, "postDate": "2021-06-15T15:50:58.850Z", "content": "

what 'Train-Label-Removed' means?

", "rawMarkdown": "what 'Train-Label-Removed' means?", "replies": [ { "id": 1350743, "postDate": "2021-06-15T17:53:28.607Z", "content": "

that you removed all the known labels from the training data in your predictions - the theory is your model is tested only on what it finds beyond those known labels.

", "rawMarkdown": "that you removed all the known labels from the training data in your predictions - the theory is your model is tested only on what it finds beyond those known labels." }, { "id": 1350946, "postDate": "2021-06-16T00:40:07.120Z", "content": "

got it, thanks

", "rawMarkdown": "got it, thanks" } ] }, { "id": 1343489, "postDate": "2021-06-10T08:31:38.957Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1339795, "author_name": "Trent", "author_url": "", "post_date": "2021-06-07T13:13:09.997000", "content": "

@chienhsianghung Thanks for putting this together! I should probably clarify that my (@trentb) 0.590 score is a rule-based model, so not strictly string matching. My strictly string matching model scores 0.648.

", "votes": 5, "replies": [ { "id": 1339996, "author_name": "Chien-Hsiang Hung", "author_url": "", "post_date": "2021-06-07T15:26:14.693000", "content": "

Thanks for your contribution!

", "votes": 0, "replies": [] }, { "id": 1343517, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2021-06-10T08:47:40.250000", "content": "

@trentb That's a great score. I actually think rule-based method will be robust in this competition, there are only that many ways you can introduce a dataset.

", "votes": 1, "replies": [] }, { "id": 1344373, "author_name": "Trent", "author_url": "", "post_date": "2021-06-10T21:26:45.307000", "content": "

Thank you. Without a local CV, it's hard to not optimize to the public LB score (even excluding the existing datasets), which could come back to haunt me. We'll find out soon enough.

", "votes": 0, "replies": [] } ] }, { "id": 1340169, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-06-07T17:31:46.523000", "content": "

Thanks for putting this together @chienhsianghung - my .490 model is a BERT classifier. My .44 model is a spacy model - neither use matching and they DO NOT remove the original training set labels from the submission. These scores are public LB scores.

\n

I have shared the spacy notebook here - https://www.kaggle.com/mlconsult/44-3-spacy-classifier-no-matching

", "votes": 3, "replies": [ { "id": 1340604, "author_name": "Chien-Hsiang Hung", "author_url": "", "post_date": "2021-06-08T06:13:05.640000", "content": "

Updated, thank you for your sharing!!! @mlconsult

", "votes": 0, "replies": [] } ] }, { "id": 1340011, "author_name": "van73", "author_url": "", "post_date": "2021-06-07T15:41:38.410000", "content": "

@chienhsianghung Nice job for collection, I'd add a NER model result of 0.367 w/o string matching.

", "votes": 3, "replies": [ { "id": 1340018, "author_name": "Chien-Hsiang Hung", "author_url": "", "post_date": "2021-06-07T15:47:03.447000", "content": "

Thank you @vanle73. It's thrilling to see you continuously improved your muti-QA model along the way!

", "votes": 1, "replies": [] }, { "id": 1340450, "author_name": "Andrew Shao", "author_url": "", "post_date": "2021-06-08T00:36:37.357000", "content": "

How does Multi-QA work if Question Answering predicts only a and index?

", "votes": 0, "replies": [] } ] }, { "id": 1341884, "author_name": "Sreeja", "author_url": "", "post_date": "2021-06-09T04:26:09.617000", "content": "

Hi,

\n

Thanks for putting this together.

\n

I got a score of 0.463 with using only the model (I added a check to see if the string is int he text to avoid false positives though. (. https://www.kaggle.com/sreejaej/colleridge-attention-inference )
\n(Looks like the score improves to 0.49 if we remove the known labels. But I need to double confirm this ).

\n

I approached the problem like that of 'translation problem'. Input a text and attempt to generate a set of words given the previous word. ( https://www.kaggle.com/sreejaej/attention-is-all-you-need-cv ).

\n

Regards,
\nSreeja

", "votes": 4, "replies": [ { "id": 1343008, "author_name": "Chien-Hsiang Hung", "author_url": "", "post_date": "2021-06-09T23:51:01.710000", "content": "

Wow, nice to see a new approach, this is pretty novel! Congratulations on getting a good result, and thank you for sharing. @sreejaej

", "votes": 1, "replies": [] } ] }, { "id": 1356124, "author_name": "AndyX", "author_url": "", "post_date": "2021-06-18T18:39:59.560000", "content": "

I'm not sure why you want to remove the train labels.

\n

The original quote from host is:
\nThere is no crossover between train and the private test set, so string matching won't be an effective strategy there.

\n

I think this statement means that there is no crossover in the train text and private test text, not that there is no crossover in the train and private test label. so I think that private test text may still contain the train label.

\n

I would like to hear your explanation

", "votes": 1, "replies": [ { "id": 1356387, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-06-19T01:57:49.623000", "content": "

The theory is your model is tested only on what it finds beyond those known labels. Essentially no leakage. Also, there could be datasets in the train set that weren't labelled, but may be in test as they say there won't be any crossover labels. I think the problem is, no one really knows and you have to have the best AI that can find govt datasets in a text…

\n

The best test, IMO, would be absolutely no datasets in the train that are in the test, but this is not clear by the directions.

", "votes": 3, "replies": [] }, { "id": 1356446, "author_name": "AndyX", "author_url": "", "post_date": "2021-06-19T03:22:22.343000", "content": "

I got it. thank you!😃

", "votes": 0, "replies": [] } ] }, { "id": 1361055, "author_name": "Mark Peng", "author_url": "", "post_date": "2021-06-22T14:21:53.877000", "content": "

Good luck everyone, finally got a non-string-matching NER model scored 0.442 (a SciBERT+Bi-LSTM-CRF model), not sure how it goes with the blending of other model outputs, fingers crossed!

", "votes": 0, "replies": [] }, { "id": 1359696, "author_name": "SanVik2000", "author_url": "", "post_date": "2021-06-21T13:12:51.950000", "content": "

Hey guys. I really wanted to know how the ground truth labels are formulated. For instance, in the train set, even in case of multiple dataset occurences/citations, there is still only one label mentioned in the train.csv (For example, just ADNI)……I just wanted to clarify if that's done on purpose (to make public test harder for the model) and that all mentioned datasets in the publication is expected as the output ?

", "votes": 0, "replies": [] }, { "id": 1358639, "author_name": "ozan.özgür", "author_url": "", "post_date": "2021-06-20T16:41:58.147000", "content": "

I got 0.400 in TLR using NER in bert-sklearn (no string matching). But I had labeled some training data by hand and I did not remove those labels. So a higher TLR score does not necessarily mean that my model is doing better in finding unseen dataset names in my case. But I expect it to do better overall.

", "votes": 0, "replies": [] }, { "id": 1351797, "author_name": "Gautier", "author_url": "", "post_date": "2021-06-16T16:00:18.693000", "content": "

This is a good idea and I hope string matching solutions will not win out in the end - but maybe that's wishful thinking.

\n

We are at 0.512/0.392 (NTM/TLR) -> using spacy mostly

\n

Sadly gains on these metrics have actually led to decrease on the LB when enabling matching - probably due to predicting somewhat similar labels twice instead of once.

", "votes": 0, "replies": [] }, { "id": 1351110, "author_name": "Kumar Shubham", "author_url": "", "post_date": "2021-06-16T04:42:11.740000", "content": "

For removing the known labels, which one of the two logics do you use?

\n

a. Remove all strings which are exactly matching to one of the train labels
\nb. Remove all string whose jaccard score (or any other text similarity metric) with any of existing labels is greater than some threshold value.

\n

So, if your existing label is \"agricultural resources management survey\" and your prediction string is \"agricultural resources management survey arms\", will you remove the prediction string?

", "votes": 0, "replies": [ { "id": 1351536, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-06-16T11:36:59.923000", "content": "

jaccard similarity .5

", "votes": 1, "replies": [] } ] }, { "id": 1350615, "author_name": "liuze", "author_url": "", "post_date": "2021-06-15T15:50:58.850000", "content": "

what 'Train-Label-Removed' means?

", "votes": 0, "replies": [ { "id": 1350743, "author_name": "Ken Miller", "author_url": "", "post_date": "2021-06-15T17:53:28.607000", "content": "

that you removed all the known labels from the training data in your predictions - the theory is your model is tested only on what it finds beyond those known labels.

", "votes": 0, "replies": [] }, { "id": 1350946, "author_name": "liuze", "author_url": "", "post_date": "2021-06-16T00:40:07.120000", "content": "

got it, thanks

", "votes": 0, "replies": [] } ] }, { "id": 1343489, "author_name": "", "author_url": "", "post_date": "2021-06-10T08:31:38.957000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1339217": "I got 0.453 `Non-Train-Matching` LB, haven't tried `Train-Label-Removed`. Below are some results I found in the Discussion.\n\n| Team | NTM/TLR Score | Model |\n| --- | --- | --- |\n| @trentb | 0.590/0.425 |\n| @zashio @paseri @makinotakuya @lystahi | 0.595/0.023 | spaCy |\n| @mlconsult | 0.490, 0.440 | classifier, spaCy |\n| @vanle73 @yunliangli @yechens | 0.505, 0.367 | QA, NER |\n| @sreejaej | 0.463/0.490 | translation |\n|@xiaonanji | 0.454 | NER |\n| @suicaokhoailang @dathudeptrai | 0.414 |\n| @osamir | 0.412, 0.393 | QA, NER |\n| @davidemariani | 0.346 | NER |\n| @cascadenite | 0.307 | MLM |\n| @udaygurugubelli | 0.300 | QA |\n| @latong | 0.299 | |\n| @hannes82 | 0.277 | |\n| @osciiart | 0.054 | |\n\nSources:\n- [What is your best score without string matching?](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/232964)\n- [Single Model Performance](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/240948#1318370)\n- [Use multi-answer QA model competition records](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/238226#1302509)\n\nThis LB only works if we're all willing to share. Please come and share your result, I'll update it accordingly so that we have a more reliable LB to check.\n\nThanks to everyone willing to contribute,\nHsiang", "1339795": "@chienhsianghung Thanks for putting this together! I should probably clarify that my (@trentb) 0.590 score is a rule-based model, so not strictly string matching. My strictly string matching model scores 0.648.", "1340169": "Thanks for putting this together @chienhsianghung - my .490 model is a BERT classifier. My .44 model is a spacy model - neither use matching and they DO NOT remove the original training set labels from the submission. These scores are public LB scores.\n\nI have shared the spacy notebook here - https://www.kaggle.com/mlconsult/44-3-spacy-classifier-no-matching", "1340011": "@chienhsianghung Nice job for collection, I'd add a NER model result of 0.367 w/o string matching.", "1341884": "Hi, \n\nThanks for putting this together. \n\nI got a score of 0.463 with using only the model (I added a check to see if the string is int he text to avoid false positives though. (. https://www.kaggle.com/sreejaej/colleridge-attention-inference ) \n(Looks like the score improves to 0.49 if we remove the known labels. But I need to double confirm this ). \n\nI approached the problem like that of 'translation problem'. Input a text and attempt to generate a set of words given the previous word. ( https://www.kaggle.com/sreejaej/attention-is-all-you-need-cv ). \n\nRegards,\nSreeja ", "1356124": "I'm not sure why you want to remove the train labels.\n\nThe original quote from host is:\n`There is no crossover between train and the private test set, so string matching won't be an effective strategy there.`\n\nI think this statement means that there is no crossover in the train text and private test text, not that there is no crossover in the train and private test label. so I think that private test text may still contain the train label.\n\nI would like to hear your explanation", "1361055": "Good luck everyone, finally got a non-string-matching NER model scored **0.442** (a SciBERT+Bi-LSTM-CRF model), not sure how it goes with the blending of other model outputs, fingers crossed!", "1359696": "Hey guys. I really wanted to know how the ground truth labels are formulated. For instance, in the train set, even in case of multiple dataset occurences/citations, there is still only one label mentioned in the train.csv (For example, just ADNI)......I just wanted to clarify if that's done on purpose (to make public test harder for the model) and that all mentioned datasets in the publication is expected as the output ?", "1358639": "I got 0.400 in TLR using NER in bert-sklearn (no string matching). But I had labeled some training data by hand and I did not remove those labels. So a higher TLR score does not necessarily mean that my model is doing better in finding unseen dataset names in my case. But I expect it to do better overall.", "1351797": "This is a good idea and I hope string matching solutions will not win out in the end - but maybe that's wishful thinking.\n\nWe are at **0.512**/**0.392** (NTM/TLR) -> using spacy mostly \n\nSadly gains on these metrics have actually led to decrease on the LB when enabling matching - probably due to predicting somewhat similar labels twice instead of once.", "1351110": "For removing the known labels, which one of the two logics do you use?\n\na. Remove all strings which are exactly matching to one of the train labels\nb. Remove all string whose jaccard score (or any other text similarity metric) with any of existing labels is greater than some threshold value.\n\nSo, if your existing label is \"agricultural resources management survey\" and your prediction string is \"agricultural resources management survey arms\", will you remove the prediction string?", "1350615": "what 'Train-Label-Removed' means?", "1343489": "" } }