The current multi-answer QA model is used for prediction, and fine-tuned on bert base. The score of the verification set (20% of all data) is 0.96, and the effect submitted on the public test set is 0.393. Continue to update…
\nupdate: singe QA model 0.425
\nupdate:0.505
", "messages": [ { "id": "1302509", "postDate": "05/11/2021 15:23:18", "content": "The current multi-answer QA model is used for prediction, and fine-tuned on bert base. The score of the verification set (20% of all data) is 0.96, and the effect submitted on the public test set is 0.393. Continue to update…
\nupdate: singe QA model 0.425
\nupdate:0.505
", "rawMarkdown": "The current multi-answer QA model is used for prediction, and fine-tuned on bert base. The score of the verification set (20% of all data) is **0.96**, and the effect submitted on the public test set is **0.393**. Continue to update...\n\n--------\nupdate: singe QA model 0.425\n\n--------\nupdate:0.505", "votes": null }, { "id": "1302553", "postDate": "05/11/2021 15:37:44", "content": "@vanle73 Nice! May I ask you: Is the submission score with or without known training labels?
", "rawMarkdown": "vanle73 Nice! May I ask you: Is the submission score with or without known training labels?", "votes": null }, { "id": "1302575", "postDate": "05/11/2021 15:50:53", "content": "Without any label match postprocessing!
", "rawMarkdown": "Without any label match postprocessing!", "votes": null }, { "id": "1312295", "postDate": "05/18/2021 01:20:32", "content": "How to you design the questions ? Only a general question like \"What is the dataset used ?\". @vanle73
", "rawMarkdown": "How to you design the questions ? Only a general question like \"What is the dataset used ?\". @vanle73", "votes": null }, { "id": "1314609", "postDate": "05/19/2021 09:10:29", "content": "@hannes82 could you pls explain what do you mean by with out training labels? my doubt is if you remove how do you give start and end indices as labels??
\nyou mean to remove at the test time???
@udaygurugubelli yes excactly, you normally predict datasets at test time with your model and as a postprocessing procedure you remove all your predictions that match with a training label with a Jaccard score >=0.5 (see also this discussion).
\nIn this way, you ensure that you measure the performance of your model on unseen datasets (useful because in the private test set there are only unseen dataset labels).
", "rawMarkdown": "udaygurugubelli yes excactly, you normally predict datasets at test time with your model and as a postprocessing procedure you remove all your predictions that match with a training label with a Jaccard score >=0.5 (see also [this](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/232964) discussion).\n\nIn this way, you ensure that you measure the performance of your model on *unseen* datasets (useful because in the private test set there are only unseen dataset labels).", "votes": null }, { "id": "1314735", "postDate": "05/19/2021 10:56:58", "content": "Thank you for your reply. But I found that once I did this operation, the result I got using the QA model changed from about 0.4 to 0.03, and there was almost no matching pair. After you perform this operation, has the public Lb increased? Your reply is the greatest help to me, thank you
", "rawMarkdown": "Thank you for your reply. But I found that once I did this operation, the result I got using the QA model changed from about 0.4 to 0.03, and there was almost no matching pair. After you perform this operation, has the public Lb increased? Your reply is the greatest help to me, thank you", "votes": null }, { "id": "1314747", "postDate": "05/19/2021 11:10:59", "content": "Yes! The simplest way
", "rawMarkdown": "Yes! The simplest way", "votes": null }, { "id": "1314766", "postDate": "05/19/2021 11:22:38", "content": "thanks for the reply. did you see that they are talking about rescoring some kind of post processing after the prediction.
", "rawMarkdown": "thanks for the reply. did you see that they are talking about rescoring some kind of post processing after the prediction.", "votes": null }, { "id": "1314768", "postDate": "05/19/2021 11:23:30", "content": "@vanle73 yes..true..mine also same..something needs to be fixed here
", "rawMarkdown": "vanle73 yes..true..mine also same..something needs to be fixed here", "votes": null }, { "id": "1314785", "postDate": "05/19/2021 11:34:03", "content": "That unfortunately means that your model is not generalizing well.
\nAfter this operation, the score is exptected to decrease because you are removing TP from your predictions. But the aim of the postprocessing operation is not to improve on the public Lb. It is to get a better measure on how well your model does on identifying onseen datasets.
", "rawMarkdown": "That unfortunately means that your model is not generalizing well. \n\nAfter this operation, the score is exptected to decrease because you are removing TP from your predictions. But the aim of the postprocessing operation is not to improve on the public Lb. It is to get a better measure on how well your model does on identifying onseen datasets.", "votes": null }, { "id": "1314874", "postDate": "05/19/2021 12:25:55", "content": "in that case there is hope to get the best then..lets c..
", "rawMarkdown": "in that case there is hope to get the best then..lets c..", "votes": null }, { "id": "1315078", "postDate": "05/19/2021 14:50:00", "content": "While there are unseen dataset labels in the private testing data, the dataset labels provided for the training data may also be in the private testing data. We don't really know how prevalent they may be though.
", "rawMarkdown": "While there are unseen dataset labels in the private testing data, the dataset labels provided for the training data may also be in the private testing data. We don't really know how prevalent they may be though.", "votes": null }, { "id": "1315111", "postDate": "05/19/2021 15:11:56", "content": "No, I don't think so:
\n\n\nThere is no crossover between train and the private test set, so string matching won't be an effective strategy there.
\n
Also important to note:
\n\n", "rawMarkdown": "No, I don't think so:\n\n> There is no crossover between train and the private test set, so string matching won't be an effective strategy there. \n\nAlso important to note:\n\n> There are a huge number of labels in test that are not mentioned in train at all.", "votes": null }, { "id": "1315227", "postDate": "05/19/2021 16:38:26", "content": "There are a huge number of labels in test that are not mentioned in train at all.
\n
In the excerpt below, I had interpreted crossover to refer to only samples/publications. For example, a large number of public test samples are NOT crossover from train is in contrast to a percentage of the public test samples ARE drawn from train. So then there is no crossover between train and the private test set means there are not any shared samples/publications between train and the private test set. The tricky part is so string matching won't be an effective strategy there. I don't know if this means that string matching of the given train labels won't be an effective strategy or if finding valid labels outside of the given train labels in the train samples/publications and string matching using those won't be an effective strategy. I've assumed it means the former, but I'm not sure now.
We'd also like to note that - as you've noticed - a percentage of the public test samples ARE drawn from train. This is deliberate. As mentioned in the dataset description, the training set is not exhaustively labeled. The public test samples that come from train have additional labels that are not included in the training CSV. It's our hope that these will serve as guides for the difficult task of labeling the private test set. A large number of public test samples are NOT crossover from train, so you will still need to solve the problem to get a high public score.\n\nThere is no crossover between train and the private test set, so string matching won't be an effective strategy there.\nFwiw, I think the attempt to create an alternative public LB metric by removing training labels is of limited utility. It still is derived from training data and I suspect most if not all of us are explicitly (or in some ways worse, implicitly) spending a lot of time trying to push our models to “pick up” more labels from the training data which we can peek into - it’s not “unseen data” by any real definition of the phrase.
", "rawMarkdown": "Fwiw, I think the attempt to create an alternative public LB metric by removing training labels is of limited utility. It still is derived from training data and I suspect most if not all of us are explicitly (or in some ways worse, implicitly) spending a lot of time trying to push our models to “pick up” more labels from the training data which we can peek into - it’s not “unseen data” by any real definition of the phrase.", "votes": null }, { "id": "1315280", "postDate": "05/19/2021 17:17:39", "content": "@leecming Does this mean that your interpretation leans toward the private test data not having labels that have been picked up from the training data?
", "rawMarkdown": "leecming Does this mean that your interpretation leans toward the private test data not having labels that have been picked up from the training data?", "votes": null }, { "id": "1315283", "postDate": "05/19/2021 17:24:18", "content": "Who knows… especially after the re-annotation exercise. That might actually have breached the “disjoint” guarantee that was true prior (in terms of labels).
\nI think there’s no real point in thinking too much about this or string matching. It’s human nature to try out stuff but I wish people hadn’t done string matching at the onset of the competition, it’s tainted the way we approach this problem when we should be focusing on model-based solutions.
", "rawMarkdown": "Who knows… especially after the re-annotation exercise. That might actually have breached the “disjoint” guarantee that was true prior (in terms of labels).\n\nI think there’s no real point in thinking too much about this or string matching. It’s human nature to try out stuff but I wish people hadn’t done string matching at the onset of the competition, it’s tainted the way we approach this problem when we should be focusing on model-based solutions.", "votes": null }, { "id": "1315372", "postDate": "05/19/2021 18:36:12", "content": "I was searching in the discussions and found this:
\n\n\nAs @philculliton has already pointed out, the private leaderboard does not contain any labels from the training set.
\n
Appart from that I also tend to think that unlabeled datasets in the training data occur in the private test set. There are a huge number of all sorts of datasets in the training data and I think it is difficult to find 8000 publications that didn't meantion any of these..
\nTo see how my models perform on truly unseen datasets, I excluded all predictions which can be found in the texts of the training data. I was surprised by the result: model1 0.000, model2 0.000 …
\nI think it is very unlikely that my models didn't pick up any of the unseen datasets (they do pick up a lot of datasets present but not labeled in the training data) - unless they aren't there or quite rare..
\nIt could however still be the case that there are many truly unseen datasets in the private test set..
", "rawMarkdown": "I was searching in the discussions and found this:\n\n> As @philculliton has already pointed out, the private leaderboard does not contain any labels from the training set.\n\nAppart from that I also tend to think that unlabeled datasets in the training data occur in the private test set. There are a huge number of all sorts of datasets in the training data and I think it is difficult to find 8000 publications that didn't meantion any of these..\n\nTo see how my models perform on truly unseen datasets, I excluded all predictions which can be found in the texts of the training data. I was surprised by the result: model1 0.000, model2 0.000 ...\n\nI think it is very unlikely that my models didn't pick up any of the unseen datasets (they do pick up a lot of datasets present but not labeled in the training data) - unless they aren't there or quite rare..\n\nIt could however still be the case that there are many truly unseen datasets in the private test set..", "votes": null }, { "id": "1315382", "postDate": "05/19/2021 18:44:29", "content": "@hannes82 Thank you for your answers, it may be true, but we may not be able to find a better way to determine the effect of our model, because the way we train the model may limit the model itself to find the inner connection of the mention. These are derived from the training data. Not fully labeled
", "rawMarkdown": "hannes82 Thank you for your answers, it may be true, but we may not be able to find a better way to determine the effect of our model, because the way we train the model may limit the model itself to find the inner connection of the mention. These are derived from the training data. Not fully labeled", "votes": null }, { "id": "1315549", "postDate": "05/19/2021 22:39:36", "content": "Another datapoint (anec-data?) which should induce concern among top public LB scorers is the public LB shake-up that took place during the re-scoring. It was very benign since a) it was additive (labels were only added; no deletions) and b) public LB auto-picks your best submission.
\nYet we saw a fairly big reshuffle within the gold and silver region which would suggest that people are overly tuning. My best submission post-rescoring was NOT the same as the best submission pre-scoring.
\nAll this suggests that we'll need luck in picking the right submissions for private. And I'm not sure how you can make a good decision beyond \"one conservative, one aggressive i.e. top scoring\".
", "rawMarkdown": "Another datapoint (anec-data?) which should induce concern among top public LB scorers is the public LB shake-up that took place during the re-scoring. It was very benign since a) it was additive (labels were only added; no deletions) and b) public LB auto-picks your best submission.\n\nYet we saw a fairly big reshuffle within the gold and silver region which would suggest that people are overly tuning. My best submission post-rescoring was NOT the same as the best submission pre-scoring. \n\nAll this suggests that we'll need luck in picking the right submissions for private. And I'm not sure how you can make a good decision beyond \"one conservative, one aggressive i.e. top scoring\".", "votes": null }, { "id": "1315551", "postDate": "05/19/2021 22:47:51", "content": "When I added the below datasets to the labels in \"is in\" file, my matching score increased over baseline. Meaning to me that datasets that are not labelled in the train set are at least in some of the test documents as public LB is based on a sample of hidden test documents.
\nI think Phil could have meant, literally, that none of the labelled datasets are in the hidden test, but there could many unlabeled datasets in the train that could be in the hidden test. IMO
\nMeaning maybe we have been exposed to many of the datasets in train that are in test, they are just not labelled for us. Doesn't really help us either way I guess. And as was pointed out, matching is not going to carry the day, and my model based system is stuck in the high .40's with a month to go.
\n(some are in the labels, but some are not - I just wanted to see what adding unlabelled, known datasets from the train did to my matching - this was before I understood matching would not win)
\nlocal food economic impact assessment
\ntvb model
\nmexican american study project
\nnational longitudinal study of adolescent health
\nnational health interview survey
\nproject talent longitudinal study
\nsea lake and overland surges from hurricanes slosh model
\nschools and staffing survey
\nprivate school universe survey
\nfast response survey system
\nnational center for education statistics schools and staffing survey
\nnational assessment of educational progress
\ncoinmon core of data
\nnational postsecondary student aid study
\npostsecondary student aid study
\nnational study of postsecondary faculty
\nsurvey on vocational programs in secondary schools
\ndistrict survey of alternative schools and programs
\nstate nonfiscal survey of public elementary secondary education
\nnational public education financial survey
\nannual survey of government finances school systems f 33 survey
\nintegrated postsecondary education data system
\nipeds
\nnsopf
\nbps data
\npostsecondary students longitudinal study
\npublic elementary secondary school universe survey
\nccd local education agency universe survey
\nccd national public education financial survey
\nnces national education longitudinal study of 1988
\nagricultural resource management survey
\naez model
\nusda corn yield data
\nargonne national laboratory s greet
\ngreet model
\nargonne national laboratory s cclub model
\nnational education longitudinal survey
\neducation longitudinal study of 2002
\nprogress in international reading literacy study
\nusgs national water quality assessment
\nipeds fall enrollment dataset
\nfall enrollment dataset
\ndelta cost project
\nannual survey of colleges standard research compilation
\nchesapeake bay watershed land cover data series
\nnccpi corn and soybeans sub model
\nnational longitudinal survey of youth
\nnational survey of teachers
did anyone beat the baseline with QA model or its just search in the wild!!!
", "rawMarkdown": "did anyone beat the baseline with QA model or its just search in the wild!!!", "votes": null }, { "id": "1318808", "postDate": "05/22/2021 15:27:37", "content": "in that case, it is really tough to decide which ones to select as best submissions
", "rawMarkdown": "in that case, it is really tough to decide which ones to select as best submissions", "votes": null }, { "id": "1336520", "postDate": "06/05/2021 02:28:10", "content": "update: I got 0.505 by QA model。
\nAt the same time, I look forward to finding other teammates to improve the ranking.
I am also looking for team to improve myself and to beat the baseline score.
", "rawMarkdown": "I am also looking for team to improve myself and to beat the baseline score.", "votes": null }, { "id": "1336644", "postDate": "06/05/2021 05:34:18", "content": "nice, have you tried ner model or some other models?
", "rawMarkdown": "nice, have you tried ner model or some other models?", "votes": null }, { "id": "1336835", "postDate": "06/05/2021 08:33:06", "content": "haa, i am working on single head QA and NER with CRF which performs little bit better
", "rawMarkdown": "haa, i am working on single head QA and NER with CRF which performs little bit better", "votes": null } ], "comments": [ { "id": 1302553, "author_name": "hannes82", "author_url": "", "post_date": "05/11/2021 15:37:44", "content": "@vanle73 Nice! May I ask you: Is the submission score with or without known training labels?
", "votes": null, "replies": [ { "id": 1302575, "author_name": "vanle73", "author_url": "", "post_date": "05/11/2021 15:50:53", "content": "Without any label match postprocessing!
", "votes": null, "replies": [] }, { "id": 1314609, "author_name": "udaygurugubelli", "author_url": "", "post_date": "05/19/2021 09:10:29", "content": "@hannes82 could you pls explain what do you mean by with out training labels? my doubt is if you remove how do you give start and end indices as labels??
\nyou mean to remove at the test time???
@udaygurugubelli yes excactly, you normally predict datasets at test time with your model and as a postprocessing procedure you remove all your predictions that match with a training label with a Jaccard score >=0.5 (see also this discussion).
\nIn this way, you ensure that you measure the performance of your model on unseen datasets (useful because in the private test set there are only unseen dataset labels).
", "votes": null, "replies": [] }, { "id": 1314735, "author_name": "vanle73", "author_url": "", "post_date": "05/19/2021 10:56:58", "content": "Thank you for your reply. But I found that once I did this operation, the result I got using the QA model changed from about 0.4 to 0.03, and there was almost no matching pair. After you perform this operation, has the public Lb increased? Your reply is the greatest help to me, thank you
", "votes": null, "replies": [] }, { "id": 1314766, "author_name": "udaygurugubelli", "author_url": "", "post_date": "05/19/2021 11:22:38", "content": "thanks for the reply. did you see that they are talking about rescoring some kind of post processing after the prediction.
", "votes": null, "replies": [] }, { "id": 1314768, "author_name": "udaygurugubelli", "author_url": "", "post_date": "05/19/2021 11:23:30", "content": "@vanle73 yes..true..mine also same..something needs to be fixed here
", "votes": null, "replies": [] }, { "id": 1314785, "author_name": "hannes82", "author_url": "", "post_date": "05/19/2021 11:34:03", "content": "That unfortunately means that your model is not generalizing well.
\nAfter this operation, the score is exptected to decrease because you are removing TP from your predictions. But the aim of the postprocessing operation is not to improve on the public Lb. It is to get a better measure on how well your model does on identifying onseen datasets.
", "votes": null, "replies": [] }, { "id": 1314874, "author_name": "udaygurugubelli", "author_url": "", "post_date": "05/19/2021 12:25:55", "content": "in that case there is hope to get the best then..lets c..
", "votes": null, "replies": [] }, { "id": 1315078, "author_name": "trentb", "author_url": "", "post_date": "05/19/2021 14:50:00", "content": "While there are unseen dataset labels in the private testing data, the dataset labels provided for the training data may also be in the private testing data. We don't really know how prevalent they may be though.
", "votes": null, "replies": [] }, { "id": 1315111, "author_name": "hannes82", "author_url": "", "post_date": "05/19/2021 15:11:56", "content": "No, I don't think so:
\n\n\nThere is no crossover between train and the private test set, so string matching won't be an effective strategy there.
\n
Also important to note:
\n\n", "votes": null, "replies": [] }, { "id": 1315227, "author_name": "trentb", "author_url": "", "post_date": "05/19/2021 16:38:26", "content": "There are a huge number of labels in test that are not mentioned in train at all.
\n
In the excerpt below, I had interpreted crossover to refer to only samples/publications. For example, a large number of public test samples are NOT crossover from train is in contrast to a percentage of the public test samples ARE drawn from train. So then there is no crossover between train and the private test set means there are not any shared samples/publications between train and the private test set. The tricky part is so string matching won't be an effective strategy there. I don't know if this means that string matching of the given train labels won't be an effective strategy or if finding valid labels outside of the given train labels in the train samples/publications and string matching using those won't be an effective strategy. I've assumed it means the former, but I'm not sure now.
We'd also like to note that - as you've noticed - a percentage of the public test samples ARE drawn from train. This is deliberate. As mentioned in the dataset description, the training set is not exhaustively labeled. The public test samples that come from train have additional labels that are not included in the training CSV. It's our hope that these will serve as guides for the difficult task of labeling the private test set. A large number of public test samples are NOT crossover from train, so you will still need to solve the problem to get a high public score.\n\nThere is no crossover between train and the private test set, so string matching won't be an effective strategy there.\nFwiw, I think the attempt to create an alternative public LB metric by removing training labels is of limited utility. It still is derived from training data and I suspect most if not all of us are explicitly (or in some ways worse, implicitly) spending a lot of time trying to push our models to “pick up” more labels from the training data which we can peek into - it’s not “unseen data” by any real definition of the phrase.
", "votes": null, "replies": [] }, { "id": 1315280, "author_name": "trentb", "author_url": "", "post_date": "05/19/2021 17:17:39", "content": "@leecming Does this mean that your interpretation leans toward the private test data not having labels that have been picked up from the training data?
", "votes": null, "replies": [] }, { "id": 1315283, "author_name": "leecming", "author_url": "", "post_date": "05/19/2021 17:24:18", "content": "Who knows… especially after the re-annotation exercise. That might actually have breached the “disjoint” guarantee that was true prior (in terms of labels).
\nI think there’s no real point in thinking too much about this or string matching. It’s human nature to try out stuff but I wish people hadn’t done string matching at the onset of the competition, it’s tainted the way we approach this problem when we should be focusing on model-based solutions.
", "votes": null, "replies": [] }, { "id": 1315372, "author_name": "hannes82", "author_url": "", "post_date": "05/19/2021 18:36:12", "content": "I was searching in the discussions and found this:
\n\n\nAs @philculliton has already pointed out, the private leaderboard does not contain any labels from the training set.
\n
Appart from that I also tend to think that unlabeled datasets in the training data occur in the private test set. There are a huge number of all sorts of datasets in the training data and I think it is difficult to find 8000 publications that didn't meantion any of these..
\nTo see how my models perform on truly unseen datasets, I excluded all predictions which can be found in the texts of the training data. I was surprised by the result: model1 0.000, model2 0.000 …
\nI think it is very unlikely that my models didn't pick up any of the unseen datasets (they do pick up a lot of datasets present but not labeled in the training data) - unless they aren't there or quite rare..
\nIt could however still be the case that there are many truly unseen datasets in the private test set..
", "votes": null, "replies": [] }, { "id": 1315382, "author_name": "vanle73", "author_url": "", "post_date": "05/19/2021 18:44:29", "content": "@hannes82 Thank you for your answers, it may be true, but we may not be able to find a better way to determine the effect of our model, because the way we train the model may limit the model itself to find the inner connection of the mention. These are derived from the training data. Not fully labeled
", "votes": null, "replies": [] }, { "id": 1315549, "author_name": "leecming", "author_url": "", "post_date": "05/19/2021 22:39:36", "content": "Another datapoint (anec-data?) which should induce concern among top public LB scorers is the public LB shake-up that took place during the re-scoring. It was very benign since a) it was additive (labels were only added; no deletions) and b) public LB auto-picks your best submission.
\nYet we saw a fairly big reshuffle within the gold and silver region which would suggest that people are overly tuning. My best submission post-rescoring was NOT the same as the best submission pre-scoring.
\nAll this suggests that we'll need luck in picking the right submissions for private. And I'm not sure how you can make a good decision beyond \"one conservative, one aggressive i.e. top scoring\".
", "votes": null, "replies": [] }, { "id": 1315551, "author_name": "mlconsult", "author_url": "", "post_date": "05/19/2021 22:47:51", "content": "When I added the below datasets to the labels in \"is in\" file, my matching score increased over baseline. Meaning to me that datasets that are not labelled in the train set are at least in some of the test documents as public LB is based on a sample of hidden test documents.
\nI think Phil could have meant, literally, that none of the labelled datasets are in the hidden test, but there could many unlabeled datasets in the train that could be in the hidden test. IMO
\nMeaning maybe we have been exposed to many of the datasets in train that are in test, they are just not labelled for us. Doesn't really help us either way I guess. And as was pointed out, matching is not going to carry the day, and my model based system is stuck in the high .40's with a month to go.
\n(some are in the labels, but some are not - I just wanted to see what adding unlabelled, known datasets from the train did to my matching - this was before I understood matching would not win)
\nlocal food economic impact assessment
\ntvb model
\nmexican american study project
\nnational longitudinal study of adolescent health
\nnational health interview survey
\nproject talent longitudinal study
\nsea lake and overland surges from hurricanes slosh model
\nschools and staffing survey
\nprivate school universe survey
\nfast response survey system
\nnational center for education statistics schools and staffing survey
\nnational assessment of educational progress
\ncoinmon core of data
\nnational postsecondary student aid study
\npostsecondary student aid study
\nnational study of postsecondary faculty
\nsurvey on vocational programs in secondary schools
\ndistrict survey of alternative schools and programs
\nstate nonfiscal survey of public elementary secondary education
\nnational public education financial survey
\nannual survey of government finances school systems f 33 survey
\nintegrated postsecondary education data system
\nipeds
\nnsopf
\nbps data
\npostsecondary students longitudinal study
\npublic elementary secondary school universe survey
\nccd local education agency universe survey
\nccd national public education financial survey
\nnces national education longitudinal study of 1988
\nagricultural resource management survey
\naez model
\nusda corn yield data
\nargonne national laboratory s greet
\ngreet model
\nargonne national laboratory s cclub model
\nnational education longitudinal survey
\neducation longitudinal study of 2002
\nprogress in international reading literacy study
\nusgs national water quality assessment
\nipeds fall enrollment dataset
\nfall enrollment dataset
\ndelta cost project
\nannual survey of colleges standard research compilation
\nchesapeake bay watershed land cover data series
\nnccpi corn and soybeans sub model
\nnational longitudinal survey of youth
\nnational survey of teachers
in that case, it is really tough to decide which ones to select as best submissions
", "votes": null, "replies": [] } ] }, { "id": 1312295, "author_name": "namgalielei", "author_url": "", "post_date": "05/18/2021 01:20:32", "content": "How to you design the questions ? Only a general question like \"What is the dataset used ?\". @vanle73
", "votes": null, "replies": [ { "id": 1314747, "author_name": "vanle73", "author_url": "", "post_date": "05/19/2021 11:10:59", "content": "Yes! The simplest way
", "votes": null, "replies": [] } ] }, { "id": 1316593, "author_name": "udaygurugubelli", "author_url": "", "post_date": "05/20/2021 17:42:11", "content": "did anyone beat the baseline with QA model or its just search in the wild!!!
", "votes": null, "replies": [] }, { "id": 1336520, "author_name": "vanle73", "author_url": "", "post_date": "06/05/2021 02:28:10", "content": "update: I got 0.505 by QA model。
\nAt the same time, I look forward to finding other teammates to improve the ranking.
I am also looking for team to improve myself and to beat the baseline score.
", "votes": null, "replies": [] }, { "id": 1336644, "author_name": "vanle73", "author_url": "", "post_date": "06/05/2021 05:34:18", "content": "nice, have you tried ner model or some other models?
", "votes": null, "replies": [] }, { "id": 1336835, "author_name": "udaygurugubelli", "author_url": "", "post_date": "06/05/2021 08:33:06", "content": "haa, i am working on single head QA and NER with CRF which performs little bit better
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1302509": "The current multi-answer QA model is used for prediction, and fine-tuned on bert base. The score of the verification set (20% of all data) is **0.96**, and the effect submitted on the public test set is **0.393**. Continue to update...\n\n--------\nupdate: singe QA model 0.425\n\n--------\nupdate:0.505", "1302553": "vanle73 Nice! May I ask you: Is the submission score with or without known training labels?", "1302575": "Without any label match postprocessing!", "1312295": "How to you design the questions ? Only a general question like \"What is the dataset used ?\". @vanle73", "1314609": "hannes82 could you pls explain what do you mean by with out training labels? my doubt is if you remove how do you give start and end indices as labels??\nyou mean to remove at the test time???", "1314725": "udaygurugubelli yes excactly, you normally predict datasets at test time with your model and as a postprocessing procedure you remove all your predictions that match with a training label with a Jaccard score >=0.5 (see also [this](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/232964) discussion).\n\nIn this way, you ensure that you measure the performance of your model on *unseen* datasets (useful because in the private test set there are only unseen dataset labels).", "1314735": "Thank you for your reply. But I found that once I did this operation, the result I got using the QA model changed from about 0.4 to 0.03, and there was almost no matching pair. After you perform this operation, has the public Lb increased? Your reply is the greatest help to me, thank you", "1314747": "Yes! The simplest way", "1314766": "thanks for the reply. did you see that they are talking about rescoring some kind of post processing after the prediction.", "1314768": "vanle73 yes..true..mine also same..something needs to be fixed here", "1314785": "That unfortunately means that your model is not generalizing well. \n\nAfter this operation, the score is exptected to decrease because you are removing TP from your predictions. But the aim of the postprocessing operation is not to improve on the public Lb. It is to get a better measure on how well your model does on identifying onseen datasets.", "1314874": "in that case there is hope to get the best then..lets c..", "1315078": "While there are unseen dataset labels in the private testing data, the dataset labels provided for the training data may also be in the private testing data. We don't really know how prevalent they may be though.", "1315111": "No, I don't think so:\n\n> There is no crossover between train and the private test set, so string matching won't be an effective strategy there. \n\nAlso important to note:\n\n> There are a huge number of labels in test that are not mentioned in train at all.", "1315227": "In the excerpt below, I had interpreted *crossover* to refer to only samples/publications. For example, `a large number of public test samples are NOT crossover from train` is in contrast to `a percentage of the public test samples ARE drawn from train`. So then `there is no crossover between train and the private test set` means there are not any shared samples/publications between train and the private test set. The tricky part is `so string matching won't be an effective strategy there`. I don't know if this means that string matching of the given train labels won't be an effective strategy or if finding valid labels outside of the given train labels in the train samples/publications and string matching using those won't be an effective strategy. I've assumed it means the former, but I'm not sure now.\n```\nWe'd also like to note that - as you've noticed - a percentage of the public test samples ARE drawn from train. This is deliberate. As mentioned in the dataset description, the training set is not exhaustively labeled. The public test samples that come from train have additional labels that are not included in the training CSV. It's our hope that these will serve as guides for the difficult task of labeling the private test set. A large number of public test samples are NOT crossover from train, so you will still need to solve the problem to get a high public score.\n\nThere is no crossover between train and the private test set, so string matching won't be an effective strategy there.\n```", "1315245": "Fwiw, I think the attempt to create an alternative public LB metric by removing training labels is of limited utility. It still is derived from training data and I suspect most if not all of us are explicitly (or in some ways worse, implicitly) spending a lot of time trying to push our models to “pick up” more labels from the training data which we can peek into - it’s not “unseen data” by any real definition of the phrase.", "1315280": "leecming Does this mean that your interpretation leans toward the private test data not having labels that have been picked up from the training data?", "1315283": "Who knows… especially after the re-annotation exercise. That might actually have breached the “disjoint” guarantee that was true prior (in terms of labels).\n\nI think there’s no real point in thinking too much about this or string matching. It’s human nature to try out stuff but I wish people hadn’t done string matching at the onset of the competition, it’s tainted the way we approach this problem when we should be focusing on model-based solutions.", "1315372": "I was searching in the discussions and found this:\n\n> As @philculliton has already pointed out, the private leaderboard does not contain any labels from the training set.\n\nAppart from that I also tend to think that unlabeled datasets in the training data occur in the private test set. There are a huge number of all sorts of datasets in the training data and I think it is difficult to find 8000 publications that didn't meantion any of these..\n\nTo see how my models perform on truly unseen datasets, I excluded all predictions which can be found in the texts of the training data. I was surprised by the result: model1 0.000, model2 0.000 ...\n\nI think it is very unlikely that my models didn't pick up any of the unseen datasets (they do pick up a lot of datasets present but not labeled in the training data) - unless they aren't there or quite rare..\n\nIt could however still be the case that there are many truly unseen datasets in the private test set..", "1315382": "hannes82 Thank you for your answers, it may be true, but we may not be able to find a better way to determine the effect of our model, because the way we train the model may limit the model itself to find the inner connection of the mention. These are derived from the training data. Not fully labeled", "1315549": "Another datapoint (anec-data?) which should induce concern among top public LB scorers is the public LB shake-up that took place during the re-scoring. It was very benign since a) it was additive (labels were only added; no deletions) and b) public LB auto-picks your best submission.\n\nYet we saw a fairly big reshuffle within the gold and silver region which would suggest that people are overly tuning. My best submission post-rescoring was NOT the same as the best submission pre-scoring. \n\nAll this suggests that we'll need luck in picking the right submissions for private. And I'm not sure how you can make a good decision beyond \"one conservative, one aggressive i.e. top scoring\".", "1315551": "When I added the below datasets to the labels in \"is in\" file, my matching score increased over baseline. Meaning to me that datasets that are not labelled in the train set are at least in some of the test documents as public LB is based on a sample of hidden test documents.\n\nI think Phil could have meant, literally, that none of the labelled datasets are in the hidden test, but there could many unlabeled datasets in the train that could be in the hidden test. IMO\n\nMeaning maybe we have been exposed to many of the datasets in train that are in test, they are just not labelled for us. Doesn't really help us either way I guess. And as was pointed out, matching is not going to carry the day, and my model based system is stuck in the high .40's with a month to go.\n\n(some are in the labels, but some are not - I just wanted to see what adding unlabelled, known datasets from the train did to my matching - this was before I understood matching would not win)\n\nlocal food economic impact assessment\ntvb model\nmexican american study project\nnational longitudinal study of adolescent health\nnational health interview survey\nproject talent longitudinal study\nsea lake and overland surges from hurricanes slosh model\nschools and staffing survey\nprivate school universe survey\nfast response survey system\nnational center for education statistics schools and staffing survey\nnational assessment of educational progress\ncoinmon core of data\nnational postsecondary student aid study\npostsecondary student aid study\nnational study of postsecondary faculty\nsurvey on vocational programs in secondary schools\ndistrict survey of alternative schools and programs\nstate nonfiscal survey of public elementary secondary education\nnational public education financial survey\nannual survey of government finances school systems f 33 survey\nintegrated postsecondary education data system\nipeds\nnsopf\nbps data\npostsecondary students longitudinal study\npublic elementary secondary school universe survey\nccd local education agency universe survey\nccd national public education financial survey\nnces national education longitudinal study of 1988\nagricultural resource management survey\naez model\nusda corn yield data\nargonne national laboratory s greet\ngreet model\nargonne national laboratory s cclub model\nnational education longitudinal survey\neducation longitudinal study of 2002\nprogress in international reading literacy study\nusgs national water quality assessment\nipeds fall enrollment dataset\nfall enrollment dataset\ndelta cost project\nannual survey of colleges standard research compilation\nchesapeake bay watershed land cover data series\nnccpi corn and soybeans sub model\nnational longitudinal survey of youth\nnational survey of teachers", "1316593": "did anyone beat the baseline with QA model or its just search in the wild!!!", "1318808": "in that case, it is really tough to decide which ones to select as best submissions", "1336520": "update: I got 0.505 by QA model。\nAt the same time, I look forward to finding other teammates to improve the ranking.", "1336560": "I am also looking for team to improve myself and to beat the baseline score.", "1336644": "nice, have you tried ner model or some other models?", "1336835": "haa, i am working on single head QA and NER with CRF which performs little bit better" }, "source": "meta" }