Our solution is composed of 6 parts below.
\nThe metric of this competition is F-score. Under this metric, assuming a current score is F, a newly detected label can improve the score when the expected value that the label is true positive is greater than 0.8F. Therefore, it is important to estimate the private test score in order to determine the detection threshold. For example, if F is 0.6, the best threshold is 0.48 and if F is 0.4, the best threshold is 0.32. For this reason, it is very important to know the number of the training-data labels in the private test data, because it affects the private test score strongly.
\nTo tackle this problem, we did LB probing. In this competition, the public test data contains duplicates of the train data. Therefore, we can create a submission only with true positive labels and with no false positive labels by applying true positive labels of the train data to their duplicates. Thus, by setting the number of true positive labels to a value related to the hidden test data, we can get information about the hidden test data from the submission score. Using this strategy, we got rough estimates of values below (actual codes are this, I shared in the competition period, and this).
From these results, we found that the public test score of train-data label string-matching is very high (0.530), but there are very few train-data labels in the test data (1,717). Therefore, at least 1,600 of the train-data labels in the test data might be in the public test data and very few might be in the private test data. Therefore, the public score of submission which discards the train-data labels from the prediction will correlate well with the private test score. The best submission can be obtained by finding the submission with the max score without train-data labels and adding train-data label string-matching to it.
\nBy this approach, we succussed to select the best-private-score submission out of our 201 submissions. We knew we can survive the big shake of the private test LB, this was a very very big advantage for our team.
Most datasets have acronyms (e.g., National Education Longitudinal Study → NELS). So, we did acronym detection to detect dataset-names that are not included in the train-data labels. The following procedure was used to extract them.
\nThe acronym itself is also detected as a dataset-name. String matching is performed on the dataset-name and the acronym. The acronym is detected as a label only when it and its long name appear more than a threshold number of times in the text.
\nBy this acronym detection, we get a score of 0.418 on the public LB and 0.436 on the private LB. Each threshold was chosen based on the public LB score.
To obtain more dataset-names, we performed a more aggressive acronym detection. We extract words that contain uppercase characters and no lowercase characters from the texts as acronym candidates. We search a chunk of words that is valid as the full name of the acronym candidate among the entire text (the actual code is this). This acronym detection does not improve the leaderboard score because it detects many false-positive labels. But we used the dataset-names detected by it for the NER model's training, which we describe in the later step.
\nWe used the external U.S. government’s dataset-names obtained from this notebook. To reduce false-positive, we apply some processing below.
\nThis approach improved the score of the public LB score from 0.418 to 0.424 and the private LB score from 0.436 to 0.486.
\nWe attempted to train a NER model as a solution to this competition, using BERT or RoBERTa with the train data or the Rich Context competition data as training data. However, NER models never outperformed rule-based approaches. We think this is because a large number of true-positive labels are intentionally excluded from the provided train data. Therefore, the train data is incomplete as training data for machine learning. However, we found NER can be used to cover the weakness of string-matching. That is, NER is useful for detecting dataset-name variations that cannot be detected by string-matching. For example, National Education Longitudinal Study is sometimes quoted as National Educational Longitudinal Survey.
\nWe used spacy library to train a NER model. We used the train-data label, the acronym-detection label, the acronym-detection-version-2 labels, and the external U.S. government label for training.
\nWe detect dataset-name candidates from the test data using the trained NER model. We calculated Jaccard scores between dataset-name candidates and any train-data labels or acronym-detection labels. We selected candidates with Jaccard scores greater than or equal to 0.5 as variations of the existing labels. This approach improved the score of the public LB score from 0.424 to 0.440 and the private LB score from 0.486 to 0.504.
Finally, we applied basic string-matching using the train-data labels. We also used the acronyms of the train-data labels for string-matching. This approach improved the score of the public LB score from 0.440 to 0.614 and the private LB score from 0.504 to 0.513.
\nKaggle_Coleridge_4th_Solution (main)
\ncoleridge_ner (sub).
\nPlease refer to README.md of the main repository to reproduce the training and the prediction.
Our solution is composed of 6 parts below.
\nThe metric of this competition is F-score. Under this metric, assuming a current score is F, a newly detected label can improve the score when the expected value that the label is true positive is greater than 0.8F. Therefore, it is important to estimate the private test score in order to determine the detection threshold. For example, if F is 0.6, the best threshold is 0.48 and if F is 0.4, the best threshold is 0.32. For this reason, it is very important to know the number of the training-data labels in the private test data, because it affects the private test score strongly.
\nTo tackle this problem, we did LB probing. In this competition, the public test data contains duplicates of the train data. Therefore, we can create a submission only with true positive labels and with no false positive labels by applying true positive labels of the train data to their duplicates. Thus, by setting the number of true positive labels to a value related to the hidden test data, we can get information about the hidden test data from the submission score. Using this strategy, we got rough estimates of values below (actual codes are this, I shared in the competition period, and this).
From these results, we found that the public test score of train-data label string-matching is very high (0.530), but there are very few train-data labels in the test data (1,717). Therefore, at least 1,600 of the train-data labels in the test data might be in the public test data and very few might be in the private test data. Therefore, the public score of submission which discards the train-data labels from the prediction will correlate well with the private test score. The best submission can be obtained by finding the submission with the max score without train-data labels and adding train-data label string-matching to it.
\nBy this approach, we succussed to select the best-private-score submission out of our 201 submissions. We knew we can survive the big shake of the private test LB, this was a very very big advantage for our team.
Most datasets have acronyms (e.g., National Education Longitudinal Study → NELS). So, we did acronym detection to detect dataset-names that are not included in the train-data labels. The following procedure was used to extract them.
\nThe acronym itself is also detected as a dataset-name. String matching is performed on the dataset-name and the acronym. The acronym is detected as a label only when it and its long name appear more than a threshold number of times in the text.
\nBy this acronym detection, we get a score of 0.418 on the public LB and 0.436 on the private LB. Each threshold was chosen based on the public LB score.
To obtain more dataset-names, we performed a more aggressive acronym detection. We extract words that contain uppercase characters and no lowercase characters from the texts as acronym candidates. We search a chunk of words that is valid as the full name of the acronym candidate among the entire text (the actual code is this). This acronym detection does not improve the leaderboard score because it detects many false-positive labels. But we used the dataset-names detected by it for the NER model's training, which we describe in the later step.
\nWe used the external U.S. government’s dataset-names obtained from this notebook. To reduce false-positive, we apply some processing below.
\nThis approach improved the score of the public LB score from 0.418 to 0.424 and the private LB score from 0.436 to 0.486.
\nWe attempted to train a NER model as a solution to this competition, using BERT or RoBERTa with the train data or the Rich Context competition data as training data. However, NER models never outperformed rule-based approaches. We think this is because a large number of true-positive labels are intentionally excluded from the provided train data. Therefore, the train data is incomplete as training data for machine learning. However, we found NER can be used to cover the weakness of string-matching. That is, NER is useful for detecting dataset-name variations that cannot be detected by string-matching. For example, National Education Longitudinal Study is sometimes quoted as National Educational Longitudinal Survey.
\nWe used spacy library to train a NER model. We used the train-data label, the acronym-detection label, the acronym-detection-version-2 labels, and the external U.S. government label for training.
\nWe detect dataset-name candidates from the test data using the trained NER model. We calculated Jaccard scores between dataset-name candidates and any train-data labels or acronym-detection labels. We selected candidates with Jaccard scores greater than or equal to 0.5 as variations of the existing labels. This approach improved the score of the public LB score from 0.424 to 0.440 and the private LB score from 0.486 to 0.504.
Finally, we applied basic string-matching using the train-data labels. We also used the acronyms of the train-data labels for string-matching. This approach improved the score of the public LB score from 0.440 to 0.614 and the private LB score from 0.504 to 0.513.
\nKaggle_Coleridge_4th_Solution (main)
\ncoleridge_ner (sub).
\nPlease refer to README.md of the main repository to reproduce the training and the prediction.