Hi all,
\nI am following the great work of @tungmphung 's shared notebook of training BERT on the masked Language Model task. It is really great work and with some minor changes gets upto 0.259 on theLB without using literal string matching - thanks Tung M Phung!
\nI'm trying to improve the training set and have a question about how to select +ve and -ve samples for a Masked Language model.
\nAt the moment this notebook: https://www.kaggle.com/tungmphung/coleridge-bert-masked-dataset-modeling does the following:
\nI can understand why negative samples are needed and I also think its important to select sentences that are similar to the positive samples (if all negative samples did not contain 'data' then the model might learn that that word alone is sufficient to predict a dataset mention for example).
\nBut currently this method is creating negative samples that DO contain dataset mentions just datasets that are not labelled in the train_df - see this example:
NEGATIVE SAMPLE:
\n*original sentence: *
\n\"ERS designed and implemented a National Household Food Acquisition and Purchase Survey FoodAPS to fill critical gaps that arose in the mid 2000s when the Agricultural Research Service discontinued its data collection on human food consumption and nutrition.
Entitity to be masked:
\n['National', 'Household', 'Food', 'Acquisition', 'and', 'Purchase', 'Survey', 'FoodAPS']
This is a dataset and now the model will treat this context as indicative of no dataset present.
\nFor me this shows the clever design of this competition- the training set is weakly labelled and there are many datasets in the texts that we a) need to predict and b)need to be careful of how we train our models.
\nCan anyone suggest how to better select training samples please?
", "messages": [ { "id": "1281834", "postDate": "04/23/2021 11:01:09", "content": "Hi all,
\nI am following the great work of @tungmphung 's shared notebook of training BERT on the masked Language Model task. It is really great work and with some minor changes gets upto 0.259 on theLB without using literal string matching - thanks Tung M Phung!
\nI'm trying to improve the training set and have a question about how to select +ve and -ve samples for a Masked Language model.
\nAt the moment this notebook: https://www.kaggle.com/tungmphung/coleridge-bert-masked-dataset-modeling does the following:
\nI can understand why negative samples are needed and I also think its important to select sentences that are similar to the positive samples (if all negative samples did not contain 'data' then the model might learn that that word alone is sufficient to predict a dataset mention for example).
\nBut currently this method is creating negative samples that DO contain dataset mentions just datasets that are not labelled in the train_df - see this example:
NEGATIVE SAMPLE:
\n*original sentence: *
\n\"ERS designed and implemented a National Household Food Acquisition and Purchase Survey FoodAPS to fill critical gaps that arose in the mid 2000s when the Agricultural Research Service discontinued its data collection on human food consumption and nutrition.
Entitity to be masked:
\n['National', 'Household', 'Food', 'Acquisition', 'and', 'Purchase', 'Survey', 'FoodAPS']
This is a dataset and now the model will treat this context as indicative of no dataset present.
\nFor me this shows the clever design of this competition- the training set is weakly labelled and there are many datasets in the texts that we a) need to predict and b)need to be careful of how we train our models.
\nCan anyone suggest how to better select training samples please?
", "rawMarkdown": "Hi all,\n\nI am following the great work of @tungmphung 's shared notebook of training BERT on the masked Language Model task. It is really great work and with some minor changes gets upto 0.259 on theLB without using literal string matching - thanks Tung M Phung!\n\nI'm trying to improve the training set and have a question about how to select +ve and -ve samples for a Masked Language model.\n\nAt the moment this notebook: [https://www.kaggle.com/tungmphung/coleridge-bert-masked-dataset-modeling](url) does the following:\n\n1. selects positive samples for the MLM to learn the context surrounding a dataset entity mention; these are selected from processed sentences that contain any of the train dataset's labels (the set of title, label and cleaned label commonly used)\n2. selects negative samples for the MLM to learn the context surrounding entities that look a bit like datasets but aren't eg Authors's names or \"United States\"; these are selected from the processed sentences that contain keywords like 'data', 'study' etc. It identifies candidate phrases that contain capitalised words and some connectors , are>1 word long and do not match the Label set well.\n\nI can understand why negative samples are needed and I also think its important to select sentences that are similar to the positive samples (if all negative samples did not contain 'data' then the model might learn that that word alone is sufficient to predict a dataset mention for example).\nBut currently this method is creating negative samples that DO contain dataset mentions just datasets that are not labelled in the train_df - see this example:\n\nNEGATIVE SAMPLE:\n*original sentence: *\n\"ERS designed and implemented a **National Household Food Acquisition and Purchase Survey FoodAPS** to fill critical gaps that arose in the mid 2000s when the Agricultural Research Service discontinued its data collection on human food consumption and nutrition.\n\n*Entitity to be masked:*\n['National', 'Household', 'Food', 'Acquisition', 'and', 'Purchase', 'Survey', 'FoodAPS']\n\nThis is a dataset and now the model will treat this context as indicative of no dataset present.\n\nFor me this shows the clever design of this competition- the training set is weakly labelled and there are many datasets in the texts that we a) need to predict and b)need to be careful of how we train our models.\n\n**Can anyone suggest how to better select training samples please?**", "votes": null }, { "id": "1282023", "postDate": "04/23/2021 14:13:56", "content": "I've had an idea which might at least make some improvement:
\nIt might take a while and for sure it won't generalise to any datasets that are not in the training set but are in the hidden test set but worth a try… : )
", "rawMarkdown": "I've had an idea which might at least make some improvement:\n1. follow the same procedure as above to get positive and negative samples\n2. then manually go through the negative sample entities to manually select those which are datasets -call these False Negative Entities (FNE)\n3. repeat the corpus creation but add FNE to the label set for creating the positive samples\n4. remove any negative samples that contain FNEs\n5. train the MLM on those.\n\nIt might take a while and for sure it won't generalise to any datasets that are not in the training set but are in the hidden test set but worth a try... : )", "votes": null }, { "id": "1286241", "postDate": "04/27/2021 17:52:24", "content": "Hi Mark, it's great that you feel my work helpful.
\nI think you are right that there are lots of datasets in the training data that are not marked as such. For example, we can just take sentences with a sequence of capitalized words followed by \"dataset\\\\b\", then most of the results seem to be real datasets, but the majority of them were labeled as negative.
The training data has high precision but low recall. Regarding the MLM approach, I think a potential improvement would be to reduce the number of false negatives in the MLM data. One possible way is to make some general rules to exclude likely-FN data points. These rules should be efficient enough to eliminate FNs while not too much affecting the TNs and biasing data. One such rule may follow the example above, that we don't take sequences of capitalized words as negative samples if they are followed by \"dataset\\\\b\".
I've had an idea which might at least make some improvement:
\nIt might take a while and for sure it won't generalise to any datasets that are not in the training set but are in the hidden test set but worth a try… : )
", "votes": null, "replies": [ { "id": 1286241, "author_name": "tungmphung", "author_url": "", "post_date": "04/27/2021 17:52:24", "content": "Hi Mark, it's great that you feel my work helpful.
\nI think you are right that there are lots of datasets in the training data that are not marked as such. For example, we can just take sentences with a sequence of capitalized words followed by \"dataset\\\\b\", then most of the results seem to be real datasets, but the majority of them were labeled as negative.
The training data has high precision but low recall. Regarding the MLM approach, I think a potential improvement would be to reduce the number of false negatives in the MLM data. One possible way is to make some general rules to exclude likely-FN data points. These rules should be efficient enough to eliminate FNs while not too much affecting the TNs and biasing data. One such rule may follow the example above, that we don't take sequences of capitalized words as negative samples if they are followed by \"dataset\\\\b\".