{ "id": 335294, "title": "Pretraining on \"Kaggle's Language\"", "url": "/competitions/AI4Code/discussion/335294", "author_name": "", "post_date": "2022-07-05T14:59:53.152958200Z", "votes": 62, "comment_count": 5, "views": 0, "content": "

Pretraining on \"Kaggle's Language\"

\n

CodeBERT is a good starting point for a code-based model.
\nBut CodeBERT isn't specifically pretrained to understand Kaggle notebooks. Just mainstream code.

\n

And we are not mainstream around here on Kaggle! 😎

\n
\n

\"Kaggle's slang\" terms - CodeBERT probably didn't train on them but they are all over the notebooks:

\n \n
\n

Needless to say, CodeBERT won't 100% the true meaning behind \"magic features\" or \"2nd stage FE\" or \"ensembling NNs\" out of the box.

\n

So the meaning of these terms is captured when fine-tuning.

\n

I've written a simple tutorial on pertaining language models because when I looked into this it was hard to find a simple self-contained code example for this.

\n
\n

You can also simply just want to use a different architecture that doesn't have a pretrained \"code\" version on huggingface.

\n
\n
\n

So you wanted to pre-train a language model, you know how it all works, it should be easy, probably there are a million implementations to do this since it all comes down to messing with some text a bit.

\n

So you went to huggingface's documentation and there was so much documentation about everything in the universe, and you ended out without really pre-training your model…

\n

If you are brave enough to try and run a pertaining session on the competition's data, this post is for you:

\n

πŸ‘‡ Here is how you pre-train a model πŸ‘‡

\n
from datasets import Dataset\nfrom transformers import AutoModel, AutoTokenizer, DataCollatorForLanguageModeling\n\ndataset = Dataset.from_pandas(train_df)\nmodel = AutoModel.from_pretrained(\"roberta-base\")\ntokenizer = AutoTokenizer.from_pretrained(\"roberta-base\")\n\ndata_collator = DataCollatorForLanguageModeling(\n    tokenizer=tokenizer, mlm=True, mlm_probability=0.15\n)\n\ntrainer = Trainer(\n    model=model,\n    args=TrainingArguments(\n        output_dir=\"./results\",\n        overwrite_output_dir=True,\n        num_train_epochs=1,\n        per_device_train_batch_size=16,\n        save_steps=10_000,\n        save_total_limit=2,\n    ),\n    data_collator=data_collator,\n    train_dataset=dataset,\n)\n\ntrainer.train()\n
\n

A simple code that gets a pandas data frame with a text column and pre-trains using a simple pretrained backbone on it.
\nFor this quick example, the training method is Masked Language Modeling (MLM) and we specify this by using the appropriate data collator class. [more on this later]

\n

But wait!

\n
How do I pre-train the BEST way?
\n
What are the best tricks?
\n

Some general pre-training tips:

\n\n

A summary of pertaining methods & Tricks

\n

Masked Language Modeling (MLM)

\n

\"\"

\n
\n

TL;DR: Mask some words with [MASK] and train the model to put the original words back where they belong

\n
\n

Example:

\n
I Love Machine Learning -> I [MASK] Machine Learning\n
\n

Permutation Language Modeling (PLM)

\n
\n

TL;DR: During training, randomly shuffle some of the tokens in the input text

\n
\n

Example:

\n
I Love Machine Learning -> I Learning [MASK] Love Machine\n
\n

Next Sentence Prediction (NSP)

\n
\n

TL;DR: Given 2 sentences, decide if the sentences are consecutive sentences from the same document.

\n
\n

Example:

\n
Sentence A: I would love to play piano -> Sentence B: I love to play football\n
\n

Whole Word Masking (WWM)

\n
\n

TL:DR: Instead of masking individual words, mask entire words.

\n
\n

Example:

\n
I Love Machine Learning -> I [WWM] Machine [WWM] Learning\n
\n

Replaced Token Detection (Electra)

\n
\n

TL;DR: Train a model to predict whether a particular token came from the input or was replaced during training via MLM

\n
\n

For Illustration:

\n
I Love Machine Learning -> I kinda like Machine Learning # replaced by a model to be \"hard\" \n
\n

Text Infilling

\n
\n

TL;DR: Given a chunk of text, replace a small contiguous portion of the text with a single [MASK] token and ask the model to put back the missing text.

\n
\n

Example:

\n
I [MASK] [MASK] Learning -> I Love Machine Learning \n
\n

Contrastive Learning

\n
\n

TL;DR: Use noise to create deceptive samples and force the model to recognize and distinguish between these deceptive samples and the real text

\n
\n

For Illustration:

\n
I Love Machine Learning -> I [MASK] achine Learning -> I Love Machine Learning.\n
\n

Other Pre-Training Objectives from famous papers

\n
RoBERTa: A Robustly Optimized BERT Pretraining Approach
\n

In the RoBERTa paper, the authors propose several improvements to the pre-training of BERT:

\n\n

In the original BERT implementation, when getting the objective for MLM, if we consider a batch of training data, that same batch is going to be used for every step (aside from the gradient updates). This means that the model will see the same sample multiple times and therefore, over-fit to that sample.

\n

RoBERTa proposes dynamic masking at each step: Each step will dynamically generate the mask and the model will no longer see the same sample multiple times.

\n\n

RoBERTa also removes the Next Sentence Prediction (NSP) objective that was included in the previous model. NSP is the task of deciding if two given sentences are consecutive sentences in the same document. Because this is an unsupervised task, the authors decided not to use it in their implementation.

\n\n

Rather than adding a single [CLS] token, a set of contiguous sentence pairs is fed into the network as follows:

\n
[CLS] The man went to the store. [SEP] He bought a gallon of milk. [SEP]\n
\n

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

\n

\"\"

\n

Unlike BERT, BART is not just pretrained on a simple Masked Language Model. BART does a few things differently:

\n\n
BART Denoising Strategies
\n

Denoising by Masking
\nIn the traditional BERT model, some of the tokens in the input sentence are masked and the task is to predict what belongs in the masked location[1]. This introduces noise and forces the Transformer language model to learn representations that are invariant to the specific tokens that make up the sentence. However, this is not always enough and the authors of BART demonstrate that they can improve performance by adding other denoising objectives to BART such as the following:

\n

Denoising by Deletion
\nSimilar to traditional BERT, tokens are randomly masked from the word. Instead of replacing these deleted tokens with [MASK], we delete these tokens from the input and try to have the BART decoder reconstruct what token is missing.

\n

Infilling
\nInstead of deleting individual tokens, deletions may be made such that contiguous spans of tokens are deleted and replaced with a single [MASK] token.

\n

Document Rotation
\nThis strategy simply reorders the words in the input sentence. In other words, we randomly select a word in the sentence and rotate the sentence such that the chosen word is now the first word in the input.

\n

TextFuse
\nIn a similar vein to Document Rotation, this strategy chooses a word in the input sentence to use as context and the model must now decode the rest of the sentence using this word as the context. In other words, the input sequence is formatted as [CONTEXT, [MASK], [MASK], [MASK]….].

\n

SpanBERT: Improving Pre-training by Representing and Predicting Spans

\n

\"\"

\n

SpanBERT Model
\nSpanBERT takes both the unsupervised pre-training objectives of BERT and modified pre-training objectives to achieve state-of-the-art performance on various NLP tasks.

\n

How is SpanBERT's pretraining different?

\n\n

\"TAKING NOTES ON THE FLY HELPS LANGUAGE PRE-TRAINING (TNF)

\n

This paper points out that BERT training is, in part, so expensive because it has to learn good representations of extremely rare words.

\n

Solution: If a rare word is detected, one simply finds the historical context vector in the rare word lookup table and adds it to the final input embedding. e.g. (word_embedding + position_embedding + TNF_embedding).

\n

ELECTRA: PRE-TRAINING TEXT ENCODERS AS DISCRIMINATORS RATHER THAN GENERATORS

\n

\"\"

\n

The key idea here is to train using replaced token detection.

\n

Replaced Token Detection

\n

Using two transformer models:

\n\n

We first mask a few of the input tokens as shown above and the generator outputs predicted tokens. Next, the discriminator (which is the key innovation in Electra) has to decide which inputs are real and which are synthetic.

\n

References

\n", "messages": [ { "id": "1844453", "postDate": "07/05/2022 14:59:53", "content": "

Pretraining on \"Kaggle's Language\"

\n

CodeBERT is a good starting point for a code-based model.
\nBut CodeBERT isn't specifically pretrained to understand Kaggle notebooks. Just mainstream code.

\n

And we are not mainstream around here on Kaggle! 😎

\n
\n

\"Kaggle's slang\" terms - CodeBERT probably didn't train on them but they are all over the notebooks:

\n \n
\n

Needless to say, CodeBERT won't 100% the true meaning behind \"magic features\" or \"2nd stage FE\" or \"ensembling NNs\" out of the box.

\n

So the meaning of these terms is captured when fine-tuning.

\n

I've written a simple tutorial on pertaining language models because when I looked into this it was hard to find a simple self-contained code example for this.

\n
\n

You can also simply just want to use a different architecture that doesn't have a pretrained \"code\" version on huggingface.

\n
\n
\n

So you wanted to pre-train a language model, you know how it all works, it should be easy, probably there are a million implementations to do this since it all comes down to messing with some text a bit.

\n

So you went to huggingface's documentation and there was so much documentation about everything in the universe, and you ended out without really pre-training your model…

\n

If you are brave enough to try and run a pertaining session on the competition's data, this post is for you:

\n

πŸ‘‡ Here is how you pre-train a model πŸ‘‡

\n
from datasets import Dataset\nfrom transformers import AutoModel, AutoTokenizer, DataCollatorForLanguageModeling\n\ndataset = Dataset.from_pandas(train_df)\nmodel = AutoModel.from_pretrained(\"roberta-base\")\ntokenizer = AutoTokenizer.from_pretrained(\"roberta-base\")\n\ndata_collator = DataCollatorForLanguageModeling(\n    tokenizer=tokenizer, mlm=True, mlm_probability=0.15\n)\n\ntrainer = Trainer(\n    model=model,\n    args=TrainingArguments(\n        output_dir=\"./results\",\n        overwrite_output_dir=True,\n        num_train_epochs=1,\n        per_device_train_batch_size=16,\n        save_steps=10_000,\n        save_total_limit=2,\n    ),\n    data_collator=data_collator,\n    train_dataset=dataset,\n)\n\ntrainer.train()\n
\n

A simple code that gets a pandas data frame with a text column and pre-trains using a simple pretrained backbone on it.
\nFor this quick example, the training method is Masked Language Modeling (MLM) and we specify this by using the appropriate data collator class. [more on this later]

\n

But wait!

\n
How do I pre-train the BEST way?
\n
What are the best tricks?
\n

Some general pre-training tips:

\n\n

A summary of pertaining methods & Tricks

\n

Masked Language Modeling (MLM)

\n

\"\"

\n
\n

TL;DR: Mask some words with [MASK] and train the model to put the original words back where they belong

\n
\n

Example:

\n
I Love Machine Learning -> I [MASK] Machine Learning\n
\n

Permutation Language Modeling (PLM)

\n
\n

TL;DR: During training, randomly shuffle some of the tokens in the input text

\n
\n

Example:

\n
I Love Machine Learning -> I Learning [MASK] Love Machine\n
\n

Next Sentence Prediction (NSP)

\n
\n

TL;DR: Given 2 sentences, decide if the sentences are consecutive sentences from the same document.

\n
\n

Example:

\n
Sentence A: I would love to play piano -> Sentence B: I love to play football\n
\n

Whole Word Masking (WWM)

\n
\n

TL:DR: Instead of masking individual words, mask entire words.

\n
\n

Example:

\n
I Love Machine Learning -> I [WWM] Machine [WWM] Learning\n
\n

Replaced Token Detection (Electra)

\n
\n

TL;DR: Train a model to predict whether a particular token came from the input or was replaced during training via MLM

\n
\n

For Illustration:

\n
I Love Machine Learning -> I kinda like Machine Learning # replaced by a model to be \"hard\" \n
\n

Text Infilling

\n
\n

TL;DR: Given a chunk of text, replace a small contiguous portion of the text with a single [MASK] token and ask the model to put back the missing text.

\n
\n

Example:

\n
I [MASK] [MASK] Learning -> I Love Machine Learning \n
\n

Contrastive Learning

\n
\n

TL;DR: Use noise to create deceptive samples and force the model to recognize and distinguish between these deceptive samples and the real text

\n
\n

For Illustration:

\n
I Love Machine Learning -> I [MASK] achine Learning -> I Love Machine Learning.\n
\n

Other Pre-Training Objectives from famous papers

\n
RoBERTa: A Robustly Optimized BERT Pretraining Approach
\n

In the RoBERTa paper, the authors propose several improvements to the pre-training of BERT:

\n\n

In the original BERT implementation, when getting the objective for MLM, if we consider a batch of training data, that same batch is going to be used for every step (aside from the gradient updates). This means that the model will see the same sample multiple times and therefore, over-fit to that sample.

\n

RoBERTa proposes dynamic masking at each step: Each step will dynamically generate the mask and the model will no longer see the same sample multiple times.

\n\n

RoBERTa also removes the Next Sentence Prediction (NSP) objective that was included in the previous model. NSP is the task of deciding if two given sentences are consecutive sentences in the same document. Because this is an unsupervised task, the authors decided not to use it in their implementation.

\n\n

Rather than adding a single [CLS] token, a set of contiguous sentence pairs is fed into the network as follows:

\n
[CLS] The man went to the store. [SEP] He bought a gallon of milk. [SEP]\n
\n

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

\n

\"\"

\n

Unlike BERT, BART is not just pretrained on a simple Masked Language Model. BART does a few things differently:

\n\n
BART Denoising Strategies
\n

Denoising by Masking
\nIn the traditional BERT model, some of the tokens in the input sentence are masked and the task is to predict what belongs in the masked location[1]. This introduces noise and forces the Transformer language model to learn representations that are invariant to the specific tokens that make up the sentence. However, this is not always enough and the authors of BART demonstrate that they can improve performance by adding other denoising objectives to BART such as the following:

\n

Denoising by Deletion
\nSimilar to traditional BERT, tokens are randomly masked from the word. Instead of replacing these deleted tokens with [MASK], we delete these tokens from the input and try to have the BART decoder reconstruct what token is missing.

\n

Infilling
\nInstead of deleting individual tokens, deletions may be made such that contiguous spans of tokens are deleted and replaced with a single [MASK] token.

\n

Document Rotation
\nThis strategy simply reorders the words in the input sentence. In other words, we randomly select a word in the sentence and rotate the sentence such that the chosen word is now the first word in the input.

\n

TextFuse
\nIn a similar vein to Document Rotation, this strategy chooses a word in the input sentence to use as context and the model must now decode the rest of the sentence using this word as the context. In other words, the input sequence is formatted as [CONTEXT, [MASK], [MASK], [MASK]….].

\n

SpanBERT: Improving Pre-training by Representing and Predicting Spans

\n

\"\"

\n

SpanBERT Model
\nSpanBERT takes both the unsupervised pre-training objectives of BERT and modified pre-training objectives to achieve state-of-the-art performance on various NLP tasks.

\n

How is SpanBERT's pretraining different?

\n\n

\"TAKING NOTES ON THE FLY HELPS LANGUAGE PRE-TRAINING (TNF)

\n

This paper points out that BERT training is, in part, so expensive because it has to learn good representations of extremely rare words.

\n

Solution: If a rare word is detected, one simply finds the historical context vector in the rare word lookup table and adds it to the final input embedding. e.g. (word_embedding + position_embedding + TNF_embedding).

\n

ELECTRA: PRE-TRAINING TEXT ENCODERS AS DISCRIMINATORS RATHER THAN GENERATORS

\n

\"\"

\n

The key idea here is to train using replaced token detection.

\n

Replaced Token Detection

\n

Using two transformer models:

\n\n

We first mask a few of the input tokens as shown above and the generator outputs predicted tokens. Next, the discriminator (which is the key innovation in Electra) has to decide which inputs are real and which are synthetic.

\n

References

\n", "rawMarkdown": "## Pretraining on \"Kaggle's Language\"\n\n`CodeBERT` is a good starting point for a code-based model.\nBut `CodeBERT` isn't specifically pretrained to understand Kaggle notebooks. Just mainstream code.\n\nAnd we are not mainstream around here on Kaggle! 😎\n\n> **\"Kaggle's slang\" terms - CodeBERT probably didn't train on them but they are all over the notebooks:**\n- \"shakeup\"\n- \"blending\" / \"to blend\" / \"blend of blend\"\n- \"magic features\"\n- \"CV\" / \"CV score\" / \"best CV score\" / \"CV split\" (variations of the term are somewhat mainstream)\n- \"LB\" / \"public LB\" / \"private LB\" / \"LB probing\" / also just \"public\", \"private\"\n-\"CV-LB gap\"\n- \"OOF\" / \"out of fold\" / \"oof\" / \"oofs\" / \"oof of oof\" / \"y_oof\" / so many variations..\n- \"ensembling\" / \"ensembled\" / \"to ensemble\" (as a verb)\n- \"meta-features\" / \"meta-model\"\n- \"kernel\"\n- \"NN\" / \"NN model\" / \"NN-based\"\n- \"FE\"\n- \"Oliver's feature selection\"\n- \"single model\" / \"single model score\" (that is made of 5 folds)\n- \"stacking\" / \"stacked\" / \"to stack\" (as a verb when stacking) / \"self stacking\"\n- \"Grandmaster\" (all ranks but mostly \"Grandmaster\". you won't find this on random github repos)\n- \"gold zone\"\n- \"submission.csv\" has an important meaning when it comes to kaggle. also \"sample_submission.csv\" and \"../input\"\n- \"id features\" / \"leak features\"\n- \"2nd stage model\" / \"2nd level\" / \"1st stage\"\n- \"Upvote if you find this useful\" / \"Upvote if you like it\" / anything \"upvote\" gets a different meaning on kaggle.\n- \"comp\"\n- \"TTA\" (nearly exclusive to kaggle)\n- \"AE\" \n- \"target encoding\" (nearly exclusive to kaggle)\n- \"PS\" / \"pseudo-labeling\" (nearly exclusive to kaggle)\n- \"EDA\" (already mainstream)\n- \"πŸ”₯\" - probably the most frequent emoji on kaggle notebooks.\n(And these are only the ones I remember..)\n\n\nNeedless to say, `CodeBERT` won't 100% the true meaning behind \"magic features\" or \"2nd stage FE\" or \"ensembling NNs\" out of the box. \n\nSo the meaning of these terms is captured when fine-tuning. \n\nI've written a simple tutorial on pertaining language models because when I looked into this it was hard to find a simple self-contained code example for this. \n\n> You can also simply just want to use a different architecture that doesn't have a pretrained \"code\" version on huggingface. \n\n_____\n\n**So you wanted to pre-train a language model,** you know how it all works, it should be easy, probably there are a million implementations to do this since it all comes down to messing with some text a bit.\n\nSo you went to [huggingface's documentation](https://huggingface.co/docs) and **there was so much documentation** about **everything in the universe**, and you ended out without really pre-training your model...\n\nIf you are brave enough to try and run a pertaining session on the competition's data, this post is for you:\n\n## πŸ‘‡ Here is how you pre-train a model πŸ‘‡\n\n```python\nfrom datasets import Dataset\nfrom transformers import AutoModel, AutoTokenizer, DataCollatorForLanguageModeling\n\ndataset = Dataset.from_pandas(train_df)\nmodel = AutoModel.from_pretrained(\"roberta-base\")\ntokenizer = AutoTokenizer.from_pretrained(\"roberta-base\")\n\ndata_collator = DataCollatorForLanguageModeling(\n tokenizer=tokenizer, mlm=True, mlm_probability=0.15\n)\n\ntrainer = Trainer(\n model=model,\n args=TrainingArguments(\n output_dir=\"./results\",\n overwrite_output_dir=True,\n num_train_epochs=1,\n per_device_train_batch_size=16,\n save_steps=10_000,\n save_total_limit=2,\n ),\n data_collator=data_collator,\n train_dataset=dataset,\n)\n\ntrainer.train()\n```\n\nA simple code that gets a pandas data frame with a text column and pre-trains using a simple pretrained backbone on it.\nFor this quick example, the training method is **Masked Language Modeling (MLM)** and we specify this by using the appropriate data collator class. [more on this later]\n\n\n#### But wait! \n\n###### How do I pre-train the BEST way? \n\n###### What are the best tricks?\n\n### Some general pre-training tips:\n\n- Pre-train for **as long as you can**\n- **More data** is better than more epochs\n- **Masking 15%** of the tokens seems to be the sweet spot\n- Use an **MLM probability schedule** (e.g. start with 0.15 and decrease to 0.05 over the course of training)\n- Don't forget to **use a warmup scheduler** (linear warmup is fine)\n- Always use an optimizer with weight decay (e.g. AdamW)\n- If you are doing a question-answering task, you may **want to use a different learning rate** for the final linear layer (e.g. 0.01)\n- When fine-tuning, always **use a small learning rate (e.g. 1e-5)**\n- Don't forget to **set the number of training epochs**\n- **If you are using a TPU**, use gradient accumulation (e.g. accumulate_grad_batches=8)\n- **If you are using a TPU**, you may want to set a different batch size for each step (e.g. batch_size_per_step=batch_size_per_device * gradient_accumulation_steps)\n- **For text classification**, use DataCollatorForClassification\n- **For question answering**, use DataCollatorForQuestionAnswering\n- **For sequence tagging**, use DataCollatorForPermutationLanguageModeling\n- **For next sentence prediction**, use DataCollatorForNextSentencePrediction\n- **For language modeling**, use DataCollatorForLanguageModeling\n- **For permutation language modeling** use DataCollatorForPermutationLanguageModeling\n\n#### A summary of pertaining methods & Tricks\n\n### Masked Language Modeling (MLM)\n\n![](https://i.ibb.co/c2kfsT4/bert.png)\n\n> **TL;DR:** Mask some words with [MASK] and train the model to put the original words back where they belong\n\n**Example:**\n```python\nI Love Machine Learning -> I [MASK] Machine Learning\n```\n\n### Permutation Language Modeling (PLM)\n\n> **TL;DR:** During training, randomly shuffle some of the tokens in the input text\n\n**Example:**\n```python\nI Love Machine Learning -> I Learning [MASK] Love Machine\n```\n\n\n### Next Sentence Prediction (NSP)\n\n> **TL;DR:** Given 2 sentences, decide if the sentences are consecutive sentences from the same document.\n\n**Example:**\n```python\nSentence A: I would love to play piano -> Sentence B: I love to play football\n```\n\n\n### Whole Word Masking (WWM)\n\n> **TL:DR:** Instead of masking individual words, mask entire words.\n\n**Example:**\n```python\nI Love Machine Learning -> I [WWM] Machine [WWM] Learning\n```\n\n\n### Replaced Token Detection (Electra)\n\n> **TL;DR:** Train a model to predict whether a particular token came from the input or was replaced during training via MLM\n\n**For Illustration:**\n```python\nI Love Machine Learning -> I kinda like Machine Learning # replaced by a model to be \"hard\" \n```\n\n### Text Infilling\n\n> **TL;DR:** Given a chunk of text, replace a small contiguous portion of the text with a single [MASK] token and ask the model to put back the missing text.\n\n**Example:**\n```python\nI [MASK] [MASK] Learning -> I Love Machine Learning \n```\n\n### Contrastive Learning\n\n> **TL;DR:** Use noise to create deceptive samples and force the model to recognize and distinguish between these deceptive samples and the real text\n\n**For Illustration:**\n```python\nI Love Machine Learning -> I [MASK] achine Learning -> I Love Machine Learning.\n```\n\n**Other Pre-Training Objectives from famous papers**\n\n##### RoBERTa: A Robustly Optimized BERT Pretraining Approach\n\nIn the RoBERTa paper, the authors propose several improvements to the pre-training of BERT:\n\n- **Dynamic Masking**\n- **Removing** the Next Sentence Prediction (NSP) objective\n- **Increasing** training from 100K steps to 350K steps\n\n- **Dynamic Masking**\n\nIn the original BERT implementation, when getting the objective for MLM, if we consider a batch of training data, that same batch is going to be used for every step (aside from the gradient updates). This means that the model will see the same sample multiple times and therefore, over-fit to that sample. \n\nRoBERTa proposes dynamic masking at each step: Each step will dynamically generate the mask and the model will no longer see the same sample multiple times.\n\n- **Removing the Next Sentence Prediction**\n\nRoBERTa also removes the Next Sentence Prediction (NSP) objective that was included in the previous model. NSP is the task of deciding if two given sentences are consecutive sentences in the same document. Because this is an unsupervised task, the authors decided not to use it in their implementation.\n\n- **Contiguous Sentence Pairs**\n\nRather than adding a single [CLS] token, a set of contiguous sentence pairs is fed into the network as follows:\n\n```python\n[CLS] The man went to the store. [SEP] He bought a gallon of milk. [SEP]\n```\n\n### BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension\n\n![](https://i.ibb.co/rdZg49h/bart.png)\n\nUnlike BERT, BART is not just pretrained on a simple Masked Language Model. BART does a few things differently:\n\n- BART **uses Denoising as a pretraining task** for the Transformer Encoder-Decoder model\n- BART **performs different corruption strategies** on the input data\n- BART **is trained to do Denoising from the start**\n\n\n##### BART Denoising Strategies\n\n**Denoising by Masking**\nIn the traditional BERT model, some of the tokens in the input sentence are masked and the task is to predict what belongs in the masked location[1]. This introduces noise and forces the Transformer language model to learn representations that are invariant to the specific tokens that make up the sentence. However, this is not always enough and the authors of BART demonstrate that they can improve performance by adding other denoising objectives to BART such as the following:\n\n**Denoising by Deletion**\nSimilar to traditional BERT, tokens are randomly masked from the word. Instead of replacing these deleted tokens with [MASK], we delete these tokens from the input and try to have the BART decoder reconstruct what token is missing.\n\n**Infilling**\nInstead of deleting individual tokens, deletions may be made such that contiguous spans of tokens are deleted and replaced with a single [MASK] token.\n\n**Document Rotation**\nThis strategy simply reorders the words in the input sentence. In other words, we randomly select a word in the sentence and rotate the sentence such that the chosen word is now the first word in the input.\n\n**TextFuse**\nIn a similar vein to Document Rotation, this strategy chooses a word in the input sentence to use as context and the model must now decode the rest of the sentence using this word as the context. In other words, the input sequence is formatted as [CONTEXT, [MASK], [MASK], [MASK]….].\n\n\n### SpanBERT: Improving Pre-training by Representing and Predicting Spans\n\n![](https://i.ibb.co/DpXRqjY/span-bart.png)\n\nSpanBERT Model\nSpanBERT takes both the unsupervised pre-training objectives of BERT and modified pre-training objectives to achieve state-of-the-art performance on various NLP tasks.\n\n**How is SpanBERT's pretraining different?**\n\n\n- **mask out contiguous spans** of text in the original sentence. In the graphic above, you can see a set of 4 consecutive tokens replaced with [MASK]. The model is tasked with predicting this missing information.\nThey introduce a Span Boundary Objective (SBO) that forces the model to use the representations of the tokens along the border of the masked region (x4 and x9 in the image above) to predict the missing tokens.\nFurther SpanBERT removes the NSP objective from BERT and only trains on single-sentence inputs.\n\n**\"TAKING NOTES ON THE FLY HELPS LANGUAGE PRE-TRAINING (TNF)**\n\nThis paper points out that BERT training is, in part, so expensive because it has to learn good representations of extremely rare words. \n\n**Solution:** If a rare word is detected, one simply finds the historical context vector in the rare word lookup table and adds it to the final input embedding. e.g. (word_embedding + position_embedding + TNF_embedding).\n\n### ELECTRA: PRE-TRAINING TEXT ENCODERS AS DISCRIMINATORS RATHER THAN GENERATORS\n\n![](https://i.ibb.co/T0k47X8/electra.png)\n\nThe key idea here is to train using replaced token detection.\n\n**Replaced Token Detection**\n\nUsing two transformer models:\n- Generator\n- Discriminator\n\nWe first mask a few of the input tokens as shown above and the generator outputs predicted tokens. Next, the discriminator (which is the key innovation in Electra) has to decide which inputs are real and which are synthetic.\n\n**References**\n\n- https://medium.com/analytics-vidhya/an-overview-of-the-various-bert-pre-training-methods-c365512342d8", "votes": null }, { "id": "1845013", "postDate": "07/06/2022 01:52:14", "content": "

Thank you @thedevastator for sharing a very detailed article.

", "rawMarkdown": "Thank you @thedevastator for sharing a very detailed article.", "votes": null }, { "id": "1845598", "postDate": "07/06/2022 12:59:13", "content": "

My pleasure, I am glad it helps!

", "rawMarkdown": "My pleasure, I am glad it helps!", "votes": null }, { "id": "1848856", "postDate": "07/09/2022 01:48:20", "content": "

thanks for sharing. why fine-tuned not pretrained? It costs too much resources.

", "rawMarkdown": "thanks for sharing. why fine-tuned not pretrained? It costs too much resources.", "votes": null }, { "id": "1852548", "postDate": "07/12/2022 06:25:21", "content": "

Thanks for your sharing, which is helpful for me to start this competition.

", "rawMarkdown": "Thanks for your sharing, which is helpful for me to start this competition.", "votes": null }, { "id": "1859839", "postDate": "07/18/2022 02:18:07", "content": "

WOW..this is gold! @thedevastator

", "rawMarkdown": "WOW..this is gold! @thedevastator", "votes": null } ], "comments": [ { "id": 1845013, "author_name": "muhammedtausif", "author_url": "", "post_date": "07/06/2022 01:52:14", "content": "

Thank you @thedevastator for sharing a very detailed article.

", "votes": null, "replies": [ { "id": 1845598, "author_name": "thedevastator", "author_url": "", "post_date": "07/06/2022 12:59:13", "content": "

My pleasure, I am glad it helps!

", "votes": null, "replies": [] } ] }, { "id": 1848856, "author_name": "dragonzhang", "author_url": "", "post_date": "07/09/2022 01:48:20", "content": "

thanks for sharing. why fine-tuned not pretrained? It costs too much resources.

", "votes": null, "replies": [] }, { "id": 1852548, "author_name": "gooooooood", "author_url": "", "post_date": "07/12/2022 06:25:21", "content": "

Thanks for your sharing, which is helpful for me to start this competition.

", "votes": null, "replies": [] }, { "id": 1859839, "author_name": "apurbapandey", "author_url": "", "post_date": "07/18/2022 02:18:07", "content": "

WOW..this is gold! @thedevastator

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1844453": "## Pretraining on \"Kaggle's Language\"\n\n`CodeBERT` is a good starting point for a code-based model.\nBut `CodeBERT` isn't specifically pretrained to understand Kaggle notebooks. Just mainstream code.\n\nAnd we are not mainstream around here on Kaggle! 😎\n\n> **\"Kaggle's slang\" terms - CodeBERT probably didn't train on them but they are all over the notebooks:**\n- \"shakeup\"\n- \"blending\" / \"to blend\" / \"blend of blend\"\n- \"magic features\"\n- \"CV\" / \"CV score\" / \"best CV score\" / \"CV split\" (variations of the term are somewhat mainstream)\n- \"LB\" / \"public LB\" / \"private LB\" / \"LB probing\" / also just \"public\", \"private\"\n-\"CV-LB gap\"\n- \"OOF\" / \"out of fold\" / \"oof\" / \"oofs\" / \"oof of oof\" / \"y_oof\" / so many variations..\n- \"ensembling\" / \"ensembled\" / \"to ensemble\" (as a verb)\n- \"meta-features\" / \"meta-model\"\n- \"kernel\"\n- \"NN\" / \"NN model\" / \"NN-based\"\n- \"FE\"\n- \"Oliver's feature selection\"\n- \"single model\" / \"single model score\" (that is made of 5 folds)\n- \"stacking\" / \"stacked\" / \"to stack\" (as a verb when stacking) / \"self stacking\"\n- \"Grandmaster\" (all ranks but mostly \"Grandmaster\". you won't find this on random github repos)\n- \"gold zone\"\n- \"submission.csv\" has an important meaning when it comes to kaggle. also \"sample_submission.csv\" and \"../input\"\n- \"id features\" / \"leak features\"\n- \"2nd stage model\" / \"2nd level\" / \"1st stage\"\n- \"Upvote if you find this useful\" / \"Upvote if you like it\" / anything \"upvote\" gets a different meaning on kaggle.\n- \"comp\"\n- \"TTA\" (nearly exclusive to kaggle)\n- \"AE\" \n- \"target encoding\" (nearly exclusive to kaggle)\n- \"PS\" / \"pseudo-labeling\" (nearly exclusive to kaggle)\n- \"EDA\" (already mainstream)\n- \"πŸ”₯\" - probably the most frequent emoji on kaggle notebooks.\n(And these are only the ones I remember..)\n\n\nNeedless to say, `CodeBERT` won't 100% the true meaning behind \"magic features\" or \"2nd stage FE\" or \"ensembling NNs\" out of the box. \n\nSo the meaning of these terms is captured when fine-tuning. \n\nI've written a simple tutorial on pertaining language models because when I looked into this it was hard to find a simple self-contained code example for this. \n\n> You can also simply just want to use a different architecture that doesn't have a pretrained \"code\" version on huggingface. \n\n_____\n\n**So you wanted to pre-train a language model,** you know how it all works, it should be easy, probably there are a million implementations to do this since it all comes down to messing with some text a bit.\n\nSo you went to [huggingface's documentation](https://huggingface.co/docs) and **there was so much documentation** about **everything in the universe**, and you ended out without really pre-training your model...\n\nIf you are brave enough to try and run a pertaining session on the competition's data, this post is for you:\n\n## πŸ‘‡ Here is how you pre-train a model πŸ‘‡\n\n```python\nfrom datasets import Dataset\nfrom transformers import AutoModel, AutoTokenizer, DataCollatorForLanguageModeling\n\ndataset = Dataset.from_pandas(train_df)\nmodel = AutoModel.from_pretrained(\"roberta-base\")\ntokenizer = AutoTokenizer.from_pretrained(\"roberta-base\")\n\ndata_collator = DataCollatorForLanguageModeling(\n tokenizer=tokenizer, mlm=True, mlm_probability=0.15\n)\n\ntrainer = Trainer(\n model=model,\n args=TrainingArguments(\n output_dir=\"./results\",\n overwrite_output_dir=True,\n num_train_epochs=1,\n per_device_train_batch_size=16,\n save_steps=10_000,\n save_total_limit=2,\n ),\n data_collator=data_collator,\n train_dataset=dataset,\n)\n\ntrainer.train()\n```\n\nA simple code that gets a pandas data frame with a text column and pre-trains using a simple pretrained backbone on it.\nFor this quick example, the training method is **Masked Language Modeling (MLM)** and we specify this by using the appropriate data collator class. [more on this later]\n\n\n#### But wait! \n\n###### How do I pre-train the BEST way? \n\n###### What are the best tricks?\n\n### Some general pre-training tips:\n\n- Pre-train for **as long as you can**\n- **More data** is better than more epochs\n- **Masking 15%** of the tokens seems to be the sweet spot\n- Use an **MLM probability schedule** (e.g. start with 0.15 and decrease to 0.05 over the course of training)\n- Don't forget to **use a warmup scheduler** (linear warmup is fine)\n- Always use an optimizer with weight decay (e.g. AdamW)\n- If you are doing a question-answering task, you may **want to use a different learning rate** for the final linear layer (e.g. 0.01)\n- When fine-tuning, always **use a small learning rate (e.g. 1e-5)**\n- Don't forget to **set the number of training epochs**\n- **If you are using a TPU**, use gradient accumulation (e.g. accumulate_grad_batches=8)\n- **If you are using a TPU**, you may want to set a different batch size for each step (e.g. batch_size_per_step=batch_size_per_device * gradient_accumulation_steps)\n- **For text classification**, use DataCollatorForClassification\n- **For question answering**, use DataCollatorForQuestionAnswering\n- **For sequence tagging**, use DataCollatorForPermutationLanguageModeling\n- **For next sentence prediction**, use DataCollatorForNextSentencePrediction\n- **For language modeling**, use DataCollatorForLanguageModeling\n- **For permutation language modeling** use DataCollatorForPermutationLanguageModeling\n\n#### A summary of pertaining methods & Tricks\n\n### Masked Language Modeling (MLM)\n\n![](https://i.ibb.co/c2kfsT4/bert.png)\n\n> **TL;DR:** Mask some words with [MASK] and train the model to put the original words back where they belong\n\n**Example:**\n```python\nI Love Machine Learning -> I [MASK] Machine Learning\n```\n\n### Permutation Language Modeling (PLM)\n\n> **TL;DR:** During training, randomly shuffle some of the tokens in the input text\n\n**Example:**\n```python\nI Love Machine Learning -> I Learning [MASK] Love Machine\n```\n\n\n### Next Sentence Prediction (NSP)\n\n> **TL;DR:** Given 2 sentences, decide if the sentences are consecutive sentences from the same document.\n\n**Example:**\n```python\nSentence A: I would love to play piano -> Sentence B: I love to play football\n```\n\n\n### Whole Word Masking (WWM)\n\n> **TL:DR:** Instead of masking individual words, mask entire words.\n\n**Example:**\n```python\nI Love Machine Learning -> I [WWM] Machine [WWM] Learning\n```\n\n\n### Replaced Token Detection (Electra)\n\n> **TL;DR:** Train a model to predict whether a particular token came from the input or was replaced during training via MLM\n\n**For Illustration:**\n```python\nI Love Machine Learning -> I kinda like Machine Learning # replaced by a model to be \"hard\" \n```\n\n### Text Infilling\n\n> **TL;DR:** Given a chunk of text, replace a small contiguous portion of the text with a single [MASK] token and ask the model to put back the missing text.\n\n**Example:**\n```python\nI [MASK] [MASK] Learning -> I Love Machine Learning \n```\n\n### Contrastive Learning\n\n> **TL;DR:** Use noise to create deceptive samples and force the model to recognize and distinguish between these deceptive samples and the real text\n\n**For Illustration:**\n```python\nI Love Machine Learning -> I [MASK] achine Learning -> I Love Machine Learning.\n```\n\n**Other Pre-Training Objectives from famous papers**\n\n##### RoBERTa: A Robustly Optimized BERT Pretraining Approach\n\nIn the RoBERTa paper, the authors propose several improvements to the pre-training of BERT:\n\n- **Dynamic Masking**\n- **Removing** the Next Sentence Prediction (NSP) objective\n- **Increasing** training from 100K steps to 350K steps\n\n- **Dynamic Masking**\n\nIn the original BERT implementation, when getting the objective for MLM, if we consider a batch of training data, that same batch is going to be used for every step (aside from the gradient updates). This means that the model will see the same sample multiple times and therefore, over-fit to that sample. \n\nRoBERTa proposes dynamic masking at each step: Each step will dynamically generate the mask and the model will no longer see the same sample multiple times.\n\n- **Removing the Next Sentence Prediction**\n\nRoBERTa also removes the Next Sentence Prediction (NSP) objective that was included in the previous model. NSP is the task of deciding if two given sentences are consecutive sentences in the same document. Because this is an unsupervised task, the authors decided not to use it in their implementation.\n\n- **Contiguous Sentence Pairs**\n\nRather than adding a single [CLS] token, a set of contiguous sentence pairs is fed into the network as follows:\n\n```python\n[CLS] The man went to the store. [SEP] He bought a gallon of milk. [SEP]\n```\n\n### BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension\n\n![](https://i.ibb.co/rdZg49h/bart.png)\n\nUnlike BERT, BART is not just pretrained on a simple Masked Language Model. BART does a few things differently:\n\n- BART **uses Denoising as a pretraining task** for the Transformer Encoder-Decoder model\n- BART **performs different corruption strategies** on the input data\n- BART **is trained to do Denoising from the start**\n\n\n##### BART Denoising Strategies\n\n**Denoising by Masking**\nIn the traditional BERT model, some of the tokens in the input sentence are masked and the task is to predict what belongs in the masked location[1]. This introduces noise and forces the Transformer language model to learn representations that are invariant to the specific tokens that make up the sentence. However, this is not always enough and the authors of BART demonstrate that they can improve performance by adding other denoising objectives to BART such as the following:\n\n**Denoising by Deletion**\nSimilar to traditional BERT, tokens are randomly masked from the word. Instead of replacing these deleted tokens with [MASK], we delete these tokens from the input and try to have the BART decoder reconstruct what token is missing.\n\n**Infilling**\nInstead of deleting individual tokens, deletions may be made such that contiguous spans of tokens are deleted and replaced with a single [MASK] token.\n\n**Document Rotation**\nThis strategy simply reorders the words in the input sentence. In other words, we randomly select a word in the sentence and rotate the sentence such that the chosen word is now the first word in the input.\n\n**TextFuse**\nIn a similar vein to Document Rotation, this strategy chooses a word in the input sentence to use as context and the model must now decode the rest of the sentence using this word as the context. In other words, the input sequence is formatted as [CONTEXT, [MASK], [MASK], [MASK]….].\n\n\n### SpanBERT: Improving Pre-training by Representing and Predicting Spans\n\n![](https://i.ibb.co/DpXRqjY/span-bart.png)\n\nSpanBERT Model\nSpanBERT takes both the unsupervised pre-training objectives of BERT and modified pre-training objectives to achieve state-of-the-art performance on various NLP tasks.\n\n**How is SpanBERT's pretraining different?**\n\n\n- **mask out contiguous spans** of text in the original sentence. In the graphic above, you can see a set of 4 consecutive tokens replaced with [MASK]. The model is tasked with predicting this missing information.\nThey introduce a Span Boundary Objective (SBO) that forces the model to use the representations of the tokens along the border of the masked region (x4 and x9 in the image above) to predict the missing tokens.\nFurther SpanBERT removes the NSP objective from BERT and only trains on single-sentence inputs.\n\n**\"TAKING NOTES ON THE FLY HELPS LANGUAGE PRE-TRAINING (TNF)**\n\nThis paper points out that BERT training is, in part, so expensive because it has to learn good representations of extremely rare words. \n\n**Solution:** If a rare word is detected, one simply finds the historical context vector in the rare word lookup table and adds it to the final input embedding. e.g. (word_embedding + position_embedding + TNF_embedding).\n\n### ELECTRA: PRE-TRAINING TEXT ENCODERS AS DISCRIMINATORS RATHER THAN GENERATORS\n\n![](https://i.ibb.co/T0k47X8/electra.png)\n\nThe key idea here is to train using replaced token detection.\n\n**Replaced Token Detection**\n\nUsing two transformer models:\n- Generator\n- Discriminator\n\nWe first mask a few of the input tokens as shown above and the generator outputs predicted tokens. Next, the discriminator (which is the key innovation in Electra) has to decide which inputs are real and which are synthetic.\n\n**References**\n\n- https://medium.com/analytics-vidhya/an-overview-of-the-various-bert-pre-training-methods-c365512342d8", "1845013": "Thank you @thedevastator for sharing a very detailed article.", "1845598": "My pleasure, I am glad it helps!", "1848856": "thanks for sharing. why fine-tuned not pretrained? It costs too much resources.", "1852548": "Thanks for your sharing, which is helpful for me to start this competition.", "1859839": "WOW..this is gold! @thedevastator" }, "source": "meta" }