Hi Guys,
\nThere had been some discussions about what exactly can we do to improve our language models since we all pretty much use the same one.
\nI am cross-posting a post I've written a couple of weeks ago about tips and tricks for yielding better performance out of hugging face pretrained models
\nLet me know what you think!
\nEnjoy!
\nbatch size in transformersTransformers are sensitive to the size of the batch we set them with, this is because each transformer layer is computing weights across the expected hidden features of the batch size it receives then projecting the hidden features down to the specified parameters of the head it is feeding. Technically you can train with any batch size you want but you should also consider changing the learning rate (more on this late). Going under the default batch size might not be the optimal solution as you might lose some of the weight quality of the model. With that in mind, finding the right batch size or learning rate can be tricky since transformers take time to train.
Adding more layers can be beneficial as you can use them to learn more advanced features, but it can also temper the fine-tuning of your large pretrained model. This can easily be seen in many past winning solutions where the winner simply used a pretrained language model followed by a simple reduction (such as mean or \"deberta reduction\").
\nHowever, as it turns out: Many of those solutions could be even further improved by incorporating another training phase after the pretrained model had been trained! This can be done by freezing the pretrained model and adding dense layers after it. You will get a tiny uplift in the upmost leaderboard with that change though.
\nWeight freezing and unfreezing in PyTorch
\n# Weight freezing\nfor param in model.parameters():\n param.requires_grad = False # unfreeze weights in at all\n# Weight unfreezing\nfor param in model.parameters():\n param.requires_grad = True # unfreeze weights in at all\nWeight freezing and unfreezing in TensorFlow
\n# Weight freezing\nlayer.trainable = False\n# Weight unfreezing\nlayer.trainable = True\nLearning rates and learning rate schedulers affect transformers' training much more than they affect the normal ML models. Changing the learning rate can have a great impact on performance and also on training convergence. Remember that the global learning rate weight decay mostly affects the unfreeze layers weights where the fine-tuning requires only a small percentage of warmup. If a model converges slower or yields a lower score than it should, try experimenting by adjusting the learning rate - or the library you are using might not support different learning rates.
\nIn recent times, One Cycle Cosine schedule has shown to provide better results on multiple NLP tasks, you can use it this way:
\nOne Cycle Cosine scheduling in PyTorch
\nfrom torch.optim.lr_scheduler import CosineAnnealingLR\noptimizer = torch.optim.AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)\nscheduler = CosineAnnealingLR(optimizer, T_max=num_train_optimization_steps)\nnum_training_steps = num_train_optimization_steps / args.gradient_accumulation_steps\n# Update the scheduler\nscheduler.step()\n# step the learning rate scheduler here, you will want to step the learning rate scheduler only once per optimizer step nothing more nothing less. So in this case, it should be called before you expect the gradients to be applied.\nOne Cycle Cosine scheduling in TensorFlow
\noptimizer = tf.keras.optimizers.Adam(learning_rate)\nscheduler = tf.keras.optimizers.schedules.CosineDecay(learning_rate, decay_steps=num_training_steps)\nThere are several things you need to know if you seek to get the best performance out of Adam optimizer:
\nbeta1 and beta2 used in the Adam optimizer, choosing the best values depends on your task and the data. many new tasks can benefit from lower beta1 and higher beta2 while on established tasks they would perform the opposite. So again: Experiments will be your best friend here.max_grad_normclip_norm` value than other models to converge. batch size based learning rate schedulers as you can peak up the learning rates whenever you deem fit. There are several notable optimizers that are important to know about:
\nAdamW: This is an extension to the Adam algorithm that prevents exponential weights decay of the model's weights in the outer layers as well as encourages penalized hyper volume below the default weights.
Adafactor: It was designed to have low memory usage and is scalable. This optimizer can offer significant optimizer performances using multiple GPUs (see below).
Novograd: Basically another Adam-like optimizer but with very better properties. It is one of the optimizers used by to train the bert-large model.
Ranger: Ranger optimizer is a very interesting optimizer that has pretty good results in winning solutions it terms of performance optimization but it is obviously not very well known or supported so I leave that up to you to research.
Lamb: GPU optimized reusable Adam optimizer developed by the GLUE and QQP competition winner.
Lookahead: A popular optimizer you can use on top of other optimizers and it will provide you with some performance gains.
I think after this topic you should have a pretty good idea about finding the best batch size for the task and model you are using. Now I want to point out some more problems that might affect you during training.
\nYou can find below some tips and tricks extended from the transformers docs.
Max length
\nWhen selecting a value for the max_length (or max_position_embeddings) argument of a transformer model, please ensure that the chosen value is less than the straight maximum sequence length you can feed to that model without padding. Using a length beyond this value will result in an exception when attempting to save the model. This behavior occurs whether you supply the value or use the default one provided.
For the new \"longformer\" tokenizer (LongformerTokenizer), the value of max_length is not enforced, but instead tokens after the max_length are silently ignored for the shape calculation. For safety, you should always use the actual maximum length your data will contain as the value for max_length.
Padding
\nWhen you are batching a dataset, a Tensor might be returned which is an array of sequences that are of uneven lengths, where each sequence is contained within a matrix. If the batch vectors are padded in the dimension exceeding the maximum length of the sequence, these vectors can propagate the errors to all the remaining tokens of that sequence.
\nIn the TensorFlow framework, this scenario is solved by right-padding a sequence to fit the maximum sequence length; whereas, it is solved in the PyTorch framework by left-padding a sequence to fit the maximum sequence length. To manage this small difference, a simple check for the framework is written in the subsequent code. This way, a similar padded seq len sequence is passed over to the TensorFlow and PyTorch frameworks.
left_pad_target=True left_pad_source=False, you want to pad the smaller sequence on the right and the longer one on the left.Total sequence count
\nmax_seq param of the data_collator class only keeps max_seq sequences at once. This means that calling data_collator.encode_batch(my_sequence_of_sequences) more than max_seq times will not encode all sequences. To encode all sequences in that batch, you need to call max_seq times data_collator.encode_batch(next_sequences), where next_sequences is a split of my_sequence_of_sequences, each with at most max_seq sequences.
Gradient accumulation
\nGradient accumulation delays the gradient computation until all the mini batches in an epoch are processed through the model. This helps in faster training when the batch size is very less, that is, in the case of large models. The GPU calculates a few steps of the model with a large batch size at regular intervals of every parameter update, thereby avoiding an expensive gradient update every step. Gradient accumulation is activated if the gradient_accumulation_steps integer is used as a parameter in the training loop.
\nvocab size
\nvocab size in Trainer is the total number of retained tokens used for training the model. The tokenizers can be further extended to support frequent tokens of the new training set we are fine tuning. This sometimes leads to better results.
Mixed Precision Training
\nTraining deep neural networks can be computationally intensive. When performing these computations, there is an inherent tradeoff between speed and performance. Mixed precision is the combined use of different numerical precisions in a computational method. Mixed precision training offers significant computational speedup by performing operations in the half-precision format while storing minimal information in single-precision to retain as much information as possible in critical parts of the network.
There are several mixed-precision approaches, the most commonly used mixed processors functioning modes are the following:
\nSubtle Differences: With this mode, the weights are stored in low precision but some parts of the network are executed in single precision.
\nAlso note that mixed-precision training is achieved without a significant accuracy degradation compared to full precision training. This can be very handy in cases where resources like GPU power and especially time are limited (in cases where multiple models have to be trained), it can also work in conjunction with boosted learning rates and high batch size techniques to yield even greater improvements by using a small percentage of data to get a rough idea what can be a large-scale performance boost.
Mixed Precision training in Pytorch
\nMake sure to install apex for this to work and to check if your GPU supports mixed-precision training
\n!git clone https://github.com/NVIDIA/apex.git\nrun apex/setup.sh \nfrom apex import amp \nnet, optimizer = amp.initialize(net, optimizer, opt_level = 'O2', keep_batchnorm_fp32 = False, loss_scale = 'dynamic')```\nMixed-Precision training in Tensorflow
\ntf.config.optimizer.set_experimental_options({\"auto_mixed_precision\": True}) \nDistributed training saves time by distributing the data across multiple GPUs so if you have multiple GPUs and even multiple machines with GPUs it can save you a lot of time. The downside is that this is a very large topic with a lot of frameworks and approaches to choose from and thus I will leave the choice up to you, a sublte tip is that you should choose the easyest one to use and integrate as a framework but as a tip distributed training can sometimes improve/yield better performance on performance-bound models such as LAMB models for example.
\nNote that distributed training and multi-GPU works best with the more efficient represantions (optimalization level from FP16 to FP32) as well as with gradient accumulation.
\nMulti GPU in Pytorch
\nif torch.multiprocessing.spawn.supports_start_method(): mp.spawn(main_worker, nprocs = args.n_gpu, args = (args, device))\nelse: mp.spawn(main_worker, args = (args, device), nprocs = args.n_gpu) \nMulti GPU in Tensorflow
\nstrategy = tf.distribute.MirroredStrategy()\nWarm-UP
\nDo not overlook the warm-up period when training transformers, the first 5% or so of training - Is a very important period of training as it is the good start your model needs to stay forcefully on the rails. However, as you may know, warm-up does not have to be constant, in studies, we have witnessed increased benefits from using a gradually increased learning rate for a warm-up and sometimes even better performance using multiple learning rates for a warm-up.
\nLearning rate and batch size
\nSometimes having a low batch size with a high learning rate can provide good results so never rule this idea out from your strategy, sometimes the highest batch size might give you worse performance with unfit learning rates while being used with higher learning rates or learning rate scheduler. So try lowing the batch size and learning rate and give them multiple learning rates with things as it might produce better results.
\nFinally, I would like to remind you that there are many solutions to the same problem and each solution is a one piece of the bigger puzzle, it is up to you to put the puzzle together and figure out which pieces go together with the best :)
", "messages": [ { "id": "1847422", "postDate": "07/07/2022 23:49:36", "content": "Hi Guys,
\nThere had been some discussions about what exactly can we do to improve our language models since we all pretty much use the same one.
\nI am cross-posting a post I've written a couple of weeks ago about tips and tricks for yielding better performance out of hugging face pretrained models
\nLet me know what you think!
\nEnjoy!
\nbatch size in transformersTransformers are sensitive to the size of the batch we set them with, this is because each transformer layer is computing weights across the expected hidden features of the batch size it receives then projecting the hidden features down to the specified parameters of the head it is feeding. Technically you can train with any batch size you want but you should also consider changing the learning rate (more on this late). Going under the default batch size might not be the optimal solution as you might lose some of the weight quality of the model. With that in mind, finding the right batch size or learning rate can be tricky since transformers take time to train.
Adding more layers can be beneficial as you can use them to learn more advanced features, but it can also temper the fine-tuning of your large pretrained model. This can easily be seen in many past winning solutions where the winner simply used a pretrained language model followed by a simple reduction (such as mean or \"deberta reduction\").
\nHowever, as it turns out: Many of those solutions could be even further improved by incorporating another training phase after the pretrained model had been trained! This can be done by freezing the pretrained model and adding dense layers after it. You will get a tiny uplift in the upmost leaderboard with that change though.
\nWeight freezing and unfreezing in PyTorch
\n# Weight freezing\nfor param in model.parameters():\n param.requires_grad = False # unfreeze weights in at all\n# Weight unfreezing\nfor param in model.parameters():\n param.requires_grad = True # unfreeze weights in at all\nWeight freezing and unfreezing in TensorFlow
\n# Weight freezing\nlayer.trainable = False\n# Weight unfreezing\nlayer.trainable = True\nLearning rates and learning rate schedulers affect transformers' training much more than they affect the normal ML models. Changing the learning rate can have a great impact on performance and also on training convergence. Remember that the global learning rate weight decay mostly affects the unfreeze layers weights where the fine-tuning requires only a small percentage of warmup. If a model converges slower or yields a lower score than it should, try experimenting by adjusting the learning rate - or the library you are using might not support different learning rates.
\nIn recent times, One Cycle Cosine schedule has shown to provide better results on multiple NLP tasks, you can use it this way:
\nOne Cycle Cosine scheduling in PyTorch
\nfrom torch.optim.lr_scheduler import CosineAnnealingLR\noptimizer = torch.optim.AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)\nscheduler = CosineAnnealingLR(optimizer, T_max=num_train_optimization_steps)\nnum_training_steps = num_train_optimization_steps / args.gradient_accumulation_steps\n# Update the scheduler\nscheduler.step()\n# step the learning rate scheduler here, you will want to step the learning rate scheduler only once per optimizer step nothing more nothing less. So in this case, it should be called before you expect the gradients to be applied.\nOne Cycle Cosine scheduling in TensorFlow
\noptimizer = tf.keras.optimizers.Adam(learning_rate)\nscheduler = tf.keras.optimizers.schedules.CosineDecay(learning_rate, decay_steps=num_training_steps)\nThere are several things you need to know if you seek to get the best performance out of Adam optimizer:
\nbeta1 and beta2 used in the Adam optimizer, choosing the best values depends on your task and the data. many new tasks can benefit from lower beta1 and higher beta2 while on established tasks they would perform the opposite. So again: Experiments will be your best friend here.max_grad_normclip_norm` value than other models to converge. batch size based learning rate schedulers as you can peak up the learning rates whenever you deem fit. There are several notable optimizers that are important to know about:
\nAdamW: This is an extension to the Adam algorithm that prevents exponential weights decay of the model's weights in the outer layers as well as encourages penalized hyper volume below the default weights.
Adafactor: It was designed to have low memory usage and is scalable. This optimizer can offer significant optimizer performances using multiple GPUs (see below).
Novograd: Basically another Adam-like optimizer but with very better properties. It is one of the optimizers used by to train the bert-large model.
Ranger: Ranger optimizer is a very interesting optimizer that has pretty good results in winning solutions it terms of performance optimization but it is obviously not very well known or supported so I leave that up to you to research.
Lamb: GPU optimized reusable Adam optimizer developed by the GLUE and QQP competition winner.
Lookahead: A popular optimizer you can use on top of other optimizers and it will provide you with some performance gains.
I think after this topic you should have a pretty good idea about finding the best batch size for the task and model you are using. Now I want to point out some more problems that might affect you during training.
\nYou can find below some tips and tricks extended from the transformers docs.
Max length
\nWhen selecting a value for the max_length (or max_position_embeddings) argument of a transformer model, please ensure that the chosen value is less than the straight maximum sequence length you can feed to that model without padding. Using a length beyond this value will result in an exception when attempting to save the model. This behavior occurs whether you supply the value or use the default one provided.
For the new \"longformer\" tokenizer (LongformerTokenizer), the value of max_length is not enforced, but instead tokens after the max_length are silently ignored for the shape calculation. For safety, you should always use the actual maximum length your data will contain as the value for max_length.
Padding
\nWhen you are batching a dataset, a Tensor might be returned which is an array of sequences that are of uneven lengths, where each sequence is contained within a matrix. If the batch vectors are padded in the dimension exceeding the maximum length of the sequence, these vectors can propagate the errors to all the remaining tokens of that sequence.
\nIn the TensorFlow framework, this scenario is solved by right-padding a sequence to fit the maximum sequence length; whereas, it is solved in the PyTorch framework by left-padding a sequence to fit the maximum sequence length. To manage this small difference, a simple check for the framework is written in the subsequent code. This way, a similar padded seq len sequence is passed over to the TensorFlow and PyTorch frameworks.
left_pad_target=True left_pad_source=False, you want to pad the smaller sequence on the right and the longer one on the left.Total sequence count
\nmax_seq param of the data_collator class only keeps max_seq sequences at once. This means that calling data_collator.encode_batch(my_sequence_of_sequences) more than max_seq times will not encode all sequences. To encode all sequences in that batch, you need to call max_seq times data_collator.encode_batch(next_sequences), where next_sequences is a split of my_sequence_of_sequences, each with at most max_seq sequences.
Gradient accumulation
\nGradient accumulation delays the gradient computation until all the mini batches in an epoch are processed through the model. This helps in faster training when the batch size is very less, that is, in the case of large models. The GPU calculates a few steps of the model with a large batch size at regular intervals of every parameter update, thereby avoiding an expensive gradient update every step. Gradient accumulation is activated if the gradient_accumulation_steps integer is used as a parameter in the training loop.
\nvocab size
\nvocab size in Trainer is the total number of retained tokens used for training the model. The tokenizers can be further extended to support frequent tokens of the new training set we are fine tuning. This sometimes leads to better results.
Mixed Precision Training
\nTraining deep neural networks can be computationally intensive. When performing these computations, there is an inherent tradeoff between speed and performance. Mixed precision is the combined use of different numerical precisions in a computational method. Mixed precision training offers significant computational speedup by performing operations in the half-precision format while storing minimal information in single-precision to retain as much information as possible in critical parts of the network.
There are several mixed-precision approaches, the most commonly used mixed processors functioning modes are the following:
\nSubtle Differences: With this mode, the weights are stored in low precision but some parts of the network are executed in single precision.
\nAlso note that mixed-precision training is achieved without a significant accuracy degradation compared to full precision training. This can be very handy in cases where resources like GPU power and especially time are limited (in cases where multiple models have to be trained), it can also work in conjunction with boosted learning rates and high batch size techniques to yield even greater improvements by using a small percentage of data to get a rough idea what can be a large-scale performance boost.
Mixed Precision training in Pytorch
\nMake sure to install apex for this to work and to check if your GPU supports mixed-precision training
\n!git clone https://github.com/NVIDIA/apex.git\nrun apex/setup.sh \nfrom apex import amp \nnet, optimizer = amp.initialize(net, optimizer, opt_level = 'O2', keep_batchnorm_fp32 = False, loss_scale = 'dynamic')```\nMixed-Precision training in Tensorflow
\ntf.config.optimizer.set_experimental_options({\"auto_mixed_precision\": True}) \nDistributed training saves time by distributing the data across multiple GPUs so if you have multiple GPUs and even multiple machines with GPUs it can save you a lot of time. The downside is that this is a very large topic with a lot of frameworks and approaches to choose from and thus I will leave the choice up to you, a sublte tip is that you should choose the easyest one to use and integrate as a framework but as a tip distributed training can sometimes improve/yield better performance on performance-bound models such as LAMB models for example.
\nNote that distributed training and multi-GPU works best with the more efficient represantions (optimalization level from FP16 to FP32) as well as with gradient accumulation.
\nMulti GPU in Pytorch
\nif torch.multiprocessing.spawn.supports_start_method(): mp.spawn(main_worker, nprocs = args.n_gpu, args = (args, device))\nelse: mp.spawn(main_worker, args = (args, device), nprocs = args.n_gpu) \nMulti GPU in Tensorflow
\nstrategy = tf.distribute.MirroredStrategy()\nWarm-UP
\nDo not overlook the warm-up period when training transformers, the first 5% or so of training - Is a very important period of training as it is the good start your model needs to stay forcefully on the rails. However, as you may know, warm-up does not have to be constant, in studies, we have witnessed increased benefits from using a gradually increased learning rate for a warm-up and sometimes even better performance using multiple learning rates for a warm-up.
\nLearning rate and batch size
\nSometimes having a low batch size with a high learning rate can provide good results so never rule this idea out from your strategy, sometimes the highest batch size might give you worse performance with unfit learning rates while being used with higher learning rates or learning rate scheduler. So try lowing the batch size and learning rate and give them multiple learning rates with things as it might produce better results.
\nFinally, I would like to remind you that there are many solutions to the same problem and each solution is a one piece of the bigger puzzle, it is up to you to put the puzzle together and figure out which pieces go together with the best :)
", "rawMarkdown": "# Ideas for training better language models\n\nHi Guys,\n\nThere had been some discussions about what exactly can we do to improve our language models since we all pretty much use the same one. \n\nI am cross-posting a post I've written a couple of weeks ago about tips and tricks for yielding better performance out of hugging face pretrained models \n\nLet me know what you think! \n\nEnjoy! \n\n_____\n\n### Beware of the default `batch size` in transformers\n\nTransformers are sensitive to the size of the `batch` we set them with, this is because each transformer layer is computing weights across the expected hidden features of the batch size it receives then projecting the hidden features down to the specified parameters of the head it is feeding. Technically you can train with any batch size you want but you should also consider changing the learning rate (more on this late). Going under the default `batch size` might not be the optimal solution as you might lose some of the weight quality of the model. With that in mind, finding the right batch size or learning rate can be tricky since transformers take time to train.\n\n### About hidden layers after the pretrained model\n\nAdding more layers can be beneficial as you can use them to learn more advanced features, but it can also temper the fine-tuning of your large pretrained model. This can easily be seen in many past winning solutions where the winner simply used a pretrained language model followed by a simple reduction (such as mean or \"deberta reduction\"). \n\nHowever, as it turns out: Many of those solutions could be even further improved by incorporating another training phase after the pretrained model had been trained! This can be done by freezing the pretrained model and adding dense layers after it. You will get a tiny uplift in the upmost leaderboard with that change though. \n\n**Weight freezing and unfreezing in PyTorch**\n\n```python\n# Weight freezing\nfor param in model.parameters():\n param.requires_grad = False # unfreeze weights in at all\n```\n\n```python\n# Weight unfreezing\nfor param in model.parameters():\n param.requires_grad = True # unfreeze weights in at all\n```\n\n**Weight freezing and unfreezing in TensorFlow**\n\n```python\n# Weight freezing\nlayer.trainable = False\n```\n```python\n# Weight unfreezing\nlayer.trainable = True\n```\n\n### Learning rates and learning rate schedulers\n\nLearning rates and learning rate schedulers affect transformers' training much more than they affect the normal ML models. Changing the learning rate can have a great impact on performance and also on training convergence. Remember that the global learning rate weight decay mostly affects the unfreeze layers weights where the fine-tuning requires only a small percentage of warmup. If a model converges slower or yields a lower score than it should, try experimenting by adjusting the learning rate - or the library you are using might not support different learning rates. \n\n#### Learning rate schedulers\n\nIn recent times, One Cycle Cosine schedule has shown to provide better results on multiple NLP tasks, you can use it this way:\n\n**One Cycle Cosine scheduling in PyTorch**\n\n```python\nfrom torch.optim.lr_scheduler import CosineAnnealingLR\noptimizer = torch.optim.AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)\nscheduler = CosineAnnealingLR(optimizer, T_max=num_train_optimization_steps)\nnum_training_steps = num_train_optimization_steps / args.gradient_accumulation_steps\n```\n\n```python\n# Update the scheduler\nscheduler.step()\n# step the learning rate scheduler here, you will want to step the learning rate scheduler only once per optimizer step nothing more nothing less. So in this case, it should be called before you expect the gradients to be applied.\n```\n\n**One Cycle Cosine scheduling in TensorFlow**\n\n```python\noptimizer = tf.keras.optimizers.Adam(learning_rate)\nscheduler = tf.keras.optimizers.schedules.CosineDecay(learning_rate, decay_steps=num_training_steps)\n```\n\n#### Hyperparameters Of Optimizers\n\nThere are several things you need to know if you seek to get the best performance out of Adam optimizer:\n\n* Finding the best weight decay value can be tricky and experiments (and luck) will be your best friends here. \n* Another important hyperparameter is the `beta1` and `beta2` used in the Adam optimizer, choosing the best values depends on your task and the data. many new tasks can benefit from lower beta1 and higher beta2 while on established tasks they would perform the opposite. So again: Experiments will be your best friend here.\n* In the world of Adam optimizer, the number one rule is not to underestimate the importance of the optimizer epsilon value. The same principle of finding the best weight decay hyperparameter applies here.\n* Do not overuse the gradient clipping norm - it might sometimes be helpful when your gradients explode while the opposite is also true - it can prevent convergence on some tasks.\n* Gradient accumulation still sows to provide some subtle benefits, I usually accumulate gradients for about 2 steps but if your GPU does not run out of memory, you can push up to 8 gradient accumulation steps. Gradient accumulation is also useful when using mixed-precision. \n\n#### Some things you need to know about transformers' learning rates\n\n* Transformers require a larger `max_grad_norm`clip_norm` value than other models to converge. \n* Transformers are sensitive to the batch size value and the training loss you will provide to them often depends on the model and the batch size. You will want to try and compare many batch size values in your experiments. \n* Transformers are robust to **multiple learning rates**, while they are usually trained at 2e-5 learning rates or below, it is possible to train them with a much higher learning rate than reducing it or peaking it up when needed during training. This provides a much better solution to `batch size` based learning rate schedulers as you can peak up the learning rates whenever you deem fit. \n* By giving transformers multiple learning rates you can train them to converge faster and also make good use of the first phase of very-high-learning-rate as a warm-up phase for the model. \n* It is possible to train transformers with no (or very low) weight decay but that requires a higher learning rate. \n* The gradients of the model's weights suffered from many more ups and downs than you might expect from ConvNets models, it is probably because of how self attentions works and how transformers are designed to use it. H\n* Finally, don’t underestimate the importance of the learning rate scheduling strategy for your model.\n\n#### Some Learning Rate Schedulers Tips\n\n* Using “Triangular” or “One Cyclic” methods for learning rate scheduling can provide subtle but significant improvements - those intelligent methods of learning rate scheduling are can overcome some of the batch size issues and this will be obvious when using them, especially when plugging them into the multiple learning rate system. \n* Take the time to research for the best learning rate scheduling method for your task and the transformers you are using, it is a very important part of how your model will converge. \n* Learning rate schedules can be used to train transformers with lower batch sizes or multiple learning rates or any combination of those. \n* Try low learning rates first to see if a dramatic learning rate increase will help or hurt your performance. \n* Increasing the learning rate at later stages of training or multiple learning rates or high batch size or gradient accumulation or learning rate schedulers sometimes will help your model converge better, it is an advanced technique as sometimes it can hurt the performance but only if you give it too large a value - remember to test it. \n* Loss scaling can help reduce loss variance and improve gradient flow when using gradient accumulation or multiple learning rates or high batch size, but if you are trying to solve that problem by increasing batch size, try to increase the learning rate instead as it can sometimes yield better performance. \n* In recent times, one cycle learning rate scheduling approaches have taken the leading positions on public leaderboards and past winning solutions. \n\n### Overfitting and regularization\n\n* Use dropout! adding layers dropouts between transformers layers usually yields more training stability and more robust results, use dropouts with your hidden layers.\n* Dropout can also be used to increase performance by small margins, experiment with setting layer dropouts before training. the task and the model.\n* **If you are into regularization: **Regularization can provide marge uplift on performance when your NN is overfitting or underfitting, for normal Machine learning models, L1 or L2 regularization are OK but for transformers, use additive and hidden layers dropout. \n* I sound like a broken record: Use experiments. Experiment. Experiments and also try vanilla models as they still show subtle signs of performance. \n* You can increase your model's robustness to overfitting by validation data augmentation or mixing pretrained models with your trained model\n\n#### Use validation sets to choose robust parameters and data augmentation\n\n* Adding validation set data augmentation will improve your model performance on test time augmentation and thus increase the absolution performance but if you seek to gain better performance use some of your validation set for data augmentation experiments or data augmentation experiments during training and use another part to increase your performance parameters, this will reduce your model overfitting.\n* You can use the test time augmentation by running the same model multiple times (or on multiple GPUs).\n* Run many validation data augmentation experiments at the same time to find the ones with the greatest impact your model performance.\n* Again I sound like a broken record: Using experiments. Experiment. Experiments and also try vanilla models as they still show subtle signs of performance. \n\n### Advanced optimizers\n\n**There are several notable optimizers that are important to know about:**\n- **AdamW:** This is an extension to the Adam algorithm that prevents exponential weights decay of the model's weights in the outer layers as well as encourages penalized hyper volume below the default weights. \n- **Adafactor:** It was designed to have low memory usage and is scalable. This optimizer can offer significant optimizer performances using multiple GPUs (see below). \n- **Novograd:** Basically another Adam-like optimizer but with very better properties. It is one of the optimizers used by to train the bert-large model. \n\n- **Ranger:** Ranger optimizer is a very interesting optimizer that has pretty good results in winning solutions it terms of performance optimization but it is obviously not very well known or supported so I leave that up to you to research. \n- **Lamb:** GPU optimized reusable Adam optimizer developed by the GLUE and QQP competition winner. \n- **Lookahead:** A popular optimizer you can use on top of other optimizers and it will provide you with some performance gains. \n\n### Batch size vs steps vs max len \n\nI think after this topic you should have a pretty good idea about finding the best batch size for the task and model you are using. Now I want to point out some more problems that might affect you during training. \nYou can find below some tips and tricks extended from the transformers docs.\n\n**Max length**\n\nWhen selecting a value for the `max_length` (or `max_position_embeddings`) argument of a transformer model, please ensure that the chosen value is less than the straight maximum sequence length you can feed to that model without padding. Using a length beyond this value will result in an exception when attempting to save the model. This behavior occurs whether you supply the value or use the default one provided. \n\nFor the new \"longformer\" tokenizer (LongformerTokenizer), the value of `max_length` is not enforced, but instead tokens after the `max_length` are silently ignored for the shape calculation. For safety, you should always use the actual maximum length your data will contain as the value for `max_length`.\n\n**Padding**\n\nWhen you are batching a dataset, a Tensor might be returned which is an array of sequences that are of uneven lengths, where each sequence is contained within a matrix. If the batch vectors are padded in the dimension exceeding the maximum length of the sequence, these vectors can propagate the errors to all the remaining tokens of that sequence.\nIn the TensorFlow framework, this scenario is solved by right-padding a sequence to fit the maximum sequence length; whereas, it is solved in the PyTorch framework by left-padding a sequence to fit the maximum sequence length. To manage this small difference, a simple check for the framework is written in the subsequent code. This way, a similar padded seq len sequence is passed over to the TensorFlow and PyTorch frameworks.\n* **To use `left_pad_target=True` `left_pad_source=False`, you want to pad the smaller sequence on the right and the longer one on the left.*\n\n**Total sequence count**\n\n`max_seq` param of the `data_collator` class only keeps `max_seq` sequences at once. This means that calling `data_collator.encode_batch(my_sequence_of_sequences)` more than `max_seq` times will not encode all sequences. To encode all sequences in that batch, you need to call `max_seq` times `data_collator.encode_batch(next_sequences)`, where `next_sequences` is a split of `my_sequence_of_sequences`, each with at most `max_seq` sequences.\n\n**Gradient accumulation**\n\nGradient accumulation delays the gradient computation until all the mini batches in an epoch are processed through the model. This helps in faster training when the batch size is very less, that is, in the case of large models. The GPU calculates a few steps of the model with a large batch size at regular intervals of every parameter update, thereby avoiding an expensive gradient update every step. Gradient accumulation is activated if the gradient_accumulation_steps integer is used as a parameter in the training loop.\n\n**vocab size**\n\n`vocab size` in `Trainer` is the total number of retained tokens used for training the model. The tokenizers can be further extended to support frequent tokens of the new training set we are fine tuning. This sometimes leads to better results. \n\n### Precision and mixed-precision\n\n**Mixed Precision Training**\nTraining deep neural networks can be computationally intensive. When performing these computations, there is an inherent tradeoff between speed and performance. Mixed precision is the combined use of different numerical precisions in a computational method. Mixed precision training offers significant computational speedup by performing operations in the half-precision format while storing minimal information in single-precision to retain as much information as possible in critical parts of the network. \n\nThere are several mixed-precision approaches, the most commonly used mixed processors functioning modes are the following:\n\n**Subtle Differences:** With this mode, the weights are stored in low precision but some parts of the network are executed in single precision.\n**Also note that mixed-precision training is achieved without a significant accuracy degradation compared to full precision training. This can be very handy in cases where resources like `GPU power` and especially time are limited (in cases where multiple models have to be trained), it can also work in conjunction with boosted learning rates and high batch size techniques to yield even greater improvements by using a small percentage of data to get a rough idea what can be a large-scale performance boost.**\n\n**Mixed Precision training in Pytorch**\n\nMake sure to install apex for this to work and to check if your GPU supports mixed-precision training\n\n```python\n!git clone https://github.com/NVIDIA/apex.git\nrun apex/setup.sh \n```\n\n```python\nfrom apex import amp \nnet, optimizer = amp.initialize(net, optimizer, opt_level = 'O2', keep_batchnorm_fp32 = False, loss_scale = 'dynamic')```\n```\n\n**Mixed-Precision training in Tensorflow**\n\n```python\ntf.config.optimizer.set_experimental_options({\"auto_mixed_precision\": True}) \n```\n\n### Distributed training and multi-GPU\n\nDistributed training saves time by distributing the data across multiple GPUs so if you have multiple GPUs and even multiple machines with GPUs it can save you a lot of time. The downside is that this is a very large topic with a lot of frameworks and approaches to choose from and thus I will leave the choice up to you, a sublte tip is that you should choose the easyest one to use and integrate as a framework but as a tip distributed training can sometimes improve/yield better performance on performance-bound models such as LAMB models for example. \n\nNote that distributed training and multi-GPU works best with the more efficient represantions (optimalization level from FP16 to FP32) as well as with gradient accumulation. \n\n**Multi GPU in Pytorch**\n\n```python\nif torch.multiprocessing.spawn.supports_start_method(): mp.spawn(main_worker, nprocs = args.n_gpu, args = (args, device))\nelse: mp.spawn(main_worker, args = (args, device), nprocs = args.n_gpu) \n```\n\n**Multi GPU in Tensorflow**\n\n```python\nstrategy = tf.distribute.MirroredStrategy()\n```\n\n### Warmup, Learning Rate, and Batch size interactions\n\n**Warm-UP**\n\nDo not overlook the warm-up period when training transformers, the first 5% or so of training - Is a very important period of training as it is the good start your model needs to stay forcefully on the rails. However, as you may know, warm-up does not have to be constant, in studies, we have witnessed increased benefits from using a gradually increased learning rate for a warm-up and sometimes even better performance using multiple learning rates for a warm-up. \n\n**Learning rate and batch size**\n\nSometimes having a low batch size with a high learning rate can provide good results so never rule this idea out from your strategy, sometimes the highest batch size might give you worse performance with unfit learning rates while being used with higher learning rates or learning rate scheduler. So try lowing the batch size and learning rate and give them multiple learning rates with things as it might produce better results.\n\nFinally, I would like to remind you that there are many solutions to the same problem and each solution is a one piece of the bigger puzzle, it is up to you to put the puzzle together and figure out which pieces go together with the best :)", "votes": null }, { "id": "1847772", "postDate": "07/08/2022 06:27:12", "content": "Wonderful post! Fantastic work!
", "rawMarkdown": "Wonderful post! Fantastic work!", "votes": null }, { "id": "1854727", "postDate": "07/14/2022 00:55:27", "content": "It seems like a good guidebook that can be used in the form of an inspection checklist to learn not only in this competition but also in actual work. Thank you very much. 👍👍
", "rawMarkdown": "It seems like a good guidebook that can be used in the form of an inspection checklist to learn not only in this competition but also in actual work. Thank you very much. 👍👍", "votes": null } ], "comments": [ { "id": 1847772, "author_name": "ravi20076", "author_url": "", "post_date": "07/08/2022 06:27:12", "content": "Wonderful post! Fantastic work!
", "votes": null, "replies": [] }, { "id": 1854727, "author_name": "heuingttiang", "author_url": "", "post_date": "07/14/2022 00:55:27", "content": "It seems like a good guidebook that can be used in the form of an inspection checklist to learn not only in this competition but also in actual work. Thank you very much. 👍👍
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1847422": "# Ideas for training better language models\n\nHi Guys,\n\nThere had been some discussions about what exactly can we do to improve our language models since we all pretty much use the same one. \n\nI am cross-posting a post I've written a couple of weeks ago about tips and tricks for yielding better performance out of hugging face pretrained models \n\nLet me know what you think! \n\nEnjoy! \n\n_____\n\n### Beware of the default `batch size` in transformers\n\nTransformers are sensitive to the size of the `batch` we set them with, this is because each transformer layer is computing weights across the expected hidden features of the batch size it receives then projecting the hidden features down to the specified parameters of the head it is feeding. Technically you can train with any batch size you want but you should also consider changing the learning rate (more on this late). Going under the default `batch size` might not be the optimal solution as you might lose some of the weight quality of the model. With that in mind, finding the right batch size or learning rate can be tricky since transformers take time to train.\n\n### About hidden layers after the pretrained model\n\nAdding more layers can be beneficial as you can use them to learn more advanced features, but it can also temper the fine-tuning of your large pretrained model. This can easily be seen in many past winning solutions where the winner simply used a pretrained language model followed by a simple reduction (such as mean or \"deberta reduction\"). \n\nHowever, as it turns out: Many of those solutions could be even further improved by incorporating another training phase after the pretrained model had been trained! This can be done by freezing the pretrained model and adding dense layers after it. You will get a tiny uplift in the upmost leaderboard with that change though. \n\n**Weight freezing and unfreezing in PyTorch**\n\n```python\n# Weight freezing\nfor param in model.parameters():\n param.requires_grad = False # unfreeze weights in at all\n```\n\n```python\n# Weight unfreezing\nfor param in model.parameters():\n param.requires_grad = True # unfreeze weights in at all\n```\n\n**Weight freezing and unfreezing in TensorFlow**\n\n```python\n# Weight freezing\nlayer.trainable = False\n```\n```python\n# Weight unfreezing\nlayer.trainable = True\n```\n\n### Learning rates and learning rate schedulers\n\nLearning rates and learning rate schedulers affect transformers' training much more than they affect the normal ML models. Changing the learning rate can have a great impact on performance and also on training convergence. Remember that the global learning rate weight decay mostly affects the unfreeze layers weights where the fine-tuning requires only a small percentage of warmup. If a model converges slower or yields a lower score than it should, try experimenting by adjusting the learning rate - or the library you are using might not support different learning rates. \n\n#### Learning rate schedulers\n\nIn recent times, One Cycle Cosine schedule has shown to provide better results on multiple NLP tasks, you can use it this way:\n\n**One Cycle Cosine scheduling in PyTorch**\n\n```python\nfrom torch.optim.lr_scheduler import CosineAnnealingLR\noptimizer = torch.optim.AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)\nscheduler = CosineAnnealingLR(optimizer, T_max=num_train_optimization_steps)\nnum_training_steps = num_train_optimization_steps / args.gradient_accumulation_steps\n```\n\n```python\n# Update the scheduler\nscheduler.step()\n# step the learning rate scheduler here, you will want to step the learning rate scheduler only once per optimizer step nothing more nothing less. So in this case, it should be called before you expect the gradients to be applied.\n```\n\n**One Cycle Cosine scheduling in TensorFlow**\n\n```python\noptimizer = tf.keras.optimizers.Adam(learning_rate)\nscheduler = tf.keras.optimizers.schedules.CosineDecay(learning_rate, decay_steps=num_training_steps)\n```\n\n#### Hyperparameters Of Optimizers\n\nThere are several things you need to know if you seek to get the best performance out of Adam optimizer:\n\n* Finding the best weight decay value can be tricky and experiments (and luck) will be your best friends here. \n* Another important hyperparameter is the `beta1` and `beta2` used in the Adam optimizer, choosing the best values depends on your task and the data. many new tasks can benefit from lower beta1 and higher beta2 while on established tasks they would perform the opposite. So again: Experiments will be your best friend here.\n* In the world of Adam optimizer, the number one rule is not to underestimate the importance of the optimizer epsilon value. The same principle of finding the best weight decay hyperparameter applies here.\n* Do not overuse the gradient clipping norm - it might sometimes be helpful when your gradients explode while the opposite is also true - it can prevent convergence on some tasks.\n* Gradient accumulation still sows to provide some subtle benefits, I usually accumulate gradients for about 2 steps but if your GPU does not run out of memory, you can push up to 8 gradient accumulation steps. Gradient accumulation is also useful when using mixed-precision. \n\n#### Some things you need to know about transformers' learning rates\n\n* Transformers require a larger `max_grad_norm`clip_norm` value than other models to converge. \n* Transformers are sensitive to the batch size value and the training loss you will provide to them often depends on the model and the batch size. You will want to try and compare many batch size values in your experiments. \n* Transformers are robust to **multiple learning rates**, while they are usually trained at 2e-5 learning rates or below, it is possible to train them with a much higher learning rate than reducing it or peaking it up when needed during training. This provides a much better solution to `batch size` based learning rate schedulers as you can peak up the learning rates whenever you deem fit. \n* By giving transformers multiple learning rates you can train them to converge faster and also make good use of the first phase of very-high-learning-rate as a warm-up phase for the model. \n* It is possible to train transformers with no (or very low) weight decay but that requires a higher learning rate. \n* The gradients of the model's weights suffered from many more ups and downs than you might expect from ConvNets models, it is probably because of how self attentions works and how transformers are designed to use it. H\n* Finally, don’t underestimate the importance of the learning rate scheduling strategy for your model.\n\n#### Some Learning Rate Schedulers Tips\n\n* Using “Triangular” or “One Cyclic” methods for learning rate scheduling can provide subtle but significant improvements - those intelligent methods of learning rate scheduling are can overcome some of the batch size issues and this will be obvious when using them, especially when plugging them into the multiple learning rate system. \n* Take the time to research for the best learning rate scheduling method for your task and the transformers you are using, it is a very important part of how your model will converge. \n* Learning rate schedules can be used to train transformers with lower batch sizes or multiple learning rates or any combination of those. \n* Try low learning rates first to see if a dramatic learning rate increase will help or hurt your performance. \n* Increasing the learning rate at later stages of training or multiple learning rates or high batch size or gradient accumulation or learning rate schedulers sometimes will help your model converge better, it is an advanced technique as sometimes it can hurt the performance but only if you give it too large a value - remember to test it. \n* Loss scaling can help reduce loss variance and improve gradient flow when using gradient accumulation or multiple learning rates or high batch size, but if you are trying to solve that problem by increasing batch size, try to increase the learning rate instead as it can sometimes yield better performance. \n* In recent times, one cycle learning rate scheduling approaches have taken the leading positions on public leaderboards and past winning solutions. \n\n### Overfitting and regularization\n\n* Use dropout! adding layers dropouts between transformers layers usually yields more training stability and more robust results, use dropouts with your hidden layers.\n* Dropout can also be used to increase performance by small margins, experiment with setting layer dropouts before training. the task and the model.\n* **If you are into regularization: **Regularization can provide marge uplift on performance when your NN is overfitting or underfitting, for normal Machine learning models, L1 or L2 regularization are OK but for transformers, use additive and hidden layers dropout. \n* I sound like a broken record: Use experiments. Experiment. Experiments and also try vanilla models as they still show subtle signs of performance. \n* You can increase your model's robustness to overfitting by validation data augmentation or mixing pretrained models with your trained model\n\n#### Use validation sets to choose robust parameters and data augmentation\n\n* Adding validation set data augmentation will improve your model performance on test time augmentation and thus increase the absolution performance but if you seek to gain better performance use some of your validation set for data augmentation experiments or data augmentation experiments during training and use another part to increase your performance parameters, this will reduce your model overfitting.\n* You can use the test time augmentation by running the same model multiple times (or on multiple GPUs).\n* Run many validation data augmentation experiments at the same time to find the ones with the greatest impact your model performance.\n* Again I sound like a broken record: Using experiments. Experiment. Experiments and also try vanilla models as they still show subtle signs of performance. \n\n### Advanced optimizers\n\n**There are several notable optimizers that are important to know about:**\n- **AdamW:** This is an extension to the Adam algorithm that prevents exponential weights decay of the model's weights in the outer layers as well as encourages penalized hyper volume below the default weights. \n- **Adafactor:** It was designed to have low memory usage and is scalable. This optimizer can offer significant optimizer performances using multiple GPUs (see below). \n- **Novograd:** Basically another Adam-like optimizer but with very better properties. It is one of the optimizers used by to train the bert-large model. \n\n- **Ranger:** Ranger optimizer is a very interesting optimizer that has pretty good results in winning solutions it terms of performance optimization but it is obviously not very well known or supported so I leave that up to you to research. \n- **Lamb:** GPU optimized reusable Adam optimizer developed by the GLUE and QQP competition winner. \n- **Lookahead:** A popular optimizer you can use on top of other optimizers and it will provide you with some performance gains. \n\n### Batch size vs steps vs max len \n\nI think after this topic you should have a pretty good idea about finding the best batch size for the task and model you are using. Now I want to point out some more problems that might affect you during training. \nYou can find below some tips and tricks extended from the transformers docs.\n\n**Max length**\n\nWhen selecting a value for the `max_length` (or `max_position_embeddings`) argument of a transformer model, please ensure that the chosen value is less than the straight maximum sequence length you can feed to that model without padding. Using a length beyond this value will result in an exception when attempting to save the model. This behavior occurs whether you supply the value or use the default one provided. \n\nFor the new \"longformer\" tokenizer (LongformerTokenizer), the value of `max_length` is not enforced, but instead tokens after the `max_length` are silently ignored for the shape calculation. For safety, you should always use the actual maximum length your data will contain as the value for `max_length`.\n\n**Padding**\n\nWhen you are batching a dataset, a Tensor might be returned which is an array of sequences that are of uneven lengths, where each sequence is contained within a matrix. If the batch vectors are padded in the dimension exceeding the maximum length of the sequence, these vectors can propagate the errors to all the remaining tokens of that sequence.\nIn the TensorFlow framework, this scenario is solved by right-padding a sequence to fit the maximum sequence length; whereas, it is solved in the PyTorch framework by left-padding a sequence to fit the maximum sequence length. To manage this small difference, a simple check for the framework is written in the subsequent code. This way, a similar padded seq len sequence is passed over to the TensorFlow and PyTorch frameworks.\n* **To use `left_pad_target=True` `left_pad_source=False`, you want to pad the smaller sequence on the right and the longer one on the left.*\n\n**Total sequence count**\n\n`max_seq` param of the `data_collator` class only keeps `max_seq` sequences at once. This means that calling `data_collator.encode_batch(my_sequence_of_sequences)` more than `max_seq` times will not encode all sequences. To encode all sequences in that batch, you need to call `max_seq` times `data_collator.encode_batch(next_sequences)`, where `next_sequences` is a split of `my_sequence_of_sequences`, each with at most `max_seq` sequences.\n\n**Gradient accumulation**\n\nGradient accumulation delays the gradient computation until all the mini batches in an epoch are processed through the model. This helps in faster training when the batch size is very less, that is, in the case of large models. The GPU calculates a few steps of the model with a large batch size at regular intervals of every parameter update, thereby avoiding an expensive gradient update every step. Gradient accumulation is activated if the gradient_accumulation_steps integer is used as a parameter in the training loop.\n\n**vocab size**\n\n`vocab size` in `Trainer` is the total number of retained tokens used for training the model. The tokenizers can be further extended to support frequent tokens of the new training set we are fine tuning. This sometimes leads to better results. \n\n### Precision and mixed-precision\n\n**Mixed Precision Training**\nTraining deep neural networks can be computationally intensive. When performing these computations, there is an inherent tradeoff between speed and performance. Mixed precision is the combined use of different numerical precisions in a computational method. Mixed precision training offers significant computational speedup by performing operations in the half-precision format while storing minimal information in single-precision to retain as much information as possible in critical parts of the network. \n\nThere are several mixed-precision approaches, the most commonly used mixed processors functioning modes are the following:\n\n**Subtle Differences:** With this mode, the weights are stored in low precision but some parts of the network are executed in single precision.\n**Also note that mixed-precision training is achieved without a significant accuracy degradation compared to full precision training. This can be very handy in cases where resources like `GPU power` and especially time are limited (in cases where multiple models have to be trained), it can also work in conjunction with boosted learning rates and high batch size techniques to yield even greater improvements by using a small percentage of data to get a rough idea what can be a large-scale performance boost.**\n\n**Mixed Precision training in Pytorch**\n\nMake sure to install apex for this to work and to check if your GPU supports mixed-precision training\n\n```python\n!git clone https://github.com/NVIDIA/apex.git\nrun apex/setup.sh \n```\n\n```python\nfrom apex import amp \nnet, optimizer = amp.initialize(net, optimizer, opt_level = 'O2', keep_batchnorm_fp32 = False, loss_scale = 'dynamic')```\n```\n\n**Mixed-Precision training in Tensorflow**\n\n```python\ntf.config.optimizer.set_experimental_options({\"auto_mixed_precision\": True}) \n```\n\n### Distributed training and multi-GPU\n\nDistributed training saves time by distributing the data across multiple GPUs so if you have multiple GPUs and even multiple machines with GPUs it can save you a lot of time. The downside is that this is a very large topic with a lot of frameworks and approaches to choose from and thus I will leave the choice up to you, a sublte tip is that you should choose the easyest one to use and integrate as a framework but as a tip distributed training can sometimes improve/yield better performance on performance-bound models such as LAMB models for example. \n\nNote that distributed training and multi-GPU works best with the more efficient represantions (optimalization level from FP16 to FP32) as well as with gradient accumulation. \n\n**Multi GPU in Pytorch**\n\n```python\nif torch.multiprocessing.spawn.supports_start_method(): mp.spawn(main_worker, nprocs = args.n_gpu, args = (args, device))\nelse: mp.spawn(main_worker, args = (args, device), nprocs = args.n_gpu) \n```\n\n**Multi GPU in Tensorflow**\n\n```python\nstrategy = tf.distribute.MirroredStrategy()\n```\n\n### Warmup, Learning Rate, and Batch size interactions\n\n**Warm-UP**\n\nDo not overlook the warm-up period when training transformers, the first 5% or so of training - Is a very important period of training as it is the good start your model needs to stay forcefully on the rails. However, as you may know, warm-up does not have to be constant, in studies, we have witnessed increased benefits from using a gradually increased learning rate for a warm-up and sometimes even better performance using multiple learning rates for a warm-up. \n\n**Learning rate and batch size**\n\nSometimes having a low batch size with a high learning rate can provide good results so never rule this idea out from your strategy, sometimes the highest batch size might give you worse performance with unfit learning rates while being used with higher learning rates or learning rate scheduler. So try lowing the batch size and learning rate and give them multiple learning rates with things as it might produce better results.\n\nFinally, I would like to remind you that there are many solutions to the same problem and each solution is a one piece of the bigger puzzle, it is up to you to put the puzzle together and figure out which pieces go together with the best :)", "1847772": "Wonderful post! Fantastic work!", "1854727": "It seems like a good guidebook that can be used in the form of an inspection checklist to learn not only in this competition but also in actual work. Thank you very much. 👍👍" }, "source": "meta" }