![\"Gradient](\"https://miro.medium.com/max/1400/1*rJIH9gPhctTLCk5G5iQ_oA.png\")
\n Translations:\n English, Chinese\n
\n\n\n\n\n Despite the stunning success of Transformers in Natural Language Processing (NLP) tasks, it is still challenging to train them on even modern Graphics Processing Units (GPUs) or deploy them in production, due to the massive number of parameters. Training or inferencing such large models, we can probably run out of memory (OOM) or the process becomes very long.
\nNevertheless, there are a lot of offered approaches to avoid such problems, so the main contribution of this article is to describe and show how the provided methods can be applied in the Training and Inference scripts. Firstly, the article will go through basic approaches such as Gradient Accumulation, Freezing, Automatic Mixed Precision, 8-bit Optimizers, and Gradient Checkpointing, and then describe NLP-specific optimizing approaches such as Dynamic Padding, Uniform Dynamic Padding, and Fast Tokenizers.\n\n
\nThe idea behind Gradient Accumulation is very simple - simulating a larger batch size. Sometimes using a large batch size is necessary for better convergence or improving the performance, however, it often requires a lot of memory. One possible solution to such an issue is to use a smaller batch size, however, on the one hand, small batch size leads to increasing training or inference time, and on the other hand, the gradient descent algorithms are very sensitive to the choice of batch size and may lead to unstable convergence and performance reduction. Instead, we can run multiply steps (accumulation steps) and accumulate (compute average) gradients a certain number of accumulation steps, and then when we have enough computed gradients perform the optimization step.\n
","metadata":{}},{"cell_type":"markdown","source":"\nFreezing is an effective way to speed up training and decrease memory utilization almost without losing final quality by toggling computing gradients in certain layers of the model.
\nA well-known fact in Deep Learning is that low layers learn input data patterns and at the same time top layers learn high-level features, which are specific to target tasks. When performing the optimization step with some kind of optimization algorithms (e.g. SGD, AdamW, or RMSprop), the low layers receive small gradients, and hence the parameters almost stay not changed, this is called Gradient Vanishing, so instead of computing \"useless\" gradients and perform optimization of such low-gradients parameters, which sometimes require a lot of time and computational power, we can just freeze them.
\nPyTorch provides a comfortable API for toggling computing gradients. Such behavior can be set by the property requires_grad of torch.Tensor.\n\n
\nAutomatic Mixed Precision (AMP) is another very simple way of reducing memory consumption and training time without losing final quality, which was introduced in \"Mixed Precision Training\" paper by NVIDIA and Baidu researchers in 2017. The key idea behind the approach is to use lower precision for keeping the model's gradients and parameters in the memory, i.e instead of using full precision (e.g float32) the proposed approach uses half-precision (e.g float16) for keeping tensors in memory. However, when computing gradients in lower precision, some values can be so small that they are treated as zeros, this phenomenon is called \"overflow\". In order to prevent \"overflow\", the authors of the original paper proposed a gradient scaling method.
\nPyTorch provides a package with necessary functionality (from lowering precision to gradient scaling) for using Automatic Mixed Precision, called torch.cuda.amp. Automatic Mixed Precision was implemented as a context manager, so it can easily be inserted in training and inferencing scripts.\n\n\n
![\"Automatic](\"https://developer-blogs.nvidia.com/wp-content/uploads/2019/01/pasted-image-0-21.png\")
\n The idea of 8-bit Optimizers is similar to Automatic Mixed Precision, where the model's parameters and gradients are kept in lower precision, but 8-bit Optimizers additionally keep the optimizer's state in lower precision too. The authors (Meta Research) detail described the 8-bit Optimizers in the original paper \"8-bit Optimizers via Block-wise Quantization\", and showed that 8-bit Optimizers lead to significant decreasing memory utilization and slightly speeding up the training. Additionally, the authors studied the impact of different hyperparameter settings and show that 8-bit Optimizers are stable to different choices of learning rate, betas and weight decay parameters without losing performance or hurting the convergence. Therefore, the authors provided a comfortable high-level library for 8-bit Optimizers, called bitsandbytes. \n \n
","metadata":{}},{"cell_type":"markdown","source":"![\"8-bit](\"https://i.ibb.co/9bj3JqG/Screenshot-3.png\")
\nSometimes even using small batch size and other optimization techniques, e.g. Gradient Accumulation, Freezing, or Automatic Precision Training, we still can run out of memory, especially in cases when the models are large enough. One of the proposed powerful solutions for solving this issue is Gradient Checkpointing, which was firstly introduced in the \"Training Deep Nets With Sublinear Memory Cost\" paper in 2016. The authors demonstrated that Gradient Checkpointing can significantly reduce memory utilization from $ O(n) $ to $ O(\\sqrt{n}) $, where $ n $ is the number of layers in the model. This approach allows training large models on a single GPU or provides more memory for increasing the batch size for better and faster convergence.\n\n
","metadata":{}},{"cell_type":"markdown","source":"\nThe idea behind Gradient Checkpoint is to compute gradients in small chunks while removing unnecessary gradients from the memory during forward and backpropagation passes, thereby reducing memory utilization, despite, such an approach requires more compute steps to reproduce the whole back propagation graph.\n\n
","metadata":{}},{"cell_type":"markdown","source":"\nPyTorch framework provides gradient checkpointing from the box via torch.utils.checkpoint.checkpoint and torch.utils.checkpoint.checkpoint_sequential functions.
\n \"Specifically, in the forward pass, function will run in torch.no_grad() manner, i.e., not storing the intermediate activations. Instead, the forward pass saves the inputs tuple and the function parameter. In the backwards pass, the saved inputs and function is retrieved, and the forward pass is computed on function again, now tracking the intermediate activations, and then the gradients are calculated using these activation values.\"
\nAdditionally, HuggingFace Transformers supports Gradient Checkpoint too. Gradient Checkpointing can be performed by gradient_checkpointing_enable method of PreTrainedModel instance.\n
\nHuggingFace Transformers provides two types of Tokenizers: Base and Fast. The main difference between them is that Fast Tokenizers are written on Rust since Python is very slow in loops, which are necessary during tokenization. This is a non-trivial way of allowing us to get a additional speed-up during the tokenization process. The types of Tokenizers can be easily changed through HuggingFace Transformers API in from_pretrained method of transformers.AutoTokenizer instance by setting use_fast property to True.\n
","metadata":{}},{"cell_type":"markdown","source":"![\"Tokenization](\"https://huggingface.co/datasets/huggingface-course/documentation-images/resolve/main/en/chapter6/tokenization_pipeline.svg\")
\nGenerally, the models are trained with a batch of inputs, and each input in the batch must have a fixed size, I.e the batch must be a representation of the matrix. The fixed size is often selected relatively on the length distribution in the dataset, the number of features, or other factors. In NLP tasks, the input size is referred to as the length of the text and called max length. Unfortunately, different texts have different lengths, so to handle such cases, the researchers proposed padding tokens and truncation. Truncation is applied, when the max length is smaller than the input text's length, so some tokens (often lasts) are removed. Padding tokens are special tokens, which is added to the end of input text when the input text's length is smaller than the max length, also worthing to note that padding tokens should not be included in calculating loss in some tasks (e.g Masked Language Modeling or Named Entity Recognition).\n
","metadata":{}},{"cell_type":"markdown","source":"![\"fixed-padding-length-1\"](\"https://i.ibb.co/J2j895v/fixed-padding-length-1.png\")
\n\n However, the padding tokens have significant drawbacks. It is very ineffective and requires more additional memory in cases where the input text is very short relatively to chosen max length. To prevent extra computational operations, the developers proposed one very effective approach - pad the inputs of the batch to the maximum input length of the batch. Although the difference in the terminology is very insufficient, such an approach can speed up training by 35% or even 50%! Despite this, speed up and memory usage depends on batch size and distribution of lengths.\n\n
","metadata":{}},{"cell_type":"markdown","source":"![\"dynamic-padding\"](\"https://i.ibb.co/BzRMzgx/dynamic-padding.png\")
\nThere is one additional possible approach, which is based on Dynamic Padding, called Uniform Dynamic Padding. The idea is to sort texts by their corresponding lengths beforehand and not shuffle samples during training or inference. This approach is very effective and needs fewer computations during training or inference than Dynamic Padding. However, it is not recommended to use Uniform Dynamic Padding during training since the training implies shuffling of inputs.\n\n
","metadata":{}},{"cell_type":"markdown","source":"![\"uniform-length-batching\"](\"https://i.ibb.co/MCBJj71/uniform-length-batching.png\")
\n\nThe optimization is a necessary step in developing models even on modern GPUs. For this reason, the article went through the most powerful and popular approaches for speeding up the training and reducing the memory consumption of large models such as Transformers. \n
","metadata":{}},{"cell_type":"markdown","source":"\nDuring writing article the following links were used:
\n