{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\nAdvancements in the context window of transformer models have paved the way for improved modeling capabilities and enhanced understanding of long-range dependencies. Traditional approaches, such as sinusoidal position embeddings, have faced challenges in extrapolating beyond their original context window. However, recent innovations in Position Interpolation offer a solution by downsizing input position indices to align with the desired context window size. This technique enables transformers to effectively model longer contexts without compromising quality. Additionally, the introduction of FlashAttention has revolutionized attention computation in transformers with large context windows, optimizing memory access and computation speed. By performing softmax reduction without complete visibility to the entire input and eliminating the need to retain intermediate attention matrices during the backward pass, FlashAttention significantly improves computational efficiency and accelerates training and inference times.\n\nThese advancements in context window management hold great potential for a wide range of natural language processing tasks. Longer context windows provide transformers with a broader understanding of text, allowing them to capture more intricate patterns and dependencies. Consequently, transformers equipped with extended context windows, leveraging Position Interpolation and FlashAttention, demonstrate improved performance on various language processing tasks, including machine translation, sentiment analysis, question answering, and text generation. By effectively modeling longer dependencies, these techniques empower transformers to better comprehend semantic relationships and contextual nuances, leading to more accurate and comprehensive language understanding.\n\nIn this paper, we delve into the details of Position Interpolation and FlashAttention, exploring their impact on the performance, efficiency, and extrapolation capabilities of transformer models. Through experimental evaluations and comparisons with existing approaches, we aim to shed light on the potential of these advancements to push the boundaries of natural language processing and advance the state-of-the-art in various language-related applications.","metadata":{}},{"cell_type":"markdown","source":"# Positional Extrapolation and interpolation\nThe transformers that uses sinosuidal approach have a very weak extraplation capability and as per Press et al(2022) this is because of its position embedding method.\nThe ALiBi biases query-key attention scores and add a penalty that is proportional to the distance between them. It provide the perplexity at par the sinosuidal which is trained with 2048 length,but uses 11% less memory and is 11% faster.[6] But it is an non trainale embedding technique and these techniques are not very popular. One popular extrapolation technique comes in the form of RoPE that is being used in LLaMa. But it has very loose bound as the distance increases the values that comes  out are highly unstable.[7]\nFor this chen at al suggest positional interpolation which has 600 times smaller upper bound which is highly stable.The process of Position Interpolation involves linearly downsizing the input position indices to align them with the original context window size. Position Interpolation provides a straightforward means to accommodate extremely long context windows, such as 32768, by requiring only 1000 fine-tuning steps on the Pile (Gao et al., 2020) dataset to achieve good quality.[7]","metadata":{}},{"cell_type":"markdown","source":"![](https://www.researchgate.net/publication/371909431/figure/fig1/AS:11431281170841288@1687922397684/An-illustration-of-our-Position-Interpolation-method-Consider-a-Llama-model-pre-trained.png)","metadata":{}},{"cell_type":"markdown","source":"[7]","metadata":{}},{"cell_type":"markdown","source":"# FlashAttention\n\nThe training of transformers with large context window is expensive and timetaking. It is because the process to increase the context window has a huge dependency on attention algorithm. The basic attention algorithm is quadically expensive as the context window increases.\nThe way to deal with it by improving the mechanism of attention computation.\nThere are various efforts made to ease the attention algorithm calculation with the help of sparse-approximation and low-rank approximation.\nBut various paper claims that the main problem behind this is slower memory access(IO)[1,2,3,4].They argue at the fact that the computation speed of modern GPUs are very high compared to the memory speed. The technique reduce HBM access for attention matrix with\n1- Performing softmax reduction without visibility to the entire input. execute the softmax reduction, For this the input is partitioned the into small chunks and softmax reduction is executed iteratively over these chunk in multiple passes\".[5]\n2- Refraining from retaining the intermediate attention matrix during the backward pass. We enhance computational efficiency by storing the softmax normalization factor obtained during the forward pass. This allows us to swiftly recompute attention on-chip in the course of backward pass.[5]\n\nIt yeild great result on training as it trains GPT-2 with 1K sequence length three times faster and on long-range where sequence length is between 1k-4k 2.4 times faster.\nWhen combined with block sparse technique it improves the perplexity of gpt-2 by 0.7 and long-document classification by 6.4.[5]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"![](https://www.adept.ai/images/blog/flashier-attention/01.png)","metadata":{}},{"cell_type":"markdown","source":"[4]","metadata":{}},{"cell_type":"markdown","source":"# Conclusion\nThe advancements in the context window of transformer models, facilitated by techniques such as Position Interpolation and FlashAttention, offer significant improvements in modeling capabilities, efficiency, and the ability to handle longer dependencies. These innovations enhance computational efficiency, accelerate training and inference times, and enable transformers to capture more nuanced patterns and contextual information. The extended context windows empower transformers to achieve improved performance in various natural language processing tasks. Continued research in this area holds promise for further enhancing language understanding and pushing the boundaries of transformer-based models.","metadata":{}},{"cell_type":"markdown","source":"# Reference\n[1] Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler. Data movement is all you need: A case study on optimizing transformers. Proceedings of Machine Learning and Systems, 3:711–732, 2021.--8\n\n[2] NVIDIA. Nvidia Tesla V100 GPU architecture, 2017.\n\n[3] NVIDIA. Nvidia A100 tensor core GPU architecture, 2020.\n\n[4] NVIDIA. Nvidia H100 tensor core GPU architecture, 2022.\n[5] Dao, Tri, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. \"Flashattention: Fast and memory-efficient exact attention with io-awareness.\" Advances in Neural Information Processing Systems 35 (2022): 16344-16359.\n\n[6] Press, Ofir, Noah A. Smith, and Mike Lewis. \"Train short, test long: Attention with linear biases enables input length extrapolation.\" arXiv preprint arXiv:2108.12409 (2021).\n\n[7] Chen, Shouyuan, Sherman Wong, Liangjian Chen, and Yuandong Tian. \"Extending Context Window of Large Language Models via Positional Interpolation.\" arXiv preprint arXiv:2306.15595 (2023).\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\ndf = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')\ndf.iloc[0,1] = 'Text data'\ndf.iloc[1,1] = 'https://www.kaggle.com/code/riyajm/advancement-in-context-window'\ndf.iloc[2,1] = 'https://www.kaggle.com/code/roshinifernando/real-time-object-detection/comments#2347059'\ndf.iloc[3,1] = 'https://www.kaggle.com/code/charleschuang/clustering-methods-for-medicine/comments#2347068'\ndf.iloc[4,1] = 'https://www.kaggle.com/code/antongc/developments-in-ai-and-ethics-in-the-past-2-years/comments#2347087'\n\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-16T18:23:20.317792Z","iopub.execute_input":"2023-07-16T18:23:20.318396Z","iopub.status.idle":"2023-07-16T18:23:20.339457Z","shell.execute_reply.started":"2023-07-16T18:23:20.318332Z","shell.execute_reply":"2023-07-16T18:23:20.338222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T18:23:20.341899Z","iopub.execute_input":"2023-07-16T18:23:20.342857Z","iopub.status.idle":"2023-07-16T18:23:20.352482Z","shell.execute_reply.started":"2023-07-16T18:23:20.342815Z","shell.execute_reply":"2023-07-16T18:23:20.351213Z"},"trusted":true},"execution_count":null,"outputs":[]}]}