{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Progress in Transformer Based Language Model

\n\n

Author: Agastya Kumar

\n

Category: text_data

\n","metadata":{"_uuid":"8217c3ee-21f0-4850-ba60-cdea107c5e95","_cell_guid":"aab6b99a-23a1-4c2c-93d8-4a5f57e3613a","jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-07-04T16:22:11.867678Z","iopub.execute_input":"2023-07-04T16:22:11.868056Z","iopub.status.idle":"2023-07-04T16:22:11.913148Z","shell.execute_reply.started":"2023-07-04T16:22:11.868026Z","shell.execute_reply":"2023-07-04T16:22:11.911667Z"}}},{"cell_type":"markdown","source":"Until recently common wisdom among people about AI was that while computers can be better than humans when it comes to data-driven decision making, it has no competition on cognitive and creative ones. That, however, is changing.In the past few years language-based AI has advanced by leaps and bounds and a remarkable progress is the launch of ChatGPT and BARD (powerful AI chatbots developed based on LLMs), which has attracted widespread attention from society.Considering this rapid technical progress, in this notebook, we present the recent advances of transformer based model particulary LLMs.","metadata":{}},{"cell_type":"markdown","source":"# Table of Contents\n\n* [Introduction](#section1)\n* [Pretraining data](#section2)\n* [Architecture Configuration of LLMs](#section3)\n* [Parameter-Efficient Fine-Tuning Methods](#section4)\n* [Important LLMs](#section5)\n* [Open Source LLMs Model](#section6)\n* [LLMs Reasoning Ability](#section7)\n* [LLMs are misaligned](#section8)\n* [API-based Conversational LLMs](#section9)\n - [ChatGPT](#sub-section9.1)\n - [GPT-4](#sub-section9.2)\n - [BARD](#sub-section9.3)\n* [Decoding-enhanced BERT with disentangled attention (DeBERTa) Models](#section10)\n - [DebertaV1](#sub-section10.1)\n - [DebertaV2](#sub-section10.2)\n - [DebertaV3](#sub-section10.3)\n* [Popular techniques used in recent NLP Kaggle competitions:](#section11)\n* [Conclusion](#section12)\n* [References](#section13)","metadata":{}},{"cell_type":"markdown","source":"\n## Introduction\n\nNLP has evolved greatly over the years from the early days of rule-based systems and statistical methods, through the development of neural networks and word embeddings, to the breakthroughs in Transformer-based architectures and pre-trained models, the progress has been astounding.\nIn 2017 **\"Attention is All You Need\"** [1] research paper published by Google AI introduced the Transformer architecture, and revolutionized the way NLP research was progressing.It was the first time the concept of transformers was referenced. Before this paper, RNN and CNN were used in the field of NLP but they had two problems:\n* Dealing with long term dependencies\n* No parallelization during training\n\nSince then, almost all the nlp research shifted towards exploring different facets of transformers (as indicated by the staggering number of citations of the original paper, i.e. 80,545 citations at the time of writing).\nIn the next year **transfer learning** approach was proposed to capture context-aware word representations by first pre-training a bidirectional LSTM [2] (biLSTM) network (instead of learning fixed word representations) and then fine-tuning the biLSTM network to specific downstream tasks. Further, based on the encoder part of Transformer architecture [2] BERT [3] was proposed by pretraining bidirectional language models with specially designed pre-training tasks on large-scale unlabeled corpora.\n\nAfter the BERT was introduced in 2018 many other encoder only models such as Roberta[4], XLM[5], Deberta[6], T5[7],BART[8] are trained with Masked Language Model(encoder) or span corruption(encoder-decoder) objective on a very large corpus of text and then finetuned to adapt to one or more specific tasks.Even though these models have achieved near state of the art across thousands of natural language tasks, the problem with these models are that they require a significant number of task-specific training examples to finetune the model which is quite expensive.\n\nIn the same year when BERT was introduced OpenAI released their first unidirectional GPT(generative pre-training) model based on decoder part of transformer architecture.Following a similar architecture of GPT-1,GPT-2 was released in 2019 with 1.5B parameters with the objective of next token prediction given the previous tokens.\n\nFollowing the scaling law [9] researchers find that scaling language models (e.g., scaling model size both in depth and width or increasing the data size) often leads to an improved model capacity on downstream tasks.GPT-3 [10] the first model successfully applied the scaling principle having 175 billion parameters and trained on massive 500 billion tokens.GPT-3 also shows that extremely large autoregressive language models (LMs) can be used for zero shot or few-shot predictions whereas previous pretrained language models(PLM) does not perform well .Thus,the research community has coined the term large language models (LLM) for these Pre-trained Language Models (PLMs) of significant size (e.g.,containing tens or hundreds of billions of parameters).The success of GPT-3 has turned the research direction towards decoder only model and encoder-only models gradually begin to fade away(Fig 1).\n\nIn the literature [11], emergent abilities of LLMs are formally defined as “the abilities that are not present in small models but arise in large models”,which is one of the most prominent features that distinguish LLMs from previous PLMs.\n\nAfter GPT-3 large number of autoregressive models were developed as can be seen from Fig 2.The most powerful of these post-GPT-3 models are GLaM [12], Gopher [13],Jurassic [14] Megatron–Turing NLG [15], Chinchilla [16], PaLM [17], LLaMA [18] all of which achieved few-shot state-of-the-art results on a significant number of tasks at the time of their release.All these model get performance improvement mainly from scaling either model or data size or both.There are also models like Language Model for Dialogue Applications(LaMDA for dialogue)[19], Galactica [20] (for science knowledge), Codex [21] (for code generation), Sparrow [22] (chatbot model finetuned from chinchilla), BloombergGPT [23] (for financial domain) that are optimized for specific tasks. These models exhibit strong capacities to understand natural language and solve complex tasks (via text generation). \n\n
\n
\n \n
\n
\n\nFig 1: The evolutionary tree of modern LLMs traces the development of language models in recent years and highlights some of the most well-known models. Models on the same branch have closer relationships. Transformer-based models are shown in non-grey colors: decoder-only models in the blue branch, encoder-only models in the pink branch, and encoder-decoder models in the green branch. The vertical position of the models on the timeline represents their release dates. Open-source models are represented by solid squares, while closed-source models are represented by hollow ones. The stacked bar plot in the bottom right corner shows the number of models from various companies and institutions. [Source](https://arxiv.org/pdf/2304.13712.pdf)\n\n\n## **Pretraining data**:\nPre-training data plays a pivotal role in the development of large language models. As the foundation of remarkable\ncapabilities of LLMs, the quality, quantitative, and diversity of pre-training data influence the performance\nof LLMs significantly. For example, PaLM and BLOOM excel in multilingual tasks and machine translation with an\nabundance of multilingual pretraining data. Moreover, PaLM’s performance in Question Answering tasks is enhanced\nby incorporating a considerable amount of social media conversations and Books corpus. Likewise, code execution\nand code completion capabilities of GPT-3.5 [24] (code-davinci-002) are amplified by the integration of code data in its pretraining dataset.ChatGPT and BARD perform significantly better in conversations as there pretraining data consist of dialogues and prompts. ","metadata":{}},{"cell_type":"markdown","source":"## **Architecture Configuration of LLMs**\nTransformers configuration has 3 major components from which it is built : position embeddings,activation functions, and attention and bias.With the recent advances in LLMs these parts have been studied thoroughly\nand new techniques are proposed to train transformers efficiently:\n\n**Position Embeddings**: Apart from Absolute and Relative position embedding which are commonly used in pre-trained language model and few LLMs. Two new position embeddings are proposed:\n* **Rotary Position Embedding** [25]. RoPE embeddings incorporate both absolute and relative positioning by:1.Encoding the absolute position with a rotation matrix 2.Incorporating relative position directly into self-attention.Due to the excellent performance and the longterm decay property, RoPE is widely adopted in the latest LLMs, e.g., PaLM [56] and LLaMA [57].\n* **ALiBi**. ALiBi [26] is similar to relative position embedding, in terms of calculating biases attention score but the penalty scores in ALiBi are pre-defined without any trainable parameters.Results shows that ALiBi has a better extrapolation performance on sequences longer than some fixed length and it also improves the training stability in BLOOM[27].\n\n**Activation Functions** :GeLU activations are widely used is existing LLMs but recently LLMs like PaLM and LaMDA uses two new variants of GeLU namely SwiGLU and GeGLU which often achieve better performance in practice.\n\n**Attention and Bias** : Attention and biases are one of the most critical components in transformers that allows the tokens across the sequence to interact with each other and compute the representations of the input and output sequence.Aside from Full attention and sparse attention which are there from many years following new attention were proposed:\n* **Multi-query attention** - Multi-query attention just shares key and value vectors between each of the attention heads, instead of performing a separate projection for each head. This change does not make training any faster, but it does significantly improve the auto-regressive decoding (i.e., used to perform inference or generation) efficiency of LLMs.\nRepresentative models with multiquery attention include PaLM [17] and StarCoder [28]\n* **Flash Attention** : Flash Attention [29] is proposed to optimize the speed and memory consumption of\nattention modules on GPUs.FlashAttention organizes the input into blocks and introduces necessary recomputation, both to make better use of the fast memory.","metadata":{}},{"cell_type":"markdown","source":"## Parameter-Efficient Fine-Tuning Methods\nSince LLMs consist of a huge amount of model parameters, it would be costly to perform the fullparameter tuning. In this section, we will discuss how to conduct efficient tuning on LLMs.we briefly review four parameter-efficient fine-tuning methods for Transformer language models, including adapter tuning, prefix tuning, prompt tuning and LoRA.\n\n**Adapter Tuning** [30]. Adapter tuning incorporates small neural network modules (called adapter) into the Transformer models. To implement the adapter module, a bottleneck architecture has been proposed in [31, 32], which first\ncompresses the original feature vector into a smaller dimension (followed by a nonlinear transformation) and then\nrecovers it to the original dimension. The adapter modules would be integrated into each Transformer layer, typically\nusing a serial insertion after each of the two core parts (i.e.,attention layer and feed-forward layer) of a Transformer layer.\n\n**Prefix Tuning**. Prefix tuning [33] prepends a sequence of prefixes, which are a set of trainable continuous vectors, to\neach Transformer layer in language models. These prefix vectors are task-specific, which can be considered as virtual\ntoken embeddings. To optimize the prefix vectors, a reparameterization trick [34] has been proposed by learning a\nMLP function that maps a smaller matrix to the parameter matrix of prefixes, instead of directly optimizing the prefixes. \n\n**Prompt Tuning**. Different from prefix tuning, prompt tuning [35, 36] mainly focuses on incorporating trainable\nprompt vectors at the input layer. Based on the discrete prompting methods, it augments the input text\nby including a group of soft prompt tokens (either in a free form [37] or a prefix form [38]), and then takes\nthe prompt-augmented input to solve specific downstream tasks.\n\n**Low-Rank Adaptation (LoRA)**. LoRA [39] imposes the low-rank constraint for approximating the update matrix at\neach dense layer, so as to reduce the trainable parameters for adapting to downstream tasks. Consider the case of\noptimizing a parameter matrix W. The update process can be written in a general form as: W ← W + ∆W.LoRA has been widely applied to open-source LLMs LLaMA and BLOOM for parameter-efficient fine-tuning.","metadata":{}},{"cell_type":"markdown","source":"\n## Important LLMs \nThere are few recent LLMs that brings major development in research:\n\n**Chinchilla** : Chinchilla breaks the tired trend of building bigger and bigger model with keeping training tokens approximately same, and proposed a new scaling law that argue both models and data sizes should be increased in equal scales.With only 70B parameters and 4 times more more data(approx 1.4 trillion tokens). Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks.\n\n\n**LLaMA(Large Language Model Meta AI)**: LLaMA released in Feb 2023 by Meta AI is a collection of state-of-the-art foundation language models from ranging from 7B to 65B parameters. Since released, LLaMA has attracted extensive attention from both research and industry communities.These models are trained using publicly available datasets with 1.4 trillion tokens for largest and 1 trillion token for smallest model(Table 1).These models outperform GPT-3(175B) on most benchmark.A huge number of researchers have extended LLaMA models by either instruction tuning(fine-tuning with a mixture of multi-task datasets on pretrained LLMs) or continual pretraining.\n\n\n**Pathways Language Model (PaLM)**: It a 540 billion parameter LLM trained using Google’s novel **Pathways** [40],a new ML system which enables highly efficient training of very large neural networks across thousands of accelerator chips and PaLM is the first large scale use of Pathways framework. By eliminating pipeline parallelism, this architecture achieves impressive training throughput, allowing PaLM to be pre-trained over a more extensive dataset which was not possible before. The model achieves state-of-the-art few-shot results across hundreds of natural language, code and difficult mathematical reasoning tasks. \nThe PaLM pretraining dataset consists of a high-quality corpus of 780 billion tokens that represent a wide\nrange of natural language use cases. The dataset is a mixture of filtered webpages,2 books, Wikipedia, news\narticles, source code, and social media conversations. This dataset is based on the datasets used to train\nLaMDA [19] and GLaM [12].\n\n\n\n**Pathways Language Model 2 (PaLM 2)** [41]:PaLM 2, a new state-of-the-art language model and the model behind BARD success.It has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 incorporates the following diverse set of recent research advances:\n* **Compute-optimal scaling**: Recently, compute-optimal scaling [21] showed that data size is at least as important as model size. PaLM 2 researchers validated this study for larger amounts of compute and similarly find that data and model size should be scaled roughly 1:1 to achieve the best performance for a given amount of training compute (as opposed to past trends, which scaled the model 3× faster than the dataset).\n* **Training objective improvements**: Instead of using a single causal or masked language modeling objective like other LLM PaLM 2 uses recently proposed new pre training method UL2[42]. UL2 is a unified framework for pretraining models that are universally effective across datasets and setups. UL2 uses Mixture-of-Denoisers (MoD), apre-training objective that combines diverse pre-training paradigms together.\nPaLM 2 uses more multilingual and diverse pre-training mixture, which extends across hundreds of languages and domains (e.g., programming languages, mathematics, and parallel multilingual documents). \n\nThe largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses\nmore training compute.Evaluation results shows that PaLM 2 models significantly outperform PaLM on a variety\nof tasks, including natural language generation, translation, and reasoning. These results suggest that model scaling\nis not the only way to improve performance. Instead, performance can be unlocked by meticulous data selection\nand efficient architecture/objectives. ","metadata":{}},{"cell_type":"markdown","source":"\n
\n
\n \n
\n
\n\nTable 1: Statistics of large language models [Source](https://arxiv.org/pdf/2303.18223.pdf)","metadata":{}},{"cell_type":"markdown","source":"## Open Source LLMs Model\n\nLLMs exhibit a tendency towards closed-sourcing. In the early stages of LLM development (before 2020), the\nmajority of models were open-sourced. However, with the introduction of GPT-3, companies have increasingly opted to close-source their models, such as PaLM, LaMDA, and GPT-4. Consequently, it has become more difficult\nfor academic researchers to conduct experiments on LLM training. Given the huge cost of model pre-training, well-trained model checkpoints are critical to the study and development of LLMs for the research community.Since the parameter scale is a key factor to consider for using LLMs, many researchers categorize these LLMs into two scale levels (i.e., tens of billions of parameters and hundreds of billions of parameters)\n\n**Models with Tens of Billions of Parameters**: Most of the models in this category have a parameter scale ranging from\n10B to 20B, except LLaMA (containing 65B parameters in the largest version), NLLB [43] (containing 54.5B parameters in the largest version), and Falcon [44] (containing 40B parameters in the largest version). Models within this range include mT5 [45], PanGu-α [46], T0 [47], GPTNeoX-20B [48], CodeGen [49], UL2 [42], Flan-T5 [50], and mT0 [51]. Among them, Flan-T5 (11B version) can serve as a premier model for research on instruction tuning, since\nit explores the instruction tuning from three aspects [54].Instruction tuning LLaMA has become a major approach to developing customized or specialized models, due to the relatively low computational costs.Instruction-tuned models (i.e., Vicuna[52], Alpaca[53] and ChatGLM[54]) mostly perform better than non-instruction-tuned\nmodels (i.e., LLaMA[19], Pythia[55] and Falcon[56]).However, after instruction tuning, Vicuna-7B and\nAlpaca-7B suffer from performance degradations on LAMBADA, a language modeling task. The reason may be that\nthe instruction data primarily focuses on enabling LLMs to\nfollow human instructions, which is not always useful for\nthe general language generation task.\n\n**Models with Hundreds of Billions of Parameters**: For models in this category, only a handful of models have been\npublicly released. For example, OPT [57], OPT-IML [58],BLOOM [40], and BLOOMZ [59] have nearly the same number of parameters as GPT-3 (175B version), while GLM [60] and Galactica [61] have 130B and 120B parameters, respectively\n\nWhen considering contributions to the open-source community, particularly those related to LLMs, Meta stands out as one of the most generous commercial companies, as all the LLMs developed by Meta are open-sourced.","metadata":{}},{"cell_type":"markdown","source":"## **LLMs Reasoning Ability**\n\nReasoning is a fundamental aspect of human intelligence that plays a crucial role in activities such as problem solving, decision making,and critical thinking. Recent research [62] shows that\nreasoning ability appears to emerge only in very large language models like GPT-3 175B,PaLM 540B,GPT-4 as evidenced by significant improvements in performance on reasoning tasks at a certain scale (e.g., 100 billion parameters).Several prompting based strategy are proposed to boost the performance of LLMs on complex reasoning tasks, such as arithmetic reasoning, commonsense reasoning, and symbolic reasoning. like Chain-of-Thought(CoT),Self-consistency CoT Prompt and Tree of Thoughts(ToT) prompt:\n\n**Chain-of-Thought (CoT) prompts** [63] [Source Code](https://github.com/FranxYao/chain-of-thought-hub)\n\nChain of thought reasoning is a simple but powerful technique that allows models to decompose complex problems into intermediate steps that are solved individually. The use of chain-of-thought (CoT) prompts improve the performance of LLMs on various reasoning tasks,Specifically, in CoT prompting,⟨input, output⟩ demonstrations are replaced with ⟨input, chain of thought, output⟩ triples, e.g.,[input] Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? [chain of thought] Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. [output] The answer is 11.” In this way, given a target question, the model learns to generate explicit rationale before producing the final answer.\n\nAuthor of the paper conducted experiments on two different sets of task i.e Arithmetic and commonsense reasoning with LaMDA models (422M to 137B parameters), and PaLM models (8B to 540B parameters).chains of thought is manually created to include in the examples for chain of thought prompting.They found that chain of thought prompting substantially outperforms standard prompting for large model sizes > 100 billion.\n\n
\n
\n \n
\n
\n\nFig 4: Chain-of-thought reasoning processes are highlighted. Source [Wei et al. (2022)](https://arxiv.org/pdf/2201.11903.pdf)\n\n\n**Self-consistency CoT Prompting**[64]\n\nIt is an advanced prompting technique that builds upon CoT prompting.This approach is inspired by human experience in a way that if humans reach to the same answer by thinking multiple distinct possible ways then he is more confident in any one way that final answer is correct.Self consistency method contains three steps:1 prompt a large language model using chain-of-thought (CoT) prompting; (2) replace the “greedy decode” in CoT and develop various diverse set of approaches to solve the problem.Each method should be unique (3) analyze the reasoning paths and choose the most consistent and accurate response as the final answer.\n\nTo compare with CoT method the author choose 4 models UL2(20 billion), LaMDA(137 billion), PaLM(540 billion), GPT-3(175 billion) parameters and evaluated the model on same set of prompts as in [62].Self-consistency improves the arithmeticreasoning performance over all four language models significantly over chain-of-thought prompting.\nThe gains become more significant when the language model’s scale increases,\n\n
\n
\n \n
\n
\n\nFig 5: The Self consistency-CoT method Source [Original Paper](https://arxiv.org/pdf/2203.11171.pdf)\n\n\n**Tree of Thoughts(ToT) prompts** [65] [Source code](https://github.com/dave1010/tree-of-thought-prompting)\n\nThe \"Tree of Thoughts\" (ToT) framework is a great approach that allows language model to take correct decision by considering multiple different reasoning paths and self-evaluating choices to decide the next action.ToT does 4 things\n**Thought Decomposition**- As the name suggest that in this part we only divide the task in our hand into various steps.Here we do not use thoughts in any steps\n**Thought generator**. So after defining the task for each step we now actually generate the thoughts.thoughts can be generated in two ways sample and propose.\nSample: we repeat the generation process k times independently.\npropose: generate k thoughts at one inference. So, these k thoughts may not be independent.\n**State evaluator**.Here, we define a state evaluation function: v(s). To expand the tree, we use this function to find the good path, like in chess programming.\n**Search algorithm**. In this step we try to expand the tree. For each leaf node, we evaluate it with the state evaluation function. To choose which leaf node for evaluation, we use a search algorithm. It could be a breadth-first search and a depth-first search.\n\nThe ToT framework combines language-based capability to generate and evaluate diverse thoughts with search algorithms, such as breadth-first search (BFS) or depth-first search (DFS), which allow systematic exploration of the tree of thoughts with lookahead and backtracking.\nEmpirical experiments[65] show that ToT significantly enhances language models' problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords.\n\n
\n
\n \n
\n
\n\nFig 6: Schematic illustrating various approaches to problem solving with LLMs. Each rectangle box represents a thought, which is a coherent language sequence that serves as an intermediate step toward problem solving.[Image Source](https://arxiv.org/pdf/2305.10601.pdf)","metadata":{}},{"cell_type":"markdown","source":"## **LLMs are misaligned**\n\nLarge language models (LLMs) can perform range of NLP tasks but they often express unintended behaviors such as making up facts, generating biased or toxic text, or simply not following user instructions.This is possibly because of the objective (next token prediction) they are trained on are differnet from the objective “follow the user’s instructions helpfully and safely” [66]. \n\nReinforcement Learning is quite complex to work with, and therefore, mostly confined to gaming and simulated environments like Atari or MuJoCo.But recently researchers started to solve the misalignemnt problem mainly through Human Feedback([67], [68])(involving humans in the training loop).InstructGPT uses an effective technique RLHF(reinforcement learning with human feedback) in tuning model,and ChatGPT represents the first use case of this technique for a model put into production.OpenAI describes their approach to alignment research in a technical article [69], which has summarized three promising directions: “training AI systems to use human feedback, to assist human evaluation and to do alignment research","metadata":{}},{"cell_type":"markdown","source":"\n## **Conversational LLMs**\n\n\"Picture an AI that truly speaks your language — and not just your words and syntax. Imagine an AI that understands context, nuance, and even humor.This is no longer just a futuristic concept — it's the reality of recently released **ChatGPT** and **BARD**\" - [forbes](https://www.forbes.com/sites/bernardmarr/2023/05/19/a-short-history-of-chatgpt-how-we-got-to-where-we-are-today/?sh=22fea8ed674f)\n\n\n\n## ChatGPT\n\nOne of the most exciting advancements in technology is the introduction of ChatGPT. ChatGPT is the large language model developed by OpenAI and released on 30 November 2022.It is based on the GPT (Generative Pre-trained Transformer) architecture, specifically GPT-3.5, which was the latest versions of the GPT series when it released. ChatGPT can generate text with remarkably high precision, detail, and coherence.\n\nIt is one of the most trending and viral Artificial Intelligence(AI) product.According to OpenAI, ChatGPT acquired 1 million users in just 5 days after launching and 100 million users in just two months making it the fastest-growing application in history.For comparison, it took nine months for TikTok to reach 100 million users.\n\nThe creators have used a combination of both Supervised Learning and Reinforcement Learning to fine-tune ChatGPT, but it is the Reinforcement Learning from Human Feedback (RLHF) that makes ChatGPT unique.RLHF enhance the performance of these models by minimizing harmful, untruthful, and/or biased outputs and making them more capable of generating human-like responses.\n\nBut exactly how ChatGPT use human feedback to attack the alignment problem?","metadata":{}},{"cell_type":"markdown","source":"
\n
\n \n
\n
\n\nFig 10: Different steps to train ChatGPT : [ChatGPT](https://openai.com/blog/chatgpt)","metadata":{}},{"cell_type":"markdown","source":"**Step 1: Supervised Fine-Tuning (SFT) model**\nChatGPT is trained using same methods as InstructGPT(GPT 3.5) [70] but with a slight differences in the data collection\nIn the first step an initial model is trained using supervised fine-tuning.Then a list of prompts is selected \nand a group of human labelers are asked to write down the expected output response. Two different sources of prompts have been used: some have been prepared directly from the Human AI trainer taking help from above trained model to compose their responses, some have been sampled from OpenAI’s API requests (i.e. from their GPT-3 customers).Finally both dataset is combined and SFT model is fine-tuned from a model in the GPT-3.5 series.\n\nDue to the limited amount of data for this step, the SFT model obtained after this process is likely to output text which is still (probabilistically) not very user-attentive and generally suffers from misalignment, in the sense explained in the above sections. The problem here is that the supervised learning step suffers from high scalability costs.\n\nTo overcome this problem, instead of asking human labelers to create a much bigger curated dataset, a slow and costly process, the strategy is now to have the labelers rank different outputs of the SFT model to create a reward model –let’s explain this in more detail in the following section.\n\nWe trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup. We trained an initial model using supervised fine-tuning: human AI trainers provided conversations in which they played both sides—the user and an AI assistant. We gave the trainers access to model-written suggestions to help them compose their responses. We mixed this new dialogue dataset with the InstructGPT dataset, which we transformed into a dialogue format.\n\n**Step 2: The reward model (RM)**\nIn this step reward model is created for Reinforcemnet learning.A random list of prompts is selected and the SFT model generates multiple outputs (anywhere between 4 and 9) for each prompt.Labelers rank the outputs from best to worst.This produces KC2 comparisons for each prompt shown to a labeler . Since comparisons are very correlated within each labeling task all KC2 comparisons from each prompt is trained as a single batch element as this avoids overfitting and is also more computationally efficient.The result is a new labeled dataset, where the rankings are the labels.\n\nNow the RM model will be trained.\nStarting from the SFT model with the final unembedding layer removed,model is trained to take in a prompt and response, and output a scalar reward.Cross-Entropy loss is used, with the comparisons as labels — the difference in rewards represents the log odds that one response will be preferred to the other by a human labeler.\n.\nSpecifically, the loss function for the reward model is\n\n
\n
\n \n
\n
\n\n\nwhere rθ(x, y) is the scalar output of the reward model for prompt x and completion y with parameters\nθ, yw is the preferred completion out of the pair of yw and yl\n, and D is the dataset of human comparisons.\n\n**Step 3: Fine-tuning the SFT model via Proximal Policy Optimization (PPO)**\nReinforcement Learning is now applied to fine-tune the SFT policy by letting it optimize the reward model. The specific algorithm used is called Proximal Policy Optimization (PPO) [61].\n\nBefore going further lets see what exactly is PPO:\n\n* PPO is an algorithm that is used to train agents in reinforcement learning. It is called an \"on-policy\" algorithm because it learns from and updates the current policy directly, rather than learning from past experiences as in \"off-policy\" algorithms like DQN (Deep Q-Network). This means that PPO is continuously adapting the current policy based on the actions that the agent is taking and the rewards it is receiving.\n* PPO uses a trust region optimization method to train the policy, which means that it constrains the change in the policy to be within a certain distance of the previous policy in order to ensure stability. This is in contrast to other policy gradient methods which can sometimes make large updates to the policy that can destabilize learning.\n\nIn the final step,SFT model once again fine tuned using PPO on bandit environment. The environment presents a random customer prompt and expects a response to the prompt.Given the prompt and response, it produces a reward determined by the reward model and ends the episode.A per-token KL penalty is added from the SFT model at each token to mitigate over optimization of the reward model.The value function is initialized from the RM and the PPo model is initialized from the SFT model.\n\nFollowing objective function was optimized during RL training:\n\n
\n
\n \n
\n
\n\n\n\n\nwhere πRLϕ is the learned RL policy, πSFT is the supervised trained model, and Dpretrain is the pretraining distribution. The KL reward coefficient, β, and the pretraining loss coefficient, γ, control the strength of the KL penalty and pretraining gradients respectively.\n\n**Step 4 Iteration**: Several iterations of the above process is performed and the best model is selected based on the holdout validation set comprising of labelers not seen in training data.\n\nLimitations of ChatGPT:\n1.ChatGPT sometimes generate hallucinations, which are texts that seem plausible but may be factually incorrect.\n2.ChatGPT is sensitive to tweaks to the input phrasing. For example for a given prompt ChatGPT may claim to not know the answer but few tweaks in the prompt may give right answer.\n3.Due to the bias in training data ChatGPT repeats certain phrases many times like it is a language model trained by OpenAI.\n4.If ChatGPT receives ambiguous query then instead to clarifying question it understand what the user intended.\n5 ChatGPT lacks knowledge of events that occur after 2021 since it does not have access to real time information.\n\n\n## GPT-4\nGPT-4[71] ,the latest milestone by OpenAI.It is a large multimodal model (accepting image and text inputs, emitting text outputs) that, while less capable than humans in many real-world scenarios, exhibits human-level performance on various professional and academic benchmarks.Although OpenAI did not share the training data on which GPT-4 was trained but the training method remains the same as that of ChatGPT as discussed above. OpenAI spent 6 months iteratively aligning GPT-4 using lessons from their adversarial testing program as well as ChatGPT, resulting in best-ever results (though far from perfect) on factuality, steerability, and refusing to go outside of guardrails.According to OpenAI GPT-4 is 82% less likely to respond to requests for disallowed content and 40% more likely to produce factual responses than GPT-3.5 on our internal evaluations .\n\nGPT-4 has a maximum token limit of 32,000 (equivalent to 25,000 words), which is a significant increase from GPT-3.5’s 4,000 tokens (equivalent to 3,125 words).GPT-4 is accessible through ChatGPT Plus, a premium service from OpenAI for which users have to pay $20.\n\nIn a casual conversation, the distinction between GPT-3.5 and GPT-4 can be subtle. The difference comes out when the complexity of the task reaches a sufficient threshold—GPT-4 is more reliable, creative, and able to handle much more nuanced instructions than GPT-3.5.\nFollowing table compares the performance of GPT-4 on traditional benchmarks designed for machine learning models against GPT 3.5 and other SOTA models:\n\n
\n
\n \n
\n
\n\nTable 3. Performance of GPT-4 on academic benchmarks.[Source](https://cdn.openai.com/papers/gpt-4.pdf) \n\n\n\n## **BARD**\n\nSeeing the unprecedented success of ChatGPT, Google launched its own chatbot in March 2023 in limited capacity and later to worldwide.Google Bard[72] is a conversational generative AI chatbot developed by Google, based initially on the LaMDA family of large language models and later the PaLM 2.Google has not disclosed the exact size of the training dataset, but it is estimated to be in the trillions of words since it is based on PaLM 2.\n\nThe training data for Google Bard is constantly being updated, which helps Bard to learn and improve over time. Bard is also able to access and process real-time information from the web. This allows Bard to stay up-to-date on current events and to provide users with the most accurate information possible.It also cites the sources from where the information is shown.BARD also has suffers the same limitations except that its knowledge is up to date unlike ChatGPT.","metadata":{}},{"cell_type":"markdown","source":"## **Practical Evaluation of LLMs**\nTo examine the effectiveness and superiority of LLMs, a\nsurge of tasks and benchmarks have been proposed for\nconducting empirical ability evaluation and analysis.\nHere, we mainly focus on four basic types of ability evaluation for LLMs, i.e.,language understanding, language generation, knowledge utilization, and complex reasoning.\n\n1.**Traditional NLU tasks** -\nTraditional NLU tasks are some fundamental tasks in NLP including text classification, named entity recognition (NER),\nentailment prediction.In most natural language understanding tasks, such as tasks in **GLUE** and **SuperGLUE**,fine-tuned models like deberta, roberta, XLM still have better performance.This is possibly because encoder models are bidirectional trained with MLM objective that allows them to have better semantic understanding of text.Recently proposed Deberta is the state of the art model in various NLU task and we will discuss in detail the architecture in next section.\n\n2.**Natural Language Generation** - tasks like summarization, machine translation, question answering require models to have a comprehensive understanding of the input contents or requirements and a certain level of creativity. This is what LLMs excel at.LLMs are remarkably adept at code synthesis as well.Existing models uses Treebank [366], WikiText-103 [73],Pile [74] and LAMBADA [75] for evaluation and LLMs.Empirical studies [76, 77] show that LLMs bring substantial performance gains over the previous state-of-the-art methods on these evaluation datasets and follows the scaling law which means that scaling language models would improve the accuracy.\n\n3 **Knowledge-intensive tasks** - Knowledge-intensive NLP refers to the tasks that need a lot of experience, understanding, background information, real- world knowledge and skills.LLMs excel at knowledge-intensive tasks due to their massive real-world knowledge.Tasks such as NaturalQuestions [78], WebQuestions [79], and TriviaQA [80] Here also LLMs tends to follow the scaling law terms of both model size and data size [81] ,[82].Newly released GPT-4 outperforms existing models by a considerable margin in massive multitask language understanding (MMLU) a highly knowledge intensive task.(Table 3)\nAlthough LLMs have achieved key progress in capturing and utilizing knowledge information, they suffer from two major issues hallucination(looks currect but facturally wrong) and knowledge recency(unable to give answer beyond the training data).\n\n\n4.**Complex Reasoning Tasks** : Reasoning is an important ability in intelligent system to accomplish human level task .GPT-4 exhibits strong reasoning ability as it outperform all other models on ARC-C commonsense reasoning benchmark, which contains difficult questions in science exams from grade 3 to grade 9.GPT-4 has been close to the performance of 100% (96.3%) (Table-3)\nIt also shows strong performance in Arithmetic reasoning and problem solving also LLMs.Datasets used for evaluatiion are GSM8k [83], SVAMP [84] and AQuA [85] and GPT-4 overperforms any other methods [86], even some huge models particularly tuned for arithmetic problems [87].Without the intervention of external tools, LLMs may occasionally\nmake mistakes in performing basic calculations, prompt based technique as discussed above can significantly\nimprove LLMs’ ability in calculations.","metadata":{}},{"cell_type":"markdown","source":"\n\n# Decoding-enhanced BERT with disentangled attention (DeBERTa) Models\nAlthough all research is mainly focused on improving LLMs(decoder only) but there are many NLP task like text classification, sentiment analysis where encoder models are still better.Deberta is the state of the art model in many NLP benchmark(Table 2).There are 3 models in deberta family Deberta[88], DebertaV2[89], DebertaV3[90].\n\n## Debertav1\nDeBERTa improves all previous encoder only models using two novel techniques\n### **Disentangled Attention**\n\nThe proposed disentangled attention mechanism differs from all existing approaches in that each input word is represented using two separate vectors that encode a word’s content and position, respectively. The key difference is that instead of adding the two vectors we treat them separately throughout the network.\nThese relative position embeddings are shared across all the layers of the transformer as inputs and the self-attention layer not only uses the query and key transformation on the context embeddings but also has separate matrices for the transformation for these positional embeddings.\n\n
\n
\n \n
\n
\n\nFig 7: [The architecture of DeBERTa.](https://www.microsoft.com/en-us/research/blog/microsoft-deberta-surpasses-human-performance-on-the-superglue-benchmark)\n\n\nFor a token at position i in a sequence is represented by two main components Hi and Pi∣j where H is the context vector and P is the relative position vector of i with respect to j. Then attention weight of a word pair (i, j) can be computed as a sum of four attention scores using disentangled matrices on their contents and positions as content-to-content, content-to-position, position-to-content, and position-to-position.\n\n\n\\begin{aligned}\nA_{i, j} &=\\left\\{\\boldsymbol{H}_{i}, \\boldsymbol{P}_{\\boldsymbol{i} \\mid j}\\right\\} \\times\\left\\{\\boldsymbol{H}_{\\boldsymbol{j}}, \\boldsymbol{P}_{\\boldsymbol{j} \\mid \\boldsymbol{i}}\\right\\}^{\\top} \\\\\n&=\\boldsymbol{H}_{\\boldsymbol{i}} \\boldsymbol{H}_{j}^{\\top}+\\boldsymbol{H}_{\\boldsymbol{i}} \\boldsymbol{P}_{\\boldsymbol{j} \\mid \\boldsymbol{i}}^{\\top}+\\boldsymbol{P}_{\\boldsymbol{i} \\mid \\boldsymbol{j}} \\boldsymbol{H}_{\\boldsymbol{j}}^{\\top}+\\boldsymbol{P}_{\\boldsymbol{i} \\mid \\boldsymbol{j}} \\boldsymbol{P}_{j \\mid \\boldsymbol{i}}^{\\top}\n\\end{aligned}\n\n\n1. The context-context portion essentially maps the semantic relation between the meaning of the two words i and j. \n2. The context-position portion is a map of how the context of the word i is related to the relative position vectors of the words around it. \n3. The position-context embedding is the map of how the relative position of the word i relates to the context of the word j. \n4. Finally, the position-position part of the overall attention would map the relation of position vectors for a word i to that of word j.Since relative position embedding is used everywhere, the position-to-position term does not provide much additional information and is removed from above equation.\n\n\n### **DeBERTa Absolute Position Incorporation in Decoding Layer**:\n\nFollowing the footsteps of BERT, DeBERTa uses the masked language modeling task (MLM) for its pre-training.The disentangled attention mechanism already considers the contents and relative positions of the context words, but not the absolute positions of these words, which in many cases are crucial for the prediction.\n\nGiven a sentence “a new store opened beside the new mall” with the words “store” and “mall”\nmasked for prediction. Using only the local context (e.g., relative positions and surrounding words)\nis insufficient for the model to distinguish store and mall in this sentence, since both follow the word\nnew with the same relative positions. To address this limitation, the model needs to take into account\nabsolute positions, as complement information to the relative positions.\n\nBERT ingests the absolute position in the input layer itself. To give the model some context of the actual positions, the authors add them right after all the transformer layers but before the softmax layer for masked token prediction. In this way, DeBERTa captures the relative positions in all the transformer layers and only uses absolute positions as complementary information when decoding the masked words. Thus, the name enhanced masked decoder.\n\n
\n
\n \n
\n
\n\nFig 8: Comparison of the decoding layer. Source: [The DebERTa paper](https://arxiv.org/abs/2006.03654) \n \n \n### **Scale Invariant FineTuning**\n\nIn addition to some major component changes, DeBerta authors also train the model using a new adversarial training algorithm, Scale Invariant FineTuning (SiFT).SiFT improves the training stability by applying the perturbations to the normalized word embeddings. Specifically, when fine-tuning DeBerta to a downstream NLP task, SiFT first normalizes the word embedding vectors into stochastic vectors and then applies the perturbations to the normalized embeddings vectors, this substantially improves the performance of the fine-tuned models.\n\nTheoretically Deberta has a max sequence length of 24528 that makes it a great choice for long sequences as opposed to RoBERTa.","metadata":{"_uuid":"10fa842c-4a98-4f3d-b59d-32fe949e262e","_cell_guid":"42fb83e4-5633-4040-b12c-a77daa4c3a74","trusted":true}},{"cell_type":"markdown","source":"\n## DeBERTa v2\n\nDeberta v2 is the scaled version of deberta. There are two variants one has 900M and other has 1.5B parameters.The significant performance boost due to scaling DeBERTa to a larger model makes the single DeBERTa1.5B surpass the human performance on SuperGLUE for the first time in terms of macroaverage score (89.9 versus 89.8). Deberta1.5B consists of 48 layers with a hidden size of 1,536 and 24 attention heads.To train such big models few optimization in model architectures were done which are summarized below:\n\n* Vocabulary: In v2 the tokenizer is changed to use a new vocabulary of size 128K built from the training data. Instead of a GPT2-based tokenizer, the tokenizer is now sentencepiece-based tokenizer.\n\n* nGiE (nGram Induced Input Encoding): The DeBERTa-v2 model uses an additional convolution layer along with the first transformer layer to induce n-gram knowledge of sub-word encodings and their outputs are summed up before feeding to the next Transformer layer. This enables the model to better learn the local dependency of input tokens.\n\n* Sharing of position projection matrices Wrq, Wrk with content projection matrices in all attention layers to reduce parameters Wcq, Wrk.\n\n* Apply bucket to encode relative positions The DeBERTa-v2 model uses a log bucket to encode relative positions similar to T5.","metadata":{"_uuid":"ebfb4a60-5a3b-47de-b433-6feeadaaccf4","_cell_guid":"74460261-5022-40c1-8cf9-331d1fdc26a4","trusted":true}},{"cell_type":"markdown","source":"\n## DeBERTa v3\nDeBERTaV3[68] improves the original model by trying two ways :\n\n**Replace Token Detection (RTD) trainng** : Following ELECTRA[69]-style training paradigm, DeBERTa v3 DeBERTa v3 has replaced the masked language model (MLM) pre-training task with a much more nuanced replace token detection task (RTD).\nRTD is a pretraining approach in which two transformer models are trained: the generator and the discriminator. The generator’s role is to replace tokens in a sequence, and is therefore trained as a masked language model. The discriminator, which is the model we’re interested in, tries to identify which tokens were replaced by the generator in the sequence.\n\n
\n
\n \n
\n
\n\nFig 9: An overview of replaced token detection(RTD). The generator is usually small masked language model is preferred that is trained jointly with the discriminator.After pre-training, we throw out the generator and only fine-tune the discriminator (the ELECTRA model) on downstream tasks. Source [ELECTRA paper](https://arxiv.org/pdf/2003.10555.pdf)\n\n**Gradient-Disentangled Embedding Sharing (GDES)** : In ELECTRA, the discriminator and the generator share the same token embeddings. However,researchers found that embedding sharing hurts training efficiency and model performance, since the training losses of the discriminator and\nthe generator pull token embeddings into opposite directions. This is because the training objectives\nbetween the generator and the discriminator are very different.To mitigate this new method called gradient-disentangled embedding sharing (GDES) was proposed where the generator shares its embeddings with the discriminator but stops the gradients from the discriminator to the generator embeddings.","metadata":{"_uuid":"508ac124-3a3b-4836-b640-c635cc374d9d","_cell_guid":"868fa203-fb9b-4b58-a5f4-7e4c77e0c424","trusted":true}},{"cell_type":"markdown","source":"The architecture and training change in deberta models significantly improve the efficiency of model pre-training and the performance of both natural language understand (NLU) and natural langauge generation (NLG) downstream tasks.Below table shows the results of deberta on various nlp benchmarks\n\n
\n
\n \n
\n
\n
\n\nTable 2: Comparison of DeBerta on various nlp benchmarks Source [DeBertaV3](https://arxiv.org/abs/2111.09543)\n","metadata":{"_uuid":"c0456ed4-c174-4ec0-bf3d-1ce20f2415eb","_cell_guid":"8b7957a0-876a-4121-9961-6e70377e19d5","trusted":true}},{"cell_type":"markdown","source":"## **Popular techniques used in recent NLP Kaggle competitions**:\nKaggle has seen an upsurge of NLP competitions in recent years.This is possibly due to the continuous advancemnts in NLP that led the companies and instituions to apply NLP in their domain.Some popular approach used are:\n* **Transfer learning** based approach is predominantly used as it performs very well.\n* Since Deberta release kaggle hosted 7 monolingual(english) NLP competitions where deep learning based solution is possible and DeBerta is the undisputed Kagglers first choice as it outperform all other models.This shows that deberta performs superior not only in major NLP benchmark but also in other datasets.Other models are also used for ensemble.For multilingual data XLM , mdeberta in [chaii - Hindi and Tamil Question Answering](https://www.kaggle.com/competitions/chaii-hindi-and-tamil-question-answering/discussion?sort=recent-comments)\n and other variants for task specific like bert for patent was used in [U.S. Patent Phrase to Phrase Matching](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion?sort=recent-comments) are mainly used.\n* **Adversarial training** methods like(Adversarial Weight Perturbation(AWP), Fast Gradient Method(FGM) ,SiFT) are getting popular as it shows better generalization and performance on unseen datasets.Almost all competition winners from [NBME - Score Clinical Patient Notes](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes),[Feedback Prize - Predicting Effective Arguments](https://www.kaggle.com/competitions/feedback-prize-english-language-learning), [Feedback Prize - English Language Learning](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/overview),[Feedback Prize - Evaluating Student Writing](https://www.kaggle.com/competitions/feedback-prize-2021) uses adversarial training.\n* Since labeling datasets is an expensive process many competitions only provide part of the datasets annotated.To exlore unannotated data various approaches like **pseudo labeling, meta-pseudo label** [2nd place solution](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/322832) are used.\n* Due to the multi-layer structure of Transformers, different layers capture different levels of representations.Many winners explore this by stacking last few layers and adding different heads like RNNs(LSTM, GRU)[1st solution](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/332243) , 1D-CNN,dense layer,or attention layer that help in improving the score during fine-tuning. \n* Recently competitons [Feedback Prize - Predicting Effective Arguments](https://www.kaggle.com/competitions/feedback-prize-english-language-learning), [Feedback Prize - Evaluating Student Writing](https://www.kaggle.com/competitions/feedback-prize-2021) introduced efficiency track which aims at reducing model and computational complexity.In these track knowledge distillation was mostly used along with all techniques discussed above to squeeze last drop of point where objective is to train efficient model in terms of speed and given metric.","metadata":{}},{"cell_type":"markdown","source":"\n# **Conclusion**\n\nIn this essay, we have reviewed the recent progress in natural language processing, and introduced various prominent techniques,models and findings developed over the past few years.Firstly we discussed how the research has progressed over the years and set the background for PLMs and LLMs.Then we discussed how pretraining data affects the model followed by important models that helps research moving forward.ChatGPT and DeBerta are thoroughly discussed to get inner insight of what makes them so powerful.Large LLMs tends to perform better than small LLMs and mostly Large models are closed source hence API-based research could become the predominant method in the academic community in future.Even though we can see some traces of artificial general intelligence(AGI) in models like ( ChatGPT,GPT-4,BARD) by seeing their generation capability and performance on various tasks but still they are far from human intelligence because they lack a complete understanding of language and world .Particulary small LLMs tends to perform poor on complex reasoning task which is essential component for LLMs to perform like humans. Future progress in NLP would be in developing novel techniques that will shorten the gap between model and human intelligence. Infact OpenAI has published a technical article\nentitled “Planning for AGI and beyond”, which discusses the short-term and long-term plans to approach AGI so we are not too far away from seeing a machine mimicking humans.","metadata":{}},{"cell_type":"markdown","source":"\n# **References**\n\n[1].Wang, B. and Komatsuzaki, A. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\n[2].Su, J., Lu, Y., Pan, S., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding.\nCoRR, abs/2104.09864, 2021. URL https://arxiv.org/abs/2104.09864.\n\n[3].Shazeer, N. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019. URL https://arxiv.org/abs/1911.02150 .\n\n[4].Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Chung, H. W., Sutton, C., Gehrmann, S., Schuh,\nP., et al. PaLM: Scaling language modeling with Pathways. arXiv preprint arXiv:2204.02311, 2022. URL\nhttps://arxiv.org/abs/2204.02311 .\n\n[5].Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin.PaLM 2 Technical Report https://arxiv.org/abs/2305.10403 .\n\n[6].Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre-training\ntext encoders as discriminators rather than generators. https://arxiv.org/abs/2003.10555 .\n\n[7].He, P., Liu, X., Gao, J., Chen, W., et al 2021. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing https://arxiv.org/abs/2111.09543 .\n\n[8].He, P., Liu, X., Gao, J., Chen, W., Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, 2020. https://arxiv.org/abs/2006.03654 .\n\n[9].Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright. Training language models to follow instructions with human feedback 2022. https://arxiv.org/abs/2203.02155 .\n\n[10].OpenAI GPT-4 Technical Report 2023 arXiv preprint arXiv:1911.02150, https://arxiv.org/abs/2303.08774 .\n\n[11].Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., et al.Training compute-optimal large language models. NeurIPS, 2022. URL https://arxiv.org/abs/2203.15556 .\n\n[12].Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Bahri, D., Schuster, T., Zheng, S.,\nZhou, D., Houlsby, N., and Metzler, D. UL2: Unifying language learning paradigms. In The Eleventh International\nConference on Learning Representations, 2023. URL https://openreview.net/forum?id=6ruVLB727MC .\n\n[13].Barham, P., Chowdhery, A., Dean, J., Ghemawat, S., Hand, S., Hurt, D., Isard, M., Lim, H., Pang, R., Roy,\n.S., Saeta, B., Schuh, P., Sepassi, R., Shafey, L. E., Thekkath, C. A., and Wu, Y. Pathways: Asynchronous\ndistributed dataflow for ML. To appear in MLSys 2022, 2022. URL https://arxiv.org/abs/2203.12533 .\n\n[14].Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker,\nL., Du, Y., et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022.\nURL https://arxiv.org/pdf/2201.08239 .\n\n[15].Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan Language models are few-shot learners.In Advances in Neural Information Processing Systems, volume 33, pp. 1877–1901, 2020. URL \nhttps://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf .\n\n[16].Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Evaluating Large Language Models Trained on Code 2021 https://arxiv.org/abs/2107.03374 .\n\n[17].Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz\nKaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information\nprocessing systems, pp. 5998–6008, 2017.https://arxiv.org/abs/1706.03762 .\n\n[18].Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama,Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems,35:27730–27744, 2022.https://arxiv.org/pdf/2203.02155.pdf\n\n[19].Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems,\npages 4299–4307.https://arxiv.org/pdf/1706.03741.pdf\n\n[20].Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D.,\nand Christiano, P. (2020). Learning to summarize from human feedback. arXiv preprint\narXiv:2009.01325.https://arxiv.org/pdf/2009.01325.pdf\n\n[21].J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown,B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and\nD. Amodei, “Scaling laws for neural language models,” CoRR, vol. abs/2001.08361, 2020.https://arxiv.org/pdf/2001.08361.pdf\n\n[22].Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In NAACL.https://arxiv.org/pdf/1802.05365.pdf\n\n[23].Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI.https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf\n\n[24].Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111–3119. Curran Associates,Inc.https://arxiv.org/pdf/1310.4546.pdf\n\n[25].Sutskever, I., Vinyals, O., and Le, Q. (2014). Sequence to sequence learning with neural networks.In Advances in Neural Information Processing Systems (NIPS 2014).\n\n[26].Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly\nlearning to align and translate. CoRR, abs/1409.0473, 2014.\n\n[27].Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642,\n\n[28].S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023 https://arxiv.org/abs/2305.10601\n\n[29].Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou Self-Consistency Improves Chain of Thought Reasoning in Language Models In ICLR, 2023, https://arxiv.org/abs/2203.11171\n\n[30] Wu, Shijie, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. “BloombergGPT: A Large Language Model for Finance.” Arxiv.\n\n[31].https://towardsdatascience.com/palm-efficiently-training-massive-language-models-b82d6cc1582\n\n[32].https://openai.com/blog/chatgpt\n\n[33].https://openai.com/research/gpt-4\n\n[34].https://jrodthoughts.medium.com/instructgpt-is-one-of-the-models-behidn-the-magic-of-chatgpt-59813dd8aabc\n\n[35]OpenAI, “Our approach to alignment research,” OpenAI Blog, August 2022.\n\n[36]N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” in Proceedings of the 36th International Conference\non Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, 2019, pp. 2790–2799.\n\n[37]V. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, “Reducing activation recomputation in large transformer models,”\nCoRR, vol. abs/2205.05198, 2022.\n\n[38] T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagne, A. S. Luccioni, “BLOOM: A 176b-parameter open-access multilingual language model,” CoRR, vol. abs/2211.05100, 2022.\n\n[39] P. F. Christiano, J. Leike, T. B. Brown, M. Martic,S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” in Advances in Neural Information Processing Systems 30: Annual Conference on\nNeural Information Processing Systems 2017, December\n4-9, 2017, Long Beach, CA, USA\n\n[40] T. Schick, J. Dwivedi-Yu, R. Dessı, R. Raileanu,: Language models can teach themselves to use tools,” CoRR, vol. abs/2302.04761,2023.\n\n[41] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang,K. Button, M. Knight, B. Chess, and J. Schulman,\n“Webgpt: Browser-assisted question-answering with human feedback,” CoRR, vol. abs/2112.09332, 2021.\n\n[42] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang,M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring\nthe limits of transfer learning with a unified textto-text transformer,” J. Mach. Learn. Res., pp. 140:1–\n140:67, 2020.\n\n[43] L. Xue, N. Constant, A. Roberts, M. Kale, R. AlRfou, A. Siddhant, A. Barua, and C. Raffel, “mt5: A\nmassively multilingual pre-trained text-to-text transformer,” in Proceedings of the 2021 Conference of the\nNorth American Chapter of the Association for Computational Linguistics: Human Language Technologies,\nNAACL-HLT 2021, Online, June 6-11, 2021, 2021, pp.\n483–498.\n[44] O. O. Buruk, “Academic writing with GPT-3.5: reflections on practices, efficacy and transparency,” CoRR,\nvol. abs/2304.11079, 2023.\n\n[45] R. Liu and N. B. Shah, “Reviewergpt? an exploratory\nstudy on using large language models for paper reviewing,” CoRR, vol. abs/2306.00622, 2023.\n\n[46] M. Kosinski, “Theory of mind may have spontaneously emerged in large language models,” CoRR,\nvol. abs/2302.02083, 2023.\n\n[47] M. M. Amin, E. Cambria, and B. W. Schuller, “Will\naffective computing emerge from foundation models\nand general ai? A first evaluation on chatgpt,” CoRR\n,\nvol. abs/2303.03186, 2023.\n\n[48] G. Sridhara, R. H. G., and S. Mazumdar, “Chatgpt: A\nstudy on its utility for ubiquitous software engineering tasks,” CoRR, vol. abs/2305.16837, 2023.\n\n[49] W. Sun, C. Fang, Y. You, Y. Miao, Y. Liu, Y. Li, G. Deng,\nS. Huang, Y. Chen, Q. Zhang, H. Qian, Y. Liu, and\nZ. Chen, “Automatic code summarization via chatgpt:\nHow far are we?” CoRR, vol. abs/2305.12865, 2023.\n\n[50] C. S. Xia and L. Zhang, “Conversational automated\nprogram repair,” CoRR, vol. abs/2301.13246, 2023.\n\n[51] H. Cho, H. J. Kim, J. Kim, S. Lee, S. Lee, K. M. Yoo,\nand T. Kim, “Prompt-augmented linear probing: Scaling beyond the limit of few-shot in-context learners,”\nCoRR, vol. abs/2212.10873, 2022.\n\n[52] S. Shin, S. Lee, H. Ahn, S. Kim, H. Kim, B. Kim, K. Cho,\nG. Lee, W. Park, J. Ha, and N. Sung, “On the effect of\npretraining corpora on in-context learning by a largescale language model,” in NAACL-HLT. Association\nfor Computational Linguistics, 2022, pp. 5168–5186.\n\n[53] Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, “Efficient transformers: A survey,” ACM Comput. Surv.,\nvol. 55, no. 6, pp. 109:1–109:28, 2023.\n\n[54] S. Chen, B. H. Kann, M. B. Foote, H. J. Aerts, G. K.\nSavova, R. H. Mak, and D. S. Bitterman, “The utility\nof chatgpt for cancer treatment information,” medRxiv,\n2023.\n\n[55] K. Malinka, M. Peres´ıni, A. Firc, O. Hujnak, and\nF. Janus, “On the educational impact of chatgpt: Is\nartificial intelligence ready to obtain a university degree?” CoRR, vol. abs/2303.11146, 2023.\n\n[56] T. Susnjak, “Chatgpt: The end of online exam integrity?” CoRR, vol. abs/2212.09292, 2022.\n\n[57] K. Tan, T. Pang, and C. Fan, “Towards applying powerful large ai models in classroom teaching: Opportunities, challenges and prospects,” 2023.\n\n[58] F. Kamalov and I. Gurrib, “A new era of artificial\nintelligence in education: A multifaceted revolution,”\nCoRR, vol. abs/2305.18303, 2023.\n\n[59] E. Kasneci, K. Seßler, S. Kuchemann, M. Bannert, ¨\nD. Dementieva, F. Fischer, U. Gasser, G. Groh,\nS. Gunnemann, E. H ¨ ullermeier ¨ et al., “Chatgpt for\ngood? on opportunities and challenges of large language models for education,” Learning and Individual\nDifferences, vol. 103, p. 102274, 2023.\n\n[60] A. Blair-Stanek, N. Holzenberger, and B. V. Durme,\n“Can GPT-3 perform statutory reasoning?” CoRR, vol.\nabs/2302.06100, 2023.\n\n[61] D. Trautmann, A. Petrova, and F. Schilder, “Legal\nprompt engineering for multilingual legal judgement\nprediction,” CoRR, vol. abs/2212.02199, 2022.\n\n[62] J. H. Choi, K. E. Hickman, A. Monahan, and\nD. Schwarcz, “Chatgpt goes to law school,” Available\nat SSRN, 2023.\n\n[63] J. J. Nay, “Law informs code: A legal informatics\napproach to aligning artificial intelligence with humans,” CoRR, vol. abs/2209.13020, 2022.\n\n[64] F. Yu, L. Quartey, and F. Schilder, “Legal prompting:\nTeaching a language model to think like a lawyer,”\nCoRR, vol. abs/2212.01326, 2022.\n\n[65] D. Trautmann, A. Petrova, and F. Schilder, “Legal\nprompt engineering for multilingual legal judgement\nprediction,” CoRR, vol. abs/2212.02199, 2022.\n\n[66] A. Tamkin, M. Brundage, J. Clark, and D. Ganguli,\n“Understanding the capabilities, limitations, and societal impact of large language models,” CoRR, vol.\nabs/2102.02503, 2021.\n\n[67] A. Abid, M. Farooqi, and J. Zou, “Persistent antimuslim bias in large language models,” in AIES ’21:\nAAAI/ACM Conference on AI, Ethics, and Society, Virtual\nEvent, USA, May 19-21, 2021, M. Fourcade, B. Kuipers,\nS. Lazar, and D. K. Mulligan, Eds. ACM, 2021, pp.\n298–306.\n\n[68] A. Shah and S. Chava, “Zero is not hero yet: Benchmarking zero-shot performance of llms for financial\ntasks,” CoRR, vol. abs/2305.16633, 2023.\n\n[69] D. Araci, “Finbert: Financial sentiment analysis\nwith pre-trained language models,” CoRR, vol.\nabs/1908.10063, 2019.\n\n[70] J. C. S. Alvarado, K. Verspoor, and T. Baldwin, “Domain adaption of named entity recognition to support credit risk assessment,” in Proceedings of the\nAustralasian Language Technology Association Workshop,\nALTA 2015, Parramatta, Australia, December 8 - 9, 2015,\nB. Hachey and K. Webster, Eds. ACL, 2015, pp. 84–90.\n\n[71] G. Son, H. Jung, M. Hahm, K. Na, and S. Jin, “Beyond\nclassification: Financial reasoning in state-of-the-art\nlanguage models,” CoRR, vol. abs/2305.01505, 2023.\n\n[72] X. Zhang, Q. Yang, and D. Xu, “Xuanyuan 2.0: A large\nchinese financial chat model with hundreds of billions\nparameters,” arXiv preprint arXiv:2305.12002, 2023.\n\n[73] H. Yang, X.-Y. Liu, and C. D. Wang, “Fingpt: Opensource financial large language models,” CoRR, vol.\nabs/2306.06031, 2023.\n\n[74] Q. Jin, B. Dhingra, Z. Liu, W. W. Cohen, and X. Lu,\n“Pubmedqa: A dataset for biomedical research question answering,” in Proceedings of the 2019 Conference\non Empirical Methods in Natural Language Processing\nand the 9th International Joint Conference on Natural\nLanguage Processing, EMNLP-IJCNLP 2019, Hong Kong,\nChina, November 3-7, 2019, 2019, pp. 2567–2577.\n\n[75] A. Krithara, A. Nentidis, K. Bougiatiotis, and\nG. Paliouras, “Bioasq-qa: A manually curated corpus\nfor biomedical question answering,” 2022.\n\n[76] C. Zhang, C. Zhang, C. Li, Y. Qiao, S. Zheng, S. K.\nDam, M. Zhang, J. U. Kim, S. T. Kim, J. Choi, G. Park,\nS. Bae, L. Lee, P. Hui, I. S. Kweon, and C. S. Hong,\n“One small step for generative ai, one giant leap for\nAGI: A complete survey on chatgpt in AIGC era,”\nCoRR, vol. abs/2304.06488, 2023.\n\n[77] M. Haman and M. Skolnik, “Using chatgpt to conduct\na literature review.” Accountability in research, 2023.\n\n[78] O. Aydın and E. Karaarslan, “Openai chatgpt gen- ¨\nerated literature review: Digital twin in healthcare,”\nSSRN Electronic Journal, 2022.\n\n[79] Y. J. Park, D. Kaplan, Z. Ren, C. Hsu, C. Li, H. Xu, S. Li,\nand J. Li, “Can chatgpt be used to generate scientific\nhypotheses?” CoRR, vol. abs/2304.12208, 2023.\n\n[80] M. M. Hassan, R. A. Knipper, and S. K. K. Santu,\n“Chatgpt as your personal data scientist,” CoRR, vol.\nabs/2305.13657, 2023.\n\n[81] L. Cheng, X. Li, and L. Bing, “Is GPT-4 a good data\nanalyst?” CoRR, vol. abs/2305.15038, 2023.\n\n[82] S. I. M. Hussam Alkaissi, “Artificial hallucinations in\nchatgpt\n\n[83] E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier,\nand G. Penedo, “Falcon-40B: an open large language\nmodel with state-of-the-art performance,” 2023.\n\n[84] S. K. K. Santu and D. Feng, “Teler: A general\ntaxonomy of LLM prompts for benchmarking\ncomplex tasks,” CoRR, vol. abs/2305.11430, 2023.\n[Online]. Available: https://doi.org/10.48550/arXiv.\n2305.11430\n\n[85] J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea,\nH. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C.\nSchmidt, “A prompt pattern catalog to enhance\nprompt engineering with chatgpt,” arXiv preprint\narXiv:2302.11382, 2023.\n\n[86] OpenAI, “Gpt best practices,” OpenAI, 2023.\n[Online]. Available: https://platform.openai.com/\ndocs/guides/gpt-best-practices\n\n[87] Contributors, “Ai short,” 2023. [Online]. Available:\nhttps://www.aishort.top/\n\n[88] ——, “Awesome chatgpt prompts,” Github, 2023.\n[Online]. Available: https://github.com/f/awesomechatgpt-prompts/\n\n[89] V. Liu and L. B. Chilton, “Design guidelines for\nprompt engineering text-to-image generative models,” in Proceedings of the 2022 CHI Conference on Human\nFactors in Computing Systems, 2022, pp. 1–23.\n\n[90] L. Beurer-Kellner, M. Fischer, and M. Vechev,\n“Prompting is programming: A query language for\nlarge language models,” Proceedings of the ACM on\nProgramming Languages, vol. 7, no. PLDI, pp. 1946–\n1969, 2023.\n\n[91] P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N.\nWu, S.-C. Zhu, and J. Gao, “Chameleon: Plug-andplay compositional reasoning with large language\nmodels,” arXiv preprint arXiv:2304.09842, 2023.\n\n[92] Y. Hou, J. Zhang, Z. Lin, H. Lu, R. Xie, J. J. McAuley,\nand W. X. Zhao, “Large language models are zeroshot rankers for recommender systems,” CoRR, vol.\nabs/2305.08845, 2023.\n\n[93] S. Chang and E. Fosler-Lussier, “How to prompt\nllms for text-to-sql: A study in zero-shot, singledomain, and cross-domain settings,” CoRR, vol.\nabs/2305.11853, 2023. [Online]. Available: https:\n//doi.org/10.48550/arXiv.2305.11853\n\n[94] R. Tang, X. Han, X. Jiang, and X. Hu, “Does synthetic\ndata generation of llms help clinical text mining?”\narXiv preprint arXiv:2303.04360, 2023.\n\n[95] O. Nov, N. Singh, and D. M. Mann, “Putting chatgpt’s medical advice to the (turing) test,” CoRR, vol.\nabs/2301.10035, 2023.\n\n[96] K. Yang, S. Ji, T. Zhang, Q. Xie, and S. Ananiadou,\n“On the evaluations of chatgpt and emotion-enhanced\nprompting for mental health analysis,” CoRR, vol.\nabs/2304.03347, 2023.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}