{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Rise of Text Data\n\nThe volume of text data generated worldwide is staggering. According to recent reports [[1]](#item-five), 329 million TBs of data is generated daily, of which 80% is unstructured, developed from data sources like social media feeds, product reviews, news and other types of articles. Processing and understanding these unstructured data thus become tremendously crucial for corporate and academic institutions. Natural Language Processing (NLP) is a subfield of artificial intelligence (AI) and computational linguistics that focuses on the interaction between computers and human language. It involves the development of algorithms, models, and techniques that enable computers to understand, interpret, and generate natural language in a meaningful way for humans. Over the years, NLP research has evolved significantly, driven by advancements in machine learning, deep neural networks, and the availability of large-scale language datasets. \n\nThis article summarises the advancements in Natural Language Processing over the past four decades. This article can act as a guide for technical and non-technical readers to understand the techniques often used to convert text data into information and how text data is being used in different applications of modern days. \n\n### Overview of the report\n\nWe showcase the initial successes of embedding-based and RNNs in Kaggle NLP competitions, followed by the rise of Transformer-based encoder models such as BERT. However, the most recent NLP competitions saw an uprise of larger capacity models - DeBERTa, BirBird, Longformer and RoBERTa-large. Although our analysis highlights that the recent winning solutions contain single models, the popularity of an ensemble of multiple models remains popular among the Kaggle community to improve results. We also highlight generative LLMs' effectiveness and reasoning capabilities, such as GPT-3, and how the community can utilize these methods in predictive applications. \n\n### Table of Contents\n\n* [The Past: Rise of Recurrent Neural Networks and Attention](#item-one)\n * [Techniques used by Kaggle Community before 2020](#item-one-two)\n\n\n* [The Present: Era of Large Language Models](#item-two)\n * [Techniques still popular among Kaggle Community after 2020](#item-two-two)\n * [Eternal Teachings from Kaggle Maestros](#item-two-three)\n * [Comparison of Large Language Models](#item-two-four)\n \n\n* [The Future - Handling Ethical and Computational Considerations of LLMs](#item-three)\n\n* [Closing Remarks](#item-four)\n\n* [References](#item-five) ","metadata":{}},{"cell_type":"code","source":"! pip install -U transformers accelerate openai tiktoken -q","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T13:39:51.521901Z","iopub.execute_input":"2023-07-16T13:39:51.522309Z","iopub.status.idle":"2023-07-16T13:40:18.537561Z","shell.execute_reply.started":"2023-07-16T13:39:51.522279Z","shell.execute_reply":"2023-07-16T13:40:18.536351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\nimport matplotlib.dates as mdates\nfrom datetime import datetime\nimport pandas as pd\nfrom dateutil import parser","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T13:40:18.540973Z","iopub.execute_input":"2023-07-16T13:40:18.541488Z","iopub.status.idle":"2023-07-16T13:40:18.547545Z","shell.execute_reply.started":"2023-07-16T13:40:18.541438Z","shell.execute_reply":"2023-07-16T13:40:18.546719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Read the timeline data\n\ntimeline = pd.read_csv(\"/kaggle/input/nlp-innovations/NLP_timeline.csv\")\nnames = timeline.Technique.tolist()\ndates = timeline['Introduced Year'].tolist()\ndates = [parser.parse(str(d)) for d in dates]","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T11:27:43.570466Z","iopub.execute_input":"2023-07-16T11:27:43.570853Z","iopub.status.idle":"2023-07-16T11:27:43.606901Z","shell.execute_reply.started":"2023-07-16T11:27:43.570821Z","shell.execute_reply":"2023-07-16T11:27:43.605906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We begin our discussion with visualization of some of the essential developments in the field of NLP over the past 40 years. ","metadata":{}},{"cell_type":"code","source":"# Reference - https://matplotlib.org/stable/gallery/lines_bars_and_markers/timeline.html\n# We do a timeline plot\n\nlevels = np.tile([-3, 3, -2, 2, -1, 1],\n int(np.ceil(len(dates)/6)))[:len(dates)]\n\n# Create figure and plot a stem plot with the date\nfig, ax = plt.subplots(figsize=(10, 6), layout=\"constrained\")\nax.set(title=\"Figure 1: Important Developments in NLP\")\n\nax.vlines(dates, 0, levels, color=\"tab:red\") # The vertical stems.\nax.plot(dates, np.zeros_like(dates), \"-o\",\n color=\"k\", markerfacecolor=\"w\") # Baseline and markers on it.\n\n# annotate lines\nfor d, l, r in zip(dates, levels, names):\n ax.annotate(r, xy=(d, l),\n xytext=(-3, np.sign(l)*3), textcoords=\"offset points\", ha='center',\n horizontalalignment=\"right\",\n verticalalignment=\"bottom\" if l > 0 else \"top\")\n\n# format x-axis with 4-month intervals\n#ax.xaxis.set_major_locator(mdates.MonthLocator(interval=4))\n#ax.xaxis.set_major_formatter(mdates.DateFormatter(\"%b %Y\"))\nplt.setp(ax.get_xticklabels(), rotation=30, ha=\"right\")\n\nplt.xlabel(\"Year\")\n# remove y-axis and spines\nax.yaxis.set_visible(False)\nax.spines[[\"left\", \"top\", \"right\"]].set_visible(False)\n\n#ax.margins(y=0.1)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T11:27:43.608999Z","iopub.execute_input":"2023-07-16T11:27:43.609542Z","iopub.status.idle":"2023-07-16T11:27:44.152286Z","shell.execute_reply.started":"2023-07-16T11:27:43.609513Z","shell.execute_reply":"2023-07-16T11:27:44.151171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Figure 1 shows the inception of some of the popular NLP techniques - RNN, Word embeddings on a timeline. Agile developments in recent 5-10 years are exciting and give us a glimpse into the future of NLP.","metadata":{}},{"cell_type":"markdown","source":"\n\n## The Past: Rise of Recurrent Neural Networks and Attention (1980 - 2020)\n\nWe first discuss the fundamental concepts of NLP and how they can be used in processing and analyzing text data. Processing text data requires different levels of abstraction (c.f. Figure 2). \n\n* **Morphology**: Focuses on word formation and the internal structure of words, including processes such as inflexion, derivation, and compounding.\n\n* **Syntax**: Deals with the arrangement and structure of words to form meaningful sentences, including phrase structure, grammatical rules, and syntactic parsing.\n\n* **Semantics**: Concerned with the meaning of words, phrases, and sentences, including lexical semantics (word meaning) and compositional semantics (sentence meaning).\n\n* **Pragmatics**: Considers the context, intentions, and implications of language use, including discourse analysis, speech acts, and conversational implicatures.\n\n

\n\n

\n

\n Figure 2: Hierarchy of Natural Language Processing (Image Credits - Nextiva)\n

\n\nEarly NLP research primarily relied on rule-based systems [[2]](#item-five) and statistical approaches [[3]](#item-five). Rule-based systems involve formulating linguistic rules and heuristics by human experts to process and analyze text. These rules were designed to capture syntactic and semantic patterns in language and enable machines to perform specific language-related tasks. However, they needed help handling the complexity and ambiguity inherent in human language. The manual construction of rules was a labour-intensive process and often required expert knowledge of linguistics.\nStatistical methods, such as n-grams [[4]](#item-five) and Hidden Markov Models (HMMs) [[5]](#item-five), were popular approaches in NLP. N-grams involved analyzing consecutive sequences of words to capture local dependencies, such as bigrams (pairs of adjacent words) and trigrams (triplets of adjacent words). These methods offered promising results in specific tasks, such as language modelling [6], speech recognition [[7]](#item-five), and spelling correction [[8]](#item-five). Another traditional technique, the term frequency-inverse document frequency (tf-idf), played a significant role in NLP. It considers the term frequency (how often a term appears in a document) and the inverse document frequency. (how common or rare a term is across the entire collection). It has been applied in information retrieval, document classification, and text mining tasks. However, they faced challenges capturing natural language's semantic nuances and contextual dependencies. N-grams and HMMs needed more contextual understanding and struggled with handling word sense ambiguity and long-range dependencies.\n\nThe breakthrough in deep learning techniques significantly improved NLP research. Deep learning models, especially neural networks, demonstrated remarkable capabilities in capturing complex patterns and dependencies in text. Recurrent Neural Networks (RNNs) [[9]](#item-five) and Convolutional Neural Networks (CNNs) [[10]](#item-five) have become widely used architectures in NLP. RNNs were particularly effective in tasks requiring sequential information processing, such as language modelling, machine translation, and sentiment analysis. The inherent recurrent connections in RNNs allowed them to capture contextual dependencies by maintaining a hidden state that retained information from previous time steps. However, standard RNNs suffered from the \"vanishing gradient\" problem, limiting their ability to capture long-range dependencies. Long Short-Term Memory (LSTM) networks were introduced to address this limitation [[11]](#item-five). LSTMs incorporated memory cells that could store and retrieve information, allowing them to capture long-term dependencies more effectively. LSTMs became a crucial component in various NLP tasks and showed improved performance over traditional methods. CNNs, known for their success in computer vision tasks, were adapted for NLP to capture local features and hierarchies in text data. CNNs excelled in text classification and sentiment analysis tasks, where local patterns and feature extraction were crucial. By applying filters of different sizes across the input text, CNNs could capture relevant local information and learn higher-level representations. The combination of CNNs and RNNs, known as the \"CNN-RNN\" architecture, further improved text understanding and generation by leveraging local and sequential information [[12]](#item-five). Traditional approaches to language representation, such as bag-of-words and word frequency-based methods, are needed to capture semantic relationships between words. Word2Vec, introduced by Tomas Mikolov et al. from Google in 2013, learns word embeddings by training a neural network model on large amounts of text data, capturing the semantic and syntactic relationships between words. \n\nDespite the advancements achieved by RNNs and CNNs, capturing long-range dependencies remained challenging. The introduction of Transformer models [[13]](#item-five) revolutionized language understanding tasks. Transformers employ self-attention mechanisms, allowing them to capture contextual relationships across the entire input sequence more efficiently. The attention mechanism enables each word to attend to different parts of the input sequence, aggregating information from relevant context. The transformer architecture, introduced in the seminal work \"Attention is All You Need,\" marked a significant breakthrough in NLP. Transformer models, such as Bidirectional Encoder Representations from Transformers (BERT) [[14]](#item-five), Generative Pre-trained Transformers (GPT) [[15]](#item-five), and their variants, have achieved state-of-the-art performance in numerous NLP tasks.\n\nBERT, for example, introduced the concept of masked language modelling and next-sentence prediction during pretraining. Pretraining involved exposing the model to a vast amount of unlabelled text data, enabling it to learn contextual representations of words. By training the model to predict missing words and sentence coherence, BERT acquired a rich understanding of language semantics and syntax. The pre-trained BERT model could then be fine-tuned on specific downstream tasks, such as question answering, named entity recognition, and sentiment analysis, yielding impressive results. The success of transformers demonstrated their ability to capture intricate linguistic patterns and contextual dependencies, surpassing previous approaches.","metadata":{}},{"cell_type":"markdown","source":"\n\n#### What were the popular techniques used by the Kaggle community before 2020?\n\nIn the next section we analyze the techniques explored by the Kaggle communities in competition winning solutions. We analyze the winning solutions from various Kaggle competitions - [Toxic Comment Classification Challenge](https://www.kaggle.com/c/8076), [Jigsaw Multilingual Toxic Comment Classification](https://www.kaggle.com/c/19018). \n\nFor this exercise we make use of GPT-3 model to extract the key techniques used in the winning solutions and visualize them using word clouds.\n\n*Disclaimer:* OpenAI API key is required to run the following cells. You can obtain the API from [OpenAI](https://platform.openai.com/) and copy in Kaggle secrets. ","metadata":{}},{"cell_type":"code","source":"kaggle_data = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv\", parse_dates=[0, 3])","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T13:40:18.548697Z","iopub.execute_input":"2023-07-16T13:40:18.548992Z","iopub.status.idle":"2023-07-16T13:40:19.396597Z","shell.execute_reply.started":"2023-07-16T13:40:18.548967Z","shell.execute_reply":"2023-07-16T13:40:19.395462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kaggle_data_till_2020 = kaggle_data[kaggle_data['Competition Launch Date'].dt.year <= 2020]\n\ncompetitions = ['Large Scale Hierarchical Text Classification', \\\n 'Tradeshift Text Classification', \\\n 'Toxic Comment Classification Challenge', \\\n 'Jigsaw Multilingual Toxic Comment Classification',]\n\n#kaggle_data_till_2020['Title of Competition'].unique()\nkaggle_data_till_2020 = kaggle_data_till_2020[kaggle_data_till_2020['Title of Competition'].isin(competitions)].reset_index(drop=True)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T11:27:50.813853Z","iopub.execute_input":"2023-07-16T11:27:50.814349Z","iopub.status.idle":"2023-07-16T11:27:50.832133Z","shell.execute_reply.started":"2023-07-16T11:27:50.814313Z","shell.execute_reply":"2023-07-16T11:27:50.830727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tqdm import tqdm\nimport time\nimport re\nimport numpy as np","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T13:40:19.400371Z","iopub.execute_input":"2023-07-16T13:40:19.400822Z","iopub.status.idle":"2023-07-16T13:40:19.408496Z","shell.execute_reply.started":"2023-07-16T13:40:19.400779Z","shell.execute_reply":"2023-07-16T13:40:19.406968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import openai\nimport tiktoken\nfrom kaggle_secrets import UserSecretsClient\n\nuser_secrets = UserSecretsClient()\nopenai.api_key = user_secrets.get_secret(\"OPENAI_API_KEY\")","metadata":{"execution":{"iopub.status.busy":"2023-07-16T13:40:19.410228Z","iopub.execute_input":"2023-07-16T13:40:19.410978Z","iopub.status.idle":"2023-07-16T13:40:20.745495Z","shell.execute_reply.started":"2023-07-16T13:40:19.410939Z","shell.execute_reply":"2023-07-16T13:40:20.744311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Extract solution rank and the type of model used - single or ensemble\n\ndef get_rank(x):\n selected_ranks = re.findall(\"\\d+\\w+ place\", x) + re.findall(\"\\d+\\w+ rank\", x) + re.findall(\"\\d+th\", x) + re.findall(\"\\d+st\", x)\n if len(selected_ranks) > 0:\n return int(re.findall(\"\\d+\", selected_ranks[0])[0])\n elif 'gold' in x:\n return 1\n else:\n return np.nan\n\ndef model_type(x):\n if \"ensemble\" in x or \"stack\" in x or \"blend\" in x:\n return \"Ensemble\"\n else:\n return \"Single\"","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T13:40:20.748377Z","iopub.execute_input":"2023-07-16T13:40:20.749387Z","iopub.status.idle":"2023-07-16T13:40:20.757557Z","shell.execute_reply.started":"2023-07-16T13:40:20.749340Z","shell.execute_reply":"2023-07-16T13:40:20.756178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load the cl100k_base tokenizer\ntokenizer = tiktoken.get_encoding(\"cl100k_base\")\n\n# reduce the token sequence length\n\nMAX_LEN = 3000\n\ndef shorten_text(text):\n tokens = tokenizer.encode(text)\n tokens = tokens[:MAX_LEN]\n return tokenizer.decode(tokens)","metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"execution":{"iopub.status.busy":"2023-07-16T13:40:20.759261Z","iopub.execute_input":"2023-07-16T13:40:20.759655Z","iopub.status.idle":"2023-07-16T13:40:22.330266Z","shell.execute_reply.started":"2023-07-16T13:40:20.759627Z","shell.execute_reply":"2023-07-16T13:40:22.328996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use GPT-3 davinci model to extract the list of techniques used in the writeups\n\nfor i in tqdm(range(kaggle_data_till_2020.shape[0])):\n try:\n prompt = f'What are the techniques used in the text?: \"text: {shorten_text(kaggle_data_till_2020.Writeup.iloc[i])}'\n response = openai.Completion.create(model=\"text-davinci-003\", prompt=prompt, \\\n temperature=0.0, max_tokens=150, stop=\"<|endoftext|>\")\n\n kaggle_data_till_2020.loc[i, \"Techniques used\"] = response['choices'][0]['text'].split(\":\")[-1].strip().replace(\".\",\"\").replace(\"and\", \"\")\n\n time.sleep(2)\n except:\n kaggle_data_till_2020.loc[i, \"Techniques used\"] = \"\"\n \nkaggle_data_till_2020['Techniques used'] = kaggle_data_till_2020['Techniques used'].apply(lambda x: re.sub(\"\\n\\d+\",\",\",x))","metadata":{"execution":{"iopub.status.busy":"2023-07-16T11:28:30.413549Z","iopub.execute_input":"2023-07-16T11:28:30.414006Z","iopub.status.idle":"2023-07-16T11:31:09.505364Z","shell.execute_reply.started":"2023-07-16T11:28:30.413974Z","shell.execute_reply":"2023-07-16T11:31:09.503922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wc_data = kaggle_data_till_2020[['Competition Launch Date','Techniques used']]\nwc_data['Year'] = wc_data['Competition Launch Date'].dt.year\nwc_data = wc_data.groupby(['Year'])['Techniques used'].apply(lambda x: \",\".join(list(x))).reset_index(drop=False)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T11:31:09.507890Z","iopub.execute_input":"2023-07-16T11:31:09.508330Z","iopub.status.idle":"2023-07-16T11:31:09.529407Z","shell.execute_reply.started":"2023-07-16T11:31:09.508297Z","shell.execute_reply":"2023-07-16T11:31:09.527879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reference - https://www.geeksforgeeks.org/generating-word-cloud-python/\n\nfrom wordcloud import WordCloud, STOPWORDS\n\ndef gen_wordcloud(text):\n comment_words = ''\n stopwords = set(STOPWORDS)\n \n tokens = text.split()\n \n for i in range(len(tokens)):\n tokens[i] = tokens[i].lower()\n \n comment_words += \" \".join(tokens)+\" \"\n \n return comment_words","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T13:40:22.331694Z","iopub.execute_input":"2023-07-16T13:40:22.332033Z","iopub.status.idle":"2023-07-16T13:40:22.358880Z","shell.execute_reply.started":"2023-07-16T13:40:22.332002Z","shell.execute_reply":"2023-07-16T13:40:22.357574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize the top techniques\n\nfig, ax = plt.subplots(1,3, figsize = (8, 16), facecolor = None)\n\nfor i, year in enumerate(wc_data.Year.unique()):\n comment_words = gen_wordcloud(wc_data[wc_data.Year == year]['Techniques used'].iloc[0])\n wordcloud = WordCloud(width = 400, height = 400,\n background_color ='white',\n stopwords = set(STOPWORDS),collocation_threshold = 5,\n min_font_size = 10).generate(comment_words)\n\n # plot the WordCloud image \n ax[i].imshow(wordcloud)\n ax[i].set_title(f\"Top techniques in Year {year}\")\n ax[i].axis(\"off\") \n \nplt.tight_layout(pad = 1)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T13:40:22.360391Z","iopub.execute_input":"2023-07-16T13:40:22.361246Z","iopub.status.idle":"2023-07-16T13:40:23.035050Z","shell.execute_reply.started":"2023-07-16T13:40:22.361214Z","shell.execute_reply":"2023-07-16T13:40:23.033560Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kaggle_data_till_2020['rank'] = kaggle_data_till_2020['Title of Writeup'].apply(get_rank)\nkaggle_data_till_2020['type'] = kaggle_data_till_2020['Writeup'].apply(model_type)\nkaggle_data_till_2020['Year'] = kaggle_data_till_2020['Competition Launch Date'].dt.year\n\nkaggle_data_till_2020.groupby(['rank','Year'])['type'].value_counts(normalize=True).head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T11:31:15.602171Z","iopub.execute_input":"2023-07-16T11:31:15.602582Z","iopub.status.idle":"2023-07-16T11:31:15.627888Z","shell.execute_reply.started":"2023-07-16T11:31:15.602544Z","shell.execute_reply":"2023-07-16T11:31:15.626463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The most common techniques used in older competitions were an ensemble of different ML-based models - logistic regression, XGBoost, and Naive Bayes. Tf-Idf was the most popular text representation learning technique. However, starting in 2017, embedding-based models and attention have gained popularity. Till 2017, variants of RNNs such as LSTM and GRUs were popular. For instance - The winning team [Toxic Crusaders](https://www.kaggle.com/c/8076/discussion/52557) used FastText and GloVe embeddings along with bidirectional GRUs in their winning submissions in Toxic Comment Classification Challenge. Similar methodologies were also adopted by [neongen & Computer says no](https://www.kaggle.com/c/8076/discussion/52612), the second-place winners. Both the teams achieved over 0.988 mean ROC-AUC on the private leaderboard.\n\nIn 2020 (in Jigsaw Multilingual Toxic Comment Classification), most winning teams used multilingual language models such as XLM-RoBERTa. As discussed by the winning team, [\nLingua Franca](https://www.kaggle.com/c/19018/discussion/160862), ensembling was still crucial in mitigating the training variability of Transformers. They also emphasized the importance of monolingual language models trained in foreign languages.\n\nAlthough most top-performing teams used larger language models, many highlighted the importance of post and preprocessing techniques like pseudo labelling, test-time augmentation and label smoothing. Also, ensembling techniques remained crucial even in 2020.\n\n","metadata":{}},{"cell_type":"markdown","source":"\n## The Present: Era of Large Language Models (2020 - 2023)\n\nLarge Language Models (LLMs), such as Generative Pre-trained Transformer 3 (GPT-3) [[17]](#item-five) and Large Language Models, have emerged as powerful tools in the field of NLP. As figure 3 highlights, these models are predominantly decoder-based, as opposed to encoder-based variants like BERT or encoder-decoder-based variants like BART [[18]](#item-five). These models are trained on vast amounts of text data and can generate coherent and contextually relevant text. GPT-3, developed by OpenAI, is one of the most prominent LLMs. It has 175 billion parameters, making it one of the largest language models. GPT-3 has demonstrated impressive capabilities in various NLP tasks, including text completion, question answering, machine translation, sentiment analysis, and creative writing. The model can generate human-like responses and adapt its language style based on input and context. As opposed to autoencoding type pre-training (*e.g.* masked language modeling or, denoising pre-training), these decoder-based models are majorly trained using next token prediction task. More sophisticated techniques such as instruction-based pre-training and reinforcement learning from human feedback (RHLF) have gained popularity in training large language models. One of the recent language models, InstructGPT [[36]](#item-five), was pre-trained to perform a specific task or follow a particular instruction by using a dataset that includes both instructional prompts and corresponding target outputs. Albeit having 100x lesser parameters than GPT-3, this language model shows comparative performances across different language understanding and generative tasks.\n\n

\n\n

\n

\n Figure 3: Developments of Large Language Models (Image credit - arxiv-vanity)\n

\n\nEmerging techniques that gained popularity with LLMs are few-shot and zero-shot learning. With few-shot learning, a model can learn even with a few labelled examples within a few gradient updates. In-context learning, an emerging property of LLMs, enables them to learn new concepts by considering the surrounding context or environment, even with limited supervision.","metadata":{}},{"cell_type":"markdown","source":"\n\n#### What are the popular techniques used by the Kaggle community after 2020?\n\nWe analyze the winning solutions from recent Kaggle NLP competitions - [U.S. Patent Phrase to Phrase Matching](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching), [Jigsaw Rate Severity of Toxic Comments](https://www.kaggle.com/competitions/jigsaw-toxic-severity-rating), [NBME - Score Clinical Patient Notes](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes) and highlight the most popular techniques used by them.","metadata":{}},{"cell_type":"code","source":"kaggle_data_after_2020 = kaggle_data[kaggle_data['Competition Launch Date'].dt.year > 2020]\n\ncompetitions = [\"U.S. Patent Phrase to Phrase Matching\", \\\n \"Feedback Prize - Predicting Effective Arguments\", \\\n \"NBME - Score Clinical Patient Notes\", \\\n \"Feedback Prize - Evaluating Student Writing\", \\\n \"Jigsaw Rate Severity of Toxic Comments\",\n]\n\n#kaggle_data_after_2020['Title of Competition'].unique()\nkaggle_data_after_2020 = kaggle_data_after_2020[kaggle_data_after_2020['Title of Competition'].isin(competitions)].reset_index(drop=True)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T13:40:29.450639Z","iopub.execute_input":"2023-07-16T13:40:29.451073Z","iopub.status.idle":"2023-07-16T13:40:29.466855Z","shell.execute_reply.started":"2023-07-16T13:40:29.451036Z","shell.execute_reply":"2023-07-16T13:40:29.465106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in tqdm(range(kaggle_data_after_2020.shape[0])):\n try:\n prompt = f'What are the techniques used in the text?: \"text: {shorten_text(kaggle_data_after_2020.Writeup.iloc[i])}'\n response = openai.Completion.create(model=\"text-davinci-003\", prompt=prompt, \\\n temperature=0.0, max_tokens=150, stop=\"<|endoftext|>\")\n\n kaggle_data_after_2020.loc[i, \"Techniques used\"] = response['choices'][0]['text'].split(\":\")[-1].strip().replace(\".\",\"\").replace(\"and\", \"\")\n\n time.sleep(2)\n except:\n kaggle_data_after_2020.loc[i, \"Techniques used\"] = \"\"\n \nkaggle_data_after_2020['Techniques used'] = kaggle_data_after_2020['Techniques used'].apply(lambda x: re.sub(\"\\n\\d+\",\",\",x))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T13:40:32.185503Z","iopub.execute_input":"2023-07-16T13:40:32.186015Z","iopub.status.idle":"2023-07-16T13:48:58.823146Z","shell.execute_reply.started":"2023-07-16T13:40:32.185967Z","shell.execute_reply":"2023-07-16T13:48:58.821982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wc_data = kaggle_data_after_2020[['Competition Launch Date','Techniques used']]\nwc_data['Year'] = wc_data['Competition Launch Date'].dt.year\nwc_data = wc_data.groupby(['Year'])['Techniques used'].apply(lambda x: \",\".join(list(x))).reset_index(drop=False)\nwc_data['Techniques used'] = wc_data['Techniques used'].apply(lambda x: x.replace(\"\",\"\").replace(\"\",\"\"))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T13:53:28.767531Z","iopub.execute_input":"2023-07-16T13:53:28.768056Z","iopub.status.idle":"2023-07-16T13:53:28.784760Z","shell.execute_reply.started":"2023-07-16T13:53:28.768010Z","shell.execute_reply":"2023-07-16T13:53:28.783572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1,wc_data.Year.nunique(), figsize = (8, 16), facecolor = None)\n\nfor i, year in enumerate(wc_data.Year.unique()):\n comment_words = gen_wordcloud(wc_data[wc_data.Year == year]['Techniques used'].iloc[0])\n wordcloud = WordCloud(width = 400, height = 400,\n background_color ='white',\n stopwords = set(STOPWORDS),collocation_threshold = 5,\n min_font_size = 10).generate(comment_words)\n\n # plot the WordCloud image \n ax[i].imshow(wordcloud)\n ax[i].set_title(f\"Top techniques in Year {year}\")\n ax[i].axis(\"off\") \n \nplt.tight_layout(pad = 1)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T13:53:33.001729Z","iopub.execute_input":"2023-07-16T13:53:33.002110Z","iopub.status.idle":"2023-07-16T13:53:33.815185Z","shell.execute_reply.started":"2023-07-16T13:53:33.002080Z","shell.execute_reply":"2023-07-16T13:53:33.814168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kaggle_data_after_2020['rank'] = kaggle_data_after_2020['Title of Writeup'].apply(get_rank)\nkaggle_data_after_2020['type'] = kaggle_data_after_2020['Writeup'].apply(model_type)\nkaggle_data_after_2020['Year'] = kaggle_data_after_2020['Competition Launch Date'].dt.year\n\nkaggle_data_after_2020.groupby(['rank','Year'])['type'].value_counts(normalize=True).head(20)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T13:54:10.396090Z","iopub.execute_input":"2023-07-16T13:54:10.396743Z","iopub.status.idle":"2023-07-16T13:54:10.419988Z","shell.execute_reply.started":"2023-07-16T13:54:10.396702Z","shell.execute_reply":"2023-07-16T13:54:10.419089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Larger language models - RoBERTa large and DeBERTa gained more popularity in recent years, highlighting the effectiveness of larger capacity models on real-life applications. Unlike the BERT model, which only calculates contextual similarity between the content of tokens, the DeBERTa model [[25]](#item-five) uses contextual as well as relative positional similarities between tokens. DeBERTa and larger variants of DeBERTa have been adopted by 75% of all the top-performing teams in all five competitions. Apart from this model, other large language models - BigBird [[38]](#item-five), Funnel-Transformer [[39]](#item-five), Longformer [[40]](#item-five) have also been used by the winning teams. Interestingly, all these models are meant for longer sequence classification tasks. Compared to the maximum sequence length of 512 of BERT, these language models can encode sequences up to 4096. Traditionally the pooled output from these language models is passed to the final classification layer for final predictions. However, Kagglers have often tried stacking traditional RNNs such as GRU or LSTMs and attention-based pooling on the hidden representations extracted by language model encoders. Notably, in U.S. Patent Phrase to Phrase Matching competitions, [gezi](https://www.kaggle.com/c/33657/discussion/332243), the competition winner, used bidirectional LSTM on top of the BERT layer. The second-place holder team [SKT](https://www.kaggle.com/c/35308/discussion/347359) adopted a similar approach in Feedback Prize - Predicting Effective Arguments competition.\n\nAnother interesting observation is the rise of single models in Kaggle leaderboards. Compared to older competitions, many winning solutions contain a single model in recent competitions. Single large-scale models have become more robust yet simpler than complex pipelines involving an ensemble of models. ","metadata":{}},{"cell_type":"markdown","source":"\n\n#### Eternal Teachings from the Kaggle Maestros\n\nAlbeit popular among the research communities, language models sometimes require nudges to boost performance in the Kaggle leaderboard. Conventional wisdom from Kaggle masters and grandmasters remains equally valuable, even in the era of language models. We highlight some of these pearls of wisdom. These techniques are valuable in the domain of NLP and across other verticles of machine learning.\n\n* **A Robust Validation Scheme -** Kaggle masters have always emphasized the importance of robust cross-validation mechanisms to validate models before submitting them to the final leaderboard. They have repeatedly argued in favour of relying on cross-validation performance than a public leaderboard. [Ryuichi & currypurin](https://www.kaggle.com/c/33607/discussion/323095), the 1st place winner of NBME - Score Clinical Patient Notes competition discussed how using GroupStratifiedKFolds helped them in achieving a better result than other cross-validation strategies. [Manuel Campos](https://www.kaggle.com/c/27935/discussion/306084), the 4th place winner of the Jigsaw Rate Severity of Toxic Comments competition, undertook two different cross-validation strategies to evaluate their submitting solution. Similar notable examples can be found even in older competitions - [rapids.nlp](https://www.kaggle.com/c/19018/discussion/160980)'s submission in Jigsaw Multilingual Toxic Comment Classification, etc.\n\n* **Weighted Ensembles -** Ensembling techniques such as stacking and blending have always been important to the Kaggle community. With appropriate ensembling, Kaggle winners have significantly boosted their single models' performances, placing them in higher leaderboard ranks. Appropriate ensembling remained crucial even in the era of pre-trained large language models. [Guanshuo Xu](https://www.kaggle.com/c/27935/discussion/306274) used genetic algorithms to determine the ensemble weights, achieving him the 1st place in Jigsaw Rate Severity of Toxic Comments. In the same competition, [steubk](https://www.kaggle.com/c/27935/discussion/308938) earned 3rd place, where he used a simple weighted average on five different model predictions. In U.S. Patent Phrase to Phrase Matching competition, [Z & T](https://www.kaggle.com/c/33657/discussion/332234), the 2nd place winner used linear regression to determine the model weights in the ensemble. Similarly, [Ryuichi & currypurin](https://www.kaggle.com/c/33607/discussion/323095), the 1st place winner of NBME - Score Clinical Patient Notes competition, used different blended weights for their various models, improving private leaderboard score by 0.0005 points. [Psi](https://www.kaggle.com/c/19018/discussion/160937) used rank-averaged blending in the Jigsaw Multilingual Toxic Comment Classification competition and secured 8th place.\n\n* **Pseudo Labelling -** Pseudo labelling is a semi-supervised technique where artificial labels are generated from unlabeled data and used to fine-tune or train a model. In many instances, pseudo-labelling has been proven effective in improving performance. Several top performers have used pseudo-labeling in recent NLP competitions. Notably - [One Chance](https://www.kaggle.com/c/27935/discussion/306366) (7th place) in Jigsaw Rate Severity of Toxic Comments, [LakeCity](https://www.kaggle.com/c/33607/discussion/322832) (3rd place), [Y.Nakama & copasta & hakubishin & Takoi](https://www.kaggle.com/c/33607/discussion/322799) (4th place), [LYZD-Fintech](https://www.kaggle.com/c/33607/discussion/322891) (9th place) s' solutions in NBME - Score Clinical Patient Notes competition utilize pseudo labeling to improve performances of their single best models.\n\nMany other noteworthy techniques such as knowledge distillation (used by [LakeCity](https://www.kaggle.com/c/33607/discussion/322832) in NBME - Score Clinical Patient Notes competition), label smoothing (used by [Ace Team](https://www.kaggle.com/c/19018/discussion/160964) in Jigsaw Multilingual Toxic Comment Classification), threshold modification (used by [LYZD-Fintech](https://www.kaggle.com/c/33607/discussion/322891) in NBME - Score Clinical Patient Notes competition), Stochastic Weight Averaging (SWA) (used by [Now You See Me](https://www.kaggle.com/c/35308/discussion/347891) in Feedback Prize - Predicting Effective Arguments and [Upstage](https://www.kaggle.com/c/31779/discussion/316071) in Feedback Prize - Evaluating Student Writing), Adversarial Weight Perturbation (AWP) (used by [world peace](https://www.kaggle.com/c/31779/discussion/313177) in Feedback Prize - Evaluating Student Writing and [SKT](https://www.kaggle.com/c/35308/discussion/347359) in Feedback Prize - Predicting Effective Arguments) are popular among the Kaggle community, even in recent years. ","metadata":{}},{"cell_type":"markdown","source":"\n\n#### Comparison of Large Language Models\n\n\nWith the inception of more large language models, the need for more robust evaluation benchmarks have arise. New benchmarks, such as GLUE (General Language Understanding Evaluation) and SuperGLUE, have emerged to evaluate models' performance across multiple NLP tasks comprehensively. These benchmarks provide a more holistic assessment of language understanding, reasoning, and inference capabilities, encouraging the development of more robust and versatile NLP models. Recently, MMLU (Massive Multitask Language Understanding) benchmark was proposed [[20]](#item-five) to evaluate language models' capabilities of zero and few-shot learning capabilities. MMLU covers 57 subjects across STEM, the humanities, and the social sciences and assesses language models' knowledge and problem-solving abilities. ","metadata":{}},{"cell_type":"code","source":"mmlu_benchmarks = pd.read_csv(\"/kaggle/input/nlp-innovations/MMLU_benchmark.csv\")","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T11:35:16.879595Z","iopub.execute_input":"2023-07-16T11:35:16.880054Z","iopub.status.idle":"2023-07-16T11:35:16.894383Z","shell.execute_reply.started":"2023-07-16T11:35:16.880010Z","shell.execute_reply":"2023-07-16T11:35:16.893368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import scipy\n\nsizes = mmlu_benchmarks['Score'] / mmlu_benchmarks['Score'].max()\n\nplt.figure(figsize=(12, 8))\nplt.scatter(mmlu_benchmarks['Model Size'], mmlu_benchmarks['Score'], \n alpha=0.5,\n c = sorted(mmlu_benchmarks['Score']*100),\n s = sizes * 10000)\n\nfor i in range(len(mmlu_benchmarks)):\n plt.annotate(mmlu_benchmarks.Model.iloc[i], (mmlu_benchmarks['Model Size'].iloc[i], \\\n mmlu_benchmarks['Score'].iloc[i] + 0.2), ha='center')\n \n\n#m, b = np.polyfit(mmlu_benchmarks['Model Size'], mmlu_benchmarks['Score'], 1)\nm, b, r_value, p_value, std_err = scipy.stats.linregress(mmlu_benchmarks['Model Size'], mmlu_benchmarks['Score'])\n\nm_ = \"%.3g\" % m\nr2 = r_value**2\n\nplt.plot(mmlu_benchmarks['Model Size'], m*mmlu_benchmarks['Model Size']+b)\n\nplt.text(4e11,60,f\"y = {m_}*x + {str(round(b,2))} \\nR2 = {round(r2,3)}\")\n\nplt.ylabel(\"% Accuracy on MMLU benchmark\", size=14)\nplt.xlabel(\"Model Size\", size=14)\nplt.xlim(4e7, 7e11)\nplt.ylim(20, 90)\nplt.title(\"Figure 4: Performance of different LLMs on MMLU Benchmark\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T11:35:16.896105Z","iopub.execute_input":"2023-07-16T11:35:16.896838Z","iopub.status.idle":"2023-07-16T11:35:17.643701Z","shell.execute_reply.started":"2023-07-16T11:35:16.896804Z","shell.execute_reply":"2023-07-16T11:35:17.642434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Figure 4 shows a linear relationship between model size and accuracy on the MMLU benchmark, highlighting the effectiveness of larger capacity models. PaLM [[21]](#item-five) is the largest language model to date, with 540 billion parameters, achieving nearly 70% accuracy on the MMLU benchmark. However, the most effective language models in relative terms could be LLaMa [[22]](#item-five) and Chinchilla [[23]](#item-five). This large-scale language model exhibits abilities to reasoning like humans. \n\nBelow we highlight a real-life application where GPT-3.5 is asked to classify a toxic comment collected from the Jigsaw Multilingual Toxic Comment Classification competition. Even in a zero-shot setting, the model correctly considers the factors before classifying the comment as toxic.","metadata":{}},{"cell_type":"code","source":"text2 = \"Why can't you believe how fat Artie is? Did you see him on his recent appearence on the Tonight Show with Jay Leno? He looks absolutely AWFUL! If I had to put money on it, I'd say that Artie Lange is a can't miss candidate for the 2007 Dead pool! \\\nKindly keep your malicious fingers off of my above comment, . Everytime you remove it, I will repost it!!!\"","metadata":{"execution":{"iopub.status.busy":"2023-07-16T11:35:17.645240Z","iopub.execute_input":"2023-07-16T11:35:17.645589Z","iopub.status.idle":"2023-07-16T11:35:17.651371Z","shell.execute_reply.started":"2023-07-16T11:35:17.645559Z","shell.execute_reply":"2023-07-16T11:35:17.650106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"message_log = [\n {\"role\": \"system\", \"content\": \"\"\"You are a helpful assistant that calculates the probability of a text being toxic with the reasoning behind it.\"\"\"\n }\n]\n\nquestion1 = \"What is the approximated probability that the text is toxic?\"\nprompt = f'{question1}: \"in: {text2}\"'\nmessage_log.append({\"role\": \"user\", \"content\": prompt})\n\nresponse = openai.ChatCompletion.create(model=\"gpt-3.5-turbo\", messages=message_log, \\\n temperature=0.0, max_tokens=200, stop=\"<|endoftext|>\")\n\nprint (response['choices'][0]['message']['content'])","metadata":{"execution":{"iopub.status.busy":"2023-07-16T11:35:17.652760Z","iopub.execute_input":"2023-07-16T11:35:17.653122Z","iopub.status.idle":"2023-07-16T11:35:23.577886Z","shell.execute_reply.started":"2023-07-16T11:35:17.653092Z","shell.execute_reply":"2023-07-16T11:35:23.576702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## The Future - Handling Ethical and Computational Considerations of LLMs\n\nThe current state of NLP showcases a significant contrast between traditional techniques and recent advancements. While valuable in their time, data-driven approaches and deep learning techniques have surpassed rule-based systems and statistical methods. Introducing transformer architectures, contextualized embeddings, and large-scale pretraining has unprecedentedly propelled NLP performance. These advancements have allowed models to capture intricate language patterns, contextual dependencies, and semantic relationships more effectively.\n\nHowever, challenges and opportunities for future research in NLP remain. One of the significant challenges observed with LLMs is *hallucination*, where LLMs generate factually incorrect statements [[34]](#item-five). Due to this, LLMs are sometimes referred to as *stochastic parrots*. Factually incorrect statement generation can pose legal and ethical challenges [[35]](#item-five) and restrict these powerful models from being deployed in sensitive industries like healthcare or finance. One possible solution to this problem can be knowledge graphs.\n\n### Knowledge Graphs with LLMs\n\nKnowledge graphs have emerged as a powerful tool for enhancing language understanding in NLP. A knowledge graph is a structured representation of knowledge that captures entities, their attributes, and the relationships between them. By organizing information in a graph-like structure, knowledge graphs enable machines to reason and navigate vast amounts of interconnected knowledge. In the context of language understanding, knowledge graphs can provide valuable semantic information that enhances text comprehension. They help establish connections between entities, disambiguate terms, and capture the context of a given text. As illustrated in Figure 5, with a more structured information storage capability, KGs can help LLMs eliminate hallucinations [[19]](#item-five).\n\n

\n\n

\n

\n Figure 5: Knowledge Graph guided LLMs (Image credit - arxiv-vanity)\n

\n\nEthical considerations, such as bias in language models and their societal impact, require further investigation [[26, 27]](#item-five). Adversarial robustness, addressing the vulnerabilities of NLP models to malicious attacks and manipulations, is another critical area of focus. The explainability and interpretability of NLP models are crucial for building trust and understanding their decision-making processes [[28]](#item-five). Larger model sizes come with increased computational demands. Recent work by *Kaplan et al.,* [[37]](#item-five) empirically computes the scaling laws - the need for larger training data and higher computational resources for training larger capacity language models. Training these massive models requires substantial computational resources and infrastructure, including specialized hardware such as GPUs or TPUs, and significant amounts of time. The training process involves processing large-scale datasets to learn complex language patterns, necessitating extensive computing power and storage capabilities. For instance, training of GPT-3 took 34 days on 1024 A100 GPUs, emitting approximately 200000 kilograms of CO2, equivalent to the yearly emissions of 40 cars. [[24]](#item-five). Addressing these challenges requires balancing model performance, computational efficiency, and responsible data collection practices. Researchers are exploring techniques like model compression [[29, 30]](#item-five), distillation [[31, 32]](#item-five), and transfer learning [[33]](#item-five) to make LLMs more resource-efficient and adaptable to specific domains or languages.\n","metadata":{}},{"cell_type":"markdown","source":"\n## Closing Remarks\n\nThe field of Natural Language Processing has witnessed remarkable progress, driven by the transition from rule-based systems to machine learning and deep learning techniques. Introducing Transformer models, contextualized embeddings, and large-scale pretraining has revolutionized language understanding and generation tasks. While traditional methods have played a significant role in developing NLP research, the recent advancements in deep learning and transformer-based models have pushed the boundaries of performance and capability in natural language processing. As the field continues to evolve, the future holds tremendous potential for further breakthroughs in NLP research, addressing challenges, and unlocking new opportunities for understanding, generating, and interacting with human language. With the learning gathered from the Kaggle community for over ten years, we can leverage these techniques in deriving more value from text-based data in any real-life application.\n\nFinally, we end this report with a final note generated by three language models. We ask GPT, GPT-2 and GPT-3 about the future of NLP. Let us look at the future state of NLP through the lenses of these models - ","metadata":{}},{"cell_type":"code","source":"from transformers import pipeline","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T11:39:08.943542Z","iopub.execute_input":"2023-07-16T11:39:08.943974Z","iopub.status.idle":"2023-07-16T11:39:24.361152Z","shell.execute_reply.started":"2023-07-16T11:39:08.943927Z","shell.execute_reply":"2023-07-16T11:39:24.359727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"generator_gpt = pipeline(model=\"openai-gpt\")\ntext = generator_gpt(\"Natural language processing has evolved so much in the recent past. The future of NLP \", \\\n max_length=500, num_return_sequences=1, return_full_text=True)[0]['generated_text'].replace(\"\\xa0\",\" \")","metadata":{"execution":{"iopub.status.busy":"2023-07-16T12:12:28.196593Z","iopub.execute_input":"2023-07-16T12:12:28.197051Z","iopub.status.idle":"2023-07-16T12:16:26.645148Z","shell.execute_reply.started":"2023-07-16T12:12:28.197009Z","shell.execute_reply":"2023-07-16T12:16:26.643494Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print ('\\033[1m' + \"Hello, I'm GPT \" + u\"\\U0001F916 \\n \\n\") \nprint ('\\033[0m' + \"I can forsee the future. Following is my view on the future of NLP. \\n \\n\")\nprint (text)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T11:47:39.423260Z","iopub.execute_input":"2023-07-16T11:47:39.423907Z","iopub.status.idle":"2023-07-16T11:47:39.429576Z","shell.execute_reply.started":"2023-07-16T11:47:39.423870Z","shell.execute_reply":"2023-07-16T11:47:39.428333Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"generator_gpt2 = pipeline(model=\"gpt2\")\ntext = generator_gpt2(\"Natural language processing has evolved so much in the recent past. The future of NLP \", \\\n max_length=500, num_return_sequences=1, return_full_text=True)[0]['generated_text'].replace(\"\\xa0\",\" \")","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T11:47:39.431844Z","iopub.execute_input":"2023-07-16T11:47:39.432725Z","iopub.status.idle":"2023-07-16T11:47:50.866432Z","shell.execute_reply.started":"2023-07-16T11:47:39.432679Z","shell.execute_reply":"2023-07-16T11:47:50.864749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print ('\\033[1m' + \"Hello, I'm GPT2 \" + u\"\\U0001F916 \\U0001F916 \\n \\n\") \nprint ('\\033[0m' + \"I can forsee the future. Following is my view on the future of NLP. \\n \\n\")\nprint (text)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T11:47:50.868553Z","iopub.execute_input":"2023-07-16T11:47:50.869135Z","iopub.status.idle":"2023-07-16T11:47:50.874311Z","shell.execute_reply.started":"2023-07-16T11:47:50.869102Z","shell.execute_reply":"2023-07-16T11:47:50.873519Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"message_log = [\n {\"role\": \"system\", \"content\": \"\"\"You are a helpful assistant that generates text based on a prompt.\"\"\"\n }\n]\n\nprompt = \"Natural language processing has evolved so much in the recent past. The future of NLP \"\nmessage_log.append({\"role\": \"user\", \"content\": prompt})\n\nresponse = openai.ChatCompletion.create(model=\"gpt-3.5-turbo\", messages=message_log, \\\n temperature=0.0, max_tokens=500, stop=\"<|endoftext|>\")\nprint ('\\033[1m' + \"Hello, I'm GPT3 \" + u\"\\U0001F916 \\U0001F916 \\U0001F916 \\n \\n\") \nprint ('\\033[0m' + \"I can also forsee the future. Following is my view on the future of NLP. \\n \\n\")\n\nprint (response['choices'][0]['message']['content'])","metadata":{"execution":{"iopub.status.busy":"2023-07-05T10:17:52.574339Z","iopub.execute_input":"2023-07-05T10:17:52.575322Z","iopub.status.idle":"2023-07-05T10:18:03.123225Z","shell.execute_reply.started":"2023-07-05T10:17:52.575286Z","shell.execute_reply":"2023-07-05T10:18:03.121863Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The linguistic differences in responses by different GPT models highlight the importances of larger depth language models and their superiority in understanding, processing and generating text data. Overall, the future of NLP and text data processing holds immense possibilities. Exciting times ahead!","metadata":{}},{"cell_type":"markdown","source":"\n\n## References\n\n[1] https://explodingtopics.com/blog/data-generated-per-day\n\n[2] F. Hayes-Roth, “Rule-based systems,” Commun. ACM, vol. 28, no. 9, pp. 921–932, 1985.\n\n[3] Y. Yang, “An evaluation of statistical approaches to text categorization,” Inf. Retr., vol. 1, no. 1–2, pp. 69–90, 1999.\n\n[4] G. Sidorov, F. Velasquez, E. Stamatatos, A. Gelbukh, and L. Chanona-Hernández, “Syntactic n-grams as machine learning features for natural language processing,” Expert Syst. Appl., vol. 41, no. 3, pp. 853–860, 2014.\n\n[5] S. Vogel, H. Ney, and C. Tillmann, “HMM-based word alignment in statistical translation,” in COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics, 1996.\n\n[6] A. Mnih and G. Hinton, “Three new graphical models for statistical language modelling,” in Proceedings of the 24th international conference on Machine learning, 2007, pp. 641–648.\n\n[7] H. Murveit and R. Moore, “Integrating natural language constraints into HMM-based speech recognition,” in International Conference on Acoustics, Speech, and Signal Processing, IEEE, 1990, pp. 573–576.\n\n[8] P. Ruch, R. Baud, and A. Geissbühler, “Using lexical disambiguation and named-entity recognition to improve spelling correction in the electronic patient record,” Artif. Intell. Med., vol. 29, no. 1–2, pp. 169–184, 2003.\n\n[9] Hochreiter, Sepp, and Jürgen Schmidhuber. \"Long short-term memory.\" Neural computation 9, no. 8 (1997): 1735-1780.\n\n[10] Zhang, Xiang, and Yann LeCun. \"Text understanding from scratch.\" arXiv preprint arXiv:1502.01710 (2015).\n\n[11] T. Young, D. Hazarika, S. Poria, and E. Cambria, “Recent trends in deep learning based natural language processing,” Ieee Comput. Intell. Mag., vol. 13, no. 3, pp. 55–75, 2018.\n\n[12] U. D. Gandhi, P. Malarvizhi Kumar, G. Chandra Babu, and G. Karthick, “Sentiment analysis on twitter data by using convolutional neural network (CNN) and long short term memory (LSTM),” Wirel. Pers. Commun., pp. 1–10, 2021.\n\n[13] Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. \"Attention is all you need.\" Advances in neural information processing systems 30 (2017).\n\n[14] Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. \"Bert: Pre-training of deep bidirectional transformers for language understanding.\" arXiv preprint arXiv:1810.04805 (2018).\n\n[15] Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. \"Language models are unsupervised multitask learners.\" OpenAI blog 1, no. 8 (2019): 9.\n\n[16] Q. Zhu and J. Luo, “Generative pre-trained transformer for design concept generation: an exploration,” Proc. Des. Soc., vol. 2, pp. 1825–1834, 2022.\n\n[17] Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan et al. \"Language models are few-shot learners.\" Advances in neural information processing systems 33 (2020): 1877-1901.\n\n[18] Lewis, Mike, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. \"Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.\" arXiv preprint arXiv:1910.13461 (2019).\n\n[19] Pan, Shirui, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. \"Unifying Large Language Models and Knowledge Graphs: A Roadmap.\" arXiv preprint arXiv:2306.08302 (2023).\n\n[20] Hendrycks, Dan, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. \"Measuring massive multitask language understanding.\" arXiv preprint arXiv:2009.03300 (2020).\n\n[21] Chowdhery, Aakanksha, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham et al. \"Palm: Scaling language modeling with pathways.\" arXiv preprint arXiv:2204.02311 (2022).\n\n[22] Touvron, Hugo, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière et al. \"Llama: Open and efficient foundation language models.\" arXiv preprint arXiv:2302.13971 (2023).\n\n[23] Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas et al. \"Training compute-optimal large language models.\" arXiv preprint arXiv:2203.15556 (2022).\n\n[24] https://shrinkthatfootprint.com/carbon-footprint-of-training-gpt-3-and-large-language-models/\n\n[25] He, Pengcheng, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. \"Deberta: Decoding-enhanced bert with disentangled attention.\" arXiv preprint arXiv:2006.03654 (2020).\n\n[26] Liang, Paul Pu, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. \"Towards understanding and mitigating social biases in language models.\" In International Conference on Machine Learning, pp. 6565-6576. PMLR, 2021.\n\n[27] Zhang, Jizhi, Keqin Bao, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. \"Is chatgpt fair for recommendation? evaluating fairness in large language model recommendation.\" arXiv preprint arXiv:2305.07609 (2023).\n\n[28] Zhang, Tianhua, Hongyin Luo, Yung-Sung Chuang, Wei Fang, Luc Gaitskell, Thomas Hartvigsen, Xixin Wu, Danny Fox, Helen Meng, and James Glass. \"Interpretable unified language checking.\" arXiv preprint arXiv:2304.03728 (2023). \n\n[29] Hu, Edward J., Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. \"Lora: Low-rank adaptation of large language models.\" arXiv preprint arXiv:2106.09685 (2021).\n\n[30] Dettmers, Tim, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. \"Qlora: Efficient finetuning of quantized llms.\" arXiv preprint arXiv:2305.14314 (2023).\n\n[31] Hsieh, Cheng-Yu, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. \"Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.\" arXiv preprint arXiv:2305.02301 (2023).\n\n[32] Gu, Yuxian, Li Dong, Furu Wei, and Minlie Huang. \"Knowledge Distillation of Large Language Models.\" arXiv preprint arXiv:2306.08543 (2023). \n\n[33] Hu, Zhiqiang, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. \"LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models.\" arXiv preprint arXiv:2304.01933 (2023).\n\n[34] McKenna, Nick, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. \"Sources of Hallucination by Large Language Models on Inference Tasks.\" arXiv preprint arXiv:2305.14552 (2023).\n\n[35] Li, Zihao. \"The dark side of chatgpt: Legal and ethical challenges from stochastic parrots and hallucination.\" arXiv preprint arXiv:2304.14347 (2023).\n\n[36] Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang et al. \"Training language models to follow instructions with human feedback.\" Advances in Neural Information Processing Systems 35 (2022): 27730-27744.\n\n[37] Kaplan, Jared, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. \"Scaling laws for neural language models.\" arXiv preprint arXiv:2001.08361 (2020).\n\n[38] Zaheer, Manzil, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham et al. \"Big bird: Transformers for longer sequences.\" Advances in neural information processing systems 33 (2020): 17283-17297.\n\n[39] Dai, Zihang, Guokun Lai, Yiming Yang, and Quoc Le. \"Funnel-transformer: Filtering out sequential redundancy for efficient language processing.\" Advances in neural information processing systems 33 (2020): 4271-4282.\n\n[40] Beltagy, Iz, Matthew E. Peters, and Arman Cohan. \"Longformer: The long-document transformer.\" arXiv preprint arXiv:2004.05150 (2020).","metadata":{}},{"cell_type":"code","source":"submission_df = pd.DataFrame()\nsubmission_df['type'] = ['essay_category', 'essay_url', 'feedback1_url', 'feedback2_url', 'feedback3_url']\nsubmission_df['value'] = ['Text data', 'https://www.kaggle.com/code/datamafia7/a-text-odyssey-the-past-present-and-future-of-nlp', \\\n 'https://www.kaggle.com/code/hodajalali/graph-learning-and-complex-networks', \\\n 'https://www.kaggle.com/code/coder98/state-of-content-recommendation-engines-in-2023', \\\n 'https://www.kaggle.com/code/bazooo/transformers-from-2020']\nsubmission_df.to_csv(\"submission.csv\", index=False)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true},"execution_count":null,"outputs":[]}]}