{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Introduction**\n\nThis essay is about the application of transformers in Natural Language Processing (NLP).\n\nTransformer models have achieved state-of-the-art results in many NLP problems. They are the technological backbone behind the recent chatbot craze. \n\nThere are hopes that transformer-based machine learning technologies could eventually power transformative AI, defined as artificially intelligent programs or agents that could radically alter human society.\n\nWhile transformers are incredibly adept at many NLP tasks, there are fundamental shortcomings in their design and architecture that inhibit them from fully mastering NLP.\n\nThis essay reviews the vanguard transformer models and their applications, and underscores their limitations. ","metadata":{}},{"cell_type":"markdown","source":"# **1. The objectives of NLP**\n\nBefore diving into transformers, we must first understand what problems they were meant to solve.\n\n[The NLP tasks assigned to machines](https://huggingface.co/learn/nlp-course/chapter1/2?fw=pt) are usually among the following:\n\n*(a) Classification*<br>\nEarly advancements of deep learning in NLP were in sentiment analysis. In industry, simple binary predictions are hugely helpful: for example, by classifying messages as toxic/non-toxic, social media companies can streamline their content moderation workflow.<br>\nMost of Kaggle's text-based machine learning competitions are classification problems.\n\n*(b) Named entity recognition*<br>\nThis refers to extracting entities (e.g. noun/verb/adjective, people/place/object) from text. Some machine learning services such as those from Microsoft Azure offer highly accurate models for picking out desired features from user-provided text.\n\n*(c) Text summarization*<br>\nWe all want a machine that can pore over large volumes of text and summarize them concisely. Text summarization is about condensing text and conveying only the most important parts.\n\n*(d) Text translation*<br>\nTranslating text from one language to another is one of the most desirable goals of NLP, and one of the most difficult to get right since how words should be translated is highly context-sensitive.\n\n*(e) Text generation*<br>\nThe most complex problem in NLP is getting machines to generate coherent and meaningful text. There has been monumental progress in this area in recent years, with the advance of conversational Large Language Models.","metadata":{}},{"cell_type":"markdown","source":"# **2. Technical review**","metadata":{}},{"cell_type":"markdown","source":"## 2.1 Preprocessing\n\nComputers cannot handle raw text. In NLP, the challenge begins with finding ways to convert letters and words into numbers that machines can process (preprocessing). \n\nFirst, we must break down the text into smaller units for analysis. This is done by applying [tokenization techniques (usually at the level of individual words and sentences)](https://www.kaggle.com/discussions/getting-started/269233). Typically, we remove punctuation, symbols and spaces from consideration. This splitting of text into smaller pieces is known as \"tokenization\", and the resultant smaller strings of text are known as \"tokens\". \n\nIn practice, this step may not be necessary since pre-trained transformers such as BERT include tokenization. If you are to build a neural network from scratch, though, tokenization can be done using Python's [NLTK library](https://www.nltk.org), which offers many simple preprocessing methods for text data.\n\nThere are two main approaches to converting word \"tokens\" into numbers: text encoding and text embedding.\n\nExamples of text encoding techniques are bag-of-words and one-hot encoding. These methods represent text data with numerical vectors. A typical method is by assigning each word in a given text a unique index, then creating a vector that contains a 1 in the index corresponding to the word and 0's everywhere else. \n\nThis approach can be efficient and adequate for some classification tasks, but it cannot capture the nuances of text well enough to be a solution to more advanced NLP problems due to its inability to account for the relationships between different words and phrases.\n\nAdvances in deep learning and the availability of cheap and fast computing on large datasets have enabled the development of text embedding techniques. \n\nThe typical text embedding uses neural networks to [iteratively learn the continuous vector representations of the words or phrases found in the entire text, giving similar numerical representations to similar words](https://www.kdnuggets.com/2021/11/guide-word-embedding-techniques-nlp.html). This process is known as \"vectorization\", and is considered a more sophisticated approach that can account for semantic context.\n\nPopular embeddings are:<br>\n- Word2Vec: a shallow neural network-based approach that produces a dense vector representation for each word by predicting the context of the word in the text. \n- FastText: a modification of Word2Vec that handles subword information by breaking words down into smaller parts (character n-grams). This allows it to produce embeddings for rare or out-of-vocabulary words.\n- BERT: a pre-trained transformer-based model that learns embeddings from huge volumes of text data. It produces context-sensitive embeddings for individual words, phrases, and sentences.\n- RoBERTa: a variant of BERT that was trained on more and more diverse text data. It has been found to outperform BERT on many downstream NLP tasks.\n\nHaving vectorized text data, we can then use deep learning models to learn the patterns in them. We will first review the traditional approach, then move on to the transformers revolution.","metadata":{}},{"cell_type":"markdown","source":"## 2.2 RNN, GRU and LSTM\n\nRecurrent neural networks (RNN) and their inspired architectures, GRU (Gated Recurrent Unit) and LSTM (Long Short-Term Memory) form the basis of modern-day deep learning for NLP tasks.\n\nRNN-based models were designed to capture the temporal dependencies in text sequences. When training an RNN, the outputs of the previous step are passed as inputs to the current step. This produces a model that can account for the order of words in the text. \n\nWhen training RNNs for NLP tasks, researchers found that they can suffer from [\"vanishing/exploding gradient problem\"](https://doi.org/10.48550/arXiv.1808.03314): the former is when gradients become too small for the updating of model weights in backpropagation, and the latter is when the gradients become too large, causing weight updates to become erratic. In both cases, the model fails to learn beyond a bottleneck.\n\nTo address this problem, RNNs with gating mechanisms such as GRUs and LSTMs were invented. GRUs selectively update the hidden state, while LSTMs selectively forget or remember information from previous steps. \n\nBoth architectures can capture long-term dependencies in text sequences without running into the vanishing or exploding gradient problem. However, they are computationally expensive and may overfit on smaller datasets.\n\nRNN-based models have made huge advancements in many NLP tasks, but remain severely limited in text summarization, translation, and generation capabilities since they have trouble accounting for the semantic nuances of language.\n\nWhile transformer-based models have shown much more impressive results in the more advanced NLP tasks, RNNs and their variants remain important tools due to their generally lower memory and computing power requirements. There have even been attempts to [reinvent RNNs to match transformer performance](https://doi.org/10.48550/arXiv.2305.13048).","metadata":{}},{"cell_type":"markdown","source":"## 2.3 Transformers","metadata":{}},{"cell_type":"markdown","source":"### 2.3.1 What are transformers?\n\nTransformers are neural network architectures that mimic the way humans think about text with mechanisms for:\n- [positional encoding](https://www.kdnuggets.com/2023/01/concepts-know-getting-transformer.html): determining the position of a word in the text\n- attention: evaluating the importance of a word in a particular sentence\n- self-attention: each word in the input can reference every other word in the input.\n\nIn the landmark 2017 [\"Attention Is All You Need\"](https://arxiv.org/abs/1706.03762) paper, transformers were demonstrated to have advantages over RNN-based architectures in that they can be parallelized (since text is processed non-sequentially rather than sequentially), allowing them to be trained on much larger corpuses of text much more time-effectively, and with superior performance on machine translation tasks to boot.\n\nSome popular transformers are:\n- BERT (Bidirectional Encoder Representations from Transformers): a pre-trained transformer that uses [bidirectional training to create contextualized word embeddings](https://machinelearningmastery.com/a-brief-introduction-to-bert)\n- RoBERTa (Robustly Optimized BERT approach): a modification of BERT that has been trained on larger amounts of data to improve performance on various NLP tasks\n- ALBERT (A Lite BERT): a smaller and faster version of BERT that achieves comparable performance on various NLP tasks while requiring less computation\n- GPT (Generative Pre-trained Transformer): an autoregressive model that uses a left-to-right training approach to generate text. ","metadata":{}},{"cell_type":"markdown","source":"### 2.3.2 How to incorporate transformers in our projects?\n\nPre-trained transformer models are freely available on [Hugging Face](https://huggingface.co), and can be accessed through the Python transformers library. As a demonstration, we will walk through how to [incorporate pre-trained transformers into code using pipelines](https://www.kdnuggets.com/2023/02/simple-nlp-pipelines-huggingface-transformers.html). ","metadata":{}},{"cell_type":"code","source":"#\"quiet\" tags are added to hide lengthy output messages\n#install the transformers library\n!pip install transformers -q\n#may be required (depending on environment) to avoid runtime errors\n!pip install --upgrade accelerate -q\n#to accelerate training\n!pip install xformers -q","metadata":{"execution":{"iopub.status.busy":"2023-06-13T11:48:06.841260Z","iopub.execute_input":"2023-06-13T11:48:06.841687Z","iopub.status.idle":"2023-06-13T11:51:16.289599Z","shell.execute_reply.started":"2023-06-13T11:48:06.841658Z","shell.execute_reply":"2023-06-13T11:51:16.286798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#import pipeline\nfrom transformers import pipeline\n\n#create pipeline \nclassifier = pipeline(task=\"sentiment-analysis\", \n                      model=\"distilbert-base-uncased-finetuned-sst-2-english\")\n#plug in the text to classify\npreds = classifier(\"This seems too good to be true.\")\nprint(f\"label: {preds[0]['label']}, score: {round(preds[0]['score'], 4)}\")","metadata":{"execution":{"iopub.status.busy":"2023-06-13T11:51:30.027189Z","iopub.execute_input":"2023-06-13T11:51:30.027758Z","iopub.status.idle":"2023-06-13T11:51:53.028066Z","shell.execute_reply.started":"2023-06-13T11:51:30.027717Z","shell.execute_reply":"2023-06-13T11:51:53.027132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can see that pre-trained transformer models may perform decently on classification tasks without further training. To optimize a transformer for a downstream task (e.g. detecting spam), [further training on a specialized dataset is required](https://www.kaggle.com/code/yeemeitsang/spam-text-classification-roberta).\n\nNext, to get a taste of the text generation capabilities of transformers, let us experiment with [text completion using an interactive graphical interface, gradio](https://huggingface.co/learn/nlp-course/chapter9/2?fw=pt):","metadata":{}},{"cell_type":"code","source":"#install gradio\n!pip install gradio -q","metadata":{"execution":{"iopub.status.busy":"2023-06-13T11:52:24.459696Z","iopub.execute_input":"2023-06-13T11:52:24.460363Z","iopub.status.idle":"2023-06-13T11:52:41.120246Z","shell.execute_reply.started":"2023-06-13T11:52:24.460307Z","shell.execute_reply":"2023-06-13T11:52:41.118621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gradio as gr\n\n#create pipeline \nmodel = pipeline(task=\"text-generation\", model='gpt2', max_new_tokens=50)\n#helper function for prediction\ndef predict(prompt):\n    completion = model(prompt)[0][\"generated_text\"]\n    return completion\n#launch GUI\ngr.Interface(fn=predict, inputs=\"text\", outputs=\"text\").launch()","metadata":{"execution":{"iopub.status.busy":"2023-06-13T11:53:25.992260Z","iopub.execute_input":"2023-06-13T11:53:25.993358Z","iopub.status.idle":"2023-06-13T11:53:50.062903Z","shell.execute_reply.started":"2023-06-13T11:53:25.993307Z","shell.execute_reply":"2023-06-13T11:53:50.061108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.3.3 How prominent are transformers?\n\nWe will be evaluating the occurrences of several deep learning-related keywords on [arXiv](https://arxiv.org), an open archive of research articles. \n\nWe will be retrieving papers on computation and language in the computer science field (**cs.CL**) published between the years 2021 and 2023 from the [metadata on arXiv articles provided by Kaggle](https://www.kaggle.com/datasets/Cornell-University/arxiv). An occurrence of plus one is counted when a selected keyword is found in the title or abstract of an arXiv article.","metadata":{}},{"cell_type":"code","source":"#import libraries\nimport json\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n#the filepath to the arXiv metadata\nfile = '/kaggle/input/2023-kaggle-ai-report/arxiv_metadata_20230510.json'\n#set the years to filter\nyears = ['2021', '2022', '2023']\n\n#helper function to compile dataframe on select topics\ndef get_dataframe(topics):\n    #create empty dataframe to store data from json file\n    df = pd.DataFrame(columns = ['date published', 'categories', 'title', 'abstract', 'doi'])\n    with open(file, 'r') as f:\n        for line in f:\n            data_dict = json.loads(line)\n            #split the category string \n            cat = data_dict['categories']\n            #select only cs.CL papers \n            if 'cs.CL' in cat:\n                date = data_dict['versions'][0]['created']\n                if any(year in date for year in years):\n                    if any(topic in data_dict['title'].lower() for topic in topics) or \\\n                       any(topic in data_dict['abstract'].lower() for topic in topics):\n                        df.loc[len(df)] = [date, \n                                           cat, \n                                           data_dict['title'], \n                                           data_dict['abstract'][:50],\n                                           data_dict['doi']]\n    return df","metadata":{"execution":{"iopub.status.busy":"2023-06-13T12:04:50.336834Z","iopub.execute_input":"2023-06-13T12:04:50.337315Z","iopub.status.idle":"2023-06-13T12:04:50.347801Z","shell.execute_reply.started":"2023-06-13T12:04:50.337280Z","shell.execute_reply":"2023-06-13T12:04:50.346617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get papers on transformers\ndf_transformer = get_dataframe(['transformer'])\ndf_transformer.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-13T12:05:12.724712Z","iopub.execute_input":"2023-06-13T12:05:12.725168Z","iopub.status.idle":"2023-06-13T12:06:17.796764Z","shell.execute_reply.started":"2023-06-13T12:05:12.725135Z","shell.execute_reply":"2023-06-13T12:06:17.795418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get papers on neural networks\ndf_nn = get_dataframe(['neural network'])\ndf_nn.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-13T12:07:16.793285Z","iopub.execute_input":"2023-06-13T12:07:16.794094Z","iopub.status.idle":"2023-06-13T12:07:53.335758Z","shell.execute_reply.started":"2023-06-13T12:07:16.794044Z","shell.execute_reply":"2023-06-13T12:07:53.334488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get papers on BERT\ndf_bert = get_dataframe(['bert'])\ndf_bert.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-13T12:09:26.453169Z","iopub.execute_input":"2023-06-13T12:09:26.453634Z","iopub.status.idle":"2023-06-13T12:10:07.908861Z","shell.execute_reply.started":"2023-06-13T12:09:26.453599Z","shell.execute_reply":"2023-06-13T12:10:07.907812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get papers on GPT\ndf_gpt = get_dataframe(['gpt'])\ndf_gpt.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-13T12:10:19.299331Z","iopub.execute_input":"2023-06-13T12:10:19.299891Z","iopub.status.idle":"2023-06-13T12:10:56.087480Z","shell.execute_reply.started":"2023-06-13T12:10:19.299846Z","shell.execute_reply":"2023-06-13T12:10:56.086231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get papers on RNN\ndf_rnn = get_dataframe(['rnn', 'recurrent neural network'])\ndf_rnn.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-13T12:12:28.784937Z","iopub.execute_input":"2023-06-13T12:12:28.785422Z","iopub.status.idle":"2023-06-13T12:13:03.102794Z","shell.execute_reply.started":"2023-06-13T12:12:28.785389Z","shell.execute_reply":"2023-06-13T12:13:03.101428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get papers on LSTM\ndf_lstm = get_dataframe(['lstm', 'long short-term memory'])\ndf_lstm.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-13T12:13:09.197815Z","iopub.execute_input":"2023-06-13T12:13:09.198255Z","iopub.status.idle":"2023-06-13T12:13:43.549539Z","shell.execute_reply.started":"2023-06-13T12:13:09.198223Z","shell.execute_reply":"2023-06-13T12:13:43.548664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#uncomment and plug in any keyword(s) to check\n#df_trial = get_dataframe(['electra'])\n#df_trial.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#plot bar chart of keywords against frequencies\nnames = ['transformer', 'BERT', 'neural network', 'GPT', 'LSTM', 'RNN']\nfreq = [len(df_transformer), len(df_bert), len(df_nn), \n        len(df_gpt), len(df_lstm), len(df_rnn)]\nplt.bar(names, freq)\nplt.title('Occurrences in arXiv cs.CL articles from 2021-2023')\nplt.xlabel('keyword')\nplt.ylabel('number of occurrences')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-13T12:18:24.020081Z","iopub.execute_input":"2023-06-13T12:18:24.021609Z","iopub.status.idle":"2023-06-13T12:18:24.282468Z","shell.execute_reply.started":"2023-06-13T12:18:24.021547Z","shell.execute_reply":"2023-06-13T12:18:24.279878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Although this search is not exhaustive, we get a sense that transformer-based models are hotter topics of research than RNN-based ones.","metadata":{}},{"cell_type":"code","source":"#uncomment to check essays with doi attached\n#df_transformer[df_transformer.doi.notnull()]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.4 Large Language Models\n\n[Large Language Models (LLMs) refer to pre-trained language models that have been \"scaled up\" to display new capabilities](https://arxiv.org/abs/2303.18223). \n\nThe most famous example is OpenAI's ChatGPT. Due to its resounding success, GPT has become somewhat synonymous with ChatGPT.","metadata":{}},{"cell_type":"code","source":"#import wordcloud\nfrom wordcloud import WordCloud\n\n#join all essay titles into a single string\ntext = ' '.join(df_gpt['title'].tolist())\n#create a word cloud object\nwordcloud = WordCloud(background_color='white', \n                      max_words=50, colormap='gist_rainbow_r').generate(text)\n# plot the word cloud\nplt.figure(figsize=(8, 8))\nplt.imshow(wordcloud, interpolation='bilinear')\nplt.axis('off')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-13T12:19:43.152810Z","iopub.execute_input":"2023-06-13T12:19:43.153251Z","iopub.status.idle":"2023-06-13T12:19:43.512394Z","shell.execute_reply.started":"2023-06-13T12:19:43.153217Z","shell.execute_reply":"2023-06-13T12:19:43.510774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The ability of GPT-based LLMs to perform general-purpose tasks without fine-tuning (few-shot performance) was first made apparent with the introduction of GPT-3 in the 2020 [\"Language Models are Few-Shot Learners\"](https://doi.org/10.48550/arXiv.2005.14165) paper.\nThe strong performance of GPT-3 and successive versions on tasks they were not trained for makes LLMs a highly promising \"technology of the future\". \n\nThe popularity of LLMs in consumer applications has established them as viable candidates for virtual assistants and artificial content creators. ChatGPT can write essays and poetry, and generate computer code to a passable standard.\n\nThe potential for GPT-based models to [improve content accuracy through feedback mechanisms](https://doi.org/10.48550/arXiv.2302.12813), and by increasing their parameters has led to the recent buzz over an incoming technological revolution where computers displace humans in many facets of the economy and public life.\n\nDespite their impressive performance and potential for growth, the sky is not the limit for LLMs. There are [limits on the amount of computational resources we can devote to training a model, thus more efficient alternatives tailored to specific tasks are preferred over scaling up general-purpose LLMs in search of capability breakthroughs](https://arxiv.org/pdf/2209.00099.pdf). ","metadata":{}},{"cell_type":"markdown","source":"# **3. Limitations**","metadata":{}},{"cell_type":"markdown","source":"While transformers have achieved remarkable progress in NLP, they are limited when it comes to solving more complex NLP problems, such as text generation. Some limitations are:\n- Lacking common sense: transformers do not have experience-based common sense knowledge. This can lead to them generating text that is plausible at first glance, but is actually nonsensical, contradictory or false. Also, for time-sensitive tasks (e.g. news reports), transformers are behind the curve.\n- Difficulty generating narratively-coherent content: modeling long-term dependencies in text remains challenging for transformers. They may thus struggle with generating lengthy passages that maintain coherence throughout the entire narrative.\n- Limited creativity: transformer-based models tend to generate output that is similar to their training data, which results in bland and repetitive output that is not sensitive enough to the differences in inputs. \n- Difficulty monitoring outputs: LLMs may generate incorrect, biased, or otherwise problematic text, especially when the training data is problematic to begin with. Modifying or filtering their outputs to ensure their appropriateness requires additional measures such as post-processing or pre-training on filtered data.<br>\n\nIn attempts to improve in these areas, new transformer architectures are being experimented with, such as:\n- [Block-Recurrent Transformers leveraging the recurrence mechanisms of LSTMs to improve the performance of transformers in language modeling tasks over long-range sequences](https://towardsdatascience.com/block-recurrent-transformer-lstm-and-transformer-combined-ec3e64af971a)\n- [augmentation with external fact-checking and filtering mechanisms](https://arxiv.org/pdf/2302.12813.pdf).","metadata":{}},{"cell_type":"markdown","source":"# **Conclusion**\n\nWith transformers, we have achieved many results in NLP that were previously out of reach. While it is easy to get caught up in the hype of what transformers could do, we have to be aware of their limitations and work within them, trying to make the models more efficient and accurate without endlessly scaling computation.\n\nThis means not applying them to tasks they are ill-suited to, carefully reviewing their outputs in critical functions instead of automating them away, and devising control mechanisms to ensure their appropriateness with respect to a given task.","metadata":{}},{"cell_type":"markdown","source":"# **References**\n(Formatted by ChatGPT)<br>\n\n**Articles**<br>\nAgarwal, N. (2022). The Ultimate Guide To Different Word Embedding Techniques In NLP. Retrieved from https://www.kdnuggets.com/2021/11/guide-word-embedding-techniques-nlp.html<br>\nKafritsas, N. (2022). Block-Recurrent Transformer: LSTM and Transformer Combined. Towards Data Science. Retrieved from https://towardsdatascience.com/block-recurrent-transformer-lstm-and-transformer-combined-ec3e64af971a<br>\nNaduvin, R. (2023). Concepts You Should Know Before Getting Into Transformers. Retrieved from https://www.kdnuggets.com/2023/01/concepts-know-getting-transformer.html<br>\nShastri, Y. (2023). Simple NLP Pipelines with HuggingFace Transformers. Retrieved from https://www.kdnuggets.com/2023/02/simple-nlp-pipelines-huggingface-transformers.html<br>\nTam, A. (2022). A Brief Introduction to BERT. Retrieved from https://machinelearningmastery.com/a-brief-introduction-to-bert\n\n**Dataset**<br>\narXiv.org submitters. (2023). arXiv Dataset. https://doi.org/10.34740/KAGGLE/DSV/5895943\n\n**Papers**<br>\nBrown, T. B., Mann, B. and Ryder, N. et al (2020). Language Models are Few-Shot Learners. https://doi.org/10.48550/arXiv.2005.14165<br>\nPeng, B., Alcaide, E. and Anthony, Q. et al (2021). RWKV: Reinventing RNNs for the Transformer Era. https://doi.org/10.48550/arXiv.2305.13048<br>\nPeng, B., Galley, M. and He, P. et al (2023). Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback. arXiv preprint arXiv:2302.12813 [cs.CL]. https://doi.org/10.48550/arXiv.2302.12813<br>\nSherstinsky, A. (2018). Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network. https://doi.org/10.48550/arXiv.1808.03314<br>\nTreviso, M., Lee, J.-U., and Ji, T. et al (2022). Efficient Methods for Natural Language Processing: A Survey. https://doi.org/10.48550/arXiv.2209.00099<br>\nVaswani, A., Shazeer, N. and Parmar, N. et al (2017). Attention Is All You Need. \nhttps://doi.org/10.48550/arXiv.1706.03762<br>\nZhao, W. X., Zhou, K. and Li, J. et al(2023). A Survey of Large Language Models. arXiv preprint arXiv:2303.18223 [cs.CL]. https://doi.org/10.48550/arXiv.2303.18223\n\n**Websites**<br>\nHugging Face. (2022). The Hugging Face Course. Retrieved from https://huggingface.co/course [Online; accessed June 11, 2023]<br>\nNatural Language Toolkit (n.d.). Natural Language Toolkit (NLTK). Retrieved from https://www.nltk.org/","metadata":{}},{"cell_type":"markdown","source":"# **Submission to the 2023 Kaggle AI Report competition**","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-13T13:07:04.605245Z","iopub.execute_input":"2023-06-13T13:07:04.605812Z","iopub.status.idle":"2023-06-13T13:07:04.629670Z","shell.execute_reply.started":"2023-06-13T13:07:04.605776Z","shell.execute_reply":"2023-06-13T13:07:04.628492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vals = [\"'Text data'\", \n        \"'https://www.kaggle.com/yeemeitsang/cracking-nlp-with-transformers'\",\n        \"''\",\n        \"''\",\n        \"''\"]\nsub.value = vals\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-13T13:07:07.118889Z","iopub.execute_input":"2023-06-13T13:07:07.120276Z","iopub.status.idle":"2023-06-13T13:07:07.132154Z","shell.execute_reply.started":"2023-06-13T13:07:07.120234Z","shell.execute_reply":"2023-06-13T13:07:07.130754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-06-13T13:07:12.046956Z","iopub.execute_input":"2023-06-13T13:07:12.047374Z","iopub.status.idle":"2023-06-13T13:07:12.060073Z","shell.execute_reply.started":"2023-06-13T13:07:12.047345Z","shell.execute_reply":"2023-06-13T13:07:12.059052Z"},"trusted":true},"execution_count":null,"outputs":[]}]}