{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"<center><font size=7>2023 Kaggle AI Report</font>\n<center><font size=6>Selected Topic: Text Data</font>\n    ","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"<font size=5 color='blue'>Table of Contents</font>\n<hr style=\"height:5px;border-width:0;color:gray;background-color:blue\">\n\n* [1. Introduction](#introduction)\n* [2. Reading and Filtering the Data](#section2)\n    - [2.1 Plotting the Categories of the Selected Papers](#section21) \n* [3. A First Overview of the Content of the Text Data Papers](#section3)\n    - [3.1 Which Languages are Considered ?](#section31)\n    - [3.2 Which Levels of Text Analysis?](#section32)\n    - [3.3 Which NLP Tasks?](#section33)\n    - [3.4 Which Machine Learning Approaches?](#section34)\n    - [3.5 Which NN Architectures?](#section35)\n* [4. Intermediate Conclusions](#section4)\n","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"<a id=\"introduction\"></a>\n# 1. Introduction\nIn this essay, we try to summarize the advancements for the last two years in natural language processing (NLP), i.e. the subfield of AI dedicated to analyzing and processing text data. For that, we first analyze the titles and abstracts of the arxiv articles devoted to this field. This analysis which aims at identifying the main trends in the field, is mainly a statistic one. It consists of counting the number of occurences of keywords related to these trends. \n\n","metadata":{}},{"cell_type":"code","source":"import time\nfrom datetime import datetime\nimport json\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport re","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:56:02.625936Z","iopub.execute_input":"2023-05-29T10:56:02.627138Z","iopub.status.idle":"2023-05-29T10:56:02.634141Z","shell.execute_reply.started":"2023-05-29T10:56:02.627092Z","shell.execute_reply":"2023-05-29T10:56:02.632631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"file_name = '/kaggle/input/2023-kaggle-ai-report/arxiv_metadata_20230510.json'","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:56:02.639409Z","iopub.execute_input":"2023-05-29T10:56:02.639814Z","iopub.status.idle":"2023-05-29T10:56:02.648015Z","shell.execute_reply.started":"2023-05-29T10:56:02.639784Z","shell.execute_reply":"2023-05-29T10:56:02.646717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section2\"></a>\n# 2. Reading and Filtering the Data\nWe consider that a paper about **Text data** necessasily belongs to the category **cs.CL** even if it can \nbelong to other categories too. We are interested in recent papers only. Therefore, when reading the json file, we first select the papers belonging to the category cs.CL, then those corresponding to the years 2021, 2022, and 2023.\nReading and filtering the *arxiv json* file will result in a list containing the papers we are intereseted in. Each paper is represented by a dictionary whose keys are the properties of the papers (title, abstract, ...). In addition to the papers, we also record the categories to which belong these papers, and their frequencies.\n","metadata":{}},{"cell_type":"code","source":"date_format = \"%a, %d %b %Y %H:%M:%S %Z\"\n    \ntstart = time.time()\nlst_Txt_2years = []\nnb_all = 0\nnb_cat_csCL = 0\nnb_2years = 0\ndic_categs = {}\nfor line in open(file_name, 'r'):\n    nb_all += 1\n    dic = json.loads(line)\n    lst_cats = list(dic['categories'].split(\" \"))\n    if 'cs.CL' in lst_cats:\n      nb_cat_csCL +=1\n      str_creation_date = dic['versions'][0]['created']\n      creation_date = datetime.strptime(str_creation_date, date_format)\n      creation_year = creation_date.year\n      if (creation_year in [2021, 2022, 2023]):\n        nb_2years +=1\n        lst_Txt_2years.append(json.loads(line))\n        for cat in lst_cats:\n          if (cat in dic_categs.keys()):\n            dic_categs[cat] +=1\n          else:\n            dic_categs[cat] =1\ntend = time.time()\ndic_categs_sorted = {k: v for k, v in sorted(dic_categs.items(), key=lambda item: item[1], reverse=True)}\n\nprint('Elpased time:',round(tend-tstart),' s')\nprint('Total number of papers: ',nb_all)\nprint('Number of papers about Text processing: ',nb_cat_csCL)\nprint('Number of recent papers about Text processing: ',nb_2years)\n","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:56:02.652387Z","iopub.execute_input":"2023-05-29T10:56:02.653116Z","iopub.status.idle":"2023-05-29T10:56:53.027400Z","shell.execute_reply.started":"2023-05-29T10:56:02.653080Z","shell.execute_reply":"2023-05-29T10:56:53.026149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section21\"></a>\n## 2.1 Plotting the Categories of the Selected Papers.\nWhen reading and filtering the json file, we recorded the categories to which belong the selected papers (in addition to the main category cs.CL). Below, we plot a summary of this information. ","metadata":{}},{"cell_type":"code","source":"lst_categs = list(dic_categs_sorted)\nlst_freqs = [dic_categs_sorted[k] for k in dic_categs_sorted.keys()]\nnb_categs = len(dic_categs)\n\nlst_categs2 = lst_categs[:10]+['Others']\nlst_freqs2 = lst_freqs[:10]+[sum(lst_freqs[10:])]\n\nX = np.arange(len(lst_categs2))\nwidth = 0.5\n\nfig, ax = plt.subplots(figsize=(15,10))\ncateg_bars = ax.bar(X, lst_freqs2, width, label='XXXX')\n\n# Add some text for labels, title and custom x-axis tick labels, etc.\nax.set_ylabel('Frequency')\nax.set_xlabel('Category')\nax.set_title('Categories of the Text data papers, and their frequencies', fontsize=18, color='blue')\nax.set_xticks(X)\nax.set_xticklabels(lst_categs2)\n\nfor b in categ_bars:\n   height = b.get_height()\n   ax.annotate('{}'.format(height), xy=(b.get_x() + b.get_width() / 2, height), ha='center', va='bottom')\n\n","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:56:53.029456Z","iopub.execute_input":"2023-05-29T10:56:53.029816Z","iopub.status.idle":"2023-05-29T10:56:53.554248Z","shell.execute_reply.started":"2023-05-29T10:56:53.029788Z","shell.execute_reply":"2023-05-29T10:56:53.553100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Unsurprisingly, we notice that the main other categories to which belong the text data papers are: cs.AI (**Artificial intelligence**), cs.LG (**Machine learning**), and eess.AS (**Audio and Speech processing**). ","metadata":{}},{"cell_type":"markdown","source":"<a id=\"section3\"></a>\n# 3. A First Overview of the Content of the Text Data Papers.\n\nIn the following, we analyze the titles and the abstracts to extract the following information: (1)Which (human, natural) languages are considered in these papers? (2)Which levels of text analysis? (3)Which NLP tasks? (4)Which machine learning (ML) approaches are applied? and (5)Which neural networks architectures are used?\n","metadata":{}},{"cell_type":"code","source":"#The following function receives a list of words,a title and an abstract and returns True if the title or the abstract \n#contain at least one of the words of the list.\n\ndef paperContainsWord(title, abstract, lst_words):\n    contains = False\n    for word in lst_words:\n      word = word.lower()\n      if ((re.search(word,title.lower())!=None)or(re.search(word,abstract.lower())!=None)):\n        contains = True\n        break\n    return contains\n","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:56:53.556182Z","iopub.execute_input":"2023-05-29T10:56:53.557170Z","iopub.status.idle":"2023-05-29T10:56:53.565123Z","shell.execute_reply.started":"2023-05-29T10:56:53.557115Z","shell.execute_reply":"2023-05-29T10:56:53.563952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section31\"></a>\n## 3.1 Which Languages are Considered ?\nWe search the title and the abstract of each paper for each language name(s). We also search for the words meaning that the work presetend in the paper considers more than one language (e.g. in the case of machine translation). ","metadata":{}},{"cell_type":"code","source":"lst_languages = [\n'Abkhazian', 'Afar', 'Afrikaans', 'Akan', 'Albanian', 'Amharic', 'Arabic', 'Aragonese', 'Armenian', 'Assamese', 'Avaric', \n'Avestan', 'Aymara', 'Azerbaijani',\n'Bambara', 'Bashkir', 'Basque', 'Belarusian', 'Bengali', 'Bihari', 'Bislama', 'Bosnian', 'Breton', 'Bulgarian', 'Burmese',\n'Catalan', 'Central Khmer', 'Chamorro', 'Chechen', 'Chichewa', 'Chinese|Mandarin', 'Chuvash', 'Cornish', 'Corsican', 'Cree', 'Croatian', \n'Czech',\n'Danish', 'Divehi', 'Dutch', 'Dzongkha',\n'English', 'Esperanto', 'Estonian', ' Ewe ',\n'Faroese', 'Fijian', 'Finnish', 'French', 'Fulah', \n'Gaelic', 'Galician', 'Ganda', 'Georgian', 'German', 'Greek', 'Guarani', 'Gujarati',\n'Haitian', 'Hausa', 'Hebrew', 'Herero', 'Hindi', 'Hiri Motu', 'Hungarian',\n'Icelandic', 'Ido', 'Igbo', 'Indonesian', 'Interlingua', 'Interlingue', 'Inuktitut', 'Inupiaq', 'Irish', 'Italian', \n'Japanese', 'Javanese',\n'Kalaallisut', 'Kannada', 'Kanuri', 'Kashmiri', 'Kazakh', 'Kikuyu', 'Kinyarwanda', 'Kirghiz', 'Komi', 'Kongo', 'Korean', \n'Kuanyama', 'Kurdish',\n'Lao', 'Latin', 'Latvian', 'Limburgan', 'Lingala', 'Lithuanian', 'Luba-Katanga', 'Luxembourgish',\n'Macedonian', 'Malagasy', 'Malay', 'Malayalam', 'Maltese', 'Manx', 'Maori', 'Marathi', 'Marshallese', 'Mongolian',\n'Nauru', 'Navajo', 'Ndebele', 'Ndonga', 'Nepali', 'Northern Sami', 'Norwegian', \n'Occitan', 'Ojibwa', 'Oriya', 'Oromo', 'Ossetian', \n'Pali', 'Panjabi', 'Persian', 'Polish', 'Portuguese', 'Pushto', \n'Quechua', \n'Romanian', 'Romansh', 'Rundi', 'Russian', \n'Samoan', 'Sango', 'Sanskrit', 'Sardinian', 'Serbian', 'Shona', 'Sichuan', 'Sindhi', 'Sinhala', 'Slavic', 'Slovak', 'Slovenian', \n'Somali', 'Sotho', 'Spanish|Castilian', 'Sundanese', 'Swahili', 'Swati', 'Swedish', \n'Tagalog', 'Tahitian', 'Tajik', 'Tamil', 'Tatar', 'Telugu', 'Thai', 'Tibetan', 'Tigrinya', 'Tonga', 'Tsonga', 'Tswana', \n'Turkish', 'Turkmen', ' Twi ', \n'Uighur', 'Ukrainian', 'Urdu', 'Uzbek', \n'Venda', 'Vietnamese', 'Volapük',\n'Walloon', 'Welsh', 'Western Frisian', 'Wolof', \n'Xhosa', \n'Yiddish', 'Yoruba', \n'Zhuang', 'Zulu']\n\nnbLanguages = len(lst_languages)\n\n","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:56:53.568396Z","iopub.execute_input":"2023-05-29T10:56:53.568872Z","iopub.status.idle":"2023-05-29T10:56:53.586119Z","shell.execute_reply.started":"2023-05-29T10:56:53.568832Z","shell.execute_reply":"2023-05-29T10:56:53.585014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#To take account of the different forms in which we find the languages names in our papers: preceded and/or followed a blank \n#or à hyphen.\ndic_lst_languages={}\nfor numl in range(nbLanguages):\n  lng = lst_languages[numl]\n  lst_words = lng.split('|')\n  new_lng = ''\n  for wrd in lst_words:\n    new_lng = new_lng + ' '+wrd+' |'\n    new_lng = new_lng + ' '+wrd+'-|'   \n    new_lng = new_lng + '-'+wrd+' |'    \n    new_lng = new_lng + '-'+wrd+'-|'\n  new_lng = new_lng[:-1]\n  dic_lst_languages[lst_words[0]]=new_lng\n\ndic_lst_languages['Multilingual']='Multilingual|Multi-Lingual|Bilingual|Bi-Lingual|Crosslingual|Cross-Lingual|Translation'","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:56:53.587584Z","iopub.execute_input":"2023-05-29T10:56:53.588048Z","iopub.status.idle":"2023-05-29T10:56:53.604054Z","shell.execute_reply.started":"2023-05-29T10:56:53.588009Z","shell.execute_reply":"2023-05-29T10:56:53.603027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nbPapers = len(lst_Txt_2years)\n\ndic_freq_lang = {}\nfor k in dic_lst_languages.keys():\n  lng = dic_lst_languages[k]\n  lst_words = lng.split('|')\n  for nump in range(nbPapers):\n     title = lst_Txt_2years[nump]['title']\n     abstract = lst_Txt_2years[nump]['abstract']\n     if (paperContainsWord(title, abstract, lst_words)==True):\n       if k not in dic_freq_lang.keys():\n         dic_freq_lang[k]=[nump]\n       else:\n         dic_freq_lang[k].append(nump)\n\ndic_freq_lang_sorted = {k: v for k, v in sorted(dic_freq_lang.items(), key=lambda item: len(item[1]), reverse=True)}","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:56:53.605574Z","iopub.execute_input":"2023-05-29T10:56:53.606021Z","iopub.status.idle":"2023-05-29T10:58:25.322765Z","shell.execute_reply.started":"2023-05-29T10:56:53.605980Z","shell.execute_reply":"2023-05-29T10:58:25.321394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#How many languages are considered ? Which are the most used?\nprint('These ',nbPapers, ' papers consider ',len(dic_freq_lang),' languages.')\nprint('Top 10 of the languages:')\nfor numl in range(10):\n  k = list(dic_freq_lang_sorted.keys())[numl]\n  print(k,':', len(dic_freq_lang_sorted[k]),' papers.')","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:25.324399Z","iopub.execute_input":"2023-05-29T10:58:25.324815Z","iopub.status.idle":"2023-05-29T10:58:25.333353Z","shell.execute_reply.started":"2023-05-29T10:58:25.324784Z","shell.execute_reply":"2023-05-29T10:58:25.332030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Plotting the most frequent languages\nX_barplot = list(dic_freq_lang_sorted.keys())[:11]\nX_barplot.sort()\nY_barplot = [len(dic_freq_lang[lng]) for lng in X_barplot]\n\nX_barplot.append('Other lang.')\nlst_papers_other_langs = []\nfor lng in list(dic_freq_lang_sorted.keys())[11:]:\n  lst_papers_other_langs = list(set(lst_papers_other_langs).union(set(dic_freq_lang[lng])))\nY_barplot.append(len(lst_papers_other_langs))\n\nX = np.arange(len(X_barplot))\nwidth = 0.5\n\nfig, ax = plt.subplots(figsize=(15,10))\nlang_bars=ax.bar(X, Y_barplot, width, label='XXXX')\n\n# Add some text for labels, title and custom x-axis tick labels, etc.\nax.set_ylabel('Number of papers')\nax.set_xlabel('language')\nax.set_title('Languages and and their frequencies', fontsize=18, color='blue')\nax.set_xticks(X)\nax.set_xticklabels(X_barplot)\n\nfor b in lang_bars:\n   height = b.get_height()\n   ax.annotate('{}'.format(height), xy=(b.get_x() + b.get_width() / 2, height), ha='center', va='bottom')\n","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:25.334655Z","iopub.execute_input":"2023-05-29T10:58:25.335037Z","iopub.status.idle":"2023-05-29T10:58:25.891246Z","shell.execute_reply.started":"2023-05-29T10:58:25.335006Z","shell.execute_reply":"2023-05-29T10:58:25.890065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section32\"></a>\n## 3.2 Which Levels of Text Analysis?\nIt is generally admitted that natural language processing has 7 levels of analysis: phonology, morphology, lexical, syntactic, semantic, discourse, and pragmatic. The morphological and lexical analysis are usually merged. In this section, we count then we plot the number of occurences of the words related to these levels. ","metadata":{}},{"cell_type":"code","source":"dic_lst_levels = {'Phonology':'Phonolog', \n             'Morhological/Lexical':'Morphol|Lexic',\n             'Systanctic':'Syntactic|Syntax|Pars',\n             'Semantic':'Semantic',\n             'Discourse':'Discourse',\n             'Pragmatic':'Pragmatic'\n             } ","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:25.892621Z","iopub.execute_input":"2023-05-29T10:58:25.892998Z","iopub.status.idle":"2023-05-29T10:58:25.899095Z","shell.execute_reply.started":"2023-05-29T10:58:25.892947Z","shell.execute_reply":"2023-05-29T10:58:25.898007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dic_freq_level = {}\nfor k in dic_lst_levels.keys():\n  lvl = dic_lst_levels[k]\n  lst_words = lvl.split('|')\n  for nump in range(nbPapers):\n     title = lst_Txt_2years[nump]['title']\n     abstract = lst_Txt_2years[nump]['abstract']\n     if (paperContainsWord(title, abstract, lst_words)==True):\n       if k not in dic_freq_level.keys():\n         dic_freq_level[k] = 1\n       else:\n         dic_freq_level[k] +=1","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:25.903508Z","iopub.execute_input":"2023-05-29T10:58:25.903907Z","iopub.status.idle":"2023-05-29T10:58:27.073338Z","shell.execute_reply.started":"2023-05-29T10:58:25.903874Z","shell.execute_reply":"2023-05-29T10:58:27.072245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dic_freq_level_sorted = {k: v for k, v in sorted(dic_freq_level.items(), key=lambda item: item[1], reverse=True)}\nfor lvl in dic_freq_level_sorted.keys():\n   print(lvl, dic_freq_level_sorted[lvl])","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:27.074507Z","iopub.execute_input":"2023-05-29T10:58:27.074818Z","iopub.status.idle":"2023-05-29T10:58:27.082268Z","shell.execute_reply.started":"2023-05-29T10:58:27.074792Z","shell.execute_reply":"2023-05-29T10:58:27.081015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_barplot = list(dic_freq_level.keys())\nY_barplot = [dic_freq_level[level] for level in X_barplot]\n\nX = np.arange(len(X_barplot))\nwidth = 0.5\n\nfig, ax = plt.subplots(figsize=(15,10))\nlevel_bars=ax.bar(X, Y_barplot, width, label='XXXX')\n\nax.set_ylabel('Number of papers')\nax.set_xlabel('level')\nax.set_title('NLP levels considered in the papers, and their frequencies', fontsize=18, color='blue')\nax.set_xticks(X)\nax.set_xticklabels(X_barplot)\n\nfor b in level_bars:\n   height = b.get_height()\n   ax.annotate('{}'.format(height), xy=(b.get_x() + b.get_width() / 2, height), ha='center', va='bottom')\n","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:27.083630Z","iopub.execute_input":"2023-05-29T10:58:27.084002Z","iopub.status.idle":"2023-05-29T10:58:27.511173Z","shell.execute_reply.started":"2023-05-29T10:58:27.083949Z","shell.execute_reply":"2023-05-29T10:58:27.510095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section33\"></a>\n## 3.3 Which NLP Tasks?\nWe consider a set of important NLP tasks (see list below). We associate to each task a list of words or subwords such that if we find these words/subwords in the title or the abstract of a paper, we are 'almost sure' that this paper is at least partly devoted to this task. Of course, tasks are not mutually exclusive.","metadata":{}},{"cell_type":"code","source":"dic_lst_tasks = {'LM':'Language model', \n             'WM':'Word embedding',\n             'TM':'Topic model|Topic detection|Topic model|Topic classification|Topic segmentation|Topic prediction|Topic identification|Topic discovery',\n             'Classification':'Document classification|Text classification',\n             'Clustering':'Document clustering|Text clustering',\n             'NLG':'Text generation|Language generation|NLG',\n             'NLU':'Text understanding|Language understanding|NLU',\n             'Explanation':'Explanation',\n             'KG':'Knowledge graph',\n             'QA':'Question answer|Question-answer',\n             'NER':'Named entity|Named-entity',\n             'Translation':'Translation',\n             'Summarization':'Summar',\n             'Chatbot':'Chatbot'\n             }","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:27.512678Z","iopub.execute_input":"2023-05-29T10:58:27.513126Z","iopub.status.idle":"2023-05-29T10:58:27.519785Z","shell.execute_reply.started":"2023-05-29T10:58:27.513089Z","shell.execute_reply":"2023-05-29T10:58:27.518642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dic_freq_task = {}\nfor k in dic_lst_tasks.keys():\n  tsk = dic_lst_tasks[k]\n  lst_words = tsk.split('|')\n  for nump in range(nbPapers):\n     title = lst_Txt_2years[nump]['title']\n     abstract = lst_Txt_2years[nump]['abstract']\n     if (paperContainsWord(title, abstract, lst_words)==True):\n       if k not in dic_freq_task.keys():\n         dic_freq_task[k] =1\n       else:\n         dic_freq_task[k] +=1\n         \n     \ndic_freq_task_sorted = {k: v for k, v in sorted(dic_freq_task.items(), key=lambda item: item[1], reverse=True)}","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:27.521311Z","iopub.execute_input":"2023-05-29T10:58:27.521732Z","iopub.status.idle":"2023-05-29T10:58:31.164085Z","shell.execute_reply.started":"2023-05-29T10:58:27.521695Z","shell.execute_reply":"2023-05-29T10:58:31.162924Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for tsk in dic_freq_task_sorted.keys():\n   print(tsk,': ', dic_freq_task_sorted[tsk],' papers')","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:31.165337Z","iopub.execute_input":"2023-05-29T10:58:31.165670Z","iopub.status.idle":"2023-05-29T10:58:31.172207Z","shell.execute_reply.started":"2023-05-29T10:58:31.165641Z","shell.execute_reply":"2023-05-29T10:58:31.171120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_barplot = list(dic_freq_task_sorted.keys())\nX_barplot.sort()\nY_barplot = [dic_freq_task[k] for k in X_barplot]\n\nX = np.arange(len(X_barplot))\nwidth = 0.5\n\nfig, ax = plt.subplots(figsize=(15,10))\ntask_bars=ax.bar(X, Y_barplot, width, label='XXXX')\n\nax.set_ylabel('Number of papers')\nax.set_xlabel('Task')\nax.set_title('NLP tasks considered in the papers, and their frequencies', fontsize=18, color='blue')\nax.set_xticks(X)\nax.set_xticklabels(X_barplot)\n\nfor b in task_bars:\n   height = b.get_height()\n   ax.annotate('{}'.format(height), xy=(b.get_x() + b.get_width() / 2, height), ha='center', va='bottom')\n","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:31.173740Z","iopub.execute_input":"2023-05-29T10:58:31.174114Z","iopub.status.idle":"2023-05-29T10:58:31.697101Z","shell.execute_reply.started":"2023-05-29T10:58:31.174042Z","shell.execute_reply":"2023-05-29T10:58:31.696195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As one may expect, the most popular subject in these papers is LM (language model). The following statistics are about the two most popular LMs at the moment: *Bert* and *GPT*.","metadata":{}},{"cell_type":"code","source":"#About Bert & GPT\nnbBERT = 0\nnbGPT = 0\nfor nump in range(nbPapers):\n   title = lst_Txt_2years[nump]['title']\n   res = re.findall(r\"BERT\", title)\n   if (res!=[]):\n     nbBERT +=1\n   res = re.findall(r\"GPT\", title)\n   if (res!=[]):\n     nbGPT +=1\n    \nprint(nbBERT,nbGPT)","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:31.698556Z","iopub.execute_input":"2023-05-29T10:58:31.699580Z","iopub.status.idle":"2023-05-29T10:58:31.783782Z","shell.execute_reply.started":"2023-05-29T10:58:31.699545Z","shell.execute_reply":"2023-05-29T10:58:31.782581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_barplot = ['BERT','GPT']\nY_barplot = [nbBERT, nbGPT]\n\nX = np.arange(len(X_barplot))\nwidth = 0.5\n\nfig, ax = plt.subplots(figsize=(10,10))\nLM_bars=ax.bar(X, Y_barplot, width, label='XXXX')\n\nax.set_ylabel('Number of papers')\nax.set_xlabel('LLM')\nax.set_title('LLM: BERT vs GPT', fontsize=18, color='blue')\nax.set_xticks(X)\nax.set_xticklabels(X_barplot)\n\nfor b in LM_bars:\n   height = b.get_height()\n   ax.annotate('{}'.format(height), xy=(b.get_x() + b.get_width() / 2, height), ha='center', va='bottom')","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:31.785377Z","iopub.execute_input":"2023-05-29T10:58:31.785723Z","iopub.status.idle":"2023-05-29T10:58:32.132888Z","shell.execute_reply.started":"2023-05-29T10:58:31.785696Z","shell.execute_reply":"2023-05-29T10:58:32.131721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section34\"></a>\n## 3.4 Which Machine Learning Approaches?\nSeveral different approaches have been applied to text processing. In this section, we aim at identifying the main ML approaches used in this field during the last two years. For that, we first simply search the titles for all the the pairs of successive words 'xxxx learning'. ","metadata":{}},{"cell_type":"code","source":"#We first the titles for pairs of successive words `xxxx learning'.\ndic_learning = {}\nfor nump in range(nbPapers):\n   title = lst_Txt_2years[nump]['title']\n   title = title.lower()\n   res = re.findall(r\"\\S*\\s+learning\", title)\n   for x in res:\n     y = x.split()\n     if (y[0]=='learning'):\n       print(x)\n     if y[0] not in dic_learning.keys():\n        dic_learning[y[0]] = 1\n     else:        \n        dic_learning[y[0]] += 1\n        \ndic_learning_sorted = {k: v for k, v in sorted(dic_learning.items(), key=lambda item: item[1], reverse=True)}\n","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:32.134318Z","iopub.execute_input":"2023-05-29T10:58:32.134657Z","iopub.status.idle":"2023-05-29T10:58:32.680395Z","shell.execute_reply.started":"2023-05-29T10:58:32.134628Z","shell.execute_reply":"2023-05-29T10:58:32.678777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Top 10 of the words preceding the word learning in the titles of the papers.\nfor wrd in list(dic_learning_sorted.keys())[:20]:\n   print(wrd, dic_learning_sorted[wrd])","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:32.682413Z","iopub.execute_input":"2023-05-29T10:58:32.682928Z","iopub.status.idle":"2023-05-29T10:58:32.694387Z","shell.execute_reply.started":"2023-05-29T10:58:32.682875Z","shell.execute_reply":"2023-05-29T10:58:32.692927Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the list above, we notice several learning approaches and frameworks: **contrastive** learning, **deep** Learning, **representation** learning, **transfer** learning, **reinforcement** learning, **few-shot** and **zero-shot** learning, **self-supervised** learning and **semi-supervised** learning. The figure below recaps the frequencies of occurence of these approaches and techniques in the papers.","metadata":{}},{"cell_type":"code","source":"X_barplot = ['contrastive', 'deep', 'representation', 'transfer', 'reinforcement', 'few-shot', 'self-supervised', 'semi-supervised', 'zero-shot']\nX_barplot.sort()\nY_barplot = [dic_learning[k] for k in X_barplot]\n\nX = np.arange(len(X_barplot))\nwidth = 0.5\n\nfig, ax = plt.subplots(figsize=(15,10))\nlearning_bars=ax.bar(X, Y_barplot, width, label='XXXX')\n\nax.set_ylabel('Number of papers')\nax.set_xlabel('Task')\nax.set_title('ML approaches and techniques, and their frequencies', fontsize=18, color='blue')\nax.set_xticks(X)\nax.set_xticklabels(X_barplot)\n\nfor b in learning_bars:\n   height = b.get_height()\n   ax.annotate('{}'.format(height), xy=(b.get_x() + b.get_width() / 2, height), ha='center', va='bottom')\n","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:32.696606Z","iopub.execute_input":"2023-05-29T10:58:32.697138Z","iopub.status.idle":"2023-05-29T10:58:33.167227Z","shell.execute_reply.started":"2023-05-29T10:58:32.697089Z","shell.execute_reply":"2023-05-29T10:58:33.166157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section35\"></a>\n## 3.5 Which NN Architectures?\nThe main ML approaches encountered above are obviously based on neural networks (NN). In this section we identify the main NN architectures used in the recent NLP works.","metadata":{}},{"cell_type":"code","source":"dic_NN_archi={'ConvNET':'convolutional','Transformer':'Transformer', 'LSTM':'LSTM',\n              'RNN-other':'RNN|Recurrent Neural', 'Attention':'Attention',\n              'Encoder-decoder':'Encode-decoder|Autoencoder|Encoders-Decoders'}\n\ndic_freq_archi = {}\nfor k in dic_NN_archi.keys():\n  archi = dic_NN_archi[k]\n  lst_words = archi.split('|')\n  for nump in range(nbPapers):\n     title = lst_Txt_2years[nump]['title']\n     abstract = lst_Txt_2years[nump]['abstract']\n     if (paperContainsWord(title, abstract, lst_words)==True):\n       if k not in dic_freq_archi.keys():\n         dic_freq_archi[k]=[nump]\n       else:\n         dic_freq_archi[k].append(nump)\n            \n         \ndic_freq_archi_sorted = {k: v for k, v in sorted(dic_freq_archi.items(), key=lambda item: len(item[1]), reverse=True)}\n","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:33.168468Z","iopub.execute_input":"2023-05-29T10:58:33.168793Z","iopub.status.idle":"2023-05-29T10:58:34.447633Z","shell.execute_reply.started":"2023-05-29T10:58:33.168764Z","shell.execute_reply":"2023-05-29T10:58:34.446334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for k in dic_freq_archi_sorted.keys():\n  print(k,':', len(dic_freq_archi_sorted[k]),' papers.')","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:34.449224Z","iopub.execute_input":"2023-05-29T10:58:34.450212Z","iopub.status.idle":"2023-05-29T10:58:34.456281Z","shell.execute_reply.started":"2023-05-29T10:58:34.450176Z","shell.execute_reply":"2023-05-29T10:58:34.455095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_barplot = list(dic_freq_archi_sorted.keys())\nX_barplot.sort()\nY_barplot = [len(dic_freq_archi[k]) for k in X_barplot]\n\nX = np.arange(len(X_barplot))\nwidth = 0.5\n\nfig, ax = plt.subplots(figsize=(15,10))\narchi_bars=ax.bar(X, Y_barplot, width, label='XXXX')\n\nax.set_ylabel('Number of papers')\nax.set_xlabel('Architecture')\nax.set_title('NN Architectures considered in the papers, and their frequencies', fontsize=18, color='blue')\nax.set_xticks(X)\nax.set_xticklabels(X_barplot)\n\nfor b in archi_bars:\n   height = b.get_height()\n   ax.annotate('{}'.format(height), xy=(b.get_x() + b.get_width() / 2, height), ha='center', va='bottom')","metadata":{"execution":{"iopub.status.busy":"2023-05-29T10:58:34.458074Z","iopub.execute_input":"2023-05-29T10:58:34.458418Z","iopub.status.idle":"2023-05-29T10:58:34.892500Z","shell.execute_reply.started":"2023-05-29T10:58:34.458390Z","shell.execute_reply":"2023-05-29T10:58:34.891332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section4\"></a>\n# 4. Intermediate Conclusions\nThe statistics presented above give us an idea about the main trends in NLP for the last two years. These trends can be summarized as follows:\n\n* The **connexionist** approach (vs for example the symbolic approach) continues to be the main approach in this field.\n* **LSTM** and other **recurrent** neural networks continue to be the main architectures used for the different NLP tasks.\n* **Attention** mechanism (introduced in the mid 2010's) continues to be frequently used. \n* **Transformer** (based on attention mechanism, introduced in 2017) is currently the most popolular architecture. \n* Nowadays, **LLM** (large language models) is the most studied subfield of NLP. This is probably due to the impressive results achieved there. \n* **Self-supervised** learning is the newest trend, the most important and the one that is developing the fastest in NLP. It is used in particular through two techniques: **contrastive** learning and **representation** learning.  \n\n\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}