{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# AI Report - Basic EDA\n\n\nThe objective of this competition is to thoroughly examine and analyze the significant advancements made in the field of artificial intelligence (AI) over the past two years. The aim is to compile a comprehensive report detailing these advancements and their impact. To facilitate this investigation, Kaggle has made available two datasets for participants to utilize: one comprising the write-ups of various competitions, and the other containing a collection of papers sourced from Arxiv.\n\nIn this kernel, we conduct a preliminary analysis of these two datasets. We address fundamental inquiries such as the total number of competitions and write-ups available, while also delving briefly into competitions and articles specifically focused on text data. However, our analysis is by no means exhaustive, and it serves as an initial exploration into the datasets to gain a high-level understanding of the provided information.","metadata":{}},{"cell_type":"code","source":"!python -m spacy download en_core_web_md\n\n!unzip -o /usr/share/nltk_data/corpora/wordnet.zip -d /usr/share/nltk_data/corpora/","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:24:46.538918Z","iopub.execute_input":"2023-06-30T06:24:46.540233Z","iopub.status.idle":"2023-06-30T06:25:05.857190Z","shell.execute_reply.started":"2023-06-30T06:24:46.540194Z","shell.execute_reply":"2023-06-30T06:25:05.855813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nfrom PIL import Image\nfrom wordcloud import WordCloud, STOPWORDS, ImageColorGenerator\n\nimport nltk\n\nnltk.download('stopwords')\nfrom nltk.corpus import stopwords\nfrom nltk.tokenize import RegexpTokenizer\nnltk.download('wordnet') \nfrom nltk.stem.wordnet import WordNetLemmatizer\nnltk.download('omw-1.4')\n\nfrom sklearn.feature_extraction.text import CountVectorizer\n\nimport spacy\n\nnlp = spacy.load(\"en_core_web_md\")\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-06-30T06:25:05.859594Z","iopub.execute_input":"2023-06-30T06:25:05.859912Z","iopub.status.idle":"2023-06-30T06:25:07.590732Z","shell.execute_reply.started":"2023-06-30T06:25:05.859880Z","shell.execute_reply":"2023-06-30T06:25:07.589486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Kaggle Writeups dataset\n\nFirst, we analyze the *kaggle_writeups* dataset to look at questions such as the total number of competitions and write-ups available.","metadata":{}},{"cell_type":"code","source":"kw = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv\", parse_dates=[0, 3])","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:07.593301Z","iopub.execute_input":"2023-06-30T06:25:07.593708Z","iopub.status.idle":"2023-06-30T06:25:08.200398Z","shell.execute_reply.started":"2023-06-30T06:25:07.593677Z","shell.execute_reply":"2023-06-30T06:25:08.198797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kw.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:08.203244Z","iopub.execute_input":"2023-06-30T06:25:08.203611Z","iopub.status.idle":"2023-06-30T06:25:08.220707Z","shell.execute_reply.started":"2023-06-30T06:25:08.203580Z","shell.execute_reply":"2023-06-30T06:25:08.219170Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kw.info()","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:08.222073Z","iopub.execute_input":"2023-06-30T06:25:08.222390Z","iopub.status.idle":"2023-06-30T06:25:08.240671Z","shell.execute_reply.started":"2023-06-30T06:25:08.222355Z","shell.execute_reply":"2023-06-30T06:25:08.239227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The dataset contains information about **3127 writeups**.","metadata":{}},{"cell_type":"markdown","source":"### How many competitions does the Kaggle write ups dataset has?","metadata":{}},{"cell_type":"code","source":"num_comp = kw[\"Title of Competition\"].nunique()\nprint(f\"The dataset has {num_comp} competitions.\")","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:08.242016Z","iopub.execute_input":"2023-06-30T06:25:08.242286Z","iopub.status.idle":"2023-06-30T06:25:08.256538Z","shell.execute_reply.started":"2023-06-30T06:25:08.242263Z","shell.execute_reply":"2023-06-30T06:25:08.255409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### What are the earliest and latest competitions?","metadata":{}},{"cell_type":"code","source":"earliest_comp = kw[\"Competition Launch Date\"].min().strftime('%Y-%m-%d')\nlatest_comp = kw[\"Competition Launch Date\"].max().strftime('%Y-%m-%d')\n\nprint(f\"The earliest competition this dataset contains is {earliest_comp} and the latest is {latest_comp}.\")","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:08.257792Z","iopub.execute_input":"2023-06-30T06:25:08.258121Z","iopub.status.idle":"2023-06-30T06:25:08.271032Z","shell.execute_reply.started":"2023-06-30T06:25:08.258094Z","shell.execute_reply":"2023-06-30T06:25:08.269722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Although the dataset spans for about 13 years, our primary goal will be to look at the past two years. We want to understand what are the most popular techniques that participants used for text data competitions.","metadata":{}},{"cell_type":"markdown","source":"### What are the competitions with most write ups?","metadata":{}},{"cell_type":"code","source":"kw[\"Title of Competition\"].value_counts().head(20)","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:08.272618Z","iopub.execute_input":"2023-06-30T06:25:08.273026Z","iopub.status.idle":"2023-06-30T06:25:08.289255Z","shell.execute_reply.started":"2023-06-30T06:25:08.272972Z","shell.execute_reply":"2023-06-30T06:25:08.287292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kw[\"Title of Competition\"].value_counts().tail()","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:08.291678Z","iopub.execute_input":"2023-06-30T06:25:08.293337Z","iopub.status.idle":"2023-06-30T06:25:08.309639Z","shell.execute_reply.started":"2023-06-30T06:25:08.293260Z","shell.execute_reply":"2023-06-30T06:25:08.308486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see the competition with more write ups is Feedback Prize - English Language Learning. On the other hand, some competitions have only one write up.","metadata":{}},{"cell_type":"markdown","source":"### How many competitions do we have from the past two years?\n\nWe consider competitions from May 2020 onwards (about 2 years ago).","metadata":{}},{"cell_type":"code","source":"past_two_years_kw = kw[(kw['Competition Launch Date'].dt.year > 2020) & (kw['Competition Launch Date'].dt.month > 4)]\nnum_comp_past_two_years  = past_two_years_kw['Title of Competition'].nunique()\n\nprint(f\"Number of competitions from the past two years: {num_comp_past_two_years}.\")","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:08.313381Z","iopub.execute_input":"2023-06-30T06:25:08.313706Z","iopub.status.idle":"2023-06-30T06:25:08.328411Z","shell.execute_reply.started":"2023-06-30T06:25:08.313672Z","shell.execute_reply":"2023-06-30T06:25:08.326883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### What is the total number of write-ups available specifically for text competitions within the past two years?\n\nTo anser this question, we check an external source that lists [the NLP competitions hosted on Kaggle](https://medium.com/@ebrahimhaqbhatti516/compilation-of-all-the-text-nlp-competitions-hosted-on-kaggle-17301835f225). From this source and Kaggle website, we identified the following five competitions that use text data:","metadata":{}},{"cell_type":"code","source":"text_data_competitions = [\n    \"U.S. Patent Phrase to Phrase Matching\",\n    \"Feedback Prize — Predicting Effective Arguments\",\n    \"NBME — Score Clinical Patient Notes\",\n    \"Feedback Prize — Evaluating Student Writing\",\n    \"Jigsaw Rate Severity of Toxic Comments\",\n]","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:08.330202Z","iopub.execute_input":"2023-06-30T06:25:08.331075Z","iopub.status.idle":"2023-06-30T06:25:08.338474Z","shell.execute_reply.started":"2023-06-30T06:25:08.330980Z","shell.execute_reply":"2023-06-30T06:25:08.337182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"text_past_two_years_kw = past_two_years_kw[past_two_years_kw[\"Title of Competition\"].isin(text_data_competitions)].copy()\ntext_past_two_years_kw  = text_past_two_years_kw.reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:08.340092Z","iopub.execute_input":"2023-06-30T06:25:08.340420Z","iopub.status.idle":"2023-06-30T06:25:08.355120Z","shell.execute_reply.started":"2023-06-30T06:25:08.340395Z","shell.execute_reply":"2023-06-30T06:25:08.353779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_text_compt_write_ups = text_past_two_years_kw.shape[0]\nprint(f\"Number of Writeups for text competitions from the past two years: {num_text_compt_write_ups}.\")","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:08.357074Z","iopub.execute_input":"2023-06-30T06:25:08.357635Z","iopub.status.idle":"2023-06-30T06:25:08.370109Z","shell.execute_reply.started":"2023-06-30T06:25:08.357590Z","shell.execute_reply":"2023-06-30T06:25:08.368963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the dataset contains only a few write-ups from competitions that use text data. ","metadata":{}},{"cell_type":"markdown","source":"### Which keywords can we extract from text competitions write-ups?","metadata":{}},{"cell_type":"markdown","source":"Now, we extract keywords from the write up column using the package [pke - python keyphrase extraction](https://github.com/boudinfl/pke).","metadata":{}},{"cell_type":"code","source":"!pip install git+https://github.com/boudinfl/pke.git\n!python -m spacy download en_core_web_sm","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:08.371439Z","iopub.execute_input":"2023-06-30T06:25:08.371742Z","iopub.status.idle":"2023-06-30T06:25:36.533026Z","shell.execute_reply.started":"2023-06-30T06:25:08.371715Z","shell.execute_reply":"2023-06-30T06:25:36.531754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pke\n\ndef extract_keywords(text):\n    extractor = pke.unsupervised.TfIdf()\n    extractor.load_document(input=text)\n    extractor.candidate_selection()\n    extractor.candidate_weighting()                     \n    keyphrases = extractor.get_n_best(n=10) \n    \n    # Extract top 5 keywords\n    keywords = [keyword[0] for keyword in keyphrases[:5]]\n    \n    return keywords","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:36.535252Z","iopub.execute_input":"2023-06-30T06:25:36.535733Z","iopub.status.idle":"2023-06-30T06:25:36.544483Z","shell.execute_reply.started":"2023-06-30T06:25:36.535687Z","shell.execute_reply":"2023-06-30T06:25:36.543290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before extracting keywords, we need to clean the data (write-ups text) and normalize it. The text contains html tags, special characters and changes of line. Therefore, we remove all of this extra text and lower case the end result.","metadata":{}},{"cell_type":"code","source":"import re\n\ndef clean_text(df, col_to_clean):\n\n    # Remove html tags\n    df['cleaned_text'] = df[col_to_clean].apply(lambda x: re.sub('<[^<]+?>', '', x))\n\n    # Remove change of line\n    df['cleaned_text'] = df['cleaned_text'].str.replace(\"\\n+\", \"\", regex=True)\n\n    # Remove special characters \n    df['cleaned_text'] = df['cleaned_text'].str.replace(\"-\", \" \", regex=False)\n    df['cleaned_text'] = df['cleaned_text'].str.replace(\"[^a-zA-Z0-9 ]\", \"\", regex=True)\n\n    # Lowercase text\n    df['cleaned_text'] = df['cleaned_text'].str.lower()\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:36.546863Z","iopub.execute_input":"2023-06-30T06:25:36.547196Z","iopub.status.idle":"2023-06-30T06:25:36.557111Z","shell.execute_reply.started":"2023-06-30T06:25:36.547168Z","shell.execute_reply":"2023-06-30T06:25:36.555892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"text_past_two_years_kw = clean_text(text_past_two_years_kw, 'Writeup')\n\n# Extract keywords\ntext_past_two_years_kw['keywords_lst'] = text_past_two_years_kw['cleaned_text'].apply(extract_keywords)","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:25:36.558638Z","iopub.execute_input":"2023-06-30T06:25:36.558943Z","iopub.status.idle":"2023-06-30T06:27:35.303277Z","shell.execute_reply.started":"2023-06-30T06:25:36.558919Z","shell.execute_reply":"2023-06-30T06:27:35.302225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"text_past_two_years_kw.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:27:35.304595Z","iopub.execute_input":"2023-06-30T06:27:35.304883Z","iopub.status.idle":"2023-06-30T06:27:35.320191Z","shell.execute_reply.started":"2023-06-30T06:27:35.304859Z","shell.execute_reply":"2023-06-30T06:27:35.319466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Split column keywords_lst\nkeywords_cols = pd.DataFrame(text_past_two_years_kw['keywords_lst'].to_list(), columns = ['keyword1', 'keyword2', 'keyword3', 'keyword4', 'keyword5'])\n\ntext_past_two_years_kw = pd.concat([text_past_two_years_kw, keywords_cols], axis=1) ","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:27:35.321323Z","iopub.execute_input":"2023-06-30T06:27:35.322291Z","iopub.status.idle":"2023-06-30T06:27:35.332216Z","shell.execute_reply.started":"2023-06-30T06:27:35.322260Z","shell.execute_reply":"2023-06-30T06:27:35.331196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We get the top 10 keywords for all of the events\nwriteups_top_keywords = text_past_two_years_kw.keyword1.value_counts().rename_axis('keyword').reset_index(name='num_entries')","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:27:35.333260Z","iopub.execute_input":"2023-06-30T06:27:35.333532Z","iopub.status.idle":"2023-06-30T06:27:35.347959Z","shell.execute_reply.started":"2023-06-30T06:27:35.333507Z","shell.execute_reply":"2023-06-30T06:27:35.346156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"writeups_top_keywords.head(n=30)","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:27:35.349769Z","iopub.execute_input":"2023-06-30T06:27:35.350214Z","iopub.status.idle":"2023-06-30T06:27:35.368957Z","shell.execute_reply.started":"2023-06-30T06:27:35.350177Z","shell.execute_reply":"2023-06-30T06:27:35.367533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By analyzing the extracted keywords, we notice that architectures such as Roberta and BERT are mentioned, indicating their relevance in the write-ups. Furthermore, we observe other keywords such as toxicity (a competition aimed at clasifying text) and techniques like pseudo labeling, ensambling, and using pretrained models. ","metadata":{}},{"cell_type":"code","source":"kw_text_data_architectures = [\n    \"fasttext\", \"roberta\", \"bert\",\n    \"gpt\", \"rnn\", \"cnn\", \"gru\",\n    \"t5\", \"electra\", \"xlnet\",\n    \"encoder\", \"decoder\", \"lstm\",\n    \"transformer\", \"deberta\", \"codebert\"\n]","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:27:35.370731Z","iopub.execute_input":"2023-06-30T06:27:35.371585Z","iopub.status.idle":"2023-06-30T06:27:35.380838Z","shell.execute_reply.started":"2023-06-30T06:27:35.371549Z","shell.execute_reply":"2023-06-30T06:27:35.379795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def count_occurrences_in_dataframe(df, column_name, strings_list):\n    # Convert the list of strings to a set for faster membership checking\n    strings_set = set(strings_list)\n    \n    # Filter the dataframe to include only rows where the text column contains any of the strings\n    filtered_df = df[df[column_name].str.contains('|'.join(strings_set))]\n    \n    # Use value_counts() to count the occurrences of each string\n    counts = filtered_df[column_name].value_counts()\n    \n    # Create a dictionary to store the results\n    results_dict = {'String': [], 'Occurrences': [], 'Actual Occurrences': []}\n    \n    # Iterate over the strings list\n    for string in strings_list:\n        # Add the string and its corresponding count to the dictionary\n        results_dict['String'].append(string)\n        results_dict['Occurrences'].append(counts.get(string, 0))\n        \n        # Count the actual occurrences in the filtered dataframe\n        actual_occurrences = filtered_df[column_name].str.count(string).sum()\n        results_dict['Actual Occurrences'].append(actual_occurrences)\n    \n    # Convert the dictionary to a dataframe\n    counts_df = pd.DataFrame(results_dict)\n    \n    return counts_df","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:27:35.382494Z","iopub.execute_input":"2023-06-30T06:27:35.383096Z","iopub.status.idle":"2023-06-30T06:27:35.394428Z","shell.execute_reply.started":"2023-06-30T06:27:35.383056Z","shell.execute_reply":"2023-06-30T06:27:35.392996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result = count_occurrences_in_dataframe(text_past_two_years_kw, 'cleaned_text', kw_text_data_architectures)","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:27:35.396708Z","iopub.execute_input":"2023-06-30T06:27:35.397173Z","iopub.status.idle":"2023-06-30T06:27:35.421430Z","shell.execute_reply.started":"2023-06-30T06:27:35.397134Z","shell.execute_reply":"2023-06-30T06:27:35.420468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sorted_result = result.sort_values('Actual Occurrences', ascending=True)\n\nfig, ax = plt.subplots()\nax.barh(sorted_result[\"String\"], sorted_result['Actual Occurrences'])\n\n# Set labels and title\nax.set_xlabel('Occurrences')\nax.set_ylabel('Architecture')\nax.set_title('Architectures used in Kaggle text data competitions')\n\n# Rotate the x-axis labels if needed\nplt.xticks(rotation=65);","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:27:35.422942Z","iopub.execute_input":"2023-06-30T06:27:35.423717Z","iopub.status.idle":"2023-06-30T06:27:35.716537Z","shell.execute_reply.started":"2023-06-30T06:27:35.423648Z","shell.execute_reply":"2023-06-30T06:27:35.714882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The most common architectures frequently used in text competitions include BERT and Roberta. Moreover, we notice that other common architectures are transformer-based alongside various other types of architectures.","metadata":{}},{"cell_type":"markdown","source":"## Arxiv metadata dataset\n\nMoving on to the Arxiv dataset, it contains metadata for articles published on the Arxiv website. This dataset provides interesting information regarding the published articles and will be our source for analyzing the problems related to text data that the scientific comunity has been working on.","metadata":{}},{"cell_type":"code","source":"df_arxiv = pd.read_json(\"/kaggle/input/2023-kaggle-ai-report/arxiv_metadata_20230510.json\", \n                     lines=True,\n                     convert_dates=True,\n                     chunksize=100000,\n                    )","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:27:35.718082Z","iopub.execute_input":"2023-06-30T06:27:35.718441Z","iopub.status.idle":"2023-06-30T06:27:35.725440Z","shell.execute_reply.started":"2023-06-30T06:27:35.718409Z","shell.execute_reply":"2023-06-30T06:27:35.724100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First, we filter out the categories that could contain NLP papers.","metadata":{}},{"cell_type":"code","source":"nlp_related_categories_arxiv = [\n    \"cs.IR\", \"cs.CL\", \"cs.LG\", \"cs.NE\",\n    \"cs.AI\", \"cs.DL\", \"cs.HC\", \"cs.SI\",\n    \"cs.SD\", \"cs.WEB\", \"stat.ML\", \"math.IT\"\n]","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:27:35.727341Z","iopub.execute_input":"2023-06-30T06:27:35.727848Z","iopub.status.idle":"2023-06-30T06:27:35.736645Z","shell.execute_reply.started":"2023-06-30T06:27:35.727799Z","shell.execute_reply":"2023-06-30T06:27:35.735496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nlp_related_categories = pd.DataFrame()\n\nfor chunk in df_arxiv:\n    chunk_nlp_papers = chunk[chunk.categories.isin(nlp_related_categories_arxiv)]\n    nlp_related_categories = pd.concat([nlp_related_categories, chunk_nlp_papers], ignore_index=True)","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:27:35.742448Z","iopub.execute_input":"2023-06-30T06:27:35.742791Z","iopub.status.idle":"2023-06-30T06:29:42.548469Z","shell.execute_reply.started":"2023-06-30T06:27:35.742757Z","shell.execute_reply":"2023-06-30T06:29:42.547444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nlp_related_categories.info()","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:29:42.549925Z","iopub.execute_input":"2023-06-30T06:29:42.550329Z","iopub.status.idle":"2023-06-30T06:29:42.788701Z","shell.execute_reply.started":"2023-06-30T06:29:42.550287Z","shell.execute_reply":"2023-06-30T06:29:42.787182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nlp_related_categories['update_date']= pd.to_datetime(nlp_related_categories['update_date'])\n\nnlp_related_categories = nlp_related_categories[(nlp_related_categories['update_date'].dt.year > 2020) & (nlp_related_categories['update_date'].dt.month > 4)].copy()\n\nnlp_related_categories = nlp_related_categories.reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:29:42.790572Z","iopub.execute_input":"2023-06-30T06:29:42.790970Z","iopub.status.idle":"2023-06-30T06:29:43.080516Z","shell.execute_reply.started":"2023-06-30T06:29:42.790933Z","shell.execute_reply":"2023-06-30T06:29:43.079268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After filtering the Arxiv dataset to obtain the papers published during the past two years, we move on to analyzing the articles that might be related to text data. However, we employ different techniques than those we used for examining the write-ups dataset since we don't have data on the papers that are related to text data.","metadata":{}},{"cell_type":"code","source":"text_data_keywords = [\n     \"text mining\",\n     \"text analytics\",\n     \"text preprocessing\",\n     \"text classification\",\n     \"text clustering\",\n     \"sentiment analysis\",\n     \"named entity recognition\",\n     \"topic modeling\",\n     \"information retrieval\",\n     \"text summarization\",\n     \"text generation\",\n     \"text similarity\",\n     \"word embeddings\",\n     \"document classification\",\n     \"text feature extraction\",\n     \"text segmentation\",\n     \"text normalization\",\n     \"text corpora\",\n     \"textual data analysis\",\n     \"question answering\",\n     \"sentiment analysis\",\n     \"language modeling\"\n]\n","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:29:43.082191Z","iopub.execute_input":"2023-06-30T06:29:43.082830Z","iopub.status.idle":"2023-06-30T06:29:43.087489Z","shell.execute_reply.started":"2023-06-30T06:29:43.082801Z","shell.execute_reply":"2023-06-30T06:29:43.086531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nlp_related_categories = clean_text(nlp_related_categories, 'abstract')","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:29:43.089289Z","iopub.execute_input":"2023-06-30T06:29:43.089686Z","iopub.status.idle":"2023-06-30T06:29:43.652939Z","shell.execute_reply.started":"2023-06-30T06:29:43.089653Z","shell.execute_reply":"2023-06-30T06:29:43.652221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result = count_occurrences_in_dataframe(nlp_related_categories, 'cleaned_text', text_data_keywords)","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:29:43.654185Z","iopub.execute_input":"2023-06-30T06:29:43.655007Z","iopub.status.idle":"2023-06-30T06:29:44.361093Z","shell.execute_reply.started":"2023-06-30T06:29:43.654966Z","shell.execute_reply":"2023-06-30T06:29:44.360334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sorted_result = result.sort_values('Actual Occurrences', ascending=True)\n\nfig, ax = plt.subplots()\nax.barh(sorted_result[\"String\"], sorted_result['Actual Occurrences'])\n\n# Set labels and title\nax.set_xlabel('Occurrences')\nax.set_ylabel('Keywords')\nax.set_title('Text data keywords mentioned in Arxiv papers')\n\n# Rotate the x-axis labels if needed\nplt.xticks(rotation=90);","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:29:44.362236Z","iopub.execute_input":"2023-06-30T06:29:44.363081Z","iopub.status.idle":"2023-06-30T06:29:44.714797Z","shell.execute_reply.started":"2023-06-30T06:29:44.363053Z","shell.execute_reply":"2023-06-30T06:29:44.713833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Over the course of the past two years, text generation has emerged as a highly prominent topic within the research community. Additionally, other tasks such as sentiment analysis, text classification, information retrieval and text summarization continue to be considered hot topics of interest.","metadata":{}},{"cell_type":"markdown","source":"## What are the prevalent model architectures frequently mentioned in Arxiv articles?\n\nFor this question, we consider the [top leading architectures as for 2023](https://www.analyticsinsight.net/top-10-leading-language-models-for-nlp-in-2023/) plus the ones we considered for the Kaggle write-ups.","metadata":{}},{"cell_type":"code","source":"top_text_data_architectures = [\n    \"bert\", \"gpt\", \"xlnet\",\n    \"roberta\", \"albert\", \"t5\",\n    \"electra\", \"deberta\", \"palm\",\n    \"structbert\"\n]\n\narxiv_text_data_architectures = list(set(top_text_data_architectures).union(set( kw_text_data_architectures)))","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:29:44.716233Z","iopub.execute_input":"2023-06-30T06:29:44.716544Z","iopub.status.idle":"2023-06-30T06:29:44.721692Z","shell.execute_reply.started":"2023-06-30T06:29:44.716518Z","shell.execute_reply":"2023-06-30T06:29:44.720554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result = count_occurrences_in_dataframe(nlp_related_categories, 'cleaned_text', arxiv_text_data_architectures)\nsorted_result = result.sort_values('Actual Occurrences', ascending=True)","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:29:44.722878Z","iopub.execute_input":"2023-06-30T06:29:44.723849Z","iopub.status.idle":"2023-06-30T06:29:45.467421Z","shell.execute_reply.started":"2023-06-30T06:29:44.723817Z","shell.execute_reply":"2023-06-30T06:29:45.465881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots()\nax.barh(sorted_result[\"String\"], sorted_result['Actual Occurrences'])\n\n# Set labels and title\nax.set_xlabel('Occurrences')\nax.set_ylabel('Architectures')\nax.set_title('Architectures mentioned in Arxiv papers')\n\n# Rotate the x-axis labels if needed\nplt.xticks(rotation=90);","metadata":{"execution":{"iopub.status.busy":"2023-06-30T06:29:45.469033Z","iopub.execute_input":"2023-06-30T06:29:45.469616Z","iopub.status.idle":"2023-06-30T06:29:45.761857Z","shell.execute_reply.started":"2023-06-30T06:29:45.469579Z","shell.execute_reply":"2023-06-30T06:29:45.760886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"BERT and transformers are the most frequently mentioned architectures in Arxiv article as they have a large impact in the research agenda. Additionally, architectures like GPT and encoder/decoder are also hot topics of discussion. Moreover, traditional approaches such as CNN and LSTM maintain their relevance and continue to be acively used for research.","metadata":{}},{"cell_type":"markdown","source":"## Conclusions\n\nBased on the analysis of both datasets, the following conclusions can be drawn:\n\n* Text data competitions on Kaggle: Although numerous Kaggle competitions have taken place in the last two years, only a limited number of them specifically focus on text data analysis.\n* Most common architectures in write-ups for text data competitions: Among the write-ups of text data competitions, the architectures frequently mentioned are BERT and Roberta. These architectures have gained significant attention for winning solutions.\n* Architectures prevalent in the Arxiv dataset: In the Arxiv dataset, BERT and transformers stand out as the prevailing architectures. Additionally, GPT is a notable architecture mentioned in both datasets, highlighting its relevance.\n* Continued relevance of traditional approaches: Traditional approaches like CNN and LSTM continue to hold relevance in both text data Kaggle competitions and research endeavors. Despite the emergence of newer architectures, these well-established methods continue to be utilized and recognized for their effectiveness.\n\nThese conclusions shed light on the current landscape of text data analysis, highlighting the popular architectures in use and the enduring significance of traditional approaches.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}