2023 Kaggle AI Report - Maturity at a glance (LSTM, GLOVE)
\n\n
\n \n \n \n
","metadata":{}},{"cell_type":"markdown","source":"This report represents a conception of the application of SLMHM/SPEM development models to analyze how mature the texts describing rapid advancements in AI from the past two years are.\n\nAcknowledgments: The author is especially grateful to Chung Wai Tang for his detailed valuable review ([Review](https://www.kaggle.com/code/dmitriygakh/kaggle-ai-report-maturity-at-a-glance/comments#2342181))\n","metadata":{}},{"cell_type":"markdown","source":"Features of this notebook:\n- Estimation of papers' maturity by ranking from chaos (level 0) up to totality (level 16);\n- Use supervised ML classification of texts (NLP), LSTM, GLOVE;\n- Findings texts with the highest maturity level;\n- Development of society, economy, and environmental issues is considered.\n\n\nResources created:\n- This notebook;\n- Training database ([data](https://www.kaggle.com/datasets/dmitriygakh/see-slmhm-spem));\n- Training database and AI tests ([tests](https://www.kaggle.com/code/dmitriygakh/see-slmhm-spem-maturity-tests-lstm-glove));\n- Web scrapping notebook ([notebook](https://www.kaggle.com/code/dmitriygakh/web-scrapping-preprints-org-basic-eda)).\n\n\n\n# Table of Contents\n\n\n\n- [1 | Introduction](#1)\n > - [1.1 Motivation](#1.1)\n > - [1.2 Topic selected](#1.2)\n > - [1.3 Introduction into SLMHM/SPEM](#1.3)\n > - [1.4 The Highest Aim](#1.4)\n > - [1.5 Methodology](#1.5)\n > - [1.6 Evaluation Algorithm](#1.6)\n > - [1.7 AI Ethics](#1.7)\n\n\n- [2 | AI Model](#2)\n > - [2.1 Training Dataset](#2.1)\n > - [2.2 Tokenizing and Word Emdedding](#2.2)\n > - [2.3 AI Model](#2.3)\n > - [2.4 Helpful Functions](#2.4)\n > - [2.5 AI Model Tests](#2.5)\n\n- [3 | Analyses of Kaggle Writeups](#3)\n\n- [4 | Analyses of Arxiv Metadata](#4)\n\n- [5 | Finding the best Kaggle Writeups](#5)\n\n- [6 | Finding the best ArXiv Papers](#6)\n\n- [7 | Kaggle Writeups EDA](#7)\n\n- [8 | ArXiv Papers EDA](#8)\n\n- [9 | Conclusion](#9)\n\n- [10 | Refernces](#10)\n\n- [11 | Future Actions](#11)\n\n- [12 | Self-Assesment](#12)\n > - [12.1 Drawbacks](#12.1)\n\n\n\n# 1 | Introduction\n\n\n\n## 1.1 Motivation\nClassical methods of evaluation of textual information require human analytical skills and deep knowledge of the subject. AI Language Models can be used to automate this task. Progress in AI texts can be evaluated quantitatively (by the number of papers on a specific subject, by the number of subjects covered, etc.) or qualitatively (impact of a paper on the economy, how accurate the narration is, how the information is, etc). The method considered in this essay demonstrates a method of classification of texts by 17 levels of maturity. For example, if the text read about some kind of improvement, it relates to level 2.2. If the text read about somewhat mutually beneficial for all elements of a system, it relates to level 3.1. If the text read about expansion or globalization, it relates to level 4.3. One can speak that as higher a level as mature the text is. But the most mature text should relate to all the levels - in other words, describe all development levels.\n\nThere were several attempts to evaluate the progress of AI advantages by studying scholarly papers and other texts. Some of the methods give a vision of advancements in AI from past years. But the following problems were discovered:\n- New technologies appear very rapidly. The best way to discover which technologies appear is to train an AI Model. But AI Model requires a continuously updated proper training set;\n- Classification models based on Natural Language Processing (NLP) seem the most helpful. Unsupervised learning seems nonsensical. Supervised learning requires a training dataset containing proper classification;\n- Training datasets based on classifiers can obsolete quite quickly;\n- AI is penetrating all spheres of human life. Multidisciplinary advantages will appear often and often. Such advantages require complex analyses of benefits and impacts.\n\nThe development of considered AI model based on universal principles in a way it will not be obsolete in the next several years is a very attractive idea. SLMHM and SPEM are new development models presented in solid conferences and published in scholarly papers can help to have quantitative/qualitative evaluation of progress in the development of AI-related technologies. Evaluating the relation of the papers to the United Nations Sustainable Development Goals allows an understanding of the impact of papers on worldwide phenomena (society, economy, natural environment).\n\n\nImportant note: Chung Wai Tang mentioned that \"With the rise of Large Language Models (LLM) and open-source tools like ChatGTP that can automate the creation of text report summaries beg the questions on the 'quality' or 'maturity' of machine-generated summaries. Presented overall evaluation framework is extremely valuable and timely as future research may look at how these ranks correlate with human's perception of 'quality'\" ([Source] (https://www.kaggle.com/code/dmitriygakh/kaggle-ai-report-maturity-at-a-glance/comments#2342181)).\n\n\nImportant note: Paul Mooney wrote: \"If there is one thing that the recent rise of generative AI has made clear, it is that subjective and qualitative evaluations are now a critical part of data science\" ([Discussion](https://www.kaggle.com/competitions/2023-kaggle-ai-report/discussion/410802)). The currently discussed methodology can be adopted to evaluate phenomena that could be evaluated only subjectively before.\n\n\n\n[↑↑↑ Table of Contents ↑↑↑](#toc)\n\n\n\n## 1.2 Topic selected\n\nThe topic of this essay is \"Maturity\" (Competition categoty \"Other\"). The maturity is assessed according to SLMHM/SPEM models [[1]](#ref1),[[2]](#ref2),[[3]](#ref3). The assessment considers the interrelations between society, economy, and environment (SEE) [[3]](#ref3). Development of these components is aligned with the [17 UN Sustainable Development Goals](https://sdgs.un.org/goals). At the same time maturity should reflect benefits for business, commerce, government, as well as for other organizations. This essay aims, to some extent, to fill the lack of research in the field of maturity, taking into account modern challenges. Although this paper presents an AI model based on NLP, the research topic is not related to word processing - it is just an evaluation method.\n\n\n\n\n## 1.3 Introduction into SLMHM/SPEM\n\nSimple Learning Motivation Hierarchy Model (SLMHM) and Social Patterns Evolution Model (SPEM) are two integrated development models capable to model socio-technical systems. Several other models were built on the base of SLMHM/SPEM. The models consist of 16 levels and 5 main patterns of development. Phenomena from different domains can be evaluated and levels/patterns can be compared showing differences in the development. Research of SLMHM in education shows the need for huge calculations [[4]](#ref4). SPEM is a more simple model and is better suited to be a starting point to learn these models. SPEM describes the main patterns of societal development. For current research SPEM patterns are ([[1]](#ref1)):\n- Initial Formation Pattern (IFP). This is a pattern of chaos. All interactions here are ad hoc. This pattern includes data, code, and phenomena that have a random origin;\n- Primary Society Pattern (PSP). This is a pattern of structure. All interactions, data, and code here are structured;\n- Adaptable Society Pattern (ASP). This is a pattern of benefits. All interactions, data, and code here are being evaluated to provide benefits. Such conceptions as logic, facts, science, research, humanism, and sustainability belong to ASP;\n- Information Society Pattern (ISP). This is a pattern of connectivity. Connectivity is represented mainly by transport and communication networks. Increased connectivity leads to increasing information flows (what's why this pattern is named \"Information Society Pattern\"). Increased connectedness between people leads to greater opportunities to express feelings and emotions. Connectivity, synergy, feelings, emotions, and neo-humanism are examples of ISP phenomena;\n- Creative Society Pattern (CSP). This is a pattern of creativity. The key value of this pattern is the generation/acquisition of information and extraction of knowledge in real-time.\n\n\n\n[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"markdown","source":"#
Structure of SLMHM/SPEM levels/patterns
\n\nSource: [[1]](#ref1)\n\n
\n \n \n \n
","metadata":{}},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"markdown","source":"\n## 1.4 The Highest Aim\n\nSLMHM requires the Highest Aim that corresponds to Level 16, Totality, and to the mission as the highest aim for SPEM (SPEM's highest aim is God, but for the current study, the mission is a better alternative). The Highest Aim for this study could be as the following: \"The complete fusion of all SEE interactions into something infinite and overall\" (see [[3]](#ref3) for more details). But, speaking strongly, the current training data set does not show a satisfactory result. So, the Highest Aim for this study is: \"A level of a worldwide long-term impact\".\n\n\n## 1.5 Methodology\n\nThe methodology of this research is based on the following steps:\n- 1. Create a synthetic training dataset containing key phrases relating to specific SLMHM levels;\n- 2. Train an AI model on the dataset;\n- 2. Make predictions for the writeups and ArXiv metadata (abstracts) for two datasets - \"earlier 2021\" and \"2021 and later\";\n- 3. Visualize the predictions.\n\n\n## 1.6 Evaluation Algorithm\n\nThe key to the analysis is the evaluation of text fragments according to SLMHM/SPEM levels/patterns. This approach allows one to evaluate whole articles, books, and so on. The resulting numerical estimates can help the further implementation of EDA. For mature analyses, the patterns should be determined for each SEE: society, economy, environment, society-economy relation, society-environment relation, economy-environment relation, and society-economy-environment relation. At a glance, it should include (16 + 1) * (7 + 1) = 136 textual patterns (16 levels of SLMHM, 1 level corresponding to ITF pattern and 8 SEE conceptions, 7 SEE issue types and 1 not relating to SEE). By the author's expert evaluation, the training dataset should contain about 5000 textual patterns to produce a more or less practical result. The last training dataset version contains about 500 rows and demonstrates evaluation according to SLMHM generally. But it is not enough to evaluate all SEE components. At the same time, there are no papers found that are quite mature according to SEE components (only papers' abstracts were used for the evaluation).\n\n\n[↑↑↑ Table of Contents ↑↑↑](#toc)\n\n\n## 1.7 AI Ethics\nAuthor's research in SLMHM/SPEM models [[1]](#ref1),[[2]](#ref2),[[3]](#ref3) shows that ASP relates to the Industrial Age, and ISP to the Information Age. ASP aimed to benefit people. Ethics in ASP is a ethics of the Industrial Age based on IQ and humanism. ISP aimed at the connectivity and benefit for expression of feelings and emotions (development of social networks and increasing demand in EQ is evidence of this claim). So, ethics relating to the Information Age is based on neo-humanism. Neo-humanism in the author's interpretation is based on connectivity where people, other life forms, and inanimate matter are parts of one connected system. Consideration of all these in one system can be modeled as a society-economy-environment system and is relevant to the [17 UN Sustainable Development Goals](https://sdgs.un.org/goals).\n\n\n\n\n# 2 | AI Model\n\n\n\n\nThe model is based on NLP and text classification.\n\n\n\n\n## 2.1 Training Dataset","metadata":{}},{"cell_type":"code","source":"# Import necessary libraries\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport os\nimport random\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.utils import pad_sequences\nimport tensorflow as tf\nfrom tensorflow.keras import Model\nfrom tensorflow.keras.layers import Input, Embedding, Bidirectional, LSTM, Conv1D, GlobalMaxPool1D, BatchNormalization, Dense, Dropout, SpatialDropout1D\nfrom tensorflow.keras.optimizers import RMSprop\nfrom tensorflow.keras.optimizers import Adam\nfrom keras.callbacks import EarlyStopping\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau\n\nfrom nltk.corpus import stopwords\n\n# We will use English stopwords\nen_stopwords = set(stopwords.words('english'))\n\n# Set random seed to have stable results for runnings\nrseed = 55\nrandom.seed(rseed)\nnp.random.seed(rseed)\ntf.random.set_seed(rseed)\nos.environ['PYTHONHASHSEED'] = str(rseed)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:35:29.489602Z","iopub.execute_input":"2023-07-16T04:35:29.490117Z","iopub.status.idle":"2023-07-16T04:35:40.471666Z","shell.execute_reply.started":"2023-07-16T04:35:29.490068Z","shell.execute_reply":"2023-07-16T04:35:40.470512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Labels for SLMHM\nllabels = ['0','1','2','3','4','5','6','7','8','9','10','11','12','13','14','15','16']\nllabels_str = [\n '0.0. Chaos',\n '1.1. Desire','1.2. Intention','1.3. Feasibility','1.4. Action',\n '2.1. Evaluation','2.2. Improvement','2.3. Alternatives','2.4. Innovations',\n '3.1. Reward','3.2. Optimization','3.3. Synergy','3.4. Extraordinaries',\n '4.1. Information','4.2. Streamlining','4.3. Expansion','4.4. Totality'\n]\n\n# Labels for SPEM\nllabels_spem = ['0','1','2','3','4']\nllabels_spem_str = ['IFP','PSP','ASP','ISP','CSP']\n# IFP - Initial Formation Pattern (chaos)\n# PSP - Primary Society Pattern (structure, discipline)\n# ASP - Adaptive Society Pattern (benefits, logic)\n# ISP - Information Society Pattern (connectivity, feelings, emotions)\n# CSP - Creative Society Pattern (creativity)\n\n# Loading the keyphrase classification data\nkpdf = pd.read_csv('../input/see-slmhm-spem/see_key_phrases.csv', delimiter = '\\t')\n\n# kpdf.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:35:40.473581Z","iopub.execute_input":"2023-07-16T04:35:40.474230Z","iopub.status.idle":"2023-07-16T04:35:40.516675Z","shell.execute_reply.started":"2023-07-16T04:35:40.474200Z","shell.execute_reply":"2023-07-16T04:35:40.515718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Feature engineering.\n# Encode training dataset into 2-columns Class/Keywords structure.\n\n# Code Class values are:\n# ndndnd[X] - not determined, SLMHM level X\n# ndndsc[X] - society, SLMHM level X\n# ndndec[X] - economy, SLMHM level X\n# ndndev[X] - environment, SLMHM level X\n# ndscec[X] - society-economy, SLMHM level X\n# ndscev[X] - society-environment, SLMHM level X\n# ndecev[X] - economy-environment, SLMHM level X\n# scecev[X] - society-economy-environment, SLMHM level X\n\nkpdf.loc[kpdf['SLMHMlvsND'] >= 0, 'Class'] = 'ndndnd' + kpdf['SLMHMlvsND'].apply(lambda x: f\"{x:.0f}\")\nkpdf.loc[kpdf['SLMHMlvsSc'] >= 0, 'Class'] = 'ndndsc' + kpdf['SLMHMlvsSc'].apply(lambda x: f\"{x:.0f}\")\nkpdf.loc[kpdf['SLMHMlvsEc'] >= 0, 'Class'] = 'ndndec' + kpdf['SLMHMlvsEc'].apply(lambda x: f\"{x:.0f}\")\nkpdf.loc[kpdf['SLMHMlvsEv'] >= 0, 'Class'] = 'ndndev' + kpdf['SLMHMlvsEv'].apply(lambda x: f\"{x:.0f}\")\nkpdf.loc[kpdf['SLMHMlvsScEc'] >= 0, 'Class'] = 'ndscec' + kpdf['SLMHMlvsScEc'].apply(lambda x: f\"{x:.0f}\")\nkpdf.loc[kpdf['SLMHMlvsScEv'] >= 0, 'Class'] = 'ndscev' + kpdf['SLMHMlvsScEv'].apply(lambda x: f\"{x:.0f}\")\nkpdf.loc[kpdf['SLMHMlvsEcEv'] >= 0, 'Class'] = 'ndecev' + kpdf['SLMHMlvsEcEv'].apply(lambda x: f\"{x:.0f}\")\nkpdf.loc[kpdf['SLMHMlvsScEcEv'] >= 0, 'Class'] = 'scecev' + kpdf['SLMHMlvsScEcEv'].apply(lambda x: f\"{x:.0f}\")\n\nkpdf = kpdf[['Class','Keywords']].copy()\n\n# Lets see the training dataset\nkpdf","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:35:40.518156Z","iopub.execute_input":"2023-07-16T04:35:40.519035Z","iopub.status.idle":"2023-07-16T04:35:40.587411Z","shell.execute_reply.started":"2023-07-16T04:35:40.519005Z","shell.execute_reply":"2023-07-16T04:35:40.586323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Training dataset contains 136 unique categories.\n# It can be later dicomposed into 17 levels * 8 categories\n# 17-level model is easier to understand\npd.DataFrame(kpdf.Class.unique())","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:35:40.589866Z","iopub.execute_input":"2023-07-16T04:35:40.590374Z","iopub.status.idle":"2023-07-16T04:35:40.606285Z","shell.execute_reply.started":"2023-07-16T04:35:40.590329Z","shell.execute_reply":"2023-07-16T04:35:40.605016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Distribution of values\n# !!! ToDo: Training set should be complimented to be balanced !!!\n# Such disbalanced Training Dataset cannot be used to assess papers according to SEE\n# But as one can see below Training Dataset can be used to asses general maturity level\n# There is firm confidence that there are no SEE-matured papers in the dataset\n# So, SEE could be considered as a future extension\n\nvc = kpdf.Class.value_counts()\n\nplt.figure(figsize = (12,4) )\nplt.bar(vc.index, vc.values)\nplt.title(\"Training Data Distribution\")\nplt.xticks(rotation = 90) # Rotates X-Axis Ticks by 45-degrees\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:35:40.607814Z","iopub.execute_input":"2023-07-16T04:35:40.608151Z","iopub.status.idle":"2023-07-16T04:35:41.107510Z","shell.execute_reply.started":"2023-07-16T04:35:40.608124Z","shell.execute_reply":"2023-07-16T04:35:41.106493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Ideally it should include all 136 categories taking into account that \n# a text can relate to more than one category.\n# But lets use only 16 + 1 development levels\n\nkpdf['ClassD'] = kpdf['Class'].apply(lambda x: x[6:])\n\n# it is better to transform column ClassD to numeric later\n# kpdf['ClassD'] = pd.to_numeric(kpdf['ClassD'])\n\nkpdf","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:35:41.108664Z","iopub.execute_input":"2023-07-16T04:35:41.108965Z","iopub.status.idle":"2023-07-16T04:35:41.123629Z","shell.execute_reply.started":"2023-07-16T04:35:41.108939Z","shell.execute_reply":"2023-07-16T04:35:41.122787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Distribution shows that now the training dataset is balanced better\nvc = kpdf.ClassD.value_counts()\n\nplt.figure(figsize = (12,4))\nplt.bar(vc.index, vc.values)\nplt.title(\"Training Data Distribution\")","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:35:41.125201Z","iopub.execute_input":"2023-07-16T04:35:41.125852Z","iopub.status.idle":"2023-07-16T04:35:41.521215Z","shell.execute_reply.started":"2023-07-16T04:35:41.125822Z","shell.execute_reply":"2023-07-16T04:35:41.520105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Modified. Original was taken from:\n# https://www.kaggle.com/code/christofhenkel/how-to-preprocessing-when-using-embeddings\n\ndef clean_text_words(x):\n \"\"\"\n Clears text excluding spesial symbols and english stop words.\n :param x: input text\n :return: list of words and number of stop words excluded\n \"\"\"\n\n stopwords_count = 0\n x = str(x)\n for punct in \"/-'\":\n x = x.replace(punct, ' ')\n for punct in '&':\n x = x.replace(punct, f' {punct} ')\n for punct in '?!.,\"#$%\\'()*+-/:;<=>@[\\\\]^_`{|}~' + '“”’':\n x = x.replace(punct, '')\n\n x = x.replace('\\n', ' ').strip().lower()\n\n words = x.split()\n words2 = []\n\n for wrd in words: \n if wrd not in en_stopwords:\n words2.append(wrd)\n else:\n stopwords_count += 1\n\n return words2, stopwords_count\n\n\ndef clean_text(x):\n \"\"\"\n Clears text excluding spesial symbols and english stop words.\n :param x: input text\n :return: text and number of stop words excluded\n \"\"\"\n\n words, stopwords_count = clean_text_words(x)\n if(len(words) == 0):\n print('Attention! Stop word(s) only in training dataset:', x)\n return x\n else:\n phrase = ''\n for wrd in words:\n if(phrase != ''):\n phrase += ' '\n phrase += wrd\n\n return phrase, stopwords_count","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:35:41.522801Z","iopub.execute_input":"2023-07-16T04:35:41.523144Z","iopub.status.idle":"2023-07-16T04:35:41.534557Z","shell.execute_reply.started":"2023-07-16T04:35:41.523115Z","shell.execute_reply":"2023-07-16T04:35:41.533403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Create train_x data\n# Clear train_x data by removing stop words\n\ntrain_x = kpdf['Keywords']\n\nstopwords_cnt = 0\n\nfor i in range(len(train_x)):\n ph_w_stop_words = train_x[i]\n train_x[i], swrds_cnt = clean_text(train_x[i])\n\n #if(swrds_cnt > 0):\n # print('Stopwords found:', ph_w_stop_words, 'Record(',i,')')\n\n stopwords_cnt += swrds_cnt\n\nprint('Stopwords removed:', stopwords_cnt)\ntrain_x","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:35:41.536123Z","iopub.execute_input":"2023-07-16T04:35:41.536509Z","iopub.status.idle":"2023-07-16T04:35:41.829186Z","shell.execute_reply.started":"2023-07-16T04:35:41.536480Z","shell.execute_reply":"2023-07-16T04:35:41.827904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Create 16 + 1 columns accordingly\ntrain_y = pd.get_dummies(kpdf.ClassD) \ntrain_y = train_y[llabels]\n\ntrain_y","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:35:41.838043Z","iopub.execute_input":"2023-07-16T04:35:41.838552Z","iopub.status.idle":"2023-07-16T04:35:41.864950Z","shell.execute_reply.started":"2023-07-16T04:35:41.838510Z","shell.execute_reply":"2023-07-16T04:35:41.864103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"markdown","source":"\n## 2.2 Tokenizing and Word Emdedding","metadata":{}},{"cell_type":"code","source":"# Need calculate max_chunk_size because it affect performance\nmax_chunk_size = 0\nwrdsmax = None\n\nfor phrase in train_x:\n sc_size = len(phrase.split())\n if(max_chunk_size < sc_size):\n max_chunk_size = sc_size\n wrdsmax = phrase\n\nprint(max_chunk_size)\nwrdsmax","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:35:41.866159Z","iopub.execute_input":"2023-07-16T04:35:41.866929Z","iopub.status.idle":"2023-07-16T04:35:41.879059Z","shell.execute_reply.started":"2023-07-16T04:35:41.866898Z","shell.execute_reply":"2023-07-16T04:35:41.877649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Tokenize train_x texts\ntokenizer = Tokenizer()\ntokenizer.fit_on_texts(train_x)\n\nword_index = tokenizer.word_index\nvlen = len(word_index) + 1\n\nvlen","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:35:41.880904Z","iopub.execute_input":"2023-07-16T04:35:41.881316Z","iopub.status.idle":"2023-07-16T04:35:41.911648Z","shell.execute_reply.started":"2023-07-16T04:35:41.881279Z","shell.execute_reply":"2023-07-16T04:35:41.910259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load GloVe vectors\nemb_dict = {}\nf = open('/kaggle/input/glove840b300dtxt/glove.840B.300d.txt','r',encoding='utf-8')\nfor line in f:\n values = line.split(' ')\n word = values[0]\n coefs = np.asarray([float(val) for val in values[1:]])\n emb_dict[word] = coefs\nf.close()\n\nlen(emb_dict)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:35:41.915965Z","iopub.execute_input":"2023-07-16T04:35:41.916333Z","iopub.status.idle":"2023-07-16T04:40:19.138469Z","shell.execute_reply.started":"2023-07-16T04:35:41.916303Z","shell.execute_reply":"2023-07-16T04:40:19.137400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Create embedding matrix\nEMB_DIM = 300\n\nemb_mtrx = np.zeros((vlen, EMB_DIM))\nfor word, i in word_index.items():\n emb_v = emb_dict.get(word)\n if emb_v is not None:\n emb_mtrx[i] = emb_v\n\nemb_mtrx.shape","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:40:19.139803Z","iopub.execute_input":"2023-07-16T04:40:19.140164Z","iopub.status.idle":"2023-07-16T04:40:19.153599Z","shell.execute_reply.started":"2023-07-16T04:40:19.140134Z","shell.execute_reply":"2023-07-16T04:40:19.152450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"markdown","source":"\n## 2.3 AI Model","metadata":{}},{"cell_type":"code","source":"# \"Chunkinize\" train textes\ntrain_x_seq = pad_sequences(tokenizer.texts_to_sequences(train_x), maxlen = max_chunk_size)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:40:19.155441Z","iopub.execute_input":"2023-07-16T04:40:19.155915Z","iopub.status.idle":"2023-07-16T04:40:19.175847Z","shell.execute_reply.started":"2023-07-16T04:40:19.155877Z","shell.execute_reply":"2023-07-16T04:40:19.175008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create Artificial Neural Network model\n# Model (modified) from https://www.kaggle.com/code/andreshg/nlp-glove-bert-tf-idf-lstm-explained\n\nembedding_layer = Embedding(\n input_dim = emb_mtrx.shape[0], \n output_dim = emb_mtrx.shape[1], \n weights = [emb_mtrx], \n input_length = max_chunk_size,\n trainable = False\n)\n\nsequence_input = Input(shape = (max_chunk_size), dtype = 'int32')\nembedding_sequences = embedding_layer(sequence_input)\nlr = SpatialDropout1D(0.2)(embedding_sequences)\nlr = Conv1D(64, 5, activation = 'relu')(lr)\nlr = Bidirectional(LSTM(32, dropout=0.2, recurrent_dropout = 0.2))(lr)\nlr = Dense(512, activation = 'relu')(lr)\nlr = Dropout(0.5)(lr)\nlr = Dense(512, activation = 'relu')(lr)\nlr = Dropout(0.5)(lr)\noutputs = Dense(17, activation = 'sigmoid')(lr)\nmodel = Model(sequence_input, outputs)\n\nmodel.compile(\n optimizer = Adam(learning_rate = 0.001),\n loss = 'binary_crossentropy',\n metrics = ['accuracy']\n)\n\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:40:19.177006Z","iopub.execute_input":"2023-07-16T04:40:19.177373Z","iopub.status.idle":"2023-07-16T04:40:19.883595Z","shell.execute_reply.started":"2023-07-16T04:40:19.177323Z","shell.execute_reply":"2023-07-16T04:40:19.882498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Train the model\n# Ovefitting is Ok for this model (prototype)\n\nRLROnPl = ReduceLROnPlateau(\n factor = 0.1,\n min_lr = 0.01,\n monitor = 'val_loss',\n verbose = 1\n)\n\nhist = model.fit(\n train_x_seq,\n train_y,\n batch_size = 1024,\n validation_data = (train_x_seq, train_y), # Ovefitting is Ok for this model (prototype)\n epochs = 500,\n callbacks = [RLROnPl],\n verbose = 0\n)\n\n# Overfitting is Ok for this model (prototype)\nhistory_df = pd.DataFrame(hist.history)\nhistory_df.loc[:, ['loss', 'val_loss']].plot();\nprint(\"Minimum validation loss: {}\".format(history_df['val_loss'].min()))","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:40:19.885393Z","iopub.execute_input":"2023-07-16T04:40:19.885844Z","iopub.status.idle":"2023-07-16T04:41:49.449514Z","shell.execute_reply.started":"2023-07-16T04:40:19.885806Z","shell.execute_reply":"2023-07-16T04:41:49.448487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"markdown","source":"\n## 2.4 Helpful Functions","metadata":{}},{"cell_type":"code","source":"def prep_str_chunks(words, chunk_size):\n \"\"\"\n Divides text to chunks\n :param words: input text presented as the list of words\n :param chunk_size: maximal size of chunks (in word count)\n :return: list of text chunks with size = chunk_size\n \"\"\"\n\n res = []\n i = 0\n while True:\n ss = ''\n iup = i + chunk_size\n if (iup > len(words)):\n iup = len(words)\n for ii in range(i, iup):\n if(len(ss) > 0):\n ss = ss + ' '\n ss = ss + words[ii]\n res.append(ss)\n i += 1\n if (i >= len(words)):\n break\n return res\n\n\ndef prep_str_with_sub_chunks(words, chunk_size):\n \"\"\"\n Divides text to chunks\n :param words: input text presented as the list of words\n :param chunk_size: maximal size of chunks (in word count)\n :return: list of text chunks with size from one word up to chunk_size\n \"\"\"\n \n res = []\n for i in range(1, chunk_size + 1):\n res.extend(prep_str_chunks(words, i))\n return res\n\n\ndef lr_predict(txt):\n \"\"\"\n Makes predictions (evaluation by 17 SLMHM levels) for the specific text\n :param txt: input text \n :return: max values of chunks predictions for 17 levels \n \"\"\"\n \n txt_cleaned, stopwords_count = clean_text_words(txt)\n wrds = prep_str_with_sub_chunks(txt_cleaned, max_chunk_size)\n\n tmt_mtrx = pad_sequences(\n tokenizer.texts_to_sequences(wrds),\n maxlen = max_chunk_size\n )\n\n preds = model.predict(tmt_mtrx, verbose = 0)\n return np.amax(preds, axis = 0)\n\n\ndef lr_predict_lst(txtar, maxcnt):\n \"\"\"\n Makes predictions for the list of texts\n :param txtar: list of input text\n :param maxcnt: maximum number of texts to be predicted\n if maxcnt < 0, whole list will be predicted\n :return: max values of chunks predictions for 17 levels \n \"\"\"\n \n predictions = []\n \n i = 0\n sz = len(txtar)\n \n if((0 < maxcnt) and (maxcnt < sz)):\n sz = maxcnt\n \n for t in txtar:\n if(i == maxcnt):\n break\n \n ar_predict = lr_predict(t)\n predictions.append(ar_predict)\n \n print ('Text:', i, 'of', sz, 'done ', end=\"\\r\")\n i += 1\n \n print ('Text:', sz, 'of', sz, 'done (total:',len(txtar),')')\n return predictions\n\n\ndef plot_SLMHM_L(preds):\n \"\"\"\n Draws predictions as bar chart (SLMHM levels)\n :param preds: predictions\n \"\"\"\n \n plt.figure(figsize = (12,4))\n df = pd.DataFrame({'levels':preds, 'labels':llabels}, index = llabels_str)\n ax = df.plot.barh(y = 'levels') \n plt.ylabel('SLMHM Level')\n plt.xlabel('Value')\n plt.title('Maturity')\n plt.show()\n\n\ndef plot_SPEM_L(preds):\n \"\"\"\n Draws predictions as bar chart (SPEM levels)\n :param preds: predictions\n \"\"\"\n\n pspem = [0.0] * 5\n pspem[0] = preds[0]\n for i in range(1,5):\n pspem[i] = max(preds[i * 4 - 3: i * 4 + 1])\n df = pd.DataFrame({'levels':pspem, 'labels':llabels_spem}, index = llabels_spem_str)\n ax = df.plot.barh(y = 'levels') \n plt.ylabel('SPEM Pattern')\n plt.xlabel('Value')\n plt.title('Maturity')\n plt.show()\n\n \ndef plot_SLMHM_L2(preds1, preds2, name1, name2):\n \"\"\"\n Draws predictions as bar chart (SLMHM levels) for 2 sources\n :param preds1: first predictions\n :param preds2: second predictions\n :param name1: label for first predictions\n :param name2: label for second predictions\n \"\"\"\n \n plt.figure(figsize = (12,4))\n df = pd.DataFrame({name1:preds1, name2:preds2, 'labels':llabels}, index = llabels_str)\n ax = df.plot.barh()\n plt.ylabel('SLMHM Level')\n plt.xlabel('Value')\n plt.title('Maturity')\n plt.show()\n\n\ndef plot_SPEM_L2(preds1, preds2, name1, name2):\n \"\"\"\n Draws predictions as bar chart (SPEM levels) for 2 sources\n :param preds1: first predictions\n :param preds2: second predictions\n :param name1: label for first predictions\n :param name2: label for second predictions\n \"\"\"\n\n pspem1 = [0.0] * 5\n pspem1[0] = preds1[0]\n for i in range(1,5):\n pspem1[i] = max(preds1[i * 4 - 3: i * 4 + 1])\n pspem2 = [0.0] * 5\n pspem2[0] = preds2[0]\n for i in range(1,5):\n pspem2[i] = max(preds2[i * 4 - 3: i * 4 + 1])\n df = pd.DataFrame({name1:pspem1, name2:pspem2, 'labels':llabels_spem}, index = llabels_spem_str)\n ax = df.plot.barh()\n plt.ylabel('SPEM Pattern')\n plt.xlabel('Value')\n plt.title('Maturity')\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:41:49.450838Z","iopub.execute_input":"2023-07-16T04:41:49.451139Z","iopub.status.idle":"2023-07-16T04:41:49.478604Z","shell.execute_reply.started":"2023-07-16T04:41:49.451112Z","shell.execute_reply":"2023-07-16T04:41:49.477768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"markdown","source":"\n## 2.5 AI Model Tests\n\nSeveral tests are presented here for better understanding the model. The following notebook includes more tests of the training dataset: https://www.kaggle.com/code/dmitriygakh/see-slmhm-spem-maturity-tests-lstm-glove","metadata":{}},{"cell_type":"code","source":"# All bars for this text shall be of length 1.0\n# To-Do: More validation texts need\ntest_mature_text = \"Chaos was a primary characteristics of universe. The research should consider desire as the starting point. Desire leads to intention. Intention is a readiness to plan and spend resources. Availability of resources is a feasibility. Feasibility should be achieved before the project implementation. Active project relates to structure. Structure should be evaluated to produce benefits on base of the evaluation. Continous improvement and upgrading allows to maximize the benefit. All alternatives are helpful to appearence of innovation. Innovations hould provide reward for society, economy, and natural environment. It creates a ground to maximize benefit for all and to minimize loss for all. Synergy in interaction between society, economy, and environment will appear and lead to the extraordinary innotations. People will have opportunity to express their feelings and emotions. Monitoring will generate real time information and produce new knowledge. Expansion based on new knowledge will lead to The complete fusion of all society-economy-environment interactions into something infinite and overall.\"\nplot_SLMHM_L(lr_predict(test_mature_text))","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:47:41.228815Z","iopub.execute_input":"2023-07-16T06:47:41.229319Z","iopub.status.idle":"2023-07-16T06:47:41.775413Z","shell.execute_reply.started":"2023-07-16T06:47:41.229285Z","shell.execute_reply":"2023-07-16T06:47:41.774225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Ok: text relates to all SPEM patterns\n# Note: SPEM generalizes SLMHM\nplot_SPEM_L(lr_predict(test_mature_text))","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:41:50.636625Z","iopub.execute_input":"2023-07-16T04:41:50.637271Z","iopub.status.idle":"2023-07-16T04:41:51.022778Z","shell.execute_reply.started":"2023-07-16T04:41:50.637239Z","shell.execute_reply":"2023-07-16T04:41:51.021655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Ok: Structure relates to SLMHM Level 1.4\nplot_SLMHM_L(lr_predict(\"structure\"))","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:41:51.025361Z","iopub.execute_input":"2023-07-16T04:41:51.026487Z","iopub.status.idle":"2023-07-16T04:41:51.504891Z","shell.execute_reply.started":"2023-07-16T04:41:51.026451Z","shell.execute_reply":"2023-07-16T04:41:51.503820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Ok: Ethics relates to SLMHM Level 3.1\n# Note: Ethics supposes a reward for all.\n# Note: In SLMHM/SPEM ehics relates to neohumanizm and level 3.1\n# it is not human-centric, but supposes rewerd for all system parts.\nplot_SLMHM_L(lr_predict(\"ethics\"))","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:41:51.506370Z","iopub.execute_input":"2023-07-16T04:41:51.506828Z","iopub.status.idle":"2023-07-16T04:41:51.984775Z","shell.execute_reply.started":"2023-07-16T04:41:51.506788Z","shell.execute_reply":"2023-07-16T04:41:51.983598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Ok: Mission relates to SLMHM Level 4.4\nplot_SLMHM_L(lr_predict(\"mission\"))","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:41:51.986285Z","iopub.execute_input":"2023-07-16T04:41:51.986677Z","iopub.status.idle":"2023-07-16T04:41:52.456853Z","shell.execute_reply.started":"2023-07-16T04:41:51.986646Z","shell.execute_reply":"2023-07-16T04:41:52.456072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Ok: Knowledge is a some kind resource (Level 1.3).\n# Note: Knowledge extraction relates to SLMHM Level 4.2\nplot_SLMHM_L(lr_predict(\"knowledge\"))","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:41:52.458157Z","iopub.execute_input":"2023-07-16T04:41:52.458969Z","iopub.status.idle":"2023-07-16T04:41:52.946291Z","shell.execute_reply.started":"2023-07-16T04:41:52.458935Z","shell.execute_reply":"2023-07-16T04:41:52.943573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# SPEM bar chart generalises SLMHM chart\nplot_SPEM_L(lr_predict(\"knowledge\")) ","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:41:52.947582Z","iopub.execute_input":"2023-07-16T04:41:52.947927Z","iopub.status.idle":"2023-07-16T04:41:53.292224Z","shell.execute_reply.started":"2023-07-16T04:41:52.947900Z","shell.execute_reply":"2023-07-16T04:41:53.291104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Ok: In SLMHM/SPEM ehics relates to neohumanizm and ISP\n# Note: Ethics can relate both = SPEM ASP (humanism) and ISP (neo-humanism)\nplot_SPEM_L(lr_predict(\"ethics\")) ","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:41:53.293933Z","iopub.execute_input":"2023-07-16T04:41:53.295024Z","iopub.status.idle":"2023-07-16T04:41:53.633113Z","shell.execute_reply.started":"2023-07-16T04:41:53.294980Z","shell.execute_reply":"2023-07-16T04:41:53.630773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Ok: Planetary relates to CSP\nplot_SPEM_L(lr_predict(\"planetary\")) ","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:41:53.634310Z","iopub.execute_input":"2023-07-16T04:41:53.634700Z","iopub.status.idle":"2023-07-16T04:41:53.965400Z","shell.execute_reply.started":"2023-07-16T04:41:53.634669Z","shell.execute_reply":"2023-07-16T04:41:53.964330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"markdown","source":"\n# 3 | Analyses of Kaggle Writeups\n","metadata":{}},{"cell_type":"code","source":"# Conpare total maturity of prev and last titles\n# There is a disbalance between number of records in prev (1865) and last (1262) sets\n\n# Part of code is taken from: https://www.kaggle.com/code/leonidkulyk/kaggle-ai-report-topic-selection\n\nkwups_df = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv\")\nkwups_df[\"Date of Writeup\"] = pd.to_datetime(kwups_df[\"Date of Writeup\"])\nkwups_fl_prev_df = kwups_df[kwups_df['Date of Writeup'].dt.year < 2021]\nkwups_fl_last_df = kwups_df[kwups_df['Date of Writeup'].dt.year >= 2021]\n\nprint(\"Records before 2021:\", len(kwups_fl_prev_df), \"and in 2021 and after:\", len(kwups_fl_last_df))","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:41:53.974896Z","iopub.execute_input":"2023-07-16T04:41:53.975218Z","iopub.status.idle":"2023-07-16T04:41:54.720200Z","shell.execute_reply.started":"2023-07-16T04:41:53.975190Z","shell.execute_reply":"2023-07-16T04:41:54.719147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Disbalance in unicue titles are greater (230 against 81)\ntxts_prev = kwups_fl_prev_df[\"Title of Competition\"].unique()\ntxts_last = kwups_fl_last_df[\"Title of Competition\"].unique()\nprint(\"Titles before 2021:\", len(txts_prev), \"and in 2021 and after:\", len(txts_last))","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:41:54.721820Z","iopub.execute_input":"2023-07-16T04:41:54.722116Z","iopub.status.idle":"2023-07-16T04:41:54.732017Z","shell.execute_reply.started":"2023-07-16T04:41:54.722090Z","shell.execute_reply":"2023-07-16T04:41:54.731081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Predict values for all texts (-1)\ntxts_predicted_ls_prev = lr_predict_lst(txts_prev, -1)\ntxts_predicted_ls_last = lr_predict_lst(txts_last, -1)\n\n# Calculate maximums by all levels for all texts\ntxts_predicted_max_prev = np.amax(txts_predicted_ls_prev, axis = 0)\ntxts_predicted_max_last = np.amax(txts_predicted_ls_last, axis = 0)\n\n# Calculate averages by all levels for all texts\ntxts_predicted_ave_prev = np.average(txts_predicted_ls_prev, axis = 0)\ntxts_predicted_ave_last = np.average(txts_predicted_ls_last, axis = 0)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:41:54.733068Z","iopub.execute_input":"2023-07-16T04:41:54.733446Z","iopub.status.idle":"2023-07-16T04:42:14.200385Z","shell.execute_reply.started":"2023-07-16T04:41:54.733413Z","shell.execute_reply":"2023-07-16T04:42:14.199156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compare maturity by SLMHM bars\n# Chart shows that generally quantitatively papers \"before 2021\" represent majority\nplot_SLMHM_L2(\n txts_predicted_max_prev,\n txts_predicted_max_last,\n \"before 2021\",\n \"2021 and after\",\n)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:42:14.202437Z","iopub.execute_input":"2023-07-16T04:42:14.202886Z","iopub.status.idle":"2023-07-16T04:42:14.680319Z","shell.execute_reply.started":"2023-07-16T04:42:14.202847Z","shell.execute_reply":"2023-07-16T04:42:14.679121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Compare maturity by SLMHM bars (Normalized)\n# Chart shows progress for several levels and decline for other\nplot_SLMHM_L2(\n txts_predicted_ave_prev,\n txts_predicted_ave_last,\n \"before 2021\",\n \"2021 and after\",\n)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:42:14.681973Z","iopub.execute_input":"2023-07-16T04:42:14.682800Z","iopub.status.idle":"2023-07-16T04:42:15.162850Z","shell.execute_reply.started":"2023-07-16T04:42:14.682742Z","shell.execute_reply":"2023-07-16T04:42:15.161871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Compare maturity by SPEM bars (Absolute number of texts)\nplot_SPEM_L2(\n txts_predicted_max_prev,\n txts_predicted_max_last,\n \"before 2021\",\n \"2021 and after\",\n)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:42:15.164882Z","iopub.execute_input":"2023-07-16T04:42:15.165321Z","iopub.status.idle":"2023-07-16T04:42:15.412819Z","shell.execute_reply.started":"2023-07-16T04:42:15.165270Z","shell.execute_reply":"2023-07-16T04:42:15.411720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Compare maturity by SPEM bars (Normalized)\n# SPEM shows some progress in ISP\nplot_SPEM_L2(\n txts_predicted_ave_prev,\n txts_predicted_ave_last,\n \"before 2021\",\n \"2021 and after\",\n)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:42:15.414487Z","iopub.execute_input":"2023-07-16T04:42:15.414922Z","iopub.status.idle":"2023-07-16T04:42:15.663963Z","shell.execute_reply.started":"2023-07-16T04:42:15.414882Z","shell.execute_reply":"2023-07-16T04:42:15.662932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"markdown","source":"\n# 4 | Analyses of Arxiv Metadata\n","metadata":{}},{"cell_type":"code","source":"# Part of code is taken from: https://www.kaggle.com/code/sophieb/ai-report-basic-eda\n\narxiv_df = pd.read_json(\"/kaggle/input/2023-kaggle-ai-report/arxiv_metadata_20230510.json\", \n lines=True,\n convert_dates=True,\n chunksize=100000)\n\nnlp_cat_arxiv = [\"cs.IR\",\"cs.CL\",\"cs.LG\",\"cs.NE\",\"cs.AI\", \"cs.DL\", \"cs.HC\", \"cs.SI\",\"cs.SD\",\"cs.WEB\",\"stat.ML\",\"math.IT\",\"physics.data-an\"]\n\nai_pprs_df = pd.DataFrame()\n\nfor chunk in arxiv_df:\n chunk_ai_pprs = chunk[chunk.categories.isin(nlp_cat_arxiv)]\n ai_pprs_df = pd.concat([ai_pprs_df, chunk_ai_pprs], ignore_index = True)\n\nai_pprs_df['update_date'] = pd.to_datetime(ai_pprs_df['update_date'])\n\nai_pprs_df.info()","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:42:15.666036Z","iopub.execute_input":"2023-07-16T04:42:15.666548Z","iopub.status.idle":"2023-07-16T04:44:58.807275Z","shell.execute_reply.started":"2023-07-16T04:42:15.666506Z","shell.execute_reply":"2023-07-16T04:44:58.806008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# There is some balance between older (<2021 / 29471) and\n# later (>=2021 / 25284) papers\n\nai_pprs_fl_prev_df = ai_pprs_df[ai_pprs_df['update_date'].dt.year < 2021]\nai_pprs_fl_last_df = ai_pprs_df[ai_pprs_df['update_date'].dt.year >= 2021]\n\nprint(\"Records before 2021:\", len(ai_pprs_fl_prev_df), \"and in 2021 and after:\", len(ai_pprs_fl_last_df))","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:44:58.809258Z","iopub.execute_input":"2023-07-16T04:44:58.810095Z","iopub.status.idle":"2023-07-16T04:44:58.921632Z","shell.execute_reply.started":"2023-07-16T04:44:58.810055Z","shell.execute_reply":"2023-07-16T04:44:58.920423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Predictions for whole ArXiv texts requres a long time.\n# Only part of them can be analyset to quickly probe the method\n# MAX_ArXiv_Samples = 100 # Only 100 first rows (demonstrative results)\nMAX_ArXiv_Samples = -1 # All rows (actual results, about 3 hours of calculations)\n\n# Predict values\nabstrs_prev = ai_pprs_fl_prev_df['abstract']\nabstrs_last = ai_pprs_fl_last_df['abstract']\n\n# Predict for first 100 pages for debug purposes\nabstrs_predicted_ls_prev = lr_predict_lst(abstrs_prev, MAX_ArXiv_Samples)\nabstrs_predicted_ls_last = lr_predict_lst(abstrs_last, MAX_ArXiv_Samples)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T04:44:58.922958Z","iopub.execute_input":"2023-07-16T04:44:58.923441Z","iopub.status.idle":"2023-07-16T06:35:56.797804Z","shell.execute_reply.started":"2023-07-16T04:44:58.923380Z","shell.execute_reply":"2023-07-16T06:35:56.796601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Calculate maximums by all levels for all texts\nabstrs_predicted_max_prev = np.amax(abstrs_predicted_ls_prev, axis = 0)\nabstrs_predicted_max_last = np.amax(abstrs_predicted_ls_last, axis = 0)\n\n# Calculate averages by all levels for all texts\nabstrs_predicted_ave_prev = np.average(abstrs_predicted_ls_prev, axis = 0)\nabstrs_predicted_ave_last = np.average(abstrs_predicted_ls_last, axis = 0)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:56.799620Z","iopub.execute_input":"2023-07-16T06:35:56.800042Z","iopub.status.idle":"2023-07-16T06:35:56.964423Z","shell.execute_reply.started":"2023-07-16T06:35:56.800002Z","shell.execute_reply":"2023-07-16T06:35:56.963152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(abstrs_prev))\nprint(len(abstrs_last))","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:56.966672Z","iopub.execute_input":"2023-07-16T06:35:56.967645Z","iopub.status.idle":"2023-07-16T06:35:56.974523Z","shell.execute_reply.started":"2023-07-16T06:35:56.967603Z","shell.execute_reply":"2023-07-16T06:35:56.973262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Maturity of papers (absolute values)\nplot_SLMHM_L2(\n abstrs_predicted_max_prev,\n abstrs_predicted_max_last,\n \"before 2021\",\n \"2021 and after\",\n)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:56.976499Z","iopub.execute_input":"2023-07-16T06:35:56.976926Z","iopub.status.idle":"2023-07-16T06:35:57.423554Z","shell.execute_reply.started":"2023-07-16T06:35:56.976885Z","shell.execute_reply":"2023-07-16T06:35:57.422392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Maturity of papers (Normalized)\n# The chart shows progress in papers' maturity on all SLMHM Levels\nplot_SLMHM_L2(\n abstrs_predicted_ave_prev,\n abstrs_predicted_ave_last,\n \"before 2021\",\n \"2021 and after\",\n)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:57.424952Z","iopub.execute_input":"2023-07-16T06:35:57.425711Z","iopub.status.idle":"2023-07-16T06:35:57.834229Z","shell.execute_reply.started":"2023-07-16T06:35:57.425673Z","shell.execute_reply":"2023-07-16T06:35:57.833112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Maturity of papers according SPEM (quantitative)\nplot_SPEM_L2(\n abstrs_predicted_max_prev,\n abstrs_predicted_max_last,\n \"before 2021\",\n \"2021 and after\",\n)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:57.835943Z","iopub.execute_input":"2023-07-16T06:35:57.836812Z","iopub.status.idle":"2023-07-16T06:35:58.082066Z","shell.execute_reply.started":"2023-07-16T06:35:57.836780Z","shell.execute_reply":"2023-07-16T06:35:58.080909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Maturity of papers according SPEM (normalized)\nplot_SPEM_L2(\n abstrs_predicted_ave_prev,\n abstrs_predicted_ave_last,\n \"before 2021\",\n \"2021 and after\",\n)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.084003Z","iopub.execute_input":"2023-07-16T06:35:58.084444Z","iopub.status.idle":"2023-07-16T06:35:58.315967Z","shell.execute_reply.started":"2023-07-16T06:35:58.084404Z","shell.execute_reply":"2023-07-16T06:35:58.314679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"markdown","source":"\n# 5 | Finding the best Kaggle Writeups\n","metadata":{}},{"cell_type":"markdown","source":"The mothods of this chapter allows select the most mature texts.","metadata":{}},{"cell_type":"code","source":"# Now we have already calculated:\n# Lists of unique titles: txts_prev, txts_last\n# List of predictions for unique titles: txts_predicted_ls_prev, txts_predicted_ls_last\n\n\ndef print_best_texts(texts, preds, level):\n \"\"\"\n Prints texts relating to the best SLMHM level (texts where the level is the highes value)\n :param texts: list of texts\n :param preds: list of texts' predictions\n :param level: level to check\n \"\"\"\n\n max_lvl = 0\n for i in range(len(preds)):\n pred_val = preds[i][level]\n if (max_lvl < pred_val):\n max_lvl = pred_val\n\n txts = []\n txt_idxs = []\n if(max_lvl > 0):\n for i in range(len(preds)):\n pred_val = preds[i][level]\n if (max_lvl == pred_val):\n txts.append(texts[i])\n txt_idxs.append(i)\n\n print('Level: ',max_lvl,'texts:',len(txts))\n for i in range(len(txts)):\n print('Text',txt_idxs[i],':',txts[i])\n #return max_lvl, txts","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.318003Z","iopub.execute_input":"2023-07-16T06:35:58.318448Z","iopub.status.idle":"2023-07-16T06:35:58.328391Z","shell.execute_reply.started":"2023-07-16T06:35:58.318409Z","shell.execute_reply":"2023-07-16T06:35:58.327222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# One of previous competitions is about a mission that is the highest maturity (level 16)\n# Rate 0.99x shows a close relation to level 16\nprint_best_texts(txts_prev, txts_predicted_ls_prev, 16)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.329925Z","iopub.execute_input":"2023-07-16T06:35:58.330333Z","iopub.status.idle":"2023-07-16T06:35:58.347624Z","shell.execute_reply.started":"2023-07-16T06:35:58.330296Z","shell.execute_reply":"2023-07-16T06:35:58.346501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Rate 0.26x is very low and we cannot speak about\n# a close relation to level 16 (Totality, Globalization, Mission, etc.)\n# There is no progress at level 16\nprint_best_texts(txts_last, txts_predicted_ls_last, 16)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.349216Z","iopub.execute_input":"2023-07-16T06:35:58.349694Z","iopub.status.idle":"2023-07-16T06:35:58.362732Z","shell.execute_reply.started":"2023-07-16T06:35:58.349655Z","shell.execute_reply":"2023-07-16T06:35:58.361642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# AI model decides that te paper relates to the expansion? i.e. level 15\n# Rate 0.99x is a good corellation\nprint_best_texts(txts_prev, txts_predicted_ls_prev, 15)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.364407Z","iopub.execute_input":"2023-07-16T06:35:58.364900Z","iopub.status.idle":"2023-07-16T06:35:58.375251Z","shell.execute_reply.started":"2023-07-16T06:35:58.364848Z","shell.execute_reply":"2023-07-16T06:35:58.374038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# AI model decides that te paper relates to the expansion? i.e. level 15\n# Rate 0.99x is a good corellation\n# There is a progress at level 15\nprint_best_texts(txts_last, txts_predicted_ls_last, 15)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.376366Z","iopub.execute_input":"2023-07-16T06:35:58.376736Z","iopub.status.idle":"2023-07-16T06:35:58.389298Z","shell.execute_reply.started":"2023-07-16T06:35:58.376707Z","shell.execute_reply":"2023-07-16T06:35:58.388011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# A good relation to level 14 (Streamlining, Knowledge extraction, etc.)\nprint_best_texts(txts_prev, txts_predicted_ls_prev, 14)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.391150Z","iopub.execute_input":"2023-07-16T06:35:58.391503Z","iopub.status.idle":"2023-07-16T06:35:58.400540Z","shell.execute_reply.started":"2023-07-16T06:35:58.391461Z","shell.execute_reply":"2023-07-16T06:35:58.399712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# A good relation to level 14 (Streamlining, Knowledge extraction, etc.)\nprint_best_texts(txts_last, txts_predicted_ls_last, 14)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.401837Z","iopub.execute_input":"2023-07-16T06:35:58.402976Z","iopub.status.idle":"2023-07-16T06:35:58.412071Z","shell.execute_reply.started":"2023-07-16T06:35:58.402945Z","shell.execute_reply":"2023-07-16T06:35:58.410931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# A good relation to level 13 (Information generation, etc.)\nprint_best_texts(txts_prev, txts_predicted_ls_prev, 13)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.413646Z","iopub.execute_input":"2023-07-16T06:35:58.414657Z","iopub.status.idle":"2023-07-16T06:35:58.426652Z","shell.execute_reply.started":"2023-07-16T06:35:58.414617Z","shell.execute_reply":"2023-07-16T06:35:58.425401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# A good relation to level 13 (Information generation, etc.)\n# Progress\nprint_best_texts(txts_last, txts_predicted_ls_last, 13)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.427942Z","iopub.execute_input":"2023-07-16T06:35:58.428706Z","iopub.status.idle":"2023-07-16T06:35:58.437140Z","shell.execute_reply.started":"2023-07-16T06:35:58.428675Z","shell.execute_reply":"2023-07-16T06:35:58.435937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"markdown","source":"\n# 6 | Finding the best ArXiv Papers\n","metadata":{}},{"cell_type":"markdown","source":"The mothods of this chapter allows select the most mature ArXiv papers.","metadata":{}},{"cell_type":"code","source":"# Now we have already calculated:\n# Lists of unique titles: abstrs_prev, abstrs_last\n# List of predictions for unique titles: abstrs_predicted_ls_prev, abstrs_predicted_ls_last\n\ndef print_best_abstracts(texts_ser, preds, level):\n \"\"\"\n Prints texts relating to the best SLMHM level.\n Prints texts where the level is the highes value.\n Prints only 3 first texts if available.\n :param texts_ser: series of texts\n :param preds: list of texts' predictions\n :param level: level to check\n \"\"\"\n \n max_lvl = 0\n\n for i in range(len(preds)):\n pred_val = preds[i][level]\n if (max_lvl < pred_val):\n max_lvl = pred_val\n\n txts = []\n txt_idxs = []\n if(max_lvl > 0):\n for i in range(len(preds)):\n pred_val = preds[i][level]\n if (max_lvl == pred_val):\n txts.append(texts_ser[texts_ser.index[i]])\n txt_idxs.append(int(texts_ser.index[i]))\n\n print('Level: ',max_lvl,'texts:',len(txts))\n\n sz = len(txts)\n if(sz > 3):\n sz = 3\n print('3 first texts:')\n \n for i in range(sz):\n print('Text',texts_ser.index[i],':',txts[i])","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.438599Z","iopub.execute_input":"2023-07-16T06:35:58.439497Z","iopub.status.idle":"2023-07-16T06:35:58.449711Z","shell.execute_reply.started":"2023-07-16T06:35:58.439453Z","shell.execute_reply":"2023-07-16T06:35:58.448616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Close relation to level 16\nprint_best_abstracts(abstrs_prev, abstrs_predicted_ls_prev, 16)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.451458Z","iopub.execute_input":"2023-07-16T06:35:58.452580Z","iopub.status.idle":"2023-07-16T06:35:58.513051Z","shell.execute_reply.started":"2023-07-16T06:35:58.452542Z","shell.execute_reply":"2023-07-16T06:35:58.511785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Close relation to level 16\nprint_best_abstracts(abstrs_last, abstrs_predicted_ls_last, 16)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.514468Z","iopub.execute_input":"2023-07-16T06:35:58.514849Z","iopub.status.idle":"2023-07-16T06:35:58.559385Z","shell.execute_reply.started":"2023-07-16T06:35:58.514819Z","shell.execute_reply":"2023-07-16T06:35:58.558236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Close relation to level 15\nprint_best_abstracts(abstrs_prev, abstrs_predicted_ls_prev, 15)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.562964Z","iopub.execute_input":"2023-07-16T06:35:58.563277Z","iopub.status.idle":"2023-07-16T06:35:58.618162Z","shell.execute_reply.started":"2023-07-16T06:35:58.563250Z","shell.execute_reply":"2023-07-16T06:35:58.616882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Close relation to level 15\nprint_best_abstracts(abstrs_last, abstrs_predicted_ls_last, 15)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.619883Z","iopub.execute_input":"2023-07-16T06:35:58.620831Z","iopub.status.idle":"2023-07-16T06:35:58.665300Z","shell.execute_reply.started":"2023-07-16T06:35:58.620799Z","shell.execute_reply":"2023-07-16T06:35:58.664215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"markdown","source":"\n# 7 | Kaggle Writeups Progress\n","metadata":{}},{"cell_type":"code","source":"# Now we have already calculated:\n# Lists of unique titles: txts_prev, txts_last\n# List of predictions for unique titles: txts_predicted_ls_prev, txts_predicted_ls_last\n\n\n# To-Do: threshold does not work\ndef get_best_pred_num(preds, level, threshold):\n \"\"\"\n Calculates number of texts relating to the best SLMHM level \n if the level is greater than threshold.\n :param preds: list of texts' predictions\n :param level: level to check\n :param threshold: text where predicted values are < threshold are included\n :return: number of texts \n \"\"\"\n \n max_lvl = 0\n for i in range(len(preds)):\n pred_val = preds[i][level]\n if (max_lvl < pred_val):\n max_lvl = pred_val\n\n len_wups_num = 0\n\n if(max_lvl > 0):\n for i in range(len(preds)):\n pred_val = preds[i][level]\n if (max_lvl == pred_val):\n len_wups_num += 1\n \n return len_wups_num","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.666757Z","iopub.execute_input":"2023-07-16T06:35:58.667075Z","iopub.status.idle":"2023-07-16T06:35:58.675100Z","shell.execute_reply.started":"2023-07-16T06:35:58.667047Z","shell.execute_reply":"2023-07-16T06:35:58.674063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Evaluate number of the best writeups by SLMHM levels\n\nwups_prev_q = []\nwups_last_q = []\n\nfor i in range(17):\n res = get_best_pred_num(txts_predicted_ls_prev, i, 0.8)\n wups_prev_q.append(res)\n res = get_best_pred_num(txts_predicted_ls_last, i, 0.8)\n wups_last_q.append(res)\n\n# Progress in better writeup by SLMHM levels\nnp.subtract(np.array(wups_last_q), np.array(wups_prev_q))","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.676656Z","iopub.execute_input":"2023-07-16T06:35:58.677632Z","iopub.status.idle":"2023-07-16T06:35:58.695892Z","shell.execute_reply.started":"2023-07-16T06:35:58.677594Z","shell.execute_reply":"2023-07-16T06:35:58.694934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Progress by SLMHM levels on the bar chart.\n# It shows progress on several levels and a decline on others\n\nplt.figure(figsize = (12,4))\ndf = pd.DataFrame({'before 2021':wups_prev_q, '2021 and after':wups_last_q, 'labels':llabels}, index = llabels_str)\nax = df.plot.bar()\nplt.ylabel('SLMHM Level')\nplt.xlabel('Value')\nplt.title('Maturity')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:58.697643Z","iopub.execute_input":"2023-07-16T06:35:58.698047Z","iopub.status.idle":"2023-07-16T06:35:59.129697Z","shell.execute_reply.started":"2023-07-16T06:35:58.698009Z","shell.execute_reply":"2023-07-16T06:35:59.128606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"markdown","source":"\n# 8 | ArXiv Papers Progress\n","metadata":{}},{"cell_type":"code","source":"# Now we have calculated:\n# Lists of unique titles: abstrs_prev, abstrs_last\n# List of predictions for unique titles: abstrs_predicted_ls_prev, abstrs_predicted_ls_last\n# And function get_best_pred_num - function we can use here","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:59.131189Z","iopub.execute_input":"2023-07-16T06:35:59.131532Z","iopub.status.idle":"2023-07-16T06:35:59.136965Z","shell.execute_reply.started":"2023-07-16T06:35:59.131498Z","shell.execute_reply":"2023-07-16T06:35:59.135730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Evaluate number of the best papers by SLMHM levels\n\narxiv_prev_q = []\narxiv_last_q = []\n\nfor i in range(17):\n res = get_best_pred_num(abstrs_predicted_ls_prev, i, 0.8)\n arxiv_prev_q.append(res)\n res = get_best_pred_num(abstrs_predicted_ls_last, i, 0.8)\n arxiv_last_q.append(res)\n\n# Progress in better paper by SLMHM levels\n# General progress seems having positive and negative progress\nnp.subtract(np.array(arxiv_last_q), np.array(arxiv_prev_q))","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:35:59.138490Z","iopub.execute_input":"2023-07-16T06:35:59.138817Z","iopub.status.idle":"2023-07-16T06:36:00.593505Z","shell.execute_reply.started":"2023-07-16T06:35:59.138787Z","shell.execute_reply":"2023-07-16T06:36:00.592520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"code","source":"# Progress by SLMHM levels on the bar chart\n# It shows a explicit progress by several levels\n\nplt.figure(figsize = (12,4))\ndf = pd.DataFrame({'before 2021':arxiv_prev_q, '2021 and after':arxiv_last_q, 'labels':llabels}, index = llabels_str)\nax = df.plot.bar()\nplt.ylabel('SLMHM Level')\nplt.xlabel('Value')\nplt.title('Maturity')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:36:00.594983Z","iopub.execute_input":"2023-07-16T06:36:00.595413Z","iopub.status.idle":"2023-07-16T06:36:01.066167Z","shell.execute_reply.started":"2023-07-16T06:36:00.595309Z","shell.execute_reply":"2023-07-16T06:36:01.064941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}},{"cell_type":"markdown","source":"\n# 9 | Conclusion\n\n\nGeneral remarks:\n- This study demonstrates how to use an AI model and SLMHM/SPEM to evaluate maturity for comparing old and new texts;\n- Generally, the AI model demonstrates an ability to assess the texts for their maturity. Several improvements of the model and training dataset improve the results. Improvements are continuing;\n- Evaluation of all SEE issues is impossible for the current training dataset and this report's scope;\n- Generally, the method works. It is expected that a more complex training set and validation set will significantly improve the model.\n\nProgress analyses:\n- The study shows different results. For some aspects the texts showed decline, for others the progress was demonstrated;\n- The model can extract the most mature papers according to specific level. Examples for several levels were demonstrated.\n\nThe model is continuously improving in the following notebook: https://www.kaggle.com/code/dmitriygakh/see-slmhm-spem-maturity-tests-lstm-glove\n\n\nThe demonstrated methodology shows fascinating results in the qualitative ranking of the papers according to 17 levels.\n\n\n[↑↑↑ Table of Contents ↑↑↑](#toc)\n\n\n# 10 | References\n\n\n\n\n- [1] D. Gakh (2023). A Look to Model of Society and Teams Development Based on Initial Formation, Primary, Adaptable, Information, and Creative Society Patterns. International Journal of Management Research and Economics. 3(1), pp. 36-56. https://doi.org/10.51483/IJMRE.3.1.2023.36-56\n\n\n- [2] D. Gakh (2022). A Look at Evolution of Teams, Society, Smart Cities, and Information Systems based on Patterns of Primary, Adaptable, Information, and Creative Society, Proceedings of the 17th Conference on Computer Science and Intelligence Systems, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 30, pp. 701–704. http://dx.doi.org/10.15439/2022F34\n\n\n- [3] Gakh, D. (2023) Societal Patterns Evolution Model in Development of Economy, Society, and Environment. Preprints.org 2023, 2023050632. https://doi.org/10.20944/preprints202305.0632.v1\n\n\n- [4] Gakh, D. (2022). Education Development Strategy on Base of the Analysis of Messages in the Russian-Speaking Segment of the Internet using SLMHM Model. https://doi.org/10.35542/osf.io/cnh6q\n\n\n[↑↑↑ Table of Contents ↑↑↑](#toc)\n\n\n# 11 | Future Actions\n\n\nFuture actions should include:\n- Further upgrade the training data set;\n- Include Society-Economics-Environment issues;\n- Create a validation dataset and build a fitting (not overfitting) model.\n\n\n[↑↑↑ Table of Contents ↑↑↑](#toc)\n\n\n# 12 | Self-Assessment\n\n\nIn order to simplify evaluation of this notebook and to double-check its quality, self-assessment is included. Self-assessment is presented according to the SLMHM (see [anoncement](https://www.kaggle.com/discussions/general/409936)).\n\nAchievements and drawbacks:\n- 1.1. Task Ststement: Evaluate AI-related papers for innovative progress in two last years.\n- 1.2. Executants: The author performed the tasks himself.\n- 1.3. Feasibility: There was software and hardware available for the assignment. But training dataset is not developed enough. Parsing of additional data from internet sites is quite time consuming. As result only abstracts of ArXiv papers were analysed. More resources need to create a proper training dataset.\n- 1.4. Action: This notebook was developed as a prototype showing possibility to realise a more advanced solution. The training dataset was updated 3 times.\n- 2.1. Evaluation: The notebook contains several tests (see chapter [2.4 AI Model Tests](#2.4)) showing correctness and accuracy of methodology and code. Chapter [1.6 Evaluation Algorithm](#1.6) describes the evaluation algorithm.\n- 2.2. Improvement: Improvement steps are performed thanks to the version control system. Improvements include code and training dataset.\n- 2.3. Alternatives: The following alternatives were considered: notebooks of other kagglers, author's alternative notebooks (several notebooks did not give an interesting result and were not published), alternative data sources (web sites can provide a valuable data, but the scrapping is not easy enough). Training dataset should should be supplemented (now it contains about 650 rows, but estimations show that it should increase up to 5000 rows). There is one simple test text. 50 test texts will give a solid results.\n- 2.4. Innovations: This notebook demostrates use of new technique to evaluate papers by means of SLMHM/SPEM models. This approach has many advantages comparing to existing methods.\n- 3.1. Reward: This notebook benefits to kaggle community because demonstrate a new approach and scientific community because demonstrate a way of evaluation of scholarly papers. Unfortunately this notebook does not contribute the society, economy, and natural environment as it was planned before. But next versions of model could include it.\n\n\nFuture development:\n- 3.2. Optimization: Optimization can be implemented by involvement of specialists in the society, economy, and natural environment. Machine learning experts can help to build better AI model.\n- 3.3. Synergy: Synergy can be considered achieved if about 20 test text will shouw relevant results.\n- 3.4. Extraordinary: Implementation of the tool in production environment in a well known facility (library for example) can be considered as a condition to recognize the tool extraordinary.\n- 4.1. Information: Implementation of the tool in production environment in a way it generate new information can be considered as a condition to recognize the tool relating to the Information level.\n- 4.2. Streamlining: Implementation of the tool in production environment in a way it generate new knowledge can be considered as a condition to recognize the tool relating to the Streamlining level.\n- 4.3. Expansion: Packaging the solution as a product to be used as a library or unit in complex solutions in a way it become well know world-wide distributed application.\n- 4.4. Totality: Total penetration in all solutions.\n\n\n\n\n## 12 | Drawbacks\n\n\nThe drawback of the current version (July 3, 2023):\n- Training Database (data) contains some incorrect phrase-level relations (levels 3.1 and 3.2);\n- Training Database is slightly imbalanced and needs a supplement;\n- The current AI model is overfitted (due to a lack of validation data);\n- Lack of Society-Economy-Ecology relations;\n- The use of other AI model architectures (such as BERT) could improve the model's performance.\n\n\n[↑↑↑ Table of Contents ↑↑↑](#toc)\n\n\n","metadata":{}},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"submission = pd.DataFrame(\n columns=['type', 'value'],\n data = [\n ['essay_category', '\\'Other\\''], \n ['essay_url', '\\'https://www.kaggle.com/code/dmitriygakh/kaggle-ai-report-maturity-at-a-glance\\''],\n ['feedback1_url', '\\'http://www.kaggle.com/code/raimondextervinluan/2023-kaggle-ai-report-ai-ethics/comments#2325889\\''],\n ['feedback2_url', '\\'http://www.kaggle.com/code/luckypen/actionable-ai-ethics/comments#2325982\\''],\n ['feedback3_url', '\\'http://www.kaggle.com/code/mpwolke/ai-ethics-my-write-oops/comments#2325807\\'']\n ]\n)\n\nsubmission","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:36:01.067862Z","iopub.execute_input":"2023-07-16T06:36:01.068202Z","iopub.status.idle":"2023-07-16T06:36:01.082210Z","shell.execute_reply.started":"2023-07-16T06:36:01.068170Z","shell.execute_reply":"2023-07-16T06:36:01.080894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index = False)","metadata":{"execution":{"iopub.status.busy":"2023-07-16T06:36:01.083827Z","iopub.execute_input":"2023-07-16T06:36:01.084262Z","iopub.status.idle":"2023-07-16T06:36:01.100811Z","shell.execute_reply.started":"2023-07-16T06:36:01.084224Z","shell.execute_reply":"2023-07-16T06:36:01.100001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[↑↑↑ Table of Contents ↑↑↑](#toc)","metadata":{}}]}