{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Summary of the Competition

","metadata":{}},{"cell_type":"markdown","source":"

🥅 The goal of the competition is to create a summary of the advancements of AI in the past 2 years, in one of the broad cathegories that Kaggle provides.

\n

🧑‍⚖️ The submissions will be graded by a panel of Kaggle grandmasters, while participants are urged to peer review at least 3 other teams to be eligible for grading and prizes.

\n

The proposed cathegories that Kaggle gives are:

\n \n","metadata":{}},{"cell_type":"markdown","source":"

Purpose of this Notebook

\n

The purpose of this notebook is to present some interesting subgroups to the broader groups given by Kaggle, so that both experienced and more novice users can read about niche topics that appeal to them, without having to go through all of the papers in each cathegory. This notebook is still a Work In Progress, so more topics will be added in the future.

\n\n

✏️ Comments/Suggestions greatly appreciated!

","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport json\nfrom datetime import datetime","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:54:57.864695Z","iopub.execute_input":"2023-05-13T12:54:57.865089Z","iopub.status.idle":"2023-05-13T12:54:57.870761Z","shell.execute_reply.started":"2023-05-13T12:54:57.865058Z","shell.execute_reply":"2023-05-13T12:54:57.869242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Basic preparation of the dataset","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv\")\ndf['Competition Launch Date'] = pd.to_datetime(df['Competition Launch Date'])\ndf['Date of Writeup'] = pd.to_datetime(df['Date of Writeup'])\ndf.drop(df[df['Date of Writeup'] < '2021-01-01'].index, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:54:58.330435Z","iopub.execute_input":"2023-05-13T12:54:58.330856Z","iopub.status.idle":"2023-05-13T12:54:59.087698Z","shell.execute_reply.started":"2023-05-13T12:54:58.330821Z","shell.execute_reply":"2023-05-13T12:54:59.086533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:54:59.089476Z","iopub.execute_input":"2023-05-13T12:54:59.089815Z","iopub.status.idle":"2023-05-13T12:54:59.115728Z","shell.execute_reply.started":"2023-05-13T12:54:59.089785Z","shell.execute_reply":"2023-05-13T12:54:59.114721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(df)","metadata":{"execution":{"iopub.status.busy":"2023-05-13T13:56:04.697644Z","iopub.execute_input":"2023-05-13T13:56:04.698600Z","iopub.status.idle":"2023-05-13T13:56:04.707037Z","shell.execute_reply.started":"2023-05-13T13:56:04.698560Z","shell.execute_reply":"2023-05-13T13:56:04.705336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Arxiv Articles","metadata":{}},{"cell_type":"markdown","source":"Following the approach of [@leonydkulik ](https://www.kaggle.com/leonidkulyk) in his notebook ( [Link to Notebook](https://www.kaggle.com/code/leonidkulyk/kaggle-ai-report-topic-selection), definitely give it a look, it is **amazing** ) we will be filtering the massive arxiv dataset firstly by date of publication, and after, by our different subtopics.","metadata":{}},{"cell_type":"code","source":"# import JSON data\ndict_arxiv = []\nfor line in open(\"/kaggle/input/2023-kaggle-ai-report/arxiv_metadata_20230510.json\", 'r'):\n dict_arxiv.append(json.loads(line))","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:55:13.948973Z","iopub.execute_input":"2023-05-13T12:55:13.949408Z","iopub.status.idle":"2023-05-13T12:56:56.747467Z","shell.execute_reply.started":"2023-05-13T12:55:13.949379Z","shell.execute_reply":"2023-05-13T12:56:56.746406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"date_format = \"%a, %d %b %Y %H:%M:%S %Z\"\nlimit_year = datetime.strptime(\"2021\", \"%Y\")\n\nfiltered_arxiv = []\nfor article_meta in dict_arxiv:\n parsed_date = datetime.strptime(\n article_meta[\"versions\"][0][\"created\"], \n date_format\n )\n if parsed_date >= limit_year:\n filtered_arxiv.append(article_meta)","metadata":{"execution":{"iopub.status.busy":"2023-05-13T12:56:56.749759Z","iopub.execute_input":"2023-05-13T12:56:56.750136Z","iopub.status.idle":"2023-05-13T12:57:30.299041Z","shell.execute_reply.started":"2023-05-13T12:56:56.750096Z","shell.execute_reply":"2023-05-13T12:57:30.298232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Subtopics","metadata":{}},{"cell_type":"markdown","source":"# 📈 Finance and Investing\n

We will first be filtering by the different Arxiv cathegories, the main cathegory will be Quantitative Finance, as well as of course, AI related cathegories.

","metadata":{}},{"cell_type":"code","source":"fincat = [\"q-fin.CP\", \"q-fin.EC\", \"q-fin.GN\", \"q-fin.MF\", \"q-fin.PM\", \"q-fin.PR\", \"q-fin.RM\", \"q-fin.ST\", \"q-fin.TR\"]\nfin_articles = []\naicat = [\"cs.AI\", \"cs.DS\", \"cs.LG\", \"cs.NE\", \"cs.RO\", \"cs.CL\"] \n\nfor article in filtered_arxiv:\n is_fin = 0\n is_ai = 0\n categories = article[\"categories\"].split(\" \")\n for cat in categories:\n if cat in fincat:\n is_fin = 1\n if cat in aicat:\n is_ai = 1\n if is_ai == 1 and is_fin == 1:\n fin_articles.append(article)\n break","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-05-13T13:22:31.433786Z","iopub.execute_input":"2023-05-13T13:22:31.434228Z","iopub.status.idle":"2023-05-13T13:22:32.214214Z","shell.execute_reply.started":"2023-05-13T13:22:31.434195Z","shell.execute_reply":"2023-05-13T13:22:32.213141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with open(\"arxiv_financial.json\", \"w\") as out_fin:\n json.dump(fin_articles, out_fin)","metadata":{"execution":{"iopub.status.busy":"2023-05-13T14:09:08.634832Z","iopub.execute_input":"2023-05-13T14:09:08.635343Z","iopub.status.idle":"2023-05-13T14:09:08.686911Z","shell.execute_reply.started":"2023-05-13T14:09:08.635309Z","shell.execute_reply":"2023-05-13T14:09:08.685580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Now we will filter the csv file provided.

","metadata":{}},{"cell_type":"code","source":"finance_dict = [\"finance\", \"return\", \"sp500\", \"s&p500\", \"invest\", \"portfolio\", \"market\", \"stock\"]\ntitles = []\nfor title in df['Title of Competition']:\n low_title = title.lower()\n for word in finance_dict:\n if word in low_title:\n titles.append(title)\n\ndf_finance = df[df['Title of Competition'].isin(titles)]\ndf_finance.sample(5)","metadata":{"execution":{"iopub.status.busy":"2023-05-13T14:03:43.393371Z","iopub.execute_input":"2023-05-13T14:03:43.393737Z","iopub.status.idle":"2023-05-13T14:03:43.412928Z","shell.execute_reply.started":"2023-05-13T14:03:43.393710Z","shell.execute_reply":"2023-05-13T14:03:43.412047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_finance.to_csv(\"finance_kaggle_writeups.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-05-13T14:03:45.946150Z","iopub.execute_input":"2023-05-13T14:03:45.946541Z","iopub.status.idle":"2023-05-13T14:03:45.957458Z","shell.execute_reply.started":"2023-05-13T14:03:45.946500Z","shell.execute_reply":"2023-05-13T14:03:45.955620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 📋 Output Files\n

The output of these cells are the following files:

\n \n

Feel free to use them in your studies if you feel they might be helpful.

\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}