{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Sources\n\n# 🟢 Usage Summary\ncontext\nIn this analytics competition, participants will write an essay on one of the following seven topics, with a prompt to describe what the community has learned over the past 2 years of working and experimenting with:\n* Text data\n* Image and/or video data\n* Tabular and/or time series \n* data\n* Kaggle Competitions\n* Generative AI \n* AI ethics\n* **Other** (anything that does not fall into any other category)\n\nKaggle has provided a collection of writeups and a collection of scholarly articles the official competition dataset, but participants are free to use any resources. Participants will also take part in a peer feedback process that involves reviewing three peer essays, and your submission will be assessed by an expert grading panel of seven Kaggle Grandmasters. Winning essays will be aggregated in a post-competition publication and given authorship credit.","metadata":{}},{"cell_type":"markdown","source":"# Kaggle Datasources\nAnalysis and comparative Data Sources in terms of factors. articles about AI are published every day is to leverage the diverse expertise of the Kaggle community to centralize and summarize the rapid progress in AI over the past two years. Respondents participating in the \"2023 Kaggle AI Report\" survey Machine Learning & Data Science Survey\"\nIn the competition data specified the file kaggle_writeups.csv - metadata from recent Kaggle Data Sources Writeups. Let's load it and see what it looks like.\n","metadata":{}},{"cell_type":"markdown","source":"# Kaggle Writeups load the data\nFirst, we look at the Kaggle Write Ups dataset to understand the Datasources for the past two years.","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns \npd.plotting.register_matplotlib_converters()\n%matplotlib inline  \nimport matplotlib.pyplot as plt\nimport time \nimport json\nfrom pathlib import Path\n#from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-07-08T14:04:35.366016Z","iopub.execute_input":"2023-07-08T14:04:35.366409Z","iopub.status.idle":"2023-07-08T14:04:35.381273Z","shell.execute_reply.started":"2023-07-08T14:04:35.366379Z","shell.execute_reply":"2023-07-08T14:04:35.379974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DATA_DIR = Path(\"/kaggle/input/2023-kaggle-ai-report/\")\nkwups_df = pd.read_csv(DATA_DIR / \"kaggle_writeups_20230510.csv\",parse_dates=[0, 3])\nsub = pd.read_csv(DATA_DIR / \"sample_submission.csv\")\nkwups_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-08T14:04:40.252851Z","iopub.execute_input":"2023-07-08T14:04:40.253956Z","iopub.status.idle":"2023-07-08T14:04:41.088373Z","shell.execute_reply.started":"2023-07-08T14:04:40.253921Z","shell.execute_reply":"2023-07-08T14:04:41.087048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Exploration ","metadata":{}},{"cell_type":"code","source":"kwups_df.shape","metadata":{"execution":{"iopub.status.busy":"2023-07-08T14:04:44.949692Z","iopub.execute_input":"2023-07-08T14:04:44.950125Z","iopub.status.idle":"2023-07-08T14:04:44.958192Z","shell.execute_reply.started":"2023-07-08T14:04:44.950094Z","shell.execute_reply":"2023-07-08T14:04:44.956821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kwups_df.info()","metadata":{"execution":{"iopub.status.busy":"2023-07-08T14:04:48.980270Z","iopub.execute_input":"2023-07-08T14:04:48.980689Z","iopub.status.idle":"2023-07-08T14:04:49.006486Z","shell.execute_reply.started":"2023-07-08T14:04:48.980656Z","shell.execute_reply":"2023-07-08T14:04:49.005378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualisation & Analysis","metadata":{}},{"cell_type":"code","source":"kwups_df.cumsum","metadata":{"execution":{"iopub.status.busy":"2023-07-08T14:04:52.923257Z","iopub.execute_input":"2023-07-08T14:04:52.923649Z","iopub.status.idle":"2023-07-08T14:04:52.940487Z","shell.execute_reply.started":"2023-07-08T14:04:52.923620Z","shell.execute_reply":"2023-07-08T14:04:52.939603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kwups_df = pd.DataFrame(kwups_df)\nkwups_df.plot();    ","metadata":{"execution":{"iopub.status.busy":"2023-07-08T14:04:59.434800Z","iopub.execute_input":"2023-07-08T14:04:59.435240Z","iopub.status.idle":"2023-07-08T14:04:59.914090Z","shell.execute_reply.started":"2023-07-08T14:04:59.435205Z","shell.execute_reply":"2023-07-08T14:04:59.912824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Select a specific competition\n","metadata":{}},{"cell_type":"code","source":"kwups_df = kwups_df[kwups_df['Title of Competition']=='M5 Forecasting - Accuracy'].reset_index()\nkwups_df ","metadata":{"execution":{"iopub.status.busy":"2023-07-08T14:05:05.073464Z","iopub.execute_input":"2023-07-08T14:05:05.074343Z","iopub.status.idle":"2023-07-08T14:05:05.102437Z","shell.execute_reply.started":"2023-07-08T14:05:05.074306Z","shell.execute_reply":"2023-07-08T14:05:05.101121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kwups_df['Title of Competition'].value_counts().head(5) ","metadata":{"execution":{"iopub.status.busy":"2023-07-08T14:05:12.458415Z","iopub.execute_input":"2023-07-08T14:05:12.459284Z","iopub.status.idle":"2023-07-08T14:05:12.468301Z","shell.execute_reply.started":"2023-07-08T14:05:12.459245Z","shell.execute_reply":"2023-07-08T14:05:12.466974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# compare total progress of previous and last titles ","metadata":{}},{"cell_type":"code","source":"# Conpare total maturity of prev and last titles\n# There is a disbalance between number of records in prev (1293) and last (703) sets\n# Part of code is taken from: https://www.kaggle.com/code/leonidkulyk/kaggle-ai-report-topic-selection\nkwups_df = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv\")\nkwups_df[\"Date of Writeup\"] = pd.to_datetime(kwups_df[\"Date of Writeup\"])\nkwups_fl_prev_df = kwups_df[kwups_df['Date of Writeup'].dt.year < 2020]\nkwups_fl_last_df = kwups_df[kwups_df['Date of Writeup'].dt.year >= 2022]\nprint(\"Records before 2020:\", len(kwups_fl_prev_df), \"and in 2022 and after:\", len(kwups_fl_last_df))","metadata":{"execution":{"iopub.status.busy":"2023-07-08T14:05:24.072628Z","iopub.execute_input":"2023-07-08T14:05:24.073077Z","iopub.status.idle":"2023-07-08T14:05:24.683287Z","shell.execute_reply.started":"2023-07-08T14:05:24.073046Z","shell.execute_reply":"2023-07-08T14:05:24.682175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Disbalance in unicue titles are greater\ntxts_prev = kwups_fl_prev_df[\"Title of Competition\"].unique()\ntxts_last = kwups_fl_last_df[\"Title of Competition\"].unique()\nprint(\"Titles before 2020:\", len(txts_prev), \"and in 2022 and after:\", len(txts_last))","metadata":{"execution":{"iopub.status.busy":"2023-07-08T14:05:40.128118Z","iopub.execute_input":"2023-07-08T14:05:40.128542Z","iopub.status.idle":"2023-07-08T14:05:40.135228Z","shell.execute_reply.started":"2023-07-08T14:05:40.128508Z","shell.execute_reply":"2023-07-08T14:05:40.134066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Ranking of \"best\" Scores, Place Solution","metadata":{}},{"cell_type":"code","source":"kwups_df['Title of Writeup'].value_counts()[0:50]","metadata":{"execution":{"iopub.status.busy":"2023-07-08T14:05:45.065887Z","iopub.execute_input":"2023-07-08T14:05:45.066289Z","iopub.status.idle":"2023-07-08T14:05:45.078234Z","shell.execute_reply.started":"2023-07-08T14:05:45.066259Z","shell.execute_reply":"2023-07-08T14:05:45.076858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nr1 = ['essay_category', 'Kaggle competitions', ]\nr2 = ['essay_url', 'https://www.kaggle.com/code/bacharacherif/recap-2023-kaggle-ai-report-eda/edit/run/136088477']\nr3 = ['feedback1_url', 'http://www.kaggle.com/.../your_1st_peer_feedback', ]\nr4 = ['feedback2_url', 'http://www.kaggle.com/.../your_2nd_peer_feedback', ]\nr5 = ['feedback3_url', 'http://www.kaggle.com/.../your_3rd_peer_feedback', ]\nsubmission = pd.DataFrame( [r1,r2,r3,r4,r5], columns=['type','value'])\nsubmission.to_csv('submission.csv', index=False)\nsubmission","metadata":{"execution":{"iopub.status.busy":"2023-07-08T14:08:02.009030Z","iopub.execute_input":"2023-07-08T14:08:02.009482Z","iopub.status.idle":"2023-07-08T14:08:02.031008Z","shell.execute_reply.started":"2023-07-08T14:08:02.009452Z","shell.execute_reply":"2023-07-08T14:08:02.029618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 💖 THANK YOU\nIf you liked this notebook, please consider upvoting. Once again, thank you for your support.","metadata":{}}]}