{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nMore than a hundred AI papers are published every day, making it exceedingly hard to keep up with current innovations. The goal of this competition is to tap into the diverse expertise of the Kaggle community to centralize and summarize the rapid advancements in AI from the past two years. The Kaggle community has a breadth and depth of AI experience which extends beyond the reach of any single individual or research group. We aim to share your collective perspective with the broader research community.\n\nWe have to write an essay on one of the following seven topics, with a prompt to describe what the community has learned over the past 2 years of working and experimenting with:\n\n    Text data\n    Image and/or video data\n    Tabular and/or time series data\n    Kaggle Competitions\n    Generative AI\n    AI ethics\n    Other (anything that does not fall into any other category)\n\nFrom the Kaggle data collection, use any resources and also take part in a peer feedback process that involves reviewing three peer essays. Foreference [notebook](https://www.kaggle.com/code/docxian/kaggle-ai-report-access-to-data). Please upvote reference book also.","metadata":{}},{"cell_type":"code","source":"# Basic Imports\nimport os\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Import plotting\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Import Word cloud\nfrom wordcloud import WordCloud, STOPWORDS, ImageColorGenerator\n\n#Import Json Lib\nimport json","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:54:11.088624Z","iopub.execute_input":"2023-05-12T10:54:11.089661Z","iopub.status.idle":"2023-05-12T10:54:11.095275Z","shell.execute_reply.started":"2023-05-12T10:54:11.089626Z","shell.execute_reply":"2023-05-12T10:54:11.094177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Enginnering\nLoad data from both of the Json file, arxiv and kaggle writeups. Explore it, manage it, modify it ..!! ","metadata":{}},{"cell_type":"code","source":"for dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-05-12T10:54:16.541474Z","iopub.execute_input":"2023-05-12T10:54:16.541882Z","iopub.status.idle":"2023-05-12T10:54:16.548648Z","shell.execute_reply.started":"2023-05-12T10:54:16.541854Z","shell.execute_reply":"2023-05-12T10:54:16.547671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Arxiv Metadata","metadata":{}},{"cell_type":"code","source":"# import arxiv meta data from JSON file \narxiv_json_file = '../input/2023-kaggle-ai-report/arxiv_metadata_20230510.json'\n\ndict_arxiv = []\nfor line in open(arxiv_json_file, 'r'):\n    dict_arxiv.append(json.loads(line))\n\n# convert to data frame\ndf_arxiv_metadata = pd.DataFrame.from_dict(dict_arxiv)\n# clean up\ndel dict_arxiv","metadata":{"execution":{"iopub.status.busy":"2023-05-12T11:00:27.293695Z","iopub.execute_input":"2023-05-12T11:00:27.294133Z","iopub.status.idle":"2023-05-12T11:02:32.717639Z","shell.execute_reply.started":"2023-05-12T11:00:27.294100Z","shell.execute_reply":"2023-05-12T11:02:32.716669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Arxiv metadata detail info \ndf_arxiv_metadata.info()","metadata":{"execution":{"iopub.status.busy":"2023-05-12T11:03:12.609489Z","iopub.execute_input":"2023-05-12T11:03:12.609953Z","iopub.status.idle":"2023-05-12T11:03:12.636249Z","shell.execute_reply.started":"2023-05-12T11:03:12.609919Z","shell.execute_reply":"2023-05-12T11:03:12.634864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Overview of Arxiv metadata\ndf_arxiv_metadata.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-12T11:03:40.201415Z","iopub.execute_input":"2023-05-12T11:03:40.201791Z","iopub.status.idle":"2023-05-12T11:03:40.244869Z","shell.execute_reply.started":"2023-05-12T11:03:40.201761Z","shell.execute_reply":"2023-05-12T11:03:40.243709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Arxiv Metadata Word Cloud\ncategory_text_arxiv = \" \".join(xx for xx in  df_arxiv_metadata[df_arxiv_metadata.categories=='cs.AI'].reset_index().abstract)\n# Stop Word for cloud\nstopwords_cloud = set(STOPWORDS)\n# Plot word cloud\nwordcloud = WordCloud(stopwords=stopwords_cloud, max_font_size=50, max_words=250,\n                      width = 600, height = 400,\n                      background_color='black').generate(category_text_arxiv)\n\nplt.figure(figsize=(12,8))\nplt.imshow(wordcloud, interpolation='bilinear')\nplt.axis('off')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-12T11:27:51.298474Z","iopub.execute_input":"2023-05-12T11:27:51.298872Z","iopub.status.idle":"2023-05-12T11:27:58.527279Z","shell.execute_reply.started":"2023-05-12T11:27:51.298844Z","shell.execute_reply":"2023-05-12T11:27:58.526481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Kaggle Writeups","metadata":{}},{"cell_type":"code","source":"# import Kaggle Writeup data from files.\ndf_kaggle_writeup = pd.read_csv('../input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv')\n# Get detailed info\ndf_kaggle_writeup.info()","metadata":{"execution":{"iopub.status.busy":"2023-05-12T11:08:32.811214Z","iopub.execute_input":"2023-05-12T11:08:32.811768Z","iopub.status.idle":"2023-05-12T11:08:33.236729Z","shell.execute_reply.started":"2023-05-12T11:08:32.811723Z","shell.execute_reply":"2023-05-12T11:08:33.235397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Overview of Kaggle Writeup data\ndf_kaggle_writeup.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-12T11:09:23.694300Z","iopub.execute_input":"2023-05-12T11:09:23.695075Z","iopub.status.idle":"2023-05-12T11:09:23.714126Z","shell.execute_reply.started":"2023-05-12T11:09:23.695030Z","shell.execute_reply":"2023-05-12T11:09:23.712689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Kaggle Writeup - Title Of comptetion\ndf_kaggle_writeup['Title of Competition'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-05-12T11:23:19.981284Z","iopub.execute_input":"2023-05-12T11:23:19.981720Z","iopub.status.idle":"2023-05-12T11:23:19.992519Z","shell.execute_reply.started":"2023-05-12T11:23:19.981687Z","shell.execute_reply":"2023-05-12T11:23:19.991465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Kaggle Writeup Word Cloud\ncategory_text_kaggle = \" \".join(xx for xx in  df_kaggle_writeup[df_kaggle_writeup['Title of Competition']=='Feedback Prize - English Language Learning'].reset_index()['Title of Writeup'])\n# Stop Word for cloud\nstopwords_cloud = set(STOPWORDS)\n# Plot word cloud\nwordcloud = WordCloud(stopwords=stopwords_cloud, max_font_size=50, max_words=250,\n                      width = 600, height = 400,\n                      background_color='black').generate(category_text_kaggle)\n\nplt.figure(figsize=(12,8))\nplt.imshow(wordcloud, interpolation='bilinear')\nplt.axis('off')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-12T11:29:47.641093Z","iopub.execute_input":"2023-05-12T11:29:47.641519Z","iopub.status.idle":"2023-05-12T11:29:48.084430Z","shell.execute_reply.started":"2023-05-12T11:29:47.641486Z","shell.execute_reply":"2023-05-12T11:29:48.083331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}