{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Published on May 13, 2023. By Marília Prata, mpwolke.","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n#Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-05-13T21:27:37.159611Z","iopub.execute_input":"2023-05-13T21:27:37.160311Z","iopub.status.idle":"2023-05-13T21:27:38.766032Z","shell.execute_reply.started":"2023-05-13T21:27:37.160273Z","shell.execute_reply":"2023-05-13T21:27:38.764813Z"},"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#IA Générative et ma NS (Naturelle, Authentique Stupidité)\n\nLe terme anglais de Generative AI, ou IA Générative en français désigne des algorithmes d'Intelligence Artificielle et de Machine Learning qui utilisent des contenus existants au service de leur apprentissage pour en générer de nouveaux.","metadata":{}},{"cell_type":"markdown","source":"#Flatenning Json ","metadata":{}},{"cell_type":"code","source":"#https://stackoverflow.com/questions/12451431/loading-and-parsing-a-json-file-with-multiple-json-objects\n\nimport pandas as pd\nmeta = pd.read_json('../input/2023-kaggle-ai-report/arxiv_metadata_20230510.json', lines=True)","metadata":{"execution":{"iopub.status.busy":"2023-05-13T21:28:25.663781Z","iopub.execute_input":"2023-05-13T21:28:25.664211Z","iopub.status.idle":"2023-05-13T21:31:18.153414Z","shell.execute_reply.started":"2023-05-13T21:28:25.664175Z","shell.execute_reply":"2023-05-13T21:31:18.151855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#https://stackoverflow.com/questions/12451431/loading-and-parsing-a-json-file-with-multiple-json-objects\n\nmeta.to_json('new_file.json')","metadata":{"execution":{"iopub.status.busy":"2023-05-13T21:31:52.547905Z","iopub.execute_input":"2023-05-13T21:31:52.548548Z","iopub.status.idle":"2023-05-13T21:32:42.676838Z","shell.execute_reply.started":"2023-05-13T21:31:52.548505Z","shell.execute_reply":"2023-05-13T21:32:42.675322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Meta: arxiv_metadata","metadata":{}},{"cell_type":"code","source":"meta.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-13T21:33:08.312187Z","iopub.execute_input":"2023-05-13T21:33:08.312605Z","iopub.status.idle":"2023-05-13T21:33:08.360165Z","shell.execute_reply.started":"2023-05-13T21:33:08.312572Z","shell.execute_reply":"2023-05-13T21:33:08.358789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Search for specific keyword in abstracts","metadata":{}},{"cell_type":"code","source":"#Chris X https://www.kaggle.com/code/docxian/cord-19-dexamethasone-articles\n\n# define keyword\nmy_keyword = 'generative AI'","metadata":{"execution":{"iopub.status.busy":"2023-05-13T21:38:04.509177Z","iopub.execute_input":"2023-05-13T21:38:04.509550Z","iopub.status.idle":"2023-05-13T21:38:04.514656Z","shell.execute_reply.started":"2023-05-13T21:38:04.509521Z","shell.execute_reply":"2023-05-13T21:38:04.513519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Chris X https://www.kaggle.com/code/docxian/cord-19-dexamethasone-articles\n\ndef word_finder(i_word, i_text):\n    found = (str(i_text).lower()).find(str(i_word).lower()) # avoid case sensitivity\n    if found == -1:\n        result = 0\n    else:\n        result = 1\n    return result\n\n# partial function for mapping\nword_indicator_partial = lambda text: word_finder(my_keyword, text)\n# build indicator vector (0/1) of hits\nkeyword_indicator = np.asarray(list(map(word_indicator_partial, meta.abstract)))","metadata":{"execution":{"iopub.status.busy":"2023-05-13T21:38:10.272195Z","iopub.execute_input":"2023-05-13T21:38:10.272621Z","iopub.status.idle":"2023-05-13T21:38:17.400585Z","shell.execute_reply.started":"2023-05-13T21:38:10.272586Z","shell.execute_reply":"2023-05-13T21:38:17.399383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# number of hits\nprint('Number of hits for keyword <', my_keyword, '> : ', keyword_indicator.sum())","metadata":{"execution":{"iopub.status.busy":"2023-05-13T21:38:22.266126Z","iopub.execute_input":"2023-05-13T21:38:22.266542Z","iopub.status.idle":"2023-05-13T21:38:22.274517Z","shell.execute_reply.started":"2023-05-13T21:38:22.266507Z","shell.execute_reply":"2023-05-13T21:38:22.272925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Chris X https://www.kaggle.com/code/docxian/cord-19-dexamethasone-articles\n\n# add index vector as additional column\nmeta['selection'] = keyword_indicator\n\n# select only hits from data frame\nmeta_hits = meta[meta['selection']==1]","metadata":{"execution":{"iopub.status.busy":"2023-05-13T21:39:50.120880Z","iopub.execute_input":"2023-05-13T21:39:50.121307Z","iopub.status.idle":"2023-05-13T21:39:50.140282Z","shell.execute_reply.started":"2023-05-13T21:39:50.121270Z","shell.execute_reply":"2023-05-13T21:39:50.139044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Les Voilá: IA Génératives \n\nLes algorithmes d'Intelligence Artificielle et de Machine Learning qui utilisent des contenus existants au service de leur apprentissage pour en générer de nouveaux.","metadata":{}},{"cell_type":"code","source":"# show results\nmeta_hits","metadata":{"execution":{"iopub.status.busy":"2023-05-13T21:40:10.000710Z","iopub.execute_input":"2023-05-13T21:40:10.001151Z","iopub.status.idle":"2023-05-13T21:40:10.069227Z","shell.execute_reply.started":"2023-05-13T21:40:10.001115Z","shell.execute_reply":"2023-05-13T21:40:10.068391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator","metadata":{"execution":{"iopub.status.busy":"2023-05-13T21:51:16.239807Z","iopub.execute_input":"2023-05-13T21:51:16.241571Z","iopub.status.idle":"2023-05-13T21:51:16.302735Z","shell.execute_reply.started":"2023-05-13T21:51:16.241481Z","shell.execute_reply":"2023-05-13T21:51:16.301566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"text = \" \".join(abst for abst in meta_hits.abstract)","metadata":{"execution":{"iopub.status.busy":"2023-05-13T21:51:30.676708Z","iopub.execute_input":"2023-05-13T21:51:30.677183Z","iopub.status.idle":"2023-05-13T21:51:30.685772Z","shell.execute_reply.started":"2023-05-13T21:51:30.677144Z","shell.execute_reply":"2023-05-13T21:51:30.684254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stopwords = set(STOPWORDS)","metadata":{"execution":{"iopub.status.busy":"2023-05-13T21:51:50.641705Z","iopub.execute_input":"2023-05-13T21:51:50.642241Z","iopub.status.idle":"2023-05-13T21:51:50.647816Z","shell.execute_reply.started":"2023-05-13T21:51:50.642201Z","shell.execute_reply":"2023-05-13T21:51:50.646566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#ChatGPT is here. But, Where's The Bard on the WordCloud? \n\nI found that single one on Meta json file:\n\nShould ChatGPT and Bard Share Revenue with Their Data Providers? A New\n  Business Model for the AI Era :\n  \nAuthor: Dong Zhang -  \nhttps://doi.org/10.48550/arXiv.2305.02555\n \n\n  \"With various AI tools such as ChatGPT becoming increasingly popular, we are\nentering a true AI era. We can foresee that exceptional AI tools will soon reap\nconsiderable profits. A crucial question arise: should AI tools share revenue\nwith their training data providers in additional to traditional stakeholders\nand shareholders? The answer is Yes. Large AI tools, such as large language\nmodels, always require more and better quality data to continuously improve,\nbut current copyright laws limit their access to various types of data. Sharing\nrevenue between AI tools and their data providers could transform the current\nhostile zero-sum game relationship between AI tools and a majority of\ncopyrighted data owners into a collaborative and mutually beneficial one, which\nis necessary to facilitate the development of a virtuous cycle among AI tools,\ntheir users and data providers that drives forward AI technology and builds a\nhealthy AI ecosystem. However, current revenue-sharing business models do not\nwork for AI tools in the forthcoming AI era, since the most widely used metrics\nfor website-based traffic and action, such as clicks, will be replaced by new\nmetrics such as prompts and cost per prompt for generative AI tools. A\ncompletely new revenue-sharing business model, which must be almost independent\nof AI tools and be easily explained to data providers, needs to establish a\nprompt-based scoring system to measure data engagement of each data provider.\nThis paper systematically discusses how to build such a scoring system for all\ndata providers for AI tools based on classification and content similarity\nmodels, and outlines the requirements for AI tools or third parties to build\nit. Sharing revenue with data providers using such a scoring system would\nencourage more data owners to participate in the revenue-sharing program. This\nwill be a utilitarian AI era where all parties benefit.\"\n\nhttps://arxiv.org/abs/2305.02555#:~:text=A%20crucial%20question%20arise%3A%20should,The%20answer%20is%20Yes.","metadata":{}},{"cell_type":"code","source":"#Chris X https://www.kaggle.com/code/docxian/cord-19-dexamethasone-articles\n\nwordcloud = WordCloud(stopwords=stopwords, max_font_size=50, max_words=500,\n                      width = 600, height = 400,\n                      colormap='Set3',\n                      background_color=\"black\").generate(text)\nplt.figure(figsize=(12,8))\nplt.imshow(wordcloud, interpolation=\"bilinear\")\nplt.axis(\"off\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-13T21:55:18.615433Z","iopub.execute_input":"2023-05-13T21:55:18.615941Z","iopub.status.idle":"2023-05-13T21:55:20.484836Z","shell.execute_reply.started":"2023-05-13T21:55:18.615901Z","shell.execute_reply":"2023-05-13T21:55:20.483079Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Show all Abstracts and save to file\n\nI think it's only 82.","metadata":{}},{"cell_type":"markdown","source":"#My favorite Titles on Ouput 82\n\nShould ChatGPT and Bard Share Revenue with Their Data Providers? A New\n  Business Model for the AI Era \n\nThe AI generation gap: Are Gen Z students more interested in adopting\n  generative AI such as ChatGPT in teaching and learning than their Gen X and\n  Millennial Generation teachers?\n  \nChatGPT is all you need to decolonize sub-Saharan Vocational Education  \n\nCausal Semantic Communication for Digital Twins: A Generalizable\n  Imitation Learning Approach \n  \nCan ChatGPT be used to generate scientific hypotheses?\n\nEKILA: Synthetic Media Provenance and Attribution for Generative Art \n\nA Word is Worth a Thousand Pictures: Prompts as AI Design Material\n\nCombining Generative Artificial Intelligence (AI) and the Internet:\n  Heading towards Evolution or Degradation?\n  ","metadata":{}},{"cell_type":"code","source":"#Chris X https://www.kaggle.com/code/docxian/cord-19-dexamethasone-articles\n\n# show all abstracts\nn = meta_hits.shape[0]\nfor i in range(0,n):\n    print(meta_hits.title.iloc[i],\":\\n\")\n    print(meta_hits.abstract.iloc[i])\n    print('\\n')","metadata":{"execution":{"iopub.status.busy":"2023-05-13T22:00:20.875337Z","iopub.execute_input":"2023-05-13T22:00:20.875838Z","iopub.status.idle":"2023-05-13T22:00:20.895765Z","shell.execute_reply.started":"2023-05-13T22:00:20.875805Z","shell.execute_reply":"2023-05-13T22:00:20.893866Z"},"_kg_hide-output":true,"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Above we have 82(?) Abstracts. Click Expand to read them all.","metadata":{}},{"cell_type":"code","source":"# make available for download\nmeta_hits.to_csv('hits.csv')","metadata":{"execution":{"iopub.status.busy":"2023-05-13T22:06:02.262551Z","iopub.execute_input":"2023-05-13T22:06:02.263091Z","iopub.status.idle":"2023-05-13T22:06:02.295824Z","shell.execute_reply.started":"2023-05-13T22:06:02.263053Z","shell.execute_reply":"2023-05-13T22:06:02.294250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Acknowledgements\n\nChris X https://www.kaggle.com/code/docxian/cord-19-dexamethasone-articles","metadata":{}}]}