{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 2023 Kaggle AI Report: Essays on the state of machine learning in 2023\n\n[//]: # (Title: Kaggle AI Report: Essays on the state of machine learning in 2023)\n[//]: # (Author: Jonas Campos)\n[//]: # (Date: Jun 14, 2023)\n[//]: # (Comment: 2023 Kaggle AI Report) \n[//]: # (Tags: #python, #IA, #Ethics, #Zeitgeist)  \n\n[![](https://img.shields.io/badge/feito%20com%20%E2%9D%A4%20por-jaac-cyan)](https://github.com/jonasaacampos)\n[![LinkedIn Badge](https://img.shields.io/badge/LinkedIn-Profile-informational?style=flat&logo=linkedin&logoColor=white&color=0D76A8)](https://www.linkedin.com/in/jonasaacampos)\n\nSeparei com esta análise os artigos que consideram ao menos citar algo referente à ética. Embora os dados destes mais de 2 milhões de artigos considerados nesta análise sejam desanimadores, revelo-me decepcionado, porém não surpreso de que a Inteligência Artificial não será a causa final do crepúsculo da humanidade, mas sim obra de algum grupo de seres humanos que acham feio o que não é espelho.\n\n> Competição link: https://www.kaggle.com/competitions/2023-kaggle-ai-report\n\n![](https://i.imgur.com/TcQP6RP.jpg?1)","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-06-14T12:17:29.895850Z","iopub.execute_input":"2023-06-14T12:17:29.896311Z","iopub.status.idle":"2023-06-14T12:17:29.912539Z","shell.execute_reply.started":"2023-06-14T12:17:29.896279Z","shell.execute_reply":"2023-06-14T12:17:29.911303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploratory Analysis\n\nLet's have an overview of the data\n\n**Análisis exploratorio**\n\nVamos tener una visión general de los datos\n\n**Análise exploratória**\n\nVamos ter uma visão geral dos dados.\n","metadata":{}},{"cell_type":"code","source":"file = '/kaggle/input/2023-kaggle-ai-report/arxiv_metadata_20230510.json'\ndf = pd.read_json(file, lines=True)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:17:29.914482Z","iopub.execute_input":"2023-06-14T12:17:29.915837Z","iopub.status.idle":"2023-06-14T12:20:30.884877Z","shell.execute_reply.started":"2023-06-14T12:17:29.915781Z","shell.execute_reply":"2023-06-14T12:20:30.883605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:20:30.911667Z","iopub.execute_input":"2023-06-14T12:20:30.912155Z","iopub.status.idle":"2023-06-14T12:20:31.004446Z","shell.execute_reply.started":"2023-06-14T12:20:30.912122Z","shell.execute_reply":"2023-06-14T12:20:31.003243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def verifica_biblioteca(nome_lib):\n    import importlib\n    \n    try:\n        importlib.import_module(nome_lib)\n        #print(f'Biblioteca {nome_lib} pronta para uso. Fazendo importação')\n    except ImportError:\n        #print(f'Biblioteca {nome_lib} não existe no projeto. Aguarde a instalação...')\n        import subprocess\n        subprocess.check_call(['pip', 'install', nome_lib])\n    finally:\n        globals()[nome_lib] = importlib.import_module(nome_lib)\n        print('\\n')\n        return True\n        \n           \n","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:20:31.005978Z","iopub.execute_input":"2023-06-14T12:20:31.006362Z","iopub.status.idle":"2023-06-14T12:20:31.013780Z","shell.execute_reply.started":"2023-06-14T12:20:31.006333Z","shell.execute_reply":"2023-06-14T12:20:31.012502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def total_linhas_df_por_extenso(df):\n    \n    if verifica_biblioteca('num2words'):\n        from num2words import num2words\n        total_linhas = num2words(df.shape[0], lang='pt-br')\n        print(f'total_linhas: {total_linhas}')\n    else:\n        print('Erro ao gerar a função.')\n\n    \n#!pip install num2words\n#from num2words import num2words\n","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:20:31.016956Z","iopub.execute_input":"2023-06-14T12:20:31.017439Z","iopub.status.idle":"2023-06-14T12:20:31.030521Z","shell.execute_reply.started":"2023-06-14T12:20:31.017408Z","shell.execute_reply":"2023-06-14T12:20:31.029226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"total_linhas_df_por_extenso(df)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:20:31.032594Z","iopub.execute_input":"2023-06-14T12:20:31.033046Z","iopub.status.idle":"2023-06-14T12:20:45.270618Z","shell.execute_reply.started":"2023-06-14T12:20:31.033006Z","shell.execute_reply":"2023-06-14T12:20:45.269316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Separating Data Sample for Exploratory Analysis\n\nSince this dataset has more than 2 million records, let's separate a sample so that we can analyze it (and avoid frying my computer. The mother earth thanks you)\n\n**Separando muestra de datos para análisis exploratorio**\n\nDado que este conjunto de datos tiene más de 2 millones de registros, vamos a separar una muestra para poder analizarla (y evitar que mi computadora se sobrecaliente. El medio ambiente te lo agradece)\n\n**Separando amostra de dados para análise exploratória**\n\nComo este conjunto de dados possui mais de 2 milhões de registros, vamos separar uma amostra para que possamos analisar. (e para que não fritar meu computador. O meio ambiente agradece)","metadata":{}},{"cell_type":"code","source":"tamanho_amostra = 0.3 #30%\nseed = 21\ndf_amostra = df.sample(frac=tamanho_amostra, random_state=seed)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:20:45.271913Z","iopub.execute_input":"2023-06-14T12:20:45.272248Z","iopub.status.idle":"2023-06-14T12:20:47.163048Z","shell.execute_reply.started":"2023-06-14T12:20:45.272219Z","shell.execute_reply":"2023-06-14T12:20:47.162002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"total_linhas_df_por_extenso(df_amostra)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:20:47.164271Z","iopub.execute_input":"2023-06-14T12:20:47.164626Z","iopub.status.idle":"2023-06-14T12:20:47.171305Z","shell.execute_reply.started":"2023-06-14T12:20:47.164596Z","shell.execute_reply":"2023-06-14T12:20:47.170022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> With 600,000 rows, it seems more reasonable to work with our data. Let's go!\n> \n> Con 600,000 filas, parece más razonable trabajar con nuestros datos. ¡Vamos allá!\n> \n> Com 600 mil linhas parece ser mais razoável trabalhar com nossos dados. Let's go!","metadata":{}},{"cell_type":"code","source":"df_amostra.columns","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:20:47.172794Z","iopub.execute_input":"2023-06-14T12:20:47.173184Z","iopub.status.idle":"2023-06-14T12:20:47.185918Z","shell.execute_reply.started":"2023-06-14T12:20:47.173151Z","shell.execute_reply":"2023-06-14T12:20:47.184560Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_amostra['abstract']","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:20:47.187547Z","iopub.execute_input":"2023-06-14T12:20:47.188020Z","iopub.status.idle":"2023-06-14T12:20:47.202011Z","shell.execute_reply.started":"2023-06-14T12:20:47.187983Z","shell.execute_reply":"2023-06-14T12:20:47.200774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"amostra_abstract = str(df_amostra.iloc[100, 10])\namostra_abstract","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:20:47.208582Z","iopub.execute_input":"2023-06-14T12:20:47.209037Z","iopub.status.idle":"2023-06-14T12:20:47.218294Z","shell.execute_reply.started":"2023-06-14T12:20:47.208994Z","shell.execute_reply":"2023-06-14T12:20:47.217235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Separating Articles Related to Ethics\n\nEthics in AI is a serious problem. So is ethics among humans. But are we as researchers paying attention to this or are we just interested in lining the pockets of a few individuals, determining who really matters on a scale of life value? I believe that as humanity, we are standing on the edge of an abyss. And the machines we have helped create will only amplify the harm we have created, on a larger scale and with greater competence than we could ever achieve.\n\n**Separando los artículos relacionados con la ética**\n\nLa ética en la inteligencia artificial es un problema serio. Lo mismo ocurre con la ética entre los seres humanos. ¿Pero estamos nosotros, como investigadores, prestando atención a esto o simplemente estamos interesados en llenar los bolsillos de unas pocas personas, determinando quién realmente importa en una escala de valor de vida? Creo que como humanidad, nos encontramos al borde de un abismo. Y las máquinas que hemos ayudado a crear solo amplificarán el daño que hemos causado, en una escala más grande y con una mayor competencia de la que podríamos lograr nosotros mismos.\n\n**Separando os artigos com temas relacionados à Ética**\n\nÉtica em IA é um problema sério. Assim entre os humanos. Mas será que nós equando pesquisadores estamos atentos à isso ou queremos é encher os bolsos de algumas pessoas, determinando quem de fato importa em uma escala de valor de vida? Acredito que enquanto humanidade, estamos diante do abismo. E as máquinas que ajudamos a criar só farão o mal que criamos em uma escala e com uma competência melhor do que nós faríamos.\n\n![](https://th.bing.com/th/id/OIG.sbQIlbFOfUD3LXwXl69y?pid=ImgGn)","metadata":{}},{"cell_type":"markdown","source":"## Separation of Articles\n\nLet's separate the articles that are relevant to our topic.\n\n**Separación de artículos**\n\nVamos a separar los artículos que son relevantes para nuestro tema.\n\n**Separação dos artigos**\n\nVamos separar os artigos que são pertinentes ao nosso tema","metadata":{}},{"cell_type":"code","source":"import re","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:20:47.220282Z","iopub.execute_input":"2023-06-14T12:20:47.220716Z","iopub.status.idle":"2023-06-14T12:20:47.228403Z","shell.execute_reply.started":"2023-06-14T12:20:47.220679Z","shell.execute_reply":"2023-06-14T12:20:47.227188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"termos = ['ethical', \n          'ethics', \n          'morality', \n          'moral', \n          'ethicality', \n          'ethical behavior', \n          'ethical principles', \n          'ethical standards', \n          'ethical dilemma', \n          'ethical decision-making', \n          'ethical considerations', \n          'ethical framework', \n          'ethical guidelines', \n          'ethical values', \n          'ethical code', \n          'ethical responsibility', \n          'ethical conduct']\n\npattern = r'\\b(' + '|'.join(termos) + r')\\b'","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:20:47.230139Z","iopub.execute_input":"2023-06-14T12:20:47.231412Z","iopub.status.idle":"2023-06-14T12:20:47.242066Z","shell.execute_reply.started":"2023-06-14T12:20:47.231359Z","shell.execute_reply":"2023-06-14T12:20:47.240911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_amostra['termos'] = df_amostra['abstract'].str.contains(pattern, flags=re.IGNORECASE)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:20:47.243724Z","iopub.execute_input":"2023-06-14T12:20:47.244905Z","iopub.status.idle":"2023-06-14T12:22:02.646972Z","shell.execute_reply.started":"2023-06-14T12:20:47.244865Z","shell.execute_reply":"2023-06-14T12:22:02.645933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_amostra['termos'].sum()","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:22:02.648352Z","iopub.execute_input":"2023-06-14T12:22:02.648701Z","iopub.status.idle":"2023-06-14T12:22:02.657858Z","shell.execute_reply.started":"2023-06-14T12:22:02.648672Z","shell.execute_reply":"2023-06-14T12:22:02.656816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_amostra['termos_quantidade'] = df_amostra['abstract'].str.count(pattern, flags=re.IGNORECASE)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:22:02.659130Z","iopub.execute_input":"2023-06-14T12:22:02.659515Z","iopub.status.idle":"2023-06-14T12:23:18.306772Z","shell.execute_reply.started":"2023-06-14T12:22:02.659486Z","shell.execute_reply":"2023-06-14T12:23:18.305807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_amostra['termos_quantidade'].sum()","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:23:18.308163Z","iopub.execute_input":"2023-06-14T12:23:18.308724Z","iopub.status.idle":"2023-06-14T12:23:18.315966Z","shell.execute_reply.started":"2023-06-14T12:23:18.308692Z","shell.execute_reply":"2023-06-14T12:23:18.314697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_amostra.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:23:18.317568Z","iopub.execute_input":"2023-06-14T12:23:18.317869Z","iopub.status.idle":"2023-06-14T12:23:18.358754Z","shell.execute_reply.started":"2023-06-14T12:23:18.317843Z","shell.execute_reply":"2023-06-14T12:23:18.357544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualizations","metadata":{"execution":{"iopub.status.busy":"2023-05-25T13:39:29.330419Z","iopub.execute_input":"2023-05-25T13:39:29.330874Z","iopub.status.idle":"2023-05-25T13:39:29.336615Z","shell.execute_reply.started":"2023-05-25T13:39:29.330835Z","shell.execute_reply":"2023-05-25T13:39:29.335057Z"}}},{"cell_type":"code","source":"from wordcloud import WordCloud\nimport matplotlib.pyplot as plt\nimport seaborn as sns","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:23:18.360534Z","iopub.execute_input":"2023-06-14T12:23:18.361732Z","iopub.status.idle":"2023-06-14T12:23:19.413742Z","shell.execute_reply.started":"2023-06-14T12:23:18.361687Z","shell.execute_reply":"2023-06-14T12:23:19.412694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def criar_wordcloud(string_unica):\n\n    wordcloud = WordCloud(collocations = False, background_color = 'white',\n                            width = 2048, height = 1080).generate(string_unica)\n    plt.figure(figsize=(10,8))\n    plt.imshow(wordcloud, interpolation='bilinear')\n    plt.axis('off')\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:23:19.414971Z","iopub.execute_input":"2023-06-14T12:23:19.415969Z","iopub.status.idle":"2023-06-14T12:23:19.422613Z","shell.execute_reply.started":"2023-06-14T12:23:19.415932Z","shell.execute_reply":"2023-06-14T12:23:19.421497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# palavras mais comuns em todos os artigos da amostra\ntext = ' '.join(df_amostra['abstract'])\ncriar_wordcloud(text)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:23:19.424818Z","iopub.execute_input":"2023-06-14T12:23:19.425217Z","iopub.status.idle":"2023-06-14T12:26:50.975329Z","shell.execute_reply.started":"2023-06-14T12:23:19.425187Z","shell.execute_reply":"2023-06-14T12:26:50.974485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# palavras mais comuns nos artigos que possuem algum de nossos termos chave\ntext2 = ' '.join(df_amostra.loc[df_amostra['termos'], 'abstract'])\ncriar_wordcloud(text2)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:50.976987Z","iopub.execute_input":"2023-06-14T12:26:50.977721Z","iopub.status.idle":"2023-06-14T12:26:56.014525Z","shell.execute_reply.started":"2023-06-14T12:26:50.977689Z","shell.execute_reply":"2023-06-14T12:26:56.011625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Saving File\nI will save the data of all the articles that are relevant to our key terms.\n\n**Guardando archivo**\n\nVoy a guardar los datos de todos los artículos que son relevantes para nuestros términos clave.\n\n**Salvar Arquivo**\n\nVou salvar os dados de todos os artigos que são pertinente aos nossos termos chave","metadata":{}},{"cell_type":"code","source":"#filtra os dados\ndf_filter = df_amostra[ df_amostra['termos'] == True]","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.016411Z","iopub.execute_input":"2023-06-14T12:26:56.016976Z","iopub.status.idle":"2023-06-14T12:26:56.030594Z","shell.execute_reply.started":"2023-06-14T12:26:56.016933Z","shell.execute_reply":"2023-06-14T12:26:56.029242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#grava um arquivo *.csv\ndf_filter.to_csv('filter_ethics_data.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.032521Z","iopub.execute_input":"2023-06-14T12:26:56.032983Z","iopub.status.idle":"2023-06-14T12:26:56.116831Z","shell.execute_reply.started":"2023-06-14T12:26:56.032940Z","shell.execute_reply":"2023-06-14T12:26:56.115583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"arquivo_filtrado = '/kaggle/working/filter_ethics_data.csv'","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.119350Z","iopub.execute_input":"2023-06-14T12:26:56.119823Z","iopub.status.idle":"2023-06-14T12:26:56.125924Z","shell.execute_reply.started":"2023-06-14T12:26:56.119780Z","shell.execute_reply":"2023-06-14T12:26:56.124636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(arquivo_filtrado)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.127826Z","iopub.execute_input":"2023-06-14T12:26:56.128392Z","iopub.status.idle":"2023-06-14T12:26:56.179596Z","shell.execute_reply.started":"2023-06-14T12:26:56.128351Z","shell.execute_reply":"2023-06-14T12:26:56.178392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.columns","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.181456Z","iopub.execute_input":"2023-06-14T12:26:56.181783Z","iopub.status.idle":"2023-06-14T12:26:56.190174Z","shell.execute_reply.started":"2023-06-14T12:26:56.181756Z","shell.execute_reply":"2023-06-14T12:26:56.188891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['license'].unique()","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.192092Z","iopub.execute_input":"2023-06-14T12:26:56.192554Z","iopub.status.idle":"2023-06-14T12:26:56.208016Z","shell.execute_reply.started":"2023-06-14T12:26:56.192511Z","shell.execute_reply":"2023-06-14T12:26:56.206216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.214298Z","iopub.execute_input":"2023-06-14T12:26:56.215412Z","iopub.status.idle":"2023-06-14T12:26:56.226968Z","shell.execute_reply.started":"2023-06-14T12:26:56.215374Z","shell.execute_reply":"2023-06-14T12:26:56.226143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"# Converter a coluna para o tipo de dados de data\ndf['update_date'] = pd.to_datetime(df['update_date'])\n# Calcular a média das datas\ndatas_media = df['update_date'].mean()","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.228189Z","iopub.execute_input":"2023-06-14T12:26:56.229022Z","iopub.status.idle":"2023-06-14T12:26:56.243106Z","shell.execute_reply.started":"2023-06-14T12:26:56.228988Z","shell.execute_reply":"2023-06-14T12:26:56.241707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calcular a diferença em relação ao valor máximo\ndf['update_days_diff'] = (df['update_date'].max() - df['update_date']).dt.days","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.244753Z","iopub.execute_input":"2023-06-14T12:26:56.245684Z","iopub.status.idle":"2023-06-14T12:26:56.262686Z","shell.execute_reply.started":"2023-06-14T12:26:56.245651Z","shell.execute_reply":"2023-06-14T12:26:56.261604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['update_days_diff'].describe()","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.264609Z","iopub.execute_input":"2023-06-14T12:26:56.264974Z","iopub.status.idle":"2023-06-14T12:26:56.284397Z","shell.execute_reply.started":"2023-06-14T12:26:56.264945Z","shell.execute_reply":"2023-06-14T12:26:56.283378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_diff_media = df['update_days_diff'].mean()\ndata_diff_desvio_padrao = df['update_days_diff'].std()\ndata_diff_variancia = df['update_days_diff'].var()\ndata_diff_dispersao = df['update_days_diff'].std() ** 2","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.285900Z","iopub.execute_input":"2023-06-14T12:26:56.287118Z","iopub.status.idle":"2023-06-14T12:26:56.298562Z","shell.execute_reply.started":"2023-06-14T12:26:56.287079Z","shell.execute_reply":"2023-06-14T12:26:56.297339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# quantidade de termos\ntermos_quantidade_media = df['termos_quantidade'].mean()\ntermos_quantidade_desvio_padrao = df['termos_quantidade'].std()\ntermos_quantidade_variancia = df['termos_quantidade'].var()\ntermos_quantidade_dispersao = df['termos_quantidade'].std() ** 2\n","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.300697Z","iopub.execute_input":"2023-06-14T12:26:56.301510Z","iopub.status.idle":"2023-06-14T12:26:56.313783Z","shell.execute_reply.started":"2023-06-14T12:26:56.301467Z","shell.execute_reply":"2023-06-14T12:26:56.312772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"\"\"Considerando a quantidade de termos relativos à etica, temos os seguintes métricas\n        Média: {termos_quantidade_media}\nDesvio Padrão: {termos_quantidade_desvio_padrao}\n    Variância: {termos_quantidade_variancia}\n    Dispersão: {termos_quantidade_dispersao}\n\"\"\")","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.315172Z","iopub.execute_input":"2023-06-14T12:26:56.316036Z","iopub.status.idle":"2023-06-14T12:26:56.328491Z","shell.execute_reply.started":"2023-06-14T12:26:56.315994Z","shell.execute_reply":"2023-06-14T12:26:56.327323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"\"\"Considerando a data de publicação (quantos dias atrás foram publicados os artigos, considerando a data atual), temos os seguintes métricas\n        Média: {data_diff_media}\nDesvio Padrão: {data_diff_desvio_padrao}\n    Variância: {data_diff_variancia}\n    Dispersão: {data_diff_dispersao}\n\"\"\")","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.329968Z","iopub.execute_input":"2023-06-14T12:26:56.330417Z","iopub.status.idle":"2023-06-14T12:26:56.340880Z","shell.execute_reply.started":"2023-06-14T12:26:56.330386Z","shell.execute_reply":"2023-06-14T12:26:56.339654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sorted_values = df.sort_values(['termos_quantidade', 'update_days_diff'], ascending=[False, True])","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.342684Z","iopub.execute_input":"2023-06-14T12:26:56.343134Z","iopub.status.idle":"2023-06-14T12:26:56.364183Z","shell.execute_reply.started":"2023-06-14T12:26:56.343092Z","shell.execute_reply":"2023-06-14T12:26:56.362780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sorted_values.columns","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.366310Z","iopub.execute_input":"2023-06-14T12:26:56.366693Z","iopub.status.idle":"2023-06-14T12:26:56.373930Z","shell.execute_reply.started":"2023-06-14T12:26:56.366661Z","shell.execute_reply":"2023-06-14T12:26:56.373016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"novas_colunas_df_filtrado = ['id', \n                             'authors_parsed', \n                             'title', \n                             'journal-ref', \n                             'categories', \n                             'update_date',  \n                             'termos_quantidade',\n                             'update_days_diff']","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.375421Z","iopub.execute_input":"2023-06-14T12:26:56.375756Z","iopub.status.idle":"2023-06-14T12:26:56.385997Z","shell.execute_reply.started":"2023-06-14T12:26:56.375728Z","shell.execute_reply":"2023-06-14T12:26:56.385177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sorted_values = df_sorted_values[novas_colunas_df_filtrado]","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.387045Z","iopub.execute_input":"2023-06-14T12:26:56.387384Z","iopub.status.idle":"2023-06-14T12:26:56.404334Z","shell.execute_reply.started":"2023-06-14T12:26:56.387348Z","shell.execute_reply":"2023-06-14T12:26:56.403384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sorted_values.head(20)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T12:26:56.406061Z","iopub.execute_input":"2023-06-14T12:26:56.406516Z","iopub.status.idle":"2023-06-14T12:26:56.434641Z","shell.execute_reply.started":"2023-06-14T12:26:56.406472Z","shell.execute_reply":"2023-06-14T12:26:56.433370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conclusão\n\nParece-me que a preocupação com o desempenho dos algoritomos, somados ao desejo de ser o primeiro a dominar (e possuir os direitos de exploração) da Inteligência Artificial é o Zeitgeist de nossa era. Temos excesso de produção de comida, e agravamento da fome. Vivemos cada vez mais, enquanto investimos fortunas em armas e guerras. O trabalho escravo e a exploração infantil aumentam, enquanto a riqueza das nações cresce. \n\nInveste-se fortunas para colonizar outros planetas, enquanto os recursos do único planeta conhecidamente habitável são exauridos pelo simples capricho da acumulação de capital.\n\nSeparei com esta análise os artigos que consideram ao menos citar algo referente à ética. Embora os dados destes mais de 2 milhões de artigos considerados nesta análise sejam desanimadores, revelo-me decepcionado, porém não surpreso de que a Inteligência Artificial não será a causa final do crepúsculo da humanidade, mas sim obra de algum grupo de seres humanos que acham feio o que não é espelho.\n\n\n**Conclusion**\n\nIt seems to me that the concern with algorithm performance, combined with the desire to be the first to dominate (and possess the exploitation rights) of Artificial Intelligence, is the Zeitgeist of our era. We have an excess of food production, yet hunger persists and worsens. We live longer lives while investing fortunes in weapons and wars. Slave labor and child exploitation are on the rise, even as the wealth of nations grows.\n\nFortunes are being invested in colonizing other planets, while the resources of the only known habitable planet are depleted for the mere whims of capital accumulation.\n\nThrough this analysis, I have separated the articles that at least mention something related to ethics. Although the data from these over 2 million articles considered in this analysis is disheartening, I find myself disappointed but not surprised that Artificial Intelligence will not be the ultimate cause of humanity's twilight, but rather the work of a group of human beings who find ugliness in what is not a mirror.\n\n**Conclusión**\n\nMe parece que la preocupación por el rendimiento de los algoritmos, junto con el deseo de ser los primeros en dominar (y poseer los derechos de explotación) la Inteligencia Artificial, es el espíritu de nuestra era. Tenemos un exceso de producción de alimentos, y a la vez el hambre se agrava. Vivimos más tiempo mientras invertimos fortunas en armas y guerras. La esclavitud laboral y la explotación infantil aumentan, mientras la riqueza de las naciones crece.\n\nSe invierten fortunas en colonizar otros planetas, mientras los recursos del único planeta conocido como habitable se agotan por el simple capricho de acumular capital.\n\nCon esta análisis, he separado los artículos que al menos mencionan algo relacionado con la ética. Aunque los datos de estos más de 2 millones de artículos considerados en este análisis son desalentadores, me encuentro decepcionado pero no sorprendido de que la Inteligencia Artificial no será la causa final del crepúsculo de la humanidad, sino más bien obra de algún grupo de seres humanos que encuentran fealdad en lo que no es un espejo.\n","metadata":{}},{"cell_type":"markdown","source":"## Cont(r)ate-me 📫:\n\n<p align='center'>\n  <a href='https://github.com/jonasaacampos'>\n    <img src='https://img.shields.io/badge/GitHub-100000?style=for-the-badge&logo=github&logoColor=white'/>\n  </a>\n  <a href='https://www.linkedin.com/in/jonasaacampos/'>\n    <img src='https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white'/>\n  </a>\n   <a href='https://dev.to/jonasaacampos'>\n    <img src='https://img.shields.io/badge/dev.to-0A0A0A?style=for-the-badge&logo=devdotto&logoColor=white'/>\n  </a>\n    <a href='https://www.buymeacoffee.com/jaac.dev'>\n    <img src='https://img.shields.io/badge/Buy_Me_A_Coffee-FFDD00?style=for-the-badge&logo=buy-me-a-coffee&logoColor=black'/>\n  </a>\n</p>\n","metadata":{}}]}