{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n#for dirname, _, filenames in os.walk('/kaggle/input'):\n#    for filename in filenames:\n#        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This competition challenges data scientists to show how publicly funded data and evidence are used to serve science and society. Data, evidence, and science are critical if government is address the many threats facing society: pandemics, climate change and coastal inundation, Alzheimer’s disease, child hunger, and support science and innovation, increase food production, maintain biodiversity, and address many other challenges. Yet much of the information about data necessary to inform evidence and science is locked inside publications.\n\nCan natural language processing find the hidden-in-plain-sight data citations? Can machine learning find the link between the words used in research articles and the data referenced in the article?"},{"metadata":{"trusted":true},"cell_type":"code","source":"# import dependencies\nimport pandas as pd\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EDA"},{"metadata":{"trusted":true},"cell_type":"code","source":"# import data\ndata = pd.read_csv('/kaggle/input/coleridgeinitiative-show-us-the-data/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Label analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"# find unique titles and sort alphabetically\nsorted(data['cleaned_label'].unique()) # clearly same datasets but written in different ways","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# label length\ncleaned_label = data['cleaned_label'].unique()\ncleaned_label_length = [len(x) for x in cleaned_label]\nplt.figure(figsize = (15,15))\nplt.bar(range(len(cleaned_label)), cleaned_label_length)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# find unique titles and sort alphabetically\nsorted(data['dataset_title'].unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"An original dataset title may be re-written/phrased in many ways\n\nFor example, **Aging Integrated Database (AGID)** is written as **aging integrated database agid** or **aging integrated database**"},{"metadata":{"trusted":true},"cell_type":"code","source":"title_count = data.groupby(['dataset_title']).count()\ntitle_count.sort_values(by = 'Id', ascending = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"title_count = data.groupby(['cleaned_label']).count()\ntitle_count.sort_values(by = 'Id', ascending = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Word Cloud of label"},{"metadata":{"trusted":true},"cell_type":"code","source":"from wordcloud import WordCloud, STOPWORDS\n\nwords_in_titles = list(data.pub_title.str.split(expand=True).stack())\n\nwordcloud = WordCloud(stopwords = STOPWORDS,\n                      background_color = \"white\",\n                      width = 3000,\n                      height = 2000\n                     ).generate(' '.join(words_in_titles))\nplt.figure(1, figsize = (18, 12))\nplt.imshow(wordcloud)\nplt.axis('off')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Publication Title Analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"data.groupby(['pub_title']).count().sort_values(by = 'Id', ascending = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Document analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"import json\n\n# read text\ntext = []\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        if filename.endswith('.json'):\n            text.append(json.load(open(os.path.join(dirname, filename))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# section length\nfrom collections import Counter\nsection_length = Counter([len(x) for x in text])\nsections = sorted(section_length.keys())\nplt.figure(figsize = (15,15))\nplt.bar(sections, [section_length[x] for x in sections])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# inspect first publication\ntext[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Section example from text[0] \n\n```\n{'section_title': 'Results', 'text': ''}\n```\n\nSomehow, section_title could be the title itself and sub-title."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}