{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-05-31T14:50:57.124156Z","iopub.execute_input":"2021-05-31T14:50:57.124699Z","iopub.status.idle":"2021-05-31T14:51:09.391348Z","shell.execute_reply.started":"2021-05-31T14:50:57.124657Z","shell.execute_reply":"2021-05-31T14:51:09.376579Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import json\nimport os\nimport glob\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom nltk.corpus import stopwords\n\nimport re\nimport string\n\nfrom wordcloud import WordCloud, STOPWORDS","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:51:09.393461Z","iopub.execute_input":"2021-05-31T14:51:09.393908Z","iopub.status.idle":"2021-05-31T14:51:09.40674Z","shell.execute_reply.started":"2021-05-31T14:51:09.393861Z","shell.execute_reply":"2021-05-31T14:51:09.405596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# What exactly is the objective of this competition?","metadata":{}},{"cell_type":"markdown","source":"The goal is to predict whether scientific articles mention datasets. The submission must include the id of the article and a string justifying the prediction (cleaned using the ``clean_text()`` function provided). Our model must predict this string.","metadata":{}},{"cell_type":"code","source":"train_0 = pd.read_json('/kaggle/input/coleridgeinitiative-show-us-the-data/train/f8b03c87-9d1a-4f20-b76b-cb6c69d447b2.json')\ntrain_csv = pd.read_csv('/kaggle/input/coleridgeinitiative-show-us-the-data/train.csv')","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:51:09.408783Z","iopub.execute_input":"2021-05-31T14:51:09.409103Z","iopub.status.idle":"2021-05-31T14:51:09.600242Z","shell.execute_reply.started":"2021-05-31T14:51:09.409073Z","shell.execute_reply":"2021-05-31T14:51:09.599311Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_csv","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:51:09.601675Z","iopub.execute_input":"2021-05-31T14:51:09.601978Z","iopub.status.idle":"2021-05-31T14:51:09.632522Z","shell.execute_reply.started":"2021-05-31T14:51:09.601936Z","shell.execute_reply":"2021-05-31T14:51:09.631474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_csv.pub_title[0]","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:51:09.633653Z","iopub.execute_input":"2021-05-31T14:51:09.633916Z","iopub.status.idle":"2021-05-31T14:51:09.638665Z","shell.execute_reply.started":"2021-05-31T14:51:09.63389Z","shell.execute_reply":"2021-05-31T14:51:09.638032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here for instance, the article title \"The Impact of Dual Enrollment on College Degree Attainment: Do Low-SES Students Benefit?\" uses data from the National Education Longitudinal Study. Remember that we must predict the dataset(s) used for all articles or publication names. There can obviously be several datasets attached to one article.","metadata":{"execution":{"iopub.status.busy":"2021-05-21T15:54:59.904697Z","iopub.execute_input":"2021-05-21T15:54:59.90508Z","iopub.status.idle":"2021-05-21T15:54:59.918664Z","shell.execute_reply.started":"2021-05-21T15:54:59.905048Z","shell.execute_reply":"2021-05-21T15:54:59.917623Z"}}},{"cell_type":"markdown","source":"# What exactly is our training data?","metadata":{}},{"cell_type":"markdown","source":"``train.csv`` is a file linking the publication and datasets referenced. We find the publication texts in the individual .json files. As for the test data, we have 4 articles to parse and link to datasets. From what I understand now, we have to find strings within the articles' text that we find are likely to be references to datasets. ","metadata":{}},{"cell_type":"markdown","source":"# How many different publications are there in the training set? And how many datasets are referenced?","metadata":{}},{"cell_type":"code","source":"print(f'There are {len(train_csv.Id.unique())} different articles and {len(train_csv.cleaned_label.unique())} different datasets.')","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:51:09.639744Z","iopub.execute_input":"2021-05-31T14:51:09.64026Z","iopub.status.idle":"2021-05-31T14:51:09.660377Z","shell.execute_reply.started":"2021-05-31T14:51:09.640228Z","shell.execute_reply":"2021-05-31T14:51:09.659397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Let's plot the distribution of publications and datasets","metadata":{}},{"cell_type":"code","source":"sns.countplot(x=train_csv.Id.value_counts())","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:51:09.662314Z","iopub.execute_input":"2021-05-31T14:51:09.662794Z","iopub.status.idle":"2021-05-31T14:51:09.918092Z","shell.execute_reply.started":"2021-05-31T14:51:09.662728Z","shell.execute_reply":"2021-05-31T14:51:09.917073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure(figsize=(13, 6))\nfig.suptitle('Distribution of articles and datasets', fontsize=20)\n\nax0 = plt.subplot2grid((1, 2), (0, 0))\nax1 = plt.subplot2grid((1, 2), (0, 1))\n\nax0.hist(train_csv.Id.value_counts())\nax0.set_xlabel(\"# of linked datasets by article\")\nax0.set_ylabel(\"# of articles\")\n\nax1.hist(train_csv.cleaned_label.value_counts())\nax1.set_xlabel(\"# of occurences of datasets in articles\")\nax1.set_ylabel(\"# of datasets\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:51:09.920349Z","iopub.execute_input":"2021-05-31T14:51:09.920672Z","iopub.status.idle":"2021-05-31T14:51:10.290907Z","shell.execute_reply.started":"2021-05-31T14:51:09.920642Z","shell.execute_reply":"2021-05-31T14:51:10.29024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.kdeplot(x=train_csv.Id.value_counts())","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:51:10.292262Z","iopub.execute_input":"2021-05-31T14:51:10.292556Z","iopub.status.idle":"2021-05-31T14:51:10.601491Z","shell.execute_reply.started":"2021-05-31T14:51:10.292528Z","shell.execute_reply":"2021-05-31T14:51:10.600529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.kdeplot(x=train_csv.cleaned_label.value_counts())","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:51:10.603041Z","iopub.execute_input":"2021-05-31T14:51:10.603444Z","iopub.status.idle":"2021-05-31T14:51:10.778238Z","shell.execute_reply.started":"2021-05-31T14:51:10.603402Z","shell.execute_reply":"2021-05-31T14:51:10.777545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The distribution of linked datasets by articles is much less smoother than that of articles by datasets. **Why?** It looks like a Laplace distibution. **Could I test it?**","metadata":{}},{"cell_type":"markdown","source":"This script allows to check the number of files in the train directory. Taking out the train.csv file we have one \nmore article than our code above found within the train.csv file.\\\n``cd ../../kaggle/input\ncd ./coleridgeinitiative-show-us-the-data/train\nls -f . | wc -l``\\\nThe answer should be 14318.","metadata":{}},{"cell_type":"markdown","source":"## What form do the articles have?","metadata":{}},{"cell_type":"code","source":"article0 = pd.read_json('/kaggle/input/coleridgeinitiative-show-us-the-data/train/d0fa7568-7d8e-4db9-870f-f9c6f668c17b.json')","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:51:10.779238Z","iopub.execute_input":"2021-05-31T14:51:10.779618Z","iopub.status.idle":"2021-05-31T14:51:10.793027Z","shell.execute_reply.started":"2021-05-31T14:51:10.779589Z","shell.execute_reply":"2021-05-31T14:51:10.792099Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"article0","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:51:10.794206Z","iopub.execute_input":"2021-05-31T14:51:10.794504Z","iopub.status.idle":"2021-05-31T14:51:10.805651Z","shell.execute_reply.started":"2021-05-31T14:51:10.794476Z","shell.execute_reply":"2021-05-31T14:51:10.804944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So our first article has 18 titles. The left column contains the title of the paragraph and the right column contains the text.","metadata":{}},{"cell_type":"code","source":"article0.text[0]","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:51:10.806892Z","iopub.execute_input":"2021-05-31T14:51:10.807314Z","iopub.status.idle":"2021-05-31T14:51:10.825958Z","shell.execute_reply.started":"2021-05-31T14:51:10.807284Z","shell.execute_reply":"2021-05-31T14:51:10.824263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at the paragraph titles.","metadata":{}},{"cell_type":"code","source":"for sentence in article0.section_title:\n    print(''.join(sentence))","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:51:10.827402Z","iopub.execute_input":"2021-05-31T14:51:10.827919Z","iopub.status.idle":"2021-05-31T14:51:10.841187Z","shell.execute_reply.started":"2021-05-31T14:51:10.827886Z","shell.execute_reply":"2021-05-31T14:51:10.840397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset representation","metadata":{}},{"cell_type":"code","source":"train_csv = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:53:10.466138Z","iopub.execute_input":"2021-05-31T14:53:10.4665Z","iopub.status.idle":"2021-05-31T14:53:10.543344Z","shell.execute_reply.started":"2021-05-31T14:53:10.46647Z","shell.execute_reply":"2021-05-31T14:53:10.542442Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stopwords = set(STOPWORDS)\n\nwordcloud = WordCloud(background_color='white',\n                      stopwords=stopwords,\n                      max_words=100,\n                      max_font_size=30,\n                      scale=3,\n                      random_state=1)\n   \ncloud = wordcloud.generate(str(train_csv['dataset_title'].unique()))\n\nfig = plt.figure(figsize=(15, 15))\nplt.axis('off')\nplt.imshow(cloud)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-05-31T14:54:18.858045Z","iopub.execute_input":"2021-05-31T14:54:18.858414Z","iopub.status.idle":"2021-05-31T14:54:19.446974Z","shell.execute_reply.started":"2021-05-31T14:54:18.858383Z","shell.execute_reply":"2021-05-31T14:54:19.445926Z"},"trusted":true},"execution_count":null,"outputs":[]}]}