{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Description\nThe objective of the competition is to identify the mention of datasets within scientific publications. Your predictions will be short excerpts from the publications that appear to note a dataset. Predictions that more accurately match the precise words used to identify the dataset within the publication will score higher. Predictions should be cleaned using the clean_text function from the Evaluation page to ensure proper matching.\n\nPublications are provided in JSON format, broken up into sections with section titles.\n\nThe goal in this competition is not just to match known dataset strings but to generalize to datasets that have never been seen before using NLP and statistical techniques. A percentage of the public test set publications are drawn from the training set - not all datasets have been identified in train, so these unidentified datasets have been used as a portion of the public test labels. These should serve as guides for the difficult task of labeling the private test set.\n\nNote that the hidden test set has roughly ~8000 publications, many times the size of the public test set. Plan your compute time accordingly."},{"metadata":{},"cell_type":"markdown","source":"# Files\n- train - the full text of the training set's publications in JSON format, broken into sections with section titles\n- test - the full text of the test set's publications in JSON format, broken into sections with section titles\n- train.csv - labels and metadata for the training set\n- sample_submission.csv - a sample submission file in the correct format"},{"metadata":{},"cell_type":"markdown","source":"# Columns\n- id - publication id - note that there are multiple rows for some training documents, indicating multiple mentioned datasets\n- pub_title - title of the publication (a small number of publications have the same title)\n- dataset_title - the title of the dataset that is mentioned within the publication\n- dataset_label - a portion of the text that indicates the dataset\n- cleaned_label - the dataset_label, as passed through the clean_text function from the Evaluation page"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\n\ntrain_df = pd.read_csv('/kaggle/input/coleridgeinitiative-show-us-the-data/train.csv')\ntrain_df.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import json\nfile = open('/kaggle/input/coleridgeinitiative-show-us-the-data/train/{}'.format(train_df['Id'].iloc[1])+'.json')\ntext_json = json.load(file)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# text_json  # [{'section_title':'', 'text':''}]\n# len(text_json)\n# text_json[8].get('section_title')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# file = open('coleridgeinitiative-show-us-the-data/test/2100032a-7c33-4bff-97ef-690822c43466.json')\n# text_json = json.load(file)\n# text_json","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_all_text(json_file):\n    \n    file = open('/kaggle/input/coleridgeinitiative-show-us-the-data/train/{}'.format(json_file)+'.json')\n    text_json = json.load(file)\n    length = len(text_json)\n    text_list = []\n    \n    for i in range(length):\n        text_list.append(text_json[i].get('text'))\n    text = ' '.join(text_list)\n    \n    return text\n\ndef get_all_section_titles(json_file):\n    \n    file = open('/kaggle/input/coleridgeinitiative-show-us-the-data/train/{}'.format(json_file)+'.json')\n    text_json = json.load(file)\n    length = len(text_json)\n    section_titles_list = []\n    \n    for i in range(length):\n        section_titles_list.append(text_json[i].get('section_title'))\n    section_titles = ' '.join(section_titles_list)\n    \n    return section_titles","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['text'] = train_df['Id'].apply(get_all_text)\ntrain_df['section_titles'] = train_df['Id'].apply(get_all_section_titles)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.drop(columns=['dataset_label','Id'],inplace=True) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import re\n\ndef clean_text(txt):\n    return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['section_titles'] = train_df['section_titles'].apply(clean_text)\ntrain_df['text'] = train_df['text'].apply(clean_text)\ntrain_df['dataset_title'] = train_df['dataset_title'].apply(clean_text)\ntrain_df['pub_title'] = train_df['pub_title'].apply(clean_text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from collections import Counter\n\nprint('No of Data Titles:',len(Counter(train_df['dataset_title'])))\nprint('No of Data Labels:',len(Counter(train_df['cleaned_label'])))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\ntest_data = []\n\nfor i in os.listdir('/kaggle/input/coleridgeinitiative-show-us-the-data/test/'):\n    \n    file = open('/kaggle/input/coleridgeinitiative-show-us-the-data/test/{}'.format(i))\n    text_json = json.load(file)\n    length = len(text_json)\n    text_list = []\n    section_titles_list=[]\n    \n    for j in range(length):\n        text_list.append(text_json[j].get('text'))\n        section_titles_list.append(text_json[j].get('section_title'))\n        text = ' '.join(text_list)\n        section_titles=' '.join(section_titles_list)\n        \n    test_data.append([i.split('.')[0],text,section_titles])\n    \ntest_df = pd.DataFrame(columns = ['Id','text','section_titles'], data =test_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['text'] = test_df['text'].apply(clean_text)\ntest_df['section_titles'] = test_df['section_titles'].apply(clean_text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.drop(columns=['Id'],inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}