{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Simple EDA and preprocessing of DataFrame\n## Coleridge Initiative - Show US the Data\n\nIn this competition, we are given scientific articles and asked to identify mentions of datasets. \n\n> The objective of the competition is to identify the mention of datasets within scientific publications. \n\nLet's just dive in!\n\n\n**If you find this helpful, please give it an upvote!**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nfrom tqdm.notebook import tqdm\nfrom fastcore.all import *\nfrom collections import Counter","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check the directory:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_path = Path('../input/coleridgeinitiative-show-us-the-data')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_path.ls()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"(dataset_path/'train').ls()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"(dataset_path/'test').ls()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have our `train.csv`, `sample_submission.csv` and the `train` & `test` folders with json files of the text of the scientific articles. Let's check our train CSV file:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(dataset_path/'train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Basically we have the following columns:\n\n* `id` - publication id - note that there are multiple rows for some training documents, indicating multiple mentioned datasets\n* `pub_title` - title of the publication (a small number of publications have the same title)\n* `dataset_title` - the title of the dataset that is mentioned within the publication\n* `dataset_label` - a portion of the text that indicates the dataset\n* `cleaned_label` - the dataset_label, as passed through the `clean_text` function:\n```\ndef clean_text(txt):\n    return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()\n```"},{"metadata":{},"cell_type":"markdown","source":"Let's see what dataset titles are mentioned in the scientific articles. Note that there can be even more dataset mentions that are not labeled."},{"metadata":{"trusted":true},"cell_type":"code","source":"Counter(train_df.dataset_title)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'There are {len(Counter(train_df.dataset_title))} dataset titles in this dataset')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These are the dataset titles, but the exact way these datasets are mentioned in the paper is provided in the `dataset_label` column. Additionally, the `dataset_label` column has a simple text cleaning function applied to it, giving us `cleaned_labels`."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'There are {len(Counter(train_df.cleaned_label))} ways the datasets are mentioned in the papers')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at an example JSON file:"},{"metadata":{"trusted":true},"cell_type":"code","source":"example_json = (dataset_path/'train'/(train_df.Id.iloc[0]+'.json')).read_json()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can check JSON file by showing the output of the below code cell:"},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"print(example_json)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here, I am finding the target `dataset_label` and providing some context text:"},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(len(example_json)):\n    position = example_json[i]['text'].find(train_df.iloc[0].dataset_label)\n    if position != -1:\n        print(f'Found in section {i}, {example_json[i][\"section_title\"]}:')\n        print(f'{example_json[i][\"text\"][position-400:position+len(train_df.iloc[0].dataset_label)+400]}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A function to find the section and position of the `dataset_label`"},{"metadata":{"trusted":true},"cell_type":"code","source":"def find_str_in_json(json, string):\n    for i in range(len(json)):\n        position = json[i]['text'].find(string)\n        if position != -1:\n            return i, position\n    print('problem')\n    return -1, -1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's process the whole dataset and add the section and position of the `dataset_label` as new columns."},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"section_list = []\nposition_list = []\nfor i in tqdm(range(len(train_df))):\n    current_json = (dataset_path/'train'/(train_df.Id.iloc[i]+'.json')).read_json()\n    current_str = train_df.iloc[i].dataset_label\n    section, position = find_str_in_json(current_json, current_str)\n    section_list.append(str(section))\n    position_list.append(str(position))\ntrain_df['section'] = section_list\ntrain_df['position'] = position_list\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interestingly, there are many `dataset_label`s that are not found in the text json and I need to investigate this further. Please let me know if you have any ideas why this is happening."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here, I merge the rows with the same `Id` column, which is how it's found in the `sample_submission.csv`:"},{"metadata":{"trusted":true},"cell_type":"code","source":"new_train_df = pd.DataFrame({'dataset_label': train_df.groupby('Id', sort=False)['dataset_label'].apply('|'.join), \n              'cleaned_label': train_df.groupby('Id', sort=False)['cleaned_label'].apply('|'.join),\n              'section': train_df.groupby('Id', sort=False)['section'].apply('|'.join),\n              'position': train_df.groupby('Id', sort=False)['position'].apply('|'.join)})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"new_train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Before we end, let's quickly take a look at `sample_submission.csv`. Currently, no example prediction strings are provided. We are basically supposed to provide the exercepts of the dataset mentions, separated by `|` character. "},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.read_csv(dataset_path/'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The following Jaccard similarity score is used for evaluation:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def jaccard_similarity(s1, s2):\n    l1 = s1.split(\" \")\n    l2 = s2.split(\" \")    \n    intersection = len(list(set(l1).intersection(l2)))\n    union = (len(l1) + len(l2)) - intersection\n    return float(intersection) / union","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For now, **we are done.**\n\nIf you enjoyed this kernel, please give it an upvote. If you have any questions or suggestions, please leave a comment!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}