{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Basic EDA and my first impression\n\n- Providing text of science paper, we are going to extract (predict) a name of dataset contained in the text.\n- `train.csv` contains 3 columns abount dataset:\n    - dataset_title: official dataset name\n    - dataset_label: dataset name exactly appears in the text\n    - cleaned_label: dataset name cleaned from dataset_label using `clean_text` operation (see [competition page](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation)). All ground-truth texts have been cleaned for matching purposes.\n- Actual train/test texts are provided via JSON format. One JSON file contains multiple sections as a list and each section is composed of a title and a text.\n    \nThis is obviously NLP task. Additionally, I think:\n- Extracting a part of input text is a popular NLP task. SQuAD is a typical dataset often used as a benchmark for extractive QA task.\n    - Huggingface introduction: https://huggingface.co/transformers/task_summary.html#extractive-question-answering\n    - Keras example: https://keras.io/examples/nlp/text_extraction_with_bert/\n    - For this purpose, I have created a CSV file contains begin/end position of the dataset statement. Feel free to use it :).\n- There are a plenty of texts. Treating all of them with a heavy deep learning model (e.g. BERT-based model) cound result in exceeding the submission time-limit.\n    - Officially announced that \"the hidden test set has roughly ~8000 publications, many times the size of the public test set\".\n- The existence of unseen datasets is announced.\n    - Is is important to recognize not only known datasets but also the surrounding text pattern indicating the existence of dataset.\n    - Some regulatization encouraging a model to focus the surrounding pattern is a possible solution (e.g. drop a part of an exact dataset name from the training texts)."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport json\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom glob import glob\nfrom tqdm import tqdm\nfrom pandas_profiling import ProfileReport\n\ndatadir = '/kaggle/input/coleridgeinitiative-show-us-the-data'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(f'{datadir}/train.csv')\ndf_train.iloc[:10000:1000]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"profile = ProfileReport(df_train, title=\"Pandas Profiling Report\")\nprofile.to_widgets()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Relation between dataset_title and dataset_label"},{"metadata":{},"cell_type":"markdown","source":"A unique `dataset_title` can be stated with a different form. For example, *Alzheimer's Disease Neuroimaging Initiative (ADNI)* is stated in 3 forms as below:"},{"metadata":{"trusted":true},"cell_type":"code","source":"unique_dataset_titles = df_train.dataset_title.value_counts().reset_index()\nunique_dataset_titles.columns = ['dataset_title', 'counts']\nprint('most appeared dataset_title:', unique_dataset_titles.iloc[0])\n\nsample_ds_title = unique_dataset_titles.loc[0, 'dataset_title']\ndf = df_train.query('dataset_title == @sample_ds_title')\ndf = df[['dataset_title', 'dataset_label', 'cleaned_label']]\n\nprint(f'\\nRetrieved dataset_label/cleaned_label from \"{sample_ds_title}\"')\ndf.pivot_table(index='dataset_title', columns=['dataset_label', 'cleaned_label'], aggfunc=len)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df_train.groupby('dataset_title').apply(lambda df: df.dataset_label.value_counts())\ndf = df.reset_index()\ndf.columns = ['dataset_title', 'dataset_label', 'counts']\ndf","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_unique_labels = df.groupby('dataset_title').size()\ndf_unique_labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For example *SARS-CoV-2 genome sequence* is stated with 17 forms"},{"metadata":{"trusted":true},"cell_type":"code","source":"# The title related with the most unique dataset_label\ntitle = 'SARS-CoV-2 genome sequence'\ndf.query('dataset_title == @title')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Conversely, a `dataset_label` can be uniquely associated with a `dataset_title`"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df_train.groupby('dataset_label').apply(lambda df: df.dataset_title.value_counts())\ndf = df.reset_index()\ndf.columns = ['dataset_label', 'dataset_title', 'counts']\ndf.sort_values('dataset_title')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Explore texts"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_paths = glob(f'{datadir}/train/*.json')\ntrain_data = []\nfor path in train_paths[:100]:\n    with open(path, 'r') as f:\n        train_data.append(json.load(f))\n\ntest_paths = glob(f'{datadir}/test/*.json')\ntest_data = []\nfor path in test_paths:\n    with open(path, 'r') as f:\n        test_data.append(json.load(f))\n        \nprint(f'Train files: {len(train_paths)}')\nprint(f'Test files: {len(test_paths)}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Single JSON file is composed of multiple sections. Each section has a title and a text."},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sections = train_data[0]\nsample_path = train_paths[0]\nfilename = os.path.basename(sample_path)\n\nwith open(sample_path, 'r') as f:\n    sections = json.load(f)\n\nprint(f'{filename} has {len(sample_sections)} sections:')\n\nfor section in sections[:10]:\n    title = section['section_title']\n    text = section['text']\n    print(f'title: {title.ljust(70, \" \")}, text: {text[:50]} ...')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Counting the number of words composing each section in order to check the scale of the problem."},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\n# This aggregation takes tens of minutes. I have saved the resulting table and use it.\n\ndf_text_train = pd.DataFrame(columns=['Id', 'title', 'n_words'])\n\ntrain_ids = df_train.Id.unique()\n\nfor train_id in tqdm(train_ids):\n    path = f'{datadir}/train/{train_id}.json'\n    with open(path, 'r') as f:\n        data = json.load(f)\n        \n    for section in data:\n        title = section['section_title']\n        text = section['text']\n        n_words = len(text.split(' '))\n        row = {'Id': train_id, 'title': title, 'n_words': n_words}\n        df_text_train = df_text_train.append(row, ignore_index=True)\n'''\n\ndf_text_train = pd.read_csv('/kaggle/input/coleridge-initiative-assets/train_text.csv')\ndf_text_train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Single publication (i.e single JSON file) often has ~40 sections but those with over 100 sections also exist.\n- Large part of a section has ~1000 words but those with over 1500 words also exist."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 7))\nax1.set_xlabel('N sections per id')\nax1.set_ylabel('counts')\nax1.hist(df_text_train.Id.value_counts(), range=(0, 100), bins=20)\n\nax2.set_xlabel('N words per section')\nax2.set_ylabel('counts')\nax2.hist(df_text_train.n_words, range=(0, 2000), bins=20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Format for Extractive QA task\n\nIn the original BERT paper, the extractive QA task is solved via predicting start/end positions of the answer (see details in [huggingface document](https://huggingface.co/transformers/task_summary.html#extractive-question-answering) or [Keras example](https://keras.io/examples/nlp/text_extraction_with_bert/).\nFor this purpose, I have converted the `train.csv` and JSON contents to the table that contains the target `dataset_label` and corresponding section and start/end positions."},{"metadata":{"trusted":true},"cell_type":"code","source":"def find_label_position(row):\n    if len(row) > 1:\n        raise ValueError('Multiple rows detected')\n    \n    file_id, pub_title, dataset_title, dataset_label, cleaned_label = row.iloc[0].values\n\n    filename = f'{datadir}/train/{file_id}.json'\n    with open(filename, 'r') as f:\n        sections = json.load(f)\n        \n    data = []\n    for i, section in enumerate(sections):\n        text = section['text']        \n        begin = text.find(dataset_label)\n        while begin >= 0:\n            data.append([i, begin, begin+len(dataset_label)])\n            begin = text.find(dataset_label, begin+1)\n    df = pd.DataFrame(data, columns=['section_id', 'ds_label_begin', 'ds_label_end'])\n    return df\n\ntqdm.pandas()\ndf = df_train.groupby(['Id', 'dataset_label']).progress_apply(find_label_position).reset_index()\ndf = df[['Id', 'dataset_label', 'section_id', 'ds_label_begin', 'ds_label_end']]\ndf = df_train.merge(df, on=['Id', 'dataset_label'])\ndf","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_row = df.iloc[0]\nprint(sample_row)\n\nfile_id, section_id, label_begin, label_end = sample_row[['Id', 'section_id', 'ds_label_begin', 'ds_label_end']]\n\npath = f'{datadir}/train/{file_id}.json'\nwith open(path, 'r') as f:\n    sections = json.load(f)\n    \ntarget_section = sections[section_id]['text']\ntarget_section[label_begin: label_end]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.to_csv('train_extactive_qa.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}