{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Where's Waldo - Dataset Edition\nThis notebook contains our approach to the Coleridge challenge. It uses a SciBERT model to find the datasets in text. We sadly had to limit our predictions to the first few 2048 sentences of each publication, since our full model took too long to predict on the full hidden test set.","metadata":{}},{"cell_type":"markdown","source":"# Getting things ready","metadata":{}},{"cell_type":"markdown","source":"## Install packages offline\n- Install `segtok` for splitting sentences into words.\n- Install the right version of `fsspec` (required by `simpletransformers`)\n- Install the right version of `seqeval` (required by `simpletransformers`)\n- Install `simpletransformers`","metadata":{}},{"cell_type":"code","source":"!pip install segtok --no-index --find-links=file:///kaggle/input/coleridgepackages/packages/\n!pip install fsspec --no-index --find-links=file:///kaggle/input/coleridgepackages/packages/\n!pip install seqeval --no-index --find-links=file:///kaggle/input/coleridgepackages/packages/\n!pip install simpletransformers --no-index --find-links=file:///kaggle/input/simpletransformers/simpletransformers-0.51.0/","metadata":{"scrolled":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-06-10T09:10:35.652491Z","iopub.execute_input":"2021-06-10T09:10:35.652881Z","iopub.status.idle":"2021-06-10T09:10:51.348559Z","shell.execute_reply.started":"2021-06-10T09:10:35.652841Z","shell.execute_reply":"2021-06-10T09:10:51.347616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Import libraries","metadata":{}},{"cell_type":"code","source":"import os, json, re\nimport pandas as pd\nimport numpy as np\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nfrom simpletransformers.ner import NERModel, NERArgs\nimport torch\n\nfrom tqdm import tqdm\nfrom sklearn.model_selection import train_test_split","metadata":{"execution":{"iopub.status.busy":"2021-06-10T09:54:59.94486Z","iopub.execute_input":"2021-06-10T09:54:59.945183Z","iopub.status.idle":"2021-06-10T09:55:00.002881Z","shell.execute_reply.started":"2021-06-10T09:54:59.945155Z","shell.execute_reply":"2021-06-10T09:55:00.001877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define functions","metadata":{}},{"cell_type":"markdown","source":"Define the function that is given to us on the evaluation page to clean text.","metadata":{}},{"cell_type":"code","source":"def clean_text(txt):\n    return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower())","metadata":{"execution":{"iopub.status.busy":"2021-06-10T08:46:35.420607Z","iopub.execute_input":"2021-06-10T08:46:35.421028Z","iopub.status.idle":"2021-06-10T08:46:35.427109Z","shell.execute_reply.started":"2021-06-10T08:46:35.420997Z","shell.execute_reply":"2021-06-10T08:46:35.426287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following functions are mainly used to get some data from a given publication.","metadata":{}},{"cell_type":"code","source":"# Get number of sentences in a given text\ndef get_sentence_count(text):\n    return len(split_single(text))\n\n# Get datasets used in publication\ndef extract_datasets(doc_id):\n    temp = train.loc[train['Id'] == doc_id]\n    return temp.dataset_label.values\n\n# Get text for training publication\ndef extract_train_text(filename):\n    file_loc = f\"{data_folder}/train/{filename}.json\"\n    \n    with open(file_loc) as f:\n        json_file = json.load(f)\n        text_list = [section['text'] for section in json_file]\n    \n    return \" \".join(text_list)\n\n# Get text for test publication\ndef extract_test_text(filename):\n    file_loc = f\"{data_folder}/test/{filename}.json\"\n    \n    with open(file_loc) as f:\n        json_file = json.load(f)\n        text_list = [section['text'] for section in json_file]\n    \n    return \" \".join(text_list)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following function adds tags for the `datasets` in a `text` and saves it to an `output` file. It only considers sentences that actually have a dataset in them. ","metadata":{}},{"cell_type":"code","source":"def preprocess_text_part(text, datasets, outfile):\n    # Keep track of the sentence we are currently adding to the output file\n    global sent_id\n    \n    # Get the sentences from the given text\n    sentences = split_single(text)    \n    \n    # Get indices for sentences with dataset in them\n    indices = []\n    for i, sent in enumerate(sentences):\n        for dataset in datasets:\n            if dataset in sent:\n                indices.append(i)\n    \n    # Remove duplicates\n    indices = list(set(indices))\n\n    # Go through all these indices\n    for ix in indices:\n        # Retrieve the sentence and split it into words\n        sent = sentences[ix]\n        sent_words = sent.split(' ')\n\n        # Find longest dataset  for current sentence\n        max_dataset_len = 0\n        max_dataset = \"\"\n        for dataset in datasets:\n            if dataset in sent and max_dataset_len < len(dataset):\n                    max_dataset_len = len(dataset)\n                    max_dataset = dataset\n\n        ds_indices = []\n\n        # Split the dataset into words\n        ds_words = max_dataset.split(' ')        \n\n        # Check if the dataset consists of 1 or more words\n        if len(ds_words) == 1:\n            # If one go through all the words in the sentence, if the cleaned dataset is\n            # equal to one of these words we add this words index to a list\n            for i, w in enumerate(sent_words):\n                    # if clean_text(dataset) == clean_text(w):\n                    if max_dataset in w:\n                        ds_indices.append([i])\n        elif len(ds_words) > 1:\n            # Else go through all the words in the sentence and add the indices of the dataset\n            # words in the sentence words to the list\n            for i in range(len(sent_words)):\n                if sent_words[i] == ds_words[0] and sent_words[i:i+len(ds_words)] == ds_words:\n                    ds_indices.append([ix for ix in range(i, i+len(ds_words))])\n        else: \n            print(\"Something strange happened...\")\n\n        # Tag every word in the sentence with 'O' except at the given indices\n        for i, word in enumerate(sent_words):\n            tag = 'O'\n\n            # Go trough all the lists of indices that contain dataset words\n            for ix_list in ds_indices:\n                # If the dataset consists of 1 word and is equal to our word\n                # we tag it with 'B-DATASET'\n                if len(ix_list) == 1 and i == ix_list[0]:\n                    tag = 'B-DATASET'\n                # If the dataset consists of more than 1 word and its first word \n                # is equal to our current word , we also tag it with 'B-DATASET'\n                elif len(ix_list) > 1 and i == ix_list[0]:\n                    tag = 'B-DATASET'\n                # If it is one of the other words in the datset, we tag it with\n                # 'I-DATASET'\n                elif len(ix_list) > 1 and i in ix_list:\n                    tag = 'I-DATASET'\n            \n            # Write result to the given csv file\n            with open(outfile, 'a+') as f:\n                word_writer = csv.writer(f, delimiter='\\t')\n                word_writer.writerow([sent_id, word, tag])\n\n        sent_id += 1","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preparing the data","metadata":{}},{"cell_type":"markdown","source":"## Loading the data","metadata":{}},{"cell_type":"markdown","source":" We get a `train.csv` that has information about the publications and the datasets that they contain.","metadata":{}},{"cell_type":"code","source":"data_folder = \"../input/coleridgeinitiative-show-us-the-data\"\ntrain_csv = os.path.join(data_folder, \"train.csv\")\ntrain = pd.read_csv(train_csv)\ntrain.sample(5)","metadata":{"execution":{"iopub.status.busy":"2021-06-10T09:43:23.568552Z","iopub.execute_input":"2021-06-10T09:43:23.568932Z","iopub.status.idle":"2021-06-10T09:43:23.713569Z","shell.execute_reply.started":"2021-06-10T09:43:23.568901Z","shell.execute_reply":"2021-06-10T09:43:23.712742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The actual text in the publication can be found in the `train/` folder, and has a `.json` file for each publication. Each of these files has section titles and their corresponding texts.","metadata":{}},{"cell_type":"code","source":"train_sample_json = os.path.join(data_folder, f\"train/{train.Id.values[13891]}.json\") \nwith open(train_sample_json) as f:\n    sample_file = json.load(f)\n    title_list = [section['section_title'] for section in sample_file]    \n    print(f\"Sample section titles of text:\")\n    for t in title_list[:5]:\n        print('    ' + t )\n    text_list = [section['text'] for section in sample_file]\n    text = \" \".join(text_list)\n    print(f\"Sample of text:\\n '{text[0:250]}'\\n\")","metadata":{"execution":{"iopub.status.busy":"2021-06-10T09:53:21.092031Z","iopub.execute_input":"2021-06-10T09:53:21.092351Z","iopub.status.idle":"2021-06-10T09:53:21.103883Z","shell.execute_reply.started":"2021-06-10T09:53:21.092323Z","shell.execute_reply":"2021-06-10T09:53:21.103014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preprocessing the data","metadata":{}},{"cell_type":"markdown","source":"We showcase an old and a new/current method of preprocessing the input data. The old method was discarded when we discovered that our new method greatly improved precision and recall.","metadata":{}},{"cell_type":"markdown","source":"### Old method","metadata":{}},{"cell_type":"markdown","source":"In the old method of preprocessing the data, we first preprocess the data and then split it into training and validation data. ","metadata":{}},{"cell_type":"code","source":"# Process all ids and save it to a csv\nsent_id = 0 \noutfile = 'train_modified_part.csv'\n\nfor i in tqdm(train.Id.values):\n    text = extract_train_text(i)\n    datasets = extract_datasets(i)\n    preprocess_text_part_v2(text, datasets, outfile)","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This split simply takes the first 80% of the (processed) sentences and uses that as the training data, and uses the rest as validation data.","metadata":{}},{"cell_type":"code","source":"# Split data in training and eval data\nn = max(data['sentence_id'].values)\nsplit = int(0.8 * n)\n\ntrain_data =  data[data['sentence_id'] <= split]\neval_data = data[data['sentence_id'] > split]","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Show where the split takes place","metadata":{}},{"cell_type":"code","source":"print(train_data.head())\nprint(eval_data.head())","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Current/new method","metadata":{}},{"cell_type":"markdown","source":"> The output of this step is done beforehand, and loaded in from `coleridgepackages/data/train_modified.part.csv` and `coleridgepackages/data/val_modified.part.csv` \n\nGo through all ids, extract the text and tag the words in the sentences that contain datasets, then save to `.csv` file.","metadata":{}},{"cell_type":"code","source":"# Split data into training and validation data\ntrain_ids, val_ids = train_test_split(train.Id.values, test_size=0.2)\n\n# Process all the training ids and save it to a csv\nsent_id = 0\noutfile = 'train_modified_part.csv'\nfor i in tqdm(train_ids):\n    text = extract_train_text(i)\n    datasets = extract_datasets(i)\n    preprocess_text_part(text, datasets, outfile)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-06-10T08:46:42.39053Z","iopub.execute_input":"2021-06-10T08:46:42.390908Z","iopub.status.idle":"2021-06-10T08:46:42.394519Z","shell.execute_reply.started":"2021-06-10T08:46:42.390876Z","shell.execute_reply":"2021-06-10T08:46:42.393634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Do the same thing for our validation data.","metadata":{}},{"cell_type":"code","source":"# Process all the validation ids and save it to a csv\nsent_id = 0 \noutfile = 'val_modified_part.csv'\nfor i in tqdm(val_ids):\n    text = extract_train_text(i)\n    datasets = extract_datasets(i)\n    preprocess_text_part(text, datasets, outfile)","metadata":{"execution":{"iopub.status.busy":"2021-06-10T08:46:42.739287Z","iopub.execute_input":"2021-06-10T08:46:42.73962Z","iopub.status.idle":"2021-06-10T08:46:42.743617Z","shell.execute_reply.started":"2021-06-10T08:46:42.739589Z","shell.execute_reply":"2021-06-10T08:46:42.742372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Creating our model","metadata":{}},{"cell_type":"markdown","source":"Load old preprocessed data","metadata":{}},{"cell_type":"code","source":"data = pd.read_csv('train_modified_part.csv', delimiter='\\t', encoding='utf8',\n                 names=[\"sentence_id\", \"words\", \"labels\"]).dropna()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Load preprocessed data when recently processed","metadata":{}},{"cell_type":"code","source":"train_data_r = pd.read_csv('train_modified_part.csv', delimiter='\\t', encoding='utf8',\n                 names=[\"sentence_id\", \"words\", \"labels\"]).dropna()\nval_data_r = pd.read_csv('val_modified_part.csv', delimiter='\\t', encoding='utf8',\n                 names=[\"sentence_id\", \"words\", \"labels\"]).dropna()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Load preprocessed data from a Kaggle dataset (to reduce running time)","metadata":{}},{"cell_type":"code","source":"# Load data offline\ntrain_data = pd.read_csv('../input/coleridgepackages/data/train_modified_part.csv', delimiter='\\t', encoding='utf8',\n                 names=[\"sentence_id\", \"words\", \"labels\"]).dropna()\neval_data = pd.read_csv('../input/coleridgepackages/data/val_modified_part.csv', delimiter='\\t', encoding='utf8',\n                 names=[\"sentence_id\", \"words\", \"labels\"]).dropna()","metadata":{"execution":{"iopub.status.busy":"2021-06-10T08:46:44.05432Z","iopub.execute_input":"2021-06-10T08:46:44.054649Z","iopub.status.idle":"2021-06-10T08:46:46.47229Z","shell.execute_reply.started":"2021-06-10T08:46:44.054617Z","shell.execute_reply":"2021-06-10T08:46:46.471339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Setting up the SimpleTransformers model","metadata":{}},{"cell_type":"markdown","source":"Configure a logger to reduce clutter in the log messages.","metadata":{}},{"cell_type":"code","source":"# Logging config\nlogging.basicConfig(level=logging.INFO)\ntransformers_logger = logging.getLogger(\"transformers\")\ntransformers_logger.setLevel(logging.WARNING)","metadata":{"execution":{"iopub.status.busy":"2021-06-10T06:44:05.747401Z","iopub.execute_input":"2021-06-10T06:44:05.747768Z","iopub.status.idle":"2021-06-10T06:44:05.751446Z","shell.execute_reply.started":"2021-06-10T06:44:05.747737Z","shell.execute_reply":"2021-06-10T06:44:05.750502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Initialize the model using various parameters.","metadata":{}},{"cell_type":"code","source":"# Configure the arguments for our model\nmodel_args = NERArgs()\n\n# Define our labels\nmodel_args.labels_list = [\"O\", \"B-DATASET\", \"I-DATASET\"]\n\n# Define batch size\nmodel_args.train_batch_size = 32\n\n# Evaluate during training\nmodel_args.evaluate_during_training = True\nmodel_args.evaluate_during_training_verbose = True\nmodel_args.use_cached_eval_features = True\n\n# Enable early stopping\nmodel_args.use_early_stopping = True\nmodel_args.early_stopping_delta = 0.01\nmodel_args.early_stopping_metric='eval_loss'\nmodel_args.early_stopping_metric_minimize=True\nmodel_args.early_stopping_patience=2\n\n# Overwrite output if already exists\nmodel_args.overwrite_output_dir=True\n\n# Check that we are using GPU\ncuda_available = torch.cuda.is_available()\n\n# Load the weights from Sci-BERT \nweights_loc =  \"../input/coleridgepackages/sci-bert\"\nmodel = NERModel(\n    \"bert\", weights_loc, args=model_args, use_cuda=cuda_available\n)\n","metadata":{"execution":{"iopub.status.busy":"2021-06-10T09:11:39.774922Z","iopub.execute_input":"2021-06-10T09:11:39.775238Z","iopub.status.idle":"2021-06-10T09:11:43.339337Z","shell.execute_reply.started":"2021-06-10T09:11:39.77521Z","shell.execute_reply":"2021-06-10T09:11:43.33853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Train the model.","metadata":{}},{"cell_type":"code","source":"# Train the model\nmodel.train_model(train_data, eval_data=eval_data)","metadata":{"execution":{"iopub.status.busy":"2021-06-10T09:11:46.089711Z","iopub.execute_input":"2021-06-10T09:11:46.09005Z","iopub.status.idle":"2021-06-10T09:35:31.806805Z","shell.execute_reply.started":"2021-06-10T09:11:46.090019Z","shell.execute_reply":"2021-06-10T09:35:31.805912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Evaluate the model on our validation data","metadata":{}},{"cell_type":"code","source":"# Evaluate the model\nresult, model_outputs, preds_list = model.eval_model(eval_data)","metadata":{"execution":{"iopub.status.busy":"2021-06-10T09:39:09.72993Z","iopub.execute_input":"2021-06-10T09:39:09.730259Z","iopub.status.idle":"2021-06-10T09:41:14.481332Z","shell.execute_reply.started":"2021-06-10T09:39:09.73023Z","shell.execute_reply":"2021-06-10T09:41:14.480432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predictions","metadata":{}},{"cell_type":"markdown","source":"Load all the test files.","metadata":{}},{"cell_type":"code","source":"test_files = os.listdir('../input/coleridgeinitiative-show-us-the-data/test')\ntest_files","metadata":{"execution":{"iopub.status.busy":"2021-06-07T10:54:50.424876Z","iopub.execute_input":"2021-06-07T10:54:50.425249Z","iopub.status.idle":"2021-06-07T10:54:50.439595Z","shell.execute_reply.started":"2021-06-07T10:54:50.425213Z","shell.execute_reply":"2021-06-07T10:54:50.438848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Go through all ids, extract text, turn into sentences, feed these to the model, extract the words that have a dataset tag, and add these to the prediction string. ","metadata":{}},{"cell_type":"code","source":"submission_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv')\nids = submission_df['Id']\n\nlabels = []\n\nfor index in ids:\n    # Keep track of all the predictions for this publication\n    prediction_list = []\n    \n    try: \n        # Retrieve the text and split it into sentences\n        text = extract_test_text(index)    \n        sentences_list = split_single(text)\n        \n        # Take all the first 1024 sentences \n        inp = sentences_list[:1024] \n        \n        # Predict on these sentences\n        preds, _ = model.predict(inp)    \n        \n        # For each prediction on a sentence\n        for sent in preds:\n            long_pred_list = []\n            in_long_dataset = False\n\n            for j, word_label in enumerate(sent):\n                current_word = clean_text(list(word_label.keys())[0]).strip()\n                current_tag = list(word_label.values())[0]\n\n                # Look at the next tag if it exists, otherwise its an empty string\n                next_tag = \"\"\n                next_ix = j+1\n                if next_ix < len(sent):\n                    next_tag = list(sent[next_ix].values())[0]\n\n                # Single word that has B-DATASET tag, add it to the final list immediately\n                # when next tag = 'O' or word is at end of sentence\n                is_latest_word = (j == len(sent)-1)\n                if (current_tag=='B-DATASET' and next_tag == 'O') or (current_tag=='B-DATASET' and is_latest_word):\n                    if current_word not in prediction_list:\n                        prediction_list.append(current_word)\n                # If we otherwise meet a word with a B-DATASET tag we must be in a long dataset name\n                elif current_tag=='B-DATASET':\n                    in_long_dataset = True\n                # No longer if we meet a word with an O tag\n                elif current_tag=='O':\n                    in_long_dataset = False\n\n                if in_long_dataset:\n                    long_pred_list.append(current_word)\n\n            # Combine long predictions\n            long_pred_str = ' '.join(long_pred_list)\n\n            # \n            if len(long_pred_list) > 0:\n                # Combine the dataset for the current sentence\n                long_pred_str = ' '.join(long_pred_list)\n\n                # Add it to the full predictions for the current publication if its not\n                # in there already\n                if long_pred_str not in prediction_list:\n                    prediction_list.append(long_pred_str)\n    except:\n        print('an error occurred')\n        pass\n    \n    # Add the predictions for the current publication to the complete predictions\n    labels.append('|'.join(prediction_list))                ","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-06-07T10:54:51.693961Z","iopub.execute_input":"2021-06-07T10:54:51.6943Z","iopub.status.idle":"2021-06-07T10:55:08.028084Z","shell.execute_reply.started":"2021-06-07T10:54:51.694271Z","shell.execute_reply":"2021-06-07T10:55:08.02705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create submission file","metadata":{}},{"cell_type":"markdown","source":"Add the ids and the prediction strings to the submission dataframe. Save the dataframe as `.csv` and submit.","metadata":{}},{"cell_type":"code","source":"submission = pd.DataFrame()\nsubmission['Id'] = ids\nsubmission['PredictionString'] = labels\n\n\nsubmission","metadata":{"execution":{"iopub.status.busy":"2021-06-07T10:55:32.369791Z","iopub.execute_input":"2021-06-07T10:55:32.370125Z","iopub.status.idle":"2021-06-07T10:55:32.386213Z","shell.execute_reply.started":"2021-06-07T10:55:32.370088Z","shell.execute_reply":"2021-06-07T10:55:32.385404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-06-04T08:15:54.739946Z","iopub.execute_input":"2021-06-04T08:15:54.740303Z","iopub.status.idle":"2021-06-04T08:15:54.976878Z","shell.execute_reply.started":"2021-06-04T08:15:54.740267Z","shell.execute_reply":"2021-06-04T08:15:54.976099Z"},"trusted":true},"execution_count":null,"outputs":[]}]}