{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nimport json\nimport glob\nimport tqdm\nimport spacy\n\nfrom transformers import RobertaTokenizer\nimport tokenizers\n\n# Define stop words\nsp = spacy.load('en_core_web_sm')\nall_stopwords = sp.Defaults.stop_words\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Gathering Data From Large Texts\n\nThe goal of this competition is to build a model where we can automatically extract the name of the data set associcated with `train.csv`. One of the first things we should look at is that text itself. The size of the text will make this competition challenging since models like BERT only allow for a max of 512 tokens so we will have to get creative or we can also check out models like the Longformer (https://huggingface.co/transformers/model_doc/longformer.html). Here is the official paper on the Longformer (https://arxiv.org/pdf/2004.05150.pdf)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load in the training data - the cleaned label names are defined here\ntrain_df = pd.read_csv('/kaggle/input/coleridgeinitiative-show-us-the-data/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Snippet to extract number of words from the different sections in each text &\n# extract the first section where the dataset name is mentioned (name comes from train.csv)\n\ntext_list = []\n\nfor data in tqdm.tqdm(glob.glob('/kaggle/input/coleridgeinitiative-show-us-the-data/train/*')):\n    full_filename = data.split('/')[-1]\n    filename = full_filename.split('.')[0]\n    \n    # Extract the dataset name for the text\n    # For some files there is more than one cleaned dataset name\n    dataset_name = (\n        train_df\n        .loc[train_df['Id'] == filename]\n        .reset_index(drop=True)['cleaned_label'][0]\n    )\n    \n    # Open the text data\n    with open(data, 'rb') as f:\n        text_data = json.load(f)\n        \n    # Snippet to count the number of words\n    number_of_words = 0\n    number_of_words_no_stop = 0\n    for idx, i in enumerate(text_data):\n        \n        # NOTE: We probably want to omit stop words but for\n        # a first take this is fine\n        text = i['text'].lower().split()\n        text_no_stop = [i for i in text if i not in all_stopwords]\n        \n        number_of_words += len(text)\n        number_of_words_no_stop += len(text_no_stop)\n    \n    # Extract the first section where the dataset name\n    # is mentioned\n    for idx, i in enumerate(text_data):\n        section = i['section_title'].lower()\n        text = i['text'].lower()\n        text_no_stop = ' '.join([i for i in text.split() if i not in all_stopwords])\n    \n        text_index = text.find(dataset_name)\n    \n        if text_index != -1:\n            start_index = text_index\n            end_index = start_index + len(dataset_name)\n            data = text[start_index:end_index]\n        \n            section_tuple = (idx, start_index, end_index, number_of_words, number_of_words_no_stop, data, text_no_stop)\n        \n            text_list.append(section_tuple)\n        \n            continue","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"extracted_data = pd.DataFrame(\n    text_list,\n    columns=['section', 'start_index', 'end_index', 'number_of_words', 'number_of_words_no_stop', 'dataset_name', 'text']\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"extracted_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Sections\n\nEach text is a list of JSON objects and each has to keys `section_title` & `text`. For a quick visualiztion on which sections the dataset name shows up in we can look at a bar plot of the value counts"},{"metadata":{"trusted":true},"cell_type":"code","source":"section_counts = extracted_data['section'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ax = section_counts[section_counts > 100].plot(kind='bar', figsize=(10, 7))\n\nax.set_title('Common Sections That Mention Dataset Name')\nax.set_xlabel('Section Number')\nax.set_ylabel('Counts')\n\nplt.grid()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are some texts with quite the number of sections as identifed by the JSON. We see that there can be a wide range where the dataset name is defined or mentioned for the first time. The idea here was to see if we could truncate the texts to be shorter. It looks like it could be possible but there will have to be some experimentation and creativity. For example, we could possible build a two-stage model. A model that goes through each section and predicts if the data set will be mentioned based on the text (binary classification) and then another model that predictions the start and ending index of the dataset.\n\nFrom the plot above we see that the dataset name of interest in most commonly found in section 1."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Simple plot that looks at the log number of words in a text\nax = extracted_data['number_of_words'].apply(np.log).hist(figsize=(10, 7), alpha=0.6, bins=100)\n\nax.set_title('Histogram For Number of Words in Text (Log)')\nax.set_xlabel('Log Number of Words')\nax.set_ylabel('Counts')\n\n# Simple plot that looks at the log number of words with stop words omitted in a text\nextracted_data['number_of_words_no_stop'].apply(np.log).hist(figsize=(10, 7), alpha=0.6, bins=100, color='orange', ax=ax)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Look at the head of the train.csv\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since we already have it defined, we can also take a quick look at some of the stats of the publication titles and the cleaned labels from the `train.csv`."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get lengths for the publication title and the cleaned label\ntrain_df['pub_title_length_by_word'] = train_df['pub_title'].apply(lambda x: len(x.split()))\ntrain_df['cleaned_label_length_by_word'] = train_df['cleaned_label'].apply(lambda x: len(x.split()))\n\ntrain_df['pub_title_length'] = train_df['pub_title'].apply(len)\ntrain_df['cleaned_label_length'] = train_df['cleaned_label'].apply(len)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ax = train_df['pub_title_length_by_word'].hist(figsize=(10, 7), bins=10)\nax.set_title('Historgram of Publication Title Lengths by Word')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ax = train_df['pub_title_length'].hist(figsize=(10, 7), bins=10)\nax.set_title('Historgram of Publication Title Lengths')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ax = train_df['cleaned_label_length_by_word'].hist(figsize=(10, 7), bins=10)\nax.set_title('Historgram of Cleaned Label Lengths by Word')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ax = train_df['cleaned_label_length'].hist(figsize=(10, 7), bins=10)\nax.set_title('Historgram of Cleaned Label Lengths')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If we want to train a model the next thing we will have to think about is tokenization of the text. I will be importing `tokenizers` which can be found here (https://pypi.org/project/tokenizers/). We will take a look at a simple example of what the `ByteLevelBPETokenizer` looks like and how we can use it on our text data. For a better understanding of BPE (Byte Pair Encoding) this artical may be useful (https://towardsdatascience.com/byte-pair-encoding-the-dark-horse-of-modern-nlp-eb36c7df4f10).\n\nFirst let's look at a simple example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load in the data for the tokenizer\nPATH = '../input/roberta-base'\nvocab_file = os.path.join(PATH, 'vocab.json')\nmerges_file = os.path.join(PATH, 'merges.txt')\n\ntokenizer = tokenizers.ByteLevelBPETokenizer(\n    vocab=vocab_file, \n    merges=merges_file, \n    lowercase=True,\n    add_prefix_space=True\n)\n\n# One thing we want to do before feeding our sequences into a model like RoBERTa is adding special tokens to the tokenizer\ntokenizer.add_special_tokens(['<s>', '</s>', '<pad>', '<mask>'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"encoding = tokenizer.encode('<s> Hello World!</s><pad>')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Look at the encoded tokens\nencoding.tokens","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Look at the offsets provided by the tokenizer\nencoding.offsets","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Look at the attention mask provided by the tokenizer\nencoding.attention_mask","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"encoding.ids","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"encoding = tokenizer.encode(extracted_data['text'][0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Look at the tokens\nencoding.tokens[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Look at the ids\nencoding.ids[:10]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is certainly a lot more work to be done here. We should probably remove punctuation & try to clean up the text a bit more before we tokenize and input into a model."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}