{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Spacy Preprocessing for Named Entity Extraction\n\nBefore we begin executing a NER model we need to transform our existing data into a format for training. I wanted to share this notebook with the Kaggle community as I found the spacy docs a bit frustrating and I hope this might just help someone get off the ground a little faster!"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport json\nimport pandas as pd\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')\nsample_sub = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv')\ntrain_files_path = '../input/coleridgeinitiative-show-us-the-data/train'\ntest_files_path = '../input/coleridgeinitiative-show-us-the-data/test'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Appending text to training dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_append_return(filename, train_files_path=train_files_path, output='text'):\n    json_path = os.path.join(train_files_path, (filename+'.json'))\n    headings = []\n    contents = []\n    combined = []\n    with open(json_path, 'r') as f:\n        json_decode = json.load(f)\n        for data in json_decode:\n            headings.append(data.get('section_title'))\n            contents.append(data.get('text'))\n            combined.append(data.get('section_title'))\n            combined.append(data.get('text'))\n    \n    all_headings = ' '.join(headings)\n    all_contents = ' '.join(contents)\n    all_data = '. '.join(combined)\n    \n    if output == 'text':\n        return all_contents\n    elif output == 'head':\n        return all_headings\n    else:\n        return all_data\n    \ntqdm.pandas()\ntrain_df['text'] = train_df['Id'].progress_apply(read_append_return)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Import Spacy\n\nImport Spacy and load the english module, we can use this to generate POS (Part of Speech) tags for our text. These part of speech tags grammatically categorise the words in the text into categories like Noun, Verb, etc.\n\nSpacy also allows you to use it's EntityRuler module to generate Entity Recognition Tags. We will likely need this in the future, when we go to build a NER pipeline."},{"metadata":{"trusted":true},"cell_type":"code","source":"import spacy\nnlp = spacy.load('en_core_web_sm')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generate patterns for all of our datasets\n\nHere I convert all of the dataset names into Spacy patterns for identification. This is what will label our entities as Datasets when we execute pass text through the spacy module. \n\nNote: As I have added all of the datasets to a single pattern which is run against all incoming text files, this could induce errors if the dataset names are used in other essays, but in reference to something that is not the dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"patterns = []\nfor dataset in train_df.cleaned_label.unique():\n    phrase = []\n    for word in nlp(dataset):\n        pattern = {}\n        pattern[\"LOWER\"] = str(word)\n        phrase.append(pattern)\n    patterns.append({\"label\": \"DATASET\", \"pattern\": phrase})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Remove default NER from Spacy pipeline\n\nNER pipeline has an existing Entity Recognition module which we need to remove before adding our own."},{"metadata":{"trusted":true},"cell_type":"code","source":"nlp.remove_pipe(\"ner\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Build Spacy Pipeline\n\nHere we add a module to the spacy pipeline which matches the patterns we created above. It is possible to add other modules into the spacy pipeline, which may be of special interest later in the project."},{"metadata":{"trusted":true},"cell_type":"code","source":"from spacy.pipeline import EntityRuler\n\nruler = EntityRuler(nlp, overwrite_ents=True)\nruler.add_patterns(patterns)\nnlp.add_pipe(ruler)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\nstart = time.time()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"indexes = [0]\n\nfor index in indexes:\n    doc = nlp(train_df.iloc[index]['text'].lower())\n    print(\"{: >20} {: >20} {: >20} {: >20}\\n\".format('TEXT', 'POS', 'TAG', 'ENTITY'))\n    for token in doc:\n        print(\"{: >20} {: >20} {: >20} {: >20}\".format(token.text, token.pos_, token.tag_, token.ent_type))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"end = time.time()\nhours, rem = divmod(end-start, 3600)\nminutes, seconds = divmod(rem, 60)\nprint(\"{:0>2}:{:0>2}:{:05.2f}\".format(int(hours),int(minutes),seconds))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}