{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<h1 style=\"background-color:DodgerBlue; color:white\" >Custom NER using Spacy 3.0+</h1>\n\nRecently, in my work, I did custom NER using production-level NLP library called spaCy.  \n\nUtilizing that experience, this notebook aims to train a custom NER transformer-based model to detect datasets as entities. For achieving this, we require spaCy 3.0+.\n\nThe whole process is quite straightforward:\n1. Make your training dataset by marking entities in it. spaCy 3.0 requires DocBin format. \n    - For our problem, the training labels help us mark the entities. (the **positive examples**)\n    - Rest lines could be our **negative examples** with start and end indexes of entity has 0,0\n    - **Caution:** In this competition, train data is not exhaustively labeled. That means, we have some positive examples inside the examples that we mark as negative. You would ideally want to increase the class-prior weight of the positive examples we already know.\n2. Initialize spacy with a config file (**spacy init** command)\n3. Train spacy model using the settings mentioned in config file (**spacy train** command)\n4. Load the model and use it like any other spacy pipeline (**spacy.load()** command)\n","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport json\nimport glob\nimport re\nfrom tqdm import tqdm","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Note:**  This notebook uses internet, therefore, you cannot submit it as submission. However, you can take the trained model and use it make your submissions.","metadata":{}},{"cell_type":"markdown","source":"## Install Spacy 3.0.+ Transformers","metadata":{}},{"cell_type":"code","source":"!pip install -U spacy[transformers]","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Predefined function for prepropossing\n\nFor preprocessing, we stick to the given function which replaces anything apart from letters and digits with a ' '. However, for training our spaCy model, we do not lowercase the text","metadata":{}},{"cell_type":"code","source":"def clean_text(txt):\n    return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower())\n\n# does not lowercase the text\ndef clean_text2(txt):\n    return re.sub('[^A-Za-z0-9]+', ' ', str(txt))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Read train csv and create a sample (for faster demo)","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"../input/coleridgeinitiative-show-us-the-data/train.csv\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# number of unique labels\nlen(df.cleaned_label.unique()) ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create a subset for quick demo\nsample = df.sample(500)\nsample.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create the training dataset by marking entries","metadata":{}},{"cell_type":"code","source":"# get positive and negative examples for entities\nPOSITIVE_DATA = []\nNEGATIVE_DATA = []\nfor idx,row in tqdm(sample.iterrows()):\n    pub = \"../input/coleridgeinitiative-show-us-the-data/train/\" + row.Id + \".json\"\n    f = open(pub)  \n    data = json.load(f)\n    paper_text = str([sec['text'] for sec in data]).strip(\"[\").strip(\"]\")\n    sentences = paper_text.split(\".\")\n    for sentence in sentences:\n        sentence2 = clean_text(sentence) # use given clean_text to find cleaned_label\n        a = re.search(row.cleaned_label,sentence2)\n        if  a != None: # if label is found, make it a positive example\n            POSITIVE_DATA.append((clean_text2(sentence),{\"entities\":[(a.span()[0],a.span()[1],\"DATASET\")]}))\n        else: # if label is not found, make it a negative example\n            if len(clean_text2(sentence))>20: # greater than 20 chars\n                NEGATIVE_DATA.append((clean_text2(sentence),{\"entities\":[(0,0,\"DATASET\")]}))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"POSITIVE_DATA[0:10]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(POSITIVE_DATA)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(NEGATIVE_DATA)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## We have an IMBALANCED CLASS problem.\n#### For brevity, let's downsample negative class to 2000 examples","metadata":{}},{"cell_type":"code","source":"import random\nNEG_SAMPLE = random.choices(NEGATIVE_DATA, k=2000) # downsampling negative class","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TRAIN_DATA = np.array(POSITIVE_DATA + NEG_SAMPLE) # our train data is positive + negative examples\nnp.random.shuffle(TRAIN_DATA) # shuffle the train data\nlen(TRAIN_DATA) # total examples in train data","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Spacy 3.0 uses DocBin format - convert train set to this format\n####  DocBin is highly efficient serializable format used by spaCy3.0 \nUse below converter to change above train_set into new format","metadata":{}},{"cell_type":"code","source":"import spacy\nfrom spacy.tokens import DocBin\n\nnlp = spacy.blank(\"en\") # load a new spacy model\ndb = DocBin() # create a DocBin object\n\nfor text, annot in tqdm(TRAIN_DATA): # data in previous format\n    doc = nlp.make_doc(text) # create doc object from text\n    ents = []\n    for start, end, label in annot[\"entities\"]: # add character indexes\n        span = doc.char_span(start, end, label=label, alignment_mode=\"contract\")\n        if span is None:\n            pass\n        else:\n            ents.append(span)\n    doc.ents = ents # label the text with the ents\n    db.add(doc)\n\ndb.to_disk(\"./train.spacy\") # save the docbin object","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train the spaCy transformer model\nhttps://spacy.io/usage/training#quickstart","metadata":{}},{"cell_type":"code","source":"# step1: Get baseconfig file from https://spacy.io/usage/training#quickstart\n!cp \"../input/spacybaseconfigcfg/base_config.cfg\" ./","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# step2: initialize the base config file. \n# Config file contains the training settings. \n# Init with spacy init initializes it with most common settings\n!python -m spacy init fill-config base_config.cfg config.cfg","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# step3: train using spacy train command\n!python -m spacy train config.cfg --output ./output --paths.train ./train.spacy --paths.dev ./train.spacy --gpu-id 0","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Explaining Training Pipeline Variables\n\n- E is epochs\n- Loss Transformer\n- Loss NER\n- ENTS_F is f score\n- ENTS_P is precision\n- ENTS_R is recall\n- Score is to score the model (in order to pick best model later)","metadata":{}},{"cell_type":"markdown","source":"# Load the custom NER model and predict.","metadata":{}},{"cell_type":"code","source":"from thinc.api import set_gpu_allocator, require_gpu\nset_gpu_allocator(\"pytorch\")\nrequire_gpu(0)\n# Use spacy.load to load your custom model\ncustom_ner_model = spacy.load(\"./output/model-best\") # output model is stored as \"model-best\" and \"model-last\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_pubs = glob.glob(\"../input/coleridgeinitiative-show-us-the-data/test/*.json\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from spacy import displacy\n\nfor index, pub in enumerate(test_pubs):\n    f = open(pub)\n    data = json.load(f)\n    paper_text = str([sec['text'] for sec in data]).strip(\"[\").strip(\"]\")\n    sentences = paper_text.split(\".\")\n    for sentence in sentences:\n        sentence = clean_text2(sentence)\n        doc = custom_ner_model(sentence)\n        if len(doc.ents) > 0:\n            displacy.render(doc, style=\"ent\", jupyter=True)\n        ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# References\n1. https://spacy.io/usage/training","metadata":{}},{"cell_type":"markdown","source":"#### This is my first notebook on Kaggle. Your feedback and suggestions would be appreciated! - Shivam","metadata":{}}]}