{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Hi everyone!\n\nThe following code allows you to extract all entities from the training data texts using Spacy's Named Entity Recognition (NER) tool. This process was designed with the purpose of producing inputs for a classification model that would break every text as a set of candidate entities that possibly refer to a dataset. \n\nWe label each entity using the 'clean_label' column provided by the competition: if clean_label is in the entity.text.lower() string, the entity is labeled with a 1 (dataset), else a 0 (not dataset).\n\nThe outputs of the classification model would be entities with their respect probability of actually being a dataset which, given a threshold for the predictions, could be postprocessed into something like *dataset_1|dataset_2|dataset_3.*\n\n\nSince the NER task is quite memory-intensive to be executed locally using all the data, this process was coded as a script that would work on chunks of the training set saving entities into different csv files corresponding to each chunk. Once run the script in the console, delimitators for the chunks come from user inputs prompts (*input()* function) in execution.","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\n\nimport pandas as pd\npd.set_option('display.max_columns', None)\nimport numpy as np\nimport json\nimport os\nimport random\nimport string\nimport re","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')\ntrain_files_path = '../input/coleridgeinitiative-show-us-the-data/train'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Credits for the *read_append_return* function to [prashansdixit](https://www.kaggle.com/prashansdixit)  at https://www.kaggle.com/prashansdixit/coleridge-initiative-eda-baseline-model.","metadata":{"execution":{"iopub.status.busy":"2021-05-25T18:41:06.447986Z","iopub.execute_input":"2021-05-25T18:41:06.450469Z","iopub.status.idle":"2021-05-25T18:41:06.468889Z","shell.execute_reply.started":"2021-05-25T18:41:06.450019Z","shell.execute_reply":"2021-05-25T18:41:06.466767Z"}}},{"cell_type":"code","source":"def read_append_return(filename, train_files_path = train_files_path, output = 'text'):\n    json_path = os.path.join(train_files_path, (filename + '.json'))\n    headings = []\n    contents = []\n    combined = []\n    with open(json_path, 'r') as f:\n        json_decode = json.load(f)\n        for data in json_decode:\n            headings.append(data.get('section_title'))\n            contents.append(data.get('text'))\n            combined.append(data.get('section_title'))\n            combined.append(data.get('text'))\n    \n    all_headings = ' '.join(headings)\n    all_contents = ' '.join(contents)\n    all_data = '. '.join(combined)\n    \n    if output == 'text':\n        return all_contents\n    elif output == 'head':\n        return all_headings\n    else:\n        return all_data\n\n\ndef text_cleaning(text):\n    '''\n    Converts all text to lower case, Removes special charecters, emojis and multiple spaces\n    text - Sentence that needs to be cleaned\n    '''\n    text = ''.join([k for k in text if k not in string.punctuation])\n    text = re.sub('[^A-Za-z0-9]+', ' ', str(text).lower()).strip()\n    text = re.sub(' +', ' ', text)\n    emoji_pattern = re.compile(\"[\"\n                               u\"\\U0001F600-\\U0001F64F\"  # emoticons\n                               u\"\\U0001F300-\\U0001F5FF\"  # symbols & pictographs\n                               u\"\\U0001F680-\\U0001F6FF\"  # transport & map symbols\n                               u\"\\U0001F1E0-\\U0001F1FF\"  # flags (iOS)\n                               \"]+\", flags=re.UNICODE)\n    text = emoji_pattern.sub(r'', text)\n    return text","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df['text'] = train_df['Id'].apply(read_append_return)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.shape","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import spacy\n\nnlp = spacy.load('en_core_web_sm')\n#nlp.max_length = 5000000\n\n# Console execution\n# low_ = int(input('low: ')) # Low end of the chunk\n# high_ = int(input('high: ')) # High end of the chunk\n\n#Kaggle kernel\nlow_ = 0\nhigh_ = 5\n\n\ndata = []\nfor ix in train_df.iloc[low_:high_,:].index:\n    df = train_df.loc[ix]\n    text = df.text\n    true_label = df.cleaned_label\n    if len(text) < nlp.max_length:\n        data.append((ix, text, true_label))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"texts = [x[1] for x in data]\nindexes = [x[0] for x in data]\nlabs = [x[2] for x in data]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using a NER spacy pipeline, we extract all entities from the selected texts. Starting and ending position of the entity's","metadata":{}},{"cell_type":"code","source":"print('spacy pipeline')\n\nidx = 0\norgs = []\nfor doc in nlp.pipe(texts, disable=[\"tok2vec\", \"tagger\", \"parser\", \"attribute_ruler\", \"lemmatizer\"]):\n    # Do something with the doc here\n    orgs.extend([[indexes[idx], ent.text, ent.start_char, ent.end_char, ent.label_, labs[idx]]for ent in doc.ents])\n    idx += 1\n    print(idx, end = '\\r')\n\nprint(len(orgs),'entities')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Once the entities are extracted, we use the actual target variable (clean_label) to label each entity as a dataset (1) or not dataset (0).","metadata":{}},{"cell_type":"code","source":"labeled = []\nfor org in orgs:\n    if org[-1] in org[1].lower():\n        labeled.append(org + [1])\n    else:\n        labeled.append(org + [0])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We export the data into a csv files using the delimitators (variables *high_* and *low_*) for naming each file.","metadata":{}},{"cell_type":"code","source":"df = pd.DataFrame(labeled, columns = ['ix','phrase','start_char',\n                                           'end_char','ner_label','dataset_label',\n                                           'label_in_phrase'])\n\nprint('Entities dataframe shape:', df.shape)\n\noutput_file = 'ents_%s_%s.csv' % (low_,high_)\nprint('Output filename:', output_file)\ndf.to_csv(output_file)","metadata":{},"execution_count":null,"outputs":[]}]}