{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Disclaimer: I did not find the solution for this competition. Please check out the Leaderboard for winning notebooks. In this blog, my focus is text processing and labeling: how to look for scientific articles in a body of text, process the raw text and subsequently classify them with predefined labels.\n\n## INPUT NOTEBOOK\n\nEDA: https://www.kaggle.com/baotramduong/show-us-the-data-eda\n\nTags: https://www.kaggle.com/baotramduong/coleridge-initiative-get-tags\n\nN-gram Models: https://www.kaggle.com/baotramduong/coleridge-initiative-n-gram-models\n\nGRU: https://www.kaggle.com/baotramduong/coleridge-initiative-gru\n\nBidirectional LSTM: https://www.kaggle.com/baotramduong/coleridge-initiative-bidirectional-lstm\n\nsep-CNN: https://www.kaggle.com/baotramduong/coleridge-initiative-sep-cnn\n\nspaCy NER: https://www.kaggle.com/baotramduong/coleridge-initiative-spacy-ner\n\nCapstone: https://www.kaggle.com/baotramduong/coleridge-initiative-ngram-lstm-gru-spacy\n\n## BLOGS\n\nCapstone (A Summary of All Models):\n\nhttps://baotramduong.medium.com/discover-how-scientific-data-is-used-for-the-public-good-with-natural-language-processing-d9f8825e1dcf\n\nN-gram Models:\n\nhttps://baotramduong.medium.com/natural-language-processing-nlp-with-n-gram-models-coleridge-initiative-f18d5714b492\n\nDeep Learning Models:\n\nhttps://baotramduong.medium.com/natural-language-processing-nlp-with-deep-learning-models-rnn-cnn-coleridge-initiative-928f8d003b6d\n\nspaCy NER:\n\nhttps://baotramduong.medium.com/spacys-ner-model-to-identify-scientific-datasets-a336bec6190a\n\n# INTRODUCTION\n\n**Natural Language Processing (NLP)** is one of the branches of AI that gives the machines the ability to read, understand, and deliver meaning. NLP system needs to understand text, sign, and semantic properly. There are many methods that help the NLP system to understand text and symbols (Meparlad, 2020):\n\n* Text classification: the process of categorizing the text into a group of words and assigning a set of predefined tags or categories based on its context.\n\n* Vector semantic: defines semantic and interprets words meaning to explain features such as similar words and opposite words. The main idea behind vector semantic is two words are alike if they have used in a similar context. Vector semantic is useful in sentiment analysis.\n\n* Word embedding: translates spares vectors into a low-dimensional space that preserves semantic relationships. Word embedding is a type of word representation that allows words with similar meaning to have a similar representation.\n\n* Probabilistic language model: calculates the probability of a sentence of a sequence of words.\n\n* Sequence labeling: assigns a class or label to each token in a given input sequence.\n\nFor this project, the focus is text processing and labeling: we want to look for scientific articles in a body of text and subsequently classify them with predefined labels. This is a big challenge since sequence prediction is considered as one of the hardest problems to solve in the data science industry.\n\nWe will discuss how to approach any NLP problem with techniques like n-gram, sequence vector, and spaCy NER. We will look at:\n\n### Steps\n1. Obtain Data\n2. Data Scrubbing & Exploration\n3. Data Annotation: Getting Train & Test Sentences\n4. Data Preprocessing\n5. Feature Extraction\n6. Part 1: Building n-gram Models\n7. Part 2: Building RNN Models: Bidirectional LSTM & GRU Model\n8. Part 3: Building CNN Model: sep-CNN Model\n9. Part 4: Building spaCy NER Model\n10. Prediction on Test Set\n11. Prepare Submission Data\n\n## BUSINESS STATEMENTS\n\n1. Can natural language processing find the hidden-in-plain-sight data citations?\n\n2. Can machine learning find the link between the words used in research articles and the data referenced in the article?\n\n## METHODOLOGY\n\nIn this competition, natural language processing (NLP) is used to automate the discovery of how scientific data are referenced in publications. I attempted:\n\n* n-gram models (blog, n-gram notebook)\n* Recurrent neural network (RNN) deep learning models (blog, Bidirectional LSTM notebook, GRU notebook)\n* Convolutional neural network (CNN) deep learning models (blog, sep-CNN notebook)\n* spaCy NER (blog, spaCy notebook)\n\n### Metrics\n\nThe performance of the models are evaluated based on:\n* Accuracy score\n* Precision: True Negative (TN) or specificity to determine how good the model is at detecting negatives (Normal)\n* Recall: True Positive (TP) or sensitivity to determine how good the model is at detecting the positives (Pneumonia)\n* F1: harmonic mean of precision and recall\n\n## SUMMARY OF FINDINGS\n\n|#  |Model                            |Accuracy|CV  |Precision|Recall|F1  |\n|---|---------------------------------|--------|----|---------|------|----|\n|0  |CLF RandomForestClassifier       |0.75    |0.78|0.44     |0.45  |0.44|\n|1  |CLF Linear Support Vector Machine|0.62    |0.62|0.21     |0.09  |0.11|\n|2  |CLF MultinomialNB                |0.64    |0.62|0.25     |0.10  |0.12|\n|3  |DL GRU                           |0.83    |-   |0.29     |0.33  |0.3 |\n|4  |DL Bidirectional LSTM            |0.83    |-   |0.28     |0.31  |0.28|\n|5  |DL sep-CNN                       |0.48    |-   |0.0      |0.01  |0.01|\n|6  |spaCy NER                        |80.73   |-   |-        |-     |-   |\n\nThis discussion only aims at how to preprocess text and build models. Although accuracy is acceptable, Precision/ Recall/ and F1 are very low. More works are needed to be done in the future.\n\n## ACTIONABLE INSIGHTS\n\n1.With supervised learning algorithms, large annotated data for training is required which are expensive and often take a lot of time. Future efforts could be dedicated on providing more effective deep transfer learning models and exploring appropriate data augmentation techniques (He et al., 2020).\n\n2.Most neural models for sequence labeling do not scale well because when the size of data grows, the parameters of models increase exponentially, leading to the high complexity of back propagation. There exists need for developing approaches to balance model complexity and scalability (He et al., 2020).\n\n3.Increasing access to confidential data presumed significantly increasing privacy risks. However, the country’s laws and practices are not currently optimized to support the use of data for evidence building, nor in a manner that best protects privacy. We need to improve data security and privacy protections beyond what exists today (US CEP, 2017).\n\n## FUTURE WORKS\n\n* Spend more time on gathering, cleaning and visualizing data\n* Correct highly imbalanced dataset\n* More focus on tweaking the hyper-parameters","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-07-10T04:20:28.129936Z","iopub.execute_input":"2021-07-10T04:20:28.130451Z","iopub.status.idle":"2021-07-10T04:20:41.360927Z","shell.execute_reply.started":"2021-07-10T04:20:28.130343Z","shell.execute_reply":"2021-07-10T04:20:41.359815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Libraries","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport random\nimport time\nimport datetime\nfrom itertools import chain\nimport re\nimport json\nfrom tqdm.autonotebook import tqdm\nimport string\nimport collections\nfrom textblob import TextBlob\nfrom pathlib import Path\nimport collections\nimport pickle\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nfrom wordcloud import WordCloud, STOPWORDS\n\nimport nltk\nfrom nltk.probability import FreqDist\nfrom nltk.tokenize import word_tokenize\nfrom nltk.stem.wordnet import WordNetLemmatizer\n\nfrom sklearn import preprocessing\nfrom sklearn.model_selection import train_test_split, cross_val_score\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.linear_model import SGDClassifier\nfrom sklearn.svm import SVC\nfrom sklearn.naive_bayes import MultinomialNB\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.metrics import classification_report, precision_score, recall_score, f1_score, accuracy_score\nfrom sklearn.metrics import confusion_matrix, plot_confusion_matrix\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n\nfrom keras.preprocessing import sequence, text\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.callbacks import EarlyStopping, ModelCheckpoint\nfrom keras import utils\nfrom keras.models import *\nfrom keras.layers import *\nfrom keras.callbacks import *\nfrom keras.models import Sequential\nfrom keras.layers.recurrent import LSTM, GRU\nfrom keras.layers import Dense, Dropout\nfrom keras.layers.embeddings import Embedding\nfrom keras.layers.normalization import BatchNormalization\nfrom keras.utils.vis_utils import plot_model\n\nimport en_core_web_lg\nimport spacy\nfrom spacy.lang.en import English\nfrom spacy.util import minibatch, compounding\nfrom spacy import displacy\n\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:20:41.362541Z","iopub.execute_input":"2021-07-10T04:20:41.362841Z","iopub.status.idle":"2021-07-10T04:20:50.615083Z","shell.execute_reply.started":"2021-07-10T04:20:41.362809Z","shell.execute_reply":"2021-07-10T04:20:50.614215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#define stopwords\nfrom nltk.corpus import stopwords\n\nstopwords_list = stopwords.words('english') + list(string.punctuation)\nstopwords_list += [\"''\", '\"\"', '...', '``']","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:20:50.616839Z","iopub.execute_input":"2021-07-10T04:20:50.617457Z","iopub.status.idle":"2021-07-10T04:20:50.628607Z","shell.execute_reply.started":"2021-07-10T04:20:50.617411Z","shell.execute_reply":"2021-07-10T04:20:50.627468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# OBTAIN, SCRUB & EDA","metadata":{}},{"cell_type":"code","source":"#define paths\nos.listdir('/kaggle/input/coleridgeinitiative-show-us-the-data/')\ntrain_path = '../input/coleridgeinitiative-show-us-the-data/train'\ntest_path = '../input/coleridgeinitiative-show-us-the-data/test'","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:20:50.630874Z","iopub.execute_input":"2021-07-10T04:20:50.631370Z","iopub.status.idle":"2021-07-10T04:20:50.642459Z","shell.execute_reply.started":"2021-07-10T04:20:50.631322Z","shell.execute_reply":"2021-07-10T04:20:50.641452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#create a function to get the text from the JSON file and append it to the new column in table\ndef read_json_pub(filename, train_path = train_path, output = 'text'):\n    json_path = os.path.join(train_path, (filename + '.json'))\n    headings = []\n    contents = []\n    combined = []\n    with open(json_path, 'r') as f:\n        json_decode = json.load(f)\n        for data in json_decode:\n            headings.append(data.get('section_title'))\n            contents.append(data.get('text'))\n            combined.append(data.get('section_title'))\n            combined.append(data.get('text'))\n    \n    all_headings = ' '.join(headings)\n    all_contents = ' '.join(contents)\n    all_data = '. '.join(combined)\n    \n    if output == 'text':\n        return all_contents\n    elif output == 'head':\n        return all_headings\n    else:\n        return all_data","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:20:50.644195Z","iopub.execute_input":"2021-07-10T04:20:50.644637Z","iopub.status.idle":"2021-07-10T04:20:50.654760Z","shell.execute_reply.started":"2021-07-10T04:20:50.644586Z","shell.execute_reply":"2021-07-10T04:20:50.653648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#read \ntrain = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')\n\n#review\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:20:50.656289Z","iopub.execute_input":"2021-07-10T04:20:50.656686Z","iopub.status.idle":"2021-07-10T04:20:50.828945Z","shell.execute_reply.started":"2021-07-10T04:20:50.656641Z","shell.execute_reply":"2021-07-10T04:20:50.827871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get text\ntqdm.pandas()\ntrain['text'] = train['Id'].progress_apply(read_json_pub)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:20:50.830398Z","iopub.execute_input":"2021-07-10T04:20:50.830758Z","iopub.status.idle":"2021-07-10T04:21:57.529973Z","shell.execute_reply.started":"2021-07-10T04:20:50.830726Z","shell.execute_reply":"2021-07-10T04:21:57.529133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#review\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:21:57.531948Z","iopub.execute_input":"2021-07-10T04:21:57.532426Z","iopub.status.idle":"2021-07-10T04:21:57.545107Z","shell.execute_reply.started":"2021-07-10T04:21:57.532391Z","shell.execute_reply":"2021-07-10T04:21:57.544131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SCRUB\n\nBefore we can create a bag of words or vectorize each document, we need to clean it up or else we will run into the following problems:\n* Counting things that aren't actually words.\n* Punctuation and capitalization would mess up our word counts\n\nText cleaning steps vary according to the type of data and the required task. Generally, the string is converted to lowercase and punctuation is removed before text gets tokenized. With this dataset where the particular task is to find the exact matched strings of the dataset's dataset_title, dataset_label, cleaned_label, only a simple clean is needed in order to preserve the exact wording of these strings.","metadata":{}},{"cell_type":"code","source":"def clean_text(txt):\n     return re.sub('[^A-Za-z0-9.]+', ' ', str(txt).lower())","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:21:57.547066Z","iopub.execute_input":"2021-07-10T04:21:57.547349Z","iopub.status.idle":"2021-07-10T04:21:57.559936Z","shell.execute_reply.started":"2021-07-10T04:21:57.547320Z","shell.execute_reply":"2021-07-10T04:21:57.558961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## EDA\n\nFor the full EDA of this dataset, see notebook: https://www.kaggle.com/baotramduong/show-us-the-data-eda ","metadata":{}},{"cell_type":"code","source":"#create a frequency distribution to see which words are used the most\nwords = list( train['cleaned_label'].values)\nstopwords = stopwords_list\nsplit_words = []\nfor word in words:\n    lo_w = []\n    list_of_words = str(word).split()\n    for w in list_of_words:\n        if w not in stopwords:\n            lo_w.append(w)\n    split_words.append(lo_w)\nallwords = []\nfor wordlist in split_words:\n    allwords += wordlist\n    \n#get 100 most common words\nmostcommon = FreqDist(allwords).most_common(100)\nmostcommon","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:21:57.561552Z","iopub.execute_input":"2021-07-10T04:21:57.561993Z","iopub.status.idle":"2021-07-10T04:21:57.857249Z","shell.execute_reply.started":"2021-07-10T04:21:57.561947Z","shell.execute_reply":"2021-07-10T04:21:57.856332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#plot frequency distributions\nwordcloud = WordCloud(width = 1600, height = 800, \n                      background_color = 'black', \n                      colormap = 'Spectral', \n                      stopwords = stopwords_list).generate(str(mostcommon))\n\nfig = plt.figure(figsize = (20, 10), facecolor = 'white')\nplt.imshow(wordcloud, interpolation = 'bilinear')\nplt.axis('off')\nplt.title('Top 100 Most Common Words in cleaned_label', fontsize = 30)\nplt.tight_layout()\n\n#save\nplt.savefig('cleaned_label_wordcloud.png')","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:21:57.858500Z","iopub.execute_input":"2021-07-10T04:21:57.858805Z","iopub.status.idle":"2021-07-10T04:22:00.668662Z","shell.execute_reply.started":"2021-07-10T04:21:57.858773Z","shell.execute_reply":"2021-07-10T04:22:00.667525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (30, 40)),\n\nsns.countplot(y = train['cleaned_label'], \n              order = train['cleaned_label'].value_counts().index, \n              palette = 'Spectral')\nplt.ylabel('Cleaned Label',fontsize = 30)\nplt.show()\n\n#save\nplt.savefig('cleaned_label.png')","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:22:00.670355Z","iopub.execute_input":"2021-07-10T04:22:00.670768Z","iopub.status.idle":"2021-07-10T04:22:04.675139Z","shell.execute_reply.started":"2021-07-10T04:22:00.670727Z","shell.execute_reply":"2021-07-10T04:22:04.674022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# DATA ANNOTATION: GETTING SENTENCES & LABELS\n\nUsing the following code, we read each individual publication and break it down into sentences using **sent_tokenize** function from the nltk (Natural Language Toolkit), which is a leading platform for building Python programs to work with human language data. For each sentence, we will use **.search** to search for matching **dataset_title**, **dataset_label**, **cleaned_label** in each of the sentences. When a match is found, it will be returned as a tuple containing starting and ending index of the matched string and labeled as **DATASET**.\n\nTo process the sentence, we use 2 functions **clean_text** and **shorten_sentences** to process the text. Since we need to preserve the exact words for **dataset_title**, the text cleaning process is simple with removing special characters and lower-casing. Then we break the sentences into shorter sentences.","metadata":{}},{"cell_type":"code","source":"MAX_LENGTH = 64\nOVERLAP = 20\n    \ndef shorten_sentences(sentences):\n    \"\"\"\n    Sentences that have more than MAX_LENGTH words will be split\n    into multiple sentences with overlappings.\n    \"\"\"\n    short_sentences = []\n    for sentence in sentences:\n        words = sentence.split()\n        if len(words) > MAX_LENGTH:\n            for p in range(0, len(words), MAX_LENGTH - OVERLAP):\n                short_sentences.append(' '.join(words[p:p+MAX_LENGTH]))\n        else:\n            short_sentences.append(sentence)\n    return short_sentences","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:22:04.676582Z","iopub.execute_input":"2021-07-10T04:22:04.676883Z","iopub.status.idle":"2021-07-10T04:22:04.683446Z","shell.execute_reply.started":"2021-07-10T04:22:04.676855Z","shell.execute_reply":"2021-07-10T04:22:04.681977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import nltk\n\nDATA = []\nlabel_count = 0\nempty_count = 0\n\nfor idx,row in tqdm(train.iterrows()):\n    pub = \"../input/coleridgeinitiative-show-us-the-data/train/\" + row.Id + \".json\"            \n    f = open(pub)  \n    data = json.load(f)      \n\n    balanced = False\n    \n    sentences = [clean_text(sentence) for sentence in nltk.sent_tokenize(str(data))]\n    sentences = shorten_sentences(sentences) # make sentences short\n    sentences = [sentence for sentence in sentences if len(sentence) > 10] # only accept sentences with length > 10 chars\n    \n    for sentence in sentences:          \n      \n        a = re.search(row.cleaned_label.lower(), sentence)      \n        b = re.search(row.dataset_label.lower(), sentence)\n        c = re.search(row.dataset_title.lower(), sentence)\n        cleaned_label = row.cleaned_label.lower()\n        dataset_label = row.dataset_label.lower()\n        dataset_title = row.dataset_title.lower()\n        \n        if  a != None:\n            DATA.append((sentence, cleaned_label))\n            label_count = label_count + 1\n            balanced = True\n        elif b != None:\n            DATA.append((sentence, dataset_label))\n            label_count = label_count + 1\n            balanced = True\n        elif c != None:\n            DATA.append((sentence, dataset_title))\n            label_count = label_count + 1\n            balanced = True            \n        else:\n            if balanced:\n                empty_count = empty_count + 1\n                balanced = False\n    \nprint('Text with dataset:', label_count)\nprint('Text without dataset:', empty_count)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:22:04.684871Z","iopub.execute_input":"2021-07-10T04:22:04.685190Z","iopub.status.idle":"2021-07-10T04:35:05.244874Z","shell.execute_reply.started":"2021-07-10T04:22:04.685157Z","shell.execute_reply":"2021-07-10T04:35:05.244084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get dataframe\ntrain_df = pd.DataFrame(DATA)\ntrain_df = train_df.rename({0: 'Sentence', 1: 'Label'}, axis = 1)\n\n#review\ntrain_df.tail(10)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:05.246328Z","iopub.execute_input":"2021-07-10T04:35:05.246626Z","iopub.status.idle":"2021-07-10T04:35:05.294924Z","shell.execute_reply.started":"2021-07-10T04:35:05.246580Z","shell.execute_reply":"2021-07-10T04:35:05.294219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_df['Sentence'][1000])\nprint('\\n')\nprint(train_df['Label'][1000])","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:05.297935Z","iopub.execute_input":"2021-07-10T04:35:05.299279Z","iopub.status.idle":"2021-07-10T04:35:05.305396Z","shell.execute_reply.started":"2021-07-10T04:35:05.299229Z","shell.execute_reply":"2021-07-10T04:35:05.304356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_df['Sentence'][15689])\nprint('\\n')\nprint(train_df['Label'][15678])","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:05.306559Z","iopub.execute_input":"2021-07-10T04:35:05.306826Z","iopub.status.idle":"2021-07-10T04:35:05.327457Z","shell.execute_reply.started":"2021-07-10T04:35:05.306793Z","shell.execute_reply":"2021-07-10T04:35:05.319415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_df['Sentence'][23456])\nprint('\\n')\nprint(train_df['Label'][23456])","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:05.329298Z","iopub.execute_input":"2021-07-10T04:35:05.329695Z","iopub.status.idle":"2021-07-10T04:35:05.336316Z","shell.execute_reply.started":"2021-07-10T04:35:05.329654Z","shell.execute_reply":"2021-07-10T04:35:05.335124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#create a frequency distribution to see which words are used the most\nwords = list( train_df['Sentence'].values)\nstopwords = stopwords_list\nsplit_words = []\nfor word in words:\n    lo_w = []\n    list_of_words = str(word).split()\n    for w in list_of_words:\n        if w not in stopwords:\n            lo_w.append(w)\n    split_words.append(lo_w)\nallwords = []\nfor wordlist in split_words:\n    allwords += wordlist\n    \n#get 100 most common words\nmostcommon = FreqDist(allwords).most_common(100)\nmostcommon","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:05.338234Z","iopub.execute_input":"2021-07-10T04:35:05.339034Z","iopub.status.idle":"2021-07-10T04:35:12.396486Z","shell.execute_reply.started":"2021-07-10T04:35:05.338992Z","shell.execute_reply":"2021-07-10T04:35:12.395546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#plot frequency distributions\nwordcloud = WordCloud(width = 1600, height = 800, \n                      background_color = 'black', \n                      colormap = 'Spectral', \n                      stopwords = stopwords_list).generate(str(mostcommon))\n\nfig = plt.figure(figsize = (20, 10), facecolor = 'white')\nplt.imshow(wordcloud, interpolation = 'bilinear')\nplt.axis('off')\nplt.title('Top 100 Most Common Words in Sentences', fontsize = 30)\nplt.tight_layout()\n\n#save\nplt.savefig('sentence_wordcloud.png')","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:12.397711Z","iopub.execute_input":"2021-07-10T04:35:12.397989Z","iopub.status.idle":"2021-07-10T04:35:15.132661Z","shell.execute_reply.started":"2021-07-10T04:35:12.397960Z","shell.execute_reply":"2021-07-10T04:35:15.131662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (30, 40)),\n\nsns.countplot(y = train_df['Label'], \n              order = train_df['Label'].value_counts().index, \n              palette = 'Spectral')\nplt.ylabel('Sentence Label',fontsize = 30)\nplt.show()\n\n#save\nplt.savefig('sentence_label.png')","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:15.133910Z","iopub.execute_input":"2021-07-10T04:35:15.134212Z","iopub.status.idle":"2021-07-10T04:35:19.070921Z","shell.execute_reply.started":"2021-07-10T04:35:15.134180Z","shell.execute_reply":"2021-07-10T04:35:19.069958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Our learning algorithm will take in a single text input **Sentence** and output a single classification **Label**.","metadata":{}},{"cell_type":"code","source":"#save to dataframe\ntrain_df = pd.DataFrame(DATA)\ntrain_df = train_df.rename({0: 'Sentence', 1: 'Label'}, axis = 1)\n\n#review\ntrain_df.tail(10)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:19.072155Z","iopub.execute_input":"2021-07-10T04:35:19.072468Z","iopub.status.idle":"2021-07-10T04:35:19.109690Z","shell.execute_reply.started":"2021-07-10T04:35:19.072437Z","shell.execute_reply":"2021-07-10T04:35:19.108574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX = train_df['Sentence'].to_numpy()\ny = train_df['Label'].to_numpy()","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:19.114212Z","iopub.execute_input":"2021-07-10T04:35:19.114480Z","iopub.status.idle":"2021-07-10T04:35:19.119194Z","shell.execute_reply.started":"2021-07-10T04:35:19.114454Z","shell.execute_reply":"2021-07-10T04:35:19.118021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# PREPROCESSING FOR N-GRAM MODELS\n\nModels that process the tokens independently i.e. not taking into account word order, is called n-gram models. An n-gram model is the simplest model that assigns probabilities to sentences and sequences of words. An n-gram is a sequence of n-words in a sentence, for example, a 2-gram or bigram is a two-word sequence of words and a 3-gram or trigram is three-word sequence of words.\n\nWith n-gram, we discard a lot of information about word order and grammar. This is called a bag-of-words approach. This representation is used in conjunction with models that don't take ordering into account, such as logistic regression, multi-layer perceptrons, gradient boosting machines, support vector machines. \n\nAs such, for this part of the project, we will build 3 n-gram models:\n* **Model 1: RandomForestClassifier**\n* **Model 2 : Linear Support Vector Machine**\n* **Model 3: MultinomialNB**\n\n## Summary of Steps\n* Tokenize text samples into word in the range of (1, 4)-grams\n* Vectorize using tf-idf encoding\n* Select only the top 20,000 features from the vector of tokens by discarding tokens that appear fewer than 2 times and using f_classif to calculate feature importance","metadata":{}},{"cell_type":"markdown","source":"## Train-Test-Split","metadata":{}},{"cell_type":"markdown","source":"We need to split our dataset into a training and validation set. We'll use 80% of the dataset as the training data and evaluate the performance on the remaining 20% (holdout set):","metadata":{}},{"cell_type":"code","source":"#split traing data into training a validation sets\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 123)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:19.122066Z","iopub.execute_input":"2021-07-10T04:35:19.122460Z","iopub.status.idle":"2021-07-10T04:35:19.139784Z","shell.execute_reply.started":"2021-07-10T04:35:19.122419Z","shell.execute_reply":"2021-07-10T04:35:19.138767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## TF-IDF Vectorize\n\nBefore our data can be fed to a model, it needs to be transformed to a format the model can understand. Vectorization turns text into vectors.\n\nTF stands for Term Frequency and IDF stands for Inverse Document Frequency. It is the most popular way to represent documents as feature vectors. TF-IDF computes the weight of a word in a specific document, taking into account the overall distribution of words, and measures how important a particular word is with respect to a document and the entire corpus. This means that very common words will have less weight overall, while rare words will weight more.","metadata":{}},{"cell_type":"code","source":"#https://developers.google.com/machine-learning/guides/text-classification/step-3\n\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nvectorizer = TfidfVectorizer(min_df = 2,                       #ignore terms that have a document frequency strictly lower than the given threshold\n                             dtype = 'int32', \n                             strip_accents = 'unicode',\n                             analyzer = 'word',\n                             token_pattern = r'\\w{1,}',\n                             ngram_range = (1, 4), \n                             use_idf = 1,                      #enable inverse-document-frequency reweighting\n                             smooth_idf = 1,                   #weights by adding one to document frequencies\n                             sublinear_tf = 1,                 #apply sublinear tf scaling\n                             stop_words = stopwords_list)\n\n#learn vocabulary from training texts and vectorize training texts\nvectorizer = vectorizer.fit(list(X_train) + list(X_test))\n\n#transform Bag-of-Words textual data to numeric\nX_train = vectorizer.transform(X_train)\nX_test = vectorizer.transform(X_test)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:19.140906Z","iopub.execute_input":"2021-07-10T04:35:19.141177Z","iopub.status.idle":"2021-07-10T04:35:36.624951Z","shell.execute_reply.started":"2021-07-10T04:35:19.141151Z","shell.execute_reply":"2021-07-10T04:35:36.623978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We need to transform our validation data the same way we transform our training data.\n\nWhen we convert all of the texts in a dataset into word ngram_range = (1, 4), we end up with 605815 tokens. However, not all of these tokens/features contribute to label prediction.","metadata":{}},{"cell_type":"code","source":"#check shape\nprint('Train sentences:', X_train.shape, '\\n', \n      'Test sentences:', X_test.shape, '\\n') ","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:36.625999Z","iopub.execute_input":"2021-07-10T04:35:36.626270Z","iopub.status.idle":"2021-07-10T04:35:36.632544Z","shell.execute_reply.started":"2021-07-10T04:35:36.626243Z","shell.execute_reply":"2021-07-10T04:35:36.631454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Feature selection\nWe can drop certain tokens, for instance those that occur extremely rarely across the dataset. We can also measure feature importance to select for the most informative tokens. We will use the following:\n\n* **top_k = 20000** which by discarding tokens that appear fewer than 2 times to select for the top 20000 most informative features.\n\n* **SelectkBest** scores the features importance using function f_classif and then removes all but the k highest scoring features.\n\n* **selector.transform** returns a new array where the feature set has been reduced to the best k.","metadata":{}},{"cell_type":"code","source":"from sklearn.feature_selection import SelectKBest\nfrom sklearn.feature_selection import f_classif\n\n#select top 'k' of the vectorized features, limit on the number of features. We use the top 20K features\ntop_k = 20000\n\nselector = SelectKBest(f_classif, k = min(top_k, X_train.shape[1]))\nselector.fit(X_train, y_train)\nX_train = selector.transform(X_train).astype('float32')\nX_test = selector.transform(X_test).astype('float32')","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:36.633748Z","iopub.execute_input":"2021-07-10T04:35:36.634016Z","iopub.status.idle":"2021-07-10T04:35:38.825686Z","shell.execute_reply.started":"2021-07-10T04:35:36.633978Z","shell.execute_reply":"2021-07-10T04:35:38.824470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The resulting training and validation set will be (notice we only have 20000 features)","metadata":{}},{"cell_type":"code","source":"print('Train sentences:', X_train.shape, '\\n', \n      'Test sentences:', X_test.shape, '\\n')","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:38.827081Z","iopub.execute_input":"2021-07-10T04:35:38.827403Z","iopub.status.idle":"2021-07-10T04:35:38.833904Z","shell.execute_reply.started":"2021-07-10T04:35:38.827356Z","shell.execute_reply":"2021-07-10T04:35:38.832894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Encode Label\n\nThe label also needs to be converted into numerical vector. We use **LabelEncoder** to achieve this:","metadata":{}},{"cell_type":"code","source":"from sklearn import preprocessing\n\n#use the LabelEncoder to convert text labels to integers, 0, 1, 2, etc.\nencoder = preprocessing.LabelEncoder()\n\n#since we have two different data set (X_train and X_test), \n#we need to fit it on all of our data otherwise there might be some categories in the test set X_test that were not in the train set X_train \n#and we will get errors\nencoder.fit(list(y_train) + list(y_test)) \ny_train = encoder.transform(y_train)\ny_test = encoder.transform(y_test)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:38.835135Z","iopub.execute_input":"2021-07-10T04:35:38.835497Z","iopub.status.idle":"2021-07-10T04:35:39.016336Z","shell.execute_reply.started":"2021-07-10T04:35:38.835459Z","shell.execute_reply":"2021-07-10T04:35:39.015549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Pre-trained N-gram Models","metadata":{}},{"cell_type":"code","source":"#load RandomForestClassifier_model\nfilename = '../input/coleridge-initiative-n-gram-models/RandomForestClassifier_model.sav'\nRandomForestClassifier_model = pickle.load(open(filename, 'rb'))","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:39.017270Z","iopub.execute_input":"2021-07-10T04:35:39.017694Z","iopub.status.idle":"2021-07-10T04:35:49.713038Z","shell.execute_reply.started":"2021-07-10T04:35:39.017661Z","shell.execute_reply":"2021-07-10T04:35:49.712159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#test accuracy \nRandomForestClassifier_score = RandomForestClassifier_model.score(X_test, y_test)  \nprint(RandomForestClassifier_score)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:49.714593Z","iopub.execute_input":"2021-07-10T04:35:49.714848Z","iopub.status.idle":"2021-07-10T04:35:51.252541Z","shell.execute_reply.started":"2021-07-10T04:35:49.714823Z","shell.execute_reply":"2021-07-10T04:35:51.251554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#load SGDClassifier_model\nfilename = '../input/coleridge-initiative-n-gram-models/SGDClassifier_model.sav'\nSGDClassifier_model = pickle.load(open(filename, 'rb'))","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:51.253625Z","iopub.execute_input":"2021-07-10T04:35:51.253946Z","iopub.status.idle":"2021-07-10T04:35:51.438992Z","shell.execute_reply.started":"2021-07-10T04:35:51.253915Z","shell.execute_reply":"2021-07-10T04:35:51.437961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#test accuracy \nSGDClassifier_score = SGDClassifier_model.score(X_test, y_test)  \nprint(SGDClassifier_score)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:51.440437Z","iopub.execute_input":"2021-07-10T04:35:51.440726Z","iopub.status.idle":"2021-07-10T04:35:51.488730Z","shell.execute_reply.started":"2021-07-10T04:35:51.440697Z","shell.execute_reply":"2021-07-10T04:35:51.487766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#load MultinomialNB_model\nfilename = '../input/coleridge-initiative-n-gram-models/MultinomialNB_model.sav'\nMultinomialNB_model = pickle.load(open(filename, 'rb'))","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:51.490080Z","iopub.execute_input":"2021-07-10T04:35:51.490604Z","iopub.status.idle":"2021-07-10T04:35:51.796734Z","shell.execute_reply.started":"2021-07-10T04:35:51.490554Z","shell.execute_reply":"2021-07-10T04:35:51.795743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#test accuracy \nMultinomialNB_score = MultinomialNB_model.score(X_test, y_test)  \nprint(MultinomialNB_score)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:51.798093Z","iopub.execute_input":"2021-07-10T04:35:51.798388Z","iopub.status.idle":"2021-07-10T04:35:51.846710Z","shell.execute_reply.started":"2021-07-10T04:35:51.798357Z","shell.execute_reply":"2021-07-10T04:35:51.845724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# PREPROCESSING FOR DEEP LEARNING\n\n## Summary of Steps\n\n* Tokenizes the texts into words\n* Creates a vocabulary library\n* Converts the tokens into sequence vectors\n* Pads the sequences to a fixed sequence length\n\nLike all other neural networks, deep-learning models don't take as input raw text, hence the first step we need to do is to transform text into numeric tensors.","metadata":{}},{"cell_type":"markdown","source":"## Train-Test-Split\n\nWe need to split our dataset into a training and validation set. We'll use 80% of the dataset as the training data and evaluate the performance on the remaining 20% (holdout set):","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX = train_df['Sentence'].to_numpy()\ny = train_df['Label'].to_numpy()\n\n#split traing data into training a validation sets\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 123)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:51.847922Z","iopub.execute_input":"2021-07-10T04:35:51.848218Z","iopub.status.idle":"2021-07-10T04:35:51.861154Z","shell.execute_reply.started":"2021-07-10T04:35:51.848186Z","shell.execute_reply":"2021-07-10T04:35:51.860146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#check shape\nprint('Train sentences:', X_train.shape, '\\n', \n      'Test sentences:', X_test.shape, '\\n', \n      'Train labels:', y_train.shape, '\\n', \n      'Test labels:', y_test.shape)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:51.862527Z","iopub.execute_input":"2021-07-10T04:35:51.863075Z","iopub.status.idle":"2021-07-10T04:35:51.872547Z","shell.execute_reply.started":"2021-07-10T04:35:51.863021Z","shell.execute_reply":"2021-07-10T04:35:51.871751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Tokenize Features\n\nMost neural networks models will begin by breaking up a sequence of strings into individual words, phrases, or whole sentences: a process known as tokenizing. A tokenizer builds the vocabulary and converts a word sequence to an integer sequence. Each integer maps to a value in a dictionary that encodes the entire corpus, with the keys in the dictionary being the vocabulary terms themselves. \n\n* **top_k = 20000** limits on the number of features to top 20K features.\n* **fit_on_texts** updates internal vocabulary based on a list of texts. This method creates the vocabulary index based on word frequency.\n* **text_to_sequence** transforms each text in texts to a sequence of integers. It takes each word in the text and replaces it with its corresponding integer value from the word_index dictionary.\n* **pad_sequences** The LSTM layers accept sequences of the same length only but most often each text sequence has different length of words. To counter this, we can use pad_sequence which simply pads the sequence of words with zeros. Therefore, every sentence represented as integers can be padded to have the same length. We will work with the max length of the longest sequence and pad the shorter sequences. The resulting feature vector contains mostly zeros, since we have a fairly short sentence.\n* **maxlen** is also a necessary parameter to specify how long the sequences should be. This cuts sequences that exceed that number.","metadata":{}},{"cell_type":"code","source":"#https://developers.google.com/machine-learning/guides/text-classification/step-3\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\n\n#limit on the number of features. We use the top 20K features\ntop_k = 20000\n\n#limit on the length of text sequences. Sequences longer than this will be truncated\nmax_sequence_length = 100\n\n#get max sequence length\nmax_length = len(max(X_train, key = len))\nif max_length > max_sequence_length:\n    max_length = max_sequence_length\n    \nmax_vocab_length = 20000 # max number of words to have in our vocabulary\n\n#method to count the unique words in vocabulary and assign each of those words to indices\ntokenizer = Tokenizer(num_words = top_k)\n\n#create vocabulary with training texts\ntokenizer.fit_on_texts(list(X_train))\n\n#convert text into integer sequences\nX_train_seq = tokenizer.texts_to_sequences(X_train)\nX_test_seq = tokenizer.texts_to_sequences(X_test)\n         \n#fix sequence length to max value. \n#sequences shorter than the length are padded in the beginning and sequences longer are truncated at the beginning\n#this turns our lists of integers into a 2D integer tensor of shape (samples, maxlen)\nX_train_pad  = pad_sequences(X_train_seq, maxlen = max_length)\nX_test_pad = pad_sequences(X_test_seq, maxlen = max_length)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:51.873911Z","iopub.execute_input":"2021-07-10T04:35:51.874604Z","iopub.status.idle":"2021-07-10T04:35:55.875384Z","shell.execute_reply.started":"2021-07-10T04:35:51.874562Z","shell.execute_reply":"2021-07-10T04:35:55.874644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"max_length","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:55.876517Z","iopub.execute_input":"2021-07-10T04:35:55.877101Z","iopub.status.idle":"2021-07-10T04:35:55.883523Z","shell.execute_reply.started":"2021-07-10T04:35:55.877057Z","shell.execute_reply":"2021-07-10T04:35:55.882553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_pad","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:55.884643Z","iopub.execute_input":"2021-07-10T04:35:55.885109Z","iopub.status.idle":"2021-07-10T04:35:55.897343Z","shell.execute_reply.started":"2021-07-10T04:35:55.885070Z","shell.execute_reply":"2021-07-10T04:35:55.896294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test_pad","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:55.898600Z","iopub.execute_input":"2021-07-10T04:35:55.898900Z","iopub.status.idle":"2021-07-10T04:35:55.909327Z","shell.execute_reply.started":"2021-07-10T04:35:55.898871Z","shell.execute_reply":"2021-07-10T04:35:55.908321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Tokenizer must be trained on the entire training dataset, which means it finds all of the unique words in the data and assigns each a unique integer. We can access the mapping of words to integers as a dictionary attribute called word_index on the Tokenizer object. Later, when we make predictions, we can convert the prediction to numbers and look up their associated words in the same mapping.\n\nWe need to know the size of the vocabulary for defining the embedding layer later. We can determine the vocabulary by calculating the size of the mapping dictionary.","metadata":{}},{"cell_type":"code","source":"#number of unique words in the training data\nsize_of_vocabulary = len(tokenizer.word_index) + 1 #+1 for padding\nprint(size_of_vocabulary)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:55.910523Z","iopub.execute_input":"2021-07-10T04:35:55.910809Z","iopub.status.idle":"2021-07-10T04:35:55.920399Z","shell.execute_reply.started":"2021-07-10T04:35:55.910783Z","shell.execute_reply":"2021-07-10T04:35:55.919516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get word index\nword_index = tokenizer.word_index","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:55.921556Z","iopub.execute_input":"2021-07-10T04:35:55.921837Z","iopub.status.idle":"2021-07-10T04:35:55.929692Z","shell.execute_reply.started":"2021-07-10T04:35:55.921804Z","shell.execute_reply":"2021-07-10T04:35:55.928991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get number of distinct characters\nmax_id = len(tokenizer.word_index)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:55.930925Z","iopub.execute_input":"2021-07-10T04:35:55.931225Z","iopub.status.idle":"2021-07-10T04:35:55.953613Z","shell.execute_reply.started":"2021-07-10T04:35:55.931196Z","shell.execute_reply":"2021-07-10T04:35:55.952635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Encode Label\n\nWe need to one hot encode the output word. This means converting it from an integer to a vector of 0 values: one for each word in the vocabulary, with a 1 to indicate the specific word at the index of the words integer value. This is so that the model learns to predict the probability distribution for the next word and the ground truth from which to learn from is 0 for all words except the actual word that comes next.","metadata":{}},{"cell_type":"code","source":"from sklearn import preprocessing\n\n#use the LabelEncoder to convert text labels to integers, 0, 1, 2, etc.\nencoder = preprocessing.LabelEncoder()\n\n#since we have two different data set (X_train and X_test), \n#we need to fit it on all of our data otherwise there might be some categories in the test set X_test that were not in the train set X_train \n#and we will get errors\nencoder.fit(list(y_train) + list(y_test)) \ny_train = encoder.transform(y_train)\ny_test = encoder.transform(y_test)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:55.954877Z","iopub.execute_input":"2021-07-10T04:35:55.955181Z","iopub.status.idle":"2021-07-10T04:35:56.151478Z","shell.execute_reply.started":"2021-07-10T04:35:55.955154Z","shell.execute_reply":"2021-07-10T04:35:56.150474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Binarize Label\n\n* **to_categoricals** to one hot encode the output words for each input-output sequence pair.","metadata":{}},{"cell_type":"code","source":"num_classes = train_df['Label'].nunique() + 1\nnum_classes","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:56.152841Z","iopub.execute_input":"2021-07-10T04:35:56.153260Z","iopub.status.idle":"2021-07-10T04:35:56.170732Z","shell.execute_reply.started":"2021-07-10T04:35:56.153212Z","shell.execute_reply":"2021-07-10T04:35:56.169889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from keras import utils\n\n#binarize the labels for the neural net\ny_train = utils.to_categorical(y_train, num_classes)\ny_test = utils.to_categorical(y_test, num_classes)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:56.171996Z","iopub.execute_input":"2021-07-10T04:35:56.172269Z","iopub.status.idle":"2021-07-10T04:35:56.194329Z","shell.execute_reply.started":"2021-07-10T04:35:56.172243Z","shell.execute_reply":"2021-07-10T04:35:56.193412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('X_train shape:', X_train_pad.shape)\nprint('X_test shape:', X_test_pad.shape)\nprint('y_train shape:', y_train.shape)\nprint('y_test shape:', y_test.shape)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:56.195452Z","iopub.execute_input":"2021-07-10T04:35:56.195690Z","iopub.status.idle":"2021-07-10T04:35:56.202017Z","shell.execute_reply.started":"2021-07-10T04:35:56.195666Z","shell.execute_reply":"2021-07-10T04:35:56.201155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:56.203141Z","iopub.execute_input":"2021-07-10T04:35:56.203406Z","iopub.status.idle":"2021-07-10T04:35:56.214517Z","shell.execute_reply.started":"2021-07-10T04:35:56.203378Z","shell.execute_reply":"2021-07-10T04:35:56.213625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Pre-trained RNN Models: Bidirectional LSTM and GRU\n\n**Recurrent neural network (RNN)** are used when the inputs are sequential i.e reading sequentially from left to right. This is perfect for modeling languages because language is a sequence of words and each word is dependent on the words that come before it. However, RNN suffers from short-term memory and thus often suffer from vanishing gradients problem. **Long short Term Memory (LSTM)** and **Gated Recurrent Unit (GRU)** models are the solution for this. These networks have internal mechanisms called gates that can regulate the flow of information and can thus remember information for long periods of time without having to deal with the vanishing gradient problem.","metadata":{}},{"cell_type":"markdown","source":"## Pre-trained Word Vector\nSince the dataset used is small, one method of addressing this lack of data in a given domain is to leverage data from a similar domain. This means using what is learned from one task and applying that to another task without learning from scratch. Words in a given dataset are most likely not unique to that dataset. Thus we can transfer an embedding learned from another dataset into our embedding layer. These embeddings are referred to as pre-trained embeddings.\nWe use **GloVe , or Global Vectors for Word Representation** for both of our models","metadata":{}},{"cell_type":"code","source":"#load the whole embedding into memory\nembeddings_index = {}\nf = open('../input/glove840b300dtxt/glove.840B.300d.txt')\n\nfor line in tqdm(f):\n    values = line.split()\n    word = values[0]\n    try:\n        coefs = np.asarray(values[1:], dtype = 'float32')\n        embeddings_index[word] = coefs\n    except ValueError: #catch the exception where there are strings in the GloVe text file, can be avoided if use glove.42B.300d.txt\n        pass\n    \nf.close()\n\nprint('Found %s word vectors.' % len(embeddings_index))","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:35:56.215839Z","iopub.execute_input":"2021-07-10T04:35:56.216247Z","iopub.status.idle":"2021-07-10T04:38:50.020593Z","shell.execute_reply.started":"2021-07-10T04:35:56.216205Z","shell.execute_reply":"2021-07-10T04:38:50.019513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#create an embedding matrix for the words we have in the dataset\nembedding_matrix = np.zeros((len(word_index) + 1, 300))\nfor word, i in tqdm(word_index.items()):\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None:\n        embedding_matrix[i] = embedding_vector","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:38:50.022360Z","iopub.execute_input":"2021-07-10T04:38:50.022763Z","iopub.status.idle":"2021-07-10T04:38:50.227092Z","shell.execute_reply.started":"2021-07-10T04:38:50.022718Z","shell.execute_reply":"2021-07-10T04:38:50.226149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Words in a given dataset are most likely not unique to that dataset. We can thus learn the relationship between the words in our dataset using other dataset(s). To do so, we can transfer an embedding learned from another dataset into our embedding layer. These embeddings are referred to as pre-trained embeddings. Using a pre-trained embedding gives the model a head start in the learning process.\n\nThere are pre-trained embeddings available that have been trained using large corpora, such as GloVe. GloVe has been trained on multiple corpora (primarily Wikipedia)","metadata":{}},{"cell_type":"code","source":"nonzero_elements = np.count_nonzero(np.count_nonzero(embedding_matrix, axis=1))\nnonzero_elements / size_of_vocabulary","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:38:50.228743Z","iopub.execute_input":"2021-07-10T04:38:50.229161Z","iopub.status.idle":"2021-07-10T04:38:50.270168Z","shell.execute_reply.started":"2021-07-10T04:38:50.229110Z","shell.execute_reply":"2021-07-10T04:38:50.269389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Long short Term Memory networks**, or **LSTM** is a recurrent neural network that has LSTM cell blocks in place of our standard neural network layers. LSTMs have been observed as the most effective solution to solve sequence prediction problems, which is considered as one of the hardest problems to solve in the data science industry.\n\nOne of the many problems in NLP is how to understand a word's context. In some cases, a word can have completely different meanings depending on the surrounding words. This is where bidirectional RNN comes in, which requires two steps because computers need to take in token input from both directions to get the full context. The first step is a forward pass (taking in words from left to right), and the second part is a backwards pass (taking in words from right to left).","metadata":{}},{"cell_type":"code","source":"#load LSTM_model\n# load model\nLSTM_model = load_model('../input/coleridge-initiative-bidirectional-lstm/lstm_model.h5')","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:38:50.271249Z","iopub.execute_input":"2021-07-10T04:38:50.271498Z","iopub.status.idle":"2021-07-10T04:38:52.114503Z","shell.execute_reply.started":"2021-07-10T04:38:50.271474Z","shell.execute_reply":"2021-07-10T04:38:52.113734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"LSTM_result = LSTM_model.evaluate(X_test_pad, y_test)\nprint(LSTM_result)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:38:52.117890Z","iopub.execute_input":"2021-07-10T04:38:52.118215Z","iopub.status.idle":"2021-07-10T04:40:16.702893Z","shell.execute_reply.started":"2021-07-10T04:38:52.118173Z","shell.execute_reply":"2021-07-10T04:40:16.701687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Gated recurrent units (GRUs)** are a gating mechanism in RNN. The GRU is like a LSTM with a forget gate, but has fewer parameters than LSTM, as it lacks an output gate.","metadata":{}},{"cell_type":"code","source":"#load GRU_model\n# load model\nGRU_model = load_model('../input/k/baotramduong/coleridge-initiative-gru/gru_model.h5')","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:40:16.704357Z","iopub.execute_input":"2021-07-10T04:40:16.704654Z","iopub.status.idle":"2021-07-10T04:40:19.105867Z","shell.execute_reply.started":"2021-07-10T04:40:16.704625Z","shell.execute_reply":"2021-07-10T04:40:19.104953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"GRU_result = GRU_model.evaluate(X_test_pad, y_test)\nprint(GRU_result)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:40:19.107186Z","iopub.execute_input":"2021-07-10T04:40:19.107488Z","iopub.status.idle":"2021-07-10T04:42:45.268490Z","shell.execute_reply.started":"2021-07-10T04:40:19.107460Z","shell.execute_reply":"2021-07-10T04:42:45.267590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Pre-trained CNN Models: sep-CNN\n\nWhen think about **Convolutional Neural Network (CNNs)**, we typically think of Computer Vision. However, CNN can also be applied to NLP. Instead of image pixels, sentences or documents represented as a matrix are the input. Each row of the matrix corresponds to one token, typically a word, but it could be a character. In vision, our filters slide over local patches of an image, but in NLP we typically use filters that slide over full rows of the matrix (words) (Britz, 2016).\n\nUsing CNN for NLP is counterintuitive since in vision, pixels close to each other are likely to be semantically related but in language, parts of phrases could be separated by several other words. However, Britz (2016) argues, if the simple Bag of Words model, which is an obvious oversimplification with incorrect assumptions, can be the standard approach for years and with good results, CNN, arguably can also be used for NLP. Moreover, compared to something like n-gram which can quickly become expensive when computing anything more than 3-grams, CNNs are much more efficient. Let's try CNN out.","metadata":{}},{"cell_type":"code","source":"#load sepcnn_model\nsepcnn_model = load_model('../input/coleridge-initiative-sep-cnn/sepcnn_model.h5')","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:42:45.269669Z","iopub.execute_input":"2021-07-10T04:42:45.269938Z","iopub.status.idle":"2021-07-10T04:42:46.077502Z","shell.execute_reply.started":"2021-07-10T04:42:45.269910Z","shell.execute_reply":"2021-07-10T04:42:46.076780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sepcnn_result = sepcnn_model.evaluate(X_test_pad, y_test)\nprint(sepcnn_result)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:42:46.078514Z","iopub.execute_input":"2021-07-10T04:42:46.078886Z","iopub.status.idle":"2021-07-10T04:42:48.177124Z","shell.execute_reply.started":"2021-07-10T04:42:46.078857Z","shell.execute_reply":"2021-07-10T04:42:48.176139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# SPACY NER\n\nFor this part of the notebook, we will be training spaCy NER to identify DATASET entities from a body of text , a task known as named-entity recognition (NER). NER is a fundamental Natural Language Processing (NLP) task to extract entities of interest.\n\nspaCy is an open-source library for advanced Natural Language Processing in Python. It features NER, POS tagging, dependency parsing, word vectors and more. It is designed specifically to build applications that process and understand large volumes of text. It can be used to build information extraction or natural language understanding systems, or to pre-process text for deep learning.\n\nThe first step is to build a universal model that is capable of distinguishing between entities and non-entities. ","metadata":{}},{"cell_type":"code","source":"DATA = []\nent_count = 0\nempty_count = 0\n\nfor idx,row in tqdm(train.iterrows()):\n    pub = '../input/coleridgeinitiative-show-us-the-data/train/' + row.Id + '.json'            \n    f = open(pub)  \n    data = json.load(f)      \n    \n    balanced = False\n\n    sentences = [clean_text(sentence) for sentence in nltk.sent_tokenize(str(data))]\n    sentences = shorten_sentences(sentences) # make sentences short\n    sentences = [sentence for sentence in sentences if len(sentence) > 10] # only accept sentences with length > 10 chars\n    \n    for sentence in sentences:          \n\n        a = re.search(row.cleaned_label.lower(), sentence)\n        b = re.search(row.dataset_label.lower(), sentence)\n        c = re.search(row.dataset_title.lower(), sentence)\n        \n        if  a != None:\n            DATA.append((sentence, {\"entities\":[(a.span()[0], a.span()[1], \"DATASET\")]}))\n            ent_count = ent_count + 1\n            balanced = True\n            \n        elif b != None:\n            DATA.append((sentence, {\"entities\":[(b.span()[0], b.span()[1], \"DATASET\")]}))\n            ent_count = ent_count + 1\n            balanced = True\n            \n        elif c != None:\n            DATA.append((sentence, {\"entities\":[(c.span()[0], c.span()[1], \"DATASET\")]}))\n            ent_count = ent_count + 1\n            balanced = True\n            \n        else:\n            if balanced:\n                DATA.append((sentence, {'entities':[]}))\n                empty_count = empty_count + 1\n                balanced = False\n                \nprint('Text with entities:', ent_count)\nprint('Text without entities:', empty_count)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:42:48.178450Z","iopub.execute_input":"2021-07-10T04:42:48.178767Z","iopub.status.idle":"2021-07-10T04:53:38.154657Z","shell.execute_reply.started":"2021-07-10T04:42:48.178699Z","shell.execute_reply":"2021-07-10T04:53:38.153457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.DataFrame(DATA)\ntrain_df = train_df.rename({0: 'Sentence', 1: 'Entities'}, axis = 1)\n\n#review\ntrain_df.tail(10)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:53:38.156381Z","iopub.execute_input":"2021-07-10T04:53:38.156783Z","iopub.status.idle":"2021-07-10T04:53:38.238958Z","shell.execute_reply.started":"2021-07-10T04:53:38.156737Z","shell.execute_reply":"2021-07-10T04:53:38.237835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#load spaCy model\nfilename = '../input/k/baotramduong/coleridge-initiative-spacy-ner/dataset_entity_extractor.p'\nnlp = pickle.load(open(filename, 'rb'))","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:53:38.240596Z","iopub.execute_input":"2021-07-10T04:53:38.241009Z","iopub.status.idle":"2021-07-10T04:53:39.540383Z","shell.execute_reply.started":"2021-07-10T04:53:38.240966Z","shell.execute_reply":"2021-07-10T04:53:39.539385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# PREDICTION","metadata":{}},{"cell_type":"code","source":"from functools import partial\n\n#read data\nsample_submission = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv')\n\n#apply the function to submission data\ntqdm.pandas()\nsample_submission['text'] = sample_submission['Id'].progress_apply(partial(read_json_pub, train_path = test_path))\n\n#review\nsample_submission.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:53:39.541911Z","iopub.execute_input":"2021-07-10T04:53:39.542322Z","iopub.status.idle":"2021-07-10T04:53:39.629562Z","shell.execute_reply.started":"2021-07-10T04:53:39.542279Z","shell.execute_reply":"2021-07-10T04:53:39.628920Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp_1 = [x.lower() for x in train['dataset_label'].unique()]\ntemp_2 = [x.lower() for x in train['dataset_title'].unique()]\ntemp_3 = [x.lower() for x in train['cleaned_label'].unique()]\n\nexisting_labels = set(temp_1 + temp_2 + temp_3)","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:53:39.630742Z","iopub.execute_input":"2021-07-10T04:53:39.631306Z","iopub.status.idle":"2021-07-10T04:53:39.642779Z","shell.execute_reply.started":"2021-07-10T04:53:39.631264Z","shell.execute_reply":"2021-07-10T04:53:39.642174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"literal_matching = True\nRandomForestClassifier_prediction = True\nSGDClassifier_prediction = True\nMultinomialNB_prediction = True\nlstm_prediction = True\ngru_prediction = True\nsepcnn_prediction = True\nspacy_prediction = True\n\nid_list = []\nlabels_list = []\n\nfor index, row in tqdm(sample_submission.iterrows()):\n\n    sample_text = row['text']\n\n    row_id = row['Id']\n    \n    #takes only the rows where train file is identical to a test file\n    temp_df = train[train['text'] == clean_text(sample_text)]\n    cleaned_labels = temp_df['cleaned_label'].to_list()\n    \n    #literal_matching \n    if literal_matching:\n        for known_label in existing_labels:\n            if known_label in sample_text.lower():    \n                cleaned_labels.append(clean_text(known_label))\n            \n        print('cleaned label:', set(cleaned_labels))   \n\n     #classifier \n    if RandomForestClassifier_prediction:\n        \n        #extract sentences\n        sentences = [clean_text(sentence) for sentence in nltk.sent_tokenize(str(sample_text))]\n        sentences = shorten_sentences(sentences) # make sentences short\n        sentences = [sentence for sentence in sentences if len(sentence) > 10] # only accept sentences with length > 10 chars\n    \n        #predict\n        RandomForestClassifier_labels = RandomForestClassifier_model.predict(selector.transform(vectorizer.transform(sentences).toarray()))\n    \n        #get labels\n        RandomForestClassifier_labels = encoder.inverse_transform(RandomForestClassifier_labels) \n        print('RandomForestClassifier_label:', set(RandomForestClassifier_labels))\n        RandomForestClassifier_labels = set(RandomForestClassifier_labels)\n        \n    cleaned_labels += RandomForestClassifier_labels\n\n    if MultinomialNB_prediction:\n        \n        #extract sentences\n        sentences = [clean_text(sentence) for sentence in nltk.sent_tokenize(str(sample_text))]\n        sentences = shorten_sentences(sentences) # make sentences short\n        sentences = [sentence for sentence in sentences if len(sentence) > 10] # only accept sentences with length > 10 chars\n        \n        #predict\n        MultinomialNB_labels = MultinomialNB_model.predict(selector.transform(vectorizer.transform(sentences).toarray()))\n    \n        #get labels\n        MultinomialNB_labels = encoder.inverse_transform(MultinomialNB_labels) \n        print('MultinomialNB_label:', set(MultinomialNB_labels))\n        MultinomialNB_labels = set(MultinomialNB_labels)\n        \n    cleaned_labels += MultinomialNB_labels\n    \n    if SGDClassifier_prediction:\n        \n        #extract sentences\n        sentences = [clean_text(sentence) for sentence in nltk.sent_tokenize(str(sample_text))]\n        sentences = shorten_sentences(sentences) # make sentences short\n        sentences = [sentence for sentence in sentences if len(sentence) > 10] # only accept sentences with length > 10 chars\n    \n        #predict\n        SGDClassifier_labels = SGDClassifier_model.predict(selector.transform(vectorizer.transform(sentences).toarray()))\n    \n        #get labels\n        SGDClassifier_labels = encoder.inverse_transform(SGDClassifier_labels) \n        print('SGDClassifier_label:', set(SGDClassifier_labels))\n        SGDClassifier_labels = set(SGDClassifier_labels)\n        \n    cleaned_labels += SGDClassifier_labels       \n        \n    #lstm_prediction \n    if lstm_prediction:\n        \n        #extract sentences\n        sentences = [clean_text(sentence) for sentence in nltk.sent_tokenize(str(sample_text))]\n        sentences = shorten_sentences(sentences) # make sentences short\n        sentences = [sentence for sentence in sentences if len(sentence) > 10] # only accept sentences with length > 10 chars\n        \n        tokenizer.fit_on_texts([sentence])\n        sentence_seq = tokenizer.texts_to_sequences([sentence])\n        sentence_pad  = pad_sequences(sentence_seq, maxlen = max_length)\n            \n        #predict\n        lstm_labels = LSTM_model.predict(sentence_pad)\n    \n        #get label\n        lstm_labels = encoder.inverse_transform([np.argmax(lstm_labels)])\n        print('lstm label:', set(lstm_labels))\n        lstm_labels = set(lstm_labels)\n        \n    cleaned_labels += lstm_labels\n\n    #gru_prediction \n    if gru_prediction:\n        \n        #extract sentences\n        sentences = [clean_text(sentence) for sentence in nltk.sent_tokenize(str(sample_text))]\n        sentences = shorten_sentences(sentences) # make sentences short\n        sentences = [sentence for sentence in sentences if len(sentence) > 10] # only accept sentences with length > 10 chars\n        \n        tokenizer.fit_on_texts(list(sentence))\n        sentence_seq = tokenizer.texts_to_sequences([sentence])\n        sentence_pad  = pad_sequences(sentence_seq, maxlen = max_length)\n            \n        #predict\n        gru_labels = GRU_model.predict(sentence_pad)\n    \n        #get label\n        gru_labels = encoder.inverse_transform([np.argmax(gru_labels)])\n        print('gru label:', set(gru_labels))\n        gru_labels = set(gru_labels)\n        \n    cleaned_labels += gru_labels   \n\n    #sepcnn_prediction \n    if sepcnn_prediction:\n        \n        #extract sentences\n        sentences = [clean_text(sentence) for sentence in nltk.sent_tokenize(str(sample_text))]\n        sentences = shorten_sentences(sentences) # make sentences short\n        sentences = [sentence for sentence in sentences if len(sentence) > 10] # only accept sentences with length > 10 chars\n        \n        tokenizer.fit_on_texts(list(sentence))\n        sentence_seq = tokenizer.texts_to_sequences([sentence])\n        sentence_pad  = pad_sequences(sentence_seq, maxlen = max_length)\n            \n        #predict\n        sepcnn_labels = sepcnn_model.predict(sentence_pad)\n    \n        #get label\n        sepcnn_labels = encoder.inverse_transform([np.argmax(sepcnn_labels)])\n        print('sepcnn label:', set(sepcnn_labels))\n        sepcnn_labels = set(sepcnn_labels)\n        \n    cleaned_labels += sepcnn_labels  \n    \n    #spaCy NER \n    if spacy_prediction:\n        \n        #extract sentences\n        sentences = [clean_text(sentence) for sentence in nltk.sent_tokenize(str(sample_text))]\n        sentences = shorten_sentences(sentences) # make sentences short\n        sentences = [sentence for sentence in sentences if len(sentence) > 10] # only accept sentences with length > 10 chars\n    \n        #for each entity, use our updated model to make a prediction on the sentence\n        for sentence in sentences:\n            \n            doc = nlp(sentence)\n\n            for ent in doc.ents:\n                spacy_labels = ent.text\n                \n        print('spacy label:', spacy_labels)\n\n    cleaned_labels.append(spacy_labels)    \n    \n    #get labels\n    cleaned_labels = set(cleaned_labels)\n    cleaned_labels = [clean_text(x) for x in cleaned_labels]    \n    labels_list.append('|'.join(cleaned_labels))\n    print('label list:', labels_list)   \n    id_list.append(row_id)\n    print('\\n')","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:53:39.644007Z","iopub.execute_input":"2021-07-10T04:53:39.644598Z","iopub.status.idle":"2021-07-10T04:54:16.124485Z","shell.execute_reply.started":"2021-07-10T04:53:39.644557Z","shell.execute_reply":"2021-07-10T04:54:16.121068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get dataframe\nsample_submission['PredictionString'] = labels_list\nsample_submission.drop(columns = 'text', axis = 1, inplace = True)\nsample_submission","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:54:16.125702Z","iopub.execute_input":"2021-07-10T04:54:16.125995Z","iopub.status.idle":"2021-07-10T04:54:16.137778Z","shell.execute_reply.started":"2021-07-10T04:54:16.125964Z","shell.execute_reply":"2021-07-10T04:54:16.136705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(sample_submission['PredictionString'][0])\nprint('\\n')\nprint(sample_submission['PredictionString'][1])\nprint('\\n')\nprint(sample_submission['PredictionString'][2])\nprint('\\n')\nprint(sample_submission['PredictionString'][3])","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:54:16.139446Z","iopub.execute_input":"2021-07-10T04:54:16.139839Z","iopub.status.idle":"2021-07-10T04:54:16.153179Z","shell.execute_reply.started":"2021-07-10T04:54:16.139797Z","shell.execute_reply":"2021-07-10T04:54:16.152038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#save\nsample_submission.to_csv('submission.csv', index = False)\n\n#check\nsubmission = pd.read_csv('submission.csv')\nsubmission","metadata":{"execution":{"iopub.status.busy":"2021-07-10T04:54:16.154377Z","iopub.execute_input":"2021-07-10T04:54:16.154636Z","iopub.status.idle":"2021-07-10T04:54:16.175006Z","shell.execute_reply.started":"2021-07-10T04:54:16.154609Z","shell.execute_reply":"2021-07-10T04:54:16.174215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# REFERENCE\n\nBanerjee, I., Ling, Y., Chen, M. C., Hasan, S. A., Langlotz, C. P., Moradzadeh, N., Chapman, B., Amrhein, T., Mong, D., Rubin, D. L., Farri, O., & Lungren, M. P. (2019). Comparative effectiveness of convolutional neural network (CNN) and recurrent neural network (RNN) architectures for radiology text report classification. Artificial intelligence in medicine, 97, 79–88. https://doi.org/10.1016/j.artmed.2018.11.004\n\nBritz, D. (2016, January 10). Understanding Convolutional Neural Networks for NLP. WildML. http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/.\n\nChollet, F. (2017). Chapter 6. Deep learning for text and sequences. Deep Learning with Python. · Deep Learning with Python. https://livebook.manning.com/book/deep-learning-with-python/chapter-6/18.\n\nGoogle. (n.d.). Step 4: Build, Train, and Evaluate Your Model. Google. https://developers.google.com/machine-learning/guides/text-classification/step-4.\n\nHe, Z., Wang, Z., Wei, W., Feng, S., Mao, X., Jiang, S. (2020). A Survey on Recent Advances in Sequence Labeling from Deep Learning Models. arXiv:2011.06727v1 [cs.CL]. Cornell University.\n\nMeparlad. (2020, December 11). Text Classification in Natural Language Processing. Analytics Vidhya. https://www.analyticsvidhya.com/blog/2020/12/understanding-text-classification-in-nlp-with-movie-review-example-example/.\n\nNatural Language Toolkit. Natural Language Toolkit — NLTK 3.6.2 documentation. (n.d.). https://www.nltk.org/.\n\nPennington, J. (n.d.). GloVe: Global Vectors for Word Representation. https://nlp.stanford.edu/projects/glove/.\n\nPhi, M. (2020, June 28). Illustrated Guide to LSTM’s and GRU’s: A step by step explanation. Medium. https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21.\n\nspaCy 101: Everything you need to know · spaCy Usage Documentation. spaCy 101: Everything you need to know. (n.d.). https://spacy.io/usage/spacy-101.\n\nUnited States Commission on Evidence-Based Policymaking. (2017). The Promise of Evidence-Based Policymaking: Report of the Commission on Evidence-Based Policymaking.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}