{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This notebook wants to record my process of learning NLP techniques from almost 0 to a level decent enough to participate to the [Coleridge Initiative - Show US the Data](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data), which ends in 3 months. Since I expect the [Natural Language Processing with Disaster Tweets](https://www.kaggle.com/c/nlp-getting-started) to be more approacheable, I will focus on that first.\n\nI never had the chance to work with NLP in my professional experience and I want to make it an option for the future.\n\nIt is meant to be more of a diary for myself and it is made public mainly to hold myself accountable for making a significant progress in these 3 months. The starting point is [the Kaggle course on NLP](https://www.kaggle.com/learn/natural-language-processing), which is giving already some nice information but I feel I have to dig a bit deeper to be able to do some analysis without copying.\n\nGiven that the course uses spaCy, a quick online search suggests it is a good starting point. We will see where to go from there.\n\n**Do not expect a brillian notebook, nor a high scoring one. It will most likely be a fairly pedantic exploration of functionalities I don't know yet. Feel free to drop a suggestion in the comments**\n\n\n***Day count = 6***","metadata":{}},{"cell_type":"code","source":"!pip install tubesml==0.4.2","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd\n\nimport spacy\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load an NLP model","metadata":{}},{"cell_type":"markdown","source":"The very first thing every tutorial I found do is to load a model with SpaCy. They do so by running a load method, the resulting object has quite a few methods and I am sure we are going to need a few of those shortly","metadata":{}},{"cell_type":"code","source":"nlp = spacy.load('en')\n[m for m in dir(nlp) if '__' not in m]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the [documentation](https://spacy.io/usage/models#languages), I can see that several languages are supported, some more than other as one would expect. For the English language, for example, I see they provide\n\n* language data: it contains stopwords, some language exceptions, and other things I can't recognize. The language expections for the tokenizer (**check it later**) does not seem a comprehensive list, for example in italian there is nord-est (north-east), but not nord-ovest (north-west), but it seems aimed to cover all the common contractions like I'm or You're. \n* pipelines: as the name suggests, they are a list of opertations you can perform in a certain order. The models are pretrained and in the documentation there is some accuracy value for each component. **I don't know accuracy against what, I will check it later**.\n\n# Tokenizer and Lemma\n\nI can use the loaded model to analyze a text as follows","metadata":{}},{"cell_type":"code","source":"doc = nlp(\"This is my first sentence I process, I don't know what is going to happen. Do you?\")\ndoc.to_json()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems it automatically detects the sentences by looking for a full stop, my bad grammar must be a nightmare for that.","metadata":{}},{"cell_type":"code","source":"for sent in doc.sents:\n    print(sent)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see if it is only the full stop","metadata":{}},{"cell_type":"code","source":"doc = nlp(\"This is my first sentence I process. I don't know what is going to happen... We will see. Do you?\")\nfor sent in doc.sents:\n    print(sent)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And then it already detected the tokens, which are units of text like words or punctuation.","metadata":{}},{"cell_type":"code","source":"for token in doc:\n    print(f'Token: {token},\\t\\tBase form: {token.lemma_},\\t\\t\\tPart of speech: {token.pos_}\\t\\t\\tSentiment: {token.sentiment} ')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looking at the json above, it also looks like the model finds, for example, if it is a pronoun or a verb. I can access this information via the [token attributes](https://spacy.io/api/token).\n\nCan I just do this for a full dataframe?","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/nlp-getting-started/train.csv')\n\ntokens = []  # from https://stackoverflow.com/questions/44395656/applying-spacy-parser-to-pandas-dataframe-w-multiprocessing\nlemma = []\npos = []\n\n# I am curious about what is the time cost of each operation, don't mind the time parts\nfrom time import time\ntok_time = []\nlem_time = []\npo_time =[]\n\ntot_s = time()\nfor doc in nlp.pipe(df['text'].values, batch_size=50, n_process=4):\n    if doc.is_parsed:\n        s = time()\n        tokens.append([n.text for n in doc])\n        tok_time.append(time() - s)\n        s = time()\n        lemma.append([n.lemma_ for n in doc])\n        lem_time.append(time() - s)\n        s = time()\n        pos.append([n.pos_ for n in doc])\n        po_time.append(time() - s)\n    else:\n        # We want to make sure that the lists of parsed results have the\n        # same number of entries of the original Dataframe, so add some blanks in case the parse fails\n        tokens.append(None)\n        lemma.append(None)\n        pos.append(None)\n        \ntot_time = time() - tot_s\n\ndf['tokens'] = tokens\ndf['lemma'] = lemma\ndf['pos'] = pos\n\ndf[['text', 'tokens', 'lemma', 'pos']].sample(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Not the greatest achievement in the history of achievements but yes, I can. **Not sure how it would work with a very large dataframe, I guess I'll find out later**.\n\nThe new thing I used here is [nlp.pipe](https://spacy.io/api/language#pipe), which process the text as a stream and allows for parallelization and batches (I suppose this would make it more memory friendly). Out of curiosity, I timed each operation in the hope of seeing where the computation time goes","metadata":{}},{"cell_type":"code","source":"print(f'Total time for entire dataframe: \\t\\t{tot_time}')\nprint(f'Mean time of tokenizer: \\t\\t{np.mean(tok_time)} +- {np.std(tok_time)}')\nprint(f'Mean time of lemmatizer: \\t\\t{np.mean(lem_time)} +- {np.std(lem_time)}')\nprint(f'Mean time of morphologizer: \\t\\t{np.mean(po_time)} +- {np.std(po_time)}')\nprint(f'Total time for the model to run without the above operations: \\t\\t{tot_time - np.sum(tok_time) - np.sum(lem_time) - np.sum(po_time)}')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In other words, almost all the time goes into the operation `nlp(...)`, which makes sense since each element we extract is simply an attribute of the resulting object. Hence, I should try to do that operation as fewer times as possible.\n\n### Takeaway of this section\n\nSpaCy offers nice pretrained models that can extract a lot of features from a string. The operation can be parallelized easily, which can compensate from the (at least for me) impossibility of leveraging the pandas indexing.\n\n# Pattern matching\n\nThe second topic in many tutorials, Kaggle's included, is about how to match tokens or phrases within a document. It feels like a natural second step of this journey.\n\nI am going to follow what SpaCy does [in its documentation](https://spacy.io/usage/rule-based-matching).\n\nTo match tokens based on rules we set, we can use `Matcher`","metadata":{}},{"cell_type":"code","source":"from spacy.matcher import Matcher\n\n[m for m in dir(Matcher) if '__' not in m]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# nlp is our model for english we loaded earlier\n\nmatcher = Matcher(nlp.vocab, validate=True)  # initialize object that shares the same vocabulary\n# validate will tell the matcher to validate the patterns provided against the vocabulary and, if necessary, raise an error\n\n# Create a pattern as a list of dictionaries\npattern = [{\"LOWER\": \"hello\"}, {\"IS_PUNCT\": True}, {\"LOWER\": \"world\"}]\n# add the pattern to the matcher with an ID\nmatcher.add(\"HelloWorld\", [pattern])\n\ndoc = nlp(\"Hello, world! Hello World! These are other words, like hello, but not world. Hello,world\")\nmatches = matcher(doc)\nfor match_id, start, end in matches:\n    string_id = nlp.vocab.strings[match_id]  # Get string representation\n    span = doc[start:end]  # The matched span\n    print(string_id, start, end, span.text)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thus to add a pattern to the matcher, we make a list of lists of patterns we look for. Each list of pattern is a list of dictionaries that define the sequence of tokens that we want.\n\nTo test this, let's make more patterns with different IDs","metadata":{}},{"cell_type":"code","source":"patterns = [\n    [{\"LOWER\": \"hello\"}, {\"IS_PUNCT\": True}, {\"LOWER\": \"world\"}],\n    [{\"LOWER\": \"hello\"}, {\"LOWER\": \"world\"}]\n]\nmatcher.add(\"HelloWorld\", patterns)\n\npattern = [{'TEXT': 'world'}]  # it should not then find World\nmatcher.add('World', [pattern])\n\nmatches = matcher(doc)\nfor match_id, start, end in matches:\n    string_id = nlp.vocab.strings[match_id]  # Get string representation\n    span = doc[start:end]  # The matched span\n    print(string_id, start, end, span.text)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It also appear you can overwrite an existing pattern by simply adding it with the same ID (as I did for `HelloWorld`). We can access the patterns with the `_patterns` attribute","metadata":{}},{"cell_type":"code","source":"matcher._patterns","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Interestingly, it doesn't really overwrite but rather add another element to the list. This is convenient to add new patterns without making new IDs every time. Indeed there are other methods to remove and get these patterns","metadata":{}},{"cell_type":"code","source":"matcher.get('HelloWorld')  # give the ID of the pattern you want","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"matcher.remove('World')  # remove a pattern \n# or throw an error if it doesn't exist","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I see from the docs we can also use regular expressions to match the patterns but the possibility of doing something when a match is found seems more interesting. \n\nFor this, there the `on_match` option when we add a new pattern. It has to be a function that takes the matcher, the document, an id of the match, and the match.","metadata":{}},{"cell_type":"code","source":"def callback_on_match(matcher, doc, id, matches):\n    print('Matched!', id)\n    \npattern = [{'LOWER': 'world'}]  \nmatcher.add('World', [pattern], on_match=callback_on_match)\n\n_ = matcher(doc)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To match phrases we can instead use the [`PhraseMatcher`](https://spacy.io/api/phrasematcher), which differs from the `Matcher` as it accepts patterns in the form of a Doc. \n\nA Doc is a sequence of Tokens","metadata":{}},{"cell_type":"code","source":"from spacy.matcher import PhraseMatcher\n\n[m for m in dir(PhraseMatcher) if '__' not in m]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It initializes as before with a vocabulary that has to be the same of the model we are using, but it also has the possibility of setting the token attribute to match on.","metadata":{}},{"cell_type":"code","source":"matcher = PhraseMatcher(nlp.vocab)\nterms = ['NLP', 'difficulty', 'interaction']\n\npatterns = [nlp.make_doc(text) for text in terms]\nmatcher.add(\"TerminologyList\", patterns)\n\ndoc = nlp(\"This journey into NLP is not easy and take patience \"\n          \"the difficulty is in finding interactions between these techinques and the one that I already know\")\nmatches = matcher(doc)\nfor match_id, start, end in matches:\n    span = doc[start:end]\n    print(span.text)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It feels just a more user-friendly version of the Matcher above.\n\nMore interesting seems to be the [`DependencyMatcher`](https://spacy.io/api/dependencymatcher), which allows to match dependency trees, but *I will leave this topic for another time.*\n\n# Text Classification\n\nThis appears to be the entry level for machine learning. Time to use those competition datasets","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/nlp-getting-started/train.csv')\ndf.head()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Machine Learning practitioner that is in me tells me I need to prepare the data for the model. It is my understanding that this would be just another component in the NLP pipeline. Following the Kaggle course","metadata":{}},{"cell_type":"code","source":"nlp = spacy.blank(\"en\")\n\n# Create the TextCategorizer with exclusive classes and \"bow\" architecture\ntextcat = nlp.create_pipe(\n              \"textcat\",\n              config={\n                \"exclusive_classes\": True,\n                \"architecture\": \"bow\"})\n\n# Add the TextCategorizer to the empty model\nnlp.add_pipe(textcat)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, here is me trying to be less confused:\n\n* `textcat` looked like a simple name to give to the pipeline step but I can't change it, so the next cell will show the available names\n* `bow` stands for bag-of-words, which is how we want to represent the data. This model disregard grammar or word order. It essentially assigns to each word in the document a number that represents the times that word occurs in the document.\n\nNow, I am struggling in finding a good list of available options but I can see that `textcat` is a name available here","metadata":{}},{"cell_type":"code","source":"nlp.factories","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And I can find some model architectures, like bag-of-words, here: https://spacy.io/api/architectures#TextCatBOW.\n\nDigging deeper into the documentation, here is the list of [built-in pipeline components](https://spacy.io/usage/processing-pipelines#built-in)\n\nMoreover, I can access to the pipeline names at any time via","metadata":{}},{"cell_type":"code","source":"nlp.pipe_names","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The list is very short because I started by using `spacy.blank()`, which creates a blank pipeline. A more traditional approach is to load a model and in that case the pipeline shows more pretrained transformers.\n\nEven longer if we load a model more complex than `en`.","metadata":{}},{"cell_type":"code","source":"nlp = spacy.load(\"en\")\n\n# Create the TextCategorizer with exclusive classes and \"bow\" architecture\ntextcat = nlp.create_pipe(\n              \"textcat\",\n              config={\n                \"exclusive_classes\": True,\n                \"architecture\": \"bow\"})\n\n# Add the TextCategorizer to the empty model\nnlp.add_pipe(textcat)\nnlp.pipe_names","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nlp = spacy.blank(\"en\")\n\n# Create the TextCategorizer with exclusive classes and \"bow\" architecture\ntextcat = nlp.create_pipe(\n              \"textcat\",\n              config={\n                \"exclusive_classes\": True,\n                \"architecture\": \"bow\"})\n\n# Add the TextCategorizer to the empty model\nnlp.add_pipe(textcat)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Moving on to our model, I need to add labels","metadata":{}},{"cell_type":"code","source":"# Add labels to text classifier\ntextcat.add_label('disaster')\ntextcat.add_label('not-disaster')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The `TextCategorizer` requires the label to be in a dictionary of boolean values for each class and each entry.","metadata":{}},{"cell_type":"code","source":"train_texts = df['text'].values\ntrain_labels = [{'cats': {'disaster': label == 1, 'not-disaster': label == 0}} for label in df['target']]\n\ntrain_data = list(zip(train_texts, train_labels))\ntrain_data[:3]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now I can train the model. This is done in batches and epochs. I need an optimizer and the Kaggle course suggests to use `begin_training`","metadata":{}},{"cell_type":"code","source":"from spacy.util import minibatch\n\nspacy.util.fix_random_seed(1)\noptimizer = nlp.begin_training()\n[d for d in dir(optimizer) if '__' not in d]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This method will be called `initialize` in the future and, by default, it returns an [SGD optimizer](https://thinc.ai/docs/api-optimizers).\n\nThe loop over epochs is very intuitive, the one over batches deserves some investigation","metadata":{}},{"cell_type":"code","source":"batches = minibatch(train_data, size=5)  # this is a generator\ni = 0\nfor batch in batches:\n    if i > 0:  # this is ugly code to just see the first batch\n        continue\n    print(batch)\n    i += 1","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So it is a list (text, label) but to perform the training we need separate lists for texts and labels, so we need to zip it.","metadata":{}},{"cell_type":"code","source":"t, l = zip(*batch)  # this is actually the last batch of the previous loop\nprint(l)\nt","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import random\n\nrandom.seed(13)\nspacy.util.fix_random_seed(48)\n\nlosses = {}\nfor epoch in range(10):\n    random.shuffle(train_data)  # to avoid getting stuck in suboptimal solutions\n    # Create the batch generator with batch size = 10\n    batches = minibatch(train_data, size=10)\n    # Iterate through minibatches\n    for batch in batches:\n        texts, labels = zip(*batch)\n        nlp.update(texts, labels, sgd=optimizer, losses=losses)\n    print(losses)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Many things are going on here, I need to pause.\n\n* `nlp.update` updates the models in the pipeline. We only have `textcat` so that's the only model that get updated. We have to provide an optimizer and it updates the dictionary of losses. It also allows for a dropout rate.\n* The loss is somewhat mysterious. I would expect it to go down and I struggle in finding what loss function are we talking about. The optimizer has and `L2` attribute so that would be a good candidate but I am not sure why is it then increasing with the training. **I need to search more about this** Further research tells me that the loss is the mean squared error and that it is increasing because I should be resetting it in each epoch. (More details at the end of the section).\n\nThe model is trained, I want to predict something with it. If I make up a sentence and ask for its classification, I need to \n* tokenize it\n* estract the pipeline component I need to make the prediction\n* make the prediction","metadata":{}},{"cell_type":"code","source":"texts = [\"this is a calm tweet, shiny day\",\n         \"everything is on fire\", \n         \"the party is on fire\", \n         'I am bombing this test']\ndocs = [nlp.tokenizer(text) for text in texts]\ndocs","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use textcat to get the scores for each doc\ntextcat = nlp.get_pipe('textcat')\nscores, _ = textcat.predict(docs)\n\nprint(scores)\n\n# From the scores, find the label with the highest score/probability\npredicted_labels = scores.argmax(axis=1)\nprint([textcat.labels[label] for label in predicted_labels])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Which is surprisingly good given my expectations. \n\nIf I want to predict on the test set, I can do the following","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/nlp-getting-started/test.csv')\ntest_texts = df['text'].values\ndocs = [nlp.tokenizer(text) for text in test_texts]\nscores, _ = textcat.predict(docs)\npredicted_labels = scores.argmax(axis=1)\nlabels = [textcat.labels[label] for label in predicted_labels]\nlabels[:3]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Easy enough, so I should be able to submit my first prediction which is going to be correct 77% of the times.","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/nlp-getting-started/sample_submission.csv')\nsub['label'] = labels\nsub['target'] = np.where(sub['label'] == 'disaster', 1, 0)\nsub[['id', 'target']].to_csv('bow_sub.csv', index=False)\nsub.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A good exercise for me is always to take what I learned and make it a bit more functional. For example, I definitely want to have a CV score when I train the model. Time to make a function or two","metadata":{}},{"cell_type":"code","source":"def prepare_train_data(text='text', target='target'):\n    df = pd.read_csv('/kaggle/input/nlp-getting-started/train.csv')\n    train_labels = [{'cats': {'disaster': label == 1, 'not-disaster': label == 0}} for label in df[target]]\n    \n    train_texts = df[text].values\n    data = list(zip(train_texts, train_labels))\n    \n    return data\n\n\ndef prepare_test_data(nlp, text='text'):\n    df = df = pd.read_csv('/kaggle/input/nlp-getting-started/test.csv')\n    test_texts = df['text'].values\n    docs = [nlp.tokenizer(text) for text in test_texts]\n    \n    return docs\n\n\ndef train_nlp(train_data, val_data, epochs=10, cv_loop=True):\n    nlp = spacy.blank(\"en\")\n    # Create the TextCategorizer with exclusive classes and \"bow\" architecture\n    textcat = nlp.create_pipe(\n                  \"textcat\",\n                  config={\n                    \"exclusive_classes\": True,\n                    \"architecture\": \"bow\"})\n    # Add the TextCategorizer to the empty model\n    nlp.add_pipe(textcat)\n    # Add labels to text classifier\n    textcat.add_label('disaster')\n    textcat.add_label('not-disaster')\n    \n    spacy.util.fix_random_seed(1)\n    optimizer = nlp.begin_training()\n    \n    #losses = {}\n    for epoch in range(epochs):\n        losses = {}  # I believe this is better\n        random.shuffle(train_data)  # to avoid getting stuck in suboptimal solutions\n        # Create the batch generator with batch size = 10\n        batches = minibatch(train_data, size=10)\n        # Iterate through minibatches\n        for batch in batches:\n            texts, labels = zip(*batch)\n            nlp.update(texts, labels, sgd=optimizer, losses=losses)\n        print(losses)\n        \n    # Use textcat to get the scores for each doc\n    textcat = nlp.get_pipe('textcat')\n    try:\n        scores, _ = textcat.predict(val_data)\n    except AttributeError:\n        val_texts = [i[0] for i in val_data]\n        docs = [nlp.tokenizer(text) for text in val_texts]\n        scores, _ = textcat.predict(docs)\n    pos_scores = scores[:, 0] # probability of being a disaster tweet\n    \n    if cv_loop:\n        test_docs = prepare_test_data(nlp)\n        # Use textcat to get the scores for each doc\n        scores, _ = textcat.predict(test_docs)\n        test_scores = scores[:, 0]  # probability of being a disaster tweet\n        \n        return pos_scores, test_scores\n    \n    return pos_scores\n\n\ndef evaluate_predictions(true_label, pred_label):\n    print(f'Accuracy: \\t\\t {round(accuracy_score(y_true=true_label, y_pred=(pred_label>0.5).astype(int)), 4)}')\n    print(f'AUC ROC score: \\t\\t {round(roc_auc_score(y_true=true_label, y_score=pred_label), 4)}')\n    print(f'Log Loss: \\t\\t {round(log_loss(y_true=true_label, y_pred=pred_label), 4)}')\n\n\ndef cv_nlp(n_folds=5):\n    df_train = np.array(prepare_train_data())\n    \n    kfolds = KFold(n_splits=n_folds, shuffle=True, random_state=2)\n    \n    oof = np.zeros(len(df_train))\n    preds = None\n    \n    for n_fold, (train_index, test_index) in enumerate(kfolds.split(df_train)):\n        \n        train_set = list(df_train[train_index])\n        test_set = list(df_train[test_index])\n        \n        print(f'Fold {n_fold}')\n        oof[test_index], fold_preds = train_nlp(train_set, test_set, epochs=10, cv_loop=True)\n        print('_'*40)\n        print('\\n')\n        print('_'*40)\n        \n        if preds is None:\n            preds = fold_preds\n        else:\n            preds += fold_preds / n_folds\n            \n    true_labels = [int(i[1]['cats']['disaster']) for i in df_train]\n    \n    evaluate_predictions(true_labels, oof)\n    df = pd.read_csv('/kaggle/input/nlp-getting-started/train.csv')\n    tml.plot_classification_probs(data=df, true_label=df['target'], pred_label=oof)\n    \n    return oof, preds","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import KFold\nfrom sklearn.metrics import accuracy_score, roc_auc_score, log_loss\nimport tubesml as tml\n\noof, preds = cv_nlp(n_folds=5)\n\nsub = pd.read_csv('/kaggle/input/nlp-getting-started/sample_submission.csv')\nsub['target'] = (preds>0.5).astype(int)\nsub[['id', 'target']].to_csv('bow_sub_5folds.csv', index=False)\nsub.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the out-of-fold prediction, we estimate an accuracy of 78% and we get a 73% on the LB. Particularly challenging to appropriately slice the list of tuples of string and dictionary that is the train set format, but with some index gymnastic and missing a lot the always convenient pandas indexing we can get the job reasonably done. The plot of the prediction vs true label is available in the [**tubesML package**](https://pypi.org/project/tubesml/).\n\n## Takeaway of this section\n\nWith a Bag of Words representation of the data, it is relatively easy to build a text classifier with a decent accuracy. This representation only accounts for the frequency of appearance of the words, not their function or context.\n\nThe more examples are shown to the model, the more unique words may appear, the more the vocabulary grows, hence leading to a very sparse matrix. A better pre-processing step would have been to clean the text a bit. For example\n\n* By removing very common words\n* By ignoring the case\n* By ignoring mispelled words\n\nAn interesting step I want to take in the future is to include all the above in a single pipeline, something I suspect being very simple given the structure of the scipy methods.\n\nStill a mystery is the interpretation of the `loss` that spacy outputs during training. So I went to the code of the text categorizer and I noticed that in the update function there is a line `losses[self.name] += loss`, which explains why the loss is always increasing. I am not sure why they do that. The loss is then computed by the `get_loss` method and it is `float(mean_square_error)`. \n\nThis highlights a mistake (I think) in the Kaggle course as well: defining `losses={}` outside the loop of the epochs leads to having this value always increased, but I would be more interested in seeing the progress of the loss over epochs, thus it must be reset at the beginning of each epoch. (I will follow up on that once I get an answer in the forum)\n\n# Word embeddings\n\nFollowing the common definition, word embeddings represent each word numerically so that the vector represents **the word meaning or its usage**. If bag of words does not consider the context of each word, these embeddings aim to do so.\n\nEach word is represented by a vector in a defined vector space. The values of the vector components are learned based on the word usage. Therefore, we can expect seeing similar representations for words with similar meaning.\n\nThe method SpaCy uses to learn the components is `Word2Vec`, which uses shallow, 2-layer neural networks trained to reconstruct the linguistic context of words. It can use either of 2 architectures:\n\n* Continuous Bag of Words (CBoW): learns the embedding by predicting the current word based on its context. \n* Continuous Skip Diagram: learns by predicting the surrounding words given the current one.\n\nIn SpaCy, it is already prebuilt in their models, but we need to pick a more complext one than the simple `en`","metadata":{}},{"cell_type":"code","source":"# Need to load the large model to get the vectors\nnlp = spacy.load('en_core_web_lg')\nprint(nlp.pipe_names)\n[m for m in dir(nlp) if '__' not in m]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The pipeline already has more steps than before. According to the [documentation](https://spacy.io/models/en#en_core_web_lg), there should be more pipelines (tok2vec, tagger, parser, ner, attribute_ruler, lemmatizer), so I am not sure what am I missing here.\n\nAs before, let's play around with it","metadata":{}},{"cell_type":"code","source":"text = \"I like vectors because you can sum them\"\ntext_tokens = nlp(text)\n[m for m in dir(text_tokens) if '__' not in m]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Not only I have a vector representation of the text we gave (which, as far as I understand, is the average of all the vectors representing each word)","metadata":{}},{"cell_type":"code","source":"text_tokens.vector[:10]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"But also a 300-dimensional vector representation for each of the 8 words","metadata":{}},{"cell_type":"code","source":"print(np.array([token.vector for token in  text_tokens]).shape)\nfor token in text_tokens:\n    print(f'{token.text},\\t {token.has_vector},\\t {token.vector_norm},\\t {token.is_oov}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Every word is in the vocabulary (`is_oov` flags words out of the vocabulary), which is not surprising as we loaded a model with 685k unique vectors\n\nBack to the Disaster tweets dataframe.","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/nlp-getting-started/train.csv')\ndoc_vectors = np.array([nlp(text).vector for text in df.text])\nprint(doc_vectors.shape)\ndoc_vectors","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The simplest thing we can do now, is to use these newly created features to perform our classification problem as any other classification problem. For better visibility, we can start by creating a dataframe.","metadata":{}},{"cell_type":"code","source":"doc_df = pd.DataFrame(doc_vectors, columns=np.arange(0,300))\ndoc_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And create some CV scores to pick a model.","metadata":{}},{"cell_type":"code","source":"import tubesml as tml\nfrom sklearn.model_selection import KFold\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.ensemble import ExtraTreesClassifier\nfrom sklearn.linear_model import LogisticRegression\nimport xgboost as xgb\nimport lightgbm as lgb","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"models = [('Extra Tree', ExtraTreesClassifier(n_estimators=1000, n_jobs=-1, max_depth=8)),\n          ('Logit', LogisticRegression(C=1, max_iter=2000)),\n          ('Xgb', xgb.XGBClassifier(n_estimators=2000, n_jobs=-1, reg_alpha=0.3, learning_rate=0.05,\n                                    reg_lambda=1, subsample=0.7, max_depth=4, \n                                    random_state=324,objective='binary:logistic',use_label_encoder=False, eval_metric='logloss')), \n          ('Lgb', lgb.LGBMClassifier(n_estimators=2000, learning_rate=0.05, reg_alpha=0.3, reg_lambda=1, subsample=0.7, n_jobs=-1))]\n\nkfolds = KFold(n_splits=5, random_state=235, shuffle=True)\n\noof_res = {}\nfor model in models:\n    print(model[0])\n\n    full_pipe = Pipeline([('scaler', tml.DfScaler()), model])\n\n    if 'gb' not in model[0]:\n        oof = tml.cv_score(data=doc_df, target=df.target, estimator=full_pipe, cv=kfolds, predict_proba=True)\n    else:\n        oof = tml.cv_score(data=doc_df, target=df.target, estimator=full_pipe, cv=kfolds, predict_proba=True, \n                           early_stopping=100, eval_metric='logloss')\n        \n    oof_res[model[0]] = oof\n\n    tml.eval_classification(doc_df, df.target, oof, plot=1, proba=True)\n    \n    print('_'*40)\n    print('_'*40)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Which is a bit higher than the score we got in the previous section and indeed scores 80.8% on the public LB, improving of 4% our previous attempt with `textcat`.","metadata":{}},{"cell_type":"code","source":"df_test = pd.read_csv('/kaggle/input/nlp-getting-started/test.csv')\ndoc_vectors = np.array([nlp(text).vector for text in df_test.text])\ndoc_df_test = pd.DataFrame(doc_vectors, columns=np.arange(0,300))\nfull_pipe = Pipeline([('scaler', tml.DfScaler()), \n                      ('Lgb', lgb.LGBMClassifier(n_estimators=200, learning_rate=0.05, \n                                                 reg_alpha=0.3, reg_lambda=1, subsample=0.7, n_jobs=-1))])\nfull_pipe.fit(doc_df, df.target)\npreds = full_pipe.predict(doc_df_test)\n\nsub = pd.read_csv('/kaggle/input/nlp-getting-started/sample_submission.csv')\nsub['target'] = preds\nsub[['id', 'target']].to_csv('word_embeddings_lgb.csv', index=False)\nsub.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}