{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Sentence Classification using Ignite\n\nbased on three examples:\n\n[Convolutional Neural Networks for Sentence Classification using Ignite](https://github.com/pytorch/ignite/blob/master/examples/notebooks/TextCNN.ipynb)\n\n[Обучение и тестирование нейронных сетей на PyTorch с помощью Ignite](https://habr.com/ru/company/ods/blog/424781/)\n\n[CNN baseline](https://www.kaggle.com/guitaricet/cnn-baseline)"},{"metadata":{},"cell_type":"markdown","source":"## Import Libraries"},{"metadata":{},"cell_type":"markdown","source":"`torchtext` is a library that provides multiple datasets for NLP tasks, similar to `torchvision`. Below we import the following:\n* **data**: A module to setup the data in the form Fields and Labels.\n* **datasets**: A module to download NLP datasets.\n* **GloVe**: A module to download and use pretrained GloVe embedings."},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:24.597525Z","start_time":"2019-04-24T11:29:24.199658Z"},"trusted":true},"cell_type":"code","source":"from torchtext import data\nfrom torchtext import datasets\nfrom torchtext.vocab import Vectors\nimport torchtext","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We import torch, nn and functional modules to create our models! "},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:24.609918Z","start_time":"2019-04-24T11:29:24.599277Z"},"trusted":true},"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nSEED = 1234\ntorch.manual_seed(SEED)\ntorch.cuda.manual_seed(SEED)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:24.626791Z","start_time":"2019-04-24T11:29:24.611489Z"},"trusted":true},"cell_type":"code","source":"import pathlib\nimport numpy as np\nimport sklearn.metrics as skm\nimport pprint\n\nimport csv\nimport sys\ncsv.field_size_limit(sys.maxsize)  # needed for torchtext","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`Ignite` is a High-level library to help with training neural networks in PyTorch. It comes with an `Engine` to setup a training loop, various metrics, handlers and a helpful contrib section! \n\nBelow we import the following:\n* **Engine**: Runs a given process_function over each batch of a dataset, emitting events as it goes.\n* **Events**: Allows users to attach functions to an `Engine` to fire functions at a specific event. Eg: `EPOCH_COMPLETED`, `ITERATION_STARTED`, etc.\n* **Accuracy**: Metric to calculate accuracy over a dataset, for binary, multiclass, multilabel cases. \n* **Loss**: General metric that takes a loss function as a parameter, calculate loss over a dataset.\n* **RunningAverage**: General metric to attach to Engine during training. \n* **ModelCheckpoint**: Handler to checkpoint models. \n* **EarlyStopping**: Handler to stop training based on a score function. \n* **ProgressBar**: Handler to create a tqdm progress bar."},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install pytorch-ignite","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:24.649391Z","start_time":"2019-04-24T11:29:24.628640Z"},"trusted":true},"cell_type":"code","source":"from ignite.engine import Engine, Events\nfrom ignite.metrics import Accuracy, Loss, RunningAverage, Precision, Recall\nfrom ignite.handlers import ModelCheckpoint, EarlyStopping\nfrom ignite.contrib.handlers import ProgressBar","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"input_path = '../input/dlinnlp-spring-2019-clf'\nvectors_path = '../input/glove840b300dtxt/glove.840B.300d.txt'\ncache_path = '../input/glove840b300dtxt'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analyze max length for body and title"},{"metadata":{"ExecuteTime":{"end_time":"2019-04-25T05:14:05.610832Z","start_time":"2019-04-25T05:14:05.577573Z"},"trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:25.448287Z","start_time":"2019-04-24T11:29:24.884259Z"},"trusted":true},"cell_type":"code","source":"df = pd.read_csv(f'{input_path}/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:25.467280Z","start_time":"2019-04-24T11:29:25.449907Z"},"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:25.483729Z","start_time":"2019-04-24T11:29:25.468722Z"},"trusted":true},"cell_type":"code","source":"df.label.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:25.637124Z","start_time":"2019-04-24T11:29:25.626869Z"},"trusted":true},"cell_type":"code","source":"MAX_TITLE_LEN = 40\nMAX_BODY_LEN = 2000","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:25.852117Z","start_time":"2019-04-24T11:29:25.638527Z"},"trusted":true},"cell_type":"code","source":"df.title.apply(lambda x: len(str(x).split())).clip_upper(MAX_TITLE_LEN).plot(kind='hist', bins=MAX_TITLE_LEN)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:27.164726Z","start_time":"2019-04-24T11:29:26.441803Z"},"trusted":true},"cell_type":"code","source":"df.text.apply(lambda x: len(str(x).split())).clip_upper(MAX_BODY_LEN).plot(kind='hist', bins=50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Processing Data"},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:24.695112Z","start_time":"2019-04-24T11:29:24.650997Z"},"trusted":true},"cell_type":"code","source":"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:36.522509Z","start_time":"2019-04-24T11:29:27.166404Z"},"trusted":true},"cell_type":"code","source":"index2label = ['news', 'clickbait', 'other']\nlabel2index = {l: i for i, l in enumerate(index2label)}\n\ntitle_field = torchtext.data.Field(lower=True, include_lengths=False, fix_length=MAX_TITLE_LEN, batch_first=True)\nbody_field = torchtext.data.Field(lower=True, include_lengths=False, fix_length=MAX_BODY_LEN, batch_first=True)\nlabel_field = torchtext.data.Field(sequential=False, is_target=True, use_vocab=False,\n                                   preprocessing=lambda x: label2index[x])\n\ntrain_dataset = torchtext.data.TabularDataset(f'{input_path}/train.csv',\n                                              format='csv',\n                                              fields={'title': ('title', title_field),\n                                                      'text': ('body', body_field),\n                                                      'label': ('label', label_field)})\n\nval_dataset = torchtext.data.TabularDataset(f'{input_path}/valid.csv',\n                                            format='csv',\n                                            fields={'title': ('title', title_field),\n                                                    'text': ('body', body_field),\n                                                    'label': ('label', label_field)})\n\ntest_dataset = torchtext.data.TabularDataset(f'{input_path}/test.csv',\n                                            format='csv',\n                                            fields={'title': ('title', title_field),\n                                                    'text': ('body', body_field)})\n\nbody_field.build_vocab(train_dataset, min_freq=2)\nlabel_field.build_vocab(train_dataset)\nvocab = body_field.vocab\ntitle_field.vocab = vocab\n\nprint('Vocab size: ', len(vocab))\nprint(train_dataset[0].title)\nprint(train_dataset[0].body[:15])\nprint(train_dataset[0].label)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we have three sets of the data - train, validation, test. Let's explore what these data objects are and how to extract data from them.\n* `train_data` is **torchtext.data.dataset.Dataset**, this is similar to **torch.utils.data.Dataset**.\n* `train_data[0]` is **torchtext.data.example.Example**, a Dataset is comprised of many Examples."},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:41.055186Z","start_time":"2019-04-24T11:29:36.555567Z"},"trusted":true},"cell_type":"code","source":"vocab.load_vectors(Vectors(vectors_path))\n\nprint ('Attributes of title_field.vocab : ', [attr for attr in dir(vocab) if '_' not in attr])\nprint ('First 5 values title_field.vocab.itos : ', vocab.itos[0:5]) \nprint ('First 5 key, value pairs of title_field.vocab.stoi : ', {key:value for key,value in list(vocab.stoi.items())[0:5]}) \nprint ('Shape of title_field.vocab.vectors.shape : ', vocab.vectors.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we must convert our split datasets into iterators, we'll take advantage of **torchtext.data.BucketIterator**! BucketIterator pads every element of a batch to the length of the longest element of the batch."},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:41.071289Z","start_time":"2019-04-24T11:29:41.056891Z"},"trusted":true},"cell_type":"code","source":"train_loader = data.Iterator(train_dataset, batch_size=64, device='cuda', shuffle=True, sort=False)\nval_loader = data.Iterator(val_dataset, batch_size=64, device='cuda', shuffle=False, sort=False)\ntest_loader = data.Iterator(test_dataset, batch_size=64, device='cuda', shuffle=False, sort=False)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:43.124160Z","start_time":"2019-04-24T11:29:43.066702Z"},"trusted":true},"cell_type":"code","source":"batch = next(iter(train_loader))\nprint(batch)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:43.065194Z","start_time":"2019-04-24T11:29:41.094248Z"},"trusted":true},"cell_type":"code","source":"next(iter(test_loader))","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:43.417457Z","start_time":"2019-04-24T11:29:43.125692Z"},"trusted":true},"cell_type":"code","source":"batch = next(iter(train_loader))\nprint('batch.label[0] : ', batch.label[0])\nprint('batch.title[0] : ', batch.title[0][batch.title[0] != 1])\nprint('batch.body[0] : ', batch.body[0][batch.body[0] != 1])\n\nlengths = []\nfor i, batch in enumerate(train_loader):\n    x = batch.title\n    lengths.append(x.shape[1])\n    if i == 10:\n        break\n\nprint ('Lengths of first 10 batches : ', lengths)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Text Model"},{"metadata":{},"cell_type":"markdown","source":"Here is the replication of the model, here are the operations of the model:\n* **Embedding**: Embeds a batch of text of shape (N, L) to (N, L, D), where N is batch size, L is maximum length of the batch, D is the embedding dimension. \n\n* **Convolutions**: Runs parallel convolutions across the embedded words with kernel sizes of 3, 4, 5 to mimic trigrams, four-grams, five-grams. This results in outputs of (N, L - k + 1, D) per convolution, where k is the kernel_size. \n\n* **Activation**: ReLu activation is applied to each convolution operation.\n\n* **Pooling**: Runs parallel maxpooling operations on the activated convolutions with window sizes of L - k + 1, resulting in 1 value per channel i.e. a shape of (N, 1, D) per pooling. \n\n* **Concat**: The pooling outputs are concatenated and squeezed to result in a shape of (N, 3D). This is a single embedding for a sentence.\n\n* **Dropout**: Dropout is applied to the embedded sentence. \n\n* **Fully Connected**: The dropout output is passed through a fully connected layer of shape (3D, 1) to give a single output for each example in the batch. sigmoid is applied to the output of this layer.\n\n* **load_embeddings**: This is a method defined for TextCNN to load embeddings based on user input. There are 3 modes - rand which results in randomly initialized weights, static which results in frozen pretrained weights, nonstatic which results in trainable pretrained weights. \n\n\nLet's note that this model works for variable text lengths! The idea to embed the words of a sentence, use convolutions, maxpooling and concantenation to embed the sentence as a single vector! This single vector is passed through a fully connected layer with sigmoid to output a single value. This value can be interpreted as the probability a sentence is positive (closer to 1) or negative (closer to 0).\n\nThe minimum length of text expected by the model is the size of the smallest kernel size of the model."},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:43.440286Z","start_time":"2019-04-24T11:29:43.419295Z"},"trusted":true},"cell_type":"code","source":"class TextModel(nn.Module):\n    def __init__(self, vocab_size, embedding_dim, kernel_sizes, num_filters,\n                 num_classes, d_prob, mode, hidden_dim, lstm_units,\n                 emb_vectors=None):\n        super(TextModel, self).__init__()\n        self.vocab_size = vocab_size\n        self.embedding_dim = embedding_dim\n        self.kernel_sizes = kernel_sizes\n        self.num_filters = num_filters\n        self.num_classes = num_classes\n        self.d_prob = d_prob\n        self.mode = mode\n        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=1)\n        self.load_embeddings(emb_vectors)\n        self.conv = nn.ModuleList([nn.Conv1d(in_channels=embedding_dim,\n                                             out_channels=num_filters,\n                                             kernel_size=k, stride=1) for k in kernel_sizes])\n        self.conv2 = nn.ModuleList([nn.Conv1d(in_channels=embedding_dim,\n                                             out_channels=num_filters,\n                                             kernel_size=k, stride=1) for k in kernel_sizes])\n        self.conv_body = nn.ModuleList([nn.Conv1d(in_channels=embedding_dim,\n                                             out_channels=num_filters,\n                                             kernel_size=k, stride=1) for k in kernel_sizes])\n        self.lstm1 = nn.LSTM(embedding_dim, lstm_units, bidirectional=True, batch_first=True)\n        self.lstm2 = nn.LSTM(lstm_units * 2, lstm_units, bidirectional=True, batch_first=True)\n        self.lstm_body = nn.LSTM(embedding_dim, lstm_units, bidirectional=True, batch_first=True)\n        self.dropout = nn.Dropout(d_prob)\n        self.fc = nn.Linear(len(kernel_sizes) * num_filters, hidden_dim)\n        self.fc_body = nn.Linear(len(kernel_sizes) * num_filters, hidden_dim)\n        self.fc_total = nn.Linear(hidden_dim * 2 + lstm_units * 6, hidden_dim)\n        self.fc_final = nn.Linear(hidden_dim, num_classes)\n\n\n    def forward(self, x):\n        title, body = x\n        batch_size, sequence_length = title.shape\n\n        title_emb = self.embedding(title)\n        title = [F.relu(conv(title_emb.transpose(1, 2))) for conv in self.conv]\n        title = [F.max_pool1d(c, c.size(-1)).squeeze(dim=-1) for c in title]\n        title = [F.relu(conv(title_emb.transpose(1, 2))) for conv in self.conv2]\n        title = [F.max_pool1d(c, c.size(-1)).squeeze(dim=-1) for c in title]\n        title = torch.cat(title, dim=1)\n        title = self.fc(self.dropout(title))\n        \n        h_lstm1, _ = self.lstm1(title_emb)\n        h_lstm2, _ = self.lstm2(h_lstm1)\n        \n        # average pooling\n        avg_pool2 = torch.mean(h_lstm2, 1)\n        # global max pooling\n        max_pool2, _ = torch.max(h_lstm2, 1)\n\n        batch_size, sequence_length = body.shape\n        body_emb = self.embedding(body)\n        body = [F.relu(conv(body_emb.transpose(1, 2))) for conv in self.conv_body]\n        body = [F.max_pool1d(c, c.size(-1)).squeeze(dim=-1) for c in body]\n        body = torch.cat(body, dim=1)\n        body = self.fc_body(self.dropout(body))\n        \n        h_lstm_body, _ = self.lstm_body(body_emb)\n        max_pool_body, _ = torch.max(h_lstm_body, 1)\n\n        x = torch.cat([title, body, avg_pool2, max_pool2, max_pool_body], dim=1)\n        x = F.relu(self.fc_total(self.dropout(x)))\n        x = self.fc_final(x)\n        \n        return x\n    \n    def load_embeddings(self, emb_vectors):\n        if 'static' in self.mode:\n            self.embedding.weight.data.copy_(emb_vectors)\n            if 'non' not in self.mode:\n                self.embedding.weight.data.requires_grad = False\n                print('Loaded pretrained embeddings, weights are not trainable.')\n            else:\n                self.embedding.weight.data.requires_grad = True\n                print('Loaded pretrained embeddings, weights are trainable.')\n        elif self.mode == 'rand':\n            print('Randomly initialized embeddings are used.')\n        else:\n            raise ValueError('Unexpected value of mode. Please choose from static, nonstatic, rand.')\n            ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating Model, Optimizer and Loss"},{"metadata":{},"cell_type":"markdown","source":"Below we create an instance of the TextModel model and load embeddings in **static** mode. The model is placed on a device and then a loss function of Binary Cross Entropy and Adam optimizer are setup. "},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:43.987324Z","start_time":"2019-04-24T11:29:43.441984Z"},"trusted":true},"cell_type":"code","source":"vocab_size, embedding_dim = vocab.vectors.shape\n\nmodel = TextModel(vocab_size=vocab_size,\n                  embedding_dim=embedding_dim,\n                  kernel_sizes=[3, 4, 5],\n                  num_filters=64,\n                  num_classes=3, \n                  d_prob=0.6,\n                  mode='nonstatic',\n                  hidden_dim=256,\n                  emb_vectors=vocab.vectors,\n                  lstm_units=64)\nmodel.to(device)\noptimizer = torch.optim.Adam(model.parameters(), lr=1e-4, weight_decay=1e-5)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:44.031909Z","start_time":"2019-04-24T11:29:44.019662Z"},"trusted":true},"cell_type":"code","source":"criterion = nn.CrossEntropyLoss()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training and Evaluating using Ignite"},{"metadata":{},"cell_type":"markdown","source":"### Trainer Engine - process_function\n\nIgnite's Engine allows user to define a process_function to process a given batch, this is applied to all the batches of the dataset. This is a general class that can be applied to train and validate models! A process_function has two parameters engine and batch. \n\n\nLet's walk through what the function of the trainer does:\n\n* Sets model in train mode. \n* Sets the gradients of the optimizer to zero.\n* Generate x and y from batch.\n* Performs a forward pass to calculate y_pred using model and x.\n* Calculates loss using y_pred and y.\n* Performs a backward pass using loss to calculate gradients for the model parameters.\n* model parameters are optimized using gradients and optimizer.\n* Returns scalar loss. \n\nBelow is a single operation during the trainig process. This process_function will be attached to the training engine."},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:44.050516Z","start_time":"2019-04-24T11:29:44.033745Z"},"trusted":true},"cell_type":"code","source":"def process_function(engine, batch):\n    model.train()\n    optimizer.zero_grad()\n    x, y = (batch.title, batch.body), batch.label\n    y_pred = model(x)\n    loss = criterion(y_pred, y)\n    loss.backward()\n    optimizer.step()\n    return loss.item()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Evaluator Engine - process_function\n\nSimilar to the training process function, we setup a function to evaluate a single batch. Here is what the eval_function does:\n\n* Sets model in eval mode.\n* Generates x and y from batch.\n* With torch.no_grad(), no gradients are calculated for any succeding steps.\n* Performs a forward pass on the model to calculate y_pred based on model and x.\n* Returns y_pred and y.\n\nIgnite suggests attaching metrics to evaluators and not trainers because during the training the model parameters are constantly changing and it is best to evaluate model on a stationary model. This information is important as there is a difference in the functions for training and evaluating. Training returns a single scalar loss. Evaluating returns y_pred and y as that output is used to calculate metrics per batch for the entire dataset.\n\nAll metrics in Ignite require y_pred and y as outputs of the function attached to the Engine. "},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:44.064872Z","start_time":"2019-04-24T11:29:44.052809Z"},"trusted":true},"cell_type":"code","source":"def eval_function(engine, batch):\n    model.eval()\n    with torch.no_grad():\n        x, y = (batch.title, batch.body), batch.label\n        y_pred = model(x)\n        return y_pred, y","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Instantiating Training and Evaluating Engines\n\nBelow we create 3 engines, a trainer, a training evaluator and a validation evaluator. You'll notice that train_evaluator and validation_evaluator use the same function, we'll see later why this was done! "},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:44.080435Z","start_time":"2019-04-24T11:29:44.066399Z"},"trusted":true},"cell_type":"code","source":"trainer = Engine(process_function)\ntrain_evaluator = Engine(eval_function)\nvalidation_evaluator = Engine(eval_function)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Metrics - RunningAverage, Accuracy and Loss\n\nTo start, we'll attach a metric of Running Average to track a running average of the scalar loss output for each batch. "},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:44.095515Z","start_time":"2019-04-24T11:29:44.082339Z"},"trusted":true},"cell_type":"code","source":"RunningAverage(output_transform=lambda x: x).attach(trainer, 'loss')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now there are two metrics that we want to use for evaluation - accuracy and loss. \n\nFor Accuracy, Ignite requires y_pred and y to be comprised of 0's and 1's only. Since our model outputs from a sigmoid layer, values are between 0 and 1. We'll need to write a function that transforms `engine.state.output` which is comprised of y_pred and y. \n\nBelow `thresholded_output_transform` does just that, it rounds y_pred to convert y_pred to 0's and 1's, and then returns rounded y_pred and y. This function is the output_transform function used to transform the `engine.state.output` to achieve `Accuracy`'s desired purpose.\n\nNow, we attach `Loss` and `Accuracy` (with `thresholded_output_transform`) to train_evaluator and validation_evaluator. \n\nTo attach a metric to engine, the following format is used:\n* `Metric(output_transform=output_transform, ...).attach(engine, 'metric_name')`\n"},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:44.116387Z","start_time":"2019-04-24T11:29:44.099968Z"},"trusted":true},"cell_type":"code","source":"def thresholded_output_transform(output):\n    y_pred, y = output\n    y_pred = torch.round(y_pred)\n    return y_pred, y","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:44.153409Z","start_time":"2019-04-24T11:29:44.135972Z"},"trusted":true},"cell_type":"code","source":"Accuracy(output_transform=thresholded_output_transform).attach(train_evaluator, 'accuracy')\nLoss(criterion).attach(train_evaluator, 'ce')\n\nprecision = Precision(average=False)\nrecall = Recall(average=False)\nF1 = (precision * recall * 2 / (precision + recall)).mean()\n\nprecision.attach(train_evaluator, 'precision')\nrecall.attach(train_evaluator, 'recall')\nF1.attach(train_evaluator, 'F1')","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:44.171059Z","start_time":"2019-04-24T11:29:44.155432Z"},"trusted":true},"cell_type":"code","source":"Accuracy(output_transform=thresholded_output_transform).attach(validation_evaluator, 'accuracy')\nLoss(criterion).attach(validation_evaluator, 'ce')\n\nprecision = Precision(average=False)\nrecall = Recall(average=False)\nF1 = (precision * recall * 2 / (precision + recall)).mean()\n\nprecision.attach(validation_evaluator, 'precision')\nrecall.attach(validation_evaluator, 'recall')\nF1.attach(validation_evaluator, 'F1')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Progress Bar\n\nNext we create an instance of Ignite's progess bar and attach it to the trainer and pass it a key of `engine.state.metrics` to track. In this case, the progress bar will be tracking `engine.state.metrics['loss']`"},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:44.187809Z","start_time":"2019-04-24T11:29:44.172654Z"},"trusted":true},"cell_type":"code","source":"pbar = ProgressBar(persist=True, bar_format=\"\")\npbar.attach(trainer, ['loss'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### EarlyStopping - Tracking Validation Loss\n\nNow we'll setup a Early Stopping handler for this training process. EarlyStopping requires a score_function that allows the user to define whatever criteria to stop trainig. In this case, if the loss of the validation set does not decrease in 3 epochs, the training process will stop early.  "},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:44.203103Z","start_time":"2019-04-24T11:29:44.189525Z"},"trusted":true},"cell_type":"code","source":"def score_function(engine):\n    val_loss = engine.state.metrics['F1']\n    return val_loss\n\nhandler = EarlyStopping(patience=3, score_function=score_function, trainer=trainer)\nvalidation_evaluator.add_event_handler(Events.COMPLETED, handler)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Attaching Custom Functions to Engine at specific Events\n\nBelow you'll see ways to define your own custom functions and attaching them to various `Events` of the training process.\n\nThe functions below both achieve similar tasks, they print the results of the evaluator run on a dataset. One function does that on the training evaluator and dataset, while the other on the validation. Another difference is how these functions are attached in the trainer engine.\n\nThe first method involves using a decorator, the syntax is simple - `@` `trainer.on(Events.EPOCH_COMPLETED)`, means that the decorated function will be attached to the trainer and called at the end of each epoch. \n\nThe second method involves using the add_event_handler method of trainer - `trainer.add_event_handler(Events.EPOCH_COMPLETED, custom_function)`. This achieves the same result as the above. "},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:44.243118Z","start_time":"2019-04-24T11:29:44.229730Z"},"trusted":true},"cell_type":"code","source":"@trainer.on(Events.EPOCH_COMPLETED)\ndef log_training_results(engine):\n    train_evaluator.run(train_loader)\n    metrics = train_evaluator.state.metrics\n    pbar.log_message(\n        \"Training Results - Epoch: {} \\nMetrics\\n{}\"\n        .format(engine.state.epoch, pprint.pformat(metrics)))\n    \ndef log_validation_results(engine):\n    validation_evaluator.run(val_loader)\n    metrics = validation_evaluator.state.metrics\n    metrics = validation_evaluator.state.metrics\n    pbar.log_message(\n        \"Validation Results - Epoch: {} \\nMetrics\\n{}\"\n        .format(engine.state.epoch, pprint.pformat(metrics)))\n    pbar.n = pbar.last_print_n = 0\n\ntrainer.add_event_handler(Events.EPOCH_COMPLETED, log_validation_results)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### ModelCheckpoint\n\nLastly, we want to checkpoint this model. It's important to do so, as training processes can be time consuming and if for some reason something goes wrong during training, a model checkpoint can be helpful to restart training from the point of failure.\n\nBelow we'll use Ignite's `ModelCheckpoint` handler to checkpoint models at the end of each epoch. "},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:29:44.755258Z","start_time":"2019-04-24T11:29:44.732320Z"},"trusted":true},"cell_type":"code","source":"checkpointer = ModelCheckpoint('checkpoint', 'textcnn', save_interval=1, n_saved=2, create_dir=True, save_as_state_dict=True)\nbest_model_save = ModelCheckpoint(\n    'best_model', 'textcnn', n_saved=1,\n    create_dir=True, save_as_state_dict=True,\n    score_function=score_function)\ntrainer.add_event_handler(Events.EPOCH_COMPLETED, checkpointer, {'textcnn': model})\nvalidation_evaluator.add_event_handler(Events.EPOCH_COMPLETED, best_model_save, {'textcnn': model})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Run Engine\n\nNext, we'll run the trainer for 20 epochs and monitor results. Below we can see that progess bar prints the loss per iteration, and prints the results of training and validation as we specified in our custom function. "},{"metadata":{"ExecuteTime":{"end_time":"2019-04-24T11:59:19.933446Z","start_time":"2019-04-24T11:29:44.757552Z"},"scrolled":false,"trusted":true},"cell_type":"code","source":"trainer.run(train_loader, max_epochs=20)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-23T11:06:27.144905Z","start_time":"2019-04-23T11:06:27.122526Z"}},"cell_type":"markdown","source":"## Inference"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_path = next(pathlib.Path('best_model').rglob('*'))\nmodel_path","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-25T05:06:37.534352Z","start_time":"2019-04-25T05:06:36.797437Z"},"trusted":true},"cell_type":"code","source":"model_state_dict = torch.load(model_path)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-25T05:06:39.611050Z","start_time":"2019-04-25T05:06:39.588786Z"},"trusted":true},"cell_type":"code","source":"model.load_state_dict(model_state_dict)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-25T05:06:41.693314Z","start_time":"2019-04-25T05:06:41.664972Z"},"trusted":true},"cell_type":"code","source":"predictions = []\nlabels = []\n\n# change model mode to 'evaluation'\n# disable dropout and use learned batch norm statistics\nmodel.eval()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-25T05:06:56.954298Z","start_time":"2019-04-25T05:06:56.931905Z"},"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-25T05:08:06.574069Z","start_time":"2019-04-25T05:08:00.915778Z"},"trusted":true},"cell_type":"code","source":"predictions = []\nlabels = []\n\n# change model mode to 'evaluation'\n# disable dropout and use learned batch norm statistics\nmodel.eval()\n\nwith torch.no_grad():\n    for batch in val_loader:\n        x, label = batch\n#         logits = model(title)\n        logits = model(x)\n\n        y_pred = torch.max(logits, dim=1)[1]\n        # move from GPU to CPU and convert to numpy array\n        y_pred_numpy = y_pred.cpu().numpy()\n\n        predictions = np.concatenate([predictions, y_pred_numpy])\n        labels = np.concatenate([labels, label.cpu().numpy()])\n","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-25T05:08:06.941802Z","start_time":"2019-04-25T05:08:06.575441Z"},"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-25T05:08:53.511672Z","start_time":"2019-04-25T05:08:53.475624Z"},"trusted":true},"cell_type":"code","source":"skm.f1_score(labels, predictions, average='micro')","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-25T05:08:59.950455Z","start_time":"2019-04-25T05:08:59.915148Z"},"trusted":true},"cell_type":"code","source":"skm.f1_score(labels, predictions, average='macro')","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-25T05:09:17.305709Z","start_time":"2019-04-25T05:09:07.823559Z"},"trusted":true},"cell_type":"code","source":"# Do not shuffle test set! You need id to label mapping\ntest_loader = torchtext.data.Iterator(test_dataset, batch_size=128, device='cuda', shuffle=False)\n\npredictions = []\n\nmodel.eval()\n\nwith torch.no_grad():\n    for batch in test_loader:\n        x, label = batch\n#         logits = model(title)\n        logits = model(x)\n\n        y_pred = torch.max(logits, dim=1)[1]\n        # move from GPU to CPU and convert to numpy array\n        y_pred_numpy = y_pred.cpu().numpy()\n\n        predictions = np.concatenate([predictions, y_pred_numpy])","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-25T05:09:39.828628Z","start_time":"2019-04-25T05:09:39.791006Z"},"trusted":true},"cell_type":"code","source":"predictions_str = [index2label[int(p)] for p in predictions]\n\n# test.csv index in a contiguous integers from 0 to len(test_set)\n# to this should work fine\nsubmission = pd.DataFrame({'id': list(range(len(predictions_str))), 'label': predictions_str})\n","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-25T05:09:44.057372Z","start_time":"2019-04-25T05:09:44.016431Z"},"trusted":true},"cell_type":"code","source":"submission.head()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-04-25T05:09:56.685403Z","start_time":"2019-04-25T05:09:56.639649Z"},"trusted":true},"cell_type":"code","source":"submission.to_csv('submission_ignite.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.8"}},"nbformat":4,"nbformat_minor":1}