{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Discriminating Power of Different Loss Functions on MNIST \nThe purpose of this kernel is to explore the discriminating power of different loss functions for classification, beyond the traditional softmax loss, on the **MNIST** dataset. The idea for maxmium discriminating capability is maximizing inter-class variance and minimizing intra-class variance. In order to determine so, we will visualize the generated embeddings using t-SNE. \n\nTLDR; skip to the end to see the visualizations.\n\nGithub Repo: https://github.com/BKHMSI/LossMNIST"},{"metadata":{"_uuid":"4470d73b152551fcb3a4de6a7a7d0bac8760a481"},"cell_type":"markdown","source":"## What's in this Kernel\n\nTraining MNIST using each of the following loss functions implemented using Keras + Tensorflow, then visualizing the generated embeddings using t-SNE in order to compare the discriminating power of each loss function\n\n- Categorical Cross-Entropy (Softmax) \n- Semi-Hard Triplet Loss:  \n    - <a href=\"https://arxiv.org/abs/1503.03832\">FaceNet: A Unified Embedding for Face Recognition and Clustering</a>\n    - <a href=\"https://arxiv.org/abs/1703.07737\">In Defense of the Triplet Loss for Person Re-Identification</a>\n- Large Margin Cosine Loss\n  - <a href=\"https://arxiv.org/abs/1801.09414\">CosFace: Large Margin Cosine Loss for Deep Face Recognition</a>\n- Intra-Enhanced Triplet Loss\n - <a href=\"https://ieeexplore.ieee.org/document/8272723\">Deep Dense Multi-level feature for partial high-resolution fingerprint matching</a>\n- Semi-Hard Triplet Loss + Softmax\n    \n"},{"metadata":{"_uuid":"1df2547dbc2777e37631b6d2cb3532875473199f"},"cell_type":"markdown","source":"## Let's First Define our Hyperparameters \nFeel free to change any of the parameters below to see the effect of the different loss functions with their hyperparameters on MNIST"},{"metadata":{"trusted":true,"_uuid":"2da613779056c12834da83d2b4cf818073035798"},"cell_type":"code","source":"config = {\n    \"train\": {\n      \"lr\": 0.001, \n      \"optim\": \"Nadam\",\n      \"epochs\": 10, \n      \"batch-size\": 400,\n      \"k_batch\": 40, # used in intra-enhanced triplet-loss\n      \"loss\": \"triplet-softmax\", # loss function can be one of \n                                 # [triplet-softmax, large-margin-cosine-loss, \n                                 #  intra-enhanced-triplet-loss, \n                                 #  semi-hard-triplet-loss, categorical-crossentropy]\n      \"alpha\": 0.2, # margin used in several loss functions \n      \"beta\": 0.1, # another margin used in intra-enhanced-triplet-loss\n      \"lambda_1\": 0.5, # weight of anchor-center-loss in intra-enhanced-triplet-loss\n      \"lambda_2\": 0.1, # weight of categorical-crossentropy when training with triplet-softmax\n      \"scale\": 20, # scale used in large-margin-cosine-loss\n      \"reg_lambda\": 0.01,# regularization weight used in large-margin-cosine-loss \n      \"lr_reduce_factor\": 0.5,\n      \"patience\": 5,\n      \"min_lr\": 1.0e-5,\n      \"shuffle\": True,\n    },\n    \"data\": {\n      \"imsize\": 28, \n      \"imchannel\": 1,\n      \"num_classes\": 10,\n      \"samples_per_id\": 5000, # there are 5000 samples for each class, used to order data for triplet-loss\n      \"val_split\": 0.1 ,\n    },\n    \"tsne\": {\n      \"n_iter\": 2500,\n      \"perplexity\": 30, \n    },   \n}\n\n# shortcuts\ntrain = config[\"train\"]\ndata  = config[\"data\"]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"37fb6b538a8dbc4d2034a9ec0ef3101f8d0975fe"},"cell_type":"markdown","source":"## Define the Data Loader\nI used the MNIST dataset provided by [Digit Recognizer](\"https://www.kaggle.com/c/digit-recognizer\") competition splitting the training data into 10% for validation (42,00) and the rest for training (37,800). I saved the model with the lowest validation loss after ~10 epochs and visualized the results of the validation set."},{"metadata":{"_uuid":"8683411cad6e391047dbfefbeb891e46b01e29c2","trusted":true},"cell_type":"code","source":"from __future__ import print_function\n\nimport os\nimport yaml\nimport argparse\nimport numpy as np \nimport pandas as pd\n\nfrom keras.utils import np_utils\nfrom keras.datasets import mnist\n\nclass DataLoader(object):\n    def __init__(self, config, one_hot = False):\n        self.config = config\n        self.one_hot = one_hot\n\n    def load(self):\n        data_train = pd.read_csv('../input/train.csv')\n        X_data = np.array(data_train.iloc[:,1:])\n        self.y_data = np.array(data_train.iloc[:,:1]).squeeze()\n                \n        self.input_shape = (-1, self.config[\"data\"][\"imsize\"], self.config[\"data\"][\"imsize\"], self.config[\"data\"][\"imchannel\"])\n        self.X_data = np.reshape(X_data, self.input_shape)\n\n        if self.one_hot:\n            self.y_data = np_utils.to_categorical(self.y_data, self.config[\"data\"][\"num_classes\"])\n\n        self.num_train = int(self.y_data.shape[0] * (1-self.config[\"data\"][\"val_split\"]))\n        self.num_val   = int(self.y_data.shape[0] * (self.config[\"data\"][\"val_split\"]))\n\n        if self.config[\"train\"][\"loss\"] in [\"intra-enhanced-triplet-loss\", \"semi-hard-triplet-loss\"]: \n            print(\"[INFO] Ordering Data\")\n            self.order_data_triplet_loss()\n            \n        self.split_data()\n        self.X_train = self.preprocess(self.X_train)\n        self.X_val   = self.preprocess(self.X_val)\n\n\n    def preprocess(self, data):\n        data = data.astype('float32')\n        return data / 255.\n\n    def order_data_triplet_loss(self):\n        data = {}\n        samples_per_id = self.config[\"data\"][\"samples_per_id\"]\n        for label in range(self.config[\"data\"][\"num_classes\"]):\n            mask = self.y_data==label\n            data[label] = [i for i, x in enumerate(mask) if x]\n            if len(data[label]) < samples_per_id:\n                data[label].extend(np.random.choice(data[label], samples_per_id - len(data[label]), replace=False))\n            data[label] = data[label][:samples_per_id]\n\n        k_batch = self.config[\"train\"][\"k_batch\"]\n        X_data, y_data = [], []\n        for i in range(samples_per_id // k_batch):\n            for label in data:\n                X_data.extend(self.X_data[data[label][i*k_batch:(i+1)*k_batch]])\n                y_data += [label] * k_batch\n\n        self.X_data = np.array(X_data)\n        self.y_data = np.array(y_data)\n\n    def split_data(self):\n        self.X_train = self.X_data[:self.num_train]\n        self.y_train = self.y_data[:self.num_train]\n\n        self.X_val = self.X_data[self.num_train:]\n        self.y_val = self.y_data[self.num_train:]\n\n        del self.X_data, self.y_data\n                    \n    def get_random_batch(self, k = 100):\n        X_batch, y_batch = [], []\n        for label in range(self.config[\"data\"][\"num_classes\"]):\n            X_mask = self.X_val[self.y_val==label]\n            X_batch.extend(np.array([X_mask[np.random.choice(len(X_mask), k, replace=False)]]) if k <= len(X_mask) and k >= 0 else X_mask)\n            y_batch += [label] * k if k <= len(X_mask) and k >= 0 else [label] * len(X_mask)\n        X_batch = np.reshape(X_batch, self.input_shape)\n        return X_batch, np.array(y_batch)\n    \n    def __str__(self):\n        return f\"# of training samples: {self.num_train} | # of validation samples: {self.num_val}\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"609ec05330583df7a585418e8b7b5f6fbc282118"},"cell_type":"code","source":"print(\"[INFO] Loading Data\")\ndataloader = DataLoader(config)\ndataloader.load()\nprint(dataloader)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d388b639c8bad38ba886c5841795f735c2bcf505"},"cell_type":"markdown","source":"##  Define the Data Generator"},{"metadata":{"trusted":true,"_uuid":"a833643b7cf86c686309d505b7e01d3316c59852"},"cell_type":"code","source":"class DataGenerator(object):\n    def __init__(self, config):\n        self.shuffle = config[\"train\"][\"shuffle\"]\n        self.batch_size = config[\"train\"][\"batch-size\"]\n        self.loss = config[\"train\"][\"loss\"]\n        self.num_classes = config[\"data\"][\"num_classes\"]\n\n    def generate(self, X, y):\n        ''' Generates batches of samples '''\n        # Infinite loop\n        while 1:\n            # Generate order of exploration of dataset\n            indexes = self.__get_exploration_order(len(y))\n            # Generate batches\n            batches = np.arange(len(indexes)//self.batch_size)\n            if not self.shuffle: np.random.shuffle(batches)\n\n            for batch in batches:\n                # Find list of ids\n                batch_indecies = indexes[batch*self.batch_size:(batch+1)*self.batch_size]\n                if self.loss == \"triplet-softmax\":\n                    y_1 = y[batch_indecies]\n                    y_2 = np_utils.to_categorical(y_1, self.num_classes)\n                    yield X[batch_indecies], [y_1, y_2]\n                else:\n                    yield X[batch_indecies], y[batch_indecies]\n\n    def __get_exploration_order(self, data_size):\n        ''' Generates order of exploration '''\n        idxs = np.arange(data_size)\n        if self.shuffle == True:\n            np.random.shuffle(idxs)\n        return idxs   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"213c2069be554d22795dd2b1958019ae11511d76"},"cell_type":"code","source":"print(\"[INFO] Creating Generators\")\ntrain_gen = DataGenerator(config).generate(dataloader.X_train, dataloader.y_train)\nval_gen = DataGenerator(config).generate(dataloader.X_val, dataloader.y_val)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3742b94b3d547c80677503a5cde7d6142786c393"},"cell_type":"markdown","source":"## Define the Model\n"},{"metadata":{"trusted":true,"_uuid":"cb293502a749d609894b6abbc38d704be0005c05"},"cell_type":"code","source":"from __future__ import print_function\n\nimport os\nimport yaml\nimport argparse\n\nimport tensorflow as tf\nimport keras.backend as K \nimport keras.optimizers as optimizers\n\nfrom keras import losses\nfrom keras.models import Model\nfrom keras.regularizers import l2\nfrom keras.layers import Conv2D, MaxPooling2D, AveragePooling2D, Input, GlobalAveragePooling2D, LeakyReLU, SeparableConv2D, BatchNormalization, Add\nfrom keras.layers.core import Dense, Dropout, Flatten, Lambda\nfrom keras.callbacks import ReduceLROnPlateau\n\ndef get_model(input_shape, config, top = True):\n    input_img = Input(input_shape)\n    num_classes = config[\"data\"][\"num_classes\"]\n\n    def __body(input_img):\n        x = Conv2D(32, kernel_size=(3, 3), activation='relu')(input_img)\n        x = Conv2D(64, (3, 3), activation='relu')(x)\n        x = MaxPooling2D(pool_size=(2, 2))(x)\n        x = Dropout(0.25)(x)\n        x = Flatten()(x)\n        embedding = Dense(128, activation='relu')(x)\n        return embedding\n\n    def __head(embedding):\n        x   = Dropout(0.5)(embedding)\n        out = Dense(num_classes, activation='softmax')(x)\n        return out\n\n    x = __body(input_img)\n    if config[\"train\"][\"loss\"] in [\"triplet-softmax\"] and top:\n        y = __head(x)\n        model = Model(inputs=input_img, outputs=[x, y])\n    else:\n        if top: x = __head(x)\n        model = Model(inputs=input_img, outputs=x)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7931ef83029b3b349e1fd6c237454481eeffbaf4"},"cell_type":"code","source":"print(\"[INFO] Building Model\")\ninput_shape = (data[\"imsize\"], data[\"imsize\"], data[\"imchannel\"])\nmodel = get_model(input_shape, config, top=train[\"loss\"] in [\"categorical-crossentropy\", \"triplet-softmax\"])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"## Define the Loss Functions"},{"metadata":{"trusted":true,"_uuid":"2f5ddee6d2ddf8b65f5cc0b27a3fc59554d282c3"},"cell_type":"code","source":"import functools\nimport numpy as np\nimport tensorflow as tf\nfrom keras import backend as K\n\ndef get_loss_function(func):\n    return {\n        'triplet-softmax': ([semi_hard_triplet_loss(config[\"train\"][\"alpha\"]), 'categorical_crossentropy'],  [1, train[\"lambda_2\"]]),\n        'large-margin-cosine-loss': (large_margin_cos_loss(config[\"train\"]), None),\n        'intra-enhanced-triplet-loss': (intra_enhanced_triplet_loss(config[\"train\"]), None),\n        'semi-hard-triplet-loss': (semi_hard_triplet_loss(config[\"train\"][\"alpha\"]), None),\n        'categorical-crossentropy': (losses.categorical_crossentropy, None),\n    }.get(func, (losses.categorical_crossentropy, None))\n\ndef __anchor_center_loss(embeddings, margin, batch_size = 400, k = 40):\n    \"\"\"Computes the anchor-center loss\n    Minimizes intra-class distances. Assumes embeddings are ordered \n    such that every k samples belong to the same class, where the \n    number of classes is batch_size // k.\n    Args:\n        embeddings: tensor of shape (batch_size, embed_dim)\n        margin: intra-class distances should be within this margin\n        batch_size: number of embeddings \n        k: number of samples per class in embeddings\n    Returns:\n        loss: scalar tensor containing the anchor-center loss\n    \"\"\"\n    loss = tf.constant(0, dtype='float32')\n    for i in range(0,batch_size,k):\n        anchors = embeddings[i:i+k] \n        center = tf.reduce_mean(anchors, 0)\n        loss = tf.add(loss, tf.reduce_sum(tf.maximum(tf.reduce_sum(tf.square(anchors - center), axis=1) - margin, 0.)))\n    return tf.reduce_mean(loss)\n\ndef __semi_hard_triplet_loss(labels, embeddings, margin = 0.2):\n    return tf.contrib.losses.metric_learning.triplet_semihard_loss(labels[:,0], embeddings, margin=margin)\n\ndef __intra_enhanced_triplet_loss(labels, embeddings, lambda_1, alpha, beta, batch_size, k):\n    return tf.add(__semi_hard_triplet_loss(labels, embeddings, alpha), tf.multiply(lambda_1, __anchor_center_loss(embeddings, beta, batch_size, k)))\n\ndef __large_margin_cos_loss(labels, embeddings, alpha, scale, regularization_lambda, num_cls = 10):\n    num_features = embeddings.get_shape()[1]\n    \n    with tf.variable_scope('centers_scope', reuse = tf.AUTO_REUSE):\n        weights = tf.get_variable(\"centers\", [num_features, num_cls], dtype=tf.float32, \n                initializer=tf.contrib.layers.xavier_initializer(), regularizer=tf.contrib.layers.l2_regularizer(1e-4), trainable=True)\n\n    embedds_feat_norm = tf.nn.l2_normalize(embeddings, 1, 1e-10)\n    weights_feat_norm = tf.nn.l2_normalize(weights, 0, 1e-10)\n\n    xw_norm = tf.matmul(embedds_feat_norm, weights_feat_norm)\n    margin_xw_norm = xw_norm - alpha\n\n    labels = tf.squeeze(tf.cast(labels, tf.int32))\n    label_onehot = tf.one_hot(labels, num_cls)\n    value = scale*tf.where(tf.equal(label_onehot, 1), margin_xw_norm, xw_norm)\n\n    cos_loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(labels=labels, logits=value))\n\n    regularization_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)\n    cos_loss = cos_loss + regularization_lambda * tf.add_n(regularization_losses)\n    return cos_loss \n\ndef semi_hard_triplet_loss(margin):\n    @functools.wraps(__semi_hard_triplet_loss)\n    def loss(labels, embeddings):\n        return __semi_hard_triplet_loss(labels, embeddings, margin)\n    return loss\n\ndef intra_enhanced_triplet_loss(config):\n    @functools.wraps(__intra_enhanced_triplet_loss)\n    def loss(labels, embeddings):\n        return __intra_enhanced_triplet_loss(labels, embeddings, config[\"lambda_1\"], config[\"alpha\"], config[\"beta\"], config[\"batch-size\"], config[\"k_batch\"])\n    return loss\n\ndef large_margin_cos_loss(config):\n    @functools.wraps(__large_margin_cos_loss)\n    def loss(labels, embeddings):\n        return __large_margin_cos_loss(labels, embeddings, config[\"alpha\"], config[\"scale\"], config[\"reg_lambda\"])\n    return loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"39301c3b41b304f8eba23583133aa046cfe24f89"},"cell_type":"code","source":"loss_func, loss_weights = get_loss_function(train[\"loss\"])\noptim = getattr(optimizers, train[\"optim\"])(train[\"lr\"])\nmodel.compile(loss=loss_func, loss_weights=loss_weights, optimizer=optim, metrics=[])\n\nreduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=train[\"lr_reduce_factor\"], patience=train[\"patience\"], min_lr=train[\"min_lr\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5680af3a87249439de2494624f4c39532ce26d54"},"cell_type":"code","source":"print(\"[INFO] Start Training\")\nmodel.fit_generator(\n    generator = train_gen,\n    steps_per_epoch = dataloader.num_train//train[\"batch-size\"],\n    validation_data = val_gen,\n    validation_steps= dataloader.num_val//train[\"batch-size\"],\n    shuffle = False,\n    workers = 0,\n    epochs = train[\"epochs\"],\n    callbacks=[reduce_lr]\n)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a614593f1315bb69e1c5d641bcb73ed8874f7360"},"cell_type":"markdown","source":"## It is Time for t-SNE Visualization "},{"metadata":{"trusted":true,"_uuid":"41378d1725ed3ada9e8abe22430ccbdc52b4937f"},"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\n\ndef scatter(x, labels, config, run_title = \"MNIST\"):\n    palette = np.array(sns.color_palette(\"hls\", config[\"data\"][\"num_classes\"]))\n\n    fig, ax = plt.subplots()\n    ax.scatter(x[:,0], x[:,1], lw=0, s=40, alpha=0.2, c=palette[labels.astype(np.int)])\n\n    for idx in range(config[\"data\"][\"num_classes\"]):\n        xtext, ytext = np.median(x[labels == idx, :], axis=0)\n        txt = ax.text(xtext, ytext, str(idx), fontsize=20)\n\n    plt.title(f\"{run_title} T-SNE\")\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"37ad14130a631b04de2d1b161a0ecdc419179961"},"cell_type":"code","source":"from sklearn.manifold import TSNE\n\ndef visualize(embeddings, run_title = \"MNIST\"):\n    tsne = TSNE(n_components=2, perplexity=config[\"tsne\"][\"perplexity\"], verbose=1, n_iter=config[\"tsne\"][\"n_iter\"])\n    tsne_embeds = tsne.fit_transform(embeddings)\n    scatter(tsne_embeds, y_batch, config, run_title)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"005a2d562aa66be3c1393e34d2944f804498ebec"},"cell_type":"markdown","source":"# Let's First Establish a Baseline\nFirst let's visualize t-SNE embeddings on the raw pixel data to give us a baseline of how the data is distributed before any clustering work is done."},{"metadata":{"trusted":true,"_uuid":"0babcf4f707643a2bb7892974d047c36f6e47581"},"cell_type":"code","source":"X_batch, y_batch = dataloader.get_random_batch(k = -1)\nembeddings = X_batch.reshape(-1, 784) \nvisualize(embeddings, \"Pixels\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5cb842063f167058829ab5e68bc8fe334ae7009"},"cell_type":"markdown","source":"## Using Semi-Hard Triplet Loss + Softmax"},{"metadata":{"trusted":true,"_uuid":"30db238163b7286dad42525d37ef3832e301355d"},"cell_type":"code","source":"X_batch, y_batch = dataloader.get_random_batch(k = -1)\nembeddings = model.predict(X_batch, batch_size=config[\"train\"][\"batch-size\"], verbose=1)\nif train[\"loss\"] in [\"triplet-softmax\"]:\n    embeddings = embeddings[0]\nvisualize(embeddings)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"14e870114318f422dc68bfb9cb28dd58fed54bc6"},"cell_type":"markdown","source":"## Conclusion\nYou can find the visualizations of the other loss functions in my github repo: https://github.com/BKHMSI/LossMNIST, or more interestingly you can try changing the configuration parameters and try out for yourself.\n\nHope that might be useful for anyone!"},{"metadata":{"_uuid":"792b3fd707b354bc4529edfaf091d042a538b60a"},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}