{"nbformat":4,"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"version":"3.6.5","nbconvert_exporter":"python","mimetype":"text/x-python","pygments_lexer":"ipython3","file_extension":".py","codemirror_mode":{"version":3,"name":"ipython"},"name":"python"}},"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"1204d67e-ed68-417c-bd67-992adc891659","_uuid":"32c591c67f5ea174da6bbfac741430df6c8d33fb"},"source":"# Introduction"},{"cell_type":"markdown","metadata":{"_cell_guid":"89364420-68a9-4155-9ea4-0c521e1872ce","_uuid":"af97e67fbac96818de2736ffd89b6d692ab6d4f8"},"source":"Learning the [CapsNet architecture](https://arxiv.org/abs/1710.09829) with Keras by applying on the MNIST dataset.\n\nI followed the [Keras implementation of CapsNet by Xifeng Guo](https://github.com/XifengGuo/CapsNet-Keras). Mine's a much more simpler, dumbed-down version.\n\nAlso, [\"Understanding Hinton’s Capsule Networks\"](https://medium.com/ai%C2%B3-theory-practice-business/understanding-hintons-capsule-networks-part-ii-how-capsules-work-153b6ade9f66) by Max Pechyonkin are an excellent series of posts for CapsNet reference (particularly part 2 and 3 for me)."},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\n\nfrom keras import backend as K\nfrom keras.layers import Layer, Input, Conv2D, Reshape, Lambda\nfrom keras.utils import to_categorical\nfrom keras import initializers","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"outputs":[],"execution_count":1},{"cell_type":"markdown","metadata":{"_cell_guid":"3baf55db-d64f-497a-8f0b-95d37c9ef6f7","_uuid":"4980119ee5f50e8f178044976e4f03afd1e4ad22"},"source":"## Data Exploration"},{"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')","metadata":{"collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"},"outputs":[],"execution_count":2},{"cell_type":"code","source":"train_Y = train.label\nprint(train_Y.head())\ntrain_Y = to_categorical(train_Y, num_classes=10)\nprint(train_Y[0])","metadata":{"_cell_guid":"7895ea5d-eee6-4324-b9e5-4ea6d08a0cfc","_uuid":"dcc11004781097096d7e5c208d73573948106a75"},"outputs":[],"execution_count":3},{"cell_type":"code","source":"train_X = train.drop('label', axis=1)\ntrain_X = train_X / 255\ntrain_X.head()\nprint(train_X.shape)\ntrain_X = train_X.values.reshape((-1,28,28,1))\nprint(train_X.shape)","metadata":{"_cell_guid":"ce908588-f815-47ef-a2b9-3486b8c78976","_uuid":"fcf2ad710ba58195e40e4919682861ea24b40303"},"outputs":[],"execution_count":4},{"cell_type":"markdown","metadata":{"_cell_guid":"2a14b0eb-743f-42a5-a3ba-f9469882bbd7","_uuid":"d42def3c7027c2132f65cf4da50935d992d241c9"},"source":"## CapsNet Architecture"},{"cell_type":"code","source":"def squash(vectors, axis=-1):\n    \"\"\"\n    In \"Dynamic Routing Between Capsules:\n    \n    We want the length of the output vector of a capsule to represent the probability that the entity\n    represented by the capsule is present in the current input. We therefore use a non-linear \"squashing\"\n    function to ensure that short vectors get shrunk to almost zero length and long vectors get shrunk to a\n    length slightly below 1. We leave it to discriminative learning to make good use of this non-linearity.\n    \"\"\"\n    squared = K.square(vectors)\n    squared_summed = K.sum(squared, axis, keepdims=True)\n    scale_factor = squared_summed / (squared_summed + 1) / K.sqrt(squared_summed + K.epsilon()) # not sure what the epsilon does...\n    \n    return scale_factor * vectors","metadata":{"collapsed":true,"_cell_guid":"75baa87c-bc9c-4b48-a943-b704f10aa155","_uuid":"d7ec4735c37a6cac3a24f9c9f4d577923b9f3a63"},"outputs":[],"execution_count":5},{"cell_type":"code","source":"def PrimaryCap(inputs):\n    \"\"\"\n    In \"Dynamic Routing Between Capsules\":\n    \n    The second layer (PrimaryCapsules) is a convolutional capsule layer with 32 channels of convolutional 8D capsules \n    (i.e. each primary capsule contains 8 convolutional units with a 9 × 9 kernel and a stride of 2).\n    \n    From Convolution Layer 1 an input of (20, 20, 256). \n    One convolution with a (9, 9, 256) capsule that strides by (2, 2) in the first and second dimensions results in a shape of (6, 6).\n    The capsule is 8 deep. Therefore the output results in (6, 6, 8)\n    There are 32 capsultes. So finally, the output is (6, 6, 8, 32)\n    \"\"\"\n    # Treat the 8 deep, 32 capsules as one super deep conv layer\n    output = Conv2D(filters=8 * 32, kernel_size=9, strides=(2, 2), padding='valid', activation=None)(inputs)\n    # Reshape into by-capsule output\n    outputs = Reshape(target_shape=(-1, 8))(output)\n    # scaling the capsule outputs\n    # this is the activation\n    return Lambda(squash)(outputs)","metadata":{"collapsed":true,"_cell_guid":"67063cae-e163-44ed-aef7-77af239df749","_uuid":"89efa7f352414a02068eaa4ddd986832ab63a338"},"outputs":[],"execution_count":6},{"cell_type":"markdown","metadata":{"_cell_guid":"b40971d6-d002-47aa-acb9-3e8ee5a16c68","_uuid":"0ad73e15ed732875b0b8d3ea599b28e2093335d1"},"source":"> Lower level capsule will send its input to the higher level capsule that “agrees” with its input. This is the essence of the dynamic routing algorithm."},{"cell_type":"markdown","metadata":{"_cell_guid":"c9db7edc-bfcb-4d99-ad45-c2de6d430c22","_uuid":"82083154f2b312c8c1aea58b6dbafdf0fa302593"},"source":"The `CapsuleLayer` class is adapted from [the mentioned Keras implementation.](https://github.com/XifengGuo/CapsNet-Keras/blob/master/capsulelayers.py#L79)"},{"cell_type":"code","source":"class CapsuleLayer(Layer):\n    def __init__(self, \n                 num_capsule, \n                 dim_capsule,\n                 routings=3,\n                 kernel_initializer='glorot_uniform',\n                 **kwargs):\n        super(CapsuleLayer, self).__init__(**kwargs)\n        self.num_capsule = num_capsule\n        self.dim_capsule = dim_capsule\n        self.routings = routings\n        self.kernel_initializer = initializers.get(kernel_initializer)\n\n    def build(self, input_shape):\n        assert len(input_shape) == 3, \"The input Tensor should have shape=[None, input_num_capsule, input_dim_capsule]\"\n        self.input_num_capsule = input_shape[1]\n        self.input_dim_capsule = input_shape[2]\n\n        # Transform matrix\n        self.W = self.add_weight(shape=(self.num_capsule, self.input_num_capsule,\n                                        self.dim_capsule, self.input_dim_capsule),\n                                 initializer=self.kernel_initializer,\n                                 name='W')\n\n        self.built = True\n\n    def call(self, inputs, training=None):\n        # inputs.shape=[None, input_num_capsule, input_dim_capsule]\n        # inputs_expand.shape=[None, 1, input_num_capsule, input_dim_capsule]\n        print(inputs)\n        print(inputs.shape)\n        inputs_expand = K.expand_dims(inputs, 1)\n        print(inputs_expand.shape)\n\n        # Replicate num_capsule dimension to prepare being multiplied by W\n        # inputs_tiled.shape=[None, num_capsule, input_num_capsule, input_dim_capsule]\n        inputs_tiled = K.tile(inputs_expand, [1, self.num_capsule, 1, 1])\n        print(inputs_tiled.shape)\n\n        # Compute `inputs * W` by scanning inputs_tiled on dimension 0.\n        # x.shape=[num_capsule, input_num_capsule, input_dim_capsule]\n        # W.shape=[num_capsule, input_num_capsule, dim_capsule, input_dim_capsule]\n        # Regard the first two dimensions as `batch` dimension,\n        # then matmul: [input_dim_capsule] x [dim_capsule, input_dim_capsule]^T -> [dim_capsule].\n        # inputs_hat.shape = [None, num_capsule, input_num_capsule, dim_capsule]\n        inputs_hat = K.map_fn(lambda x: K.batch_dot(x, self.W, [2, 3]), elems=inputs_tiled)\n\n        # Begin: Routing algorithm ---------------------------------------------------------------------#\n        # The prior for coupling coefficient, initialized as zeros.\n        # b.shape = [None, self.num_capsule, self.input_num_capsule].\n        b = tf.zeros(shape=[K.shape(inputs_hat)[0], self.num_capsule, self.input_num_capsule])\n\n        assert self.routings > 0, 'The routings should be > 0.'\n        for i in range(self.routings):\n            # c.shape=[batch_size, num_capsule, input_num_capsule]\n            c = tf.nn.softmax(b, axis=1)\n\n            # c.shape =  [batch_size, num_capsule, input_num_capsule]\n            # inputs_hat.shape=[None, num_capsule, input_num_capsule, dim_capsule]\n            # The first two dimensions as `batch` dimension,\n            # then matmal: [input_num_capsule] x [input_num_capsule, dim_capsule] -> [dim_capsule].\n            # outputs.shape=[None, num_capsule, dim_capsule]\n            outputs = squash(K.batch_dot(c, inputs_hat, [2, 2]))  # [None, 10, 16]\n\n            if i < self.routings - 1:\n                # outputs.shape =  [None, num_capsule, dim_capsule]\n                # inputs_hat.shape=[None, num_capsule, input_num_capsule, dim_capsule]\n                # The first two dimensions as `batch` dimension,\n                # then matmal: [dim_capsule] x [input_num_capsule, dim_capsule]^T -> [input_num_capsule].\n                # b.shape=[batch_size, num_capsule, input_num_capsule]\n                b += K.batch_dot(outputs, inputs_hat, [2, 3])\n        # End: Routing algorithm -----------------------------------------------------------------------#\n\n        return outputs\n\n    def compute_output_shape(self, input_shape):\n        return tuple([None, self.num_capsule, self.dim_capsule])","metadata":{"collapsed":true,"_cell_guid":"3c4a7ec8-4fd9-4116-aa40-491af7363bb4","_uuid":"1df7d5127426d65283cfc6495d2609461822cc6a"},"outputs":[],"execution_count":7},{"cell_type":"code","source":"def capsnet(input_shape):\n    input_ = Input(shape=input_shape)\n    \"\"\"\n    Convolution Layer 1\n    \n    A \"window\" of (9, 9) strides with (1, 1) on a (28, 28) image, it can step (20, 20) times.\n    There are 256 of these \"windows\" stacked.\n    \n    Therefore, the output of this convolution is (20, 20, 256).\n    \"\"\"\n    conv_1 = Conv2D(filters=256, kernel_size=9, strides=(1, 1), padding='valid', activation='relu')(input_)\n    \n    primary_caps = PrimaryCap(inputs=conv_1)\n    \n    digit_caps = CapsuleLayer(num_capsule=10, dim_capsule=16, routings=3)(primary_caps)\n    \n    \n    ","metadata":{"collapsed":true,"_cell_guid":"35ade5bb-55b6-476a-ab22-b9d1f78d14e9","_uuid":"80b52fc419d9888d80b4672b50d7957f916b749f"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"capsnet(input_shape=train_X.shape[1:])","metadata":{"collapsed":true,"_cell_guid":"4900e9ec-c0e6-4722-bb82-0ea8e8618e94","_uuid":"cfdf190132f41d84f612bb6702358716d3b7804e"},"outputs":[],"execution_count":null}],"nbformat_minor":1}