{"metadata": {"widgets": {"version": "1.2.0", "state": {"e1ab5b4a39ce44d6b971fa88c5239f96": {"views": [{"cell_index": 31}]}, "8849e11e93984ab8bc28344665c6d169": {"views": [{"cell_index": 31}]}, "2adc54b180ad4163a0ca046b79566286": {"views": [{"cell_index": 31}]}, "a03c633ff1904d758ea694a986d9a937": {"views": [{"cell_index": 31}]}, "18170041285649418703000e7d39cf5a": {"views": [{"cell_index": 31}]}, "52274d2d51984dda8df4eb3099093211": {"views": [{"cell_index": 31}]}, "c32932c9f6114309849d0ade2445e223": {"views": [{"cell_index": 13}]}, "d7163e941c304038af7e492059642124": {"views": [{"cell_index": 31}]}, "2d3b8ee4bb7f4e9480372ac82f3cd2f2": {"views": [{"cell_index": 31}]}, "d1ed10fc731449b4b48a5b617a591dc6": {"views": [{"cell_index": 13}]}, "006cabe51a2b4329818dd0ece816991f": {"views": [{"cell_index": 31}]}, "82130659dd7748239e932b7ab3ffa9b9": {"views": [{"cell_index": 13}]}, "289fe67b25104e2b9cca19712b363328": {"views": [{"cell_index": 31}]}, "34c902f8d21b499a871bea2b8ae54a03": {"views": [{"cell_index": 31}]}, "87641e89f34e48d199517c2dd42694a9": {"views": [{"cell_index": 31}]}}}, "language_info": {"version": "3.6.3", "pygments_lexer": "ipython3", "nbconvert_exporter": "python", "codemirror_mode": {"version": 3, "name": "ipython"}, "file_extension": ".py", "name": "python", "mimetype": "text/x-python"}, "anaconda-cloud": {}, "kernelspec": {"display_name": "Python 3", "name": "python3", "language": "python"}}, "nbformat_minor": 1, "cells": [{"metadata": {"_uuid": "f87a89f3a087c93479ba68f792b6c1439192d60d", "_cell_guid": "5ccb7d2f-124d-4cf8-8dd2-a1ac27e04dd0", "collapsed": true}, "cell_type": "markdown", "source": ["# Deep learning approach - CNN with chainer\n", "\n", "We train CNN (Convolutional Neural Network), which is widely used in image recognition.\n", "\n", "Also, I will explain several technics such as \"data augmentation\" and \"model ensemble\" \n", "to further improve the score.\n", "\n", "I achieved the score \"0.99700\" in below code.\n", "\n", "chainer is deep learning framework written in python.\n", "It supports dynamic graph construction (define-by-run).\n", "Chainer is used in research field as well, since it is quite flexible library.\n", "\n", " - https://github.com/chainer/chainer\n", " \n", " \n", "## Other reference for beginners\n", "\n", "If you are not familiar with deep learning/any deep learning framework, you might want to see more simple quick start kernel at first.\n", "Below kernel shows much more easy way to apply deep learning training, with sklearn interface.\n", "\n", " - https://www.kaggle.com/corochann/simple-deep-learning-with-chainer-by-sklearn-if\n", " \n", "If you want to learn about chainer, you can see the chainer tutorial.\n", " - http://corochann.com/deep-learning-tutorial-with-chainer\n", " - https://github.com/mitmul/chainer-handson\n", " \n", " "]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "10b73d9e9357a7a60bba61793538997604fe63ef", "_cell_guid": "8174d4df-2e82-4bfc-8414-918f756be8dc", "collapsed": true}, "cell_type": "code", "source": ["# Make sure you installed latest version of scikit-learn\n", "#!pip install -U scikit-learn\n", "\n", "# You may install `chaineripy` to show training progress bar nicely on jupyter notebook.\n", "# https://github.com/grafi-tt/chaineripy\n", "#!pip install chaineripy"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "59a82e5e331e23ff99f0e3ebb8f8e884fe3113a6", "_cell_guid": "b49b7504-92b7-4b56-aa53-d9fb268b7422", "collapsed": true}, "cell_type": "code", "source": ["# Make it False when you want to execute full training.\n", "# It takes a long time to train deep CNN with CPU, but much less time with GPU.\n", "# It is nice if you can utilize GPU, when DEBUG = False.\n", "DEBUG = True"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "26d9514d0b2886713f9b63e48804d904752c1aac", "_cell_guid": "3ecfd2dc-c9eb-4210-a246-51a76d286207"}, "cell_type": "code", "source": ["import os\n", "\n", "import pandas as pd\n", "import numpy as np\n", "\n", "# Load data\n", "print('Loading data...')\n", "DATA_DIR = '../input'\n", "#DATA_DIR = './input'\n", "train = pd.read_csv(os.path.join(DATA_DIR, 'train.csv'))\n", "test = pd.read_csv(os.path.join(DATA_DIR, 'test.csv'))\n", "\n", "train_x = train.iloc[:, 1:].values.astype('float32')\n", "train_y = train.iloc[:, 0].values.astype('int32')\n", "test_x = test.values.astype('float32')\n", "\n", "print('train_x', train_x.shape)\n", "print('train_y', train_y.shape)\n", "print('test_x', test_x.shape)"]}, {"metadata": {"_uuid": "e7430b41ac73a1f9fd50a4b48dbc4b98ce532ef3", "_cell_guid": "bd90a4e1-2337-4636-97b1-9d0a2fa4372d"}, "cell_type": "markdown", "source": ["Image pixel is aligned to one-dimension now,\n", "we need to reshape it to form image tensor.\n", "Each axis represents as follows,\n", " - 1st axis: Mini batch (42000 training imgs and 28000 test images)\n", " - 2nd axis: Channel (here is 1, because this is gray scale image)\n", " - 3rd axis: Height (28 px for MNIST)\n", " - 4th axis: Width (28 px for MNIST)\n", " \n", "Also black intensity is represented by 0 - 255 in original images,\n", "let's rescale it to 0 - 1 so that neural network can handle the range."]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "0bf6d3b9bb37e82341876bbc3056178756477f00", "_cell_guid": "c2c538f2-d192-4053-80e1-be2fc5fc9841"}, "cell_type": "code", "source": ["# reshape and rescale value\n", "train_imgs = train_x.reshape((-1, 1, 28, 28)) / 255.\n", "test_imgs = test_x.reshape((-1, 1, 28, 28)) / 255.\n", "print('train_imgs', train_imgs.shape, 'test_imgs', test_imgs.shape)"]}, {"metadata": {"_uuid": "788590033e0e39af2c221442abc33f0e85a7ad38", "_cell_guid": "c1f8fe33-3421-4418-b185-4e6240bbb6c8"}, "cell_type": "markdown", "source": ["## Visualize image\n", "\n", "Let's quickly check train_imgs by visualizing this hand-writtein digit image."]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "dc344d4b952c51cbcc9806b457ccbe664439cbb9", "_cell_guid": "c2a9bafd-3c75-4e8d-9e70-14bf04e16084"}, "cell_type": "code", "source": ["%matplotlib inline\n", "#import matplotlib\n", "#matplotlib.use('agg')\n", "import matplotlib.pyplot as plt\n", "\n", "\n", "def show_image(img):\n", "    plt.figure(figsize=(1.5, 1.5))\n", "    plt.axis('off')\n", "    if img.ndim == 3:\n", "        img = img[0, :, :]\n", "    plt.imshow(img, cmap=plt.cm.binary)           \n", "    plt.show()\n", "\n", "print('index0, label {}'.format(train_y[0]))\n", "show_image(train_imgs[0])\n", "print('index1, label {}'.format(train_y[1]))\n", "show_image(train_imgs[1])\n", "#show_image(train_imgs[2])\n", "#show_image(train_imgs[3])"]}, {"metadata": {"_uuid": "28002b2a782211032e404f3f19a2502d6ed5d426", "_cell_guid": "1df4a92a-6ba3-44dd-8628-4a1a01c70bb9"}, "cell_type": "markdown", "source": ["## Define CNN (Convolutional Neural Network)\n", "\n", "Required package:\n", " - chainer >= 2.0\n", "\n", "Convolutional neural network consists of convolution layer followed by fully connected layer."]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "d5cdc4d861a77d824355e4e903650900a66649b0", "_cell_guid": "8067aa9a-8391-46a8-9d8b-74f6eb894006"}, "cell_type": "code", "source": ["import chainer\n", "import chainer.links as L\n", "import chainer.functions as F\n", "from chainer.dataset.convert import concat_examples\n", "\n", "\n", "class CNNMedium(chainer.Chain):\n", "    def __init__(self, n_out):\n", "        super(CNNMedium, self).__init__()\n", "        with self.init_scope():\n", "            self.conv1 = L.Convolution2D(None, 16, 3, 1)\n", "            self.conv2 = L.Convolution2D(16, 32, 3, 1)\n", "            self.conv3 = L.Convolution2D(32, 32, 3, 1)\n", "            self.conv4 = L.Convolution2D(32, 32, 3, 2)\n", "            self.conv5 = L.Convolution2D(32, 64, 3, 1)\n", "            self.conv6 = L.Convolution2D(64, 32, 3, 1)\n", "            self.fc7 = L.Linear(None, 30)\n", "            self.fc8 = L.Linear(30, n_out)\n", "\n", "    def __call__(self, x):\n", "        h = F.leaky_relu(self.conv1(x), slope=0.05)\n", "        h = F.leaky_relu(self.conv2(h), slope=0.05)\n", "        h = F.leaky_relu(self.conv3(h), slope=0.05)\n", "        h = F.leaky_relu(self.conv4(h), slope=0.05)\n", "        h = F.leaky_relu(self.conv5(h), slope=0.05)\n", "        h = F.leaky_relu(self.conv6(h), slope=0.05)\n", "        h = F.leaky_relu(self.fc7(h), slope=0.05)\n", "        h = self.fc8(h)\n", "        return h\n", "\n", "    def _predict_batch(self, x_batch):\n", "        with chainer.no_backprop_mode(), chainer.using_config('train', False):\n", "            h = self.__call__(x_batch)\n", "            return F.softmax(h)\n", "\n", "    def predict_proba(self, x, batchsize=32, device=-1):\n", "        if device >= 0:\n", "            chainer.cuda.get_device_from_id(device).use()\n", "            self.to_gpu()  # Copy the model to the GPU\n", "\n", "        y_list = []\n", "        for i in range(0, len(x), batchsize):\n", "            x_batch = concat_examples(x[i:i + batchsize], device=device)\n", "            y = self._predict_batch(x_batch)\n", "            y_list.append(chainer.cuda.to_cpu(y.data))\n", "        y_array = np.concatenate(y_list, axis=0)\n", "        return y_array\n", "\n", "    def predict(self, x, batchsize=32, device=-1):\n", "        proba = self.predict_proba(x, batchsize=batchsize, device=device)\n", "        return np.argmax(proba, axis=1)"]}, {"metadata": {"_uuid": "4853fab3be1f4dbcd4d614b9e16d4bc9086a4fa2", "_cell_guid": "f25ea8af-08df-4b36-85b4-3fc08eb2794c"}, "cell_type": "markdown", "source": ["## Training code -- step 1."]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "d71b4a56d1955c595078c48d2bd7dbe37eca7a8f", "_cell_guid": "15e148c3-1b04-472a-93c8-d96401e6c296"}, "cell_type": "code", "source": ["if DEBUG:\n", "    print('DEBUG mode, reduce training data...')\n", "    # Use only first 1000 example to reduce training time\n", "    train_x = train_x[:1000]\n", "    train_imgs = train_imgs[:1000]\n", "    train_y = train_y[:1000]\n", "else:\n", "    print('No DEBUG mode')"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "5c59c2c8b8d0d068b133c821b83e9147f46bb1bf", "_cell_guid": "41535a2b-5a9d-41a3-a137-7bf3ef64cde9"}, "cell_type": "code", "source": ["from chainer import iterators, training, optimizers, serializers\n", "from chainer.datasets import TupleDataset\n", "from chainer.training import extensions\n", "\n", "\n", "# -1 indicates to use CPU, \n", "# positive value indicates GPU device id.\n", "device = -1  # If you use CPU.\n", "#device = 0  # If you use GPU. (You need to install chainer & cupy with CUDA/cudnn installed)\n", "batchsize = 16\n", "class_num = 10\n", "out_dir = '.'\n", "if DEBUG:\n", "    epoch = 5  # This value is small. Change to more than 20 for Actual running.\n", "else:\n", "    epoch = 20\n", "\n", "\n", "def train_main(train_x, train_y, val_x, val_y, model_path='cnn_model.npz'):\n", "    # 1. Setup model    \n", "    model = CNNMedium(n_out=class_num)\n", "    classifier_model = L.Classifier(model)\n", "    if device >= 0:\n", "        chainer.cuda.get_device(device).use()  # Make a specified GPU current\n", "        classifier_model.to_gpu()  # Copy the model to the GPU\n", "\n", "    # 2. Setup an optimizer\n", "    optimizer = optimizers.Adam()\n", "    #optimizer = optimizers.MomentumSGD(lr=0.001)\n", "    optimizer.setup(classifier_model)\n", "\n", "    # 3. Load the dataset\n", "    #train_data = MNISTTrainImageDataset()\n", "    train_dataset = TupleDataset(train_x, train_y)\n", "    val_dataset = TupleDataset(val_x, val_y)\n", "\n", "    # 4. Setup an Iterator\n", "    train_iter = iterators.SerialIterator(train_dataset, batchsize)\n", "    #train_iter = iterators.MultiprocessIterator(train, args.batchsize, n_prefetch=10)\n", "    val_iter = iterators.SerialIterator(val_dataset, batchsize, repeat=False, shuffle=False)\n", "    \n", "    # 5. Setup an Updater\n", "    updater = training.StandardUpdater(train_iter, optimizer, \n", "                                       device=device)\n", "    # 6. Setup a trainer (and extensions)\n", "    trainer = training.Trainer(updater, (epoch, 'epoch'), out=out_dir)\n", "\n", "    # Evaluate the model with the test dataset for each epoch\n", "    trainer.extend(extensions.Evaluator(val_iter, classifier_model, device=device), trigger=(1, 'epoch'))\n", "\n", "    trainer.extend(extensions.dump_graph('main/loss'))\n", "    trainer.extend(extensions.snapshot(), trigger=(1, 'epoch'))\n", "    trainer.extend(extensions.LogReport())\n", "    trainer.extend(extensions.PlotReport(\n", "        ['main/loss', 'validation/main/loss'],\n", "        x_key='epoch', file_name='loss.png'))\n", "    trainer.extend(extensions.PlotReport(\n", "        ['main/accuracy', 'validation/main/accuracy'],\n", "        x_key='epoch',\n", "        file_name='accuracy.png'))\n", "\n", "    try:\n", "        # Use extension library, chaineripy's PrintReport & ProgressBar\n", "        from chaineripy.extensions import PrintReport, ProgressBar\n", "        trainer.extend(ProgressBar(update_interval=5))\n", "        trainer.extend(PrintReport(\n", "            ['epoch', 'main/loss', 'validation/main/loss',\n", "             'main/accuracy', 'validation/main/accuracy', 'elapsed_time']))\n", "\n", "    except:\n", "        print('chaineripy is not installed, run `pip install chaineripy` to show rich UI progressbar')\n", "        # Use chainer's original ProgressBar & PrintReport\n", "        # trainer.extend(extensions.ProgressBar(update_interval=5))\n", "        trainer.extend(extensions.PrintReport(\n", "            ['epoch', 'main/loss', 'validation/main/loss',\n", "             'main/accuracy', 'validation/main/accuracy', 'elapsed_time']))\n", "\n", "    # Resume from a snapshot\n", "    # serializers.load_npz(args.resume, trainer)\n", "\n", "    # Run the training\n", "    trainer.run()\n", "    # Save the model\n", "    serializers.save_npz('{}/{}'\n", "                         .format(out_dir, model_path), model)\n", "    return model"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "7abad5e4d65d50d40131bec490dfed0952007f37", "_cell_guid": "57190b78-7555-4f6b-8718-c737213d84b2"}, "cell_type": "code", "source": ["import numpy as np\n", "from sklearn.model_selection import train_test_split\n", "seed = 777\n", "model_simple = 'cnn_model_simple.npz'\n", "\n", "train_idx, val_idx = train_test_split(np.arange(len(train_x)),\n", "                                      test_size=0.20, random_state=seed)\n", "print('train size', len(train_idx), 'val size', len(val_idx))\n", "train_main(train_imgs[train_idx], train_y[train_idx], train_imgs[val_idx], train_y[val_idx], model_path=model_simple)"]}, {"metadata": {"_uuid": "044d4c9e9684282680d81b8db30e01b83e9b3a36", "_cell_guid": "83d8e9c3-3726-47e9-9a5e-970371cf4395", "collapsed": true}, "cell_type": "markdown", "source": ["## Predict the label and make submission file\n", "\n", "Load trained model and create submit file."]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "6561eaae186250baab459e28dddac66d83b50c04", "_cell_guid": "126b4b22-d85c-4e18-828b-f3ea7320ec34", "collapsed": true}, "cell_type": "code", "source": ["class_num = 10\n", "\n", "def predict_main(model_path='cnn_model.npz'):\n", "    # 1. Setup model\n", "    model = CNNMedium(n_out=class_num)\n", "    classifier_model = L.Classifier(model)\n", "    if device >= 0:\n", "        chainer.cuda.get_device(device).use()  # Make a specified GPU current\n", "        classifier_model.to_gpu()  # Copy the model to the GPU\n", "\n", "    # load trained model\n", "    serializers.load_npz(model_path, model)\n", "\n", "    # 2. Prepare the dataset --> it's already prepared\n", "    # test_imgs\n", "\n", "    # 3. predict the result\n", "    t = model.predict(test_imgs, device=device)\n", "    return t\n", " \n", "\n", "def create_submission(submission_path, t):\n", "    result_dict = {\n", "        'ImageId': np.arange(1, len(t) + 1),\n", "        'Label': t\n", "    }\n", "    df = pd.DataFrame(result_dict)\n", "    df.to_csv(submission_path,\n", "              index_label=False, index=False)\n", "    print('submission file saved to {}'.format(submission_path))"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "afbb1934ef76b7e420df4a31c8dd428ad0c0014a", "_cell_guid": "97657348-d27d-414c-aa12-8a666a6e04c5"}, "cell_type": "code", "source": ["predict_label = predict_main(model_path=model_simple)\n", "print('predict_label = ', predict_label, predict_label.shape)\n", "\n", "create_submission('submission_simple.csv', predict_label)"]}, {"metadata": {"_uuid": "4b866b9a198e42fcb65b7cd5af747d9219b6c5af", "_cell_guid": "45e9ccd3-188f-4940-a1bf-2271bb372a39"}, "cell_type": "markdown", "source": ["## Submit file: score 0.96400\n", "\n", "We have finished the basic course of Convolutional Neural Network training and prediction by traind CNN.\n", "The above submission file scored 0.96400 in my environment.\n", "\n", "How to further improve the score? Let's consider the advanced step below..."]}, {"metadata": {"_uuid": "e239f1d1d2f152066c098b612f4de08bbb85b5fe", "_cell_guid": "086be860-b5d8-4ad7-b289-e5c2f5cd58a1"}, "cell_type": "markdown", "source": ["## Improve model performance\n", "\n", "The simple model training is introduced above, \n", "but the accuracy is not enough.\n", "\n", "We can further increase the model performance by considering\n", "\n", " - Data augmentation\n", " - model emsemble\n", "\n", "From below, I will explain how to achieve data augmentation training with chainer, \n", "and train multiple models to emsemble its prediction to get final output."]}, {"metadata": {"_uuid": "b258fecfbb3d35c9ce0f461466c750400f75f722", "_cell_guid": "8b515245-f820-4885-b89a-70e03f784356"}, "cell_type": "markdown", "source": ["## Data augmentation\n", "\n", "\n", "To increase validation score, the number of training data is important.\n", "When we can use more number of training data, we can reduce overfitting and validation score becomes high.\n", "\n", "\"Data augmentation\" is a technic to virtually create extra training data, based on the given training data.\n", "For this MNIST task, data augmentation can be achieved by utilizing affine transformation.\n", "\n", "1. Rotation\n", "AffineTransformation\n", "2. Translation\n", "3. Scale\n", "4. Shear\n", "\n", "I show example below."]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "02f800985419eeaa73926ef6af631b553e82ad3b", "_cell_guid": "759e9cac-66c4-4658-8901-815533911245"}, "cell_type": "code", "source": ["from chainer.datasets import TransformDataset\n", "import skimage\n", "import skimage.transform\n", "from skimage.transform import AffineTransform, warp\n", "import numpy as np\n", "\n", "def affine_image(img):\n", "    #ch, h, w = img.shape\n", "    #img = img / 255.\n", "    \n", "    # --- scale ---\n", "    min_scale = 0.8\n", "    max_scale = 1.2\n", "    sx = np.random.uniform(min_scale, max_scale)\n", "    sy = np.random.uniform(min_scale, max_scale)\n", "    \n", "    # --- rotation ---\n", "    max_rot_angle = 7\n", "    rot_angle = np.random.uniform(-max_rot_angle, max_rot_angle) * np.pi / 180.\n", "    \n", "    # --- shear ---\n", "    max_shear_angle = 10\n", "    shear_angle = np.random.uniform(-max_shear_angle, max_shear_angle) * np.pi / 180.\n", "    \n", "    # --- translation ---\n", "    max_translation = 4\n", "    tx = np.random.randint(-max_translation, max_translation)\n", "    ty = np.random.randint(-max_translation, max_translation)\n", "    \n", "    tform = AffineTransform(scale=(sx, sy), rotation=rot_angle, shear=shear_angle, \n", "                            translation=(tx, ty))\n", "    transformed_image = warp(img[0, :, :], tform.inverse, output_shape=(28, 28))\n", "    return transformed_image\n", "\n", "transformed_imgs = TransformDataset(train_imgs / 255., affine_image)\n", "\n", "print('Affine transformation, image: ', transformed_imgs[0].shape)\n", "#print(train_imgs[0])\n", "print('Original image')\n", "show_image(train_imgs[3])\n", "print('Transformed image')\n", "show_image(transformed_imgs[3])\n", "show_image(transformed_imgs[3])"]}, {"metadata": {"_uuid": "1fdf935cda34e608766b2e709e76348efb3a7257", "_cell_guid": "246162ae-b366-43df-8a61-f2838d4eecc4"}, "cell_type": "markdown", "source": ["Does transformed image look like the same number with the original image (in this case 4)?\n", "\n", "Yes, even when the image is slightly rotated, shifted (transformed) or scaled, the image looks like the same label. \n", "We can virtually create another image data from one image in such a way."]}, {"metadata": {"_uuid": "bc2b265d9269a07fd6727c2b94274f83951bfd2f", "_cell_guid": "e84e0a13-a6fb-4a54-9642-5cd554e43be6"}, "cell_type": "markdown", "source": ["### Define dataset class\n", "\n", "Chainer provides a way to define your own dataset which can be customized.\n", "\n", "We can define the class by extending `DatasetMixin` class.\n", "Here, we can implement task-specific data augmentation code."]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "5c94a9d251d48a79c4f1ecde91e35703138be7ac", "_cell_guid": "b2569d79-81dd-4f87-bc6d-b0ebe8da6cf6", "collapsed": true}, "cell_type": "code", "source": ["class MNISTTrainImageDataset(chainer.dataset.DatasetMixin):\n", "    def __init__(self, imgs, labels, train=True, augmentation_rate=1.0,\n", "                 min_scale=0.90, max_scale=1.10, max_rot_angle=4,\n", "                 max_shear_angle=2, max_translation=3):\n", "        self.imgs = imgs.reshape((-1, 1, 28, 28))\n", "        self.labels = labels\n", "        self.train = train\n", "\n", "        # affine parameters\n", "        self.augmentation_rate = augmentation_rate\n", "        self.min_scale = min_scale  # 0.85\n", "        self.max_scale = max_scale  # 1.15\n", "        self.max_rot_angle = max_rot_angle  # 5\n", "        self.max_shear_angle = max_shear_angle  # 5\n", "        self.max_translation = max_translation\n", "\n", "    def __len__(self):\n", "        \"\"\"return length of this dataset\"\"\"\n", "        return len(self.labels)\n", "\n", "    def affine_image(self, img):\n", "        # ch, h, w = img.shape\n", "\n", "        # --- scale ---\n", "        sx = np.random.uniform(self.min_scale, self.max_scale)\n", "        sy = np.random.uniform(self.min_scale, self.max_scale)\n", "\n", "        # --- rotation ---\n", "        rot_angle = np.random.uniform(-self.max_rot_angle,\n", "                                      self.max_rot_angle) * np.pi / 180.\n", "\n", "        # --- shear ---\n", "        shear_angle = np.random.uniform(-self.max_shear_angle,\n", "                                        self.max_shear_angle) * np.pi / 180.\n", "\n", "        # --- translation ---\n", "        tx = np.random.randint(-self.max_translation, self.max_translation)\n", "        ty = np.random.randint(-self.max_translation, self.max_translation)\n", "\n", "        tform = AffineTransform(scale=(sx, sy), rotation=rot_angle,\n", "                                shear=shear_angle,\n", "                                translation=(tx, ty))\n", "        transformed_image = warp(img[0, :, :], tform.inverse,\n", "                                 output_shape=(28, 28))\n", "        return transformed_image.astype('float32').reshape(1, 28, 28)\n", "\n", "    def get_example(self, i):\n", "        \"\"\"Return i-th data\"\"\"\n", "        img = self.imgs[i]\n", "        label = self.labels[i]\n", "\n", "        # Data augmentation...\n", "        if self.train:\n", "            if np.random.uniform() < self.augmentation_rate:\n", "                img = self.affine_image(img)\n", "\n", "        return img, label"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "983ca231277c22e3318d3142e159a29883779746", "_cell_guid": "b38b282f-3040-499d-9294-093366236cc5"}, "cell_type": "code", "source": ["# 3. Load the dataset\n", "train_data = MNISTTrainImageDataset(train_imgs, train_y)"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "3b31c6dd7940e9690a9bb54732cd3dbdb545eae3", "_cell_guid": "e48d6185-cb82-4857-93a9-6f08361040c3"}, "cell_type": "code", "source": ["# train_data[i] is `i`-th dataset, with format (img, label)\n", "\n", "# extract 3rd dataset\n", "index = 3\n", "img, label = train_data[index]\n", "\n", "show_image(train_data[index][0])\n", "show_image(train_data[index][0])\n", "show_image(train_data[index][0])"]}, {"metadata": {"_uuid": "90cd7f335714fcad24eb2bc1e440c7366273ca81", "_cell_guid": "4a3f3396-d60b-4de0-85fb-7dc96395c113"}, "cell_type": "markdown", "source": ["What happed? I have used the same code, `show_image(train_data[index][0])` but the resulting image is different!!\n", "\n", "When the customized dataset `MNISTTrainImageDataset` is defined,\n", "`get_example(self, i)` is always called when we access the data by index (`train_data[index]`).\n", "\n", "The result was different each time since we have randomly determine the scale, rotation, shear, translation parameter each time.\n", "\n", "We can use `train_data` dataset for chainer's CNN model training, which automatically applies data augmentation for you:)."]}, {"metadata": {"_uuid": "4aa8bc3e8ba3bc03a6bf0641ad8557c1740cdb32", "_cell_guid": "4d6f2d5b-c13f-4820-b419-5052b8ab32bd"}, "cell_type": "markdown", "source": []}, {"metadata": {"_uuid": "2255ae59c9852a9534a2009d421dc30330095fec", "_cell_guid": "ee2e4778-f3d4-47ad-b213-e430cf91ccb5"}, "cell_type": "markdown", "source": ["## Training code -- step 2.\n", "\n", "Step 1, we tried a simple training script.\n", "Let's train the model with \n", " - data augmentation\n", " - cross validation\n"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "a0df209da54053ffcb5296df4cfddfaf7121be28", "_cell_guid": "b23c8984-15ee-459b-9a0a-b751e51f4163", "collapsed": true}, "cell_type": "code", "source": ["# -1 indicates to use CPU, \n", "# positive value indicates GPU device id.\n", "device = -1  # If you use CPU.\n", "#device = 0  # If you use GPU. (You need to install chainer & cupy with CUDA/cudnn installed)\n", "\n", "batchsize = 16\n", "class_num = 10\n", "out_dir = '.'\n", "if DEBUG:\n", "    epoch = 5  # This value is small. Change to more than 20 for Actual running.\n", "else:\n", "    epoch = 30\n", "\n", "\n", "def train_main2(train_x, train_y, val_x, val_y, model_path='cnn_model.npz', model_class=CNNMedium):\n", "    # 1. Setup model\n", "    model = model_class(n_out=class_num)\n", "    classifier_model = L.Classifier(model)\n", "    if device >= 0:\n", "        chainer.cuda.get_device(device).use()  # Make a specified GPU current\n", "        classifier_model.to_gpu()  # Copy the model to the GPU\n", "\n", "    # 2. Setup an optimizer\n", "    optimizer = optimizers.Adam()\n", "    # optimizer = optimizers.MomentumSGD(lr=0.01)\n", "    optimizer.setup(classifier_model)\n", "\n", "    # 3. Load the dataset\n", "    # --- Use custom dataset to train model with data augmentation ---\n", "    train_dataset = MNISTTrainImageDataset(train_x, train_y, augmentation_rate=0.5,\n", "                                           min_scale=0.95, max_scale=1.05, max_rot_angle=7,\n", "                                           max_shear_angle=3, max_translation=2)\n", "    val_dataset = MNISTTrainImageDataset(val_x, val_y, train=False)\n", "    # --- end of modification ---\n", "\n", "    # 4. Setup an Iterator\n", "    train_iter = iterators.SerialIterator(train_dataset, batchsize)\n", "    #train_iter = iterators.MultiprocessIterator(train, args.batchsize, n_prefetch=10)\n", "    val_iter = iterators.SerialIterator(val_dataset, batchsize, repeat=False, shuffle=False)\n", "    \n", "    # 5. Setup an Updater\n", "    updater = training.StandardUpdater(train_iter, optimizer, \n", "                                       device=device)\n", "    # 6. Setup a trainer (and extensions)\n", "    trainer = training.Trainer(updater, (epoch, 'epoch'), out=out_dir)\n", "\n", "    # Evaluate the model with the test dataset for each epoch\n", "    trainer.extend(extensions.Evaluator(val_iter, classifier_model, device=device), trigger=(1, 'epoch'))\n", "\n", "    # --- Learning rate decay scheduling ---\n", "    def decay_lr(trainer):\n", "        print('decay_lr at epoch {}'.format(trainer.updater.epoch_detail))\n", "        # optimizer.lr *= 0.1  # for MomentumSGD optimizer\n", "        optimizer.alpha *= 0.1\n", "        print('optimizer lr has changed to {}'.format(optimizer.lr))\n", "    trainer.extend(decay_lr, \n", "                   trigger=chainer.training.triggers.ManualScheduleTrigger([10, 20], 'epoch'))\n", "    # --- end of modification ---\n", "\n", "    trainer.extend(extensions.dump_graph('main/loss'))\n", "    trainer.extend(extensions.snapshot(), trigger=(1, 'epoch'))\n", "    trainer.extend(extensions.LogReport())\n", "    trainer.extend(extensions.PlotReport(\n", "        ['main/loss', 'validation/main/loss'],\n", "        x_key='epoch', file_name='loss.png'))\n", "    trainer.extend(extensions.PlotReport(\n", "        ['main/accuracy', 'validation/main/accuracy'],\n", "        x_key='epoch',\n", "        file_name='accuracy.png'))\n", "\n", "    try:\n", "        # Use extension library, chaineripy's PrintReport & ProgressBar\n", "        from chaineripy.extensions import PrintReport, ProgressBar\n", "        trainer.extend(ProgressBar(update_interval=5))\n", "        trainer.extend(PrintReport(\n", "            ['epoch', 'main/loss', 'validation/main/loss',\n", "             'main/accuracy', 'validation/main/accuracy', 'elapsed_time']))\n", "\n", "    except:\n", "        print('chaineripy is not installed, run `pip install chaineripy` to show rich UI progressbar')\n", "        # Use chainer's original ProgressBar & PrintReport\n", "        # trainer.extend(extensions.ProgressBar(update_interval=5))\n", "        trainer.extend(extensions.PrintReport(\n", "            ['epoch', 'main/loss', 'validation/main/loss',\n", "             'main/accuracy', 'validation/main/accuracy', 'elapsed_time']))\n", "\n", "    # Resume from a snapshot\n", "    # serializers.load_npz(args.resume, trainer)\n", "\n", "    # Run the training\n", "    trainer.run()\n", "    # Save the model\n", "    serializers.save_npz('{}/{}'.format(out_dir, model_path), model)\n", "                         \n", "    return model"]}, {"metadata": {"_uuid": "d71770234f2dfc3fea00a3b508c93c05f3abf025", "_cell_guid": "84b0a500-caaf-4e8b-94ad-08e6c63698f1"}, "cell_type": "markdown", "source": ["### Train the model with cross validation\n", "\n", "Sklearn provides simple API to execute cross validation. The code \n", "\n", "`for train_idx, valid_idx in StratifiedKFold(n_splits=N_SPLIT_CV).split(train_imgs, train_y):`\n", "\n", "is used to apply cross validation. \n", "`StratifiedKFold` automatically keeps the balance of each label in train and validation split.\n", "\n", "Let's run below code to train the model with cross validation.\n", "\n", "Note that each training takes longer time to converge to get the minimum train loss value.\n", "\n", "This is because the input data is augmented and the model will get different image in each epoch."]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "e9600ac684fb97153187008d0efb2c2280fa3231", "_cell_guid": "0934c493-c71c-4566-b054-4a8efcb5144f"}, "cell_type": "code", "source": ["from sklearn.model_selection import KFold, StratifiedKFold\n", "\n", "if DEBUG:\n", "    N_SPLIT_CV = 2\n", "else:\n", "    N_SPLIT_CV = 5\n", "\n", "cv_step = 0\n", "# for train_idx, valid_idx in StratifiedKFold(n_splits=N_SPLIT_CV, shuffle=True, random_state=7).split(train_imgs, train_y):\n", "for train_idx, valid_idx in StratifiedKFold(n_splits=N_SPLIT_CV).split(train_imgs, train_y):\n", "    print('Training cv={} ...'.format(cv_step))\n", "    train_main2(train_imgs[train_idx], train_y[train_idx], \n", "                train_imgs[val_idx], train_y[val_idx], \n", "                model_path='cnn_model_cv{}.npz'.format(cv_step))\n", "    cv_step += 1"]}, {"metadata": {"_uuid": "ae3c4e939b2cfc974cf09bf9f4e6dfd4139c88a3", "_cell_guid": "815224b4-6f3f-4876-b356-10080c1201d6"}, "cell_type": "markdown", "source": ["## Predict with model emsemble\n", "\n"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "e5d931c40838a9050762ac550e0f4e7d7214d24e", "_cell_guid": "dd5ca199-d962-4632-91dd-7e1fad0046cb", "collapsed": true}, "cell_type": "code", "source": ["class_num = 10\n", "# device= -1\n", "\n", "def predict_proba_main(model_path='cnn_model.npz'):\n", "    # 1. Setup model\n", "    model = CNNMedium(n_out=class_num)\n", "    classifier_model = L.Classifier(model)\n", "    if device >= 0:\n", "        chainer.cuda.get_device(device).use()  # Make a specified GPU current\n", "        classifier_model.to_gpu()  # Copy the model to the GPU\n", "\n", "    # load trained model\n", "    serializers.load_npz(model_path, model)\n", "\n", "    # 2. Load the dataset\n", "    # test_imgs is already prepared\n", "\n", "    # 3. predict the result\n", "    proba = model.predict_proba(test_imgs, device=device)\n", "    return proba\n", "\n", "\n", "def create_submission(submission_path, t):\n", "    result_dict = {\n", "        'ImageId': np.arange(1, len(t) + 1),\n", "        'Label': t\n", "    }\n", "    df = pd.DataFrame(result_dict)\n", "    df.to_csv(submission_path,\n", "              index_label=False, index=False)\n", "    print('submission file saved to {}'.format(submission_path))"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "73cebfd57ecb80839bd88830a8119c19c3d8a770", "_cell_guid": "b3fa8005-f9b6-49fc-975c-082d979229e4"}, "cell_type": "code", "source": ["proba_list = []\n", "for i in range(N_SPLIT_CV):\n", "    print('predicting {}-th model...'.format(i))\n", "    proba = predict_proba_main(model_path='./cnn_model_cv{}.npz'.format(i))\n", "    proba_list.append(proba)\n", "  \n", "proba_array = np.array(proba_list)"]}, {"metadata": {"_uuid": "470fbe26a18f7f5374dad81200ec8cde072c9974", "_cell_guid": "3bf27d8b-7327-45d6-9f5a-2620144dd806"}, "cell_type": "markdown", "source": ["Take each model's probability mean as final ensemble prediction"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "9cc06ec86077b9c878044e1cbf4c71c4cd90d3d7", "_cell_guid": "28fb9ce3-4307-4f5b-9eab-e92dccb67f5c"}, "cell_type": "code", "source": ["proba_ensemble = np.mean(proba_array, axis=0)  # Take each model's mean as ensembled prediction.\n", "predict_ensemble = np.argmax(proba_ensemble, axis=1)\n", "\n", "# --- Check shape ---\n", "# 0th axis represents each model, 1st axis represents test data index, 2nd axis represents the probability of each label\n", "print('proba_array', proba_array.shape)\n", "# 0th axis represents test data index, 1st axis represents the probability of each label\n", "print('proba_ensemble', proba_ensemble.shape)\n", "# 0th axis represents final label prediction for test data index\n", "print('predict_ensemble', predict_ensemble.shape, predict_ensemble)"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "c3288d982a09804b8414e3a003be8c0f115e52aa", "_cell_guid": "c86ac646-d093-4056-a4e5-aa3b268c3de6"}, "cell_type": "code", "source": ["create_submission('submission_ensemble.csv', predict_ensemble)"]}, {"metadata": {"_uuid": "8897924637fc898abb48be241e7658ff72274c1c", "_cell_guid": "60d1a35d-7045-4484-8646-036e78682503", "collapsed": true}, "cell_type": "markdown", "source": ["### Submit file: score 0.99700\u00b6\n", "\n", "That is the end of intermidiate course of Convolutional Neural Network training and prediction by traind CNN.\n", "\n", "I have introduced major technics used in deep learning field to increase the accuracy such as, \n", " - data augmentation\n", " - cross validation\n", " - model ensemble\n", " - learning rate scheduling for CNN training\n", "\n", "After these effort, the above submission file scored 0.99700 in my environment :)\n"]}, {"metadata": {"_uuid": "84f4eef1bb8ebc00b20fe5fb6dee779279b4bf7d", "_cell_guid": "d4357fd9-d5d6-4846-977f-25906e9bfaaa", "collapsed": true}, "cell_type": "markdown", "source": ["## Appendix\n", "\n", "### Visualize data with high loss error\n", "\n", "Some may ask can we achieve 100 % accuracy for this digit-recognizer task?\n", "\n", "To understand how difficult is to increase the accuracy, let's see the \"most difficult data\" \n", "for the trained model to predict.\n", "\n", "The codes below calculates the softmax cross entropy loss for each image data, and shows most highest loss images."]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "0cd45310b76369cb2c9c4c84f4c2964db47e23a4", "_cell_guid": "2bbd0414-9627-42c3-aaf5-aced2265e513", "collapsed": true}, "cell_type": "code", "source": ["from chainer import cuda\n", "\n", "def calc_loss_and_prob(model_path='cnn_model.npz'):\n", "    # 1. Setup model\n", "    model = CNNMedium(n_out=class_num)\n", "    classifier_model = L.Classifier(model)\n", "    if device >= 0:\n", "        chainer.cuda.get_device(device).use()  # Make a specified GPU current\n", "        classifier_model.to_gpu()  # Copy the model to the GPU\n", "\n", "    # load trained model\n", "    serializers.load_npz(model_path, model)\n", "\n", "    # 2. Load the dataset\n", "    # test_imgs is already prepared\n", "\n", "    # 3. predict the result\n", "    if device >= 0:\n", "        h = model(cuda.to_gpu(train_imgs))\n", "        loss = F.softmax_cross_entropy(h, cuda.to_gpu(train_y), reduce='no')\n", "        return cuda.to_cpu(loss.data), cuda.to_cpu(h.data)\n", "    else:\n", "        h = model(train_imgs)\n", "        loss = F.softmax_cross_entropy(h, train_y, reduce='no')\n", "        return loss.data, h.data"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "73cf545e5756b713f1f228e2ab3583c5705b372b", "_cell_guid": "8deb9b0b-be8a-440a-b17e-f7aa4b4d5a9b"}, "cell_type": "code", "source": ["loss_list = []\n", "prob_list = []\n", "for i in range(N_SPLIT_CV):\n", "    print('calc loss for {}-th model...'.format(i))\n", "    loss, prob = calc_loss_and_prob(model_path='./cnn_model_cv{}.npz'.format(i))\n", "    loss_list.append(loss)\n", "    prob_list.append(prob)\n", "\n", "loss_array = np.array(loss_list)\n", "prob_array = np.array(prob_list)"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "64867ef64d5b4e306db79de03fe68d7a258f0e20", "_cell_guid": "6384c11b-2417-4bb1-9903-fcc6577f3e2f"}, "cell_type": "code", "source": ["loss_ensemble = np.mean(loss_array, axis=0)  # Take each model's mean loss.\n", "predict_ensemble = np.mean(prob_array, axis=0)  # Take each model's mean as ensembled prediction.\n", "\n", "# --- Check shape ---\n", "# 0th axis represents each model, 1st axis represents test data index, 2nd axis represents the probability of each label\n", "print('loss_array', loss_array.shape)\n", "# 0th axis represents test data index, 1st axis represents the probability of each label\n", "print('loss_ensemble', loss_ensemble.shape)"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "687a48abfd385b68fa29e7605f7502ee7314820e", "_cell_guid": "1dc24fa2-8874-44b0-b64e-bac08241b1b6"}, "cell_type": "code", "source": ["loss_index = np.argsort(loss_ensemble)\n", "print('BEST100:  ', loss_index[:30], loss_ensemble[loss_index[:30]])\n", "print('WORST100: ', loss_index[::-1][:100], loss_ensemble[loss_index[::-1][:100]])"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "5c695523a8d5863263b973fcabc61599c3e7cb11", "_cell_guid": "050a9d02-070d-4a17-bffa-0ef84e221397"}, "cell_type": "code", "source": []}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "e4e95f05180a32c85298807b77088f889dbe39a2", "_cell_guid": "c3feb1e5-0acd-47cc-94d7-00730400ef65", "collapsed": true}, "cell_type": "code", "source": ["def show_images(imgs):\n", "    num_imgs = len(imgs)\n", "    fig, axs = plt.subplots(nrows=1, ncols=num_imgs)\n", "    # plt.figure(figsize=(1.5, 1.5))\n", "    plt.axis('off')\n", "    #if img.ndim == 3:\n", "    #    img = img[0, :, :]\n", "    print('imgs shape', imgs.shape)\n", "    for i in range(num_imgs):\n", "        axs[i].imshow(imgs[i, 0, :, :], cmap=plt.cm.binary)           \n", "        axs[i].axis('off')\n", "    plt.show()"]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "6f11547bbbd1825b7002e0ad15eac488346c1f12", "_cell_guid": "55a39ce4-ba50-419f-aa31-64b3c8c182f8"}, "cell_type": "code", "source": ["worst_index = loss_index[::-1][:10]\n", "print('label ', train_y[worst_index])\n", "print('predict ', np.argmax(predict_ensemble, axis=1)[worst_index])\n", "show_images(train_imgs[worst_index])\n", "\n", "#for i in loss_index[::-1][:10]:\n", "#    print('label ', train_y[i])\n", "#    show_images(train_imgs[])"]}, {"metadata": {"_uuid": "7e7a5d78d038effea0ec52c4f1978763d33123cf", "_cell_guid": "87fcd757-34b0-494c-8b85-1c4c2a5e9e5a"}, "cell_type": "markdown", "source": ["If you run with DEBUG=False, you can see more interesting result that the images with \n", "high loss has a label which is very confusing.\n", "\n", "// --- Below comments was wrote when I executed the code with DEBUG=False. ---\n", "\n", "What do you think of these images & labels (answer)?\n", "\n", "The first image really looks like 7 (and CNN model predict it as 7), but the label is 4.\n", "This is most probably the case that \"labelling is wrong\", and the trained CNN model answer is quite reasonable.\n", "\n", "These confusing data is included in the dataset and that is why it is difficult or impossible to achieve 100 % accuracy."]}, {"outputs": [], "execution_count": null, "metadata": {"_uuid": "1996e7fdab199fc3cc849bc565381574d11453bd", "_cell_guid": "9ff47550-6c1b-4281-8caa-a3052c22b830", "collapsed": true}, "cell_type": "code", "source": []}], "nbformat": 4}