{"cells":[{"metadata":{"_uuid":"75fc05ba6debeab20154186debbba9645c8011aa"},"cell_type":"markdown","source":"# MNIST Data Augmentation with Elastic Distortion\n\nIn this kernel I'll outline an elastic distortion method for image data augmentation. \nThen I'll visualize elastic distortion and select Keras data augmentations on the MNIST data set.\nThe data augmentations described in \"Good Settings\" helped me get 99.7% test accuracy in the [Digit Recognizer](https://www.kaggle.com/c/digit-recognizer) competition.\n\n- [Elastic Distortion](#Elastic-Distortion)\n    - [Elastic Distortion Function](#Elastic-Distortion-Function)\n    - [Use Elastic Distortion with Keras](#Use-Elastic-Distortion-with-Keras)\n- [MNIST Visualizations](#MNIST-Visualizations)\n    - [Elastic Distortion Arguments Comparison](#Elastic-Distortion-Arguments-Comparison)\n    - [Interpolation Blur](#Interpolation-Blur)\n    - [Good Settings](#Good-Settings)"},{"metadata":{"_uuid":"1afebf9903a5218ccaab3b24f14e589fece10326"},"cell_type":"markdown","source":"## Elastic Distortion\n\nElastic distortion is another method of data augmentation, as opposed to affine distortion which is the method Keras uses. Elastic distortion does a good job of mimicking variations in human hand writing. A method for applying elastic distortion to the MNIST data set is described by Simard, Steinkraus, and Plattin (2003) in [\"Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis\"](https://www.microsoft.com/en-us/research/wp-content/uploads/2003/08/icdar03.pdf).\n\nThe method outline:\n1. Create random displacement fields for height and width, with values randomly sampled from $\\mathrm{unif}(-1,1)$. A displacement field defines a direction and magnitude to move a pixel. \n2. Smooth the fields with a gaussian filter. Since $\\mu = 0$ for $\\mathrm{unif}(-1,1)$, most values will be close to $0$ after the gaussian filter is applied. Thus most of the changes made by the fields will be small (assuming the gaussian filter's sigma value is large enough).\n3. Multiply the fields by a scaling factor to control intensity of the deformations.\n4. Use interpolation to apply the displacement fields to the image.\n\n#### Elastic Distortion Function\n\nCredit to the following gists for the basic function:\n- https://gist.github.com/fmder/e28813c1e8721830ff9c\n- https://gist.github.com/chsasank/4d8f68caf01f041a6453e67fb30f8f5a\n- https://gist.github.com/erniejunior/601cdf56d2b424757de5"},{"metadata":{"trusted":true,"_uuid":"2fc4a0ce3d63681b9ba70a615e8797fff8cd40bd"},"cell_type":"code","source":"import numpy as np\n\nfrom scipy.ndimage.filters import gaussian_filter\nfrom scipy.ndimage.interpolation import map_coordinates\n\ndef elastic_transform(image, alpha_range, sigma, random_state=None):\n    \"\"\"Elastic deformation of images as described in [Simard2003]_.\n    .. [Simard2003] Simard, Steinkraus and Platt, \"Best Practices for\n       Convolutional Neural Networks applied to Visual Document Analysis\", in\n       Proc. of the International Conference on Document Analysis and\n       Recognition, 2003.\n       \n   # Arguments\n       image: Numpy array with shape (height, width, channels). \n       alpha_range: Float for fixed value or [lower, upper] for random value from uniform distribution.\n           Controls intensity of deformation.\n       sigma: Float, sigma of gaussian filter that smooths the displacement fields.\n       random_state: `numpy.random.RandomState` object for generating displacement fields.\n    \"\"\"\n    \n    if random_state is None:\n        random_state = np.random.RandomState(None)\n        \n    if np.isscalar(alpha_range):\n        alpha = alpha_range\n    else:\n        alpha = np.random.uniform(low=alpha_range[0], high=alpha_range[1])\n\n    shape = image.shape\n    dx = gaussian_filter((random_state.rand(*shape) * 2 - 1), sigma) * alpha\n    dy = gaussian_filter((random_state.rand(*shape) * 2 - 1), sigma) * alpha\n\n    x, y, z = np.meshgrid(np.arange(shape[0]), np.arange(shape[1]), np.arange(shape[2]), indexing='ij')\n    indices = np.reshape(x+dx, (-1, 1)), np.reshape(y+dy, (-1, 1)), np.reshape(z, (-1, 1))\n\n    return map_coordinates(image, indices, order=1, mode='reflect').reshape(shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5258723b50f3b35aace3f45cddafdcda90e7674d"},"cell_type":"markdown","source":"#### Use Elastic Distortion with Keras\n\nNote that you can pass `elastic_transform()` to a Keras `ImageDataGenerator()` with\n\n    preprocessing_function=lambda x: elastic_transform(x, alpha_range=alpha_range, sigma=sigma)."},{"metadata":{"_uuid":"9c507195e23ca1d75511a0d4bd8e125ac0502388"},"cell_type":"markdown","source":"## MNIST Visualizations"},{"metadata":{"trusted":true,"_uuid":"14b9588574e0fa9d6896cff455a34a3bccedc1c3"},"cell_type":"code","source":"%matplotlib inline\n\nimport keras\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b7ed093a67f810d433e8943f1a6fde4ee96bdd8"},"cell_type":"markdown","source":"Load MNIST data set."},{"metadata":{"trusted":true,"_uuid":"9a74f2f5b21380888017f6f2268ca220987cb312"},"cell_type":"code","source":"x_train = np.loadtxt('../input/train.csv', dtype=int, delimiter=',', skiprows=1)\nx_train = np.reshape(x_train[:, 1:], (42000, 28, 28))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0c7fd0b82e1a1d2a3fc0314a6f1b6604079ed9eb"},"cell_type":"markdown","source":"Define plotting functions."},{"metadata":{"trusted":true,"_uuid":"9593bb1c54d4f1d831c52efc7a70e29d1f80a733"},"cell_type":"code","source":"def plot_digits(examples, title=None, size_mult=1):\n    \"\"\"Intended for graphing MNIST digits. \n    \n    # Arguments\n        examples: Numpy array with shape (num_examples, height, width, num_iterations).\n        title: Plot title string.\n        size_mult: Multiply figsize by `size_mult`.\n    \"\"\"\n   \n    num_iterations = examples.shape[-1]\n    num_examples = examples.shape[0]    \n    \n    plt.rcParams['figure.figsize'] = (num_examples * size_mult, num_iterations * size_mult)\n    plt.rcParams['image.interpolation'] = 'nearest'\n    plt.rcParams['image.cmap'] = 'gray'\n    \n    for c in range(num_iterations):\n        for i, ex in enumerate(examples):\n            plt.subplot(num_iterations, num_examples, num_examples * c + i + 1)            \n            plt.imshow(ex[:,:,c])  \n            plt.axis('off')\n            if c == 0 and i == 0 and title is not None:\n                # only way I found to keep title placement \n                # semi-consistent for different channel counts\n                plt.text(\n                    x=0,\n                    y=-ex.shape[1] // 4 // size_mult,\n                    s=title,\n                    fontsize=13,\n                    horizontalalignment='left', \n                    verticalalignment='bottom')\n\n    plt.show()\n    \n    \ndef plot_augmented(examples, alpha_range=0, sigma=0, \n                   width_shift_range=0, height_shift_range=0, zoom_range=0.0, \n                   iterations=1, title=None, size_mult=1):\n    \"\"\"Plot output after elastic distortion and select Keras data augmentations.\n    \n    # Arguments\n        examples: Numpy array with shape (num_examples, height, width, num_iterations).\n        alpha_range, sigma: arguments for `elastic_transform()`.\n        width_shift_range, height_shift_range, zoom_range: arguments for Keras `ImageDataGenerator()`.\n        iterations: Int, number of times to randomly augment the examples.\n        title: Plot title string.\n        size_mult: Multiply figsize by `size_mult`.\n    \"\"\"\n    \n    datagen = keras.preprocessing.image.ImageDataGenerator(\n        width_shift_range=width_shift_range, \n        height_shift_range=height_shift_range, \n        zoom_range=zoom_range,  \n        preprocessing_function=lambda x: elastic_transform(x, alpha_range=alpha_range, sigma=sigma)\n    )\n    x = [datagen.flow(examples, shuffle=False).next() for i in range(iterations)]\n    x = np.concatenate(x, axis=-1)\n    plot_digits(x, title=title, size_mult=size_mult)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"49d90ae01f433da9f8132233f36d217c37e427b5"},"cell_type":"markdown","source":"### Elastic Distortion Arguments Comparison\n\nRecall that `alpha` controls deformation intensity and `sigma` controls displacement field smoothing. \n\n- `alpha=8, sigma=2` does not have enough smoothing and you get squiggly lines.\n\n- `alpha=8, sigma=3` is close to human handwriting. \n\n- `alpha=10, sigma=3` some deformations are exaggerated to the point of not looking human made (you may have to run this cell multiple times to see an example of this happening)."},{"metadata":{"trusted":true,"_uuid":"4c7a4fc8d541cc6e3f5032ba2796815f48c01a22"},"cell_type":"code","source":"num_examples = 10\ned_examples = np.expand_dims(x_train[np.random.choice(x_train.shape[0], num_examples)], -1)\n\nplot_digits(ed_examples, title='Input Images')\n\n# elastic distortion\nplot_augmented(ed_examples, alpha_range=8, sigma=2, \n               iterations=3, title='Elastic Distortion | alpha=8, sigma=2')\nplot_augmented(ed_examples, alpha_range=8, sigma=3, \n               iterations=3, title='Elastic Distortion | alpha=8, sigma=3')\nplot_augmented(ed_examples, alpha_range=10, sigma=3,\n               iterations=3, title='Elastic Distortion | alpha=10, sigma=3')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fdbcf993b78d8a773b31ea858c94bdd046684c4c"},"cell_type":"markdown","source":"### Interpolation Blur\n\nWhen transforming an image, interpolation often needs to approximate pixel values, which has a blurring effect. However, a shift by an exact pixel value does not require any approximation. Keras allows you to do this by passing integer values for the shift arguments. From the Keras [documentation](https://keras.io/preprocessing/image/):\n> - int: integer number of pixels from interval `(-width_shift_range, +width_shift_range)`\n> - With `width_shift_range=2` possible values are integers `[-1, 0, +1]`, same as with `width_shift_range=[-1, 0, +1]`, while with `width_shift_range=1.0` possible values are floats in the interval `[-1.0, +1.0)`.\n\nIt's debatable whether or not blur should be avoided. Below are examples of blur caused by interpolation. Note that the affine transformations done by Keras only use interpolation once after the net result of all the transformations is calculated (for example when using `width_shift_range=1.0` and `zoom_range=0.1` together, the image is still only interpolated once). The `elastic_transform()` function also uses interpolation, which will stack on top of any interpolation Keras does, resulting in exaggerated blurring. So we get the following results:\n- Integer shift → no blur\n- Float shift → some blur\n- Elastic distortion and integer shift → some blur\n- Elastic distortion and float shift → more blur"},{"metadata":{"trusted":true,"_uuid":"5181c643385c727271c454c31ff7e357f0e08fc3"},"cell_type":"code","source":"num_examples = 6\nb_examples = np.expand_dims(x_train[np.random.choice(x_train.shape[0], num_examples)], -1)\n\nplot_digits(b_examples, title='Input Images', size_mult=2)\n\n# shift\nplot_augmented(b_examples, width_shift_range=2, height_shift_range=2,\n               title='Integer Shift', size_mult=2)\nplot_augmented(b_examples, width_shift_range=1., height_shift_range=1.,\n               title='Float Shift', size_mult=2)\n\n# elastic distortion & shift\nplot_augmented(b_examples, alpha_range=[8, 10], sigma=3, \n               width_shift_range=2, height_shift_range=2,\n               title='Elastic Distortion and Integer Shift', size_mult=2)\nplot_augmented(b_examples, alpha_range=[8, 10], sigma=3, \n               width_shift_range=1., height_shift_range=1.,\n               title='Elastic Distortion and Float Shift', size_mult=2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dd1cd595422c7842b041e8bc90924002fbbfcfaf"},"cell_type":"markdown","source":"### Good Settings\n\nThese are the data augmentation settings I have gotten good results with. Even though `alpha=10` can result in exaggerated writing, I can still tell what the number is supposed to be. Thus, I think exaggerated writing can be useful for training a CNN to recognize what features of the number are most important for identification. By using `alpha_range=[8, 10]`, the CNN will be trained on 'normal' looking numbers as well."},{"metadata":{"scrolled":false,"trusted":true,"_uuid":"a5b18659a6253fc734333d9ae6b6079e38b12698"},"cell_type":"code","source":"num_examples = 10\nexamples = np.expand_dims(x_train[np.random.choice(x_train.shape[0], num_examples)], -1)\n\nplot_digits(examples, title='Input Images')\n\nplot_augmented(examples, alpha_range=[8, 10], sigma=3, \n               width_shift_range=2, height_shift_range=2, zoom_range=0, \n               iterations=10, title='Elastic Distortion and Integer Shift | alpha_range=[8, 10], sigma=3, shift_range=2')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"toc":{"base_numbering":1,"nav_menu":{},"number_sections":false,"sideBar":true,"skip_h1_title":true,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":false,"toc_position":{"height":"calc(100% - 180px)","left":"10px","top":"150px","width":"336px"},"toc_section_display":true,"toc_window_display":false}},"nbformat":4,"nbformat_minor":1}