{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "816fec69-c9c7-10fb-f353-69ef8c840784" }, "source": [ "Greedy layer-wise pretraining in Keras\n", "==========\n", "\n", "Apparently this method has fallen out of favor, but it's a good Keras exercise. Also, the 20 minute time limit is pretty restrictive for the number of things that have to happen here, but it appears to be working.\n", "\n", "The code is set up for three layers of autoencoders, followed by a classification task, but the third layer is commented out due to time. Thus **this model uses two layers of pretrained autoencoders, followed by a dense layer attached to a softmax**. Dropout is used for regularization and Batch Normalization is used to speed up training. If this published kernel works the way it did in interactive mode, the final accuracy should be in the mid to high 90s.\n", "\n", "Even if pretraining autoencoders is no longer considered a good idea, one useful trick shown in this script is how to take layers with trained weights, and \"copy\" them into different models using **get_weights()** and **set_weights()**.\n", "\n", "For more on autoencoders in Keras, see The Keras Blog's [Building Autoencoders in Keras][1]\n", "\n", " [1]: https://blog.keras.io/building-autoencoders-in-keras.html" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ee9f752d-7bf3-fee5-7411-64db1922714c" }, "outputs": [], "source": [ "%env KERAS_BACKEND=theano\n", "%reset\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", " \n", "import keras\n", "import keras.backend as K\n", "from keras.layers import Input, Convolution2D, Activation, MaxPooling2D, \\\n", " Dense, BatchNormalization, Dropout\n", "from keras.layers.core import Flatten\n", "from keras.optimizers import SGD\n", "from keras.models import Model\n", "from keras.utils import np_utils\n", "from keras.constraints import maxnorm\n", "from keras.regularizers import l2\n", "from keras.callbacks import LearningRateScheduler\n", "from keras.layers.normalization import BatchNormalization\n", "\n", "print(keras.__version__)\n", "\n", "from subprocess import check_output\n", "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "c6db01ee-ee2b-84fe-d4b6-3ac1f41f4ddf" }, "source": [ "## Scaling\n", "\n", "Scaling inputs to be between 0 and 1. That makes the decoding model simple, because we can pretend like we're working with a binary output.\n", "\n", "\n", "From the Theano docs: [binary_crossentropy][1]\n", "\n", "```\n", "crossentropy(t,o) = -(t * log(o) + (1 - t) * log(1 - o)).\n", "```\n", "\n", " [1]: http://deeplearning.net/software/theano/library/tensor/nnet/nnet.html#theano.tensor.nnet.nnet.binary_crossentropy" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e71b38df-370a-15d0-1ae4-ae720a77e245" }, "outputs": [], "source": [ "N_train = 30000 # Out of 42000, to reduce processing time\n", "train = np.genfromtxt('../input/train.csv', delimiter = ',', skip_header = 1)\n", "training_inputs = train[0:N_train, 1:] / 255.0\n", "training_targets = np_utils.to_categorical(train[:, int(0)])[0:N_train]\n", "\n", "val_inputs = train[(N_train+1):42000, 1:] / 255.0\n", "val_targets = np_utils.to_categorical(train[:, int(0)])[(N_train+1):42000]\n", "\n", "#test = np.genfromtxt('../input/test.csv', delimiter = ',', skip_header = 1)\n", "#test_inputs = test[:, ] / 255.0" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d14475d6-c257-c5fe-44ea-b330ae58c43f" }, "outputs": [], "source": [ "# For 2D data (e.g. image), ordering type \"tf\" assumes (rows, cols, channels)\n", "# type \"th\" assumes (channels, rows, cols). See https://keras.io/backend/\n", "print('We are using image ordering type', K.image_dim_ordering())\n", "\n", "training_inputs = training_inputs.reshape(training_inputs.shape[0], 784)\n", "#test_inputs = test_inputs.reshape(test_inputs.shape[0], 784)\n", "print(training_inputs.shape)\n", "print(val_inputs.shape)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "f44a1d19-f107-ba95-1f08-6274dc950da2" }, "outputs": [], "source": [ "# Layer by layer pretraining Models\n", "\n", "# Layer 1\n", "input_img = Input(shape = (784, ))\n", "distorted_input1 = Dropout(.1)(input_img)\n", "encoded1 = Dense(800, activation = 'sigmoid')(distorted_input1)\n", "encoded1_bn = BatchNormalization()(encoded1)\n", "decoded1 = Dense(784, activation = 'sigmoid')(encoded1_bn)\n", "\n", "autoencoder1 = Model(input = input_img, output = decoded1)\n", "encoder1 = Model(input = input_img, output = encoded1_bn)\n", "\n", "# Layer 2\n", "encoded1_input = Input(shape = (800,))\n", "distorted_input2 = Dropout(.2)(encoded1_input)\n", "encoded2 = Dense(400, activation = 'sigmoid')(distorted_input2)\n", "encoded2_bn = BatchNormalization()(encoded2)\n", "decoded2 = Dense(800, activation = 'sigmoid')(encoded2_bn)\n", "\n", "autoencoder2 = Model(input = encoded1_input, output = decoded2)\n", "encoder2 = Model(input = encoded1_input, output = encoded2_bn)\n", "\n", "# Layer 3 - which we won't end up fitting in the interest of time\n", "encoded2_input = Input(shape = (400,))\n", "distorted_input3 = Dropout(.3)(encoded2_input)\n", "encoded3 = Dense(200, activation = 'sigmoid')(distorted_input3)\n", "encoded3_bn = BatchNormalization()(encoded3)\n", "decoded3 = Dense(400, activation = 'sigmoid')(encoded3_bn)\n", "\n", "autoencoder3 = Model(input = encoded2_input, output = decoded3)\n", "encoder3 = Model(input = encoded2_input, output = encoded3_bn)\n", "\n", "# Deep Autoencoder\n", "encoded1_da = Dense(800, activation = 'sigmoid')(input_img)\n", "encoded1_da_bn = BatchNormalization()(encoded1_da)\n", "encoded2_da = Dense(400, activation = 'sigmoid')(encoded1_da_bn)\n", "encoded2_da_bn = BatchNormalization()(encoded2_da)\n", "encoded3_da = Dense(200, activation = 'sigmoid')(encoded2_da_bn)\n", "encoded3_da_bn = BatchNormalization()(encoded3_da)\n", "decoded3_da = Dense(400, activation = 'sigmoid')(encoded3_da_bn)\n", "decoded2_da = Dense(800, activation = 'sigmoid')(decoded3_da)\n", "decoded1_da = Dense(784, activation = 'sigmoid')(decoded2_da)\n", "\n", "deep_autoencoder = Model(input = input_img, output = decoded1_da)\n", "\n", "# Not as Deep Autoencoder\n", "nad_encoded1_da = Dense(800, activation = 'sigmoid')(input_img)\n", "nad_encoded1_da_bn = BatchNormalization()(nad_encoded1_da)\n", "nad_encoded2_da = Dense(400, activation = 'sigmoid')(nad_encoded1_da_bn)\n", "nad_encoded2_da_bn = BatchNormalization()(nad_encoded2_da)\n", "nad_decoded2_da = Dense(800, activation = 'sigmoid')(nad_encoded2_da_bn)\n", "nad_decoded1_da = Dense(784, activation = 'sigmoid')(nad_decoded2_da)\n", "\n", "nad_deep_autoencoder = Model(input = input_img, output = nad_decoded1_da)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "444064c5-1e69-dfe1-a207-30cb6dda5eac" }, "outputs": [], "source": [ "sgd1 = SGD(lr = 5, decay = 0.5, momentum = .85, nesterov = True)\n", "sgd2 = SGD(lr = 5, decay = 0.5, momentum = .85, nesterov = True)\n", "sgd3 = SGD(lr = 5, decay = 0.5, momentum = .85, nesterov = True)\n", "\n", "autoencoder1.compile(loss='binary_crossentropy', optimizer = sgd1)\n", "autoencoder2.compile(loss='binary_crossentropy', optimizer = sgd2)\n", "autoencoder3.compile(loss='binary_crossentropy', optimizer = sgd3)\n", "\n", "encoder1.compile(loss='binary_crossentropy', optimizer = sgd1)\n", "encoder2.compile(loss='binary_crossentropy', optimizer = sgd1)\n", "encoder3.compile(loss='binary_crossentropy', optimizer = sgd1)\n", "\n", "deep_autoencoder.compile(loss='binary_crossentropy', optimizer = sgd1)\n", "nad_deep_autoencoder.compile(loss='binary_crossentropy', optimizer = sgd1)\n", "\n", "# What will happen to the learnning rates under this decay schedule?\n", "lr = 5\n", "for i in range(12):\n", " lr = lr - lr * .15\n", " print(lr) " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "eeb7f99b-61fe-ea31-8c50-f5d8b62a85a8" }, "outputs": [], "source": [ "autoencoder1.fit(training_inputs, training_inputs,\n", " nb_epoch = 8, batch_size = 512,\n", " validation_split = 0.30,\n", " shuffle = True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ceb6f3aa-0364-76b1-7313-655064eee389" }, "outputs": [], "source": [ "first_layer_code = encoder1.predict(training_inputs)\n", "print(first_layer_code.shape)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "cc7abb88-27bd-bdc3-429e-26e5b321597c" }, "outputs": [], "source": [ "autoencoder2.fit(first_layer_code, first_layer_code,\n", " nb_epoch = 8, batch_size = 512,\n", " validation_split = 0.25,\n", " shuffle = True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "638391d5-20df-731f-9bb3-fe49e792f41c" }, "outputs": [], "source": [ "#second_layer_code = encoder2.predict(first_layer_code)\n", "#print(second_layer_code.shape)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e387eec3-8a45-e899-c592-21d430796bec" }, "outputs": [], "source": [ "# Not enough time!!\n", "#autoencoder3.fit(second_layer_code, second_layer_code,\n", "# nb_epoch = 8, batch_size = 512,\n", "# validation_split = 0.30,\n", "# shuffle = True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "782d2ad0-6825-404b-7591-197b88b0e7d2" }, "outputs": [], "source": [ "# Setting the weights of the deep autoencoder\n", "#deep_autoencoder.layers[1].set_weights(autoencoder1.layers[2].get_weights()) # first dense layer\n", "#deep_autoencoder.layers[2].set_weights(autoencoder1.layers[3].get_weights()) # first bn layer\n", "#deep_autoencoder.layers[3].set_weights(autoencoder2.layers[2].get_weights()) # second dense layer\n", "#deep_autoencoder.layers[4].set_weights(autoencoder2.layers[3].get_weights()) # second bn layer\n", "#deep_autoencoder.layers[5].set_weights(autoencoder3.layers[2].get_weights()) # thrird dense layer\n", "#deep_autoencoder.layers[6].set_weights(autoencoder3.layers[3].get_weights()) # third bn layer\n", "#deep_autoencoder.layers[7].set_weights(autoencoder3.layers[4].get_weights()) # first decoder\n", "#deep_autoencoder.layers[8].set_weights(autoencoder2.layers[4].get_weights()) # second decoder\n", "#deep_autoencoder.layers[9].set_weights(autoencoder1.layers[4].get_weights()) # third decoder\n", "\n", "# Setting up the weights of the not-as-deep autoencoder\n", "nad_deep_autoencoder.layers[1].set_weights(autoencoder1.layers[2].get_weights()) # first dense layer\n", "nad_deep_autoencoder.layers[2].set_weights(autoencoder1.layers[3].get_weights()) # first bn layer\n", "nad_deep_autoencoder.layers[3].set_weights(autoencoder2.layers[2].get_weights()) # second dense layer\n", "nad_deep_autoencoder.layers[4].set_weights(autoencoder2.layers[3].get_weights()) # second bn layer\n", "nad_deep_autoencoder.layers[5].set_weights(autoencoder2.layers[4].get_weights()) # second decoder\n", "nad_deep_autoencoder.layers[6].set_weights(autoencoder1.layers[4].get_weights()) # third decoder" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "8d3e7ea0-d13e-955e-64cd-5e108bfe1356" }, "outputs": [], "source": [ "# you can see the degredation by uncommenting these one at a time and plotting\n", "#decoded_inputs = autoencoder1.predict(training_inputs[0:25, ])\n", "decoded_inputs = nad_deep_autoencoder.predict(training_inputs[0:25,])\n", "#decoded_inputs = deep_autoencoder.predict(training_inputs[0:25,])\n", "decoded_inputs.shape\n", "\n", "fig = plt.figure(figsize = (8, 8))\n", "fig.suptitle('Deep autoencoder reconstructions', fontsize=24, fontweight='bold')\n", "\n", "ax1 = fig.add_subplot(231)\n", "plt.imshow(training_inputs[2].reshape(28, 28))\n", "\n", "ax2 = fig.add_subplot(234)\n", "plt.imshow(decoded_inputs[2].reshape(28, 28))\n", "\n", "ax3 = fig.add_subplot(232)\n", "plt.imshow(training_inputs[6].reshape(28, 28))\n", "\n", "ax4 = fig.add_subplot(235)\n", "plt.imshow(decoded_inputs[6].reshape(28, 28))\n", "\n", "ax5 = fig.add_subplot(233)\n", "plt.imshow(training_inputs[4].reshape(28, 28))\n", "\n", "ax6 = fig.add_subplot(236)\n", "plt.imshow(decoded_inputs[4].reshape(28, 28))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "6423d112-d3db-3fea-6052-cb9c35747153" }, "source": [ "## On to \"fine tuning\" for classification\n", "Although it will have to be more than fine tuning" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e63968e5-3bc2-9750-135e-1864bf6adef5" }, "outputs": [], "source": [ "dense1 = Dense(500, activation = 'relu')(nad_decoded1_da)\n", "dense1_drop = Dropout(.3)(dense1)\n", "#dense1_bn = BatchNormalization()(dense1_drop)\n", "dense2 = Dense(10, activation = 'sigmoid')(dense1_drop)\n", "\n", "classifier = Model(input = input_img, output = dense2)\n", "sgd4 = SGD(lr = .1, decay = 0.001, momentum = .95, nesterov = True)\n", "classifier.compile(loss='categorical_crossentropy', optimizer = sgd4, metrics=['accuracy'])\n", " \n", "classifier.fit(training_inputs, training_targets,\n", " nb_epoch = 6, batch_size = 600,\n", " validation_split = 0.25,\n", " shuffle = True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "809bac5f-e50c-09e5-3a0d-96d7cbefdcde" }, "outputs": [], "source": [ "val_preds = classifier.predict(val_inputs)\n", "predictions = np.argmax(val_preds, axis = 1)\n", "true_digits = np.argmax(val_targets, axis = 1)\n", "predictions[0:25]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "38ef26e9-8c92-d396-ebbc-592e410e81f9" }, "outputs": [], "source": [ "n_correct = np.sum(np.equal(predictions, true_digits).astype(int))\n", "total = float(len(predictions))\n", "print(\"Validation Accuracy:\", round(n_correct / total, 3))" ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 0 }