{"cells":[{"metadata":{"_uuid":"8abac6e9d07e55338b420f9789904bc957f8fa0c","_cell_guid":"ae26a5da-aab9-47a2-b7dd-644778615a16"},"cell_type":"markdown","source":"**Author:** Raoul Malm  \n\n**Abstract:** \n\nWe implement a deep neural network consisting of convolutional and fully connected layers to classify  handwritten digits of the MNIST dataset. The labeled dataset consists of 42000 images of size 28x28 = 784 pixels (one gray-scale number) including the corresponding labels from 0,..,9. The test set consists of 28000 images. Each image is normalized such that each pixel takes on values in the range [0,1]. First, we try out basic models like logistic regression, random forest and so on. After that the images are fed into the neural network, which has the following architecture:\n\n- input layer: [.,784]\n- layer: Conv1 -> ReLu -> MaxPool: [.,14,14,36] \n- layer: Conv2 -> ReLu -> MaxPool: [.,7,7,36]\n- layer: Conv3 -> ReLu -> MaxPool: [.,4,4,36]\n- layer: FC -> ReLu: [.,576]\n- output layer: FC -> ReLu: [.,10]\n\nThis architecture is implemented with TensorFlow. In order to prevent the network from overfitting during learning we implement dropout and data augmentation, i.e. new images are generated from the original ones via rotation, translation and zooming. Finally, we predict the digit classes for the test set and write the submission file.     \n\n**Results:** \n\n- The best results are achieved by using 10-fold cross validation, by stacking the neural networks on top of each other and then by training a meta model. Since each neural network is trained for 15 epochs including data augmentation which takes roughly 30 minutes on kaggle hardware, it takes in total roughly 5 hours. The final accuracy is 99.51% on the public test set. Note that we have attached saver and summary tensors to the graph, which slows down the computation.  \n<br>\n\n- We can also train one neural network and implement a training/validation split of 95%/5% on the labeled original images. Training on 39900 original images and including data augmentation we can achieve after 15 epochs an accuracy of roughly 99.43% on the validation set of 2100 images. Of course this can vary depending on the specific training/validation splits. It also takes roughly 30 minutes on kaggle hardware. On the public test set it can achieve an accuracy of about 99.30%. Training on all data one can actually achieve the 99.43%.\n\n**Update:** \n\n- Stacking of models and training of a meta-model is now implemented.\n\n- The neural network is now implemented as a python class and the complete TensorFlow session can be saved to or restored from a file. We also implement tensor summaries, which can be visualized with TensorBoard.\n\n**Outline:**\n\n1. [Libraries and settings](#1-bullet)\n2. [Analyze data](#2-bullet)\n3. [Manipulate data](#3-bullet)\n4. [Try out some basic models with sklearn](#4-bullet)\n5. [Build the neural network with TensorFlow](#5-bullet)\n6. [Train and validate the neural network](#6-bullet)\n7. [Stacking of models and training a meta-model](#7-bullet)\n8. [Submit the test results](#8-bullet)\n\n\n**Reference:** \n\n[TensorFlow deep NN by Kirill Kliavin](https://www.kaggle.com/kakauandme/tensorflow-deep-nn?scriptVersionId=164725)\n\n\n# 1. Libraries and settings <a class=\"anchor\" id=\"1-bullet\"></a>\n- import relevant libraries\n- set number of features, neurons and filter size of the neural network "},{"metadata":{"_uuid":"420cf9cf5d64f79d38869ba265edfd759f1e4865","_cell_guid":"8fc7ff0a-f896-485a-9628-2f4c639942d0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport tensorflow as tf\nimport keras.preprocessing.image\nimport sklearn.preprocessing\nimport sklearn.model_selection\nimport sklearn.metrics\nimport sklearn.linear_model\nimport sklearn.naive_bayes\nimport sklearn.tree\nimport sklearn.ensemble\nimport os;\nimport datetime  \nimport cv2 \nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm  \n%matplotlib inline\n\n#display parent directory and working directory\nprint(os.path.dirname(os.getcwd())+':', os.listdir(os.path.dirname(os.getcwd())));\nprint(os.getcwd()+':', os.listdir(os.getcwd()));","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"15744e6161356ffdfaba3da0eb55c469f187a246","_cell_guid":"e1bd0032-cf12-45cd-840d-bbe8b36b8383"},"cell_type":"markdown","source":"# 2. Analyze data <a class=\"anchor\" id=\"2-bullet\"></a> \n- load images and have a first look\n- normalize images"},{"metadata":{"_uuid":"cd465ec1f9f87cfe2432ad669e81bf142aac1352","_cell_guid":"1c4dc51a-add1-410d-a586-58e934d0afd5","trusted":true},"cell_type":"code","source":"## load and check data\n\nif os.path.isfile('../input/train.csv'):\n    data_df = pd.read_csv('../input/train.csv') # on kaggle \n    print('train.csv loaded: data_df({0[0]},{0[1]})'.format(data_df.shape))\nelif os.path.isfile('data/train.csv'):\n    data_df = pd.read_csv('data/train.csv') # on local environment\n    print('train.csv loaded: data_df({0[0]},{0[1]})'.format(data_df.shape))\nelse:\n    print('Error: train.csv not found')\n\n# basic info about data\n#print('')\n#print(data_df.info())\n\n# no missing values\nprint('')\nprint(data_df.isnull().any().describe())\n\n# 10 different labels ranging from 0 to 9\nprint('')\nprint('distinct labels ', data_df['label'].unique())\n\n# data are approximately balanced (less often occurs 5, most often 1)\nprint('')\nprint(data_df['label'].value_counts())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4021b771e17a5f9787c9b71ad5c6733c92ec860c","_cell_guid":"64f637d5-34c4-49ea-a743-fbc1b2cd917b","trusted":true},"cell_type":"code","source":"## normalize data and split into training and validation sets\n\n# function to normalize data\ndef normalize_data(data): \n    # scale features using statistics that are robust to outliers\n    #rs = sklearn.preprocessing.RobustScaler()\n    #rs.fit(data)\n    #data = rs.transform(data)\n    #data = (data-data.mean())/(data.std()) # standardisation\n    data = data / data.max() # convert from [0:255] to [0.:1.]\n    #data = ((data / 255.)-0.5)*2. # convert from [0:255] to [-1.:+1.]\n    return data\n\n# convert class labels from scalars to one-hot vectors e.g. 1 => [0 1 0 0 0 0 0 0 0 0]\ndef dense_to_one_hot(labels_dense, num_classes):\n    num_labels = labels_dense.shape[0]\n    index_offset = np.arange(num_labels) * num_classes\n    labels_one_hot = np.zeros((num_labels, num_classes))\n    labels_one_hot.flat[index_offset + labels_dense.ravel()] = 1\n    return labels_one_hot\n\n# convert one-hot encodings into labels\ndef one_hot_to_dense(labels_one_hot):\n    return np.argmax(labels_one_hot,1)\n\n# computet the accuracy of label predictions\ndef accuracy_from_dense_labels(y_target, y_pred):\n    y_target = y_target.reshape(-1,)\n    y_pred = y_pred.reshape(-1,)\n    return np.mean(y_target == y_pred)\n\n# computet the accuracy of one-hot encoded predictions\ndef accuracy_from_one_hot_labels(y_target, y_pred):\n    y_target = one_hot_to_dense(y_target).reshape(-1,)\n    y_pred = one_hot_to_dense(y_pred).reshape(-1,)\n    return np.mean(y_target == y_pred)\n\n# extract and normalize images\nx_train_valid = data_df.iloc[:,1:].values.reshape(-1,28,28,1) # (42000,28,28,1) array\nx_train_valid = x_train_valid.astype(np.float) # convert from int64 to float32\nx_train_valid = normalize_data(x_train_valid)\nimage_width = image_height = 28\nimage_size = 784\n\n# extract image labels\ny_train_valid_labels = data_df.iloc[:,0].values # (42000,1) array\nlabels_count = np.unique(y_train_valid_labels).shape[0]; # number of different labels = 10\n\n#plot some images and labels\nplt.figure(figsize=(15,9))\nfor i in range(50):\n    plt.subplot(5,10,1+i)\n    plt.title(y_train_valid_labels[i])\n    plt.imshow(x_train_valid[i].reshape(28,28), cmap=cm.binary)\n    \n# labels in one hot representation\ny_train_valid = dense_to_one_hot(y_train_valid_labels, labels_count).astype(np.uint8)\n\n# dictionaries for saving results\ny_valid_pred = {}\ny_train_pred = {}\ny_test_pred = {}\ntrain_loss, valid_loss = {}, {}\ntrain_acc, valid_acc = {}, {}\n\nprint('x_train_valid.shape = ', x_train_valid.shape)\nprint('y_train_valid_labels.shape = ', y_train_valid_labels.shape)\nprint('image_size = ', image_size )\nprint('image_width = ', image_width)\nprint('image_height = ', image_height)\nprint('labels_count = ', labels_count)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"abcfe8ed446bf9707289a75946f5eb579b1bee80","_cell_guid":"7e7a795a-cfc9-470f-bfe7-5f869ccc5ddb"},"cell_type":"markdown","source":"# 3. Manipulate data <a class=\"anchor\" id=\"3-bullet\"></a> \n- generate new images via rotations, translations and zooming"},{"metadata":{"_uuid":"8422a258e2fb6b7f299fde94797e82b76631a0bd","_cell_guid":"5b8f694a-ee8d-41ba-af20-44939ba941b8","trusted":true},"cell_type":"code","source":"## augment data\n\n# generate new images via rotations, translations, zoom using keras\ndef generate_images(imgs):\n    \n    # rotations, translations, zoom\n    image_generator = keras.preprocessing.image.ImageDataGenerator(\n        rotation_range = 10, width_shift_range = 0.1 , height_shift_range = 0.1,\n        zoom_range = 0.1)\n\n    # get transformed images\n    imgs = image_generator.flow(imgs.copy(), np.zeros(len(imgs)),\n                                batch_size=len(imgs), shuffle = False).next()    \n  \n    return imgs[0]\n\n# check image generation\nfig,axs = plt.subplots(5,10, figsize=(15,9))\nfor i in range(5):\n    n = np.random.randint(0,x_train_valid.shape[0]-2)\n    axs[i,0].imshow(x_train_valid[n:n+1].reshape(28,28),cmap=cm.binary)\n    axs[i,1].imshow(generate_images(x_train_valid[n:n+1]).reshape(28,28), cmap=cm.binary)\n    axs[i,2].imshow(generate_images(x_train_valid[n:n+1]).reshape(28,28), cmap=cm.binary)\n    axs[i,3].imshow(generate_images(x_train_valid[n:n+1]).reshape(28,28), cmap=cm.binary)\n    axs[i,4].imshow(generate_images(x_train_valid[n:n+1]).reshape(28,28), cmap=cm.binary)\n    axs[i,5].imshow(generate_images(x_train_valid[n:n+1]).reshape(28,28), cmap=cm.binary)\n    axs[i,6].imshow(generate_images(x_train_valid[n:n+1]).reshape(28,28), cmap=cm.binary)\n    axs[i,7].imshow(generate_images(x_train_valid[n:n+1]).reshape(28,28), cmap=cm.binary)\n    axs[i,8].imshow(generate_images(x_train_valid[n:n+1]).reshape(28,28), cmap=cm.binary)\n    axs[i,9].imshow(generate_images(x_train_valid[n:n+1]).reshape(28,28), cmap=cm.binary)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3357085bb78d116e1527e6fa10121a2bca05fc94","_cell_guid":"3b7ef6b4-9eab-4b49-86dc-282413409a76"},"cell_type":"markdown","source":"# 4. Try out some basic models with sklearn <a class=\"anchor\" id=\"4-bullet\"></a> "},{"metadata":{"_uuid":"ba27309ed0d3ee72a68f9536070d60c820f346f9","_cell_guid":"6bd950e2-1288-4496-8e9b-3f8011aa5305","trusted":true},"cell_type":"code","source":"## First try out some basic sklearn models\n\nlogreg = sklearn.linear_model.LogisticRegression(verbose=0, solver='lbfgs',\n                                                 multi_class='multinomial')\ndecision_tree = sklearn.tree.DecisionTreeClassifier()\nextra_trees = sklearn.ensemble.ExtraTreesClassifier(verbose=0)\ngradient_boost = sklearn.ensemble.GradientBoostingClassifier(verbose=0)\nrandom_forest = sklearn.ensemble.RandomForestClassifier(verbose=0)\ngaussianNB = sklearn.naive_bayes.GaussianNB()\n\n# store models in dictionary\nbase_models = {'logreg': logreg, 'extra_trees': extra_trees,\n               'gradient_boost': gradient_boost, 'random_forest': random_forest, \n               'decision_tree': decision_tree, 'gaussianNB': gaussianNB}\n\n# choose models for out-of-folds predictions\ntake_models = ['logreg','random_forest','extra_trees']\n\nfor mn in take_models:\n    train_acc[mn] = []\n    valid_acc[mn] = []\n\n# cross validations\ncv_num = 10 # cross validations default = 20 => 5% validation set\nkfold = sklearn.model_selection.KFold(cv_num, shuffle=True, random_state=123)\n\nfor i,(train_index, valid_index) in enumerate(kfold.split(x_train_valid)):\n\n    # start timer\n    start = datetime.datetime.now();\n\n    # train and validation data of original images\n    x_train = x_train_valid[train_index].reshape(-1,784)\n    y_train = y_train_valid[train_index]\n    x_valid = x_train_valid[valid_index].reshape(-1,784)\n    y_valid = y_train_valid[valid_index]\n\n    for mn in take_models:\n\n        # create cloned model from base models\n        model = sklearn.base.clone(base_models[mn])\n        model.fit(x_train, one_hot_to_dense(y_train))\n\n        # predictions\n        y_train_pred[mn] = model.predict_proba(x_train)\n        y_valid_pred[mn] = model.predict_proba(x_valid)\n        train_acc[mn].append(accuracy_from_one_hot_labels(y_train_pred[mn], y_train))\n        valid_acc[mn].append(accuracy_from_one_hot_labels(y_valid_pred[mn], y_valid))\n\n        print(i,': '+mn+' train/valid accuracy = %.3f/%.3f'%(train_acc[mn][-1], \n                                                             valid_acc[mn][-1]))\n    # only one iteration\n    if False:\n        break;\n\nprint(mn+': averaged train/valid accuracy = %.3f/%.3f'%(np.mean(train_acc[mn]),\n                                                        np.mean(valid_acc[mn])))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3b5558fd025925b3b3c3fe8cc28428a04f4dd057","_cell_guid":"661bf3e7-4ef7-448d-8daf-b8d608030283","trusted":true},"cell_type":"code","source":"## compare accuracies of base models\n\n# boxplot algorithm comparison\nfig = plt.figure(figsize=(20,8))\nax = fig.add_subplot(1,2,1)\nplt.title('Train accuracy')\nplt.boxplot([train_acc[mn] for mn in train_acc.keys()])\nax.set_xticklabels([mn for mn in train_acc.keys()])\nax.set_ylabel('Accuracy');\nax.set_ylim([0.90,1.0])\n\nax = fig.add_subplot(1,2,2)\nplt.title('Valid accuracy')\nplt.boxplot([valid_acc[mn] for mn in train_acc.keys()])\nax.set_xticklabels([mn for mn in train_acc.keys()])\nax.set_ylabel('Accuracy');\nax.set_ylim([0.90,1.0])\n\nfor mn in train_acc.keys():\n    print(mn + ' averaged train/valid accuracy = %.3f/%.3f'%(np.mean(train_acc[mn]),\n                                                             np.mean(valid_acc[mn])))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"21a49993c6e32b06ca3cb76a76d4c688ac15a161","_cell_guid":"1eff9850-60ba-4f02-8de7-941865a8d91c"},"cell_type":"markdown","source":"# 5. Build the neural network with tensorflow <a class=\"anchor\" id=\"5-bullet\"></a> "},{"metadata":{"_uuid":"57d7213a23ff15c677946c5ee3c81b04579c87ec","_cell_guid":"d3bedd5b-c40d-46a7-8e7e-19af7f2a5773","trusted":true},"cell_type":"code","source":"## build the neural network class\n\nclass nn_class:\n# class that implements the neural network\n\n    # constructor\n    def __init__(self, nn_name = 'nn_1'):\n\n        # tunable hyperparameters for nn architecture\n        self.s_f_conv1 = 3; # filter size of first convolution layer (default = 3)\n        self.n_f_conv1 = 36; # number of features of first convolution layer (default = 36)\n        self.s_f_conv2 = 3; # filter size of second convolution layer (default = 3)\n        self.n_f_conv2 = 36; # number of features of second convolution layer (default = 36)\n        self.s_f_conv3 = 3; # filter size of third convolution layer (default = 3)\n        self.n_f_conv3 = 36; # number of features of third convolution layer (default = 36)\n        self.n_n_fc1 = 576; # number of neurons of first fully connected layer (default = 576)\n\n        # tunable hyperparameters for training\n        self.mb_size = 50 # mini batch size\n        self.keep_prob = 0.33 # keeping probability with dropout regularization \n        self.learn_rate_array = [10*1e-4, 7.5*1e-4, 5*1e-4, 2.5*1e-4, 1*1e-4, 1*1e-4,\n                                 1*1e-4,0.75*1e-4, 0.5*1e-4, 0.25*1e-4, 0.1*1e-4, \n                                 0.1*1e-4, 0.075*1e-4,0.050*1e-4, 0.025*1e-4, 0.01*1e-4, \n                                 0.0075*1e-4, 0.0050*1e-4,0.0025*1e-4,0.001*1e-4]\n        self.learn_rate_step_size = 3 # in terms of epochs\n        \n        # parameters\n        self.learn_rate = self.learn_rate_array[0]\n        self.learn_rate_pos = 0 # current position pointing to current learning rate\n        self.index_in_epoch = 0 \n        self.current_epoch = 0\n        self.log_step = 0.2 # log results in terms of epochs\n        self.n_log_step = 0 # counting current number of mini batches trained on\n        self.use_tb_summary = False # True = use tensorboard visualization\n        self.use_tf_saver = False # True = use saver to save the model\n        self.nn_name = nn_name # name of the neural network\n        \n        # permutation array\n        self.perm_array = np.array([])\n        \n    # function to get the next mini batch\n    def next_mini_batch(self):\n\n        start = self.index_in_epoch\n        self.index_in_epoch += self.mb_size\n        self.current_epoch += self.mb_size/len(self.x_train)  \n        \n        # adapt length of permutation array\n        if not len(self.perm_array) == len(self.x_train):\n            self.perm_array = np.arange(len(self.x_train))\n        \n        # shuffle once at the start of epoch\n        if start == 0:\n            np.random.shuffle(self.perm_array)\n\n        # at the end of the epoch\n        if self.index_in_epoch > self.x_train.shape[0]:\n            np.random.shuffle(self.perm_array) # shuffle data\n            start = 0 # start next epoch\n            self.index_in_epoch = self.mb_size # set index to mini batch size\n            \n            if self.train_on_augmented_data:\n                # use augmented data for the next epoch\n                self.x_train_aug = normalize_data(self.generate_images(self.x_train))\n                self.y_train_aug = self.y_train\n                \n        end = self.index_in_epoch\n        \n        if self.train_on_augmented_data:\n            # use augmented data\n            x_tr = self.x_train_aug[self.perm_array[start:end]]\n            y_tr = self.y_train_aug[self.perm_array[start:end]]\n        else:\n            # use original data\n            x_tr = self.x_train[self.perm_array[start:end]]\n            y_tr = self.y_train[self.perm_array[start:end]]\n        \n        return x_tr, y_tr\n               \n    # generate new images via rotations, translations, zoom using keras\n    def generate_images(self, imgs):\n    \n        print('generate new set of images')\n        \n        # rotations, translations, zoom\n        image_generator = keras.preprocessing.image.ImageDataGenerator(\n            rotation_range = 10, width_shift_range = 0.1 , height_shift_range = 0.1,\n            zoom_range = 0.1)\n\n        # get transformed images\n        imgs = image_generator.flow(imgs.copy(), np.zeros(len(imgs)),\n                                    batch_size=len(imgs), shuffle = False).next()    \n\n        return imgs[0]\n\n    # weight initialization\n    def weight_variable(self, shape, name = None):\n        initial = tf.truncated_normal(shape, stddev=0.1)\n        return tf.Variable(initial, name = name)\n\n    # bias initialization\n    def bias_variable(self, shape, name = None):\n        initial = tf.constant(0.1, shape=shape) #  positive bias\n        return tf.Variable(initial, name = name)\n\n    # 2D convolution\n    def conv2d(self, x, W, name = None):\n        return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME', name = name)\n\n    # max pooling\n    def max_pool_2x2(self, x, name = None):\n        return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1],\n                              padding='SAME', name = name)\n\n    # attach summaries to a tensor for TensorBoard visualization\n    def summary_variable(self, var, var_name):\n        with tf.name_scope(var_name):\n            mean = tf.reduce_mean(var)\n            stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean)))\n            tf.summary.scalar('mean', mean)\n            tf.summary.scalar('stddev', stddev)\n            tf.summary.scalar('max', tf.reduce_max(var))\n            tf.summary.scalar('min', tf.reduce_min(var))\n            tf.summary.histogram('histogram', var)\n    \n    # function to create the graph\n    def create_graph(self):\n\n        # reset default graph\n        tf.reset_default_graph()\n\n        # variables for input and output \n        self.x_data_tf = tf.placeholder(dtype=tf.float32, shape=[None,28,28,1], \n                                        name='x_data_tf')\n        self.y_data_tf = tf.placeholder(dtype=tf.float32, shape=[None,10], name='y_data_tf')\n\n        # 1.layer: convolution + max pooling\n        self.W_conv1_tf = self.weight_variable([self.s_f_conv1, self.s_f_conv1, 1,\n                                                self.n_f_conv1], \n                                               name = 'W_conv1_tf') # (5,5,1,32)\n        self.b_conv1_tf = self.bias_variable([self.n_f_conv1], name = 'b_conv1_tf') # (32)\n        self.h_conv1_tf = tf.nn.relu(self.conv2d(self.x_data_tf, \n                                                 self.W_conv1_tf) + self.b_conv1_tf, \n                                     name = 'h_conv1_tf') # (.,28,28,32)\n        self.h_pool1_tf = self.max_pool_2x2(self.h_conv1_tf, \n                                            name = 'h_pool1_tf') # (.,14,14,32)\n\n        # 2.layer: convolution + max pooling\n        self.W_conv2_tf = self.weight_variable([self.s_f_conv2, self.s_f_conv2, \n                                                self.n_f_conv1, self.n_f_conv2], \n                                               name = 'W_conv2_tf')\n        self.b_conv2_tf = self.bias_variable([self.n_f_conv2], name = 'b_conv2_tf')\n        self.h_conv2_tf = tf.nn.relu(self.conv2d(self.h_pool1_tf, \n                                                 self.W_conv2_tf) + self.b_conv2_tf, \n                                     name ='h_conv2_tf') #(.,14,14,32)\n        self.h_pool2_tf = self.max_pool_2x2(self.h_conv2_tf, name = 'h_pool2_tf') #(.,7,7,32)\n\n        # 3.layer: convolution + max pooling\n        self.W_conv3_tf = self.weight_variable([self.s_f_conv3, self.s_f_conv3, \n                                                self.n_f_conv2, self.n_f_conv3], \n                                               name = 'W_conv3_tf')\n        self.b_conv3_tf = self.bias_variable([self.n_f_conv3], name = 'b_conv3_tf')\n        self.h_conv3_tf = tf.nn.relu(self.conv2d(self.h_pool2_tf, \n                                                 self.W_conv3_tf) + self.b_conv3_tf, \n                                     name = 'h_conv3_tf') #(.,7,7,32)\n        self.h_pool3_tf = self.max_pool_2x2(self.h_conv3_tf, \n                                            name = 'h_pool3_tf') # (.,4,4,32)\n\n        # 4.layer: fully connected\n        self.W_fc1_tf = self.weight_variable([4*4*self.n_f_conv3,self.n_n_fc1], \n                                             name = 'W_fc1_tf') # (4*4*32, 1024)\n        self.b_fc1_tf = self.bias_variable([self.n_n_fc1], name = 'b_fc1_tf') # (1024)\n        self.h_pool3_flat_tf = tf.reshape(self.h_pool3_tf, [-1,4*4*self.n_f_conv3], \n                                          name = 'h_pool3_flat_tf') # (.,1024)\n        self.h_fc1_tf = tf.nn.relu(tf.matmul(self.h_pool3_flat_tf, \n                                             self.W_fc1_tf) + self.b_fc1_tf, \n                                   name = 'h_fc1_tf') # (.,1024)\n      \n        # add dropout\n        self.keep_prob_tf = tf.placeholder(dtype=tf.float32, name = 'keep_prob_tf')\n        self.h_fc1_drop_tf = tf.nn.dropout(self.h_fc1_tf, self.keep_prob_tf, \n                                           name = 'h_fc1_drop_tf')\n\n        # 5.layer: fully connected\n        self.W_fc2_tf = self.weight_variable([self.n_n_fc1, 10], name = 'W_fc2_tf')\n        self.b_fc2_tf = self.bias_variable([10], name = 'b_fc2_tf')\n        self.z_pred_tf = tf.add(tf.matmul(self.h_fc1_drop_tf, self.W_fc2_tf), \n                                self.b_fc2_tf, name = 'z_pred_tf')# => (.,10)\n\n        # cost function\n        self.cross_entropy_tf = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n            labels=self.y_data_tf, logits=self.z_pred_tf), name = 'cross_entropy_tf')\n     \n        # optimisation function\n        self.learn_rate_tf = tf.placeholder(dtype=tf.float32, name=\"learn_rate_tf\")\n        self.train_step_tf = tf.train.AdamOptimizer(self.learn_rate_tf).minimize(\n            self.cross_entropy_tf, name = 'train_step_tf')\n\n        # predicted probabilities in one-hot encoding\n        self.y_pred_proba_tf = tf.nn.softmax(self.z_pred_tf, name='y_pred_proba_tf') \n        \n        # tensor of correct predictions\n        self.y_pred_correct_tf = tf.equal(tf.argmax(self.y_pred_proba_tf, 1),\n                                          tf.argmax(self.y_data_tf, 1),\n                                          name = 'y_pred_correct_tf')  \n        \n        # accuracy \n        self.accuracy_tf = tf.reduce_mean(tf.cast(self.y_pred_correct_tf, dtype=tf.float32),\n                                         name = 'accuracy_tf')\n\n        # tensors to save intermediate accuracies and losses during training\n        self.train_loss_tf = tf.Variable(np.array([]), dtype=tf.float32, \n                                         name='train_loss_tf', validate_shape = False)\n        self.valid_loss_tf = tf.Variable(np.array([]), dtype=tf.float32, \n                                         name='valid_loss_tf', validate_shape = False)\n        self.train_acc_tf = tf.Variable(np.array([]), dtype=tf.float32, \n                                        name='train_acc_tf', validate_shape = False)\n        self.valid_acc_tf = tf.Variable(np.array([]), dtype=tf.float32, \n                                        name='valid_acc_tf', validate_shape = False)\n     \n        # number of weights and biases\n        num_weights = (self.s_f_conv1**2*self.n_f_conv1 \n                       + self.s_f_conv2**2*self.n_f_conv1*self.n_f_conv2 \n                       + self.s_f_conv3**2*self.n_f_conv2*self.n_f_conv3 \n                       + 4*4*self.n_f_conv3*self.n_n_fc1 + self.n_n_fc1*10)\n        num_biases = self.n_f_conv1 + self.n_f_conv2 + self.n_f_conv3 + self.n_n_fc1\n        print('num_weights =', num_weights)\n        print('num_biases =', num_biases)\n        \n        return None  \n    \n    def attach_summary(self, sess):\n        \n        # create summary tensors for tensorboard\n        self.use_tb_summary = True\n        self.summary_variable(self.W_conv1_tf, 'W_conv1_tf')\n        self.summary_variable(self.b_conv1_tf, 'b_conv1_tf')\n        self.summary_variable(self.W_conv2_tf, 'W_conv2_tf')\n        self.summary_variable(self.b_conv2_tf, 'b_conv2_tf')\n        self.summary_variable(self.W_conv3_tf, 'W_conv3_tf')\n        self.summary_variable(self.b_conv3_tf, 'b_conv3_tf')\n        self.summary_variable(self.W_fc1_tf, 'W_fc1_tf')\n        self.summary_variable(self.b_fc1_tf, 'b_fc1_tf')\n        self.summary_variable(self.W_fc2_tf, 'W_fc2_tf')\n        self.summary_variable(self.b_fc2_tf, 'b_fc2_tf')\n        tf.summary.scalar('cross_entropy_tf', self.cross_entropy_tf)\n        tf.summary.scalar('accuracy_tf', self.accuracy_tf)\n\n        # merge all summaries for tensorboard\n        self.merged = tf.summary.merge_all()\n\n        # initialize summary writer \n        timestamp = datetime.datetime.now().strftime('%d-%m-%Y_%H-%M-%S')\n        filepath = os.path.join(os.getcwd(), 'logs', (self.nn_name+'_'+timestamp))\n        self.train_writer = tf.summary.FileWriter(os.path.join(filepath,'train'), sess.graph)\n        self.valid_writer = tf.summary.FileWriter(os.path.join(filepath,'valid'), sess.graph)\n\n    def attach_saver(self):\n        # initialize tensorflow saver\n        self.use_tf_saver = True\n        self.saver_tf = tf.train.Saver()\n\n    # function to train the graph\n    def train_graph(self, sess, x_train, y_train, x_valid, y_valid, n_epoch = 1, \n                    train_on_augmented_data = False):\n\n        # train on original or augmented data\n        self.train_on_augmented_data = train_on_augmented_data\n        \n        # training and validation data\n        self.x_train = x_train\n        self.y_train = y_train\n        self.x_valid = x_valid\n        self.y_valid = y_valid\n        \n        # use augmented data\n        if self.train_on_augmented_data:\n            print('generate new set of images')\n            self.x_train_aug = normalize_data(self.generate_images(self.x_train))\n            self.y_train_aug = self.y_train\n        \n        # parameters\n        mb_per_epoch = self.x_train.shape[0]/self.mb_size\n        train_loss, train_acc, valid_loss, valid_acc = [],[],[],[]\n        \n        # start timer\n        start = datetime.datetime.now();\n        print(datetime.datetime.now().strftime('%d-%m-%Y %H:%M:%S'),': start training')\n        print('learnrate = ',self.learn_rate,', n_epoch = ', n_epoch,\n              ', mb_size = ', self.mb_size)\n        # looping over mini batches\n        for i in range(int(n_epoch*mb_per_epoch)+1):\n\n            # adapt learn_rate\n            self.learn_rate_pos = int(self.current_epoch // self.learn_rate_step_size)\n            if not self.learn_rate == self.learn_rate_array[self.learn_rate_pos]:\n                self.learn_rate = self.learn_rate_array[self.learn_rate_pos]\n                print(datetime.datetime.now()-start,': set learn rate to %.6f'%self.learn_rate)\n            \n            # get new batch\n            x_batch, y_batch = self.next_mini_batch() \n\n            # run the graph\n            sess.run(self.train_step_tf, feed_dict={self.x_data_tf: x_batch, \n                                                    self.y_data_tf: y_batch, \n                                                    self.keep_prob_tf: self.keep_prob, \n                                                    self.learn_rate_tf: self.learn_rate})\n             \n            \n            # store losses and accuracies\n            if i%int(self.log_step*mb_per_epoch) == 0 or i == int(n_epoch*mb_per_epoch):\n             \n                self.n_log_step += 1 # for logging the results\n                \n                feed_dict_train = {\n                    self.x_data_tf: self.x_train[self.perm_array[:len(self.x_valid)]], \n                    self.y_data_tf: self.y_train[self.perm_array[:len(self.y_valid)]], \n                    self.keep_prob_tf: 1.0}\n                \n                feed_dict_valid = {self.x_data_tf: self.x_valid, \n                                   self.y_data_tf: self.y_valid, \n                                   self.keep_prob_tf: 1.0}\n                \n                # summary for tensorboard\n                if self.use_tb_summary:\n                    train_summary = sess.run(self.merged, feed_dict = feed_dict_train)\n                    valid_summary = sess.run(self.merged, feed_dict = feed_dict_valid)\n                    self.train_writer.add_summary(train_summary, self.n_log_step)\n                    self.valid_writer.add_summary(valid_summary, self.n_log_step)\n                \n                train_loss.append(sess.run(self.cross_entropy_tf,\n                                           feed_dict = feed_dict_train))\n\n                train_acc.append(self.accuracy_tf.eval(session = sess, \n                                                       feed_dict = feed_dict_train))\n                \n                valid_loss.append(sess.run(self.cross_entropy_tf,\n                                           feed_dict = feed_dict_valid))\n\n                valid_acc.append(self.accuracy_tf.eval(session = sess, \n                                                       feed_dict = feed_dict_valid))\n\n                print('%.2f epoch: train/val loss = %.4f/%.4f, train/val acc = %.4f/%.4f'%(\n                    self.current_epoch, train_loss[-1], valid_loss[-1],\n                    train_acc[-1], valid_acc[-1]))\n     \n        # concatenate losses and accuracies and assign to tensor variables\n        tl_c = np.concatenate([self.train_loss_tf.eval(session=sess), train_loss], axis = 0)\n        vl_c = np.concatenate([self.valid_loss_tf.eval(session=sess), valid_loss], axis = 0)\n        ta_c = np.concatenate([self.train_acc_tf.eval(session=sess), train_acc], axis = 0)\n        va_c = np.concatenate([self.valid_acc_tf.eval(session=sess), valid_acc], axis = 0)\n   \n        sess.run(tf.assign(self.train_loss_tf, tl_c, validate_shape = False))\n        sess.run(tf.assign(self.valid_loss_tf, vl_c , validate_shape = False))\n        sess.run(tf.assign(self.train_acc_tf, ta_c , validate_shape = False))\n        sess.run(tf.assign(self.valid_acc_tf, va_c , validate_shape = False))\n        \n        print('running time for training: ', datetime.datetime.now() - start)\n        return None\n  \n    # save tensors/summaries\n    def save_model(self, sess):\n        \n        # tf saver\n        if self.use_tf_saver:\n            #filepath = os.path.join(os.getcwd(), 'logs' , self.nn_name)\n            filepath = os.path.join(os.getcwd(), self.nn_name)\n            self.saver_tf.save(sess, filepath)\n        \n        # tb summary\n        if self.use_tb_summary:\n            self.train_writer.close()\n            self.valid_writer.close()\n        \n        return None\n  \n    # forward prediction of current graph\n    def forward(self, sess, x_data):\n        y_pred_proba = self.y_pred_proba_tf.eval(session = sess, \n                                                 feed_dict = {self.x_data_tf: x_data,\n                                                              self.keep_prob_tf: 1.0})\n        return y_pred_proba\n    \n    # function to load tensors from a saved graph\n    def load_tensors(self, graph):\n        \n        # input tensors\n        self.x_data_tf = graph.get_tensor_by_name(\"x_data_tf:0\")\n        self.y_data_tf = graph.get_tensor_by_name(\"y_data_tf:0\")\n        \n        # weights and bias tensors\n        self.W_conv1_tf = graph.get_tensor_by_name(\"W_conv1_tf:0\")\n        self.W_conv2_tf = graph.get_tensor_by_name(\"W_conv2_tf:0\")\n        self.W_conv3_tf = graph.get_tensor_by_name(\"W_conv3_tf:0\")\n        self.W_fc1_tf = graph.get_tensor_by_name(\"W_fc1_tf:0\")\n        self.W_fc2_tf = graph.get_tensor_by_name(\"W_fc2_tf:0\")\n        self.b_conv1_tf = graph.get_tensor_by_name(\"b_conv1_tf:0\")\n        self.b_conv2_tf = graph.get_tensor_by_name(\"b_conv2_tf:0\")\n        self.b_conv3_tf = graph.get_tensor_by_name(\"b_conv3_tf:0\")\n        self.b_fc1_tf = graph.get_tensor_by_name(\"b_fc1_tf:0\")\n        self.b_fc2_tf = graph.get_tensor_by_name(\"b_fc2_tf:0\")\n        \n        # activation tensors\n        self.h_conv1_tf = graph.get_tensor_by_name('h_conv1_tf:0')  \n        self.h_pool1_tf = graph.get_tensor_by_name('h_pool1_tf:0')\n        self.h_conv2_tf = graph.get_tensor_by_name('h_conv2_tf:0')\n        self.h_pool2_tf = graph.get_tensor_by_name('h_pool2_tf:0')\n        self.h_conv3_tf = graph.get_tensor_by_name('h_conv3_tf:0')\n        self.h_pool3_tf = graph.get_tensor_by_name('h_pool3_tf:0')\n        self.h_fc1_tf = graph.get_tensor_by_name('h_fc1_tf:0')\n        self.z_pred_tf = graph.get_tensor_by_name('z_pred_tf:0')\n        \n        # training and prediction tensors\n        self.learn_rate_tf = graph.get_tensor_by_name(\"learn_rate_tf:0\")\n        self.keep_prob_tf = graph.get_tensor_by_name(\"keep_prob_tf:0\")\n        self.cross_entropy_tf = graph.get_tensor_by_name('cross_entropy_tf:0')\n        self.train_step_tf = graph.get_operation_by_name('train_step_tf')\n        self.z_pred_tf = graph.get_tensor_by_name('z_pred_tf:0')\n        self.y_pred_proba_tf = graph.get_tensor_by_name(\"y_pred_proba_tf:0\")\n        self.y_pred_correct_tf = graph.get_tensor_by_name('y_pred_correct_tf:0')\n        self.accuracy_tf = graph.get_tensor_by_name('accuracy_tf:0')\n        \n        # tensor of stored losses and accuricies during training\n        self.train_loss_tf = graph.get_tensor_by_name(\"train_loss_tf:0\")\n        self.train_acc_tf = graph.get_tensor_by_name(\"train_acc_tf:0\")\n        self.valid_loss_tf = graph.get_tensor_by_name(\"valid_loss_tf:0\")\n        self.valid_acc_tf = graph.get_tensor_by_name(\"valid_acc_tf:0\")\n  \n        return None\n    \n    # get losses of training and validation sets\n    def get_loss(self, sess):\n        train_loss = self.train_loss_tf.eval(session = sess)\n        valid_loss = self.valid_loss_tf.eval(session = sess)\n        return train_loss, valid_loss \n        \n    # get accuracies of training and validation sets\n    def get_accuracy(self, sess):\n        train_acc = self.train_acc_tf.eval(session = sess)\n        valid_acc = self.valid_acc_tf.eval(session = sess)\n        return train_acc, valid_acc \n    \n    # get weights\n    def get_weights(self, sess):\n        W_conv1 = self.W_conv1_tf.eval(session = sess)\n        W_conv2 = self.W_conv2_tf.eval(session = sess)\n        W_conv3 = self.W_conv3_tf.eval(session = sess)\n        W_fc1_tf = self.W_fc1_tf.eval(session = sess)\n        W_fc2_tf = self.W_fc2_tf.eval(session = sess)\n        return W_conv1, W_conv2, W_conv3, W_fc1_tf, W_fc2_tf\n    \n    # get biases\n    def get_biases(self, sess):\n        b_conv1 = self.b_conv1_tf.eval(session = sess)\n        b_conv2 = self.b_conv2_tf.eval(session = sess)\n        b_conv3 = self.b_conv3_tf.eval(session = sess)\n        b_fc1_tf = self.b_fc1_tf.eval(session = sess)\n        b_fc2_tf = self.b_fc2_tf.eval(session = sess)\n        return b_conv1, b_conv2, b_conv3, b_fc1_tf, b_fc2_tf\n    \n    # load session from file, restore graph, and load tensors\n    def load_session_from_file(self, filename):\n        tf.reset_default_graph()\n        filepath = os.path.join(os.getcwd(), filename + '.meta')\n        #filepath = os.path.join(os.getcwd(),'logs', filename + '.meta')\n        saver = tf.train.import_meta_graph(filepath)\n        print(filepath)\n        sess = tf.Session()\n        saver.restore(sess, mn)\n        graph = tf.get_default_graph()\n        self.load_tensors(graph)\n        return sess\n    \n    # receive activations given the input\n    def get_activations(self, sess, x_data):\n        feed_dict = {self.x_data_tf: x_data, self.keep_prob_tf: 1.0}\n        h_conv1 = self.h_conv1_tf.eval(session = sess, feed_dict = feed_dict)\n        h_pool1 = self.h_pool1_tf.eval(session = sess, feed_dict = feed_dict)\n        h_conv2 = self.h_conv2_tf.eval(session = sess, feed_dict = feed_dict)\n        h_pool2 = self.h_pool2_tf.eval(session = sess, feed_dict = feed_dict)\n        h_conv3 = self.h_conv3_tf.eval(session = sess, feed_dict = feed_dict)\n        h_pool3 = self.h_pool3_tf.eval(session = sess, feed_dict = feed_dict)\n        h_fc1 = self.h_fc1_tf.eval(session = sess, feed_dict = feed_dict)\n        h_fc2 = self.z_pred_tf.eval(session = sess, feed_dict = feed_dict)\n        return h_conv1,h_pool1,h_conv2,h_pool2,h_conv3,h_pool3,h_fc1,h_fc2\n    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"180614885e56beb180592796a1267c0fbcab30ac","_cell_guid":"e2b7c61e-88bb-40dd-bde0-5ec1977d8365"},"cell_type":"markdown","source":"# 6. Train and validate the neural network <a class=\"anchor\" id=\"6-bullet\"></a> \n- first try out some sklearn models\n- train the neural network \n- visualize the losses, accuracies, the weights and the activations\n- tune the hyperparameters\n"},{"metadata":{"_uuid":"98c599d07113cfa8bfee30244e31655400737184","_cell_guid":"2ca5e137-2767-4261-ac20-65d86042e856","trusted":true},"cell_type":"code","source":"## train the neural network graph\n\n#nn_name = ['nn0','nn1','nn2','nn3','nn4','nn5','nn6','nn7','nn8','nn9']\n\nnn_name = ['tmp']\n\n# cross validations\ncv_num = 10 # cross validations default = 20 => 5% validation set\nkfold = sklearn.model_selection.KFold(cv_num, shuffle=True, random_state=123)\n\nfor i,(train_index, valid_index) in enumerate(kfold.split(x_train_valid)):\n    \n    # start timer\n    start = datetime.datetime.now();\n    \n    # train and validation data of original images\n    x_train = x_train_valid[train_index]\n    y_train = y_train_valid[train_index]\n    x_valid = x_train_valid[valid_index]\n    y_valid = y_train_valid[valid_index]\n    \n    # create neural network graph\n    nn_graph = nn_class(nn_name = nn_name[i]) # instance of nn_class\n    nn_graph.create_graph() # create graph\n    nn_graph.attach_saver() # attach saver tensors\n    \n    # start tensorflow session\n    with tf.Session() as sess:\n        \n        # attach summaries\n        nn_graph.attach_summary(sess) \n        \n        # variable initialization of the default graph\n        sess.run(tf.global_variables_initializer()) \n    \n        # training on original data\n        nn_graph.train_graph(sess, x_train, y_train, x_valid, y_valid, n_epoch = 1.0)\n        \n        # training on augmented data\n        nn_graph.train_graph(sess, x_train, y_train, x_valid, y_valid, n_epoch = 14.0,\n                            train_on_augmented_data = True)\n\n        # save tensors and summaries of model\n        nn_graph.save_model(sess)\n        \n    # only one iteration\n    if True:\n        break;\n        \n    \nprint('total running time for training: ', datetime.datetime.now() - start)\n    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7fcb229c82c067efe82f5e8676bf850b315cafe4","_cell_guid":"e45f3f59-97db-455d-830c-440fa5bf1ab5","trusted":true},"cell_type":"code","source":"## visualization with tensorboard\n\nif False:\n    !tensorboard --logdir=./logs","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c80e97ad0f7f3061e087fc2acce93bea77385ece","_cell_guid":"e379fdcc-852a-4eb4-a207-aaa177482c97","trusted":true},"cell_type":"code","source":"## show confusion matrix\n\nmn = nn_name[0]\nnn_graph = nn_class()\nsess = nn_graph.load_session_from_file(mn)\ny_valid_pred[mn] = nn_graph.forward(sess, x_valid)\nsess.close()\n\ncnf_matrix = sklearn.metrics.confusion_matrix(\n    one_hot_to_dense(y_valid_pred[mn]), one_hot_to_dense(y_valid)).astype(np.float32)\n\nlabels_array = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']\nfig, ax = plt.subplots(1,figsize=(10,10))\nax = sns.heatmap(cnf_matrix, ax=ax, cmap=plt.cm.Greens, annot=True)\nax.set_xticklabels(labels_array)\nax.set_yticklabels(labels_array)\nplt.title('Confusion matrix of validation set')\nplt.ylabel('True digit')\nplt.xlabel('Predicted digit')\nplt.show();","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6541c6ca89fd856531b2f0280cbe78d91da87183","_cell_guid":"7e351ab9-f9dc-41d1-80ee-802b1c8a9114","trusted":true},"cell_type":"code","source":"## loss and accuracy curves\n\nmn = nn_name[0]\nnn_graph = nn_class()\nsess = nn_graph.load_session_from_file(mn)\ntrain_loss[mn], valid_loss[mn] = nn_graph.get_loss(sess)\ntrain_acc[mn], valid_acc[mn] = nn_graph.get_accuracy(sess)\nsess.close()\n\nprint('final train/valid loss = %.4f/%.4f, train/valid accuracy = %.4f/%.4f'%(\n    train_loss[mn][-1], valid_loss[mn][-1], train_acc[mn][-1], valid_acc[mn][-1]))\n\nplt.figure(figsize=(10, 5));\nplt.subplot(1,2,1);\nplt.plot(np.arange(0,len(train_acc[mn])), train_acc[mn],'-b', label='Training')\nplt.plot(np.arange(0,len(valid_acc[mn])), valid_acc[mn],'-g', label='Validation')\nplt.legend(loc='lower right', frameon=False)\nplt.ylim(ymax = 1.1, ymin = 0.0)\nplt.ylabel('accuracy')\nplt.xlabel('log steps');\n\nplt.subplot(1,2,2)\nplt.plot(np.arange(0,len(train_loss[mn])), train_loss[mn],'-b', label='Training')\nplt.plot(np.arange(0,len(valid_loss[mn])), valid_loss[mn],'-g', label='Validation')\nplt.legend(loc='lower right', frameon=False)\nplt.ylim(ymax = 3.0, ymin = 0.0)\nplt.ylabel('loss')\nplt.xlabel('log steps');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5532356d3a392b9d76205c02cdc17d92fdfe2481","_cell_guid":"add9e626-e259-4b3f-82ef-078778113782","trusted":true},"cell_type":"code","source":"## visualize weights\n\nmn = nn_name[0]\nnn_graph = nn_class()\nsess = nn_graph.load_session_from_file(mn)\nW_conv1, W_conv2, W_conv3, _, _ = nn_graph.get_weights(sess)\nsess.close()\n\nprint('W_conv1: min = ' + str(np.min(W_conv1)) + ' max = ' + str(np.max(W_conv1))\n      + ' mean = ' + str(np.mean(W_conv1)) + ' std = ' + str(np.std(W_conv1)))\nprint('W_conv2: min = ' + str(np.min(W_conv2)) + ' max = ' + str(np.max(W_conv2))\n      + ' mean = ' + str(np.mean(W_conv2)) + ' std = ' + str(np.std(W_conv2)))\nprint('W_conv3: min = ' + str(np.min(W_conv3)) + ' max = ' + str(np.max(W_conv3))\n      + ' mean = ' + str(np.mean(W_conv3)) + ' std = ' + str(np.std(W_conv3)))\n\ns_f_conv1 = nn_graph.s_f_conv1\ns_f_conv2 = nn_graph.s_f_conv2\ns_f_conv3 = nn_graph.s_f_conv3\n\nW_conv1 = np.reshape(W_conv1,(s_f_conv1,s_f_conv1,1,6,6))\nW_conv1 = np.transpose(W_conv1,(3,0,4,1,2))\nW_conv1 = np.reshape(W_conv1,(s_f_conv1*6,s_f_conv1*6,1))\n\nW_conv2 = np.reshape(W_conv2,(s_f_conv2,s_f_conv2,6,6,36))\nW_conv2 = np.transpose(W_conv2,(2,0,3,1,4))\nW_conv2 = np.reshape(W_conv2,(6*s_f_conv2,6*s_f_conv2,6,6))\nW_conv2 = np.transpose(W_conv2,(2,0,3,1))\nW_conv2 = np.reshape(W_conv2,(6*6*s_f_conv2,6*6*s_f_conv2))\n\nW_conv3 = np.reshape(W_conv3,(s_f_conv3,s_f_conv3,6,6,36))\nW_conv3 = np.transpose(W_conv3,(2,0,3,1,4))\nW_conv3 = np.reshape(W_conv3,(6*s_f_conv3,6*s_f_conv3,6,6))\nW_conv3 = np.transpose(W_conv3,(2,0,3,1))\nW_conv3 = np.reshape(W_conv3,(6*6*s_f_conv3,6*6*s_f_conv3))\n\nplt.figure(figsize=(15,5))\nplt.subplot(1,3,1)\nplt.gca().set_xticks(np.arange(-0.5, s_f_conv1*6, s_f_conv1), minor = False);\nplt.gca().set_yticks(np.arange(-0.5, s_f_conv1*6, s_f_conv1), minor = False);\nplt.grid(which = 'minor', color='b', linestyle='-', linewidth=1)\nplt.title('W_conv1 ' + str(W_conv1.shape))\nplt.colorbar(plt.imshow(W_conv1[:,:,0], cmap=cm.binary));\n\nplt.subplot(1,3,2)\nplt.gca().set_xticks(np.arange(-0.5, 6*6*s_f_conv2, 6*s_f_conv2), minor = False);\nplt.gca().set_yticks(np.arange(-0.5, 6*6*s_f_conv2, 6*s_f_conv2), minor = False);\nplt.grid(which = 'minor', color='b', linestyle='-', linewidth=1)\nplt.title('W_conv2 ' + str(W_conv2.shape))\nplt.colorbar(plt.imshow(W_conv2[:,:], cmap=cm.binary));\n\nplt.subplot(1,3,3)\nplt.gca().set_xticks(np.arange(-0.5, 6*6*s_f_conv3, 6*s_f_conv3), minor = False);\nplt.gca().set_yticks(np.arange(-0.5, 6*6*s_f_conv3, 6*s_f_conv3), minor = False);\nplt.grid(which = 'minor', color='b', linestyle='-', linewidth=1)\nplt.title('W_conv3 ' + str(W_conv3.shape))\nplt.colorbar(plt.imshow(W_conv3[:,:], cmap=cm.binary));\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f169ddac4c3639ba734f67048ba41b775d68094","_cell_guid":"ab004898-a7ec-4e89-9be2-503db6e24d66","trusted":true},"cell_type":"code","source":"## visualize activations\n\nimg_no = 10;\nmn = nn_name[0]\nnn_graph = nn_class()\nsess = nn_graph.load_session_from_file(mn)\n(h_conv1, h_pool1, h_conv2, h_pool2,h_conv3, h_pool3, h_fc1,\n h_fc2) = nn_graph.get_activations(sess, x_train_valid[img_no:img_no+1])\nsess.close()\n    \n# original image\nplt.figure(figsize=(15,9))\nplt.subplot(2,4,1)\nplt.imshow(x_train_valid[img_no].reshape(28,28),cmap=cm.binary);\n\n# 1. convolution\nplt.subplot(2,4,2)\nplt.title('h_conv1 ' + str(h_conv1.shape))\nh_conv1 = np.reshape(h_conv1,(-1,28,28,6,6))\nh_conv1 = np.transpose(h_conv1,(0,3,1,4,2))\nh_conv1 = np.reshape(h_conv1,(-1,6*28,6*28))\nplt.imshow(h_conv1[0], cmap=cm.binary);\n\n# 1. max pooling\nplt.subplot(2,4,3)\nplt.title('h_pool1 ' + str(h_pool1.shape))\nh_pool1 = np.reshape(h_pool1,(-1,14,14,6,6))\nh_pool1 = np.transpose(h_pool1,(0,3,1,4,2))\nh_pool1 = np.reshape(h_pool1,(-1,6*14,6*14))\nplt.imshow(h_pool1[0], cmap=cm.binary);\n\n# 2. convolution\nplt.subplot(2,4,4)\nplt.title('h_conv2 ' + str(h_conv2.shape))\nh_conv2 = np.reshape(h_conv2,(-1,14,14,6,6))\nh_conv2 = np.transpose(h_conv2,(0,3,1,4,2))\nh_conv2 = np.reshape(h_conv2,(-1,6*14,6*14))\nplt.imshow(h_conv2[0], cmap=cm.binary);\n\n# 2. max pooling\nplt.subplot(2,4,5)\nplt.title('h_pool2 ' + str(h_pool2.shape))\nh_pool2 = np.reshape(h_pool2,(-1,7,7,6,6))\nh_pool2 = np.transpose(h_pool2,(0,3,1,4,2))\nh_pool2 = np.reshape(h_pool2,(-1,6*7,6*7))\nplt.imshow(h_pool2[0], cmap=cm.binary);\n\n# 3. convolution\nplt.subplot(2,4,6)\nplt.title('h_conv3 ' + str(h_conv3.shape))\nh_conv3 = np.reshape(h_conv3,(-1,7,7,6,6))\nh_conv3 = np.transpose(h_conv3,(0,3,1,4,2))\nh_conv3 = np.reshape(h_conv3,(-1,6*7,6*7))\nplt.imshow(h_conv3[0], cmap=cm.binary);\n\n# 3. max pooling\nplt.subplot(2,4,7)\nplt.title('h_pool2 ' + str(h_pool3.shape))\nh_pool3 = np.reshape(h_pool3,(-1,4,4,6,6))\nh_pool3 = np.transpose(h_pool3,(0,3,1,4,2))\nh_pool3 = np.reshape(h_pool3,(-1,6*4,6*4))\nplt.imshow(h_pool3[0], cmap=cm.binary);\n\n# 4. FC layer\nplt.subplot(2,4,8)\nplt.title('h_fc1 ' + str(h_fc1.shape))\nh_fc1 = np.reshape(h_fc1,(-1,24,24))\nplt.imshow(h_fc1[0], cmap=cm.binary);\n\n# 5. FC layer\nnp.set_printoptions(precision=2)\nprint('h_fc2 = ', h_fc2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aaa323f005713a891ab3a1faa7aa9549fd88ff30","_cell_guid":"f7508494-8533-492d-8772-f14a99e35ade","trusted":true},"cell_type":"code","source":"## show misclassified images\n\nmn = nn_name[0]\nnn_graph = nn_class()\nsess = nn_graph.load_session_from_file(mn)\ny_valid_pred[mn] = nn_graph.forward(sess, x_valid)\nsess.close()\n\ny_valid_pred_label = one_hot_to_dense(y_valid_pred[mn])\ny_valid_label = one_hot_to_dense(y_valid)\ny_val_false_index = []\n\nfor i in range(y_valid_label.shape[0]):\n    if y_valid_pred_label[i] != y_valid_label[i]:\n        y_val_false_index.append(i)\n\nprint('# false predictions: ', len(y_val_false_index),'out of', len(y_valid))\n\nplt.figure(figsize=(10,15))\nfor j in range(0,5):\n    for i in range(0,10):\n        if j*10+i<len(y_val_false_index):\n            plt.subplot(10,10,j*10+i+1)\n            plt.title('%d/%d'%(y_valid_label[y_val_false_index[j*10+i]],\n                               y_valid_pred_label[y_val_false_index[j*10+i]]))\n            plt.imshow(x_valid[y_val_false_index[j*10+i]].reshape(28,28),cmap=cm.binary)    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"176c76821d5518569b5e33569dd127fd91039cc4","_cell_guid":"069e66b4-fb6c-4a55-9dd0-5f0d23717388"},"cell_type":"markdown","source":"# 7. Stacking of models and training a meta-model  <a class=\"anchor\" id=\"7-bullet\"></a> "},{"metadata":{"_uuid":"fb420ac72ef704929764ae4bb0c86c5031fb05d5","_cell_guid":"9ee6b6bb-3865-499f-94ec-627f8c73673c","trusted":true},"cell_type":"code","source":"## read test data\n\n# read test data from CSV file \nif os.path.isfile('../input/test.csv'):\n    test_df = pd.read_csv('../input/test.csv') # on kaggle \n    print('test.csv loaded: test_df{0}'.format(test_df.shape))\nelif os.path.isfile('data/test.csv'):\n    test_df = pd.read_csv('data/test.csv') # on local environment\n    print('test.csv loaded: test_df{0}'.format(test_df.shape))\nelse:\n    print('Error: test.csv not found')\n    \n# transforma and normalize test data\nx_test = test_df.iloc[:,0:].values.reshape(-1,28,28,1) # (28000,28,28,1) array\nx_test = x_test.astype(np.float)\nx_test = normalize_data(x_test)\nprint('x_test.shape = ', x_test.shape)\n\n# for saving results\ny_test_pred = {}\ny_test_pred_labels = {}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eb9f7f9af44b85966a82e301d5a027e9269cbb80","_cell_guid":"a58c4c30-67f4-44b1-909a-28899c03e436","trusted":true},"cell_type":"code","source":"## Stacking of neural networks\n\nif False:\n    \n    take_models = ['nn0','nn1','nn2','nn3','nn4','nn5','nn6','nn7','nn8','nn9']\n\n    # cross validations\n    # choose the same seed as was done for training the neural nets\n    kfold = sklearn.model_selection.KFold(len(take_models), shuffle=True, random_state = 123)\n\n    # train and test data for meta model\n    x_train_meta = np.array([]).reshape(-1,10)\n    y_train_meta = np.array([]).reshape(-1,10)\n    x_test_meta = np.zeros((x_test.shape[0], 10))\n\n    print('Out-of-folds predictions:')\n\n    # make out-of-folds predictions from base models\n    for i,(train_index, valid_index) in enumerate(kfold.split(x_train_valid)):\n\n        # training and validation data\n        x_train = x_train_valid[train_index]\n        y_train = y_train_valid[train_index]\n        x_valid = x_train_valid[valid_index]\n        y_valid = y_train_valid[valid_index]\n\n        # load neural network and make predictions\n        mn = take_models[i] \n        nn_graph = nn_class()\n        sess = nn_graph.load_session_from_file(mn)\n        y_train_pred[mn] = nn_graph.forward(sess, x_train[:len(x_valid)])\n        y_valid_pred[mn] = nn_graph.forward(sess, x_valid)\n        y_test_pred[mn] = nn_graph.forward(sess, x_test)\n        sess.close()\n\n        # create cloned model from base models\n        #model = sklearn.base.clone(base_models[take_models[i]])\n        #model.fit(x_train, y_train)\n        #y_train_pred_proba['tmp'] = model.predict_proba(x_train)[:,1]\n        #y_valid_pred_proba['tmp'] = model.predict_proba(x_valid)[:,1]\n        #y_test_pred_proba['tmp'] = model.predict_proba(x_test)[:,1]\n\n        # collect train and test data for meta model \n        x_train_meta = np.concatenate([x_train_meta, y_valid_pred[mn]])\n        y_train_meta = np.concatenate([y_train_meta, y_valid]) \n        x_test_meta += y_test_pred[mn]\n\n        print(take_models[i],': train/valid accuracy = %.4f/%.4f'%(\n            accuracy_from_one_hot_labels(y_train_pred[mn], y_train[:len(x_valid)]),\n            accuracy_from_one_hot_labels(y_valid_pred[mn], y_valid)))\n\n        if False:\n            break;\n\n    # take average of test predictions\n    x_test_meta = x_test_meta/(i+1)\n    y_test_pred['stacked_models'] = x_test_meta\n\n    print('')\n    print('Stacked models: valid accuracy = %.4f'%accuracy_from_one_hot_labels(x_train_meta,\n                                                                               y_train_meta))\n     ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be369d6a79943bc6244ab6497f4a048668c96796","_cell_guid":"203eacc1-761b-47c2-b44b-b12ae0e04ba9","trusted":true},"cell_type":"code","source":"## use meta model\n\nif False:\n    \n    logreg = sklearn.linear_model.LogisticRegression(verbose=0, solver='lbfgs',\n                                                     multi_class='multinomial')\n    \n    # choose meta model\n    take_meta_model = 'logreg'\n\n    # train meta model\n    model = sklearn.base.clone(base_models[take_meta_model]) \n    model.fit(x_train_meta, one_hot_to_dense(y_train_meta))\n    \n    y_train_pred['meta_model'] = model.predict_proba(x_train_meta)\n    y_test_pred['meta_model'] = model.predict_proba(x_test_meta)\n\n    print('Meta model: train accuracy = %.4f'%accuracy_from_one_hot_labels(x_train_meta, \n                                                           y_train_pred['meta_model']))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7cd749b33c0196b0b4364aa0ea6b9cfb963c07f4","_cell_guid":"749a20b3-ae70-4f65-99c1-96b935b2fc17","trusted":true},"cell_type":"code","source":"## choose one single model for test prediction\n\nif True:\n    \n    mn = nn_name[0] # choose saved model\n    nn_graph = nn_class() # create instance\n    sess = nn_graph.load_session_from_file(mn) # receive session \n    y_test_pred = {}\n    y_test_pred_labels = {}\n\n    # split evaluation of test predictions into batches\n    kfold = sklearn.model_selection.KFold(40, shuffle=False) \n    for i,(train_index, valid_index) in enumerate(kfold.split(x_test)):\n        if i==0:\n            y_test_pred[mn] = nn_graph.forward(sess, x_test[valid_index])\n        else: \n            y_test_pred[mn] = np.concatenate([y_test_pred[mn],\n                                              nn_graph.forward(sess, x_test[valid_index])])\n\n    sess.close()\n    \n    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"516501f9de8bbf528bc863a50cc8e494727b674d","_cell_guid":"6e02c22d-2755-4b0d-a5ee-85ea9c5d0c10"},"cell_type":"markdown","source":"# 8. Submit the test results <a class=\"anchor\" id=\"8-bullet\"></a> "},{"metadata":{"scrolled":true,"_uuid":"dd5c3638434505b4491ddd8c804bf53e9d20dfa6","_cell_guid":"2286dd81-8199-44ab-a119-1f7d5470b757","trusted":true},"cell_type":"code","source":"# choose the test predictions and submit the results\n\n#mn = 'meta_model'\nmn = nn_name[0]\ny_test_pred_labels[mn] = one_hot_to_dense(y_test_pred[mn])\n\nprint(mn+': y_test_pred_labels[mn].shape = ', y_test_pred_labels[mn].shape)\nunique, counts = np.unique(y_test_pred_labels[mn], return_counts=True)\nprint(dict(zip(unique, counts)))\n\n# save predictions\nnp.savetxt('submission.csv', \n           np.c_[range(1,len(x_test)+1), y_test_pred_labels[mn]], \n           delimiter=',', \n           header = 'ImageId,Label', \n           comments = '', \n           fmt='%d')\n\nprint('submission.csv completed')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"85b4ef639e68901b701d9281fcd65933f699c05a","_cell_guid":"7689a5ba-fdd8-4dcd-ab67-23fc2e3bf0fa","trusted":true},"cell_type":"code","source":"## look at some test images and predicted labels\n\nplt.figure(figsize=(10,15))\nfor j in range(0,5):\n    for i in range(0,10):\n        plt.subplot(10,10,j*10+i+1)\n        plt.title('%d'%y_test_pred_labels[mn][j*10+i])\n        plt.imshow(x_test[j*10+i].reshape(28,28), cmap=cm.binary)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e7f0bf61f5b2748fd2f1e2e8a35fcd560347f2fb","_cell_guid":"49194fc8-4179-4ed0-980a-47bb6b23acaa","collapsed":true,"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}