{"cells":[{"metadata":{"_uuid":"9a79bd7d809325484556b17575e4d4077d46af0f"},"cell_type":"markdown","source":"# Neural Networks training on CIFAR-10\nVassilis Krikonis\n\nCIFAR-10 is a dataset containing 60,000 images of 10 classes and is considered a typical dataset for computer vision problems.\n\nIn this assignment, I'll try out a few neural network configurations and hyperparameters and compare the results.\n\n## Preperation\n\nThe dataset comes in 6 pickled files. To use them, we need to serialize them using `pickle.load()` and store them into one big numpy array.\n\nEach image is originally stored as a flattened array of \\\\(width*height*channels\\\\) elements representing each pixel in one colour channel. The colour intensity i.e. value in each pixel varies from 0 to 255. By normalising the values to the range \\\\([0,1]\\\\) the performance of the neural networks I've tried training, increased immensely. Without this normalisation, I would get near random accuracy of 10% for the 10 classes in the dataset.\n\nAs a last step for the preparation of the data, we need to  convert the labels to a _one-hot-vector_ encoding, that is, a vector representing each possible label with binary flags. This step is necessary because the output of the neural network is actually a vector like this, a prediction vector with probability for each label."},{"metadata":{"trusted":true,"_uuid":"0f7484f0b27c3da1d60c1fb7285f68ada1c563a0"},"cell_type":"code","source":"import numpy as np","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"def unpickle(file):\n    import pickle\n    with open(file, 'rb') as fo:\n        dict = pickle.load(fo, encoding='latin')\n    return dict\n\nimport os\ndef load_batch_file(batch_filename):\n    filepath = os.path.join('../input/cifar-10-batches-py/', batch_filename)\n    unpickled = unpickle(filepath)\n    return unpickled","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a20a89c0a09202f69f208881ce56dd53dec75702"},"cell_type":"code","source":"train_batch_1 = load_batch_file('data_batch_1')\ntrain_batch_2 = load_batch_file('data_batch_2')\ntrain_batch_3 = load_batch_file('data_batch_3')\ntrain_batch_4 = load_batch_file('data_batch_4')\ntrain_batch_5 = load_batch_file('data_batch_5')\ntest_batch = load_batch_file('test_batch')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c6f660d083b4057ef00f5f681c1d2e6a63558a37"},"cell_type":"code","source":"from keras.utils import np_utils\nnum_classes = 10\ntrain_x = np.concatenate([train_batch_1['data'], train_batch_2['data'], train_batch_3['data'], train_batch_4['data'], train_batch_5['data']])\ntrain_x = train_x.astype('float32') # this is necessary for the division below\ntrain_x /= 255\ntrain_y = np.concatenate([np_utils.to_categorical(labels, num_classes) for labels in [train_batch_1['labels'], train_batch_2['labels'], train_batch_3['labels'], train_batch_4['labels'], train_batch_5['labels']]])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fb62c65f2f0edc422050d4f77ab58f31626399cc"},"cell_type":"code","source":"test_x = test_batch['data'].astype('float32') / 255\ntest_y = np_utils.to_categorical(test_batch['labels'], num_classes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3bbc07ba05e4ebe3dabff14629aaddf5d4246b3d"},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Dense","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"006b4200baedb34d94270903421e05503359d5ff"},"cell_type":"code","source":"img_rows = img_cols = 32\nchannels = 3","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f84141ccc2e673aa3a03c0f1529cf10eac55bf85"},"cell_type":"markdown","source":"## Attempts at training Neural Networks\n\n### Simple fully connected neural network with one hidden layer\n\nThe first (and naively basic) attempt was to train a dense neural network of 1 hidden layer, feeding the flattened arrays of the images as input. So we have the input layer with \\\\(32*32*3\\\\) inputs, a dense hidden layer of 10,000 nodes and _ReLU_ activation and the output layer with the 10 class nodes with _Softmax_ activation function to give out probabilities for the different classes."},{"metadata":{"trusted":true,"_uuid":"689d5bcfab4ee9d2d9aab9435e896f0f56cfa77c","_kg_hide-output":false},"cell_type":"code","source":"simple_model = Sequential()\nsimple_model.add(Dense(10_000, input_shape=(img_rows*img_cols*channels,), activation='relu'))\nsimple_model.add(Dense(10, activation='softmax'))\n\nsimple_model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy'])\nsimple_model_history = simple_model.fit(train_x, train_y, batch_size=100, epochs=8, validation_data=(test_x, test_y))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d2a9a1ec2528656b2d2f181a1254b786bb0d980"},"cell_type":"markdown","source":"To help us with visualising the loss and accuracy of the model over time, Keras provides a _History_ object to use that has stored the various metrics in each epoch. The following code was taken from the Keras documentation"},{"metadata":{"trusted":true,"_uuid":"b8e8b0925b9c4f03217bb04523eb4621f17c446a"},"cell_type":"code","source":"import matplotlib.pyplot as plt\ndef plot_history(history, title):\n    plt.figure(figsize=(10,3))\n    # Plot training & validation accuracy values\n    plt.subplot(121)\n    plt.plot(history.history['acc'])\n    plt.plot(history.history['val_acc'])\n    plt.title('Model accuracy')\n    plt.ylabel('Accuracy')\n    plt.xlabel('Epoch')\n    plt.legend(['Train', 'Test'], loc='upper left')\n\n    # Plot training & validation loss values\n    plt.subplot(122)\n    plt.plot(history.history['loss'])\n    plt.plot(history.history['val_loss'])\n    plt.title('Model loss')\n    plt.ylabel('Loss')\n    plt.xlabel('Epoch')\n    plt.legend(['Train', 'Test'], loc='upper left')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"16301a1df8be02b9ca306ec9990751e56cc086d9"},"cell_type":"markdown","source":"Even that shallow neural network, in such a few epochs, was able to achieve near 50% success and it's a hilariously dumb version of a neural network. Judging by the plots below, more epochs would achieve better results."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"80916600bb06adc67f9143d670581e6ce3401650"},"cell_type":"code","source":"plot_history(simple_model_history, 'Simple NN with 100 batch size')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"005db7b45471f1e5aef89a66c3465e50afa3d750"},"cell_type":"markdown","source":"### Changing the batch size\n\nWhen setting _batch size_ we tell the neural network to train with samples in batches of fixed size. The error from those samples accumulates and the network backprops the gradient descent after each step. That makes computations faster because we don't have to update the weights after each input. \n\nBut what about accuracy?"},{"metadata":{"trusted":true,"_uuid":"59b241acc73744ca7bfca0cd49d6a5fe615d370e"},"cell_type":"code","source":"simple_model_smaller_batch = Sequential()\nsimple_model_smaller_batch.add(Dense(10_000, input_shape=(img_rows*img_cols*channels,), activation='relu'))\nsimple_model_smaller_batch.add(Dense(10, activation='softmax'))\n\nsimple_model_smaller_batch.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy'])\nsimple_model_smaller_batch_history = simple_model_smaller_batch.fit(train_x, train_y, batch_size=50, epochs=8, validation_data=(test_x, test_y))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"baba3ab7d536dded9f056e652942d4ab992f9a28"},"cell_type":"markdown","source":"As expected, using smaller batch size (slightly) increases accuracy but hinders performance. More steps are needed to complete an epoch but the model updates more often as well."},{"metadata":{"trusted":true,"_uuid":"07dfa745ab5d1853b547d1c3ac054818a154cedb"},"cell_type":"code","source":"plot_history(simple_model_smaller_batch_history, 'Simple NN with 50 batch size')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"855b78afecaf6d3cb896ee7880d4b139d4802fbd"},"cell_type":"markdown","source":"### Increasing the depth of the simple neural network\n\nThe previous configurations had only one hidden layer. Due to hardware constraints I couldn't test it with more nodes but this is already an extremely shallow network. Could adding-in more layers improve the performance?"},{"metadata":{"trusted":true,"_uuid":"6639d49e36f475858a15431bfdb3c4308c3d8387"},"cell_type":"code","source":"simple_model_more_layers = Sequential()\nsimple_model_more_layers.add(Dense(10_000, input_shape=(img_rows*img_cols*channels,), activation='relu'))\nsimple_model_more_layers.add(Dense(1_000, activation='relu'))\nsimple_model_more_layers.add(Dense(100, activation='relu'))\nsimple_model_more_layers.add(Dense(10, activation='softmax'))\n\nsimple_model_more_layers.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy'])\nsimple_model_more_layers_history = simple_model_more_layers.fit(train_x, train_y, batch_size=100, epochs=8, validation_data=(test_x, test_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4fbcf80ad7fdbcb1621b8ab004be2f0e8d72ffb7"},"cell_type":"code","source":"plot_history(simple_model_more_layers_history, 'Simple NN with more layers')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e0b66b4553bf8382d79ae8a3f7256cc24e0b5d89"},"cell_type":"markdown","source":"It's not easy to tell if it improves accuracy. It seems that validation accuracy doesn't diverge from the training accuracy so if the model was left training for more epochs it could give out better results."},{"metadata":{"_uuid":"f43743e79067682ada1f8427f35b650ec90f00fb"},"cell_type":"markdown","source":"### Simple Convolution Neural Network\n\nOur input is an image and by treating it as a flat array we lose the locality on the features. Moreover, having over 3000 inputs adds computational overhead. \n\nConvolution Neural Networks on the other hand are ideal for images (among others) and by learning on the filter parameters and not on the transformation of the image itself we can achieve deeper and more complex architectures with better results.\n\nA C-NN is fed each image as a tensor i.e a multidimensional matrix so we need to _reshape_ the flattened arrays into stacked matrices for each colour channel. This way we end up with a \\\\(60000 \\times 32 \\times 32 \\times 3 \\\\) tensor.\n\nIn the example below, we train a _C-NN_ with 3 convolution layers of \\\\(3 \\times 3 \\\\) convolution matrices. These of course will be _flattened_ to feed the usual output layer."},{"metadata":{"trusted":true,"_uuid":"a033b4e2a53903c0b8be8e747fd94034330efa6a"},"cell_type":"code","source":"train_x_reshaped = train_x.reshape(len(train_x), img_rows, img_cols, channels)\ntest_x_reshaped = test_x.reshape(len(test_x), img_rows, img_cols, channels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"864791da4d8afa1c76be425316987bc11abf8a1a"},"cell_type":"code","source":"from keras.layers import Conv2D, Flatten\nsimple_cnn_model = Sequential()\nsimple_cnn_model.add(Conv2D(32, (3,3), input_shape=(img_rows,img_cols,channels), activation='relu'))\nsimple_cnn_model.add(Conv2D(32, (3,3), activation='relu'))\nsimple_cnn_model.add(Conv2D(32, (3,3), activation='relu'))\nsimple_cnn_model.add(Flatten())\nsimple_cnn_model.add(Dense(10, activation='softmax'))\n\nsimple_cnn_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\nsimple_cnn_model_history = simple_cnn_model.fit(train_x_reshaped, train_y, batch_size=100, epochs=8, validation_data=(test_x_reshaped, test_y))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1fa80594e9db6e778104c40e8aed892d669c8c06"},"cell_type":"markdown","source":"Plotting how well this configuration goes reveals something interesting: the model overfits after the 3rd epoch! We can see the validation accuracy slowly dropping while the training accuracy steadily increases. This is more visible in the plot of the model loss during the epochs. The accumulated Loss in the validation set clearly goes up after the 3rd epoch while the network keeps minimising the training set loss value.\n\nIn situation where our model overfits we need some sort of regularisation and in the next attempt I add in dropout layers.\n\nThat said, we do use _Adam_ as the model's optimizer so it may need some tuning."},{"metadata":{"trusted":true,"_uuid":"044627077e0693b813b95d2a4d451cfd8933a562"},"cell_type":"code","source":"plot_history(simple_cnn_model_history, 'CNN with 3 convolution layers')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1cd77351cd9349ee5130a80b02bb67b453a93018"},"cell_type":"markdown","source":"### Convolution Neural Network with Dropout layers\n\nSince the previous neural network had overfitting issues, I thought I'd use dropout layers to combat this. With dropout we essentially tell the network to randomly leave out a fraction of the nodes from updating the learned parameters. In each step we chose another set of nodes to leave out. \n\nHere, I've added dropout after each convolution layer to leave out 20% of nodes from updating in each step."},{"metadata":{"trusted":true,"_uuid":"815aac06281075a8db3b3d1fb0277a530b8e0446"},"cell_type":"code","source":"from keras.layers import Dropout\nsimple_cnn_model_2 = Sequential()\nsimple_cnn_model_2.add(Conv2D(32, (3,3), input_shape=(img_rows,img_cols,channels), activation='relu'))\nsimple_cnn_model_2.add(Dropout(0.2))\n\nsimple_cnn_model_2.add(Conv2D(32, (3,3), activation='relu'))\nsimple_cnn_model_2.add(Dropout(0.2))\n\nsimple_cnn_model_2.add(Conv2D(32, (3,3), activation='relu'))\nsimple_cnn_model_2.add(Dropout(0.2))\n\nsimple_cnn_model_2.add(Flatten())\nsimple_cnn_model_2.add(Dense(10, activation='softmax'))\n\nsimple_cnn_model_2.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\nsimple_cnn_model_2_history = simple_cnn_model_2.fit(train_x_reshaped, train_y, batch_size=100, epochs=8, validation_data=(test_x_reshaped, test_y))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e67e115254c36be40969d3207a0215f3a9aaccdd"},"cell_type":"markdown","source":"We can see that by adding dropout to the neural network, no overfitting seems to take place in such early stage and we can seemingly leave the training continue for a few more epochs. I suspect the accuracy to slowly increase at this rate. "},{"metadata":{"trusted":true,"_uuid":"67a6d7b9d57e91c730175fcb1e0f880bf4339986"},"cell_type":"code","source":"plot_history(simple_cnn_model_2_history, '3 convolution layers with dropout')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"92a10fba84c3a0eea13e97836c4e8a3f6ad597b7"},"cell_type":"code","source":"simple_cnn_model_3 = Sequential()\nsimple_cnn_model_3.add(Conv2D(64, (3,3), input_shape=(img_rows,img_cols,channels), activation='relu'))\nsimple_cnn_model_3.add(Dropout(0.2))\n\nsimple_cnn_model_3.add(Conv2D(64, (3,3), activation='relu'))\nsimple_cnn_model_3.add(Dropout(0.2))\n\nsimple_cnn_model_3.add(Conv2D(64, (3,3), activation='relu'))\nsimple_cnn_model_3.add(Dropout(0.2))\n\nsimple_cnn_model_3.add(Flatten())\nsimple_cnn_model_3.add(Dense(128, activation='relu'))\nsimple_cnn_model_3.add(Dense(10, activation='softmax'))\n\nsimple_cnn_model_3.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\nsimple_cnn_model_3_history = simple_cnn_model_3.fit(train_x_reshaped, train_y, batch_size=100, epochs=8, validation_data=(test_x_reshaped, test_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"30c7eb71b1f8a11877503cbd4e579cf7182a2399"},"cell_type":"code","source":"plot_history(simple_cnn_model_3_history, 'CNN with more layers and filters')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4672e4775365c194965676aa2c87f564025a4339"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}