{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","collapsed":true},"cell_type":"markdown","source":"# Drop the Dropout\nIn this kernel, we train an (existing) kernel that use Dropout, and in a second stage we suppress the Dropout and prevent overfitting by training only a thousand parameters. The base kernel we use is the excellent public one intitled \"[25 Millions Images! [0.99757] MNIST](https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist/notebook)\".\n\nBesides the code past from there, we create a second model where we remove dropout layers and introduce new layers to multiply activations from previous layers. This is done in keras using depthwise-pointwise convolution. While this method is natural for the first layers when we have 3D tensors with the CNNs, it is a little awkward for the Dense layer , since we need to reshape the 1D tensor into a 3D tensor.\n\nThe Dropout technique is very interesting to prevent overfitting, but it has the drawback that the model used to train is different from the model used to make inference.  Here we first train for 45 epochs 15 neural nets, then train for 20 epochs 15 new neural nets based on the previous ones, with \"multiplicative\" layers, all the old Convolutional and Dense layers set to not trainable, and without Dropout. Our rational is that Dropout is important/essential when we have millions of parameters, but not so when the model has less than two thousands trainable parameters. Then we use the 30 models built to make the inference.\n\nBelow we indicate \"(copy)\" when there is no modification from the code this notebook is based."},{"metadata":{"_uuid":"d69a400517648ced2b378cafdecd844d4f85bbc4"},"cell_type":"markdown","source":"# Load Libraries (copy)"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"code","source":"# LOAD LIBRARIES\nimport pandas as pd\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils.np_utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, BatchNormalization, DepthwiseConv2D, Reshape, Activation\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import LearningRateScheduler\nimport keras","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd31c62c12088bfa6f2b26dcecc714182627c767"},"cell_type":"markdown","source":"# Load Kaggle's 42,000 training images (copy)"},{"metadata":{"_uuid":"d71b3fa2b10620dc8870352fc18d4548f824a88a","trusted":false},"cell_type":"code","source":"# LOAD THE DATA\ntrain = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"b3c56055d1ba56d28d982f9647c33439c46753ff","trusted":false},"cell_type":"code","source":"# PREPARE DATA FOR NEURAL NETWORK\nY_train = train[\"label\"]\nX_train = train.drop(labels = [\"label\"],axis = 1)\nX_train = X_train / 255.0\nX_test = test / 255.0\nX_train = X_train.values.reshape(-1,28,28,1)\nX_test = X_test.values.reshape(-1,28,28,1)\nY_train = to_categorical(Y_train, num_classes = 10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cfcb89d7d2dab632986e80d9f68d194c3c1c9e9f"},"cell_type":"markdown","source":"# Generate 25 million more images!! (copy)\nby randomly rotating, scaling, and shifting Kaggle's 42,000 images."},{"metadata":{"_uuid":"3bde117b9f33a7442ce12d34891cc26189d325b3","trusted":false},"cell_type":"code","source":"# Fixing the seed\nnp.random.seed(82)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1e61e07d14b9b012748fdaac9eaf02e5263a475e","trusted":false},"cell_type":"code","source":"# CREATE MORE IMAGES VIA DATA AUGMENTATION\ndatagen = ImageDataGenerator(\n        rotation_range=10,  \n        zoom_range = 0.10,  \n        width_shift_range=0.1, \n        height_shift_range=0.1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9ea116cd3688cb26ac79b9fecc7309a1aebf3b63"},"cell_type":"markdown","source":"# Build 15 Convolutional Neural Networks! (copy)"},{"metadata":{"_uuid":"f6703f3f53c659e95579122755454899d842722a","trusted":false},"cell_type":"code","source":"# BUILD CONVOLUTIONAL NEURAL NETWORKS\nnets = 8\nmodel = [0] *nets\nfor j in range(nets):\n    model[j] = Sequential()\n    model[j].add(Conv2D(32, kernel_size = 3, activation='relu', input_shape = (28, 28, 1)))#0\n    model[j].add(BatchNormalization())#1\n    model[j].add(Conv2D(32, kernel_size = 3, activation='relu'))#2\n    model[j].add(BatchNormalization())#3\n    model[j].add(Conv2D(32, kernel_size = 5, strides=2, padding='same', activation='relu'))#4\n    model[j].add(BatchNormalization())#5\n    model[j].add(Dropout(0.4))#6\n    model[j].add(Conv2D(64, kernel_size = 3, activation='relu'))#7\n    model[j].add(BatchNormalization())#8\n    model[j].add(Conv2D(64, kernel_size = 3, activation='relu'))#9\n    model[j].add(BatchNormalization())#10\n    model[j].add(Conv2D(64, kernel_size = 5, strides=2, padding='same', activation='relu'))#11\n    model[j].add(BatchNormalization())#12\n    model[j].add(Dropout(0.4))#13\n    model[j].add(Conv2D(128, kernel_size = 4, activation='relu'))#14\n    model[j].add(BatchNormalization())#15\n    model[j].add(Flatten())#16\n    model[j].add(Dropout(0.4))#17\n    model[j].add(Dense(10, activation='softmax'))#18\n    # COMPILE WITH ADAM OPTIMIZER AND CROSS ENTROPY COST\n    model[j].compile(optimizer=\"adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"825fae667871bcf24f244afbce1e1458d9f5c9de"},"cell_type":"markdown","source":"# Build 15 Neural Networks without Dropout (new)\nWe remove all the Dropout Layers. After each convolutional layer, we add a depthwise-pointwise convolution (which will be initialized to one) that will multiply by learned parameters each previous features. So after a convolution layer with 32 features, the new layer will have 32 parameters. \nThese new layers commute with the ReLu activation, which is not the case for other types of activations. In those cases, we could separate the activations from the layers, and we could place depthwise-pointwise layers before and after the activations."},{"metadata":{"_uuid":"1d01f96295ad3cb17238eedf0a78243a156ec41c","trusted":false},"cell_type":"code","source":"model2 = [0] *nets\nfor j in range(nets):\n    model2[j] = Sequential()\n    model2[j].add(Conv2D(32, kernel_size = 3, activation='relu', input_shape = (28, 28, 1), trainable=False))#0\n    model2[j].add(BatchNormalization())#1\n    model2[j].add(DepthwiseConv2D((1,1), use_bias=False))#2\n    model2[j].add(Conv2D(32, kernel_size = 3, activation='relu', trainable=False))#3\n    model2[j].add(BatchNormalization())#4\n    model2[j].add(DepthwiseConv2D((1,1), use_bias=False))#5\n    model2[j].add(Conv2D(32, kernel_size = 5, strides=2, padding='same', activation='relu', trainable=False))#6\n    model2[j].add(BatchNormalization())#7\n    model2[j].add(DepthwiseConv2D((1,1), use_bias=False))#8\n    model2[j].add(Conv2D(64, kernel_size = 3, activation='relu', trainable=False))#9\n    model2[j].add(BatchNormalization())#10\n    model2[j].add(DepthwiseConv2D((1,1), use_bias=False))#11\n    model2[j].add(Conv2D(64, kernel_size = 3, activation='relu', trainable=False))#12\n    model2[j].add(BatchNormalization())#13\n    model2[j].add(DepthwiseConv2D((1,1), use_bias=False))#14\n    model2[j].add(Conv2D(64, kernel_size = 5, strides=2, padding='same', activation='relu', trainable=False))#15\n    model2[j].add(BatchNormalization())#16\n    model2[j].add(DepthwiseConv2D((1,1), use_bias=False))#17\n    model2[j].add(Conv2D(128, kernel_size = 4, activation='relu', trainable=False))#18\n    model2[j].add(BatchNormalization())#19\n    model2[j].add(DepthwiseConv2D((1,1), use_bias=False))#20\n    model2[j].add(Flatten())#21\n    model2[j].add(Dense(10, trainable=False))#22\n    model2[j].add(Reshape((1,1,10)))#23\n    model2[j].add(DepthwiseConv2D((1,1), use_bias=False))#24\n    model2[j].add(Flatten())#25\n    model2[j].add(Activation(\"softmax\"))#26\n    # COMPILE WITH ADAM OPTIMIZER AND CROSS ENTROPY COST\n    model2[j].compile(optimizer=keras.optimizers.Adadelta(), loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5182834d86a5181c291bce74a2b9966d980922d4"},"cell_type":"markdown","source":"# A function to init the weights of the new Models \nThe layers similar to the previous ones are initialized with the weights of the previous models, the depthwise-pointwise layers are initalized to one."},{"metadata":{"_uuid":"c45b575f901045953c714961c975ef6c12a14755","trusted":false},"cell_type":"code","source":"def initModel2(j):\n    model2[j].get_layer(index=0).set_weights(model[j].get_layer(index=0).get_weights())\n    model2[j].get_layer(index=1).set_weights(model[j].get_layer(index=1).get_weights())#batchnorm1\n    model2[j].get_layer(index=3).set_weights(model[j].get_layer(index=2).get_weights())\n    model2[j].get_layer(index=4).set_weights(model[j].get_layer(index=3).get_weights())#batchnorm4\n    model2[j].get_layer(index=6).set_weights(model[j].get_layer(index=4).get_weights())\n    model2[j].get_layer(index=7).set_weights(model[j].get_layer(index=5).get_weights())#batchnorm7\n    model2[j].get_layer(index=9).set_weights(model[j].get_layer(index=7).get_weights())\n    model2[j].get_layer(index=10).set_weights(model[j].get_layer(index=8).get_weights())#batchnorm10\n    model2[j].get_layer(index=12).set_weights(model[j].get_layer(index=9).get_weights())\n    model2[j].get_layer(index=13).set_weights(model[j].get_layer(index=10).get_weights())#batchnorm13\n    model2[j].get_layer(index=15).set_weights(model[j].get_layer(index=11).get_weights())\n    model2[j].get_layer(index=16).set_weights(model[j].get_layer(index=12).get_weights())#batchnorm16\n    model2[j].get_layer(index=18).set_weights(model[j].get_layer(index=14).get_weights())\n    model2[j].get_layer(index=19).set_weights(model[j].get_layer(index=15).get_weights())#batchnorm19\n    model2[j].get_layer(index=22).set_weights(model[j].get_layer(index=18).get_weights())\n    a=np.ones((1,1,1,32,1))\n    model2[j].get_layer(index=2).set_weights(a)\n    a=np.ones((1,1,1,32,1))\n    model2[j].get_layer(index=5).set_weights(a)\n    a=np.ones((1,1,1,32,1))\n    model2[j].get_layer(index=8).set_weights(a)\n    a=np.ones((1,1,1,64,1))\n    model2[j].get_layer(index=11).set_weights(a)\n    a=np.ones((1,1,1,64,1))\n    model2[j].get_layer(index=14).set_weights(a)\n    a=np.ones((1,1,1,64,1))\n    model2[j].get_layer(index=17).set_weights(a)\n    a=np.ones((1,1,1,128,1))\n    model2[j].get_layer(index=20).set_weights(a)\n    a=np.ones((1,1,1,10,1))\n    model2[j].get_layer(index=24).set_weights(a)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2b04bdd1651526a358071b2e49ffdf793af3aaf3","trusted":false},"cell_type":"code","source":"initModel2(0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e433661c7762b947c0fbfc4ad3f5e1d2e056312c"},"cell_type":"markdown","source":"# Train 15+15 CNNs (modified to add the training of the new models)"},{"metadata":{"_uuid":"9f1dd8a54aa0fab8530c0095f7d4c4b35984ea6d","collapsed":true,"trusted":false},"cell_type":"code","source":"# DECREASE LEARNING RATE EACH EPOCH\nannealer = LearningRateScheduler(lambda x: 1e-3 * 0.95 ** x)\n# TRAIN NETWORKS\nhistory = [0] * nets\nhistory2 = [0] * nets\nepochs = 45\nfor j in range(nets):\n    X_train2, X_val2, Y_train2, Y_val2 = train_test_split(X_train, Y_train, test_size = 0.1)\n    history[j] = model[j].fit_generator(datagen.flow(X_train2,Y_train2, batch_size=64),\n        epochs = epochs, steps_per_epoch = X_train2.shape[0]//64,  \n        validation_data = (X_val2,Y_val2), callbacks=[annealer], verbose=0)\n    print(\"CNN {0:d}: Epochs={1:d}, Train accuracy={2:.5f}, Validation accuracy={3:.5f}\".format(\n        j+1,epochs,max(history[j].history['acc']),max(history[j].history['val_acc']) ))\n    initModel2(j)\n    history2[j] = model2[j].fit_generator(datagen.flow(X_train2,Y_train2, batch_size=64),\n        epochs = 5, steps_per_epoch = X_train2.shape[0]//64,  \n        validation_data = (X_val2,Y_val2), verbose=0)\n    print(\"CNN {0:d}: Epochs={1:d}, Train accuracy={2:.5f}, Validation accuracy={3:.5f}\".format(\n        j+1,epochs,max(history2[j].history['acc']),max(history2[j].history['val_acc']) ))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"28b78b6502d2d1c993555725383f3e30728fa5be"},"cell_type":"markdown","source":"# Ensemble CNN predictions"},{"metadata":{"_uuid":"6e4e01ffe692c34c555bdbf5d606611f9a128b9c","collapsed":true,"trusted":false},"cell_type":"code","source":"# ENSEMBLE PREDICTIONS AND SUBMIT\nresults = np.zeros( (X_test.shape[0],10) ) \nfor j in range(nets):\n    results = results + model[j].predict(X_test)\nresultsa = np.argmax(results,axis = 1)\nresultsa = pd.Series(resultsa,name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),resultsa],axis = 1)\nsubmission.to_csv(\"MNIST-CNN-ENSEMBLE1.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"367639589d6c686a0114e1b81f31d8e4bbd8dda6"},"cell_type":"code","source":"results2 = np.zeros( (X_test.shape[0],10) ) \nfor j in range(nets):\n    results2 = results2 + model2[j].predict(X_test)\nresultsa = np.argmax(results2,axis = 1)\nresultsa = pd.Series(resultsa,name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),resultsa],axis = 1)\nsubmission.to_csv(\"MNIST-CNN-ENSEMBLE2.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"172f2d947c8d6e194f87606e88bf0e93d0294728"},"cell_type":"code","source":"res=results+results2\nresultsa = np.argmax(res,axis = 1)\nresultsa = pd.Series(resultsa,name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),resultsa],axis = 1)\nsubmission.to_csv(\"MNIST-CNN-ENSEMBLE3.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1d55f8c054432beabeed62024a998234c7cdb7b6"},"cell_type":"markdown","source":"# Credits\nAs already said, the code is based on \"[25 Millions Images! [0.99757] MNIST](https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist/notebook)\", in which credits was given to :\n\n* [Yassine Ghouzam][1] - [Introduction to CNN Keras - 0.997 (top 6%)][2]\n* [Peter Grenholm][5] - [Welcome to deep learning (CNN 99%)][6]\n* [Ding Li][3] - [Digits Recognition With CNN Keras][4]\n* [Aditya Soni][7] - [MNIST with Keras for Beginners(.99457)][8]\n\n[1]:https://www.kaggle.com/yassineghouzam\n[2]:https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6\n[3]:https://www.kaggle.com/dingli\n[4]:https://www.kaggle.com/dingli/digits-recognition-with-cnn-keras\n[5]:https://www.kaggle.com/toregil\n[6]:https://www.kaggle.com/toregil/welcome-to-deep-learning-cnn-99/\n[8]:https://www.kaggle.com/adityaecdrid/mnist-with-keras-for-beginners-99457/\n[7]:https://www.kaggle.com/adityaecdrid"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}