{"cells":[{"metadata":{"_uuid":"596c01624ea116b4dd8b56c7956b24142cbbc6e9","_cell_guid":"94475abd-a914-4b16-be4c-eb1a7a929775"},"cell_type":"markdown","source":"## A Fast and Easy CNN Kernel in Keras for Beginners\n\nThe aim of this kernel is to provide beginners (including me) a fast and easy platform to build a Convolutional Neural Network (CNN) model. This  notebook is focused on the coding side. For detailed description, please visit my web site at http://www.codeastar.com/convolutional-neural-network-python/ ."},{"metadata":{"_uuid":"640c26b21cc9aefcdadb9089697eae84ec3d7b51","_cell_guid":"74bf4287-26c6-453e-833d-c8fc3813ce5a"},"cell_type":"markdown","source":"First thing first, let's load training and testing data sets from Kaggle. "},{"metadata":{"_uuid":"23d66211319996bc2703b131e2e3f74fc64e67cf","collapsed":true,"_cell_guid":"c70a3347-f9c8-44ca-b217-0233fb962ac7","trusted":false},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\ndf_train = pd.read_csv('../input/train.csv')\ndf_test = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ceb6a84a5387c1bb69b7618140d7663282007065","_cell_guid":"856e7053-ce35-49ed-9d0c-5030b3bb6d96"},"cell_type":"markdown","source":"Then import required modules. "},{"metadata":{"_uuid":"b54f5b1f14ee438dc1c60e231cc43bfe553e8aa1","_cell_guid":"cb0575da-4177-4046-b80d-0649ed67da7d","trusted":false,"collapsed":true},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten\nfrom keras.layers.convolutional import Conv2D, MaxPooling2D\nfrom keras.utils import np_utils\nfrom keras.optimizers import Adam\nfrom keras.callbacks import ReduceLROnPlateau\nfrom keras.preprocessing.image import ImageDataGenerator\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8240449bfa3deb032c719b818464742c72451333","_cell_guid":"f0bea6e7-0a1a-49e3-a323-4e02986a2620"},"cell_type":"markdown","source":"Have a quick look on our training data."},{"metadata":{"_uuid":"0bd801f790a6cef333c0a52b7bede3dfefe38f2d","_cell_guid":"b79af2ce-e5a2-4f89-9ecf-cb310866573b","trusted":false,"collapsed":true},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4feaf5def18f5918aa00a344ad56d1d7496fa91e","_cell_guid":"4606d1f4-23b1-46a2-b758-4240411e1ea3"},"cell_type":"markdown","source":"We load the 2nd to 785th columns, the pixel value columns, as our input X.\nAnd load the first label column as output Y. "},{"metadata":{"_uuid":"a13cc8cb331e297fbda3c4e413550f188330713b","collapsed":true,"_cell_guid":"4b8ff14e-74dc-462e-b164-e087dc2a520a","trusted":false},"cell_type":"code","source":"#every columns but the first\ndf_train_x = df_train.iloc[:,1:] \n#only the first column\ndf_train_y = df_train.iloc[:,:1] ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f650d28a76e907b0388c8de4a8b1230e33b94e7d","_cell_guid":"0d078c6a-562a-4aff-bc38-2ebc7c1d7772"},"cell_type":"markdown","source":"We can use *imshow()* from *matplotlab* to display the our input X as images. See rather those images can map with the values of output Y. "},{"metadata":{"_uuid":"d0143d564c1e723dc2a36f022a72e578696340d6","_cell_guid":"45e11072-3854-4ea9-a862-df0629060cf6","trusted":false,"collapsed":true},"cell_type":"code","source":"ax = plt.subplots(1,5)\nfor i in range(0,5):   #validate the first 5 records\n    ax[1][i].imshow(df_train_x.values[i].reshape(28,28), cmap='gray')\n    ax[1][i].set_title(df_train_y.values[i])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f3cbfa93fa86efe90e9460061c95de5917167871","_cell_guid":"c130034d-9595-421c-ad34-87df1e40c0eb"},"cell_type":"markdown","source":"### It's building time. \nWe use Keras to build our CNN model, with typical LeNet architecture (http://yann.lecun.com/exdb/lenet/).  But unlike other popular Keras kernels on Kaggle, I do not apply the stacking convolutional layers here. As I find this is just unnecssary to spend that much resources on simple digit image sets. "},{"metadata":{"_uuid":"ce53b4ffdca0eb8dbc8c5a16a849c53d28e1b0b6","collapsed":true,"_cell_guid":"d0fc68f8-9cfa-4df1-a775-07fea07b5c22","trusted":false},"cell_type":"code","source":"def cnn_model(result_class_size):\n    model = Sequential()\n    #use Conv2D to create our first convolutional layer, with 32 filters, 5x5 filter size, \n    #input_shape = input image with (height, width, channels), activate ReLU to turn negative to zero\n    model.add(Conv2D(32, (3, 3), input_shape=(28,28,1), activation='relu', padding=\"valid\"))\n    #add a pooling layer for down sampling\n    model.add(MaxPooling2D(pool_size=(3, 3)))\n    # add another conv layer with 16 filters, 3x3 filter size, \n    model.add(Conv2D(16, (2, 2), activation='relu'))\n    #set 20% of the layer's activation to zero, to void overfit\n    model.add(Dropout(0.2))\n    #convert a 2D matrix in a vector\n    model.add(Flatten())\n    #add fully-connected layers, and ReLU activation\n    model.add(Dense(128, activation='relu'))\n    model.add(Dense(128, activation='relu'))\n    #add a fully-connected layer with softmax function to squash values to 0...1 \n    model.add(Dense(result_class_size, activation='softmax'))   \n    #model.compile(loss='categorical_crossentropy', optimizer=RMSprop(), metrics=['accuracy'])\n    adamOPT = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-8, decay=0.0, amsgrad=False)\n    model.compile(loss='categorical_crossentropy', optimizer=adamOPT, metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ce4223c6eddcc5759b96bc77b2624ce06176ec37","_cell_guid":"1f13eea1-f092-4664-aa1d-dd07c1fc580a"},"cell_type":"markdown","source":"Model is done, let's take a look on our model summary. "},{"metadata":{"_uuid":"8b8bbf206eb8d146ad2f02bf52a121d6f0992240","_cell_guid":"7ec2d8b1-bfe4-4b00-a3db-cd3301cd9116","trusted":false,"collapsed":true},"cell_type":"code","source":"#turn the label to 42000 binary class matrix \narr_train_y = np_utils.to_categorical(df_train_y['label'].values)\nmodel = cnn_model(arr_train_y.shape[1])\nmodel.summary()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f172399666d3b91d63aa5311989509e0fdf2dff8","_cell_guid":"a0c149f4-fd43-4027-8c36-6134281eea5d"},"cell_type":"markdown","source":"When there is more trainable params, more time is needed. And in the world of machine learning and optimization, more parameters may not always provide better result, as it may cause an overfitting issue. Anyway, we are good at current status."},{"metadata":{"_uuid":"2e111d8ccf3fea71cc4bad5bbf96bba9d317fb0d","collapsed":true,"_cell_guid":"222d9cc1-2b7b-424c-ad96-ca0dd4b18427","trusted":false},"cell_type":"code","source":"#normalize 255 grey scale to values between 0 and 1 \ndf_test = df_test / 255\ndf_train_x = df_train_x / 255","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ae7e4145ab6f9e93a5e7d4c9de199b8d1549dc12","_cell_guid":"b44f0e1b-153a-453d-a25b-fea5bdd41d4e"},"cell_type":"markdown","source":"Since our model uses 28x28 pixel matrix as input, we then need to reshape both our training and testing input Xs. "},{"metadata":{"_uuid":"0e9789f54c1dab101b3cf522cb7b9fb9909cf803","collapsed":true,"_cell_guid":"70f9cc76-aacf-491f-82e8-4166cd333b99","trusted":false},"cell_type":"code","source":"#reshape training X and text x to (number, height, width, channels)\narr_train_x_28x28 = np.reshape(df_train_x.values, (df_train_x.values.shape[0], 28, 28, 1))\narr_test_x_28x28 = np.reshape(df_test.values, (df_test.values.shape[0], 28, 28, 1))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f63e48119cbafbc884b2783c7aaf180832a81409","collapsed":true,"_cell_guid":"9d128991-416f-4621-ba32-af38d7679a8f","trusted":false},"cell_type":"code","source":"random_seed = 3\n#validate size = 5%\nsplit_train_x, split_val_x, split_train_y, split_val_y, = train_test_split(arr_train_x_28x28, arr_train_y, test_size = 0.05, random_state=0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"edde0f62db0e70d8d9b17f57524e96a3ca3eddbb","_cell_guid":"cc1cc253-2f25-47a4-94d6-51601d92642c"},"cell_type":"markdown","source":"print sizes"},{"metadata":{"_uuid":"cdece4d1485ef7c94750012d3291ae5803240108","_cell_guid":"caacae54-fccb-4805-992b-40b361748cdb","trusted":false,"collapsed":true},"cell_type":"code","source":"print(\"Train: \", split_train_x.shape)\nprint(\"Train label: \", split_train_y.shape)\nprint(\"Test: \", split_val_x.shape)\nprint(\"Test label: \", split_val_y.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"89eb7d877a2424cd74840aace82cbec5a87f5579","_cell_guid":"cb9bf7d8-7307-45cf-861c-580d1283a950"},"cell_type":"markdown","source":"A reduce learinging callback is prepared for later use. When there is no improvement in our model after **3** training rounds, the new learining rate will be calculated as \"current learning rate x factor( **0.5** )\" . "},{"metadata":{"_uuid":"ecb450840f6416baeac5ad578a84d07b5a604995","collapsed":true,"_cell_guid":"a8406f96-6b3e-4162-85ef-f9ddd1fbf36c","trusted":false},"cell_type":"code","source":"reduce_lr = ReduceLROnPlateau(monitor='val_acc', \n                              factor=0.5,\n                              patience=3, \n                              min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6074062dd9ca22fa89ac437c325193114c4e48c3","_cell_guid":"9ad867c2-d34e-4c93-af74-6a26fb012e1d"},"cell_type":"markdown","source":"We also apply the image generator function to cover more  variance of data distribution. "},{"metadata":{"_uuid":"759ce1cbb9cc5629ef3ca5581f35a4070c349e46","collapsed":true,"_cell_guid":"c6e07831-d736-44c3-918a-1d1169f10b89","trusted":false},"cell_type":"code","source":"datagen = ImageDataGenerator(\n        rotation_range=10,  # randomly rotate images in the range \n        zoom_range = 0.1, # Randomly zoom image \n        width_shift_range=0.1,  # randomly shift images horizontally\n        height_shift_range=0.1  # randomly shift images vertically\n        )\n\ndatagen.fit(split_train_x)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"19e608feb99312769d03b991cd0b934b69665013","_cell_guid":"cb65dc86-70ed-44b0-b65f-fa0ef2c20132"},"cell_type":"markdown","source":"After we add the image generator and learning rate callback to model, we can start training it in 30 epochs."},{"metadata":{"_uuid":"8c0f8fd658d637c66b601fa20701e83b16f5f9e7","scrolled":false,"_cell_guid":"24f5a61e-d038-468e-9f40-8be70563874a","trusted":false,"collapsed":true},"cell_type":"code","source":"model.fit_generator(datagen.flow(split_train_x, split_train_y, batch_size=1024),\n                              epochs = 30, validation_data = (split_val_x, split_val_y),\n                              verbose = 2, callbacks=[reduce_lr])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d0bcdf1e908f207d76327dd09f7a279c5392aeb3","_cell_guid":"8a2c760e-621e-4815-a231-e1e5c9670238"},"cell_type":"markdown","source":"It needs around 1200 seconds (20 minutes) to run the training process. "},{"metadata":{"_uuid":"02193192ff17a925515c8d8adf46c382cf8228ce","collapsed":true,"_cell_guid":"71a23a85-bad6-41eb-9ac9-999c5cd55ea6","trusted":false},"cell_type":"code","source":"prediction = model.predict_classes(arr_test_x_28x28, verbose=0)\ndata_to_submit = pd.DataFrame({\"ImageId\": list(range(1,len(prediction)+1)), \"Label\": prediction})\ndata_to_submit.to_csv(\"submission.csv\", header=True, index = False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f1c76748c0bb2ae4f0b9741ff409de443b1d0045","_cell_guid":"c8a43140-e231-4ec6-9c4d-900964f7e34f","trusted":false,"collapsed":true},"cell_type":"code","source":"data_to_submit.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa6de3cd124f489588f942b1a548b70db402fc75","_cell_guid":"6a46abe3-6af3-428f-8ebe-a9dd7006111d"},"cell_type":"markdown","source":"After a cup of coffee (a tea for myself, as I don't drink coffee usually), we can get the prediction, save it as a csv file and submit to Kaggle. But before sending out our submission, let' see how well our trained model is, by our own eyes."},{"metadata":{"_uuid":"23d61dea6c58bfdea0816dcfbba51b300323596f","collapsed":true,"_cell_guid":"60fe7957-0b19-4791-a18d-7b4691d92af5","trusted":false},"cell_type":"code","source":"from random import randrange\n#pick 10 images from testing data set\nstart_idx = randrange(df_test.shape[0]-10) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dea2ec826c9b38cd31765d2b0d4cea9350a7ece7","_cell_guid":"f0ef79e4-6cc3-4a42-b1e0-a607c95749e3","trusted":false,"collapsed":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,5, figsize=(15,8))\nfor j in range(0,2): \n  for i in range(0,5):\n     ax[j][i].imshow(df_test.values[start_idx].reshape(28,28), cmap='gray')\n     ax[j][i].set_title(\"Index:{} \\nPrediction:{}\".format(start_idx, prediction[start_idx]))\n     start_idx +=1","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}