{"cells":[{"metadata":{"_uuid":"724469e0f265ace6818c5413ad9b046edf534c6f"},"cell_type":"markdown","source":"# LeNet-5 CNN with Keras:\n** I am developing a series of kernels for different Deep Learning Models: **\n\n* [L-Layered Neural Network from scratch](https://www.kaggle.com/curiousprogrammer/l-layered-neural-network-from-scratch)\n* [TensorFlow NN with Augmentation](https://www.kaggle.com/curiousprogrammer/digit-recognizer-tensorflow-nn-with-augmentation)\n* [Data Augmentation in Python, TF, Keras, Imgaug](https://www.kaggle.com/curiousprogrammer/data-augmentation-in-python-tf-keras-imgaug)\n* [Deep NN with Keras](https://www.kaggle.com/curiousprogrammer/deep-nn-with-keras-97-5) \n* [CNN with TensorFlow](https://www.kaggle.com/curiousprogrammer/lenet-5-cnn-with-tensorflow-98-5) \n* CNN with Keras - This one\n* AutoEncoders with TensorFlow\n* AutoEncoders with Keras\n* GANs with TensorFlow\n* GANs with Keras"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport keras \nfrom keras.models import Sequential\nfrom keras.layers import Conv2D, Dense, MaxPool2D, Dropout, Flatten\nfrom keras.optimizers import Adam\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\nfrom sklearn.model_selection import train_test_split\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"# Load Data"},{"metadata":{"trusted":true,"_uuid":"b1f812f18030149719ed56d54a4a91f46bdef8f0"},"cell_type":"code","source":"df_train = pd.read_csv('../input/train.csv')\nX_train = df_train.iloc[:, 1:]\nY_train = df_train.iloc[:, 0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fea9de32145e1f9f9e592e77f939c721a51a0c68"},"cell_type":"code","source":"X_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b5f7101b15f4884313291e9fa1f4dc8c5b53ba2e"},"cell_type":"code","source":"Y_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"446814df733cfa6d1af6b1c824ae07d506d4394a"},"cell_type":"code","source":"X_train = np.array(X_train)\nY_train = np.array(Y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cc3ea8d43dfde2a4cb7f1debf9dbeed9572a44dc"},"cell_type":"code","source":"# Normalize inputs\nX_train = X_train / 255.0","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"77329c63e08164e2b0bb11e34de7b14b5cd0cdef"},"cell_type":"markdown","source":"# Plot Digits"},{"metadata":{"trusted":true,"_uuid":"42f1019254336a51191b9881f3b7baa3566f8850"},"cell_type":"code","source":"def plot_digits(X, Y):\n    for i in range(20):\n        plt.subplot(5, 4, i+1)\n        plt.tight_layout()\n        plt.imshow(X[i].reshape(28, 28), cmap='gray')\n        plt.title('Digit:{}'.format(Y[i]))\n        plt.xticks([])\n        plt.yticks([])\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"565f266da9791e6bc4c50cfb2d1d617fcb3e5958"},"cell_type":"code","source":"plot_digits(X_train, Y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f520ed8fafb9fe8ac557d24bb0a4814c8b3ffd2b"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(8,8))\nsns.countplot(Y_train)\nax.set_title('Distribution of Digits', fontsize=14)\nax.set_xlabel('Digits', fontsize=12)\nax.set_ylabel('Count', fontsize=14)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fbb906464bc6be7872b8eca49e6cc4604941816c"},"cell_type":"code","source":"#Train-Test Split\nX_dev, X_val, Y_dev, Y_val = train_test_split(X_train, Y_train, test_size=0.03, shuffle=True, random_state=2019)\nT_dev = pd.get_dummies(Y_dev).values\nT_val = pd.get_dummies(Y_val).values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"549624ad1e62e1186eef71eb5e34533bc4d054b7"},"cell_type":"code","source":"#Reshape the input \nX_dev = X_dev.reshape(X_dev.shape[0], 28, 28, 1)\nX_val = X_val.reshape(X_val.shape[0], 28, 28, 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3ff1224e24fd7e67c6c6fffd7f7e17adfcc95e74"},"cell_type":"markdown","source":"# CNN Architecture\n\nWe will LeNet-5 CNN architeture to build our model.\n\n** LeNet - 5 Architecture: **\n\n![LeNet-5 Architecture](https://engmrk.com/wp-content/uploads/2018/09/LeNet_Original_Image.jpg)\n\n** Convolution Operation: **\n\n![Convolution Operation](https://www.researchgate.net/profile/Ihab_S_Mohamed/publication/324165524/figure/fig3/AS:611103423860736@1522709818959/An-example-of-convolution-operation-in-2D-2.png)\n\n### Input : Flattened 784px grayscale images, which can be represented as dimension (n, 28, 28, 1)\n### Output: 0 - 9 \n\n### Let's decode the operations we will be performing in each layer \n** First Layer:  Convolutional Layer (CONV1): **\n* Parameters: Input (N) = 28, Padding (P) = 2, Filter (F) = 5 x 5, Stride (S) = 1\n* Conv Operation: ((N + 2P - F) / S) + 1 = ((28 + 4 - 5) / 1) + 1 = 28 x 28 \n* We will apply 6 filters / kernels so we will get a 28 x 28 x 6 dimensional output\n\n** Second Layer:  Average Pooling Layer (POOL1): **\n* Parameters: Input (N) = 28, Filter (F) = 2 x 2, Stride (S) = 2\n* AVG Pooling Operation: ((N + 2P -F) / S) + 1 = ((28 - 2) / 2) + 1 = 14 x 14\n* We will have a 14 x 14 x 6 dimensional output at the end of this pooling\n\n** Third Layer:  Convolutional Layer (CONV2): **\n* Parameters: Input (N) = 14, Filter (F) = 5 x 5, Stride (S) = 1\n* Conv Operation: ((N + 2P - F) / S) + 1 = ((14 - 5) / 1) + 1 = 10 x 10\n* We will apply 16 filters / kernels so we will get a 10 x 10 x 16 dimensional output \n\n** Fourth Layer: Average Pooling Layer (POOL2): **\n* Parameters: Input (N) = 10, Filter (F) = 2 x 2, Stride (S) = 2\n* AVG Pooling Operation: ((N + 2P -F) / S) + 1 = ((10 - 2) / 2) + 1 = 5 x 5\n* We will have a 5 x 5 x 16 dimensional output at the end of this pooling\n\n** Fifth Layer: Fully Connected layer(FC1): **\n* Parameters: W: 400 * 120, b: 120\n* We will have an output of 120 x 1 dimension\n\n** Sixth Layer: Fully Connected layer(FC2): **\n* Parameters: W: 120 * 84, b: 84\n* We will have an output of 84 x 1 dimension\n\n** Seventh Layer: Output layer(Softmax): **\n* Parameters: W: 84 * 10, b: 10\n* We will get an output of 10 x 1 dimension\n\nWe will tweak the pooling layers from average to max and activation functions. With this architecture as per book, I was not able to achieve accuracy > 98.5%. Let's imcrease the filters and check."},{"metadata":{"trusted":true,"_uuid":"39f4eb788b9d8d1fd6d90bf8207aef93e0bf97a7"},"cell_type":"code","source":"model = Sequential()\nmodel.add(Conv2D(filters=32, kernel_size=(5,5), padding='same', activation='relu', input_shape=(28, 28, 1)))\nmodel.add(MaxPool2D(strides=2))\nmodel.add(Conv2D(filters=48, kernel_size=(5,5), padding='valid', activation='relu'))\nmodel.add(MaxPool2D(strides=2))\nmodel.add(Flatten())\nmodel.add(Dense(256, activation='relu'))\nmodel.add(Dense(84, activation='relu'))\nmodel.add(Dense(10, activation='softmax'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5fd1c9e9fbb72c4b56b1e960821cf5d2b6c8c618"},"cell_type":"code","source":"model.build()\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d34ba8b4723050c33b3390d2a05220636c2f0f8a"},"cell_type":"code","source":"adam = Adam(lr=5e-4)\nmodel.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer=adam)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f0dfd9488061194f9ef01a57d62f10e0de8fa028"},"cell_type":"code","source":"# Set a learning rate annealer\nreduce_lr = ReduceLROnPlateau(monitor='val_acc', \n                                patience=3, \n                                verbose=1, \n                                factor=0.2, \n                                min_lr=1e-6)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e1e99d749bd0ab368b73884d761055491a0e3153"},"cell_type":"code","source":"# Data Augmentation\ndatagen = ImageDataGenerator(\n            rotation_range=10, \n            width_shift_range=0.1, \n            height_shift_range=0.1, \n            zoom_range=0.1)\ndatagen.fit(X_dev)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b6441c7f0b7ad33d1e3ca91b3d82f91a28b6f42a"},"cell_type":"code","source":"model.fit_generator(datagen.flow(X_dev, T_dev, batch_size=100), steps_per_epoch=len(X_dev)/100, \n                    epochs=30, validation_data=(X_val, T_val), callbacks=[reduce_lr])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"12a3ce31e8a5084b53979cba8e4d901a9354162b"},"cell_type":"code","source":"score = model.evaluate(X_val, T_val, batch_size=32)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c106f7458a98728316239490047772da57063300"},"cell_type":"code","source":"score","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"53f5237ef36125cc68508a0158fef0fda1a5b53e"},"cell_type":"markdown","source":"# Let's predict test data"},{"metadata":{"trusted":true,"_uuid":"ae3743fcad78547af06fd7f07a423a9c3903cd70"},"cell_type":"code","source":"df_test = pd.read_csv('../input/test.csv')\nX_test = np.array(df_test)\nX_test = X_test/255.0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"065f63d7fe1610827e0a88473c9c4e7504fddcd6"},"cell_type":"code","source":"X_test = X_test.reshape((X_test.shape[0], 28, 28, 1))\nY_test = model.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d6cdaea6d12289142fa021de35eb1db88cda4591"},"cell_type":"code","source":"Y_test = np.argmax(Y_test, axis=1)\nY_test[:5]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"02b0637a6557e16144538c4970889f9c00dcac59"},"cell_type":"markdown","source":"# Create submission file"},{"metadata":{"trusted":true,"_uuid":"df2bddece73c5cb68ce6d63d3ec2be7e6882b02d"},"cell_type":"code","source":"df_out = pd.read_csv('../input/sample_submission.csv')\ndf_out['Label'] = Y_test\ndf_out.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2689782f0dda65d13046cd5859c82c1a4b6ccc4d"},"cell_type":"code","source":"df_out.to_csv('out.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0ad1c947ccf9e9501c019822039281558b231751"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}