{"cells":[{"metadata":{"_uuid":"10a4de1073ff75be867045984c3b4448eb5c25ef"},"cell_type":"markdown","source":"# LeNet-5 CNN with TensorFlow:\n** I am developing a series of kernels for different Deep Learning Models: **\n\n* [L-Layered Neural Network from scratch](https://www.kaggle.com/curiousprogrammer/l-layered-neural-network-from-scratch)\n* [TensorFlow NN with Augmentation](https://www.kaggle.com/curiousprogrammer/digit-recognizer-tensorflow-nn-with-augmentation)\n* [Data Augmentation in Python, TF, Keras, Imgaug](https://www.kaggle.com/curiousprogrammer/data-augmentation-in-python-tf-keras-imgaug)\n* [Deep NN with Keras](https://www.kaggle.com/curiousprogrammer/deep-nn-with-keras-97-5) \n* CNN with TensorFlow - This one\n* CNN with Keras\n* AutoEncoders with TensorFlow\n* AutoEncoders with Keras\n* GANs with TensorFlow\n* GANs with Keras"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport tensorflow as tf\nfrom sklearn.model_selection import train_test_split\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bf87adf722be15820e8c6b8d2f743bc94feac6d0"},"cell_type":"markdown","source":"# Load Data"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"df_train = pd.read_csv('../input/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a2c4075b60bbb2d461573245ef7af592daa66b99"},"cell_type":"code","source":"X_train = df_train.iloc[:, 1:]\nY_train = df_train.iloc[:, 0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"259909af85a28463f3d9bac505c67e16de606aa1"},"cell_type":"code","source":"X_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f66374ce852848c4702c72bed472543241d5d6ca"},"cell_type":"code","source":"Y_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4743abde907b4f5251076a3c4a2c5a120f6fd742"},"cell_type":"markdown","source":"# Normalization"},{"metadata":{"trusted":true,"_uuid":"71605890497fcdd311330689af76085067f436e9"},"cell_type":"code","source":"X_train = np.array(X_train)\nY_train = np.array(Y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bfb684590770a4670094758c74e5705600e2aa64"},"cell_type":"code","source":"X_train = X_train/255.0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8be9ae400b3bde99212186c7cfe9b8eaaf047903"},"cell_type":"code","source":"# dev-val split\nX_dev, X_val, Y_dev, Y_val = train_test_split(X_train, Y_train, test_size=0.03, shuffle=True, random_state=2019)\n\n#Reshape the arrays to match the input in tensorflow graph\nX_dev = X_dev.reshape((X_dev.shape[0], 28, 28, 1))\nX_val = X_val.reshape((X_val.shape[0], 28, 28, 1))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2738ac5a875d41cc0bc4e790494c3c1c085b61d3"},"cell_type":"markdown","source":"# Plot digits"},{"metadata":{"trusted":true,"_uuid":"55913e345ce75d69e3d39da5ecf597bf2384b4a1"},"cell_type":"code","source":"def plot_digits(X, Y):\n    for i in range(20):\n        plt.subplot(4, 5, i+1)\n        plt.tight_layout()\n        plt.imshow(X[i].reshape((28, 28)), cmap='gray')\n        plt.title('Digit:{}'.format(Y[i]))\n        plt.xticks([])\n        plt.yticks([])\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fea6f518a69b1be09af28bbfdbb2370f0702a6c9"},"cell_type":"code","source":"plot_digits(X_train[-20:], Y_train[-20:])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d1a5d36cbad2598e4c29908cb2ed43befe2b06c"},"cell_type":"markdown","source":"# CNN Architecture\n\nWe will LeNet-5 CNN architeture to build our model.\n\n** LeNet - 5 Architecture: **\n\n![LeNet-5 Architecture](https://engmrk.com/wp-content/uploads/2018/09/LeNet_Original_Image.jpg)\n\n** Convolution Operation: **\n\n![Convolution Operation](https://www.researchgate.net/profile/Ihab_S_Mohamed/publication/324165524/figure/fig3/AS:611103423860736@1522709818959/An-example-of-convolution-operation-in-2D-2.png)\n\n### Input : Flattened 784px grayscale images, which can be represented as dimension (n, 28, 28, 1)\n### Output: 0 - 9 \n\n### Let's decode the operations we will be performing in each layer \n** First Layer:  Convolutional Layer (CONV1): **\n* Parameters: Input (N) = 28, Padding (P) = 2, Filter (F) = 5 x 5, Stride (S) = 1\n* Conv Operation: ((N + 2P - F) / S) + 1 = ((28 + 4 - 5) / 1) + 1 = 28 x 28 \n* We will apply 6 filters / kernels so we will get a 28 x 28 x 6 dimensional output\n\n** Second Layer:  Average Pooling Layer (POOL1): **\n* Parameters: Input (N) = 28, Filter (F) = 2 x 2, Stride (S) = 2\n* AVG Pooling Operation: ((N + 2P -F) / S) + 1 = ((28 - 2) / 2) + 1 = 14 x 14\n* We will have a 14 x 14 x 6 dimensional output at the end of this pooling\n\n** Third Layer:  Convolutional Layer (CONV2): **\n* Parameters: Input (N) = 14, Filter (F) = 5 x 5, Stride (S) = 1\n* Conv Operation: ((N + 2P - F) / S) + 1 = ((14 - 5) / 1) + 1 = 10 x 10\n* We will apply 16 filters / kernels so we will get a 10 x 10 x 6 dimensional output \n\n** Fourth Layer: Average Pooling Layer (POOL2): **\n* Parameters: Input (N) = 10, Filter (F) = 2 x 2, Stride (S) = 2\n* AVG Pooling Operation: ((N + 2P -F) / S) + 1 = ((10 - 2) / 2) + 1 = 5 x 5\n* We will have a 5 x 5 x 16 dimensional output at the end of this pooling\n\n** Fifth Layer: Fully Connected layer(FC1): **\n* Parameters: W: 400 * 120, b: 120\n* We will have an output of 120 x 1 dimension\n\n** Sixth Layer: Fully Connected layer(FC2): **\n* Parameters: W: 120 * 84, b: 84\n* We will have an output of 84 x 1 dimension\n\n** Seventh Layer: Output layer(Softmax): **\n* Parameters: W: 84 * 10, b: 10\n* We will get an output of 10 x 1 dimension\n\nWe will tweak the pooling layers from average to max and activation functions."},{"metadata":{"_uuid":"0c3dde6f12cac030bd1f6e3661b80f1ea67cdf29"},"cell_type":"markdown","source":"# Input placeholder"},{"metadata":{"trusted":true,"_uuid":"c80f89f0415bbb334c645450a13eb0daea03b4e7"},"cell_type":"code","source":"x = tf.placeholder(dtype=tf.float32, shape=(None, 28, 28, 1), name='X')\ny = tf.placeholder(dtype=tf.float32, shape=(None, 10), name='Y')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"213e15a42d14955ad8bb3713ea0040e6ac4c6299"},"cell_type":"markdown","source":"# Graph Creation - TF Layers"},{"metadata":{"trusted":true,"_uuid":"2f620bd24182968c59bc2763325b0c9556662734"},"cell_type":"code","source":"conv1 = tf.layers.conv2d(x, filters=6, kernel_size=5, padding='same', strides=1, activation='relu', name='CONV1')\npool1 = tf.layers.max_pooling2d(conv1, pool_size=2, strides=2, name='POOL1')\nconv2 = tf.layers.conv2d(pool1, filters=16, kernel_size=5, strides=1, activation='relu', name='CONV2')\npool2 = tf.layers.max_pooling2d(conv2, pool_size=2, strides=2, name='POOL2')\nflatten1 = tf.layers.Flatten()(pool2)\nfc1 = tf.layers.Dense(120, activation='relu')(flatten1)\nfc2 = tf.layers.Dense(84, activation='relu')(fc1)\nout = tf.layers.Dense(10, activation='softmax')(fc2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"da5887b3891f97735600ba574da23aeeb61fc43f"},"cell_type":"markdown","source":"# Hyperparameters"},{"metadata":{"trusted":true,"_uuid":"c3d4c5bd90703e24b96a85bf4db9bc2b4ab981c0"},"cell_type":"code","source":"batch_size = 100\nlearning_rate = 5e-4\nepochs = 20","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a6d3637d27905eddac5168519702481af6ef0a54"},"cell_type":"markdown","source":"# Cost, Accuracy, Optimizer"},{"metadata":{"trusted":true,"_uuid":"60e4cf018b291049150bed706eac53cf0c81265c"},"cell_type":"code","source":"cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=y, logits=out, name='cost'))\nopt = tf.train.AdamOptimizer(learning_rate).minimize(cost)\nequal_pred = tf.equal(tf.argmax(y, 1), tf.argmax(out, 1))\nacc = tf.reduce_mean(tf.cast(equal_pred, tf.float32))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c9e6c09682891fc197f5f9e1a940b2f7dfca0120"},"cell_type":"markdown","source":"# Initialise the variables"},{"metadata":{"trusted":true,"_uuid":"a6d34230873c9051e981c23920896ab80953333a"},"cell_type":"code","source":"init = tf.global_variables_initializer()\nsess = tf.Session()\nsess.run(init)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"174e4dd75333533d0a5a195916eec6fa8632e1e1"},"cell_type":"markdown","source":"# Training the Model"},{"metadata":{"trusted":true,"_uuid":"e2c8fa7cec8ec291f85857612c99049a0cd99296","scrolled":false},"cell_type":"code","source":"T_dev = pd.get_dummies(Y_dev).values\nT_val = pd.get_dummies(Y_val).values\nfor epoch in range(epochs):\n    start_index = 0\n    s = np.arange(X_dev.shape[0])\n    np.random.shuffle(s)\n    X_dev = X_dev[s, :]\n    T_dev = T_dev[s]\n    while start_index < X_dev.shape[0]:\n        end_index = start_index + batch_size\n        if end_index > X_dev.shape[0]:\n            end_index = X_dev.shape[0]\n        x_dev = X_dev[start_index:end_index, :]\n        t_dev = T_dev[start_index:end_index]\n        dev_cost, dev_acc, _ = sess.run([cost, acc, opt], feed_dict={x:x_dev, y:t_dev})\n        start_index = end_index\n    dev_cost, dev_acc = sess.run([cost, acc], feed_dict={x:X_dev, y:T_dev})\n    val_cost, val_acc = sess.run([cost, acc], feed_dict={x:X_val, y:T_val})\n    print('Epoch:{0} Cost:{1:5f} Acc:{2:.5f} Val_Cost:{3:5f} Val_Accuracy:{4:.5f}'.\n          format(epoch+1, dev_cost, dev_acc, val_cost, val_acc))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a437f30a3cedbec1b0594215d491ed1ce5e28b88"},"cell_type":"code","source":"X_test = pd.read_csv('../input/test.csv')\nX_test = np.array(X_test)\nX_test = X_test / 255.0\nX_test = X_test.reshape((X_test.shape[0], 28, 28, 1))\nT_test = sess.run([out], feed_dict={x:X_test})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f19d85c6435bcbd7d3db65f4fc237e3cad23ddf9"},"cell_type":"code","source":"Y_test = np.argmax(T_test[0], axis=1)\nY_test[:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9d3651a3e023bcd309202c8c62335e5281144623"},"cell_type":"code","source":"df_out = pd.read_csv('../input/sample_submission.csv')\ndf_out['Label'] = Y_test\ndf_out.to_csv('out.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"54eb2c2f194c63571b943f44bec8a71515b09380"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}