{"cells": [{"source": ["# INTRODUCTION\n", "\n", "Hi, in this notebook, I'll try to solve MNIST dataset classification problem using a deep neural network using Tensorflow. I've tried many hyperparameters combination before, and decide using width of 2000 and depth of 5.\n", "\n", "Let's get started by importing necessary libraries"], "metadata": {"_uuid": "d219a2381d07bd1c45a36d912538c0ad8551ce86", "_cell_guid": "c105aaf6-72ea-464c-860a-9dd321a0c769"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "e6412a450d529a464d1d83a95141c57afc2963bb", "_cell_guid": "a024166f-4ee7-40f5-a291-2b31cbc84fc3"}, "outputs": [], "cell_type": "code", "source": ["import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "\n", "import warnings\n", "warnings.filterwarnings('ignore')\n", "\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.preprocessing import OneHotEncoder\n", "import tensorflow as tf"]}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "90c5d56622ebc9621dd60525186621685ee3ba1a", "_cell_guid": "cdea3762-33b2-467f-8d39-71d91af684a9"}, "outputs": [], "cell_type": "code", "source": ["df = pd.read_csv('../input/train.csv')\n", "test = pd.read_csv('../input/test.csv')"]}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "41ae69e05d47d8d679144ef09acd281dc14c5913", "_cell_guid": "e99557e9-2d02-49dc-a359-010c00eb99b1"}, "outputs": [], "cell_type": "code", "source": ["df.describe()"]}, {"source": ["From above information, we got some information:\n", "* there are 10 classes: 0,1,2,3,4,5,6,7,8,9\n", "* there aren't any missing values\n", "* there are 784 columns (784) pixels. sqrt(784) = 28.  **so the image shape is 28x28**\n", "* the pixels ranged from 0-255 (**we need to cenvert them to 0.0 - 1.0**)\n", "\n", "next, let's check first 3 images that we need to predict the labels"], "metadata": {"_uuid": "c48848db4a20ec622b784121d2e05ab920e6cb89", "_cell_guid": "ca26c3a3-226f-4a7a-97b5-075d8e3dfcfc"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "8ddddd02c376956a616112e1c0d82b1396d4e08d", "_cell_guid": "e3491aed-2b17-4023-937a-e7e231e9d208"}, "outputs": [], "cell_type": "code", "source": ["f,ax = plt.subplots(1,3,figsize=(15,6))\n", "ax[0].imshow(test.iloc[0].reshape(28,28),cmap='binary')\n", "ax[1].imshow(test.iloc[1].reshape(28,28),cmap='binary')\n", "ax[2].imshow(test.iloc[2].reshape(28,28),cmap='binary')"]}, {"source": ["The first 3 images show number 2, 0, and 9 respectively. We'll check our prediction result with them later, just to make sure we have a good prediction result.\n", "\n", "Okay, next I'll do this project in 6 parts:\n", "1. Data Preparation\n", "2. Model Preparation\n", "3. Objective Function Preparation\n", "4. Optimizer Preparation\n", "5. Learning\n", "6. Deployment"], "metadata": {"_uuid": "b48ba4f86f742566da8d0705fd5ba997c712d2ee", "_cell_guid": "d6068a70-26d4-400c-9f82-421053087e83"}, "cell_type": "markdown"}, {"source": ["**Part 1: Data Preparation**\n", "\n", "There are 10 classes, but they are represented in integer values (0-9) which is ordinal or showing some level (0<1<..<9). In order to solve that, I'll use One-Hot encoding. Here's the formula:"], "metadata": {"_uuid": "026872aa4ac04677f5ed938a265e534706622c40", "_cell_guid": "d89772e3-e6e1-45d3-8188-f26fed670aea"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "3ce88d026cd38c2bafbee17f3ecf76976ca2a765", "_cell_guid": "1faf4e8f-0ade-482a-80a6-41d4c24224d4"}, "outputs": [], "cell_type": "code", "source": ["#one-hot formula\n", "np.array([np.array([int(i == label) for i in range(10)]) for label in [5,2,3,9]])\n", "#basically generate a numpy array for all elements (5,2,and 3), then doing a loop of 10 loops\n", "#for 10 loops, do a check if current i == label, return 0 if false, 1 if true\n", "\n", "#this will create a one-hot encoded number"]}, {"source": ["Now, let's apply the formula on df['label']"], "metadata": {"_uuid": "0bdded5a53bd25f6741211da094b9dce6f7276d9", "_cell_guid": "8ed37d1c-06b4-4ccc-84b4-fa2d890d8c7e"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "6f866554d571bc9dd248b00433ddae4c5b08a412", "_cell_guid": "c0dc97a2-0ed4-4874-9326-a450a5be8549"}, "outputs": [], "cell_type": "code", "source": ["labels_encoded = np.array([np.array([int(i == label) for i in range(10)]) for label in df.iloc[:,0].values])"]}, {"source": ["For the inputs (the pixels), I need to change the values range from 0 - 255 to 0.0 - 1.0. Here's how I do it:\n", "\n", "*note: tensorflow only works with array, so I use arrays here instead of dataframes*"], "metadata": {"_uuid": "6c36b4bbb554ae610a252c04233464baa943090c", "_cell_guid": "6911093a-eda3-427f-95d9-dd39963f8bd2"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "18980db2a4635ca14f41757bb3c2aeca2ae2754c", "_cell_guid": "208ad951-e34e-4561-af02-df6b91cf8b39"}, "outputs": [], "cell_type": "code", "source": ["dataset = df.drop('label',axis=1)\n", "# convert from [0:255] => [0.0:1.0]\n", "dataset = np.multiply(dataset.values.astype(np.float32), 1.0 / 255.0)\n", "test = np.multiply(test.values.astype(np.float32), 1.0 / 255.0)\n", "\n", "dataset.shape,labels_encoded.shape"]}, {"source": ["In order to prevent overfitting, we need to split the dataset into train dataset and validation dataset. I set the train dataset size to 40000 and validation dataset size to 2000."], "metadata": {"_uuid": "9f908e76a4eebeabd69d2599257b2fae55deae4a", "_cell_guid": "753f1612-26f8-4f36-a9d2-74fc3ae1b37d"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "9ca4ad53f59a5694eb5ae99ee7a8a25ae2af10e0", "_cell_guid": "41fb7b35-95bb-4589-9f38-6bf9ee06cb2d"}, "outputs": [], "cell_type": "code", "source": ["train_size = 40000\n", "validation_size = 2000"]}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "4b99246ffe8c1a6876aac6b0df76c2da1f2ad809", "_cell_guid": "eae1757a-605d-4164-82af-bd0d6df46ca9"}, "outputs": [], "cell_type": "code", "source": ["train = dataset[:train_size]\n", "train_targets = labels_encoded[:train_size]\n", "validation = dataset[train_size:]\n", "validation_targets = labels_encoded[train_size:]\n", "\n", "train.shape, train_targets.shape, validation.shape, validation_targets.shape, test.shape "]}, {"source": ["**Part 2: Model Preparation**\n", "\n", "In this part, I'll prepare the model to have 2000 width and 5 layers"], "metadata": {"_uuid": "b2d49a790c7b8dfe087cdb43e1ea926f44589144", "_cell_guid": "f561d5ab-2b8a-4769-b0ff-7ba1dc318a08"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "2d38b4e953887e9a83b490b233ff27823b4869b7", "_cell_guid": "03bca42e-e71a-42cd-b019-d3387050f1da"}, "outputs": [], "cell_type": "code", "source": ["input_size = 784\n", "output_size = 10\n", "hidden_layer_size = 2000\n", "\n", "tf.reset_default_graph()"]}, {"source": ["In tensorflow, we need to set the placeholder for our inputs, and the targets. Targets is basically the **true labels we are going to compare** with our outputs from the model"], "metadata": {"_uuid": "b52e39c5859a26e8531778bede2c49277c8eb576", "_cell_guid": "f1ed9373-14dd-4f83-9726-d524bd570a4e"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "0437541ab4c318ddf26c2a327bd0be21f422fe99", "_cell_guid": "d804e20f-bcb9-489a-9759-68a4e68b06f3"}, "outputs": [], "cell_type": "code", "source": ["#set the input and targets placeholders\n", "inputs = tf.placeholder(tf.float32,[None,input_size])\n", "targets = tf.placeholder(tf.float32,[None,output_size])"]}, {"source": ["the model concept always simple, **y = weights \\* x + biases**, only the implementation maybe getting complex"], "metadata": {"_uuid": "06c9576c6d51f7e90e290ca345f0970260b73fcd", "_cell_guid": "42b9f529-5b9d-45b0-9451-6d33c25bcf9e"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "36f86bfa09e7a2a5faa59502095a80eff09a4809", "_cell_guid": "a9cddac1-1796-4a95-8ce0-b3ae9ef21092"}, "outputs": [], "cell_type": "code", "source": ["#setting the 1st layer\n", "w_1 = tf.get_variable('w_1',[input_size,hidden_layer_size])\n", "b_1 = tf.get_variable('b_1',[hidden_layer_size])\n", "\n", "#to deal with non-linearity, activation function must be applied on every outputs\n", "#I use relu on this model, you can choose another function like sigmoid,tanH,or softmax\n", "o_1 = tf.nn.relu(tf.matmul(inputs,w_1) + b_1)"]}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "820975720075dc34fc8a9bb2d2d5d152e710b33b", "_cell_guid": "69546699-0b92-4f50-83e0-4f0500044b46"}, "outputs": [], "cell_type": "code", "source": ["#setting the 2nd layer\n", "w_2 = tf.get_variable('w_2',[hidden_layer_size,hidden_layer_size])\n", "b_2 = tf.get_variable('b_2',[hidden_layer_size])\n", "\n", "#still using relu\n", "o_2 = tf.nn.relu(tf.matmul(o_1,w_2) + b_2)"]}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "fae7d1a1f1c12e61c3060b3f18635a72c6017c1c", "_cell_guid": "fbf435a4-e1b3-4fde-a78a-5a59652269a7"}, "outputs": [], "cell_type": "code", "source": ["#setting the 3rd layer\n", "w_3 = tf.get_variable('w_3',[hidden_layer_size,hidden_layer_size])\n", "b_3 = tf.get_variable('b_3',[hidden_layer_size])\n", "\n", "#still using relu\n", "o_3 = tf.nn.relu(tf.matmul(o_2,w_3) + b_3)"]}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "38941c7e761c3f7c7cf6b5bea697e0a6b2a3a6f9", "_cell_guid": "b1b933a7-5e6a-429c-868c-905174c956c7"}, "outputs": [], "cell_type": "code", "source": ["#setting the 4th layer\n", "w_4 = tf.get_variable('w_4',[hidden_layer_size,hidden_layer_size])\n", "b_4 = tf.get_variable('b_4',[hidden_layer_size])\n", "\n", "#still using relu\n", "o_4 = tf.nn.relu(tf.matmul(o_3,w_4) + b_4)"]}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "8a5fa42b5f66cc294eea3ea1d337d6740c60b5ed", "_cell_guid": "4842c59d-9b5e-466a-bc5f-7b8e7421b4e7"}, "outputs": [], "cell_type": "code", "source": ["#setting the 5th layer\n", "w_5 = tf.get_variable('w_5',[hidden_layer_size,hidden_layer_size])\n", "b_5 = tf.get_variable('b_5',[hidden_layer_size])\n", "\n", "#still using relu\n", "o_5 = tf.nn.relu(tf.matmul(o_4,w_5) + b_5)"]}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "4a795851e4c3f685d93f8c524a4aa8feaaba0d21", "_cell_guid": "3e30914d-2353-496c-80b4-e7a39bf77bce"}, "outputs": [], "cell_type": "code", "source": ["#setting the outputs layer\n", "w_6 = tf.get_variable('w_6',[hidden_layer_size,output_size])\n", "b_6 = tf.get_variable('b_6',[output_size])\n", "\n", "#for the last part, I don't use any activation function, later I'll use softmax on it\n", "outputs = tf.matmul(o_5,w_6)+ b_6"]}, {"source": ["**Part 3: Objective Function Preparation**\n", "\n", "In this part, I'll set the objective function. on loss function, I'll use cross entropy, since it is the most often loss function used for classification problem."], "metadata": {"_uuid": "311edf29c50973e46a3fa7e0c4e69d1437150e89", "_cell_guid": "c74bae8b-c781-49a5-9be7-fc7fb8d1d868"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "196d2cc4858b8fcd5e1dbb7e151cf79518f7c8e9", "_cell_guid": "41946442-4fed-49a8-9c80-b0fef58bb70c"}, "outputs": [], "cell_type": "code", "source": ["loss = tf.nn.softmax_cross_entropy_with_logits(logits=outputs,labels=targets)\n", "mean_loss = tf.reduce_mean(loss)"]}, {"source": ["**Part 4: Optimizer Preparation**\n", "\n", "for the optimizer, I use AdamOptimizer."], "metadata": {"_uuid": "24f5f2067b534a446c29ed13fa8e3a5b251a7754", "_cell_guid": "cc804c8d-a870-4ccd-906f-3a3ad4d31e64"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "4e5832d0e03b5ef4bf5d01a45b16c1739084479e", "_cell_guid": "d3464a30-6148-4bf6-a063-34b58c341439"}, "outputs": [], "cell_type": "code", "source": ["optimize = tf.train.AdamOptimizer(learning_rate=0.0002).minimize(mean_loss)\n", "\n", "#argmax simply return the index of the outputs to be compare with the targets\n", "#example(output = [0,0,1] will return 2, targets = [0,0,1] return 2, so the result = True or 1)\n", "#for all samples, let's say 60 out of 100 is true, so the accuracy = 60/100 = 60%\n", "out_equal_target = tf.equal(tf.argmax(outputs,1),tf.argmax(targets,1))\n", "accuracy = tf.reduce_mean(tf.cast(out_equal_target,tf.float32))"]}, {"source": ["**Part 5: Learning**\n", "\n", "1st, initialize all variables (weights and biases). In default, tensorflow use Xavier Initializer, so we don't need to worry about initialization part, just use the default ones."], "metadata": {"_uuid": "e2eddd210533b4be5c3fab10fd2e7b1284daf56a", "_cell_guid": "c991d1ee-c703-4f89-8787-7b3c498aa103"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "13dc193e8bc3026f18eb0290d743ade5cd4ff352", "_cell_guid": "3782add0-0906-459f-a35d-e9901b7bbfb8"}, "outputs": [], "cell_type": "code", "source": ["sess = tf.InteractiveSession()\n", "initializer = tf.global_variables_initializer()\n", "sess.run(initializer)"]}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "5adcd452ef3db6979cc3b9d9ed7cfc552ab4aff0", "_cell_guid": "805eeb21-8335-429a-887f-382bbfdf7a61"}, "outputs": [], "cell_type": "code", "source": ["#to make the learning process running in timely manner, batching is a good practice.\n", "batch_size = 150\n", "batch_number = train.shape[0]//batch_size\n", "max_epoch = 15\n", "prev_validation_loss = 9999999."]}, {"source": ["*Learning Process*"], "metadata": {"_uuid": "a1833e347b4ab60b521dadf39350f4de09896ee5", "_cell_guid": "02240b45-7a38-47b1-a594-adb2ed5f795a"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "f6176d24d24bee2b1f2726f44e06bc20338908b7", "_cell_guid": "4af70321-e1e0-4a91-b8f7-0a33b0ab93a8"}, "outputs": [], "cell_type": "code", "source": ["for epoch_counter in range(max_epoch):\n", "    curr_epoch_loss = 0\n", "    start = 0\n", "    end = start+batch_size\n", "    \n", "    #batch training\n", "    for batch_counter in range(batch_number):\n", "        #set the input and target batch equals to defined size\n", "        input_batch = train[start:end]\n", "        target_batch = train_targets[start:end]\n", "        start = end\n", "        end = start+batch_size\n", "        \n", "        #running optimizer, feeding the model with current batch dataset\n", "        #the model will continously set the weight and biases with forward and back propagation\n", "        _, batch_loss = sess.run([optimize,mean_loss],\n", "                                 feed_dict={inputs:input_batch, targets:target_batch}                                )\n", "        curr_epoch_loss += batch_loss\n", "    \n", "    curr_epoch_loss /= batch_number\n", "    \n", "    #validation, forward propagate only, to see the accuracy on the model using new dataset     \n", "    val_loss, val_accuracy = sess.run([mean_loss,accuracy],\n", "                                 feed_dict={inputs:validation, targets:validation_targets})\n", "    \n", "    print('Epoch '+str(epoch_counter+1)+\n", "         '. Training loss: '+'{0:.3f}'.format(curr_epoch_loss)+\n", "         '. Validation loss: '+'{0:.3f}'.format(val_loss)+\n", "         '. Validation accuracy: '+'{0:.2f}'.format(val_accuracy*100.)+'%')\n", "    \n", "    #the rule to prevent overfitting\n", "    #1st, we already set max epoch to prevent the model continously iterating causing overfit\n", "    #2nd, if validation loss starts increasing, we need to stop learning\n", "    if val_loss > prev_validation_loss:\n", "        break\n", "    \n", "    prev_validation_loss = val_loss\n", "print('End of Training')"]}, {"source": [], "metadata": {"_uuid": "c2f968b66c814df95b9a04b9219badda1885321f", "_cell_guid": "d9369514-264d-44ab-945a-b2638e241e3e"}, "cell_type": "markdown"}, {"source": ["# Submission\n", "\n", "for the submission, we need to apply softmax function to outputs in order to get values ranged between 0 and 1"], "metadata": {"_uuid": "888bfc8b7a421a56a2ab97419912f8357722f8d5", "_cell_guid": "ee7360b3-2c2e-4d6a-89ca-f4752c0476df"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "cce91e8d7ad28216badc0910b5b07eed78403fae", "_cell_guid": "78b01da2-a0cc-499d-9ca1-753287f90cf0"}, "outputs": [], "cell_type": "code", "source": ["predict = tf.argmax(tf.nn.softmax(outputs),1)"]}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "75e0731662ab8745e4ddff27e2096bbb9e929e47", "_cell_guid": "b327721b-5d4e-43e5-869f-88192f8d7a32"}, "outputs": [], "cell_type": "code", "source": ["#forward propagate using trained model to get prediction results\n", "predictions = predict.eval(feed_dict={inputs: test})"]}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "564f6c3a51c6781ab5a53c4402bf45bc86199358", "_cell_guid": "791d1131-fdd8-45a2-bc9f-52f6a7cd1e4d"}, "outputs": [], "cell_type": "code", "source": ["submission = pd.DataFrame({\n", "    'ImageId': range(1,len(predictions)+1),\n", "    'Label': predictions\n", "})"]}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "22317f5027d7ba407406e7676c9c7f02ef79bf0d", "_cell_guid": "8f2c1524-101b-4d25-86bb-1c86f4fb520b"}, "outputs": [], "cell_type": "code", "source": ["submission.head(3)"]}, {"source": ["well, just by checking first 3 images with the predicted label, it seems we've done a great job"], "metadata": {"_uuid": "62a7184323289bafc52ebdda00f733aa4a265005", "_cell_guid": "d8ffbee3-67ab-43a3-a6c5-91cd428c4a2f"}, "cell_type": "markdown"}, {"execution_count": null, "metadata": {"collapsed": true, "_uuid": "2b5c0dad73154db206becc7d13fa889f8defebd6", "_cell_guid": "014e86f2-8f21-4bd7-92b5-af90a2605774"}, "outputs": [], "cell_type": "code", "source": ["submission.to_csv('submission.csv',index=False)"]}, {"source": ["**Credits:**\n", "\n", "I'm still learning, and I'd like to give credits to following contributors:\n", "* https://www.kaggle.com/bhushan23/mnist-with-softmax-tensorflow-tutorial\n", "* https://www.kaggle.com/kakauandme/tensorflow-deep-nn"], "metadata": {"collapsed": true, "_uuid": "c1b40ad5b92f1aacef890d6120fc58adfb17087b", "_cell_guid": "6146b5ad-d76a-4066-ad7b-96f9045c0243"}, "cell_type": "markdown"}], "nbformat_minor": 1, "metadata": {"language_info": {"codemirror_mode": {"version": 3, "name": "ipython"}, "file_extension": ".py", "name": "python", "version": "3.6.3", "pygments_lexer": "ipython3", "mimetype": "text/x-python", "nbconvert_exporter": "python"}, "kernelspec": {"language": "python", "name": "python3", "display_name": "Python 3"}}, "nbformat": 4}