{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\nThis notebook is meant to demonstrate an implementation of the LeNet neural network architecture as written by Adrian Rosebrock in his book **Deep Learning for Computer Vision with Python**.\n\nThe LeNet architecture is a small and simple feed-forward convolutional neural network excellent for a first introduction to deep learning.\n\nHere, we will implement the LeNet architecture and train it on the Kaggle-Specific MNIST dataset to make predictions on the test data and explain each step on the way.","metadata":{}},{"cell_type":"markdown","source":"# Import Packages\nFirst, we must import the packages necessary for building, training and evaluating our model.\n\n***numpy*** and ***pandas*** are imported to access *numpy arrays*, *pandas DataFrames* and other functions we'll be using to manage our data.\n\nFrom ***keras.models*** we import ***Sequential***, a model type which is a stack of layers with one input and one output.\n\nFrom ***keras.layers.convolutional*** we import ***Conv2D*** and ***MaxPooling2D***, which are the *Convolutional* layers and *Pooling* layers we will be using, respectively.\n\nFrom ***keras.layers.core*** we import ***Activation***, ***Flatten*** and ***Dense***, which are our *Activation* functions, *Flatten* method and *Dense* layers that we need to build our model.\n\nFrom ***keras.optimizers*** we import ***SGD*** which stands for *Stochastic Gradient Descent*, the optimization method we'll be using for our model.\n\nFrom ***sklearn.model_selection*** we import ***train_test_split***, the function we'll be using to split our dataset into a training set and test set.\n\nFrom ***sklearn.preprocessing*** we import ***LabelBinarizer***, the function we'll be using to encode our data labels to a format suitable for fitting the model.","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \n\nfrom keras.models import Sequential\nfrom keras.layers.convolutional import Conv2D, MaxPooling2D\nfrom keras.layers.core import Activation, Flatten, Dense\nfrom keras import backend as K\nfrom keras.optimizers import SGD\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import LabelBinarizer","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Import Data","metadata":{}},{"cell_type":"markdown","source":"First, we'll use pandas to import our dataset, which is given in the form of a CSV (Comma Seperated Values) file and convert it into a pandas DataFrame.\n\nThen, we split our data into the labels (which consist of the entire first column) and the numerical intensities of each pixel (which consist of all other columns).\n\nFinally, we convert the data we split into numpy arrays and print their shape to explore their dimensions.","metadata":{}},{"cell_type":"code","source":"data = pd.read_csv(\"../input/digit-recognizer/train.csv\")\n\nlabels = data.iloc[:, 0]\npixels = data.iloc[:, 1:]\n\nnp_labels = labels.to_numpy()\nnp_pixels = pixels.to_numpy()\n\nprint(np_labels.shape)\nprint(np_pixels.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocess Data","metadata":{}},{"cell_type":"markdown","source":"Using the ***train_test_split*** function import from the ***sklearn*** library, we split the numpy arrays containing our pixel data and labels into a set for training and a set for testing our model.","metadata":{}},{"cell_type":"markdown","source":"**The parameters we use for the train_test_split function are as follows:**\n\n*(np_pixels / 255.0)* This is our numpy array of pixel data divided by 255.0, the reason for dividing by 255.0 is to normalize our data between [0-1] to make training easier.\n\n*(np_labels)* This is our numpy array of label data.\n\n*(test_size = 0.2)* This represents how we intend to split our data. With a value of 0.2, 20% of our data will be allocated into our test set, and the remaining 80% will be used for training.\n\n*(random_state = 777)* This is set to 777 to ensure that every time we run our code, the dataset is split in exactly the same way. Any value can be used here as long as it isn't changed during evaluation.","metadata":{}},{"cell_type":"code","source":"train_data, test_data, train_labels, test_labels = train_test_split(np_pixels / 255.0, np_labels, test_size = 0.2, random_state = 777)\n\ntrain_data = train_data.reshape(train_data.shape[0], 28, 28, 1).astype(\"float32\")\ntest_data = test_data.reshape(test_data.shape[0], 28, 28, 1).astype(\"float32\")\n\nlb = LabelBinarizer()\ntrain_labels = lb.fit_transform(train_labels)\ntest_labels = lb.transform(test_labels)\n\nprint(train_data.shape)\nprint(train_labels.shape)\nprint(test_data.shape)\nprint(test_labels.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Define Model","metadata":{}},{"cell_type":"code","source":"class Model:\n    def build(width, height, depth, classes):\n        model = Sequential()\n        inputShape = (height, width, depth)\n        \n        model.add(Conv2D(20, (5, 5), padding = \"same\", input_shape = inputShape))\n        model.add(Activation(\"relu\"))\n        model.add(MaxPooling2D(pool_size = (2, 2), strides = (2, 2)))\n        \n        model.add(Conv2D(50, (5, 5), padding = \"same\", input_shape = inputShape))\n        model.add(Activation(\"relu\"))\n        model.add(MaxPooling2D(pool_size = (2, 2), strides = (2, 2)))\n        \n        model.add(Flatten())\n        model.add(Dense(500))\n        model.add(Activation(\"relu\"))\n        \n        model.add(Dense(classes))\n        model.add(Activation(\"softmax\"))\n        \n        return model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Build Model and Fit Data","metadata":{}},{"cell_type":"code","source":"model = Model.build(28, 28, 1, 10)\nopt = SGD(lr = 0.01)\n\nmodel.compile(loss = \"categorical_crossentropy\", optimizer = opt, metrics = [\"accuracy\"])\n\nX = model.fit(train_data, train_labels, validation_data = (test_data, test_labels), batch_size = 128, epochs = 20, verbose = 1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Calculate Score and Summary","metadata":{}},{"cell_type":"code","source":"score = model.evaluate(test_data, test_labels, batch_size = 128)\nprint(score)\n\nmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generate Predictions on Test Data","metadata":{}},{"cell_type":"code","source":"kaggle_test_data = pd.read_csv(\"../input/digit-recognizer/test.csv\")\nkaggle_test_data = kaggle_test_data.to_numpy()\nkaggle_test_data = kaggle_test_data.reshape(kaggle_test_data.shape[0], 28, 28, 1).astype(\"float32\")\n\nkaggle_test_predictions = model.predict(kaggle_test_data, batch_size = 128)\n\nkaggle_test_submission = pd.DataFrame(kaggle_test_predictions)\n\nkaggle_test_submission[\"Label\"] = pd.DataFrame(kaggle_test_submission.idxmax(axis = 1))\nkaggle_test_submission[\"ImageId\"] = kaggle_test_submission.index + 1\n\nkaggle_test_submission = kaggle_test_submission[[\"ImageId\", \"Label\"]]\n\nkaggle_test_submission.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generate a CSV File","metadata":{}},{"cell_type":"code","source":"kaggle_test_submission.to_csv(\"./submission.csv\", encoding = \"utf-8\", index = False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Check if CSV File is Valid","metadata":{}},{"cell_type":"code","source":"kaggle_test = pd.read_csv(\"submission.csv\")\nkaggle_test.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}