{"cells": [{"cell_type": "markdown", "metadata": {"_cell_guid": "3255adcc-f482-4f72-9b13-1b27a1bdc4e8", "_uuid": "42ec1a90ecdb55711bd695aea2696a42bb89d2ea"}, "source": ["A quick walkthrough to apply Covolutional Neural Network (CNN) via Keras on the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset in the [Kaggle Digit Recognizer competition](https://www.kaggle.com/c/digit-recognizer).\n", "\n", "If you want to begin somewhere, with clearly annotated code but fewer paragraphs, try here ;)\n", "\n", "Kudos for [vishwas](https://www.kaggle.com/vishwasgpai)'s [nice starter](https://www.kaggle.com/vishwasgpai/guide-for-creating-cnn-model-using-csv-file) I came across where this model is based. Please do also read & upvote his if you find this notebook userful.\n", "\n", "I rewritten the explanations and did the following code edits:\n", "* adding `train_test_split` to test & improve results\n", "* update to [Keras 2 syntax](https://github.com/keras-team/keras/issues/6006)\n", "* wrap up the kernel to generate and save output\n", "* adding colored visualization check for label prediction\n", "\n", "To focus on the application, I catalog links to relevant documentation/details for fellow beginners.
\n", "\n", "This is my first Kaggle kernel. Please let me know if you have any suggestions :)
\n", "\n", "*Let's go!*\n", "\n", "> #keras #cnn #conv2d #mnist #concise #beginner"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "cc2682e9-8cc3-4198-8dba-a598de8c4203", "_uuid": "d2ab2886a90e0c84ce112bf227c1898ed7e56194"}, "source": ["---\n", "# Import modules\n", "Import the usual modules\n", "* [numpy](http://www.numpy.org): linear algebra, e.g matrix calculation __\"all-the-math\"__\n", "* [pandas](http://pandas.pydata.org/pandas-docs/stable/): __\"tables\"__ (\"DataFrame\") I/O & manipulation\n", "* [matplotlib](https://matplotlib.org): data visualization, ie __\"draw plots\"__\n", "* [sklearn](http://scikit-learn.org) (\"sci-kit learn\"): __\"data mining toolset\"__ usu. for supervised machine learning and data processing, e.g.\n", " - data preprocessing \n", " - classification\n", " - regression\n", "* [keras](https://keras.io): __neural network__ library\n", " - uses [tensorflow](https://www.tensorflow.org) by default, [theano](https://deeplearning.net/software/theano/), or [CNTK](https://github.com/Microsoft/CNTK/wiki) backend for __tensor manipulation__\n", " - Sequential() is among the most commonly used models to pipeline operations in each layer\n", "\n", "tensor: take it as scalar, vector, or matrix - vector of vector for now. Look at [here](https://math.stackexchange.com/questions/412423/differences-between-a-matrix-and-a-tensor/412429#412429) and scroll up for the (much) more mathematically accurate and detailed answer. What we really \"see\" in notebooks are matrix tables as Pandas DataFrames. "]}, {"cell_type": "code", "metadata": {"_cell_guid": "0fac0ff9-7352-48f8-8ecb-9bb6bdb17f01", "scrolled": true, "_uuid": "920a1f76a5d5a4c7e57c99e952657aa8f375bae0"}, "outputs": [], "execution_count": null, "source": ["# Basic data processing\n", "import numpy as np\n", "import pandas as pd \n", "\n", "# Try visualize pixel values as an image\n", "import matplotlib.pyplot as plt\n", "\n", "# Split labelled dataset into training and testing data to test & improve our model\n", "from sklearn.model_selection import train_test_split \n", "# Change label formats between input/human-readable/output-required & better for model training formats\n", "from sklearn.preprocessing import LabelEncoder, LabelBinarizer\n", "\n", "# To build our CNN sequential model\n", "from keras.models import Sequential\n", "from keras.layers import Dense, Dropout, Flatten\n", "from keras.layers.convolutional import Convolution2D, MaxPooling2D # CNN\n", "from keras import backend as K\n", "\n", "# List the data files in store\n", "from subprocess import check_output\n", "print(check_output([\"ls\", \"../input/\"]).decode(\"utf8\"))"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "8816aa4b-c2b6-4cfa-9efb-c8dcf0596859", "_uuid": "826c14b66a1ae45e15a0eaa3dde73011f2224050"}, "source": ["---\n", "# Loading data"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "0833d1d2-8126-4c23-b8fb-0f22a19d54eb", "_uuid": "39dcdb4dafbf4f4804717bb563a7be0e3ab1c0cb"}, "source": ["Not all raw data format are as standardized as in Kaggle.
\n", "I have a habit to use `!head {yourfile}` to inspect input data before loading.
\n", "You will then know the right arguments for delimiters, header, comments etc for pd.[read_csv](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html)"]}, {"cell_type": "code", "metadata": {"_cell_guid": "0eb063a2-3c3d-45cd-a024-b8169bda8171", "scrolled": true, "_uuid": "3a01bf60d7ee1a77bf3a9934eab256d19d5732fe"}, "outputs": [], "execution_count": null, "source": ["# Read in 2 lines of the training data to inspect\n", "!head -n2 ../input/train.csv"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "65d95a16-a4ac-4c5e-ae1a-69e1ee921328", "_uuid": "d60e1715df661dc789e855fc0584a53418e7ff77"}, "source": ["Read the CSV tables and check the data dimensions."]}, {"cell_type": "code", "metadata": {"_cell_guid": "787fbed3-b610-4922-9034-acbee2a1ca28", "scrolled": true, "_uuid": "1ce1e111fea00a711c052ef1d3b24dabbaa8c9bd"}, "outputs": [], "execution_count": null, "source": ["train = pd.read_csv(\"../input/train.csv\")\n", "test = pd.read_csv(\"../input/test.csv\")\n", "print(train.shape)\n", "print(test.shape)"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "11099c25-1fce-42a0-be92-3070339daf6e", "_uuid": "7ff51c982ec1407248aa2c2fcc1bd77a4e15e19e"}, "source": ["---\n", "# Data preprocessing"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "1af2f19a-d8bd-43b4-93a3-767a09e7b435", "_uuid": "35129997692636e3eab8eb59e558a96d90ee2d10"}, "source": ["We saw that the data are shaped as linear arrays of pixel intensities, e.g.
\n", "oooooooxoooo#ooooxooooooo\n", "\n", "But images are 2D matrix of pixel values (, ie a 3D matrix), like\n", "\n", "ooooo
\n", "ooxoo
\n", "oo#oo
\n", "ooxoo
\n", "ooooo
\n", "\n", "Also, it's faster to store and calculate with [0,1] decimals than [0,255] (8-bit) values.\n", "\n", "So, we\n", "1. Reshape the matrix\n", "2. Normalize the values from [0,255] to [0,1]"]}, {"cell_type": "code", "metadata": {"_cell_guid": "fa532dbc-c20b-4fad-b144-96d19df2a2f6", "collapsed": true, "_uuid": "a19ebc3331aaf432e699e87ff05fb76e174961a7"}, "outputs": [], "execution_count": null, "source": ["# Reshape and normalize training data\n", "# drop the \"label\" column for training dataset\n", "X = train.drop(\"label\",axis=1).values.reshape(-1,1,28,28).astype('float32')/255.0\n", "y = train[\"label\"]\n", "\n", "# Reshape and normalize test data\n", "X_test1 = test.values.reshape(-1,1,28,28).astype('float32')/255.0"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "fc1096ac-dfae-4d05-8aa1-36206cf9a69f", "_uuid": "fe8137321073bf8039659b44666abafcdcd6ef2e"}, "source": ["Let's take a look at the one of the images."]}, {"cell_type": "code", "metadata": {"_cell_guid": "8656967e-5fad-47df-9b22-29c8116a0ffa", "_uuid": "8e9669e2a2b5034b323cbce579dd2a3804d04ec4"}, "outputs": [], "execution_count": null, "source": ["plt.imshow(train.drop(\"label\",axis=1).iloc[0].values.reshape(28,28),cmap=plt.get_cmap('binary'))\n", "plt.show()"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "4847d6ae-c638-4adc-b043-91b2b443c228", "_uuid": "0ed896b42c9bb67d1b4ccde83a537209837b8e49"}, "source": ["# Prepare training dataset"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "230dc522-2e8f-4664-9809-7b6675d0b022", "_uuid": "1c69992c731a351fd7514a0d8f916d2d2fd6cc1d"}, "source": ["Split out some data from the training set to test and improve our model.
\n", "(`random_state` set for better chance of reproducibility only)"]}, {"cell_type": "code", "metadata": {"_cell_guid": "ab2d8c06-befe-4f6e-9b74-80674f260c37", "collapsed": true, "_uuid": "1e03e02cb7ff7b417258409df08f379be403fd10"}, "outputs": [], "execution_count": null, "source": ["X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## LabelBinarizer\n", "Binarize the labels from digits to [\"one-hot\"](https://www.quora.com/What-is-one-hot-encoding-and-when-is-it-used-in-data-science) encoding. (linked to the most concise explanation for OneHot seen so far)"]}, {"cell_type": "code", "metadata": {"scrolled": true}, "outputs": [], "execution_count": null, "source": ["# Before binarization\n", "y_train.head()"]}, {"cell_type": "code", "metadata": {"collapsed": true}, "outputs": [], "execution_count": null, "source": ["lb = LabelBinarizer()\n", "y_train = lb.fit_transform(y_train)\n", "y_test = lb.fit_transform(y_test)"]}, {"cell_type": "code", "metadata": {}, "outputs": [], "execution_count": null, "source": ["# After binarization\n", "y_train"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "8ca86c95-2eff-4b88-aebb-803d55a97bcf", "_uuid": "32f2456ea31f4f30be05585b07d73402e573a5e0"}, "source": ["# Training model"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Designing a Conv2D sequential model"]}, {"cell_type": "code", "metadata": {"_cell_guid": "935d1f4c-efae-4d8b-8d5c-78175deac639", "_uuid": "6a4cb804196c60a9ec4fa0ec1d718516aa3dcee5"}, "outputs": [], "execution_count": null, "source": ["# Start a Keras sequential model\n", "model = Sequential()\n", "# Before Keras 2: K.set_image_dim_ordering('th')\n", "K.set_image_data_format('channels_first')\n", "# Before Keras 2: model.add(Convolution2D(30,5,5, border_mode= 'valid', input_shape=(1,28,28),activation= 'relu' ))\n", "model.add(Convolution2D(30, (5,5),padding='valid',input_shape=(1,28,28),activation='relu'))\n", "model.add(MaxPooling2D(pool_size=(2, 2)))\n", "model.add(Convolution2D(15, (3, 3), activation='relu'))\n", "model.add(MaxPooling2D(pool_size=(2, 2)))\n", "# Drop out 20% of training data in each batch\n", "model.add(Dropout(0.2))\n", "\n", "model.add(Flatten())\n", "model.add(Dense(128, activation= 'relu' ))\n", "model.add(Dense(50, activation= 'relu' ))\n", "model.add(Dense(10, activation= 'softmax' ))\n", " # Compile model\n", "model.compile(loss= 'categorical_crossentropy' , optimizer= 'adam' , metrics=['accuracy'])"]}, {"cell_type": "code", "metadata": {"_cell_guid": "5aae3a55-efa3-4f4b-86b1-a4a101a6c07b", "scrolled": true, "_uuid": "d00562509c5e723086f94d4737f392282dbac18a"}, "outputs": [], "execution_count": null, "source": ["model.fit(X_train, y_train,\n", " epochs=25,\n", " batch_size= 128)"]}, {"cell_type": "code", "metadata": {"_cell_guid": "5314a1ee-d4c0-4344-9b8c-90d8035743c0", "_uuid": "08a6798f8dcc389073704ae32374ca95ecd638fe"}, "outputs": [], "execution_count": null, "source": ["score = model.evaluate(X_test, y_test, batch_size=128)"]}, {"cell_type": "code", "metadata": {"_cell_guid": "4d455c99-cb07-4414-b9cd-5a1e5c8654b8", "scrolled": true, "_uuid": "133a38213041769424fe320f42d7f26fb55b2dca"}, "outputs": [], "execution_count": null, "source": ["score"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "0941837b-a42f-42af-b7a2-ad42b3b4b491", "_uuid": "d2cf45a34fb2742d9f03ddd942a00d7d86160947"}, "source": ["---\n", "# Predict test dataset\n", "We're almost there. Let's label the test dataset."]}, {"cell_type": "code", "metadata": {"_cell_guid": "c4109bbb-b4be-4f43-920e-6153ba18ec3b", "_uuid": "5767208fdf66278ab9e460d062447a04da55ad54"}, "outputs": [], "execution_count": null, "source": ["y_test1 = model.predict(X_test1)\n", "y_test1"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Convert the predictions from probabilities to be each of the labels (1-10 here) into the most probably label."]}, {"cell_type": "code", "metadata": {"_cell_guid": "93885153-48ab-40db-afec-1aa1aeecead4", "scrolled": true, "_uuid": "302c4538b013ea484f51341dae9fe9620ed07b95"}, "outputs": [], "execution_count": null, "source": ["y_test1 = lb.fit_transform(np.round(y_test1))\n", "y_test1"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "6068da54-01ed-482a-8221-ff307338a251", "_uuid": "2847919fcfea94ff8c564492640aa74061dc6f28"}, "source": ["Convert label from one-hot to digits, e.g. [0, 1, 0, 0, 0, 0, 0, 0, 0, 0] to 2."]}, {"cell_type": "code", "metadata": {"_cell_guid": "ab06c8e7-e0e9-4c7c-a15b-a7fc1e3075bd", "scrolled": true, "_uuid": "b40f6d487b5afe9c80977f913cd26830414f7414"}, "outputs": [], "execution_count": null, "source": ["predicted_labels = np.argmax(y_test1, axis=1)\n", "predicted_labels"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# See how we're doing\n", "Visualize some test images and see if our labels match.\n", "\n", "The following code shows the first 100 images in the test data set, each colored according to their label.\n", "We can easily spot out if any particular number is more easily misrecognized.[](http://)"]}, {"cell_type": "code", "metadata": {"scrolled": true}, "outputs": [], "execution_count": null, "source": ["cmaps = ['binary','gray','summer','YlOrRd_r','Set3','BuGn_r','spring','tab20b','PuRd_r','winter']\n", "fig,axarr = plt.subplots(10,10)\n", "\n", "for i,ax in enumerate(axarr.flat):\n", " ax.imshow(test.iloc[i].values.reshape(28,28),cmap=plt.get_cmap(cmaps[predicted_labels[i]]))\n", "plt.setp(axarr, xticks=[], yticks=[])\n", "plt.subplots_adjust(hspace=-0.2,wspace=-0.2)\n", "plt.show()"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "9597afcd-e32e-4dbe-a199-2af96f548127", "_uuid": "2d0ea7e0b294806b4f5efc5915cf1b54104f4f1a"}, "source": ["---\n", "# Save output\n", "Save your output for submission"]}, {"cell_type": "code", "metadata": {"_cell_guid": "774c4357-8d8f-4ce1-bc52-0271f61dcc5b", "collapsed": true, "_uuid": "13c60e56dceeaf946c60d1372851b0f531677776"}, "outputs": [], "execution_count": null, "source": ["np.savetxt('submission_kagglekernel_cnn25epochs.csv', \n", " np.c_[range(1,len(X_test1)+1),predicted_labels], \n", " delimiter=',', \n", " header = 'ImageId,Label', \n", " comments = '', \n", " fmt='%d')"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# What next?\n", "Before further, why not also try tune the different parameters?\n", "* dropout (enlarge training dataset vs overfitting)\n", "* change (silly, but yes, why always waste a particular set of data\n", "* number of layers\n", "* number of nodes in each layers\n", "* number of epochs (if you have the equipment and/or time)\n", "* loss function\n", "* optimizer\n", "* metric for optimization\n"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "a773c721-b0bd-4c50-b261-c869ba5bbbdf", "collapsed": true, "_uuid": "546503c6b1abd0c09dd0f73ccbc45870a297594d"}, "source": ["---\n", "Thank you for reading! Hope you enjoyed the kernel. :)
\n", "See you next time!"]}, {"cell_type": "code", "metadata": {"collapsed": true}, "outputs": [], "execution_count": null, "source": []}], "metadata": {"kernelspec": {"name": "python3", "display_name": "Python 3", "language": "python"}, "language_info": {"name": "python", "mimetype": "text/x-python", "codemirror_mode": {"name": "ipython", "version": 3}, "version": "3.6.4", "file_extension": ".py", "nbconvert_exporter": "python", "pygments_lexer": "ipython3"}}, "nbformat": 4, "nbformat_minor": 1}