{"cells":[{"metadata":{"_uuid":"d1a76b97dda99cd85550a2eaf2b91b258706ed03"},"cell_type":"markdown","source":"<center>\n<h1> Danbury AI June 2018: Workshop Part 3</h1>\n<h2>MNIST</h2>\n</center>\nWe begin our journey into using neural networks for modeling image data by starting with the classic [MNIST dataset](http://yann.lecun.com/exdb/mnist/). This academic dataset consists of well pre-processed images of digits 0 through 9. Due to the preprocessing, this dataset is simple enough to model with simple neural networks, but still maintains sufficient complexity to be a non-novel dataset for evaluating the effectivenss of neural networks versus other methods. In our next notebook we will be using a more complicated natural image dataset which will require more sophisticated neural architectures to model appropriately. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"collapsed":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nfrom ipywidgets import interact\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout\nfrom keras.models import Sequential, Model\nfrom sklearn.model_selection import train_test_split\nimport os\nprint(os.listdir(\"../input/digit-recognizer\"))","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"eede6e7960a5b523ecf0ae4bf9c2003fff77e036"},"cell_type":"markdown","source":"Let's read in our data and display the first few rows. "},{"metadata":{"trusted":true,"_uuid":"fed0217dc7f3206c6dc6285f5c479d1449398e42","collapsed":true},"cell_type":"code","source":"mnistTrainingData = pd.read_csv(\"../input/digit-recognizer/train.csv\")\nmnistTrainingData.head()","execution_count":2,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"72ce7a346ea732b2a1873db5a6143ed13716a2f4"},"cell_type":"markdown","source":"Since the labels and features are in the same matrix, we will need to split the matrix into a feature matrix *x* and a label vector *y*."},{"metadata":{"trusted":true,"_uuid":"0483f1aef5a562849067e8bb3d7ec4d623693c4d","collapsed":true},"cell_type":"code","source":"X = mnistTrainingData.values[:,1:]\ny = mnistTrainingData.values[:,0]","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"65f7a6febe3274635ff33db1aead633fcb7bd930"},"cell_type":"markdown","source":"We can use the interactive ipython components to visualize the digits. The label of the digit is displayed above the images. "},{"metadata":{"trusted":true,"_uuid":"cb6869265e43a75dd39e01c86a3bee99ee4e296e","collapsed":true},"cell_type":"code","source":"def disp(imSelIdx=0):\n    plt.title(y[imSelIdx])\n    plt.imshow(X[imSelIdx].reshape(28,28), cmap=\"gray\")\n\ninteract(disp,imSelIdx=(0,X.shape[0]))","execution_count":15,"outputs":[]},{"metadata":{"_uuid":"a7395028b8ee3feb04daedcd502e93a03072a4aa"},"cell_type":"markdown","source":"We will now convert the integer labels contained in *y* to [one-hot vectors](https://www.youtube.com/watch?v=2Uyr93f3C2M). "},{"metadata":{"trusted":true,"_uuid":"2165c5f5f405b24c5a18b38efb25ce26d483e961","collapsed":true},"cell_type":"code","source":"def oneHotEncoder(integerVal,maxClasses):\n    out = np.zeros(maxClasses)\n    out[integerVal] = 1\n    return out\n\ny_onehot = []\n\nfor i in y:\n    y_onehot.append(oneHotEncoder(i,y.max()+1))\n\ny_onehot = np.stack(y_onehot)\n\nprint(y_onehot)\nprint(\"Shape of y vector: {0}\".format(y.shape))\nprint(\"Shape of y one-hot matrix: {0}\".format(y_onehot.shape))","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"0951751cccf3dbfcb735f632b697b32dad24d76a"},"cell_type":"markdown","source":"Now we will split our data 80/20 into our *training* and *validation* sets. "},{"metadata":{"trusted":true,"_uuid":"ad4f890fa0bf2a8aa02a7e72c49d05aa31dbde1f","collapsed":true},"cell_type":"code","source":"X_train, X_validation, y_train , y_validation = train_test_split(X,y_onehot, test_size=0.2)","execution_count":12,"outputs":[]},{"metadata":{"_uuid":"93b3d775b25c56320894bf98ea7acbecd321650e"},"cell_type":"markdown","source":"At this point we are ready to train our first simple *feed-forward neural network*. "},{"metadata":{"trusted":true,"_uuid":"f6418fbc2c7dce110c9ecd60fd2aa53dec87dd6a","collapsed":true},"cell_type":"code","source":"inputs = Input(shape=(X.shape[1],))\n\nx = Dense(100, activation='sigmoid')(inputs)\nx = Dense(y_onehot.shape[1], activation='softmax')(x)\n\nmodel = Model(inputs=inputs, outputs=x)\nmodel.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])\n\nmodel.summary()","execution_count":16,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"afec5d86c80f687aff7c4cdc3f8a4d0879d3c590","scrolled":true,"collapsed":true},"cell_type":"code","source":"hist = model.fit(X_train,y_train,epochs=50, batch_size=100, validation_data=(X_validation,y_validation)) ","execution_count":18,"outputs":[]},{"metadata":{"_uuid":"500c99dc7ac0448188116571826f7586539c56b8"},"cell_type":"markdown","source":"Here we will define a function which allows us to plot the training history of our neural network. "},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"2e704db2dcdc8459653f3cf00fec5fb2cd1fc516"},"cell_type":"code","source":"def learningCurves(hist):\n    histAcc_train = hist.history['acc']\n    histLoss_train = hist.history['loss']\n    histAcc_validation = hist.history['val_acc']\n    histLoss_validation = hist.history['val_loss']\n    maxValAcc = np.max(histAcc_validation)\n    minValLoss = np.min(histLoss_validation)\n\n    plt.figure(figsize=(12,12))\n    epochs = len(histAcc_train)\n\n    plt.plot(range(epochs),histLoss_train, label=\"Training Loss\", color=\"#acc6ef\")\n    plt.plot(range(epochs),histLoss_validation, label=\"Validation Loss\", color=\"#a7e295\")\n\n    plt.scatter(np.argmin(histLoss_validation),minValLoss,zorder=10,color=\"green\")\n\n    plt.xlabel('Epochs',fontsize=14)\n    plt.title(\"Learning Curves\",fontsize=20)\n\n    plt.legend()\n    plt.show()\n\n    print(\"Max validation accuracy: {0}\".format(maxValAcc))\n    print(\"Minimum validation loss: {0}\".format(minValLoss))\n","execution_count":26,"outputs":[]},{"metadata":{"_uuid":"ee03d03cecb4d9ee9faa7e1284359666b46be650"},"cell_type":"markdown","source":"We will now use the function defined above to visualize the learning curves/training curves of our neural network. "},{"metadata":{"trusted":true,"_uuid":"7c8331bb73305ce701b0884b893a9dab9eb5e80f","collapsed":true},"cell_type":"code","source":"learningCurves(hist)","execution_count":27,"outputs":[]},{"metadata":{"_uuid":"93dbc5092640c70411ab5d8a6b962f2bcd71ae80"},"cell_type":"markdown","source":"We will now add a dropout layer between the hidden layer and output layer. "},{"metadata":{"trusted":true,"_uuid":"c699679e3448bc665c15041198a745922b282456","collapsed":true},"cell_type":"code","source":"inputs = Input(shape=(X.shape[1],))\n\nx = Dense(100, activation='sigmoid')(inputs)\nx = Dropout(0.5)(x)\nx = Dense(y_onehot.shape[1], activation='softmax')(x)\n\nmodel3 = Model(inputs=inputs, outputs=x)\nmodel3.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])\n\nmodel3.summary()","execution_count":33,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1cee6984ea0da8abbd38fba49786e1d6b1fb720a","collapsed":true},"cell_type":"code","source":"hist3 = model3.fit(X_train,y_train,epochs=50, batch_size=100, validation_data=(X_validation,y_validation)) ","execution_count":34,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"55803912e19616fcde27fd77fc94dba99b55387a","collapsed":true},"cell_type":"code","source":"learningCurves(hist3)","execution_count":35,"outputs":[]},{"metadata":{"_uuid":"fb5048c1c332e127264fe165f7b321860857d130"},"cell_type":"markdown","source":"**Workshop Problems**\n* In part 2 we saw how to use sklearn models like linear regression, random forests, and boosted trees. How do these models compare to the effectiveness of neural networks on this problem? Apply these models to MNIST here. \n* Modify the networks above by adding layers, adjusting layer widths, and changing how much dropout is used. How do these changes impact the learning curves? "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}