{"cells":[{"metadata":{},"cell_type":"markdown","source":"welcome to my notebook, here i will guide you through my first attempt of constructing a simple CNN classifier for the MNIST dataset. First, we load the data, as we can see the training set consists of 42000 samples and the test set consists of 28000 samples."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport matplotlib as mpl\n\ndata=pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ndatates=pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")\nprint(data.shape)\nprint(datates.shape)\nprint(data)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"My next step is to split the data into several smaller dataset as 42000 samples is quite a lot and will be time consuming to process all of it. Here i used 40% of the original training set for training the algorithm(datrainx and y) and another smaller batch of 1000 (modtestx and y) for testing several NN architectures. This will still take quite a while to train but i still find it acceptable."},{"metadata":{"trusted":true},"cell_type":"code","source":"#choose percentage of data to be used for training\nrandom=np.random.rand(42000)<0.9#<<by changing this\n#training data 40%\ndatrainx=data.loc[random]\ndatrainy=data.loc[random,'label']\ndatrainx.drop('label',axis=1,inplace=True)\ndatrainx.reset_index(drop=True,inplace=True)\ndatrainy=datrainy.reset_index(drop=True)\n\n#validation data 15% of the data\ndaval=data.loc[~random]\nr=np.random.rand(daval.shape[0])<0.15\ndavalx=daval.loc[r]\ndavalx.drop('label',axis=1,inplace=True)\ndavaly=daval.loc[r,'label']\ndavalx.reset_index(drop=True,inplace=True)\ndavaly=davaly.reset_index(drop=True)\n#data for model selection\n#model selection data 20% of validation\nrt=np.random.rand(davalx.shape[0])<0.20\nmodtestx=davalx.loc[rt]\nmodtesty=davaly.loc[rt]\nmodtestx.reset_index(drop=True,inplace=True)\nmodtesty=modtesty.reset_index(drop=True)\nprint(\"Training Data:\")\nprint(datrainx.shape)\nprint(datrainy.shape)\nprint(\"Validation Data:\")\nprint(davalx.shape)\nprint(davaly.shape)\nprint(\"Model Selection Data:\")\nprint(modtestx.shape)\nprint(modtesty.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, i created a function to reshape the data to feed the algorithm, 28,28 per sample for the mlp and 28,28,1 for the CNN. We also need to scale the data for better performance (you could try using the original data and compare the performance vs the scaled one) by dividing it with 255 (minmax scaling)."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def shapeshift(a):\n    return np.array(a).reshape(a.shape[0],28,28)\ndef shapeshiftconv(a):\n    return np.array(a).reshape(a.shape[0],28,28,1)\n#change the shape of the data to reflect the image's characteristic, conv for those that will be fed to\n#CNNs (28,28,1)\ndatatest=shapeshiftconv(datates)\ndatrainx=shapeshift(datrainx)\ndavalx=shapeshift(davalx)\nmodtestx=shapeshift(modtestx)\ndatrainxconv=shapeshiftconv(datrainx)\ndavalxconv=shapeshiftconv(davalx)\nmodtestxconv=shapeshiftconv(modtestx)\nprint(\"Model Selection Data:\")\nprint(modtestx.shape)\nprint(modtestxconv.shape)\nprint(\"Training Data:\")\nprint(datrainxconv.shape)\nprint(\"Validation Data:\")\nprint(davalxconv.shape)\nprint(\"Label of Image:\")\nplt.imshow(datrainx[3],cmap=mpl.cm.binary)\nprint(datrainy[3])\nplt.show()\ndatatest=datatest/255\ndatrainx=datrainx/255\ndavalx=davalx/255\nmodtestx=modtestx/255\nmodtestxconv=modtestxconv/255\ndatrainxconv=datrainxconv/255\nprint(davalxconv.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next is the construction of the NN. I tried to keep the number of layers/ parameter as low as possible (or at least until a level that i found acceptable) while minding the accuracy. Some simpler NN could perform better than the deeper/more complex ones as they tend to generalize better. I tested three models in this section: a MLP, CNN with alternating convolutional and pooling layer, and a CNN with double convolutional layers as the first layers. I tried it several times using different parameters and number of neurons, during which some more complex models with more neurons performed worse than simpler ones."},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nimport tensorflow.keras as keras\nfrom sklearn.model_selection import train_test_split\n\n\nmodel=keras.models.Sequential()\nmodel.add(keras.layers.Flatten(input_shape=[28,28]))\nmodel.add(keras.layers.Dense(200,activation='relu'))\nmodel.add(keras.layers.Dense(200,activation='relu'))\nmodel.add(keras.layers.Dense(100,activation='relu'))\nmodel.add(keras.layers.Dense(10,activation='softmax'))\nmodel.summary()\n\nmodel1=keras.models.Sequential()\nmodel1.add(keras.layers.Conv2D(64,(4,4),activation='relu',input_shape=[28,28,1],data_format='channels_last'))\nmodel1.add(keras.layers.MaxPool2D((2,2)))\nmodel1.add(keras.layers.Conv2D(128,(4,4),activation='relu'))\nmodel1.add(keras.layers.MaxPool2D((2,2)))\nmodel1.add(keras.layers.Flatten())\nmodel1.add(keras.layers.Dense(100,activation='relu'))\nmodel1.add(keras.layers.Dense(10,activation='softmax'))\nmodel1.summary()\n\nmodel2=keras.models.Sequential()\nmodel2.add(keras.layers.Conv2D(32,(4,4),activation='relu',input_shape=[28,28,1],data_format='channels_last'))\nmodel2.add(keras.layers.Conv2D(64,(4,4),activation='relu'))\nmodel2.add(keras.layers.MaxPool2D((2,2)))\nmodel2.add(keras.layers.Conv2D(48,(4,4),activation='relu'))\nmodel2.add(keras.layers.MaxPool2D((2,2)))\nmodel2.add(keras.layers.Flatten())\nmodel2.add(keras.layers.Dense(200,activation='relu'))\nmodel2.add(keras.layers.Dense(10,activation='softmax'))\nmodel2.summary()\n\nmodel.compile(loss='sparse_categorical_crossentropy',optimizer='sgd',metrics=['accuracy'])\nmodel1.compile(loss='sparse_categorical_crossentropy',optimizer='sgd',metrics=['accuracy'])\nmodel2.compile(loss='sparse_categorical_crossentropy',optimizer='sgd',metrics=['accuracy'])\nhisnn=model.fit(modtestx[:700],modtesty[:700],epochs=30,validation_data=(modtestx[701:],modtesty[701:]))\nhisconv=model1.fit(modtestxconv[:700],modtesty[:700],epochs=30,validation_data=(modtestxconv[701:],modtesty[701:]))\nhisconv2=model2.fit(modtestxconv[:700],modtesty[:700],epochs=30,validation_data=(modtestxconv[701:],modtesty[701:]))\nx=range(30)\nplt.plot(x,hisnn.history['loss'],color='r',label='mlp')\nplt.plot(x,hisconv.history['loss'],color='b',label='Conv')\nplt.plot(x,hisconv2.history['loss'],color='g',label='Doubleconv')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, the model with double convolutional layers outperforms the other two in terms of accuracy although it took far longer to train than the other 2. I chose the double convolutional CNN as our model. The last step is to train the model on the training data and then predict the result."},{"metadata":{"trusted":true},"cell_type":"code","source":"#reinitializing the model\nmodel2.compile(loss='sparse_categorical_crossentropy',optimizer='sgd',metrics=['accuracy'])\nmodel2.fit(datrainxconv,datrainy,epochs=30,validation_data=(davalxconv,davaly))\n\nresult=pd.DataFrame({'ImageId':range(1,datates.shape[0]+1)})\nresult['Label']=model2.predict_classes(datatest)\nresult.set_index('ImageId',inplace=True)\nprint(result)\nresult.to_csv('result.csv')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}