{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Let's classify some numbers"},{"metadata":{},"cell_type":"markdown","source":"First thing first, upload data into two dataframes. Run code below."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"test = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\ntrain = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\nw = pd.read_csv('/kaggle/input/digit-recognizer/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check the data out, see what we are working with."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Let's look at test data first. This is what we will use to test our model.\n\ntest.head() #We have 5 rows and 784 columns. The column 'label' appears to be missing.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head() # Train appears to have the column label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.shape # It appears we have 28,000 rows of data and 784 columns. For test data.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape # We have 42,000 and 785 columns for trian dataframe","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interesting, the 'test' data has only 784 columns, while the 'train' data has 785 columns. Let's compare 'test.head()' to 'train.head()'"},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Okay, so it appears 'test' data is missing the column 'label'. The code below appears to show that the dataframe 'w' may hold the column 'label'."},{"metadata":{"trusted":true},"cell_type":"code","source":"w.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"w.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interesting, so the number of rows (observations) in dataframe 'w' equals the number of rows (observations) in 'test' dataframe. Let's take a look at the two columns in dataframe 'w' \"ImageId\" and \"Label\" and see what values they take. We will use the .unique() method."},{"metadata":{"trusted":true},"cell_type":"code","source":"w['ImageId'].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"w['Label'].unique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interesting, so the column 'Label' in dataframe (df) 'w' only have values of 0. Something weird is going on. So first, let's build our model from the 'train' data set. First thing first, we will want to split our 'train' dataset into a training set and a test set.\n\n* Note, since the original 'test' df does not appear to have the column 'label', at this point we will use the 'train' data set to build our model."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nfrom sklearn.model_selection import train_test_split\n\nX = train.drop(columns=['label'])\ny = train['label']\n\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=40)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First, we split our 'train' df into two differnt df's. Dataframe (df) 'X' is a df containing of all columns except 'label'. DF 'y' is a df consisting of only column 'label'.\n\nWe then split our 'X' and 'y' into 'X_train', 'X_test', 'y_train', and 'y_test'. We will use 'X_train' and 'y_train' df's to train our model. We will use 'X_test' and 'y_test' to test our model (aka see how well the model does). \n\nBelow we will produce the first 5 rows of each dataframe to make sure the X_train, X_test, y_train, and y_test worked."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So it appears the code worked. Next, that's plot a barchart of 'y_test' and 'y_train'. Make sure we have a even representation of digits in both dfs."},{"metadata":{"trusted":true},"cell_type":"code","source":"spread_of_y_test = y_test.value_counts(ascending=True)\n\nspread_of_y_test = dict(spread_of_y_test)\nprint(spread_of_y_test)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"spread_of_y_train = y_train.value_counts(ascending=True)\n\nspread_of_y_train = dict(spread_of_y_train)\nprint(spread_of_y_train)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.bar(spread_of_y_test.keys(), spread_of_y_test.values(), color='g')\n('Spread of digitis 0 - 9 for Test Data')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.bar(spread_of_y_train.keys(), spread_of_y_train.values(), color='b')\nplt.title('Spread of digitis 0 - 9 for Train Data')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the two, above bar graphs, it does appear that the spread of the digits 0-9 are consistent between 'y_test' and 'y_train'.\n\nOk cool, so now we have our data that we are going to use to \"train\" our model ('X_trian' and 'y_train') and we have our data that we are going to use to \"test\" our model ('X_test' and 'y_test')."},{"metadata":{},"cell_type":"markdown","source":"So what are we actualy trying to do in this problem set and with this code? Well, we should remember that this is a 'multi-classification' problem. In short, we want to be able to use the columns in X_train df, (and respectively the X_test df) to predict if a digit is a '0', '1', '2', '3', '4' , '5', '6', '7', '8', or '9'. As you can seem this is multi-classification."},{"metadata":{},"cell_type":"markdown","source":"Below, let's get a better sense of what we are trying to do. I have displayed two images below. (One image is from 'training' dataset, the other from the 'testing' dataset.\n\nYou can see, once all the pixels are combined together, we form an image that looks a digit between 0 -9."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train_2d = X_train.values.reshape(-1,28,28)\n\nplt.imshow(X_train_2d[25]) # looks like the image below is the number three\n\nprint('When combining all the pixels together, it looks like we produce the number 3')\n\n\nprint('The corresponding value in y_train is: ' + str(y_train[25]))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Alright cool, let's see if the same holds true for the 'test' data set. Note I will be using the same exact code, but this time just pulling from the 'test' data set, instead of the 'train' data set."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test_2d = X_test.values.reshape(-1,28,28) #Note my code changed from 'X_train' to 'X_test'\n\nplt.imshow(X_test_2d[24]) # looks like the image below is the number three\n\nprint('When combining all the pixels together, it looks like we produce the number 9')\n\n\nlist_y_test = list(y_test)\n\nprint('The corresponding value in y_test is: ' + str(list_y_test[24]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Great, so now that we can visulize what we are trying to do, what type of model should we use to train digit recognition? \n\nLet's remind ourselves on what we are trying to do:\n\n1) We have y_train & X_train dataframes, which we will use to train our model.\n\n2) We have y_test & X_test dataframes, which we will use to test out model.\n\n3) We have 784 columns (features) of pixel data, when combined together, form a solid image of an integer (0-9).\n\n4) We want to train a moodel on these 784 columns to predict the image.\n\n5) Remember, this is a multiclassification problem, not binary. "},{"metadata":{},"cell_type":"markdown","source":"### Convolutional Neural Networks - CNN's"},{"metadata":{},"cell_type":"markdown","source":"The basic premise behind Convolutional Neural Networks (CNN), is the following. A CNN consists of several layers, like floors of an office building.  Each layer of a CNN consists of neurons, which will 'fire' (aka produce a certain output) if a particular threshold is met. Another way to think about it, say you are in the office building and in an elevator delivering a package, you can't remember what floor you are suppose to get off at in this 100 story building, all you do remeber is that the floor number, where you are suppose to get off at, started with a '2'. Having just this information (input) you would skip floors '10', '17', '33', etc, since they do not start with a '2'. In short, each floor can be seen as a neuron, and you, being a time constraint individual, will only get off at floors which start with a '2'. So floors (neurons) '22', '23' will fire, neurons (floors) '33', '55' will not fire. "},{"metadata":{},"cell_type":"markdown","source":"So, as one individual, in our model you would consist of one layer, with '100' possible neurons that could or could not fire. So, wanting to increase your accuracy and speed at which you can find the correct floors to deliver the packages, you invite a friend to help. This friend would add another layer, to our model. But before your friend starts to explore the '100' floor office building, he/she waits for radio communication from you on what you have learned. \n\nThis is what makes the model 'sequential'. Each person (layer) utilizes there neurons to deliver input (information) to the next layer of neurons (individual)."},{"metadata":{},"cell_type":"markdown","source":"# Model 1"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport tensorflow as tf\nimport keras\nimport matplotlib.pyplot as plt\n\ntf.keras.backend.clear_session()\n\ninput_shape = []\ninput_shape.append(len(X_train.columns))\n\n#Configure the model\nmodel_1 = tf.keras.models.Sequential([tf.keras.layers.Flatten(), \n                                    tf.keras.layers.Dense(128, activation=tf.nn.relu, input_shape = input_shape),\n                                    tf.keras.layers.Dense(64, activation=tf.nn.relu), \n                                    tf.keras.layers.Dense(32, activation=tf.nn.relu), \n                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)]) #Softmax is the function we use for multi-classification\n\n\nmodel_1.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['accuracy'])\nmodel_1.fit(X_train,y_train, epochs=10)\n\n\n#Remember now we need to test the model\n\nprint(\" \")\nprint('As you can see, in our simple first model, accuracy increases with each epoch')\n\n#Here are some important parameters for Sequential models\n#1 -- 'input_shape' is how many columns (features) of information our first layer can expect. Since we are using pixels to create images of digits we will use all\n# 784 columns (features) of data. That is why you see the code 'input_shape = len(X_train.columns)'.'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see above, with each epoch, which basically means iteration, our model improved in correctly identifying an image of a digit. Our simple model was able to increase all the way to 97% accuracy! Crazy. However, remember, this is only our 'training' data, we now need to see how the model performs on data it has never seen before, aka the 'test' data."},{"metadata":{"trusted":true},"cell_type":"code","source":"model_1.evaluate(X_test, y_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Okay, so on the test dataset, we were able to predict the correct image ~ 95% of the time. Not bad, but defintely lower then ~97% from our training data.\n\nMore importantly, when you look at the two visuals below, it appears our model is..."},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model_1.fit(X_train, y_train,\n                    epochs=10,\n                    validation_data=(X_test, y_test))\n\n\n# Plot training & validation accuracy values\nplt.plot(history.history['accuracy'])\nplt.plot(history.history['val_accuracy'])\nplt.title('Model accuracy')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Test'], loc='upper left')\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('Model loss')\nplt.ylabel('Loss')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Test'], loc='upper left')\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}