{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Steps in Creating a Data Science Project\n\n* Defining the project\n* Preparing the data\n* Exploratory Data Analysis & Preprocessing\n* Creating a machine learning model\n* Predictions\n* Presenting your findings","metadata":{"id":"xrtsNeb0TTtv"}},{"cell_type":"markdown","source":"## 1. Defining the project\nMNIST (\"Modified National Institute of Standards and Technology\") is the de facto “hello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.\n\nIn this problem, our goal is to correctly identify digits from a dataset of tens of thousands of handwritten images.","metadata":{"id":"gVA8LhLXXES0"}},{"cell_type":"markdown","source":"##2. Preparing the data","metadata":{"id":"n3JBT0TGXZGg"}},{"cell_type":"code","source":"#Import libraires\n\nimport tensorflow as tf\n#tf.__version__\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Ignore the warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# display all dataframe columns & rows\npd.options.display.max_columns = None\npd.options.display.max_rows = None\n\n# to set the limit to 3 decimals\npd.options.display.float_format = '{:.7f}'.format\n","metadata":{"id":"VTvvjzynYC8S","execution":{"iopub.status.busy":"2021-09-20T08:28:33.904266Z","iopub.execute_input":"2021-09-20T08:28:33.904630Z","iopub.status.idle":"2021-09-20T08:28:40.932049Z","shell.execute_reply.started":"2021-09-20T08:28:33.904535Z","shell.execute_reply":"2021-09-20T08:28:40.931309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The MNIST database contains 60,000 training images and 10,000 testing images taken from American Census Bureau employees and American high school students. ","metadata":{"id":"Zkpv_6aDXxyw"}},{"cell_type":"code","source":"train = pd.read_csv('../input/digit-recognizer/train.csv')\ntrain.head()","metadata":{"id":"FRkmbBxcY1S_","outputId":"c4885723-6425-4253-e869-115cc6f7e8a9","execution":{"iopub.status.busy":"2021-09-20T08:28:42.723532Z","iopub.execute_input":"2021-09-20T08:28:42.724606Z","iopub.status.idle":"2021-09-20T08:28:46.831110Z","shell.execute_reply.started":"2021-09-20T08:28:42.724552Z","shell.execute_reply":"2021-09-20T08:28:46.830018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = pd.read_csv('../input/digit-recognizer/test.csv')\ntest.head()","metadata":{"id":"WDsepKZAabHJ","outputId":"e4edae0a-7d9e-4810-9a40-9ff807b15b6b","execution":{"iopub.status.busy":"2021-09-20T08:28:55.840434Z","iopub.execute_input":"2021-09-20T08:28:55.841606Z","iopub.status.idle":"2021-09-20T08:28:58.238792Z","shell.execute_reply.started":"2021-09-20T08:28:55.841551Z","shell.execute_reply":"2021-09-20T08:28:58.237744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Exploratory Data Analysis & Preprocessing","metadata":{"id":"EyOfOzZAYk2F"}},{"cell_type":"code","source":"train.describe()","metadata":{"id":"d4GILuPdIdEh","outputId":"f77c0dad-5075-4ebf-b8e1-ebe32b4c4033","execution":{"iopub.status.busy":"2021-09-20T08:29:19.741630Z","iopub.execute_input":"2021-09-20T08:29:19.742047Z","iopub.status.idle":"2021-09-20T08:29:22.695842Z","shell.execute_reply.started":"2021-09-20T08:29:19.742010Z","shell.execute_reply":"2021-09-20T08:29:22.694865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10,7))\nsns.countplot(train['label'])\nplt.tight_layout(); #for better visualization","metadata":{"id":"FOyq23_MZE5c","outputId":"b4fb4257-7e14-4854-b51c-a4107dbdb23e","execution":{"iopub.status.busy":"2021-09-20T08:29:32.008179Z","iopub.execute_input":"2021-09-20T08:29:32.008523Z","iopub.status.idle":"2021-09-20T08:29:32.285401Z","shell.execute_reply.started":"2021-09-20T08:29:32.008485Z","shell.execute_reply":"2021-09-20T08:29:32.284437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nplt.figure(figsize=(12,6))\n#distribution plot\nsns.distplot(train['label'], axlabel=\"Labels\", color='red')","metadata":{"id":"qc9PJtpRNqEk","outputId":"99b0ec29-a279-4af8-c421-d4e326f67611","execution":{"iopub.status.busy":"2021-09-20T08:30:24.207146Z","iopub.execute_input":"2021-09-20T08:30:24.207942Z","iopub.status.idle":"2021-09-20T08:30:24.901003Z","shell.execute_reply.started":"2021-09-20T08:30:24.207890Z","shell.execute_reply":"2021-09-20T08:30:24.899828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's extract the features for train and test to be used while modeling","metadata":{}},{"cell_type":"code","source":"# Extract features\nfeatures = train.drop('label', axis=1)\n\n# Extract label\ny_train = train['label']\n\n# Train images\nX_ = np.array(features)\nX_train = X_.reshape(X_.shape[0], 28, 28)\n\n# Test images\nX_test = np.array(test)","metadata":{"id":"8vmzmr89b_2w","execution":{"iopub.status.busy":"2021-09-20T08:31:39.349186Z","iopub.execute_input":"2021-09-20T08:31:39.349545Z","iopub.status.idle":"2021-09-20T08:31:39.651575Z","shell.execute_reply.started":"2021-09-20T08:31:39.349512Z","shell.execute_reply":"2021-09-20T08:31:39.650702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To see the image associated with the index number use ``imshow()``,  with   ``X_train[]``","metadata":{"id":"CPTbqytvG3Eb"}},{"cell_type":"code","source":"plt.imshow(X_train[0])\nplt.colorbar()\nplt.grid(False)\nplt.show()","metadata":{"id":"kgiT_lpAljG6","outputId":"6cb60a26-9525-44f9-fc33-d5b5ad22e7c1","execution":{"iopub.status.busy":"2021-09-20T08:31:48.998631Z","iopub.execute_input":"2021-09-20T08:31:48.999436Z","iopub.status.idle":"2021-09-20T08:31:49.308843Z","shell.execute_reply.started":"2021-09-20T08:31:48.999391Z","shell.execute_reply":"2021-09-20T08:31:49.307805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To verify that the data is in the correct format and that you're ready to build and train the network, let's display the first 16 images from the training set.","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(10,5))\n\nfor i in range(16):\n  fig.add_subplot(4,4, i+1)\n  plt.xticks([])\n  plt.yticks([])\n  plt.imshow(X_train[i], cmap='gray')\n  plt.xlabel('Digit: ' + str(y_train[i]))\n  plt.tight_layout(); # to see clear graph\nplt.show();\n","metadata":{"id":"ZptRPjCLlD-8","outputId":"4970db6f-1a68-4edd-d5d7-3bac39d746c2","execution":{"iopub.status.busy":"2021-09-20T08:32:19.467191Z","iopub.execute_input":"2021-09-20T08:32:19.467521Z","iopub.status.idle":"2021-09-20T08:32:21.517292Z","shell.execute_reply.started":"2021-09-20T08:32:19.467490Z","shell.execute_reply":"2021-09-20T08:32:21.516204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check the shape of train and test set.","metadata":{}},{"cell_type":"code","source":"X_train.shape, X_test.shape","metadata":{"id":"1NdhwWLO7bT2","outputId":"ae573eed-815a-448c-fb05-cd5b68364d3d","execution":{"iopub.status.busy":"2021-09-20T08:32:50.577590Z","iopub.execute_input":"2021-09-20T08:32:50.578270Z","iopub.status.idle":"2021-09-20T08:32:50.584240Z","shell.execute_reply.started":"2021-09-20T08:32:50.578216Z","shell.execute_reply":"2021-09-20T08:32:50.583628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Our data is not in the same shape so here there is a need to reshape our train and test data respectively. \nEach image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels. (28*28 = 784)","metadata":{"id":"MAcOl8O78Es9"}},{"cell_type":"code","source":"X_train = X_train.reshape(X_train.shape[0], 28, 28,1)\nX_test = X_test.reshape(X_test.shape[0], 28, 28,1)","metadata":{"id":"k31NHdGt8ucm","execution":{"iopub.status.busy":"2021-09-20T08:33:26.015390Z","iopub.execute_input":"2021-09-20T08:33:26.016437Z","iopub.status.idle":"2021-09-20T08:33:26.021411Z","shell.execute_reply.started":"2021-09-20T08:33:26.016398Z","shell.execute_reply":"2021-09-20T08:33:26.020579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# lets check the shape again\nX_train.shape, X_test.shape","metadata":{"id":"ZME1Ai8B9LN1","outputId":"3dbd7ce0-e564-40d1-b470-3427cda45883","execution":{"iopub.status.busy":"2021-09-20T08:33:29.581444Z","iopub.execute_input":"2021-09-20T08:33:29.582278Z","iopub.status.idle":"2021-09-20T08:33:29.589894Z","shell.execute_reply.started":"2021-09-20T08:33:29.582231Z","shell.execute_reply":"2021-09-20T08:33:29.588728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now its time to scale, normalize in other words, our data. \n> To do so we need to divide the train and test dataset with pixel-value as an integer between 0 and 255, inclusive. (In simple words dividing with maximum number to bring them in a format of 0 and 1)","metadata":{"id":"vXEq1DwT9Q9F"}},{"cell_type":"code","source":"X_train.min(), X_train.max()","metadata":{"id":"XvN_0GZ59wVI","outputId":"1cf4047b-1325-40d7-c586-6e80bfd32d66","execution":{"iopub.status.busy":"2021-09-20T08:33:41.896124Z","iopub.execute_input":"2021-09-20T08:33:41.896472Z","iopub.status.idle":"2021-09-20T08:33:41.978543Z","shell.execute_reply.started":"2021-09-20T08:33:41.896427Z","shell.execute_reply":"2021-09-20T08:33:41.977561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test.min(), X_train.max()","metadata":{"id":"bzhn-xzU93bg","outputId":"c1053bf1-a88f-4a74-f9bb-68da640d8adb","execution":{"iopub.status.busy":"2021-09-20T08:33:44.979471Z","iopub.execute_input":"2021-09-20T08:33:44.979778Z","iopub.status.idle":"2021-09-20T08:33:45.045722Z","shell.execute_reply.started":"2021-09-20T08:33:44.979747Z","shell.execute_reply":"2021-09-20T08:33:45.044767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train = X_train/255.\nX_test = X_test/255.","metadata":{"id":"SM-8zRlJ-Nwy","execution":{"iopub.status.busy":"2021-09-20T08:33:47.622800Z","iopub.execute_input":"2021-09-20T08:33:47.623097Z","iopub.status.idle":"2021-09-20T08:33:47.806893Z","shell.execute_reply.started":"2021-09-20T08:33:47.623069Z","shell.execute_reply":"2021-09-20T08:33:47.805919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check the ``min()`` , ``max()`` of data to confirm it is now normalized. ","metadata":{"id":"Kp2B7bVr-eeB"}},{"cell_type":"code","source":"X_train.min(), X_train.max()","metadata":{"id":"s9JKzSS4-amI","outputId":"dfefbf4c-a8b3-41c0-e57e-57ebec50bbf7","execution":{"iopub.status.busy":"2021-09-20T08:33:54.253749Z","iopub.execute_input":"2021-09-20T08:33:54.254109Z","iopub.status.idle":"2021-09-20T08:33:54.309499Z","shell.execute_reply.started":"2021-09-20T08:33:54.254078Z","shell.execute_reply":"2021-09-20T08:33:54.308404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4. Creating a machine learning model","metadata":{"id":"Ryg9hfpjB0f0"}},{"cell_type":"markdown","source":"Since we are working on a binary classification problem, So lets create our neural network with tensorflow. Steps in Modeling to classify whether a circle is (Blue or Red)\nThe steps in modeling with Tensorflow are typically:\n\n* Create or import a model\n* Compile a model\n* Fit the model\n* Evaluate the model\n* Tweak\n* Evaluate.","metadata":{"id":"DS-3JFgaBbLx"}},{"cell_type":"markdown","source":"> The first layer in this network, tf.keras.layers.Flatten, transforms the format of the images from a two-dimensional array (of 28 by 28 pixels) to a one-dimensional array (of 28 * 28 = 784 pixels). Think of this layer as unstacking rows of pixels in the image and lining them up. This layer has no parameters to learn; it only reformats the data.","metadata":{"id":"o0h0PcwqCuUO"}},{"cell_type":"markdown","source":"The from_logits=True attribute inform the loss function that the output values generated by the model are not normalized, a.k.a. logits. In other words, the softmax function has not been applied on them to produce a probability distribution.","metadata":{"id":"I9RjMWy4CxGU"}},{"cell_type":"code","source":"# 1. Create the model\n\nmodel = tf.keras.Sequential([\n                             tf.keras.layers.Flatten(input_shape=(28,28)),\n                             tf.keras.layers.Dense(128, activation=\"relu\"),\n                             tf.keras.layers.Dense(10) #10 is the number of classes\n])\n\n#2 Compile the model\n\nmodel.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n              optimizer=tf.keras.optimizers.Adam(),\n              metrics=[\"accuracy\"])\n\n# 3. Fit the model\n\nhistory = model.fit(X_train, y_train, epochs=50)\n","metadata":{"id":"0SkoeUphB_Q5","outputId":"1e908dec-dc57-42ea-dd85-735e8ca72bc3","execution":{"iopub.status.busy":"2021-09-20T08:34:08.495579Z","iopub.execute_input":"2021-09-20T08:34:08.495923Z","iopub.status.idle":"2021-09-20T08:36:28.174783Z","shell.execute_reply.started":"2021-09-20T08:34:08.495867Z","shell.execute_reply":"2021-09-20T08:36:28.173621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"History variable here will be used in the next step to plot a graph of loss and accuracy parameter to see where trend is heading to.","metadata":{"id":"p2XFUlBMDg7n"}},{"cell_type":"markdown","source":"**Accuracy & Loss plot:**","metadata":{"id":"c7YIV3AWEGKP"}},{"cell_type":"code","source":"#ploting the loss and accuracy graph\n\npd.DataFrame(history.history).plot(figsize=(12,7))","metadata":{"id":"uN6nl_QWELt3","outputId":"04b90103-66ab-4ca8-f9de-af1551dbd436","execution":{"iopub.status.busy":"2021-09-20T08:37:41.091492Z","iopub.execute_input":"2021-09-20T08:37:41.092482Z","iopub.status.idle":"2021-09-20T08:37:41.473272Z","shell.execute_reply.started":"2021-09-20T08:37:41.092436Z","shell.execute_reply":"2021-09-20T08:37:41.472304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5: Predictions","metadata":{"id":"CxafDxbOCgic"}},{"cell_type":"code","source":"predictions = np.argmax(model.predict(X_test), axis=1)","metadata":{"id":"9TwIbfxdCjZ8","execution":{"iopub.status.busy":"2021-09-20T08:37:48.509154Z","iopub.execute_input":"2021-09-20T08:37:48.509431Z","iopub.status.idle":"2021-09-20T08:37:49.612688Z","shell.execute_reply.started":"2021-09-20T08:37:48.509401Z","shell.execute_reply":"2021-09-20T08:37:49.611905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions","metadata":{"id":"YCPVDC4CNtCX","outputId":"1f45114d-7f48-48c7-8e26-af742e5457b7","execution":{"iopub.status.busy":"2021-09-20T08:37:52.118507Z","iopub.execute_input":"2021-09-20T08:37:52.119195Z","iopub.status.idle":"2021-09-20T08:37:52.125337Z","shell.execute_reply.started":"2021-09-20T08:37:52.119144Z","shell.execute_reply":"2021-09-20T08:37:52.124406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Making a submission file for competition.","metadata":{}},{"cell_type":"code","source":"submission = pd.read_csv('../input/digit-recognizer/sample_submission.csv')\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-20T08:41:26.775697Z","iopub.execute_input":"2021-09-20T08:41:26.776348Z","iopub.status.idle":"2021-09-20T08:41:26.797097Z","shell.execute_reply.started":"2021-09-20T08:41:26.776300Z","shell.execute_reply":"2021-09-20T08:41:26.796231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Shape has 28000 rows of data, and 2 columns ``ImageId`` , ``Label``","metadata":{}},{"cell_type":"code","source":"submission.shape","metadata":{"execution":{"iopub.status.busy":"2021-09-20T08:39:53.879767Z","iopub.execute_input":"2021-09-20T08:39:53.880644Z","iopub.status.idle":"2021-09-20T08:39:53.886340Z","shell.execute_reply.started":"2021-09-20T08:39:53.880602Z","shell.execute_reply":"2021-09-20T08:39:53.885402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission['Label']  = predictions","metadata":{"execution":{"iopub.status.busy":"2021-09-20T08:41:50.781659Z","iopub.execute_input":"2021-09-20T08:41:50.782532Z","iopub.status.idle":"2021-09-20T08:41:50.788336Z","shell.execute_reply.started":"2021-09-20T08:41:50.782479Z","shell.execute_reply":"2021-09-20T08:41:50.787704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv',index=False)\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-20T08:43:21.640226Z","iopub.execute_input":"2021-09-20T08:43:21.640585Z","iopub.status.idle":"2021-09-20T08:43:21.712104Z","shell.execute_reply.started":"2021-09-20T08:43:21.640549Z","shell.execute_reply":"2021-09-20T08:43:21.711310Z"},"trusted":true},"execution_count":null,"outputs":[]}]}