{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Digit Recognizer**\n\n---\n\n## Introduction\n\nIn this notebook i'll implement the classifier which can recognize the digits with much higher accuracy. In this notebook i'll try to follow *Fucntional Programming* as much as i can becuase it help to make more sense to the code reader. Data given in the CSV form which dimentions of `(-1, 785)` for training data and `(-1, 784)` for testing models.\n<br>\nThis notebook will divided into many small sections following the steps suggested by Data Scientist community like visualization, preprocessing etc.","metadata":{}},{"cell_type":"markdown","source":"## 1. Importing Libraries\n---\nIn this section we import the libraries required for our task this section need not to be fully defined before starting task, you can edit this section as many times as you want try to import different libraries for same task to broaden your understandings.","metadata":{}},{"cell_type":"code","source":"# Helping Libraries\nimport os\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# Specific Libraries\nimport tensorflow as tf\nimport tensorflow.keras as keras","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-06-30T09:30:05.810734Z","iopub.execute_input":"2021-06-30T09:30:05.811100Z","iopub.status.idle":"2021-06-30T09:30:07.566314Z","shell.execute_reply.started":"2021-06-30T09:30:05.811047Z","shell.execute_reply":"2021-06-30T09:30:07.565443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#This code-block is for configuring notebook and imported library behaviours\nsns.set_style(\"darkgrid\")\nsns.set_context('talk')","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:07.571975Z","iopub.execute_input":"2021-06-30T09:30:07.572332Z","iopub.status.idle":"2021-06-30T09:30:07.577090Z","shell.execute_reply.started":"2021-06-30T09:30:07.572297Z","shell.execute_reply":"2021-06-30T09:30:07.575944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Data Preprocessing\n---\nThis section can also be divided into many smaller sub-sections as per suite the requirements of given task with given dataset at given time. This sub-sections be like:\n- Data Prepration\n- Data Visualization\n- Data Preprocessing for Desired model","metadata":{}},{"cell_type":"markdown","source":"#### A. Data Prepration","metadata":{"execution":{"iopub.status.busy":"2021-06-29T09:06:50.377153Z","iopub.execute_input":"2021-06-29T09:06:50.377518Z","iopub.status.idle":"2021-06-29T09:06:50.381646Z","shell.execute_reply.started":"2021-06-29T09:06:50.377489Z","shell.execute_reply":"2021-06-29T09:06:50.380839Z"}}},{"cell_type":"code","source":"# This var is used to store location of training and testing data.\nDIRS = {\"training\": '/kaggle/input/digit-recognizer/train.csv',\"testing\": '/kaggle/input/digit-recognizer/test.csv'}","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:07.578952Z","iopub.execute_input":"2021-06-30T09:30:07.579469Z","iopub.status.idle":"2021-06-30T09:30:07.586425Z","shell.execute_reply.started":"2021-06-30T09:30:07.579430Z","shell.execute_reply":"2021-06-30T09:30:07.585603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_df(mode='training'):\n    ''' \n    Returns the loaded pd.DataFrame accoring to given mode\n    \n    Params:\n        mode: str (optional) | ['training', 'testing']\n        \n    Return:\n        df.Dataframe according to given mode.\n    '''\n    assert mode in ['training', 'testing'], \"mode should be one of the (training, testing).\"\n    if mode == 'training':\n        return pd.read_csv(DIRS['training'])\n    else:\n        return pd.read_csv(DIRS['testing'])\n    \ndef extract_and_reshape(df):\n    ''' \n    Returns converted and reshaped data to numpy\n    \n    Params:\n        df: pd.Dataframe (required)\n        \n    Return:\n        reshaped: It first convert df to numpy array then it return reshaped array into img of shape (28x28x1).\n    '''\n    assert type(df) == pd.core.frame.DataFrame , \"df should be of type pd.Dataframe\"\n    converted = np.array(df)\n    reshaped = converted.reshape(-1, 28, 28, 1)\n    print('Extracted and Reshaped to {}'.format(reshaped.shape))\n    return reshaped\n\ndef normalize_values(array):\n    ''' \n    Returns array with normalize value (0.-1.) \n    \n    Params:\n        array: np.array/np.ndarray (required)\n        \n    Return:\n        array: But with dividing every element of of array with 255.0 (highest value in image) so\n               that value lies between 0 and 1.\n    '''\n    assert type(array) == np.ndarray , \"array should be of type numpy\"\n    return array/255.0\n\ndef print_df_info(df, Keyword):\n    ''' \n    Print dataframe info\n    \n    Params:\n        df: pd.Dataframe (required)\n        keyword: str (required)\n        \n    Working:\n        Prints the info by calling df.info() with some decoration using keyword.\n    '''\n    assert type(df) == pd.core.frame.DataFrame , \"df should be of type pd.Dataframe\"\n    print('------------{}-----------'.format(Keyword.upper()))\n    print(df.info())\n    print('-------------------------------')","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:07.587701Z","iopub.execute_input":"2021-06-30T09:30:07.588114Z","iopub.status.idle":"2021-06-30T09:30:07.598340Z","shell.execute_reply.started":"2021-06-30T09:30:07.588079Z","shell.execute_reply":"2021-06-30T09:30:07.597113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = load_df(mode='training')\nprint_df_info(train_df, 'training')\n\ntest_df = load_df(mode='testing')\nprint_df_info(test_df, 'testing')","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:07.599534Z","iopub.execute_input":"2021-06-30T09:30:07.599935Z","iopub.status.idle":"2021-06-30T09:30:11.384432Z","shell.execute_reply.started":"2021-06-30T09:30:07.599898Z","shell.execute_reply":"2021-06-30T09:30:11.383576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df, labels_df = train_df.drop(columns=['label']), pd.DataFrame(train_df['label'])","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:11.385720Z","iopub.execute_input":"2021-06-30T09:30:11.386083Z","iopub.status.idle":"2021-06-30T09:30:11.469193Z","shell.execute_reply.started":"2021-06-30T09:30:11.386044Z","shell.execute_reply":"2021-06-30T09:30:11.467796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### B. Data Visualization","metadata":{}},{"cell_type":"code","source":"def plot_images(images, labels):\n    '''\n    This fuction is used to plot (images and labels)(25) in subplots\n    \n    Params:\n        images: np.ndarray (requried)\n        labesl: np.ndarray (required)\n    \n    Working:\n        It plot the images one by one in 5x5 grid with label as title of\n        respective image.\n    '''\n    f = plt.figure(figsize=(20, 20))\n    for i, image in enumerate(images):\n        f.add_subplot(5, 5, i+1)\n        plt.imshow(np.squeeze(image, axis=2), cmap='gray')\n        plt.title(labels[i])\n        plt.xticks([])\n        plt.yticks([])\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:11.470826Z","iopub.execute_input":"2021-06-30T09:30:11.471627Z","iopub.status.idle":"2021-06-30T09:30:11.478883Z","shell.execute_reply.started":"2021-06-30T09:30:11.471587Z","shell.execute_reply":"2021-06-30T09:30:11.477995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_np = extract_and_reshape(train_df)\ntrain_np = normalize_values(train_np)\n\n# Convert to numpy with removing last axis\nlabels_np = np.array(labels_df).reshape(-1)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:11.482326Z","iopub.execute_input":"2021-06-30T09:30:11.482701Z","iopub.status.idle":"2021-06-30T09:30:11.687060Z","shell.execute_reply.started":"2021-06-30T09:30:11.482672Z","shell.execute_reply":"2021-06-30T09:30:11.685995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_images(train_np[:25], labels_np)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:11.689279Z","iopub.execute_input":"2021-06-30T09:30:11.689643Z","iopub.status.idle":"2021-06-30T09:30:12.786553Z","shell.execute_reply.started":"2021-06-30T09:30:11.689602Z","shell.execute_reply":"2021-06-30T09:30:12.785597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we will plot the labels distribution for given images. As you can see in following figure we generally have around 4000 labels each.","metadata":{}},{"cell_type":"code","source":"sns.countplot(x='label', palette='inferno', data=labels_df)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:12.787904Z","iopub.execute_input":"2021-06-30T09:30:12.788273Z","iopub.status.idle":"2021-06-30T09:30:12.964137Z","shell.execute_reply.started":"2021-06-30T09:30:12.788234Z","shell.execute_reply":"2021-06-30T09:30:12.963203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### C. Data Preprocessing for Model and/or Visualization","metadata":{}},{"cell_type":"code","source":"def convert_to_tfds(ds):\n    '''\n    This fuction convert np.ndarray to tf.data.Dataset\n    '''\n    assert type(ds) == np.ndarray , \"array should be of type numpy\"\n    return tf.data.Dataset.from_tensor_slices(ds)\n\ndef make_onehot(array, num_class=10):\n    '''\n    This fuction convert labels to onehot array with max size of num_class.\n    '''\n    assert len(array.shape) == 1 or ( len(array.shape) == 2 and array.shape[-1] == 1), \"array shoulf be of 1 or 2 dimensions.\"\n    return np.eye(num_class)[array]\n\ndef onehot_to_label(value):\n    '''\n    This fuction is revesible of make_onehot function.\n    '''\n    return np.argmax(value, axis=1)\n\ndef zip_ds(datasets):\n    '''\n    This fuction zip two tf.data.Dataset into one tf.data.Dataset.\n    '''\n    assert len(datasets) == 2, \"length of datasets should be 2 (images, labels).\"\n    return tf.data.Dataset.zip(datasets)\n\ndef split_ds(ds, percentage):\n    '''\n    This fuction split ds into train_ds and val_ds for training and validating model while training.\n    \n    Params:\n        ds: tf.data.Dataset (requried)\n            the dataset which we want to split into train and val.\n        percentage: float (required) | between 0 and 1\n            percentage of the data to split as val and other to train.\n    \n    Return:\n        (train, val): (tf.data.Dataset, tf.data.Dataset)\n            splitted dataset into train and val.\n    '''\n    assert 0 < percentage < 1, \"percentage should be between 0 and 1.\"\n    ds = ds.shuffle(1000, reshuffle_each_iteration=True)\n    val_length = int(len(ds) * percentage)\n    val = ds.take(val_length)\n    train = ds.skip(val_length)   \n    return train, val\n\ndef preprocess_ds(ds, batch_size, prefetch_size):\n    '''\n    This fuction batch dataset into batch_size and prefetch batched dataset according to prefetch_size.\n    \n    Params:\n        ds: tf.data.Dataset (required)\n            dataset to be preprocess\n        batch_size: int (required)\n        prefetch_size: int(required)\n            used in ds.prefetch() which will prefetch prefetch_size batch of dataset ahead of time.\n    '''\n    ds = ds.batch(128)\n    ds = ds.prefetch(10)\n    return ds","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:12.965597Z","iopub.execute_input":"2021-06-30T09:30:12.965954Z","iopub.status.idle":"2021-06-30T09:30:12.975819Z","shell.execute_reply.started":"2021-06-30T09:30:12.965916Z","shell.execute_reply":"2021-06-30T09:30:12.974638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ds = convert_to_tfds(train_np)\nlabels_ds = convert_to_tfds(make_onehot(labels_np))\n\nds_train = zip_ds((train_ds, labels_ds))\nds_train, ds_val = split_ds(ds_train, 0.15)\n\nbatch_size = 128\nprefetch_size = 10\nds_train, ds_val = preprocess_ds(ds_train, batch_size, prefetch_size), preprocess_ds(ds_val, batch_size, prefetch_size)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:12.977280Z","iopub.execute_input":"2021-06-30T09:30:12.977655Z","iopub.status.idle":"2021-06-30T09:30:13.922114Z","shell.execute_reply.started":"2021-06-30T09:30:12.977618Z","shell.execute_reply":"2021-06-30T09:30:13.921220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch = next(iter(ds_train))\nimages, labels = batch[0].numpy()[:25], batch[1].numpy()[:25]\nlabels = onehot_to_label(labels)\nplot_images(images, labels)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:13.926303Z","iopub.execute_input":"2021-06-30T09:30:13.928307Z","iopub.status.idle":"2021-06-30T09:30:15.319322Z","shell.execute_reply.started":"2021-06-30T09:30:13.928266Z","shell.execute_reply":"2021-06-30T09:30:15.318524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Model Training\n---\nWe can also split this section in other smaller sections like:\n- Model Definition\n- Model Declaration\n- Model Training with validation\n- Model Learning's Visualization","metadata":{}},{"cell_type":"markdown","source":"#### A. Model Definition","metadata":{}},{"cell_type":"code","source":"def cnn_model(input_shape):\n    model = keras.models.Sequential()\n    \n    model.add(keras.layers.Conv2D(32, kernel_size=3, padding='valid', use_bias=False, input_shape=input_shape, activation='elu'))\n    model.add(keras.layers.MaxPool2D(2))\n    model.add(keras.layers.BatchNormalization())\n    model.add(keras.layers.Dropout(rate=0.3))\n    \n    model.add(keras.layers.Conv2D(64, kernel_size=3, padding='valid', use_bias=False, activation='elu'))\n    model.add(keras.layers.MaxPool2D(2))\n    model.add(keras.layers.BatchNormalization())\n    model.add(keras.layers.Dropout(rate=0.3))\n    \n    model.add(keras.layers.Flatten())\n    model.add(keras.layers.Dense(256, activation='relu'))\n    model.add(keras.layers.Dropout(rate=0.2))\n    \n    model.add(keras.layers.Dense(10, activation='softmax'))\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:15.320692Z","iopub.execute_input":"2021-06-30T09:30:15.321039Z","iopub.status.idle":"2021-06-30T09:30:15.329706Z","shell.execute_reply.started":"2021-06-30T09:30:15.321002Z","shell.execute_reply":"2021-06-30T09:30:15.328660Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### B. Model Declaration\n---\nHere we also compiling model (`cnn.compile()`) which is required and specific to tf.keras. ","metadata":{}},{"cell_type":"code","source":"cnn = cnn_model((28, 28, 1))\ncnn.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:15.331111Z","iopub.execute_input":"2021-06-30T09:30:15.331686Z","iopub.status.idle":"2021-06-30T09:30:15.897730Z","shell.execute_reply.started":"2021-06-30T09:30:15.331647Z","shell.execute_reply":"2021-06-30T09:30:15.896762Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### C. Model Training with Validation","metadata":{}},{"cell_type":"code","source":"early_stopper = tf.keras.callbacks.EarlyStopping(\n    monitor='val_loss', min_delta=0, patience=3, verbose=0,\n    mode='auto', restore_best_weights=True)\n\nhistory = cnn.fit(ds_train, validation_data=ds_val, epochs=10, callbacks=[early_stopper])","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:15.898990Z","iopub.execute_input":"2021-06-30T09:30:15.899348Z","iopub.status.idle":"2021-06-30T09:30:37.068040Z","shell.execute_reply.started":"2021-06-30T09:30:15.899312Z","shell.execute_reply":"2021-06-30T09:30:37.067119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### D. Visualization of Model's Learning","metadata":{}},{"cell_type":"code","source":"pd.DataFrame.from_dict(history.history)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:37.069487Z","iopub.execute_input":"2021-06-30T09:30:37.069838Z","iopub.status.idle":"2021-06-30T09:30:37.087136Z","shell.execute_reply.started":"2021-06-30T09:30:37.069800Z","shell.execute_reply":"2021-06-30T09:30:37.085971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_history(history):\n    '''\n    This fuction is used to plot history generated by model.fit()\n    \n    Params:\n        history: tf.keras.callback.History(required)\n    \n    Working:\n        If you notice we plotting from 1:10 rather than 0:10 this is due to at the end of first epoch \n        model take high jump in loss and accuracy (you can see that in above cell output notice first \n        row loss and accuracy comparision to val_loss and val_acc respectively) which will increase \n        the range in Y-axis leads to less understandable plots.\n    '''\n    assert type(history) == tf.keras.callbacks.History, \"please use history callback generated by model.fit() call.\"\n    epochs = history.params['epochs']\n    to_plot = ['loss', 'accuracy']\n    \n    #plotting\n    f= plt.figure(figsize=(16, 8))\n    history = history.history\n    for i, value in enumerate(to_plot):\n        #######\n        # Required vars\n        #######\n        f.add_subplot(1, 2, i+1)\n        val_value = r'val_{}'.format(value)\n        range_ = np.arange(epochs-1)\n        #######\n        # Plotting values\n        #######\n        plt.plot(range_, history[value][1:], label=value)\n        plt.plot(range_, history[val_value][1:], label=val_value)\n        plt.legend(loc=r\"{} right\".format('upper' if value=='loss' else 'lower'))\n    plt.plot()\n    \n    return epochs\n\nplot_history(history)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:37.088661Z","iopub.execute_input":"2021-06-30T09:30:37.088998Z","iopub.status.idle":"2021-06-30T09:30:37.429057Z","shell.execute_reply.started":"2021-06-30T09:30:37.088963Z","shell.execute_reply":"2021-06-30T09:30:37.428290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4. Model Inference\n- Dataset Prepation\n- Prediction Visualization","metadata":{}},{"cell_type":"markdown","source":"#### A. Dataset Prepration","metadata":{}},{"cell_type":"code","source":"test_np = extract_and_reshape(test_df)\ntest_ds = convert_to_tfds(test_np)\ntest_ds = preprocess_ds(test_ds, batch_size, prefetch_size)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:37.430118Z","iopub.execute_input":"2021-06-30T09:30:37.430482Z","iopub.status.idle":"2021-06-30T09:30:37.697288Z","shell.execute_reply.started":"2021-06-30T09:30:37.430445Z","shell.execute_reply":"2021-06-30T09:30:37.696308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### B. Prediction Visualization","metadata":{}},{"cell_type":"code","source":"batch_iter = iter(test_ds)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:37.698559Z","iopub.execute_input":"2021-06-30T09:30:37.699043Z","iopub.status.idle":"2021-06-30T09:30:37.826944Z","shell.execute_reply.started":"2021-06-30T09:30:37.699002Z","shell.execute_reply":"2021-06-30T09:30:37.825964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch = next(batch_iter)\nimages = batch.numpy()\nlabels = onehot_to_label(cnn.predict(batch))\nplot_images(images[:25], labels[:25])","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:30:37.828290Z","iopub.execute_input":"2021-06-30T09:30:37.828776Z","iopub.status.idle":"2021-06-30T09:30:39.140888Z","shell.execute_reply.started":"2021-06-30T09:30:37.828735Z","shell.execute_reply":"2021-06-30T09:30:39.140049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prediction Submission","metadata":{}},{"cell_type":"code","source":"predictions = cnn.predict(test_ds)\npredictions = onehot_to_label(predictions)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:33:01.986339Z","iopub.execute_input":"2021-06-30T09:33:01.986680Z","iopub.status.idle":"2021-06-30T09:33:02.620048Z","shell.execute_reply.started":"2021-06-30T09:33:01.986649Z","shell.execute_reply":"2021-06-30T09:33:02.619150Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df = pd.DataFrame.from_dict({'ImageId': np.arange(1, len(predictions)+1), 'Label': predictions})","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:35:56.285749Z","iopub.execute_input":"2021-06-30T09:35:56.286071Z","iopub.status.idle":"2021-06-30T09:35:56.291114Z","shell.execute_reply.started":"2021-06-30T09:35:56.286041Z","shell.execute_reply":"2021-06-30T09:35:56.290293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.to_csv('./submission.csv', index = False)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T09:37:18.967478Z","iopub.execute_input":"2021-06-30T09:37:18.967840Z","iopub.status.idle":"2021-06-30T09:37:19.019205Z","shell.execute_reply.started":"2021-06-30T09:37:18.967808Z","shell.execute_reply":"2021-06-30T09:37:19.018422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}