{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Image Classification with dimentionality reduction\n    * Load data and visualize it \n    * Create Scikit learn data pipline\n    * Create tensorflow dataset\n    * Create tensorflow model\n    * Create Callbacks for the model\n    * Train model\n    * Check how model is learning during training\n    * Visualize model performance with tensorboard\n    * Do inferance on created model"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as pyplot\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.base import BaseEstimator, TransformerMixin\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.decomposition import PCA\nimport functools\nimport tensorflow as tf\nimport datetime\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n        \n\n# tf.enable_eager_execution()\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tf.__version__","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"TRAIN_PATH = '../input/digit-recognizer/train.csv'\nTEST_PATH = '../input/digit-recognizer/test.csv'\nBATCH_SIZE = 1000","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load data\nLoad the input data with pandas and seperate labels from it. It good to have training set and validation dataset, so with scikit learn ```train_test_split``` we are creating training and validation dataset with 80, 20 percent for input training data."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(TRAIN_PATH)\ntrain_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = train_df['label']\ntrain_df.drop('label', axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(train_df.values, labels.values, test_size=0.2, random_state=42)\n# X_train = train_df\n# y_train = labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualize data\nSince we are dealing with images let's plot some images from dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_img = X_train\nX_img.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_img = X_img.reshape(X_img.shape[0], 28, 28)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pyplot.imshow(X_img[0])\npyplot.title(y_train[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating data pipline\nIn this dataset we have pixel densities as training data ranging from 0-255. For better performance of our model let's do scaling first and then we will apply dimentionality reduction.Since we need to apply dimentionality reduction to all of the data, lets create a pipeline todo the same. Here we are using scikit learn's PCA to do the same. 95% of variance will give upto 153 dimentions, so let's take 169 dimentions which will be helpful for visualization as well. Using transform method of estimators, apply inverse trasform of PCA data, to get an approximation of training dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"pca = PCA(169)\nclass SqueezeDataset(BaseEstimator, TransformerMixin):\n    def __init__(self, expected_variance_ratio = 0.95):\n        self.variance_ratio = expected_variance_ratio\n        self.pca = pca\n    \n    def fit(self, X, y=None):\n        self.pca.fit(X)\n        return self\n    \n    def transform(self, X, y=None):\n        X_transformed = self.pca.transform(X)\n        return X_transformed","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Combining both estimator to create a full pipeline and transforming data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data_pipeline = Pipeline([\n    ('scale_dataset', StandardScaler()),\n    ('squeeze_dataset', SqueezeDataset())\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_processed = train_data_pipeline.fit_transform(X_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_processed.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Appying inverse transform to get an approximation of given training data after PCA"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_inverse_trans = pca.inverse_transform(X_processed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_inverse_trans = x_inverse_trans.reshape(x_inverse_trans.shape[0], 28, 28)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_inverse_trans.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visialize approximation of training data"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = pyplot.subplots(1, 3)\nax[0].imshow(x_inverse_trans[0])\nax[0].set_title('inverse transform')\nax[1].imshow(X_processed[0].reshape(13, 13))\nax[1].set_title('processed image')\nax[2].imshow(X_img[0])\nax[2].set_title('original')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create tensorflow dataset\nWith the approximation we have, let's create a dataset and an iterator, which can do shuffling, batching on out dataset.\nCreate a one shot iterator to iterate over data."},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_dataset(features, labels):\n    print(features.shape)\n    dataset = (\n        tf.data.Dataset.from_tensor_slices((features, labels))\n            .shuffle(len(labels))\n            .repeat()\n            .batch(BATCH_SIZE)\n            .prefetch(1)\n    )\n    iterator = dataset.make_one_shot_iterator()\n    return iterator","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Declare some layers with common variables to avoid repeating of code."},{"metadata":{"trusted":true},"cell_type":"code","source":"Input = tf.keras.Input\nConv2D = functools.partial(\n        tf.keras.layers.Conv2D,\n        activation='elu',\n        padding='same'\n    )\nBatchNormalization = tf.keras.layers.BatchNormalization\nDense = tf.keras.layers.Dense\nAveragePooling2D = tf.keras.layers.AveragePooling2D\nDropout = tf.keras.layers.Dropout\nFlatten = tf.keras.layers.Flatten\nL2 = functools.partial(\n        tf.keras.regularizers.l2,\n        l=0.2\n    )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating CNN with tensorflow"},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_model():\n    input = Input(shape=(28,28,1,))\n    conv1 = Conv2D(8, (3, 3))(input)\n    conv2 = Conv2D(16, (3, 3))(conv1)\n    batch_norm1 = BatchNormalization(axis=3)(conv2)\n    conv3 = Conv2D(32, (5, 5))(batch_norm1)\n    dropout = Dropout(0.3)(conv3)\n#     avg_pool_1 = AveragePooling2D((2, 2))(dropout)\n    conv4 = Conv2D(32, (5, 5))(dropout)\n    batch_norm2 = BatchNormalization(axis=3)(conv4)\n#     avg_pool_2 = AveragePooling2D(2, 2)(conv4)\n    conv5 = Conv2D(16, (7, 7))(batch_norm2)\n    conv6 = Conv2D(8, (7, 7))(conv5)\n    flt1 = Flatten()(conv6)\n    output = Dense(10, activation='softmax')(flt1)\n    model = tf.keras.Model(input, output)\n    model.compile(optimizer='adam',loss='sparse_categorical_crossentropy',metrics=['accuracy']) # Since labels are not one hot encoded using just categorial_crossentropy dosen't make sence here.\n    model.summary()\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Get iterator from prepare dataset function defined above for both training and validation datasets."},{"metadata":{"trusted":true},"cell_type":"code","source":"iterator = prepare_dataset(X_train.reshape(X_train.shape[0], 28, 28, 1), y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_iterator = prepare_dataset(X_test.reshape(X_test.shape[0], 28, 28, 1), y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = prepare_model()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Callbacks to model\nCreate callbacks to save model, visualize its performance with tensorboard, and for early stoping to avoid vanishing gradient problem"},{"metadata":{"trusted":true},"cell_type":"code","source":"logdir = os.path.join(\"logs\", datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\"))\ntensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)\ncallbacks = [\n    tf.keras.callbacks.ModelCheckpoint(filepath='../input/weights.hdf5', verbose=1, save_best_only=True),\n    tensorboard_callback,\n    tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3)\n]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training model\nTrain model using both training and valiadion iterator so that it will train and validate itself in every epoch. Set epochs to 50, if there is no improvement in learning for 3 epoch model will stop training as mentioned in callbacks."},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(iterator, steps_per_epoch=34, epochs=50,  verbose=1, callbacks=callbacks,\n                   validation_data=val_iterator, validation_steps=9)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visualize some testing data"},{"metadata":{"trusted":true},"cell_type":"code","source":"pyplot.imshow(X_test[0].reshape(28, 28))\npyplot.title(y_test[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Check how model is learning during training\nLet's check how model is learning during training. get outputs of 3rd layer from the model and try to visulaze it."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check how model is learning internally\ninputs = model.input\noutputs = [layer.output for layer in model.layers][3:]\nfunctor = tf.keras.backend.function([inputs, tf.keras.backend.learning_phase()], outputs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Testing\ntest = X_test[0].reshape(1, 28, 28, 1)\nlayer_outs = functor([test, 1.])\nprint(layer_outs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"layer_outs[0].shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = np.squeeze(layer_outs[0])\nsample = np.transpose(sample)\nsample[0].shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pyplot.imshow(sample[10])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualize model performance with tensorboard"},{"metadata":{"trusted":true,"_kg_hide-output":false},"cell_type":"code","source":"%load_ext tensorboard.notebook\n%tensorboard --logdir logs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inferance\nAfter all training and understanding how model learns and performs, let's do inferance on our model with test dataset. Load test dataset and send it to model to predict labels and write it to CSV file."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv(TEST_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = test_df.values.reshape(test_df.shape[0], 28, 28, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = model.predict(test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = np.argmax(predictions, axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df = pd.concat([pd.Series(range(1, 28001), name=\"ImageId\"), pd.Series(predictions, name=\"Label\")], axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pyplot.imshow(test_df[3].reshape(28, 28))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df.to_csv(\"cnn_mnist_datagen.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}