{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"1. Introduction\n1. Data preparation\n    1. Load data\n    1. Normalization, Reshape and Label encoding\n    1. Visualize test and train sample\n1. Model Building\n    1. Split training and valdiation set\n    1. Define the model architechture\n    1. Set the optimizer and annealer\n    1. Data augmentation\n    1. Train model\n1. Evaluate the model\n    1. Training and validation curves\n    1. Visualize Prediction\n1. Prediction and submition\n    1. Predict and Submit results\n* * * *\n### 1. Introduction\n\nThis kernel is basic start in deep learning. \n\nCIFAR-10 (Canadian Institute For Advanced Research) is the type “hello world” dataset of computer vision. This dataset is a collection of images that are commonly used to train machine learning and computer vision algorithms. It is one of the most widely used datasets for machine learning research. The CIFAR-10 dataset contains 60,000 32x32 color images in 10 different classes. The 10 different classes represent airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. There are 6,000 images of each class. In this competition, your goal is to correctly identify different object from a dataset of tens of thousands of color images.\n* * * *","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# import backend\nimport tensorflow as  tf\n\n# Model architecture\nfrom sklearn.model_selection import train_test_split\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D\nfrom keras.layers import MaxPool2D, Activation, MaxPooling2D\nfrom keras.layers.normalization import BatchNormalization\n\n\n# model optimisation and scores\nfrom sklearn import metrics\nfrom sklearn.metrics import accuracy_score, f1_score\nfrom tensorflow.keras.optimizers import Adam, SGD\nfrom sklearn.metrics import confusion_matrix, classification_report\nfrom sklearn.metrics import roc_auc_score, auc, roc_curve\n\n\n# Annealer\nfrom keras.callbacks import LearningRateScheduler\nfrom tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint\n\n\n# Data processing\n# from keras.preprocessing.image import ImageDataGenerator\nfrom keras.utils import to_categorical\n# from keras.preprocessing import image\n\n# Visualization\nimport matplotlib.pyplot as plt\n\nimport h5py\n","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:10:59.539464Z","iopub.execute_input":"2021-07-17T20:10:59.539773Z","iopub.status.idle":"2021-07-17T20:10:59.549584Z","shell.execute_reply.started":"2021-07-17T20:10:59.539720Z","shell.execute_reply":"2021-07-17T20:10:59.548608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**File Structure:**\n\n* **train.7z -** a folder containing the training images\n* **test.7z -** a folder containing the test images\n* **trainLabels.csv -** the training labels\n* **sampleSubmission.csv -** We have to predict labels for all 300,000 images.\n\nTrain and Test files are present a .7z compressed format. You can process file from .7z extension; it will take long enough time to process.\n<pre><code>\n# Reading .7z content\n!pip install pyunpack # install decoder package\n!pip install patool # install requirement\n\nfrom pyunpack import Archive\n\nprint(\"start unpacking train data\")\n%time Archive('/kaggle/input/cifar-10/train.7z').extractall('/kaggle/working/dataset/', auto_create_dir=True, patool_path=None)\nprint(\"unpacking finished\")\nprint(\"start unpacking test data\")\n%time Archive('/kaggle/input/cifar-10/test.7z').extractall('/kaggle/working/dataset/', auto_create_dir=True, patool_path=None)\nprint(\"unpacking finished\")\n\n### This will take approx 2-3 hr to read file from test.7z.\n\n</code></pre>\n\nI downloaded the zip file, and processed it and created files in batches.You can use this data for your work, data is organised as follow:\n* **train_data.h5-** it is generated from train.7z, contain training samples.\n* **test_data_1.h5-** first 50k examples from test.7z\n* **test_data_2.h5-** index 50k to 100k examples from test.7z\n* **test_data_3.h5- ** index 100k to 150k examples from test.7z\n* **test_data_4.h5-** index 150k to 200k examples from test.7z\n* **test_data_5.h5- ** index 200k to 250k examples from test.7z\n* **test_data_6.h5-** index 250k to 300k examples from test.7z\n","metadata":{}},{"cell_type":"markdown","source":"### 2. Data Prepration\n**Load Data**","metadata":{}},{"cell_type":"code","source":"# Read File\nsample_submission = pd.read_csv('../input/cifar-10/sampleSubmission.csv')\ntrain_labels = pd.read_csv('../input/cifar-10/trainLabels.csv')\n\nprint(\"Number of training sample: \",train_labels.shape[0])\nprint(\"Number of test samples: \", sample_submission.shape[0])","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2021-07-17T20:11:04.182793Z","iopub.execute_input":"2021-07-17T20:11:04.183113Z","iopub.status.idle":"2021-07-17T20:11:04.333566Z","shell.execute_reply.started":"2021-07-17T20:11:04.183059Z","shell.execute_reply":"2021-07-17T20:11:04.332703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:11:05.582883Z","iopub.execute_input":"2021-07-17T20:11:05.583207Z","iopub.status.idle":"2021-07-17T20:11:05.613405Z","shell.execute_reply.started":"2021-07-17T20:11:05.583152Z","shell.execute_reply":"2021-07-17T20:11:05.612239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import h5py\nwith h5py.File('../input/cifardata/train_data.h5', 'r') as file:\n    #for key in file.keys():\n     #   print(key)\n    train_ids = pd.DataFrame(np.array(np.squeeze(file['train_ids'])),columns=['id'])\n    train_data = np.array(file['train_images']).reshape(-1, 32, 32, 3)\n    \nwith h5py.File('../input/cifardata/test_data_1.h5', 'r') as file:\n    test_ids_1 = pd.DataFrame(np.array(np.squeeze(file['test_ids'])),columns=['id'])\n    test_data_1 = np.array(file['test_images']).reshape(-1, 32, 32, 3)\n    \nwith h5py.File('../input/cifardata/test_data_2.h5', 'r') as file:\n    test_ids_2 = pd.DataFrame(np.array(np.squeeze(file['test_ids'])),columns=['id'])\n    test_data_2 = np.array(file['test_images']).reshape(-1, 32, 32, 3)\n    \n    \nwith h5py.File('../input/cifardata/test_data_3.h5', 'r') as file:\n    test_ids_3 = pd.DataFrame(np.array(np.squeeze(file['test_ids'])),columns=['id'])\n    test_data_3 = np.array(file['test_images']).reshape(-1, 32, 32, 3)\n    \n    \nwith h5py.File('../input/cifardata/test_data_4.h5', 'r') as file:\n    test_ids_4 = pd.DataFrame(np.array(np.squeeze(file['test_ids'])),columns=['id'])\n    test_data_4 = np.array(file['test_images']).reshape(-1, 32, 32, 3)\n    \n    \nwith h5py.File('../input/cifardata/test_data_5.h5', 'r') as file:\n    test_ids_5 = pd.DataFrame(np.array(np.squeeze(file['test_ids'])),columns=['id'])\n    test_data_5 = np.array(file['test_images']).reshape(-1, 32, 32, 3)\n    \n    \nwith h5py.File('../input/cifardata/test_data_6.h5', 'r') as file:\n    test_ids_6 = pd.DataFrame(np.array(np.squeeze(file['test_ids'])),columns=['id'])\n    test_data_6 = np.array(file['test_images']).reshape(-1, 32, 32, 3)\n    \n","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:11:59.842852Z","iopub.execute_input":"2021-07-17T20:11:59.843136Z","iopub.status.idle":"2021-07-17T20:12:34.515172Z","shell.execute_reply.started":"2021-07-17T20:11:59.843086Z","shell.execute_reply":"2021-07-17T20:12:34.514433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Concatenate test into one dataframe\ntest_ids = pd.concat([test_ids_1, test_ids_2, test_ids_3, test_ids_4, test_ids_5, test_ids_6], axis=0)\ntest_data = np.concatenate([test_data_1, test_data_2, test_data_3, test_data_4, test_data_5, test_data_6], axis=0)\n# test_data = test_data_1\n# test_ids = test_ids_1\n\ntrain_data = train_data\ntarget_variable = train_labels.label\n","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:12:34.517188Z","iopub.execute_input":"2021-07-17T20:12:34.517462Z","iopub.status.idle":"2021-07-17T20:12:36.109853Z","shell.execute_reply.started":"2021-07-17T20:12:34.517417Z","shell.execute_reply":"2021-07-17T20:12:36.108957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After building dataset, let's check data is consistent or not.","metadata":{}},{"cell_type":"code","source":"# checking loaded test data is consistent or not.\n\nprint(\"all test id and samples are consistent = \", np.all(test_ids.id.value_counts().sort_index() == sample_submission.id.value_counts().sort_index()))\nprint(\"all train ids and train_labels are consistent = \", np.all(train_ids.id == train_labels.id))","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:12:36.111245Z","iopub.execute_input":"2021-07-17T20:12:36.111546Z","iopub.status.idle":"2021-07-17T20:12:36.245267Z","shell.execute_reply.started":"2021-07-17T20:12:36.111491Z","shell.execute_reply":"2021-07-17T20:12:36.244442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Class Distribution","metadata":{}},{"cell_type":"code","source":"distribution = train_labels.label.value_counts()\nfor category, size in zip(distribution.index, distribution.values):\n    print(f\"{category} {size} images\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot Distribution\n\nplt.figure(figsize=(10, 5))\ntrain_labels[\"label\"].value_counts().plot(kind='bar',\n                                          title='Distribution of classes'\n                                         )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualize Samples","metadata":{}},{"cell_type":"code","source":"# function to draw samples\n\ndef draw_sample_images(data, labels = False):\n    nrows = 4\n    ncols = 10\n    \n    total_image = data.shape[0]\n    \n\n    samples = np.random.choice(total_image, nrows*ncols)\n\n    plt.figure(figsize=(20, 5))\n    for i in range(nrows*ncols):\n        plt.subplot(nrows, ncols, i+1)\n        plt.imshow(data[samples[i]])\n        if(labels):\n            plt.title(train_labels.label[samples[i]])\n        plt.axis(\"off\")\n    plt.tight_layout()\n    plt.show()\n            \n\ndraw_sample_images(train_data, labels = True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Split dataset","metadata":{}},{"cell_type":"code","source":"# Label encoding\nfrom sklearn.preprocessing import LabelEncoder\n\nlb_make = LabelEncoder()\n\ntarget_label = lb_make.fit_transform(target_variable)\ntarget_label","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:13:02.383086Z","iopub.execute_input":"2021-07-17T20:13:02.383378Z","iopub.status.idle":"2021-07-17T20:13:02.406724Z","shell.execute_reply.started":"2021-07-17T20:13:02.383329Z","shell.execute_reply":"2021-07-17T20:13:02.405916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y = pd.get_dummies(target_label).values","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:13:03.373489Z","iopub.execute_input":"2021-07-17T20:13:03.373800Z","iopub.status.idle":"2021-07-17T20:13:03.384396Z","shell.execute_reply.started":"2021-07-17T20:13:03.373744Z","shell.execute_reply":"2021-07-17T20:13:03.383668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# split training and validation set.\nX_train, X_val, Y_train, Y_val = train_test_split(train_data, y, random_state=41, test_size=0.25)\n\nprint(\"X_train shape: \", X_train.shape)\nprint(\"Y_train shape: \", Y_train.shape)\nprint(\"X_val shape: \", X_val.shape)\nprint(\"Y_val shape: \", Y_val.shape)","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:39:02.713138Z","iopub.execute_input":"2021-07-17T20:39:02.713703Z","iopub.status.idle":"2021-07-17T20:39:03.079267Z","shell.execute_reply.started":"2021-07-17T20:39:02.713456Z","shell.execute_reply":"2021-07-17T20:39:03.078446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Base Model","metadata":{}},{"cell_type":"code","source":"def create_model():\n        \"\"\"\n        Build CNN model and Perform the following operations:\n\n        1. Flatten the output of our base model to 1 dimension\n        2. Add a fully connected layer with 1,024 hidden units and ReLU activation\n        3. This time, we will go with a dropout rate of 0.2\n        4. Add a final Fully Connected Sigmoid Layer\n        \"\"\"\n        \n        \n        model = Sequential()\n        model.add(Conv2D(32,  kernel_size = 3,kernel_initializer='he_normal', activation='relu', input_shape = (32, 32, 3)))\n        model.add(BatchNormalization())\n        \n        model.add(Dropout(0.2))\n        \n        model.add(Conv2D(64, kernel_size = 3, kernel_initializer='he_normal', strides=1, activation='relu'))\n        model.add(BatchNormalization())\n        \n        model.add(MaxPooling2D((2, 2)))\n        model.add(Conv2D(128, kernel_size = 3, strides=1, kernel_initializer='he_normal' ,padding='same', activation='relu'))\n        model.add(BatchNormalization())\n        \n        model.add(MaxPooling2D((2, 2)))\n        model.add(Conv2D(64, kernel_size = 3,kernel_initializer='he_normal', activation='relu'))\n        model.add(BatchNormalization())\n        \n        model.add(MaxPooling2D((4, 4)))\n        model.add(Dropout(0.2))\n\n\n        model.add(Flatten())\n        model.add(Dense(256,kernel_initializer='he_normal', activation = \"relu\"))\n        model.add(Dropout(0.1))\n        model.add(Dense(10, kernel_initializer='glorot_uniform', activation = \"softmax\"))\n\n\n        # Compile the model\n        model.compile(loss=\"categorical_crossentropy\", optimizer=\"adam\", metrics=[\"acc\"])\n\n        \n        \n#         model = Sequential()\n#         model.add(self.i_model)\n#         model.add(GlobalAveragePooling2D())\n#         model.add(Dense(128))\n#         model.add(Dropout(0.1))\n#         model.add(Dense(10, activation = 'softmax'))\n#         model.compile(optimizer=SGD(lr=0.0001, momentum=0.99, decay=0.01), loss='categorical_crossentropy', metrics=['acc'])\n        return model\n    \n    \n    \n\ndef train(model):\n        \"\"\"\n        Train the model with parameters:\n        epochs = 5\n        steps_per_epoch=100\n        validation_steps=50\n        \n        \"\"\"\n   \n        epochs=50\n#         steps_per_epoch=10\n#         validation_steps=5\n        \n            \n        # We'll stop training if no improvement after some epochs\n        earlystopper = EarlyStopping(monitor='val_acc', patience=5, verbose=1)\n        reduce_lr = ReduceLROnPlateau(monitor='val_acc', factor=0.5, patience=5, \n                                   verbose=1, mode='max', min_lr=0.00001)\n        # Save the best model during the traning\n#         checkpointer = ModelCheckpoint('best_model1.h5'\n#                                         ,monitor='val_acc'\n#                                         ,verbose=1\n#                                         ,save_best_only=True\n#                                         ,save_weights_only=True)\n        # Train\n        training = model.fit(x=X_train,\n                             y=Y_train,\n                             batch_size=32,\n                             epochs=epochs,\n                             verbose=1,\n                             callbacks=[earlystopper, reduce_lr],\n                             validation_data=(X_val, Y_val),\n                             \n                            )\n        \n        # Get the best saved weights\n#         model.load_weights('best_model1.h5')\n        return training\n\n    \nmodel = create_model()\n\nmodel.summary()\n","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:39:07.665372Z","iopub.execute_input":"2021-07-17T20:39:07.665711Z","iopub.status.idle":"2021-07-17T20:39:07.973658Z","shell.execute_reply.started":"2021-07-17T20:39:07.665640Z","shell.execute_reply":"2021-07-17T20:39:07.972746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training=train(model)\nprint(\"Trained\")\n\n","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:39:09.210220Z","iopub.execute_input":"2021-07-17T20:39:09.210545Z","iopub.status.idle":"2021-07-17T20:40:22.607355Z","shell.execute_reply.started":"2021-07-17T20:39:09.210487Z","shell.execute_reply":"2021-07-17T20:40:22.606433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f, ax = plt.subplots(1,2, figsize=(12,3))\nax[0].plot(training.history['loss'], label=\"Loss\")\nax[0].plot(training.history['val_loss'], label=\"Validation loss\")\nax[0].set_title('Loss')\nax[0].set_xlabel('Epoch')\nax[0].set_ylabel('Loss')\nax[0].legend()\n\n# Accuracy\nax[1].plot(training.history['acc'], label=\"Accuracy\")\nax[1].plot(training.history['val_acc'], label=\"Validation accuracy\")\nax[1].set_title('Accuracy')\nax[1].set_xlabel('Epoch')\nax[1].set_ylabel('Accuracy')\nax[1].legend()\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:40:38.487271Z","iopub.execute_input":"2021-07-17T20:40:38.487571Z","iopub.status.idle":"2021-07-17T20:40:38.976251Z","shell.execute_reply.started":"2021-07-17T20:40:38.487519Z","shell.execute_reply":"2021-07-17T20:40:38.975291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Prediction\ny_pred_raw = model.predict(X_val)\ny_pred = np.argmax(y_pred_raw, axis=1)\ny_true = np.argmax(Y_val, axis = 1)\n","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:41:11.887438Z","iopub.execute_input":"2021-07-17T20:41:11.887766Z","iopub.status.idle":"2021-07-17T20:41:13.197038Z","shell.execute_reply.started":"2021-07-17T20:41:11.887702Z","shell.execute_reply":"2021-07-17T20:41:13.196185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Print classification report\n# test_accuracy = model.evaluate(X_val)*100\nprint(f\"Test accuracy is {np.mean(y_true == y_pred)}%\")\n\nprint(\"Confusion Matrix\")\nprint(confusion_matrix(y_true,y_pred ))\nprint('F1 score is',f1_score(y_true,y_pred, average = 'weighted') *100, \"%\")\n\npredicted_probab =model.predict(X_val)\nprint(\"ROC- AUC score is\", roc_auc_score( Y_val, predicted_probab)*100)\n","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:41:13.198712Z","iopub.execute_input":"2021-07-17T20:41:13.198994Z","iopub.status.idle":"2021-07-17T20:41:14.328961Z","shell.execute_reply.started":"2021-07-17T20:41:13.198947Z","shell.execute_reply":"2021-07-17T20:41:14.328139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"def predict_for_submit():\n        \"\"\"\n        Predict submission test data and form dataframe to submit\n        \"\"\"\n        print(\"Forming submission dataframe...\")\n        \n        # Predict\n        y_pred = model.predict(test_data)\n        y_pred = np.argmax(y_pred, axis=1)\n       \n        sample_submission.label = y_pred\n    \n        print(f\"Submission dataframe created. Rows:{len(sample_submission.label.values)}\")\n        \n        # Write to csv\n        sample_submission.to_csv('submission.csv', index=False)\n        print(\"Submission completed: written submission.csv\")\n        return sample_submission\n\n    \nsample_submission = predict_for_submit()","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:16:22.455015Z","iopub.execute_input":"2021-07-17T20:16:22.455316Z","iopub.status.idle":"2021-07-17T20:16:49.675711Z","shell.execute_reply.started":"2021-07-17T20:16:22.455267Z","shell.execute_reply":"2021-07-17T20:16:49.674999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-17T20:16:49.677352Z","iopub.execute_input":"2021-07-17T20:16:49.677629Z","iopub.status.idle":"2021-07-17T20:16:49.692812Z","shell.execute_reply.started":"2021-07-17T20:16:49.677583Z","shell.execute_reply":"2021-07-17T20:16:49.691881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### References: ","metadata":{}},{"cell_type":"markdown","source":"https://keras.io/models/sequential/\nhttps://keras.io/layers/core/\nhttps://keras.io/layers/convolutional/\nhttps://keras.io/layers/pooling/\nhttps://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6\nhttps://www.kaggle.com/toregil/welcome-to-deep-learning-cnn-99\nhttps://www.kaggle.com/kanncaa1/convolutional-neural-network-cnn-tutorial\nhttps://github.com/MazenAly/Cifar100","metadata":{}}]}