{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport keras\nfrom keras.datasets import cifar10\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Activation, Flatten\nfrom keras.layers import Conv2D, MaxPooling2D\nimport os\n\nimport numpy as np\n\nimport seaborn as sns\nimport matplotlib\nimport matplotlib.pyplot as plt\n\nfrom sklearn.metrics import confusion_matrix, classification_report\nimport itertools\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = pd.read_csv('/kaggle/input/cifar-10/sampleSubmission.csv')\nsample.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trainLabel = pd.read_csv('/kaggle/input/cifar-10/trainLabels.csv')\ntrainLabel","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install pylzma\n!pip install py7zlib","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **1、 7z file**"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"import py7zlib\nimport time\n\nfp = open(\"/kaggle/input/cifar-10/train.7z\",'rb')\n#生成一个archive对象\narchive = py7zlib.Archive7z(fp)\n\n#读取文件中所有的文件名\nnames = archive.getnames()\n#search\nstartTime = time.time()\n\n#根据文件名返回文件的archiveFile类\nmember = archive.getmember(names[0])\nend_1_time = time.time()\nprint(\"search time is {}\".format(end_1_time-startTime))\n\n#read data\n#读取文件的所有数据\ndata = member.read()\nend_2_time = time.time()\nprint(\"read time is {}\".format(end_2_time-end_1_time))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"names","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **2、Replacement method**"},{"metadata":{},"cell_type":"markdown","source":"The above method is not very easy to control"},{"metadata":{},"cell_type":"markdown","source":"**2.1 Introduction**"},{"metadata":{},"cell_type":"markdown","source":"The CIFAR-10 dataset contains 60,000 color images of 32 x 32 pixels in 3 channels divided into 10 classes. Each class contains 6,000 images. The training set contains 50,000 images, while the test sets provides 10,000 images. This image taken from the CIFAR repository ( https://www.cs.toronto.edu/~kriz/cifar.html ). This is a classification problem with 10 classes(muti-label classification). We can take a view on this image for more comprehension of the dataset."},{"metadata":{},"cell_type":"markdown","source":"The challenge is to recognize previously unseen images and assign them to one of the 10 classes."},{"metadata":{"trusted":true},"cell_type":"code","source":"(x_train, y_train), (x_test, y_test) = cifar10.load_data()\nprint('x_train shape:', x_train.shape)\nprint('y_train shape:', y_train.shape)\nprint(x_train.shape[0], 'train samples')\nprint(x_test.shape[0], 'test samples')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axs = plt.subplots(1,2,figsize=(15,5))\nsns.countplot(y_train.ravel(),ax=axs[0])\naxs[0].set_title('Training data')\naxs[0].set_xlabel('Classes')\n\nsns.countplot(y_test.ravel(),ax=axs[1])\naxs[1].set_title('Testing data')\naxs[1].set_xlabel('Classes')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, each classe contain exacly 6000 examples( 5000 for training and 1000 for test).\n\nThe graph above is very important for the training, for example if we had just 1000 samples of label 1 that will be a problem , the model will find difficulties to detect label 1\"less accuracy \", so that's not going to happend everything look fine. It's important to know the distribution of dataset behind different classes because the goodness of our model depend on it.\n\nNow let's doing some preprocessing.\n\nThe output variable have 10 posible values. This is a multiclass classification problem. We need to encode these lables to one hot vectors (ex : \"bird\" -> [0,0,1,0,0,0,0,0,0,0])."},{"metadata":{},"cell_type":"markdown","source":"# **3、Normalize**"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = x_train.astype('float32')\nx_test = x_test.astype('float32')\nx_train /= 255\nx_test /= 255","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_classes = 10\ny_train = keras.utils.to_categorical(y_train, num_classes)\ny_test = keras.utils.to_categorical(y_test, num_classes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train.shape[:]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **4、Defining the model architecture Using ConVnets**"},{"metadata":{},"cell_type":"markdown","source":"In the first stage, Our net will learn 32 convolutional filters, each of which with a 3 x 3 size. The output dimension is the same one of the input shape, so it will be 32 x 32 and activation is relu, which is a simple way of introducing non-linearity; folowed by another 32 convolutional filters, each of which with a 3 x 3 size and activation is also relu. After that we have a max-pooling operation with pool size 2 x 2 and a dropout at 25%.\n"},{"metadata":{},"cell_type":"markdown","source":"In the next stage in the deep pipeline, Our net will learn 64 convolutional filters, each of which with a 3 x 3 size. The output dimension is the same one of the input shape and activation is relu; folowed by another 64 convolutional filters, each of which with a 3 x 3 size and activation is also relu. After that we have a max-pooling operation with pool size 2 x 2 and a dropout at 25%.\n"},{"metadata":{},"cell_type":"markdown","source":"And the Final stage in the deep pipeline is a dense network with 512 units and relu activation followed by a dropout at 50% and by a softmax layer with 10 classes as output, one for each category."},{"metadata":{"trusted":true},"cell_type":"code","source":"model =Sequential()\nmodel.add(Conv2D(32,(3,3),padding='same',input_shape=x_train.shape[1:]))\nmodel.add(Activation('relu'))\nmodel.add(Conv2D(32,(3,3)))\nmodel.add(MaxPooling2D(3,strides=2))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(64,(3,3),padding='same'))\nmodel.add(Activation('relu'))\nmodel.add(Conv2D(64,(3,3)))\nmodel.add(MaxPooling2D(3,strides=2))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(128,(3,3),padding='same'))\nmodel.add(Activation('relu'))\nmodel.add(Conv2D(128,(3,3)))\nmodel.add(MaxPooling2D(3,strides=2))\nmodel.add(Dropout(0.25))\n\nmodel.add(Flatten())\nmodel.add(Dense(512))\nmodel.add(Activation('relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(num_classes))\n\nmodel.add(Activation('softmax'))\n\nmodel.summary()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **5、Model training**"},{"metadata":{},"cell_type":"markdown","source":"Before making network ready for training we have to make sure to add below things:\n\nA loss function: to measure how good the network is\nAn optimizer: to update network as it sees more data and reduce loss value\nMetrics: to monitor performance of network\nAlso note that for data augmentation:\n\nOne of the most commun tehnique to avoid overfitting is data augmentation. And We know that overfitting is generaly occur when we don't have enought data for training the model. To avoid this overfitting problem, we need to expand artificially our dataset. The idea is to alter the training data with small transformations to reproduce the variations occuring when someone is writing a digit.\n\nDifferent data aumentation techniques are as follows: Cropping, Rotating, Scaling, Translating, Flipping, Adding Gaussian noise to input images, etc..."},{"metadata":{"trusted":true},"cell_type":"code","source":"opt = keras.optimizers.Adam(learning_rate=0.001,decay=1e-6,epsilon=1e-08,beta_1=0.9,beta_2=0.999)\nmodel.compile(loss='categorical_crossentropy',\n              optimizer=opt,\n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = None\nprint('Not using data argumentation.')\nhistory = model.fit(x_train,y_train,\n                   batch_size=128,\n                   epochs=5,\n                   validation_data=(x_test,y_test),\n                   shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **6、Evaluate the model**"},{"metadata":{},"cell_type":"markdown","source":"6.1 Training and validation curves.¶\nLet's see the training and validation process by the visualization of history of fitting. This allow us to quickly know if how our model fit our data (overfitting, underfitting, model convergence, etc...)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def eva(history):\n    fig, axs = plt.subplots(1,2,figsize=(15,5))\n    axs[0].plot(history.history['accuracy'])\n    axs[0].plot(history.history['val_accuracy'])\n    axs[0].set_title('Model Accuracy')\n    axs[0].set_ylabel('Accuracy')\n    axs[0].set_xlabel('Epoch')\n    axs[0].legend(['train','validate'], loc='upper left')\n    \n    axs[1].plot(history.history['loss'])\n    axs[1].plot(history.history['val_loss'])\n    axs[1].set_title('Model Loss')\n    axs[1].set_ylabel('Loss')\n    axs[1].set_xlabel('Epoch')\n    axs[1].legend(['train','validate'], loc='upper left')\n    plt.show()\n    \nprint(history.history.keys())\neva(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **7、Score trained model and prediction**"},{"metadata":{"trusted":true},"cell_type":"code","source":"scores = model.evaluate(x_test,y_test)\nprint('Test loss:',scores[0])\nprint('Test accuracy:',scores[1])\n\npred = model.predict(x_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **8、Confusion matrix**"},{"metadata":{},"cell_type":"markdown","source":"Confusion matrix can be very helpfull to see your model drawbacks. We plot the confusion matrix of the validation results. For good vizualization of our confusion matrix, we have to define to fonction"},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = ['Airplane', 'Automobile', 'Bird', 'Cat', 'Deer', 'Dog', 'Frog', 'Horse', 'Ship', 'Truck']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **9、Check the predictions**"},{"metadata":{"trusted":true},"cell_type":"code","source":"R = 5\nC = 5\nfig, axes = plt.subplots(R,C,figsize=(12,12))\naxes = axes.ravel()\n\nfor i in np.arange(0,R*C):\n    axes[i].imshow(x_test[i])\n    axes[i].set_title(\"True:%s \\n Predict:%s\" %(labels[Y_true[i]], labels[Y_pred_classes[i]]))\n    axes[i].axis('off')\n    plt.subplots_adjust(wspace=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_test(number):\n    fig = plt.figure(figsize = (3,3))\n    test_image = np.expand_dims(x_test[number], axis=0)\n    test_result = model.predict_classes(test_image)\n    plt.imshow(x_test[number])\n    dict_key = test_result[0]\n    plt.title(\"Predicted: {} \".format(labels[dict_key]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_test(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **10、Save**"},{"metadata":{"trusted":true},"cell_type":"code","source":"path = os.path.join(os.getcwd(),'save_models')\nmodel_name = 'keras_cifar10_trained_model.h5'\nif not os.path.isdir(path):\n    os.mkdir(path)\n\nmodel_path = os.path.join(path,model_name)\nmodel.save(model_path)\nprint('Saved trained model at %s ' % model_path)\n\nscores = model.evaluate(x_test,y_test,verbose=1)\nprint('Test loss:',scores[0])\nprint('Test accuracy:', scores[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}