{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Convolutional XGBOOST Tutorial \n## Accuracy=99.557\n\nThis Kernal compares the classification capability of DenseNet Layers in CNN and XGBoost and Proposing a new model of Convolutional XGBoost which uses Convolutional layers for feature extraction and XGBoost for classification.\n\nWe Know that XGBoost is one of the best ensemble learning algorithms which works well for various ML applications. But they don't perform accurately when it comes to Image Classification or other deep learning application. While When we analyse the working of CNN we understand that convolution layers are used to select and extract features for the classification.\n\nSo I tried Experimenting on Intermediate results started with a hybrid approach between [GNB, XGB &CNN](https://www.kaggle.com/chekoduadarsh/hybrid-cnn-xgboost-gnb-accuracy-99) in K-MNIST which lead to my find out in the **most and most of the conditions XGB outperforms CNN with a minimum of 0.01 and maximum of 0.1 accuracies**"},{"metadata":{},"cell_type":"markdown","source":"## Algorithm\n![](https://i.ibb.co/4Th58KC/bg3.png)\n\nAs shown in Figure given above, our framework consists of three parts: data preprocessing, feature extraction, and regression analysis. \n\nIn the data processing phase, we divide the raw meta-data features into two parts, i.e., the time-related ones, and the time-unrelated ones. The time-related ones include the “postdate” and “timezone”. The rest are time-unrelated ones, i.e., related to users and items. For the time-related ones, we deal with the “postdate” from the perspective of multiple time scales, and add the “timezone” as an additional feature. Furthermore, we propose a hybrid model \nfor the subsequent two phases. More specifically, we use a CNN model consisting of four convolution layers and three full-linear layers on social cues to learn a high-level representation of the data in the feature extraction phase. In the regression analysis phase, we use XGBoost directly to make popularity predictions given the high-level features extracted by CNN.\n\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd \nimport seaborn as sns\nimport os\n%matplotlib inline\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\nprint(data.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\nprint(test_data.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = pd.read_csv('/kaggle/input/digit-recognizer/sample_submission.csv')\nprint(sample_submission.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Lets visualize, understand and convert data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = data[:]\nval_data = data[40000:]\ntrain_label = np.float32(train_data.label)\nval_label = np.float32(val_data.label)\ntrain_image = np.float32(train_data[train_data.columns[1:]])\nval_image = np.float32(val_data[val_data.columns[1:]])\ntest_image = np.float32(test_data[test_data.columns])\nprint('train shape: %s'%str(train_data.shape))\nprint('val shape: %s'%str(val_data.shape))\nprint('train_label shape: %s'%str(train_label.shape))\nprint('val_label shape: %s'%str(val_label.shape))\nprint('train_image shape: %s'%str(train_image.shape))\nprint('val_image shape: %s'%str(val_image.shape))\nprint('test_image shape: %s'%str(test_image.shape))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"g = sns.countplot(train_label)\ng","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(train_image[13].reshape(28,28))\nplt.show()\nprint(train_image[13].shape)\n\ntrain_image = train_image/255.0\nval_image = val_image/255.0\ntest_image = test_image/255.0\n\ntrain_image = train_image.reshape(train_image.shape[0],28,28,1)\nval_image = val_image.reshape(val_image.shape[0],28,28,1)\ntest_image = test_image.reshape(test_image.shape[0],28,28,1)\nprint('train_image shape: %s'%str(train_image.shape))\n\nprint('train_image shape: %s'%str(train_image.shape))\nprint('val_image shape: %s'%str(val_image.shape))\n\ntrain_label1 = train_label\nval_label1 = val_label\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import OneHotEncoder\n\nencoder = OneHotEncoder(sparse=False,categories='auto')\nyy = [[0],[1],[2],[3],[4],[5],[6],[7],[8],[9]]\nencoder.fit(yy)\n# transform\ntrain_label = train_label.reshape(-1,1)\nval_label = val_label.reshape(-1,1)\n\n\ntrain_label = encoder.transform(train_label)\nval_label = encoder.transform(val_label)\n\n\n\nprint('train_label shape: %s'%str(train_label.shape))\nprint('val_label shape: %s'%str(val_label.shape))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\nimport numpy as np\nimport keras\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten\nfrom keras.layers import Conv2D, MaxPooling2D\nfrom keras.optimizers import SGD\nfrom keras.layers.normalization import BatchNormalization\nfrom keras.layers import LeakyReLU\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Lets Model our CNN\n\nI used the Keras Sequential API, where you have just to add one layer at a time, starting from the input.\n\nThe first is the convolutional (Conv2D) layer. It is like a set of learnable filters. I choosed to set 32 filters for the two firsts conv2D layers and 64 filters for the two last ones. Each filter transforms a part of the image (defined by the kernel size) using the kernel filter. The kernel filter matrix is applied on the whole image. Filters can be seen as a transformation of the image.\n\nThe CNN can isolate features that are useful everywhere from these transformed images (feature maps).\n\nThe second important layer in CNN is the pooling (MaxPool2D) layer. This layer simply acts as a downsampling filter. It looks at the 2 neighboring pixels and picks the maximal value. These are used to reduce computational cost, and to some extent also reduce overfitting. We have to choose the pooling size (i.e the area size pooled each time) more the pooling dimension is high, more the downsampling is important.\n\nCombining convolutional and pooling layers, CNN are able to combine local features and learn more global features of the image.\n\nDropout is a regularization method, where a proportion of nodes in the layer are randomly ignored (setting their wieghts to zero) for each training sample. This drops randomly a propotion of the network and forces the network to learn features in a distributed way. This technique also improves generalization and reduces the overfitting.\n\n'relu' is the rectifier (activation function max(0,x). The rectifier activation function is used to add non linearity to the network.\n\nThe Flatten layer is use to convert the final feature maps into a one single 1D vector. This flattening step is needed so that you can make use of fully connected layers after some convolutional/maxpool layers. It combines all the found local features of the previous convolutional layers.\n\nIn the end i used the features in two fully-connected (Dense) layers which is just artificial an neural networks (ANN) classifier. In the last layer(Dense(10,activation=\"softmax\")) the net outputs distribution of probability of each class.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\n# input: 28x28 images with 1 channels -> (28, 28, 1) tensors.\n# this applies 32 convolution filters of size 3x3 each.\nmodel.add(Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1),padding='same'))\nmodel.add(BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"))\nmodel.add(LeakyReLU(alpha=0.1))\nmodel.add(Conv2D(32, (3, 3), activation='relu',padding='same'))\nmodel.add(BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"))\nmodel.add(LeakyReLU(alpha=0.1))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(64, (3, 3), activation='relu',padding='same'))\nmodel.add(BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"))\nmodel.add(LeakyReLU(alpha=0.1))\nmodel.add(Conv2D(64, (3, 3), activation='relu',padding='same'))\nmodel.add(BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"))\nmodel.add(LeakyReLU(alpha=0.1))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.25))\n\n\nmodel.add(Conv2D(128, kernel_size=5, activation='relu',padding='same'))\nmodel.add(BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"))\nmodel.add(LeakyReLU(alpha=0.1))\nmodel.add(Conv2D(128, kernel_size=5, activation='relu',padding='same'))\nmodel.add(BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"))\nmodel.add(LeakyReLU(alpha=0.1))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.25))\n\n\nmodel.add(Conv2D(256, kernel_size=5, activation='relu',padding='same'))\nmodel.add(BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"))\nmodel.add(LeakyReLU(alpha=0.1))\nmodel.add(Conv2D(256, kernel_size=5, activation='relu',padding='same'))\nmodel.add(BatchNormalization(momentum=0.9, epsilon=1e-5, gamma_initializer=\"uniform\"))\nmodel.add(LeakyReLU(alpha=0.1))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.25))\n\n\nmodel.add(Flatten())\n\n\n\n#model = keras.applications.inception_v3.InceptionV3(weights= None, include_top=False, input_shape= (28,28,1))\nmodel.add(Dense(256, activation='relu', name='my_dense'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(128, activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation='softmax'))\n\n#sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)\n#model.compile(loss='categorical_crossentropy', optimizer=sgd)\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating a intermediate layer model to extract the data from 'my_dense' layer"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import Model\nlayer_name='my_dense'\nintermediate_layer_model = Model(inputs=model.input,\n                                 outputs=model.get_layer(layer_name).output)\n\nintermediate_layer_model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Using ImageDataGenerator to bring in augmentaion in data"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\ndatagen = ImageDataGenerator(\n    rotation_range=15,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range = 15,\n    horizontal_flip = False,\n    zoom_range = 0.20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Adam Optimizer\n\nThe Adam optimization algorithm is an extension to stochastic gradient descent that has recently seen broader adoption for deep learning applications in computer vision and natural language processing.Adam is an optimization algorithm that can be used instead of the classical stochastic gradient descent procedure to update network weights iterative based in training data.\n\nAdam was presented by Diederik Kingma from OpenAI and Jimmy Ba from the University of Toronto in their 2015 ICLR paper (poster) titled “Adam: A Method for Stochastic Optimization“. I will quote liberally from their paper in this post, unless stated otherwise.\n\nYou can try \"RMSprop\",\"Adadelta\",\"SGD\", etc.\n\ncategorical_crossentropy is mentioned thrice for 3 diffrenet outputs\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.optimizers import Adam, Adadelta, RMSprop\n\nmodel.compile(loss='categorical_crossentropy',optimizer=Adam(),metrics=['accuracy'])\n\ndatagen.fit(train_image)\n\n# training\nhistory = model.fit_generator(datagen.flow(train_image,train_label, batch_size=32),\n                              epochs = 75,\n                              shuffle=True,\n                              validation_data = (val_image,val_label),\n                              verbose = 1,\n                              steps_per_epoch=train_image.shape[0] // 32)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(history.history['accuracy'])\nplt.plot(history.history['val_accuracy'])\nplt.title('Model accuracy')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Test'], loc='upper left')\nplt.show()\n\n\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('Model loss')\nplt.ylabel('Loss')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Test'], loc='upper left')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Extract the intermediate output from CNN"},{"metadata":{"trusted":true},"cell_type":"code","source":"intermediate_output = intermediate_layer_model.predict(train_image) \nintermediate_output = pd.DataFrame(data=intermediate_output)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_data = intermediate_output[40000:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_cnn = model.predict(test_image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"intermediate_test_output = intermediate_layer_model.predict(test_image)\nintermediate_test_output = pd.DataFrame(data=intermediate_test_output)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create XGB model and training it for intermediate values\n\nXGBoost is a decision-tree-based ensemble Machine Learning algorithm that uses a gradient boosting framework. In prediction problems involving unstructured data (images, text, etc.) artificial neural networks tend to outperform all other algorithms or frameworks. However, when it comes to small-to-medium structured/tabular data, decision tree based algorithms are considered best-in-class right now. Please see the chart below for the evolution of tree-based algorithms over the years.\n![](https://miro.medium.com/max/1400/1*U72CpSTnJ-XTjCisJqCqLg.jpeg)\n\n### Algorithmic Enhancements:\n\n**Regularization:** It penalizes more complex models through both LASSO (L1) and Ridge (L2) regularization to prevent overfitting.\n\n**Sparsity Awareness:** XGBoost naturally admits sparse features for inputs by automatically ‘learning’ best missing value depending on training loss and handles different types of sparsity patterns in the data more efficiently.\n\n**Weighted Quantile Sketch:** XGBoost employs the distributed weighted Quantile Sketch algorithm to effectively find the optimal split points among weighted datasets.\n\n**Cross-validation:** The algorithm comes with built-in cross-validation method at each iteration, taking away the need to explicitly program this search and to specify the exact number of boosting iterations required in a single run."},{"metadata":{"trusted":true},"cell_type":"code","source":"from xgboost import XGBClassifier\n\nxgbmodel = XGBClassifier(objective='multi:softprob', \n                      num_class= 10)\nxgbmodel.fit(intermediate_output, train_label1)\nxgbmodel.score(val_data, val_label1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prediction for res"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_xgb = xgbmodel.predict(intermediate_test_output)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Arrange the results and submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_cnn = submission_cnn.astype(int)\nsubmission_xgb = submission_xgb.astype(int)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_cnn\nlabel = np.argmax(submission_cnn,1)\nid_ = np.arange(0,label.shape[0])\nlabel","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#if (xgbmodel.score(val_data, val_label1) > )\nfinal_sub = submission_xgb","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"save = pd.DataFrame({'ImageId':sample_submission.ImageId,'label':final_sub})\nprint(save.head(10))\nsave.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}