{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n\n# The objective of the notebook is to show how to build simple ensemble algorithms from same cnn models which are using  different loss functions.\n\n# The 5 different ensemble methods is shown here, these methods can also be used in other domains as well.\n  \n     \n> **Here I have used**\n\n\n> Cross_entropy\n\n> Huber\n\n> KLDivergence\n\n> Then I implemented these Ensemble methods on three CNN networks which are using these loss functions    \n\nspecial mention to this notebook https://www.kaggle.com/code/rohitgadhwar/image-classification-using-cnn-transfer-learning \n\n![ensemble.png](https://miro.medium.com/max/600/1*ZzXwFueV-Beh9MapLgZ5QA.png)\n\n> Suppose model1 on imag1 gives the output of [p11,p21,... p101] , where pij is the probability of each where i represents the class and j represents the number of the model\n> \n> Similarly model2 with also give us the similar output [p12,p22,... p102]\n> \n> The ensemble of the results are given below\n> \n> Rij = argmax((p01 + p02 + p03) + (p11 + p12 + p13 …. ) + (p101 + p102 + p103)..)\n> \n> Rij = argmax((2.1p01 + 2.3p02 + 1.9p03) + (2.1p11 + 2.3p12 + 1.9p13) + (2.1p101 + 2.3p102 + 1.9*p103)......)\n> \n> Rij = argmax(max(p01 , p02 , p03) + max(p11 ,p12 , p13) + max(p101 , p102 , p103))......) \n> \n> Rij = argmax((p01 * p02 * p03) + (p11 * p12 * p13 …. ) + (p101 * p102 +* p103)..)","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow import keras\nimport os\n\nimport matplotlib.pyplot as plt\nimport matplotlib.image as img\nimport seaborn as sns\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import *\nseed = 42\n\nfrom sklearn.metrics import ConfusionMatrixDisplay, classification_report, confusion_matrix , precision_score, recall_score, f1_score ,  accuracy_score\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-12-10T13:33:50.716899Z","iopub.execute_input":"2022-12-10T13:33:50.718118Z","iopub.status.idle":"2022-12-10T13:33:53.825450Z","shell.execute_reply.started":"2022-12-10T13:33:50.717998Z","shell.execute_reply":"2022-12-10T13:33:53.824387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load and Extract data","metadata":{}},{"cell_type":"code","source":"import zipfile\n\nzip_files = ['test1', 'train']\n\nfor zip_file in zip_files:\n    with zipfile.ZipFile(\"../input/dogs-vs-cats/{}.zip\".format(zip_file),\"r\") as z:\n        z.extractall(\".\")\n        print(\"{} unzipped\".format(zip_file))","metadata":{"execution":{"iopub.status.busy":"2022-12-10T13:33:53.827737Z","iopub.execute_input":"2022-12-10T13:33:53.828710Z","iopub.status.idle":"2022-12-10T13:34:07.186690Z","shell.execute_reply.started":"2022-12-10T13:33:53.828673Z","shell.execute_reply":"2022-12-10T13:34:07.185689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TRAIN_DIR_PATH = './train'\nfile_names = os.listdir(TRAIN_DIR_PATH)\nprint('There are {} number of images in directory.'.format(len(file_names)))","metadata":{"execution":{"iopub.status.busy":"2022-12-10T13:34:07.188257Z","iopub.execute_input":"2022-12-10T13:34:07.188667Z","iopub.status.idle":"2022-12-10T13:34:07.208320Z","shell.execute_reply.started":"2022-12-10T13:34:07.188629Z","shell.execute_reply":"2022-12-10T13:34:07.207324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def to_dataframe(file_names):\n    files, labels = list(), list()\n    for file in file_names:\n        files.append(file)\n        labels.append(file[:3])\n    df = pd.DataFrame({'filename':files, 'label':labels})\n    return df\n\ndf = to_dataframe(file_names)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T13:34:07.211514Z","iopub.execute_input":"2022-12-10T13:34:07.212007Z","iopub.status.idle":"2022-12-10T13:34:07.228321Z","shell.execute_reply.started":"2022-12-10T13:34:07.211968Z","shell.execute_reply":"2022-12-10T13:34:07.227331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2022-12-10T13:34:07.231638Z","iopub.execute_input":"2022-12-10T13:34:07.231986Z","iopub.status.idle":"2022-12-10T13:34:07.249084Z","shell.execute_reply.started":"2022-12-10T13:34:07.231944Z","shell.execute_reply":"2022-12-10T13:34:07.248094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.tail()","metadata":{"execution":{"iopub.status.busy":"2022-12-10T13:34:07.250551Z","iopub.execute_input":"2022-12-10T13:34:07.250892Z","iopub.status.idle":"2022-12-10T13:34:07.260112Z","shell.execute_reply.started":"2022-12-10T13:34:07.250859Z","shell.execute_reply":"2022-12-10T13:34:07.259045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Split data","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\ntrain_set, valid_set = train_test_split(df, test_size=0.15,random_state= seed)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T13:34:07.261673Z","iopub.execute_input":"2022-12-10T13:34:07.262167Z","iopub.status.idle":"2022-12-10T13:34:07.277366Z","shell.execute_reply.started":"2022-12-10T13:34:07.262073Z","shell.execute_reply":"2022-12-10T13:34:07.276437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_set.shape, valid_set.shape","metadata":{"execution":{"iopub.status.busy":"2022-12-10T13:34:07.280096Z","iopub.execute_input":"2022-12-10T13:34:07.280395Z","iopub.status.idle":"2022-12-10T13:34:07.287586Z","shell.execute_reply.started":"2022-12-10T13:34:07.280353Z","shell.execute_reply":"2022-12-10T13:34:07.286327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Image data Generator and Data Augmentation","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.preprocessing.image import ImageDataGenerator\n\nWIDTH, HEIGHT = 128, 128\nbatch_size = 32","metadata":{"execution":{"iopub.status.busy":"2022-12-10T13:34:07.289026Z","iopub.execute_input":"2022-12-10T13:34:07.289890Z","iopub.status.idle":"2022-12-10T13:34:07.297032Z","shell.execute_reply.started":"2022-12-10T13:34:07.289850Z","shell.execute_reply":"2022-12-10T13:34:07.295859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_datagenerator = ImageDataGenerator(rotation_range=15, \n                                  rescale=1./255, \n                                  shear_range=0.1,\n                                  zoom_range=0.2,\n                                  horizontal_flip=True,\n                                  width_shift_range=0.1,\n                                  height_shift_range=0.1)\n\ntraining_data = train_datagenerator.flow_from_dataframe(dataframe=train_set, \n                                                        directory='./train',\n                                                       x_col='filename',\n                                                       y_col='label',\n                                                       target_size=(WIDTH, HEIGHT),\n                                                       class_mode='categorical',\n                                                       batch_size=batch_size)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T13:34:07.301802Z","iopub.execute_input":"2022-12-10T13:34:07.302139Z","iopub.status.idle":"2022-12-10T13:34:07.490403Z","shell.execute_reply.started":"2022-12-10T13:34:07.302111Z","shell.execute_reply":"2022-12-10T13:34:07.489327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_datagenerator = ImageDataGenerator(rescale=1./255)\nvalidation_data = valid_datagenerator.flow_from_dataframe(dataframe=valid_set,\n                                                         directory='./train',\n                                                         x_col='filename',\n                                                         y_col='label',\n                                                         target_size=(WIDTH,HEIGHT),\n                                                         class_mode='categorical',\n                                                         batch_size=batch_size)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T13:34:07.491820Z","iopub.execute_input":"2022-12-10T13:34:07.492458Z","iopub.status.idle":"2022-12-10T13:34:07.535607Z","shell.execute_reply.started":"2022-12-10T13:34:07.492419Z","shell.execute_reply":"2022-12-10T13:34:07.534565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model building","metadata":{}},{"cell_type":"code","source":"def create_model(IMG_SIZE = 128 , n = 2 ,show = True ):\n    \n    IMG_SHAPE = (IMG_SIZE, IMG_SIZE, 3) \n    i = Input(shape= IMG_SHAPE)\n    x = Conv2D(32, (3, 3), activation='relu', padding='same')(i)\n    x = BatchNormalization()(x)\n    x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)\n    x = BatchNormalization()(x)\n    x = MaxPooling2D((2, 2))(x)\n\n    x = Conv2D(64, (3, 3), activation='relu', padding='same')(x)\n    x = BatchNormalization()(x)\n    x = Conv2D(64, (3, 3), activation='relu', padding='same')(x)\n    x = BatchNormalization()(x)\n    x = MaxPooling2D((2, 2))(x)\n    \n\n\n    x = Flatten()(x)\n    x = Dropout(0.2)(x)\n\n    # Hidden layer\n    x = Dense(512, activation='relu')(x)\n    x = Dropout(0.2)(x)\n    x = BatchNormalization()(x)\n    x = Dense(512, activation='relu')(x)\n    x = Dropout(0.2)(x)\n    x = BatchNormalization()(x)\n    x = Dense(512, activation='relu')(x)\n    x = Dropout(0.2)(x)\n    x = BatchNormalization()(x)\n\n    # last hidden layer i.e.. output layer\n    x = Dense(n, activation='softmax')(x)\n\n    model = Model(i, x)\n\n    # model description\n\n    if(show):\n\n        print(model.summary())\n    \n    return model \n","metadata":{"execution":{"iopub.status.busy":"2022-12-10T13:34:07.536971Z","iopub.execute_input":"2022-12-10T13:34:07.537432Z","iopub.status.idle":"2022-12-10T13:34:07.548597Z","shell.execute_reply.started":"2022-12-10T13:34:07.537397Z","shell.execute_reply":"2022-12-10T13:34:07.547558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loss Functions","metadata":{}},{"cell_type":"code","source":"\n    \nlosses = {\"Cross_entropy\" : tf.keras.losses.CategoricalCrossentropy(),\n          \"Huber\" : tf.keras.losses.Huber(),\n          \"KLDivergence\" : tf.keras.losses.KLDivergence()}\n","metadata":{"execution":{"iopub.status.busy":"2022-12-10T13:34:07.550206Z","iopub.execute_input":"2022-12-10T13:34:07.550857Z","iopub.status.idle":"2022-12-10T13:34:07.561409Z","shell.execute_reply.started":"2022-12-10T13:34:07.550819Z","shell.execute_reply":"2022-12-10T13:34:07.560490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training ","metadata":{}},{"cell_type":"markdown","source":"# CNN which is using Cross entropy Loss","metadata":{}},{"cell_type":"code","source":"    m1 = create_model()\n    \n    m1.compile(loss= losses[\"Cross_entropy\"], \n                    optimizer= tf.keras.optimizers.Adam(learning_rate=0.0001, beta_1=0.9, beta_2=0.999, decay=0.0001),\n                    metrics=['accuracy'])\n\n\n    \n    # Fit data to model\n    h1 = m1.fit(training_data,\n                        epochs=15,\n                        validation_data=validation_data,\n                        steps_per_epoch = 300 \n                \n                              #validation_steps=valid_set.shape[0]//batch_size,\n                        )    #steps_per_epoch=train_set.shape[0]//batch_size,\n                                                ","metadata":{"execution":{"iopub.status.busy":"2022-12-10T13:34:07.562967Z","iopub.execute_input":"2022-12-10T13:34:07.563392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_set.shape[0]//batch_size,train_set.shape[0]//batch_size,","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CNN which is using Huber Loss","metadata":{}},{"cell_type":"code","source":"    m2 = create_model()\n    \n    m2.compile(loss= losses[\"Huber\"], \n                    optimizer= tf.keras.optimizers.Adam(learning_rate=0.0001, beta_1=0.9, beta_2=0.999, decay=0.0001),\n                    metrics=['accuracy'])\n\n\n    \n    # Fit data to model\n    h2 = m2.fit(training_data,\n                        epochs= 15,\n                        validation_data=validation_data,\n                      \n                        steps_per_epoch = 300 \n                           #validation_steps=valid_set.shape[0]//batch_size,\n                        )    #steps_per_epoch=train_set.shape[0]//batch_size,\n                                                ","metadata":{"execution":{"iopub.status.idle":"2022-12-10T14:12:44.816996Z","shell.execute_reply.started":"2022-12-10T13:53:04.932116Z","shell.execute_reply":"2022-12-10T14:12:44.815711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CNN which is using KLDivergence Loss","metadata":{}},{"cell_type":"code","source":"    m3 = create_model()\n    \n    m3.compile(loss= losses[\"KLDivergence\"], \n                    optimizer= tf.keras.optimizers.Adam(learning_rate=0.0001, beta_1=0.9, beta_2=0.999, decay=0.0001),\n                    metrics=['accuracy'])\n\n\n    \n    # Fit data to model\n    h3 = m3.fit(training_data,\n                        epochs=15,\n                        validation_data=validation_data,\n                  \n                        steps_per_epoch = 300\n                       #validation_steps=valid_set.shape[0]//batch_size,\n                        )    #steps_per_epoch=train_set.shape[0]//batch_size,\n                                                ","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:12:44.818858Z","iopub.execute_input":"2022-12-10T14:12:44.819249Z","iopub.status.idle":"2022-12-10T14:32:48.958374Z","shell.execute_reply.started":"2022-12-10T14:12:44.819209Z","shell.execute_reply":"2022-12-10T14:32:48.957085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluation","metadata":{}},{"cell_type":"code","source":"X_test, y_test = next(validation_data)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:32:48.960109Z","iopub.execute_input":"2022-12-10T14:32:48.960499Z","iopub.status.idle":"2022-12-10T14:32:49.110570Z","shell.execute_reply.started":"2022-12-10T14:32:48.960460Z","shell.execute_reply":"2022-12-10T14:32:49.109398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels = []\n\nfor val in y_test:\n    \n    labels.append(np.argmax(val))\n\nlabels = np.array(labels)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:32:49.111905Z","iopub.execute_input":"2022-12-10T14:32:49.112266Z","iopub.status.idle":"2022-12-10T14:32:49.121787Z","shell.execute_reply.started":"2022-12-10T14:32:49.112229Z","shell.execute_reply":"2022-12-10T14:32:49.120425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def evaluate(preds1,y_val, n = 2 ):\n    \n    y_preds = []\n    for pred in preds1 : \n      y_preds.append(np.argmax(pred))\n    \n    \n    \n    acc = accuracy_score(y_val,y_preds)\n    pre = sum(precision_score(y_val,y_preds , average= None ))/n\n    recall = sum(recall_score(y_val,y_preds , average= None ))/n\n    F1 = sum(f1_score(y_val,y_preds , average= None))/n\n    \n    \n    \n    print('Accuracy Score: ',acc)\n    print('Precision Score : ',pre)\n    print('Recall Score : ', recall)\n    print('F1 Score(Class wise): ',F1)\n    print('Conf Matrix Score(Class wise):\\n ',confusion_matrix(y_val,y_preds ))\n    \n    \n    res = { \"acc\" : acc , \n            \"pre\" : pre , \n            \"recall\" : recall , \n            \"F1\" : F1\n    }\n    \n    return res ","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:47:26.768146Z","iopub.execute_input":"2022-12-10T14:47:26.768552Z","iopub.status.idle":"2022-12-10T14:47:26.777501Z","shell.execute_reply.started":"2022-12-10T14:47:26.768510Z","shell.execute_reply":"2022-12-10T14:47:26.776395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot(r):\n\n    plt.figure(figsize=(12, 16))\n\n    plt.subplot(4, 2, 1)\n    plt.plot(r.history['loss'], label='Loss')\n    plt.plot(r.history['val_loss'], label='val_Loss')\n    plt.title('Loss Function Evolution')\n    plt.legend()\n\n    plt.subplot(4, 2, 2)\n    plt.plot(r.history['accuracy'], label='accuracy')\n    plt.plot(r.history['val_accuracy'], label='val_accuracy')\n    plt.title('Accuracy Function Evolution')\n    plt.legend()","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:38:54.740862Z","iopub.execute_input":"2022-12-10T14:38:54.741557Z","iopub.status.idle":"2022-12-10T14:38:54.748462Z","shell.execute_reply.started":"2022-12-10T14:38:54.741519Z","shell.execute_reply":"2022-12-10T14:38:54.747338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluation of Cross Entropy","metadata":{}},{"cell_type":"code","source":"res1 = evaluate(m1.predict(X_test),labels)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:39:42.566456Z","iopub.execute_input":"2022-12-10T14:39:42.566829Z","iopub.status.idle":"2022-12-10T14:39:42.623538Z","shell.execute_reply.started":"2022-12-10T14:39:42.566798Z","shell.execute_reply":"2022-12-10T14:39:42.622571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cv_metrics = pd.DataFrame(res1,\n                          index=[0])\n\ncv_metrics.T.plot.bar(title=\"comparison\",\n                      legend=False);","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:40:02.179930Z","iopub.execute_input":"2022-12-10T14:40:02.180323Z","iopub.status.idle":"2022-12-10T14:40:02.384928Z","shell.execute_reply.started":"2022-12-10T14:40:02.180269Z","shell.execute_reply":"2022-12-10T14:40:02.384044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluation of Huber loss ","metadata":{}},{"cell_type":"code","source":"res2 = evaluate(m2.predict(X_test),labels)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:39:55.719819Z","iopub.execute_input":"2022-12-10T14:39:55.720182Z","iopub.status.idle":"2022-12-10T14:39:55.786201Z","shell.execute_reply.started":"2022-12-10T14:39:55.720152Z","shell.execute_reply":"2022-12-10T14:39:55.785161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cv_metrics = pd.DataFrame(res2,\n                          index=[0])\n\ncv_metrics.T.plot.bar(title=\"comparison\",\n                      legend=False);","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:32:49.855526Z","iopub.execute_input":"2022-12-10T14:32:49.856401Z","iopub.status.idle":"2022-12-10T14:32:50.052987Z","shell.execute_reply.started":"2022-12-10T14:32:49.856363Z","shell.execute_reply":"2022-12-10T14:32:50.052134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluation of Kl divergence","metadata":{}},{"cell_type":"code","source":"res3 = evaluate(m3.predict(X_test),labels)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:40:29.591345Z","iopub.execute_input":"2022-12-10T14:40:29.594327Z","iopub.status.idle":"2022-12-10T14:40:29.659152Z","shell.execute_reply.started":"2022-12-10T14:40:29.594289Z","shell.execute_reply":"2022-12-10T14:40:29.658070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cv_metrics = pd.DataFrame(res3,\n                          index=[0])\n\ncv_metrics.T.plot.bar(title=\"comparison\",\n                      legend=False);","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:40:35.414788Z","iopub.execute_input":"2022-12-10T14:40:35.415165Z","iopub.status.idle":"2022-12-10T14:40:35.617291Z","shell.execute_reply.started":"2022-12-10T14:40:35.415133Z","shell.execute_reply":"2022-12-10T14:40:35.616407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Graph","metadata":{}},{"cell_type":"markdown","source":"**Cross Entropy **","metadata":{}},{"cell_type":"code","source":"plot(h1)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:40:39.482485Z","iopub.execute_input":"2022-12-10T14:40:39.482876Z","iopub.status.idle":"2022-12-10T14:40:39.850257Z","shell.execute_reply.started":"2022-12-10T14:40:39.482843Z","shell.execute_reply":"2022-12-10T14:40:39.849289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Huber loss","metadata":{}},{"cell_type":"code","source":"plot(h2)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:40:40.535939Z","iopub.execute_input":"2022-12-10T14:40:40.536521Z","iopub.status.idle":"2022-12-10T14:40:40.904047Z","shell.execute_reply.started":"2022-12-10T14:40:40.536487Z","shell.execute_reply":"2022-12-10T14:40:40.903159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Kl Divergence **","metadata":{}},{"cell_type":"code","source":"plot(h3)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:40:45.143179Z","iopub.execute_input":"2022-12-10T14:40:45.143995Z","iopub.status.idle":"2022-12-10T14:40:45.494269Z","shell.execute_reply.started":"2022-12-10T14:40:45.143958Z","shell.execute_reply":"2022-12-10T14:40:45.493323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Ensemble\n\n\n> Ensemble modeling is a process where multiple diverse models are created to predict an outcome, either by using many different modeling algorithms or using different training data sets. The ensemble model then aggregates the prediction of each base model and results in once final prediction for the unseen data. [ref](https://www.sciencedirect.com/topics/computer-science/ensemble-modeling#:~:text=Ensemble%20modeling%20is%20a%20process,prediction%20for%20the%20unseen%20data.).","metadata":{}},{"cell_type":"markdown","source":"![ensemble_dia.png](https://www.researchgate.net/publication/293194221/figure/fig1/AS:342131643305984@1458581948247/The-common-ensemble-architecture.png)","metadata":{}},{"cell_type":"markdown","source":"**Our ensemble will be probability based ensemble**\n> \n> Suppose model1 on imag1 gives the output of [p11,p21,... p101] , where pij is the probability of each where i represents the class and j represents the number of the model\n> \n> Similarly model2 with also give us the similar output [p12,p22,... p102]\n> \n> The ensemble of the results are given below\n> \n> Rij = argmax((p01 + p02 + p03) + (p11 + p12 + p13 …. ) + (p101 + p102 + p103)..)\n> \n> Rij = argmax((2.1p01 + 2.3p02 + 1.9p03) + (2.1p11 + 2.3p12 + 1.9p13) + (2.1p101 + 2.3p102 + 1.9*p103)......)\n> \n> Rij = argmax((2.7p01 + 2.1p02 + 1.9p03) + (2.9p11 + 2.1p12 + 1.9p13) + (2.9p101 + 2.1p102 + 1.9*p103)......)\n> \n> Rij = argmax(max(p01 , p02 , p03) + max(p11 ,p12 , p13) + max(p101 , p102 , p103))......) \n> \n> Rij = argmax((p01 * p02 * p03) + (p11 * p12 * p13 …. ) + (p101 * p102 +* p103)..)","metadata":{}},{"cell_type":"code","source":"def ensemble(p1,p2,p3):\n    \n    m,n = p1.shape\n    \n    \n    z1 = np.zeros((m,n))\n    \n    z2 = np.zeros((m,n))\n    \n    z3 = np.zeros((m,n))\n    \n    z4 = np.zeros((m,n))\n    \n    z5 = np.zeros((m,n))\n    \n    \n    \n    \n    for i in range(0,m):\n        \n        for j in range(0,n):\n            \n            z1[i][j] = p1[i][j] + p2[i][j] + p3[i][j]\n            \n            z2[i][j] = 2.1*p1[i][j] + 2.3*p2[i][j] + 1.9*p3[i][j]\n            \n            z3[i][j] = 2.7*p1[i][j] + 2.1*p2[i][j] + 1.9*p3[i][j]\n            \n            z4[i][j] = max(p1[i][j] , p2[i][j] , p3[i][j])\n            \n            z5[i][j] = p1[i][j] * p2[i][j] * p3[i][j]\n    \n    \n    return z1,z2,z3,z4,z5","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:43:26.820220Z","iopub.execute_input":"2022-12-10T14:43:26.820972Z","iopub.status.idle":"2022-12-10T14:43:26.830473Z","shell.execute_reply.started":"2022-12-10T14:43:26.820933Z","shell.execute_reply":"2022-12-10T14:43:26.829484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluation of Ensemble methods","metadata":{}},{"cell_type":"code","source":"p1 = m1.predict(X_test)\np2 = m2.predict(X_test)\np3 = m3.predict(X_test) \n\nz1,z2,z3,z4,z5 = ensemble(p1,p2,p3)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:43:30.687148Z","iopub.execute_input":"2022-12-10T14:43:30.687880Z","iopub.status.idle":"2022-12-10T14:43:30.845905Z","shell.execute_reply.started":"2022-12-10T14:43:30.687843Z","shell.execute_reply":"2022-12-10T14:43:30.844863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"en1 = evaluate(z1,labels)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:43:31.882030Z","iopub.execute_input":"2022-12-10T14:43:31.882868Z","iopub.status.idle":"2022-12-10T14:43:31.896966Z","shell.execute_reply.started":"2022-12-10T14:43:31.882821Z","shell.execute_reply":"2022-12-10T14:43:31.895906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"en2 = evaluate(z2,labels)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:43:36.137381Z","iopub.execute_input":"2022-12-10T14:43:36.138534Z","iopub.status.idle":"2022-12-10T14:43:36.150019Z","shell.execute_reply.started":"2022-12-10T14:43:36.138485Z","shell.execute_reply":"2022-12-10T14:43:36.148859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"en3 = evaluate(z3,labels)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:47:46.126298Z","iopub.execute_input":"2022-12-10T14:47:46.126689Z","iopub.status.idle":"2022-12-10T14:47:46.137120Z","shell.execute_reply.started":"2022-12-10T14:47:46.126658Z","shell.execute_reply":"2022-12-10T14:47:46.135863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"en4 = evaluate(z4,labels)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:47:40.142960Z","iopub.execute_input":"2022-12-10T14:47:40.143488Z","iopub.status.idle":"2022-12-10T14:47:40.159824Z","shell.execute_reply.started":"2022-12-10T14:47:40.143456Z","shell.execute_reply":"2022-12-10T14:47:40.158998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"en5 = evaluate(z5,labels)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:45:09.925008Z","iopub.execute_input":"2022-12-10T14:45:09.925430Z","iopub.status.idle":"2022-12-10T14:45:09.938759Z","shell.execute_reply.started":"2022-12-10T14:45:09.925395Z","shell.execute_reply":"2022-12-10T14:45:09.937331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation**\n> \n> It is seen that the results are better than the base classifiers.  previously the best base classifier accuracy result was  **0.781** coming from the **kl divergence loss**, where as after the ensemble the results improved. The best result was **0.812**, which is coming from **ensemble method-3** The same trend can be seen in the other metrics as well\n","metadata":{}},{"cell_type":"code","source":"model_scores_acc = {\n    \n    \"base\" : 0.781, \n    \"Ensemble\" : 0.812, \n \n}","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:53:57.261344Z","iopub.execute_input":"2022-12-10T14:53:57.262073Z","iopub.status.idle":"2022-12-10T14:53:57.266891Z","shell.execute_reply.started":"2022-12-10T14:53:57.262030Z","shell.execute_reply":"2022-12-10T14:53:57.265910Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cv_metrics = pd.DataFrame(model_scores_acc,\n                          index=[0])\n\ncv_metrics.T.plot.bar(title=\"comparison of accuracy\",\n                      legend=False);","metadata":{"execution":{"iopub.status.busy":"2022-12-10T14:53:57.685502Z","iopub.execute_input":"2022-12-10T14:53:57.686154Z","iopub.status.idle":"2022-12-10T14:53:57.895163Z","shell.execute_reply.started":"2022-12-10T14:53:57.686113Z","shell.execute_reply":"2022-12-10T14:53:57.894229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion  \n\n\n> The purpose of this notebook is not to get high accuracy, rather the objective is to explore different loss functions and some simple ensemble methods on them. It is observed that the result after the ensemble got better. \n \n>The models can be trained for longer epochs to get more accuracy, thus resultiing better results after ensemble. Transfer learning and Fine tunning can be used for the ensemble methods.\n\n>**If you find this notebook usefull please give it an upvote it. Thank you so much for your time.**","metadata":{}},{"cell_type":"markdown","source":"# References\n\n\n[Heterogeneous versus Homogeneous Machine Learning Ensembles](https://www.researchgate.net/publication/293194221_Heterogeneous_versus_Homogeneous_Machine_Learning_Ensembles)\n\n[Building an Ensemble Learning Model Using Scikit-learn](https://towardsdatascience.com/ensemble-learning-using-scikit-learn-85c4531ff86a)\n\n[Data Mining Process](https://www.sciencedirect.com/topics/computer-science/ensemble-modeling#:~:text=Ensemble%20modeling%20is%20a%20process,prediction%20for%20the%20unseen%20data.)\n","metadata":{}}]}