{"metadata":{"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3649,"databundleVersionId":46718,"sourceType":"competition"}],"dockerImageVersionId":30733,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.13"},"papermill":{"default_parameters":{},"duration":9819.664326,"end_time":"2024-06-30T18:49:26.732527","environment_variables":{},"exception":true,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-06-30T16:05:47.068201","version":"2.5.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.741427,"end_time":"2024-06-30T16:05:50.541956","exception":false,"start_time":"2024-06-30T16:05:49.800529","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**LIBRARIES**","metadata":{"papermill":{"duration":0.007806,"end_time":"2024-06-30T16:05:50.558112","exception":false,"start_time":"2024-06-30T16:05:50.550306","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#importing libraries\n\nimport pandas as pd\npd.pandas.set_option('display.max_columns', None)\nimport numpy as np\n\nimport matplotlib.pyplot as plt \n%matplotlib inline\nimport seaborn as sns\n\nimport os\nfrom os import listdir\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nfrom sklearn.preprocessing import OneHotEncoder\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.model_selection import GridSearchCV\n\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"papermill":{"duration":1.370313,"end_time":"2024-06-30T16:05:51.936645","exception":false,"start_time":"2024-06-30T16:05:50.566332","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#tensorflow anfd keras\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import Sequential,datasets,layers,models,Model\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Dropout, BatchNormalization, Concatenate, Dense, Activation, Embedding, Flatten, LeakyReLU\nfrom tensorflow.keras.models import Model, Sequential\nfrom tensorflow.keras.regularizers import l1, l2\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\n#from tensorflow.keras.wrappers.scikit_learn import KerasClassifier\nfrom tensorflow.keras.activations import relu, sigmoid","metadata":{"papermill":{"duration":11.425457,"end_time":"2024-06-30T16:06:03.370572","exception":false,"start_time":"2024-06-30T16:05:51.945115","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from kerastuner.tuners import RandomSearch","metadata":{"papermill":{"duration":0.234022,"end_time":"2024-06-30T16:06:03.613060","exception":false,"start_time":"2024-06-30T16:06:03.379038","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**UNZIPPING THE FILE**\n\n* A .7z file is a compressed archive file created by the 7-Zip file archiver, a popular open-source compression tool. It's similar to other archive file formats like **.zip**, **.rar** and **.tar***","metadata":{"papermill":{"duration":0.007927,"end_time":"2024-06-30T16:06:03.629320","exception":false,"start_time":"2024-06-30T16:06:03.621393","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install py7zr\n\nfrom py7zr import unpack_7zarchive\nimport shutil\nshutil.register_unpack_format('7zip', ['.7z'], unpack_7zarchive)\nshutil.unpack_archive('/kaggle/input/cifar-10/train.7z', '/kaggle/working')","metadata":{"papermill":{"duration":40.969053,"end_time":"2024-06-30T16:06:44.607384","exception":false,"start_time":"2024-06-30T16:06:03.638331","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**READING THE FILE**","metadata":{"papermill":{"duration":0.00949,"end_time":"2024-06-30T16:06:44.627215","exception":false,"start_time":"2024-06-30T16:06:44.617725","status":"completed"},"tags":[]}},{"cell_type":"code","source":"tr_path = '/kaggle/working/train/'\ntr_lst = listdir(tr_path)\nX_lst = [0] * len(tr_lst)\n\nfor img in tr_lst:\n    num_img = int(img[:-4]) -1\n    image = Image.open(tr_path + img)\n    image_numpy = np.uint8(image)\n    X_lst[num_img] = image_numpy\n\nX = np.array(X_lst)\nprint(X.shape)","metadata":{"papermill":{"duration":8.372585,"end_time":"2024-06-30T16:06:53.009571","exception":false,"start_time":"2024-06-30T16:06:44.636986","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**DATA PREPROCESSING**","metadata":{"papermill":{"duration":0.009671,"end_time":"2024-06-30T16:06:53.030072","exception":false,"start_time":"2024-06-30T16:06:53.020401","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#applying OHE on the variables as they are in the format \n\ntr_df = pd.read_csv('/kaggle/input/cifar-10/trainLabels.csv')\nOHE = OneHotEncoder()\ny = OHE.fit_transform(tr_df[['label']])\ny = y.toarray()","metadata":{"papermill":{"duration":0.082431,"end_time":"2024-06-30T16:06:53.122341","exception":false,"start_time":"2024-06-30T16:06:53.039910","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#cheacking the distribution of categories\n#it should be equal or near to equal \n\nlabels_df = tr_df.groupby(['label']).count()\nlabels_matrix = y.sum(axis=0)\nprint(labels_df)\nprint(labels_matrix)","metadata":{"papermill":{"duration":0.034215,"end_time":"2024-06-30T16:06:53.166602","exception":false,"start_time":"2024-06-30T16:06:53.132387","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#splitting the data \n\nX_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.4)","metadata":{"papermill":{"duration":0.079789,"end_time":"2024-06-30T16:06:53.256653","exception":false,"start_time":"2024-06-30T16:06:53.176864","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**MODELS (ANN & CNN)**","metadata":{"papermill":{"duration":0.00964,"end_time":"2024-06-30T16:06:53.276748","exception":false,"start_time":"2024-06-30T16:06:53.267108","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"","metadata":{"papermill":{"duration":0.009616,"end_time":"2024-06-30T16:06:53.296312","exception":false,"start_time":"2024-06-30T16:06:53.286696","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#basic ANN model\n\ndef ann_model_1(ann1):\n    model = keras.Sequential()\n    for i in range(ann1.Int('num_layers', 2, 20)):\n        model.add(layers.Dense(units=ann1.Int('units_' + str(i),\n                                            min_value=32,\n                                            max_value=512,\n                                            step=32),\n                               activation='relu'))\n    model.add(layers.Dense(1, activation='linear'))\n    model.compile(\n        optimizer=keras.optimizers.Adam(\n            ann1.Choice('learning_rate', [1e-2, 1e-3, 1e-4])),\n        loss='mean_absolute_error',\n        metrics=['mean_absolute_error'])\n    return model\n","metadata":{"papermill":{"duration":0.019101,"end_time":"2024-06-30T16:06:53.325074","exception":false,"start_time":"2024-06-30T16:06:53.305973","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#hyperband tuner \n#concept in hyperparameter tuning \n\ntuner_rs = RandomSearch(\n    ann_model_1,\n    objective='val_mean_absolute_error',\n    max_trials=5,\n    executions_per_trial=3)","metadata":{"papermill":{"duration":0.717018,"end_time":"2024-06-30T16:06:54.052143","exception":false,"start_time":"2024-06-30T16:06:53.335125","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tuner_rs.search_space_summary()","metadata":{"papermill":{"duration":0.019206,"end_time":"2024-06-30T16:06:54.083523","exception":false,"start_time":"2024-06-30T16:06:54.064317","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#basic CNN model\n\nnormalized_X_train = X_train/255.0\nnormalized_X_train = normalized_X_train.astype('float32')\n\ncnn_1 = keras.Sequential([\n    keras.layers.Conv2D(64,(3,3),activation='relu',input_shape=(32,32,3)),\n    keras.layers.Flatten(),\n    keras.layers.Dense(64,activation='relu'),\n    keras.layers.Dense(32,activation='relu'),\n    keras.layers.Dense(10,activation='softmax')\n])\n\ncnn_1.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])\n\ncnn_1.fit(normalized_X_train,y_train, epochs=50)\n\ncnn_1_loss, cnn_1_score = cnn_1.evaluate(X_test,y_test,verbose=2)\n\nprint(\"The accuracy of the neural network on the test data is {0:4.3f}\".format(cnn_1_score,))","metadata":{"papermill":{"duration":121.429552,"end_time":"2024-06-30T16:08:55.533507","exception":false,"start_time":"2024-06-30T16:06:54.103955","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#complex & advanced CNN bt(interpretable)\n\n\n#param_dict\n\"\"\"\niterator that provides the training data = trai, \niterator that provides the testing data = tsai, \nnumber of filters tobe used in the convolutional layers = nfi, \nnumber of neurons = nni, \nnumber of convolutional layers = nci, \nnumber of max pooling layers = nmi \n\"\"\"\n\n\ndef create_neural_network(trai,tsai,nfi,nni,nci,nmi,overfitting,label=\"train vs. test accuracy\",verbosity=0):\n\n    \n#nearest integer rounding \n#overfitting regualarization(l1, l2)\n    numof_filters = round(nfi)\n    numof_neurons = round(nni)\n    numof_conv = round(nci)\n    numof_maxpool = round(nmi)\n    if overfitting == 'l1':\n        regularizer = l1(0.01)\n    elif overfitting == 'l2':\n        regularizer = l2(0.01)\n    else:\n        regularizer = None\n\n        \n#building neural network\n#handling errors \n    cnn_2 = Sequential()\n    cnn_2.add(Conv2D(numof_filters,(3,3),activation='relu',input_shape=(32,32,3)))\n    for max_pool in range(numof_maxpool):\n        try:\n            for i_conv in range(numof_conv):\n                cnn_2.add(Conv2D(numof_filters, (3, 3), activation='relu', padding='same',\n                                          kernel_regularizer=regularizer, bias_regularizer=regularizer))\n            cnn_2.add(MaxPooling2D(pool_size=(2,2)))\n            if overfitting == 'Dropout':\n                cnn_2.add(Dropout(0.25))\n            elif overfitting == 'Batch_Normalization':\n                cnn_2.add(BatchNormalization(momentum=0.99))\n        except:\n            break\n\n            \n#adding layers\n    cnn_2.add(keras.layers.Flatten())\n    cnn_2.add(keras.layers.Dense(numof_neurons,activation='relu'))\n    cnn_2.add(keras.layers.Dense(round(numof_neurons/2),activation='relu'))\n    cnn_2.add(keras.layers.Dense(10,activation='softmax'))\n    cnn_2.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n    print('Created a neural network with the following structure:')\n    print(cnn_2.summary())\n    \n    \n#caallbacks\n    reduce_learning_rate = keras.callbacks.ReduceLROnPlateau(patience=15, verbose=1)\n    early_stopping = keras.callbacks.EarlyStopping(patience=35, restore_best_weights=True, verbose=1)\n    print('Fitting the neural network')\n    historia = cnn_2.fit(trai, validation_data=tsai, epochs=50, verbose=verbosity, callbacks=[reduce_learning_rate,early_stopping])\n    \n    \n\n#plottting train vs. test accuracy \n    plt.figure(figsize=[15,15])\n    plt.figure()\n    if label is None:\n        plt.plot(historia.history['accuracy'], label='train accuracy')\n        plt.plot(historia.history['val_accuracy'], label='test accuracy')\n    else:\n        plt.plot(historia.history['accuracy'], label='train accuracy ({0})'.format(label))\n        plt.plot(historia.history['val_accuracy'], label='test accuracy ({0})'.format(label))\n    plt.legend()\n    \n    return cnn_2\n","metadata":{"papermill":{"duration":0.21047,"end_time":"2024-06-30T16:08:55.935928","exception":false,"start_time":"2024-06-30T16:08:55.725458","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#basic batch normalization\n\n\n#variable_dict \n\"\"\"\ntrain preprocessor = tr_preprocess\ntest preprocessor = ts_preprocess\ntrain iterator = tr_iter\ntest iterator = ts_iter\n\"\"\"\n\ntr_preprocess = ImageDataGenerator(rescale=1.0/255.0)\nts_preprocess = ImageDataGenerator(rescale=1.0/255.0)\n\ntr_iter = tr_preprocess.flow(X_train,y_train,batch_size=256)\nts_iter = ts_preprocess.flow(X_test,y_test,batch_size=256)\n\nbas_nn = create_neural_network(tr_iter,ts_iter,64,32,2,2,None)","metadata":{"papermill":{"duration":121.706022,"end_time":"2024-06-30T16:10:57.831850","exception":false,"start_time":"2024-06-30T16:08:56.125828","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#advanced & complex batch normalization\n\n\ntr_preprocess = ImageDataGenerator(rescale=1.0/255.0)\nts_preprocess = ImageDataGenerator(rescale=1.0/255.0)\n\ntr_iter = tr_preprocess.flow(X_train,y_train,batch_size=256)\nts_iter = ts_preprocess.flow(X_test,y_test,batch_size=256)\n\nnormalization = {}\n\nfor norm in [None,'l1','l2','Dropout','Batch_normalization']:\n    adv_nn = create_neural_network(tr_iter,ts_iter,64,32,2,2,norm,norm)\n    nn_loss, nn_score = adv_nn.evaluate(ts_iter,verbose=2)\n    normalization[norm] = nn_score\n\nprint(normalization)","metadata":{"papermill":{"duration":692.737847,"end_time":"2024-06-30T16:22:30.762656","exception":false,"start_time":"2024-06-30T16:10:58.024809","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#drop out is performing good amongst all \n#batch normalization is giving good train accuracy but test is not good comparing \n#using create_neural_netwrok\n\n\n#loop_dict\n\"\"\"\nnorm = normalization, \nconv = convolution,\nmx_pool = maxpool, \nneuron = neuros ,\nfilters = filters \n\"\"\"\n\nts_nn = {}\nts_score = {}\n\nfor norm in [None, 'Dropout']:\n    for conv in range(1,4):\n        for mx_pool in [1,2]:\n            for neuron in [8,32]:\n                for filters in [64,128]:\n                    indice = '{0}, {1}, {2}, {3}, {4}'.format(norm,conv,mx_pool,neuron,filters)\n                    txt = 'Creating the neural network for parameters: ' + indice\n                    print(txt)\n                    cnn_3 = create_neural_network(tr_iter, ts_iter, filters, neuron, conv, mx_pool, norm, indice)\n                    nn_loss, nn_score = cnn_3.evaluate(ts_iter, verbose=2)\n                    print('Test score: {0}'.format(nn_score))\n                    ts_score[indice] = nn_score\n                    ts_nn[indice] = cnn_3","metadata":{"papermill":{"duration":8114.812675,"end_time":"2024-06-30T18:37:45.783664","exception":false,"start_time":"2024-06-30T16:22:30.970989","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#selecting top 5 best performing neural networks\n\nbst_nn = sorted(ts_score.items(), key=lambda x : x[1], reverse=True)\nprint(bst_nn)\nbst_nn_indices = [ x[0] for x in bst_nn ]\nbst_5_nn_indices = bst_nn_indices[0:5]","metadata":{"papermill":{"duration":0.314484,"end_time":"2024-06-30T18:37:46.405970","exception":false,"start_time":"2024-06-30T18:37:46.091486","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#ensemble learning aka model stacking\n\ninpt = keras.Input(shape=(32,32,3))\n\nnn_legs = []\nfor key in bst_5_nn_indices:\n    output_i = ts_nn[key](inpt)\n    nn_legs.append(output_i)\n\nx = Concatenate()(nn_legs)\nx = Dense(1024,activation='relu')(x) \noutpt = Dense(10,activation='softmax')(x)\n\nmerged_model = Model(inputs=inpt,outputs=outpt)\nmerged_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n\nreduce_learning_rate = keras.callbacks.ReduceLROnPlateau(patience=15, verbose=1)\nearly_stopping = keras.callbacks.EarlyStopping(patience=35, restore_best_weights=True, verbose=1)\nmerged_model.fit(tr_iter, validation_data=ts_iter, epochs=50, callbacks=[reduce_learning_rate, early_stopping])","metadata":{"papermill":{"duration":402.23082,"end_time":"2024-06-30T18:44:28.938180","exception":false,"start_time":"2024-06-30T18:37:46.707360","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#unzipping test dataset\n\nshutil.unpack_archive('/kaggle/input/cifar-10/test.7z', '/tmp/testfiles')","metadata":{"papermill":{"duration":191.35605,"end_time":"2024-06-30T18:47:40.946093","exception":false,"start_time":"2024-06-30T18:44:29.590043","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! ls /tmp/testfiles/test | wc -l\n\n\n#file_count = len(os.listdir('/tmp/testfiles/test'))\n#print(file_count)","metadata":{"papermill":{"duration":2.543321,"end_time":"2024-06-30T18:47:44.199474","exception":false,"start_time":"2024-06-30T18:47:41.656153","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#predcting om test dataset \n\npred_preprocess = ImageDataGenerator(rescale=1.0 / 255.0)\npred_iter = pred_preprocess.flow_from_directory(directory='/tmp/testfiles',target_size=(32,32),color_mode='rgb',class_mode=None,batch_size=256,shuffle=False)\n\npred_final = merged_model.predict(pred_iter)","metadata":{"papermill":{"duration":93.317775,"end_time":"2024-06-30T18:49:18.189749","exception":false,"start_time":"2024-06-30T18:47:44.871974","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#one hot emcoding on prediction \n\npred_label = OHE.inverse_transform(pred_final)","metadata":{"papermill":{"duration":0.833764,"end_time":"2024-06-30T18:49:19.766834","exception":false,"start_time":"2024-06-30T18:49:18.933070","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#check dimensions of arrays in pred_label\ndimensions = set(len(arr) for arr in pred_label)\nif len(dimensions) > 1:\n    raise ValueError(\"Arrays in pred_label have different dimensions\")\n\n#create a submission dataframe with the predicted labels\nsubmission_df = pd.DataFrame({'id': range(1, len(pred_label[0]) + 1), 'label': pred_label[0]})\n\n#save the submission dataframe to a CSV file\nsubmission_df.to_csv('submission.csv', index=False)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#create a submission dataframe with the predicted labels\nsubmission_df = pd.DataFrame({'id': range(1, len(pred_label) + 1), 'label': pred_label.flatten()})\n\n#save the submission dataframe to a CSV file\nsubmission_df.to_csv('submission.csv', index=False)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}