{"cells":[{"metadata":{"_uuid":"8121e7c2eb13c540a89dc0a40365378bdc066bed"},"cell_type":"markdown","source":"**Aim**: In this kernel we are going to make a small CNN and train it on the dataset.\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nfrom sklearn.model_selection import train_test_split\nfrom tqdm import  tqdm\nprint(os.listdir(\"../input\"))\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten,GlobalAvgPool2D\nfrom keras.layers import Conv2D, MaxPooling2D\nimport keras\nfrom keras.preprocessing import image\nfrom keras.applications.vgg16 import VGG16\nfrom keras.applications.resnet50 import ResNet50\nfrom keras.applications import xception\nfrom keras.applications import inception_v3\nfrom keras.applications.vgg16 import preprocess_input, decode_predictions\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import log_loss, accuracy_score","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6436f302c700e89c7a39126dcb469d01535124af","_kg_hide-input":true},"cell_type":"code","source":"df = pd.read_csv(\"../input/labels.csv\")\ndf.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5ee1c1c9675ee3798e52afa86aed72759103694a","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"sam = pd.read_csv(\"../input/sample_submission.csv\")\nsam.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"286b0da7cd5e6c57cdec0f6cf5e670d25e78755c","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"breed_ls = list(df.groupby('breed').count().sort_values(by='id', ascending=False).index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"69b59e62cae8fa4b601b2441b62097e7846c8baf","_kg_hide-input":true},"cell_type":"code","source":"import random\nimport cv2\nfrom keras.preprocessing import image\n\ndef getRandomImageList(breed_name, no_of_samples=100):\n    global df\n    random_images = []\n    for index, row in df.iterrows():\n        if row['breed'] == breed_name:\n            random_images.append(row['id'])\n    random_images = random.sample(random_images, no_of_samples)\n    return random_images\n\ndef readImgResize(name,path):\n    img = cv2.imread(path+name)\n    img = cv2.resize(img,(128,128))\n    return image.img_to_array(img)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b0961fa4e770f62e7d516d3d791ae9154d06ddd"},"cell_type":"markdown","source":"**Input Size**:  As there are resource restriction, I am using 128x128x3 by default. However, the input shape is a choice. If you have a network big enough, you can use a bigger network size. I have trained U-Nets on 1024x1024x3 resolution but then again they don't contain dense layer. The point is that the performance of CNNs is also dependent on the input shape same as the depth of network. If I were to train it on something which has a bit more power, I'd use 350x350x3 as input size as I fount out in EDA\n\n**Sample per class**: I am including 65 samples from each class. The only reason is that I don't want to use the entire dataset as it is. The top frequent classes are twices as likely to be encountred during training as compared to the bottom classes. Hence to provide a bit of balance, I am going to take 65 random samples per class. "},{"metadata":{"trusted":true,"_uuid":"2b5a2c9090c15f5c8a967aa005f4bbf7cdb14d3c","scrolled":true},"cell_type":"code","source":"INPUT_SIZE = 128\nnum_class=120\n# breedToTrain = breed_ls[num_class]#incase we just want to train top-n freq classes\nsamples = 65\n\n# There must exist a better way of doing the sampling. This is pretty slow \n\nimage_label = []\nnum = 0\nimport tqdm\nfor i,breed in tqdm.tqdm( enumerate(breed_ls[:num_class])):\n\n    ls = getRandomImageList(breed,samples) \n    image_label.extend(ls)\n    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1b3fe1ab89810ee0db08a755864eaf8f6c68f381"},"cell_type":"markdown","source":"Let's create a new balanced dataframe using the sampled data points."},{"metadata":{"trusted":true,"_uuid":"d3c6525b8756c833c77f8a6ade78bedaba90f119"},"cell_type":"code","source":"new_df = pd.DataFrame({\"id\":image_label})\nnew_df = pd.merge(new_df, df, how='inner', on=['id'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"061c483338b4393d46dcc411755c9d9855f8f131"},"cell_type":"markdown","source":"Label Encode the \"breed\" column of the new data frame.  "},{"metadata":{"trusted":true,"_uuid":"01edfd94281b33f3477dc2ff0e51538ed6ca31a7"},"cell_type":"code","source":"\nfrom sklearn.preprocessing import LabelEncoder\nle = LabelEncoder()\ncat = le.fit_transform(new_df.breed)\nfrom keras.utils.np_utils import to_categorical\nmat = to_categorical(cat)\n\n\ntraining_data = np.zeros(shape=(len(new_df.id),128,128,3))\nfor i,j in tqdm.tqdm(enumerate(new_df[\"id\"])):\n    training_data[i]=readImgResize(j+\".jpg\",path=\"../input/train/\")\n    \n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"12dd0fbb81e3a042279af2571144bee342bfa96c"},"cell_type":"markdown","source":"Split the training data using train_test_split.  After that del the existing training data and label encoded matrix in case you run into memory error later on."},{"metadata":{"trusted":true,"_uuid":"d1b27a83b5b4350dfa588c4fddd2d98017ea3cc3"},"cell_type":"code","source":"# from keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten,GlobalAvgPool2D\nfrom keras.layers import Conv2D, MaxPooling2D\nimport keras\nfrom sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split( training_data, mat, test_size=0.05, random_state=11)\ndel training_data, mat, new_df, df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f0c2801f956bd3be24fc5bc85e3daabc2d66ef1"},"cell_type":"markdown","source":"Perform a sanity check on the inputs and targets before fitting"},{"metadata":{"trusted":true,"_uuid":"6c389e08c5ca4a5ebfedbc99a9cf30bec9e1297e","_kg_hide-input":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.figure(figsize=(15,7))\nfor i in range(6):\n    idx = random.randint(0,len(X_train))\n    itemindex = np.squeeze(np.where(y_train[idx]==1.)).tolist()\n    plt.subplot(2,3,i+1)\n    plt.imshow(X_train[idx]/255)\n    name = breed_ls[itemindex]\n    plt.xlabel(str(name))\n    # np.squeeze(itemindex[0]).tolist()\n    name\n# itemindex","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"771cba4df6206b87625aba54e2e3c58652118a0f"},"cell_type":"markdown","source":"Create a basic CNN "},{"metadata":{"trusted":true,"_uuid":"0b05555f741303b777e2458a774863e4b3fd21aa"},"cell_type":"code","source":"def createModel(nClasses=120):\n    model = Sequential()\n    model.add(Conv2D(32, (3, 3), padding='same', activation='relu', input_shape=(128,128,3)))\n    model.add(Conv2D(32, (3, 3), activation='relu'))\n    model.add(MaxPooling2D(pool_size=(2, 2)))\n#     model.add(Dropout(0.25))\n\n    model.add(Conv2D(64, (3, 3), padding='same', activation='relu'))\n    model.add(Conv2D(64, (3, 3), activation='relu'))\n    model.add(MaxPooling2D(pool_size=(2, 2)))\n    model.add(Dropout(0.15))\n\n    model.add(Conv2D(128, (3, 3), padding='same', activation='relu'))\n    model.add(Conv2D(128, (3, 3), activation='relu'))\n    model.add(MaxPooling2D(pool_size=(2, 2)))\n    model.add(Dropout(0.2))\n\n    \n    \n    model.add(Flatten())\n    model.add(Dense(512, activation='relu'))\n    model.add(Dropout(0.25))\n    model.add(Dense(nClasses, activation='softmax'))\n     \n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"78102b58e3586b0594a9ca5cd0f32ba4c676f252"},"cell_type":"markdown","source":"If the model seems to be overfitting, use dropout or increase the number of dropout neurons.  If the model is underfitting, use a larger network or train longer."},{"metadata":{"trusted":true,"_uuid":"fe029ec2d76b61216006a45b40ef970883622771"},"cell_type":"code","source":"from keras.optimizers import adam\nmodel1 = createModel()\nbatch_size = 16\nepochs = 100\n\nmodel1.compile(adam(lr=0.00001), loss='categorical_crossentropy', metrics=['accuracy'])\n ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"64e70049ec8d0dd01d93d4ea8c9c40b0b87833b7"},"cell_type":"markdown","source":"Using ImageDataGenerator for image augmentation."},{"metadata":{"trusted":true,"_uuid":"d66537de9cae87934bcf121295592bffc374ae4e"},"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\ntrain_datagen = ImageDataGenerator(\n    rescale=1./255,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range=0.2,\n    horizontal_flip=True,\n    fill_mode='nearest')\n\ntest_datagen=ImageDataGenerator(rescale=1./255)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0fb6f150f38e7cc553c24de45adef6b6ceca56e8"},"cell_type":"code","source":"train_set=train_datagen.flow(X_train,y=y_train,batch_size=64)\ntest_set=test_datagen.flow(X_test,y=y_test,batch_size=64)\nmodel1.fit_generator(train_set,\n                      steps_per_epoch = 256,\n                      validation_data = test_set,\n                      validation_steps = 4,\n                      epochs = 25,\n                      verbose = 1)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9e1bcc6e3d12ab690e2fcce019ff7eee94c8bd4e"},"cell_type":"markdown","source":"The model is trained for a small number of steps. However, the validation and training loss kept deceasing. The accuracies are also in unison. Thus, we are not over fitting, at least yet. You can increase the number of epochs on your local machine.\n\nperform a bit of memory cleaning to get prediction on test data."},{"metadata":{"trusted":true,"_uuid":"6565cddbc08d14eb4499c8d672b1c689cfc51c21"},"cell_type":"code","source":"model1.save('my_model.h5')\ndel train_set, test_set, image_label\ndel df ,new_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"043712864054d37b54472b2bb9fc76a10874ad0a"},"cell_type":"code","source":"te = os.listdir(\"../input/test/\")\nte_in = np.zeros((len(te),128,128,3))\nfor num , i in enumerate(te):\n    img = readImgResize(i,path=\"../input/test/\")/255\n    te_in[num]=img\n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d21eea5e9a9474e4578ce9840d8ed382798e55a6"},"cell_type":"code","source":"# create submission\npred = model1.predict(te_in) \nsubmission = pd.DataFrame(pred , columns =le.classes_.tolist())\nsubmission[\"id\"]=[i.split(\".\")[0] for i in os.listdir(\"../input/test/\")]\nsubmission = submission[[\"id\"]+submission.columns[:-1].tolist()]\nsubmission.to_csv(\"submission.csv\",index = False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}