{"cells":[{"metadata":{"_cell_guid":"768dba63-784f-4d6f-90c7-43efde4865b3","_uuid":"d272d5da0e6bea1768711dde0b9bedd4b85f4d1e"},"cell_type":"markdown","source":"#### Dogs vs. Cats Redux : Kernels Edition  - Raphaël \n\n*Hello, I am a French student. I learn statistics, math and computers at the University. I really enjoyed this project. It helped me to better understand convolutional neural networks. The objective here is to predict if it is a cat or a dog through photos. **I hope you will enjoy it and do not hesitate to comment. :-) ** *"},{"metadata":{"_cell_guid":"00b3f299-e546-4d38-a1e3-f6a7389aa4b8","_uuid":"277fd536343fd76bd615ab78c4e9fb068db08ced"},"cell_type":"markdown","source":"- *In the first part, I will create the project architecture: the train and validation and test directory that contains resized images (150x150) of cats and dogs.*\n\n- *In the second part, I will try to use some tools like the PIL package and the keras package to transform images to avoid overfitting.*\n\n- *In the third part, I will designed a CNN architecture to perform prediction and train it.*"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":false,"collapsed":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib\nfrom matplotlib.pyplot import plot\n\nimport zipfile # Preprocessing\n\nimport cv2 # Preprocessing\nimport imageio # Preprocessing\n\nimport PIL # Preprocessing\nimport keras\n\nimport shutil # Preprocessing\nimport os # Preprocessing\n\nfrom tqdm import tqdm # Progress bar\n# import keras_tqdm # Progress bar \n\nimport os\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"918ff281-1349-40b1-8fa3-6cb4ff5eb679","_uuid":"ce40532e3d502c5216e54766cf6bbb6f94b1f50c","collapsed":true,"trusted":false},"cell_type":"code","source":"from keras import backend \nfrom keras import applications\nfrom keras.preprocessing import image # Preprocessing\nfrom keras.preprocessing.image import ImageDataGenerator #DataAugmentation\nfrom keras.callbacks import * \nfrom keras.models import Sequential\nfrom keras.layers import Activation, Dropout, Flatten, Dense, BatchNormalization, Conv2D, MaxPooling2D","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"508a743c-f72b-4b0d-9087-20e081176ee4","_uuid":"433211de35b9877bb57f23182669a4239bbe193a"},"cell_type":"markdown","source":"##### List of pictures and labels :"},{"metadata":{"_cell_guid":"49551186-87ae-4ad0-ad09-15ceffecb79e","_uuid":"5159df40a5dd3e4c9472d3db8c5f2a334cc9098d","collapsed":true,"trusted":false},"cell_type":"code","source":"list_picture = os.listdir(\"../input/dogs-vs-cats-redux-kernels-edition/train/\")","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8a65e042-cdd9-44b8-8c93-3236a113ef83","_uuid":"ac4fc44e3f8245ab32b6062695dc8742aa62b345"},"cell_type":"markdown","source":"##### *Label :* \n- *Cat : 0*\n- *Dog : 1*"},{"metadata":{"_cell_guid":"12cb66f4-08da-4438-a82e-845dca36e2c4","_uuid":"ee8cebb6a4cc7b091d386a307d0102660031f418","collapsed":true,"trusted":false},"cell_type":"code","source":"df = pd.DataFrame({\"file\" : list_picture})\ndf['label'] = df['file'].apply(lambda x : 0 if x.split('.')[0] == 'cat' else 1)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b69d3fbd-ff28-469c-8a5c-2ab72bcd0038","_uuid":"914026eb5df78a88aa1a62619da8aa8b960d5853"},"cell_type":"markdown","source":"##### Validation dataset :"},{"metadata":{"_cell_guid":"3961cf46-d49e-4366-bb16-6ef5943480ef","_uuid":"dfda305e937b47fb8f035df8fbbf2d70411bb3b2","collapsed":true,"trusted":false},"cell_type":"code","source":"df['validation'] = df['label'].apply(lambda x : 1 if np.random.randint(0,11) <= 2 else 0)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"4dd36ec9-605c-49a0-8392-4083168acaa8","_uuid":"6ed3bf2c9d040e80fb1e6b1de6d1394172814d50","collapsed":true,"trusted":false},"cell_type":"code","source":"df.sample(10)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"988f8491-f787-4fa4-a2d4-a717bbb81fb4","_uuid":"d30489556b6a8bbc34e7fbd78190e9b009ed4ea6","collapsed":true,"trusted":false},"cell_type":"code","source":"print('Percentage of validation data : {}'.format(len(df[df['validation']==1])/len(df)*100))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8d58e30f-d1e8-439e-aa70-bb5d7102a7d4","_uuid":"e615832afde6330bccfb882ee1b1032f66f34b82"},"cell_type":"markdown","source":"*\"Train Test Split\" function could be better here. I should reduce the number of images in the validation dataset.*"},{"metadata":{"_cell_guid":"eb0f4bbf-edf7-42be-a263-f1f469c4ef43","_uuid":"6e629895a14815ee2f7e0ceff664de867831264f"},"cell_type":"markdown","source":"*We are going to create the architecture below :*\n- *The \"cat\" and \"dog\" folders contain associated images. This architecture works very well for binary classification with keras generators.*"},{"metadata":{"_cell_guid":"34beec55-b7eb-4b0c-a7e0-ab10b36e0079","_uuid":"2d3b6cb0db96fbea4503da4b72c312df0e71d502"},"cell_type":"markdown","source":"```\n.\n├── notebook.ypnb\n|\n|\n├── _data\n|    ├── _train\n|    |   ├── _cat\n|    |   └── _dog\n|    |\n|    └── _validation  \n|        ├── _cat\n|        └── _dog\n└── _test   \n        \n```"},{"metadata":{"_cell_guid":"1cadbe72-2d2a-4fa9-b745-9814e415ab03","_uuid":"ed42c8f6fa0fdf50dfd86370c2f8af4d9cc6d59d"},"cell_type":"markdown","source":"*The code below allows you to delete and rebuild the train architecture and validation above.*"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"trusted":false},"cell_type":"code","source":"try : \n    shutil.rmtree('data/train/cat/')\n    shutil.rmtree('data/train/dog/')\n    shutil.rmtree('data/validation/cat/')\n    shutil.rmtree('data/validation/dog/')\nexcept : \n    print('No folders to delete')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"89c29cfe-de83-4bdb-b024-ad7198caf050","_uuid":"5d5a4270ab8826723c01fd60079af4b96acbfde6","collapsed":true,"trusted":false},"cell_type":"code","source":"os.makedirs('data/train/cat/')\nos.makedirs('data/train/dog/')\nos.makedirs('data/validation/cat/')\nos.makedirs('data/validation/dog/')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"_cell_guid":"095d6970-2cc0-4229-93e2-50f3afee0c35","_uuid":"8cb9180a1d5bac8b218fe8c8e63ad01f57b05ca5","_kg_hide-input":false,"collapsed":true},"cell_type":"markdown","source":"```\nfor index, row in tqdm(df.iterrows(), total=len(df)):\n    \n    file_name = row['file']\n    img = cv2.imread('../input/dogs-vs-cats-redux-kernels-edition/train/{}'.format(file_name), cv2.IMREAD_COLOR)\n    # We resized picture thanks to open cv which is optionnal : \n    # Generator already resized images during training \n    #img = cv2.resize(img, (150, 150), interpolation=cv2.INTER_LINEAR) # INTER_CUBIC\n    \n    # cat if row['label'] == 0 else dog\n    if row['label'] == 0 :\n        file_name = 'cat/{}'.format(file_name)\n    else :\n        file_name = 'dog/{}'.format(file_name)\n    \n    # train if row['validation'] == 0 else validation\n    if row['validation'] == 0 : \n        imageio.imwrite('data/train/{}'.format(file_name), img)\n    else :\n        imageio.imwrite('data/validation/{}'.format(file_name), img)\n```"},{"metadata":{"_cell_guid":"065dc851-6bb6-4b6d-96f4-18646e734e4e","_uuid":"e5af2015829735d0f8fb39f3f09bcb9655d2e2a7","collapsed":true,"trusted":false},"cell_type":"code","source":"list_picture_test = os.listdir(\"../input/dogs-vs-cats-redux-kernels-edition/test/\")","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"37cdd051-8cf6-483c-9c0e-ec9a32582bf1","_uuid":"c61bff34abcfcbbb761f7b8103f525ed2bfa5fc0","collapsed":true,"trusted":false},"cell_type":"code","source":"try : \n    shutil.rmtree('reshape_test')\nexcept : \n    print('No folder to delete')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"9b23f0f2-7db6-4e3a-b0ce-f66f5b11b610","_uuid":"ce504572821c9b80be94085e1240368d55153133","collapsed":true,"trusted":false},"cell_type":"code","source":"os.makedirs('reshape_test')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a262777a-9c3e-4722-961f-3bd9788b7043","_uuid":"c9ebc1007c3741d47b96ed04663c8c2695c9f77c","collapsed":true},"cell_type":"markdown","source":"```\nfor file_path in tqdm(list_picture_test, total=len(list_picture_test)) :\n    \n    img = cv2.imread('../input/dogs-vs-cats-redux-kernels-edition/test/{}'.format(file_path), cv2.IMREAD_COLOR)\n    img = cv2.resize(img, (150, 150), interpolation=cv2.INTER_LINEAR) # INTER_CUBIC\n    imageio.imwrite('reshape_test/{}'.format(file_path), img)\n```"},{"metadata":{"_cell_guid":"5a0d4536-819f-4f76-9374-3d9beb743f56","_uuid":"e1b716cc3ef471db8556370d04b3f11b890d1508"},"cell_type":"markdown","source":"*We'll try to use some tools to transform Chucky's image.*"},{"metadata":{"_cell_guid":"782d1a02-b2e1-4e7d-86eb-e9ff90196f4c","_uuid":"3f02cbd253fe5b728d5ad714181fea756c4b85ef","collapsed":true,"trusted":false},"cell_type":"code","source":"img = keras.preprocessing.image.load_img('../input/dogs-vs-cats-redux-kernels-edition/train/dog.11931.jpg')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"460d16eb-d47b-4d88-bcad-eaee24226c18","_uuid":"930edf60fb01fea587759005303ac281e491438c","collapsed":true,"trusted":false},"cell_type":"code","source":"img","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"eb067acb-7df4-4b63-820a-93e895edcd0c","_uuid":"e6be8fd2947e79ee60c09ae9cc55f599b469409e"},"cell_type":"markdown","source":"*Convert an image as numpy array :*"},{"metadata":{"_cell_guid":"45d6c5e1-91da-4752-823a-0f32ea9e58e1","_uuid":"d3c45f543ca4c848ef248e6e4da3e44b6282452f","collapsed":true,"trusted":false},"cell_type":"code","source":"np.array(img)[0]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e8fe19eb-e079-4126-b967-657ec9ad9d59","_uuid":"7f2e8bf11049b5bd75ee77631e6ed8249884b7ae"},"cell_type":"markdown","source":"*Shape of the matrix :*"},{"metadata":{"_cell_guid":"fd66cd90-873c-4e13-8db5-7e32e5048bfa","_uuid":"95992dd2c799bea0efa2607f56e68cc32a8f9272","collapsed":true,"trusted":false},"cell_type":"code","source":"np.array(img).shape","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"db0190b1-3688-47de-922f-29d1283331e8","_uuid":"651fcbc5ae3f0f966ccd7381dbc37920ca740b03"},"cell_type":"markdown","source":"##### Example of preprocessing : "},{"metadata":{"_cell_guid":"92f53d95-492f-4417-b4e6-1fd0379e23a5","_uuid":"2a491aeafce5cfabdd738a7e271fa7cc0fb0e8b8","collapsed":true,"trusted":false},"cell_type":"code","source":"img_preprocessed = np.array(img.convert('L').rotate(45).transpose(PIL.Image.TRANSPOSE))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"cf20da21-d491-40a0-b978-ace2bc3457d1","_uuid":"fb1b1c4f1927980d94f53e30921386229590091b","collapsed":true,"trusted":false},"cell_type":"code","source":"matplotlib.pyplot.imshow(img_preprocessed, interpolation='nearest')\nmatplotlib.pyplot.show()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"05f21531-e3b2-4406-b3b6-d5409d507d60","_uuid":"c247cfd97ed63095d0ba89df60cac3c2f5d674de"},"cell_type":"markdown","source":"##### Keras : ImageDataGenerator"},{"metadata":{"_cell_guid":"5f0569ca-841f-4927-9321-98d8d0c98da5","_uuid":"a79fb761712a6bb6a95726e8a8d9458bc79ffaa6"},"cell_type":"markdown","source":"*The keras blog helped me understand how to make preprocessing on picture to avoid overfitting and to build a classifier without having a lot of data : \"Our model would never see twice the exact same picture. This helps prevent overfitting and helps the model generalize better\". (source : [Keras blog's](https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html))* "},{"metadata":{"_cell_guid":"df83fd27-972b-4f7f-a024-e11f78f2966c","_uuid":"d7bddcf5eda7cb708d81ba1e73513fe941306e9e","collapsed":true,"trusted":false},"cell_type":"code","source":"datagen = keras.preprocessing.image.ImageDataGenerator(\n        rotation_range=40,\n        width_shift_range=0.2,\n        height_shift_range=0.2,\n        rescale=1./255,\n        shear_range=0.2,\n        zoom_range=0.2,\n        horizontal_flip=True,\n        fill_mode='nearest')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"d9e37b33-f9ce-4be7-ba5e-457891789196","_uuid":"02b1f7b82b30ab53f81c58c0aa856c4f5be06b8d","collapsed":true,"trusted":false},"cell_type":"code","source":"x = keras.preprocessing.image.img_to_array(img)  # this is a Numpy array with shape (3, 150, 150)\nx = x.reshape((1,) + x.shape)  # this is a Numpy array with shape (1, 3, 150, 150)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"3a38eb3a-c45b-4ff9-b6b7-7df42c792b98","_uuid":"165b04635108c5c0f9509bae24c1866bfee1b963","collapsed":true,"trusted":false},"cell_type":"code","source":"try : \n    os.makedirs('example')\nexcept: \n    print('Folder already exist')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"93c2b26e-bf73-4442-82e0-0ca9c40cbace","_uuid":"25cf695c04b650ba5259048f54da3e76aa49f303"},"cell_type":"markdown","source":"*the .flow() command below generates batches of randomly transformed images and saves the results to the \"example\" directory :*"},{"metadata":{"_cell_guid":"34bf38b4-68aa-4b36-af76-9d5f368bb1bf","_uuid":"56559eb9e3f00693f386c56eaa272f1cc7d287ec","collapsed":true,"trusted":false},"cell_type":"code","source":"#i = 0\nfor batch in datagen.flow(x, save_to_dir='example', save_prefix='preprocessed', save_format='jpg'):\n    #Create 20 pictures  : \n    #i += 1\n    #if i > 20:\n    break  # otherwise the generator would loop indefinitely","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8e6bbd29-45c9-4289-80aa-8072c2fa5e24","_uuid":"7832700015222d070e8a6b9b2403efc9f0e04d9d","collapsed":true,"trusted":false},"cell_type":"code","source":"keras.preprocessing.image.load_img('example/{}'.format(os.listdir(\"example\")[0]))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"be0d6b94-1bd8-4760-a58d-d7faaf81d521","_uuid":"df02442768f9776912df4e32f5094cc9939eb2d2"},"cell_type":"markdown","source":"*Above we can see how the tools of the keras library have transformed Chucky's image.*"},{"metadata":{"_cell_guid":"1648bf63-0317-4a57-aa57-bb21e4ea1c44","_uuid":"577276578c734cd3ccc47ba14f59975a33092d8a"},"cell_type":"markdown","source":"### *Keras convolutionnal neural newtork : * "},{"metadata":{"_cell_guid":"8b328369-ae76-40fd-ab3e-5807f8a5d1ed","_uuid":"a67553eab9863d6c35aebf54a27cea85062d56b4"},"cell_type":"markdown","source":"![](https://cdn-images-1.medium.com/max/634/1*-r7EkRUvzkqDyyr2kwdeDg.png)"},{"metadata":{"_cell_guid":"2d60ee1e-aa7d-4c91-8f50-9e9f0f92e4ef","_uuid":"02dbfa58e07a81434e567690ee879022b930101d"},"cell_type":"markdown","source":"- *Here we defined the input shape :*"},{"metadata":{"_cell_guid":"62dc44ea-f6b9-461a-a1a6-5591fc8a5e6b","_uuid":"bdf87d5d843a159f7f320d110b341d0c29efd56c","collapsed":true,"trusted":false},"cell_type":"code","source":"img_width, img_height = 150, 150\n\nif backend.image_data_format() == 'channels_first':\n    input_shape = (3, img_width, img_height)\nelse:\n    input_shape = (img_width, img_height, 3)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"dec1de48-0168-475a-9896-8cee39e3359e","_uuid":"4e3342d72bb1a510e24cb2f22a6d2f2667e059d3","collapsed":true,"trusted":false},"cell_type":"code","source":"input_shape","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c5bad902-f8ce-4abb-b279-c5bf302037b5","_uuid":"cc5e3f210347f760a7ab8eee88be8637238917e0"},"cell_type":"markdown","source":"##### Model  : \n*I do not have a Nvidia GPU on my computer. I used batch normalization to accelerate model convergence and reduce overfitting. The batch normalization does not seem to improve my model but allows me to obtain satisfactory results with fewer iterations.\n(Source : [towardsdatascience](https://towardsdatascience.com/batch-normalization-in-neural-networks-1ac91516821c), [dlology](https://www.dlology.com/blog/one-simple-trick-to-train-keras-model-faster-with-batch-normalization/) and first introduced in the paper [Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://arxiv.org/pdf/1502.03167.pdf))*"},{"metadata":{"_cell_guid":"767c0f17-1318-4822-aff7-2d96c6af5508","_uuid":"24633f4a0ed26bf370e3ecf6ee8a3c22bc7eee10","collapsed":true,"trusted":false},"cell_type":"code","source":"model = Sequential()\n\nmodel.add(Conv2D(32, (3, 3), input_shape= input_shape, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\n\nmodel.add(Conv2D(32, (3, 3), use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Conv2D(64, (3, 3), use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\n\nmodel.add(Conv2D(64, (3, 3), use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Conv2D(128, (3, 3), use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\n\nmodel.add(Conv2D(128, (3, 3), use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Flatten()) # This converts our 3D feature maps to 1D feature vectors 3*3*128 \n\nmodel.add(Dense(128, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\nmodel.add(Dropout(0.5)) # The Dropout is aggresive but it allow to reduce overfiting.\n\nmodel.add(Dense(64, use_bias=False)) \nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\nmodel.add(Dropout(0.2))\n\nmodel.add(Dense(1)) # Binary classification\nmodel.add(BatchNormalization())\nmodel.add(Activation('sigmoid'))# Binary classification","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"51fca024-8a44-4531-9fe4-655cacfd1c8c","_uuid":"e16296b6a045f26da87474e01af6113ff0e09b1f"},"cell_type":"markdown","source":"*There is a very interesting article comparing the optimization function for this challenge : [shaoanlu](https://shaoanlu.wordpress.com/2017/05/29/sgd-all-which-one-is-the-best-optimizer-dogs-vs-cats-toy-experiment/)*"},{"metadata":{"_cell_guid":"f34e982d-14b8-49ec-89e9-efcb9db1c6da","_uuid":"98a89b9a452ee2ddf2a077651736e35d259b4cf2","collapsed":true,"trusted":false},"cell_type":"code","source":"model.compile(loss='binary_crossentropy',\n              optimizer='rmsprop',\n              metrics=['accuracy'])\n\n# optimizer = SGD(momentum=0.9, nesterov=True) could be better here ","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a35e7d4d-7f29-4212-965e-7d92250bf417","_uuid":"9161b57207f54c7d5149eb96a3cac3aa39e2d6c0"},"cell_type":"markdown","source":"*Summary of the model : *"},{"metadata":{"_cell_guid":"74ad172d-9c54-44f2-b104-fd544f43cf4b","_uuid":"928c1aa68f709abfbf72e4de3fb8beb8ad4bfd74","collapsed":true,"trusted":false},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c2d84fb0-51a3-4b1e-b291-e6365534e4b9","_uuid":"ac62fe3a91e6c4a367cb47968fe5ba7b8ac79210"},"cell_type":"markdown","source":"##### This is the augmentation configuration we will use for training\n*Our original images consist of RGB coefficients in the 0-255, to help the model process the images, we will scale these values.*"},{"metadata":{"_cell_guid":"9c776c4d-9b4e-4b2d-a83f-6c5526973334","_uuid":"dc82d09136b9c005c6267b592b6138f2d288cde8","collapsed":true,"trusted":false},"cell_type":"code","source":"train_datagen = keras.preprocessing.image.ImageDataGenerator(\n        rescale=1./255,\n        shear_range=0.2,\n        zoom_range=0.2,\n        horizontal_flip=True)\n\ntest_datagen = ImageDataGenerator(rescale=1./255)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"062d8bc8-8f61-4d9d-abfd-37af4d2481c5","_uuid":"044e6ca16e6d416854f1b3c9af068be853713c7f","collapsed":true,"trusted":false},"cell_type":"code","source":"batch_size = 32","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"af5f7bca-3c93-482d-9952-b63d80b90f49","_uuid":"b36e6c7c04004b692900791c6eb787bea4721298"},"cell_type":"markdown","source":"*These tools are generators. They allow images to be read in the specified directories.\nThey will generate batches of augmented images data. We will not load all images in the data set into memory. Instead, we will stream the images in batches.*"},{"metadata":{"_cell_guid":"4d16e65d-b844-429b-ae92-86448acbf20b","_uuid":"578d5941c7a0f8b039d98cf4dc4d950b19aff6e5","collapsed":true,"trusted":false},"cell_type":"code","source":"train_generator = train_datagen.flow_from_directory(\n        'data/train',  # target directory\n        target_size = (img_width, img_height),\n        batch_size = batch_size,\n        class_mode = 'binary')  # since we use binary_crossentropy loss, we need binary labels\n\n# this is a similar generator, for validation data\nvalidation_generator = test_datagen.flow_from_directory(\n        'data/validation',\n        target_size=(img_width, img_height),\n        batch_size=batch_size,\n        class_mode='binary')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"5b001c45-40c8-4833-be8a-ba5f31ba5e22","_uuid":"092290be681afc65fc904e8ea96f7de6a38a3d30"},"cell_type":"markdown","source":"***There is a great tutorial for this challenge : [ahmedbesbes.com](https://ahmedbesbes.com/understanding-deep-convolutional-neural-networks-with-a-practical-use-case-in-tensorflow-and-keras.html). This site allowed me to analyze the training phase of the model and visualize the results. I also learned to stop the convolutionnal neural network training before it overfit with the \"early stopping\" function thanks to this website.***"},{"metadata":{"_cell_guid":"44a89d18-4d01-4d8a-9d89-f8ba948057a0","_uuid":"66c800419a9789978a877f192f73455eed7d9874","collapsed":true,"trusted":false},"cell_type":"code","source":"# This class will allow me to visualize results of the training\nclass LossHistory(Callback):\n    \n    def on_train_begin(self, logs={}):\n        self.losses = []\n        self.val_losses = []\n        \n    def on_epoch_end(self, batch, logs={}):\n        self.losses.append(logs.get('loss'))\n        self.val_losses.append(logs.get('val_loss'))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8afe2274-3db1-4f8b-9e81-527a190a7955","_uuid":"28e89315e564228d2f4e3789a8580646012b30eb","collapsed":true,"trusted":false},"cell_type":"code","source":"history = LossHistory()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b935cb13-9001-486d-8daa-10b0219ae64c","_uuid":"cec431fe1b3f93016e645ebca9ba7a29896bc9d6"},"cell_type":"markdown","source":"*The function below allow to stop the model training when there are too much epochs without improvment performance. I will call it with the \"callback\" parameter when I fit the model.*"},{"metadata":{"_cell_guid":"9b55aac7-6e7e-4e84-87e9-3e49c93e60ae","_uuid":"587660c9bfb9f5100af224c1cb611336f8ce5fe8","collapsed":true,"trusted":false},"cell_type":"code","source":"early_stopping = keras.callbacks.EarlyStopping(monitor='val_loss', \n                              min_delta=0,\n                              patience=15, # Maximum number of epochs without improvment of val_loss, here I disabled early stopping\n                              verbose=0, \n                            mode='auto')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"dce5464e-d946-4cbf-820b-134c8e0a8cba","_uuid":"af8f3a2d5c5e734a0b50c87961d70ff39e7ef356"},"cell_type":"markdown","source":"*On Kaggle, the kernel can run for at least an hour, I can't train the model here. I'll just load the weights. keras_tqdm is a package that provides a nice output (javscript widget) when you train your neural network.*"},{"metadata":{"_cell_guid":"8ab5abc0-e276-4e33-a453-3bd7f709efb5","_uuid":"259e89a454092245b0bb49635fef9eaa3e45f150","collapsed":true,"trusted":false},"cell_type":"code","source":"model.load_weights(\"../input/weights-2/model_weights_2.h5\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"_cell_guid":"8b5efa32-fafc-412e-b40d-414a1d22d70e","_uuid":"2b092ee8621368afc1666b5d17dc586c1d03415d","_kg_hide-input":true,"collapsed":true},"cell_type":"markdown","source":"```fitted_model = model.fit_generator(\n                    train_generator,\n                    steps_per_epoch = 400,\n                    epochs = 15,\n                    validation_data = validation_generator,\n                    validation_steps =  800 // batch_size,\n                    verbose = 0,\n                    callbacks=[keras_tqdm.TQDMNotebookCallback(leave_inner=True, leave_outer=True), early_stopping, history])\nmodel.save_weights('model.h5')```\n"},{"metadata":{"_cell_guid":"5f7fdc34-d378-4d47-ad10-d12ce1953c91","_uuid":"6afe302065b0f290e72653c9977866f5ff40452e"},"cell_type":"markdown","source":"![](https://image.noelshack.com/fichiers/2018/17/3/1524679368-fit.png)"},{"metadata":{"_cell_guid":"9c4b35d1-826b-4815-8682-49409fab882e","_uuid":"2d5e143b9df57397fbe270699867e48b57c2a917"},"cell_type":"markdown","source":"The graph below crosses the evolution of the loss indicator according to the training and validation data sets."},{"metadata":{"_cell_guid":"46beb040-7c62-4a0f-b433-52f2fd002221","_uuid":"be8c3bf3a19261999cbdeb008a54ed2ab0f46196","collapsed":true},"cell_type":"markdown","source":"```\nlosses, val_losses = history.losses, history.val_losses\nfig = matplotlib.pyplot.figure(figsize=(15, 5))\nmatplotlib.pyplot.plot(fitted_model.history['loss'], 'g', label=\"train losses\")\nmatplotlib.pyplot.plot(fitted_model.history['val_loss'], 'r', label=\"val losses\")\nmatplotlib.pyplot.grid(True)\nmatplotlib.pyplot.title('Training loss vs. Validation loss')\nmatplotlib.pyplot.xlabel('Epochs')\nmatplotlib.pyplot.ylabel('Loss')\nmatplotlib.pyplot.legend()\nmatplotlib.pyplot.show()\n```"},{"metadata":{"_cell_guid":"3490e57e-005a-47ac-8090-66b946267900","_uuid":"7ba776792a7fbaf78028171023b981f0b180001a"},"cell_type":"markdown","source":"![](https://image.noelshack.com/fichiers/2018/17/3/1524679364-log-loss.png)\n\n"},{"metadata":{"_cell_guid":"0d3f1cfb-c969-4b61-a257-658eb4cba8fd","_uuid":"2fb94397590739d926ad5815730bd1a961d3a783"},"cell_type":"markdown","source":"*The graph below crosses the evolution of the accuracy indicator according to the training and validation data sets.*"},{"metadata":{"_cell_guid":"8c104efe-4c0e-47f6-a0f6-729fd436041a","_uuid":"ea69ab451aba046bc0f2f48e122edbeb19bbca5b"},"cell_type":"markdown","source":"```losses, val_losses = history.losses, history.val_losses\nfig = matplotlib.pyplot.figure(figsize=(15, 5))\nmatplotlib.pyplot.plot(fitted_model.history['acc'], 'g', label=\"accuracy on train set\")\nmatplotlib.pyplot.plot(fitted_model.history['val_acc'], 'r', label=\"accuracy on validation set\")\nmatplotlib.pyplot.grid(True)\nmatplotlib.pyplot.title('Training Accuracy vs. Validation Accuracy')\nmatplotlib.pyplot.xlabel('Epochs')\nmatplotlib.pyplot.ylabel('Accuracy')\nmatplotlib.pyplot.legend()\nmatplotlib.pyplot.show()```"},{"metadata":{"_cell_guid":"91fb15a2-094f-4c95-a041-fd55c543b612","_uuid":"7f6b805f80b037bb70ede905c1a4b914596d4468"},"cell_type":"markdown","source":"![](https://image.noelshack.com/fichiers/2018/17/3/1524679641-accuracy.png)"},{"metadata":{"_cell_guid":"f6d7b043-3b39-4e63-8512-8bff6a67eebc","_uuid":"81b7840f61efdb2e3ce1988cf07308a85f0629ce"},"cell_type":"markdown","source":"##### There is a large variance of the accuracy of validation. To reduce it, I would multiply the number of steps per epoch. Change the optimization function could reduce this variance too. \n\n##### I could initialize an early stopping rule: 3 times without improving the accuracy of validation and we stop the training phase. I should train the model on Amazon Web Service : [Keras_AWS](https://blog.keras.io/running-jupyter-notebooks-on-gpu-on-aws-a-starter-guide.html)."},{"metadata":{"_cell_guid":"12004822-6dfb-4385-8cf9-e4d2518ad16a","_uuid":"e12b051f85166c97d843ab15a7f366ceabfeb3c0"},"cell_type":"markdown","source":"*Now it's time to predict test dataset :*"},{"metadata":{"_cell_guid":"07d59a58-383c-4137-a413-d52731b70d85","_uuid":"a22372ece00c3e40d62aaf05d06e4f879af8a241","collapsed":true,"trusted":false},"cell_type":"code","source":"list_picture_test = [int(file.split('.')[0]) for file in os.listdir('reshape_test')]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"1d520d30-8eac-4bcf-8bd9-36fec6f5bff2","_uuid":"93476acca6849e64af795bd4f322fcbef7e5e772","collapsed":true,"trusted":false},"cell_type":"code","source":"list_picture_test.sort()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f432a398-20da-4589-a968-0c5560068585","_uuid":"264eace635229ad4192253926373a19ca69dbfb5","collapsed":true,"trusted":false},"cell_type":"code","source":"list_picture_test = ['{}.jpg'.format(file) for file in list_picture_test]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"1bdd403e-279f-410c-a961-1d63e3c5b093","_uuid":"b0c00adb5396f5a1ecddaa47e3dc54ecb8b21b00","collapsed":true,"trusted":false},"cell_type":"code","source":"classes = []","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"3d3d9d71-b9b0-4663-9c3c-622ce9eec2b8","_uuid":"44b2d698270957ca4826fabf823eb99184047ed9"},"cell_type":"markdown","source":"*Kaggle evaluate the probability associate to the class dog : *"},{"metadata":{"_cell_guid":"e95b33ef-8ffa-46b7-ae3f-4567429bb348","_uuid":"e0abc63a792723463b68d6004591ca6b62339b44","collapsed":true},"cell_type":"markdown","source":"```\nfor file in tqdm(list_picture_test) : \n    # Preprocessing images to predict :\n    img = image.load_img('reshape_test/{}'.format(file), target_size=(img_width, img_height))\n    img = image.img_to_array(img)\n    img = np.expand_dims(img, axis=0)\n    img = img/255 # Scaling image \n    classes.append(model.predict_proba(img))\n```"},{"metadata":{"_cell_guid":"bd8194b3-e9a3-4571-8b49-e3ccdbd9e6c5","_uuid":"70ca64f6b567ed92bc364de6ea390fdbc20b9bfc","collapsed":true},"cell_type":"markdown","source":"```classes = [x[0][0] for x in classes]```"},{"metadata":{"_cell_guid":"1e2ad850-4e29-4f4e-a5ca-72789e03a5b0","_uuid":"e8ae4276abd7643d1e8549565ded251e66546be3","collapsed":true},"cell_type":"markdown","source":"```list_id = list(range(1,12501))```"},{"metadata":{"_cell_guid":"09147e30-0dde-482b-89c6-33f63eaff769","_uuid":"0bb611b38869a347864cb98629974fe5b79c46de","collapsed":true},"cell_type":"markdown","source":"```submission = pd.DataFrame({\n            'id': list_id,\n            'label':classes\n            }, columns=['id','label'])```"},{"metadata":{"_cell_guid":"a7fea300-650b-4a88-9d95-ff1ad6bce272","_uuid":"58f4ada437f5729c7e72ad5bfe7025745d337b3b","collapsed":true},"cell_type":"markdown","source":"```submission.head()```"},{"metadata":{"_cell_guid":"8152a195-d95b-426e-92bd-f3247d750add","_uuid":"bd10ce76e2d7f5d07fc331563df0f8db4346a5ce"},"cell_type":"markdown","source":"*Save submission as csv file : *"},{"metadata":{"_cell_guid":"e0be3461-e273-49ab-aacd-51a532868a82","_uuid":"8396929a5467375a47a768d2269c69e6555cad58","collapsed":true},"cell_type":"markdown","source":"```submission.to_csv('submission_.csv', sep=\",\", index=False)```"},{"metadata":{"_cell_guid":"dc8aae6f-205e-4687-a2ca-01e30e87b8e5","_uuid":"ada574ec6aa5be6c3f41efbc8ff1c68b5d34176c"},"cell_type":"markdown","source":"### Public score : 0.26122 (log loss) which is a pretty good result, I expected less because of the few number of iteration of the model."},{"metadata":{"_cell_guid":"8f6cd251-c07f-47a5-836c-749521b77678","_uuid":"c3394cf9826e5ffebe8bf96f69b2ab06fe1f613f"},"cell_type":"markdown","source":"* *Will you guess if it's a dog or a cat?  This image has been the subject of a major Twitter debate.*"},{"metadata":{"_cell_guid":"0670b523-6960-439e-86dc-f5bfc6e87ade","_uuid":"8fa7092bae16a631e5cc7cadb23be6eb8cffef70"},"cell_type":"markdown","source":"![](https://image.noelshack.com/fichiers/2018/17/3/1524684284-atchoum.jpeg)"},{"metadata":{"_cell_guid":"00c389dd-c95f-4b33-9c90-eab0b4c2670c","_uuid":"49362285078b293dcfff5380bc8e1b9d92a6c72f","collapsed":true,"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"73ea7922-1d13-4034-b018-30798cec40b6","_uuid":"c4c669fea62de2aa77bb13aa9c0e6fcc44f23786"},"cell_type":"markdown","source":"It's a cat ! Instagram of \"Atchoumthecat\" [here](https://www.instagram.com/atchoumthecat/?utm_source=ig_embed&action=profilevisit) and his personnal website [here](https://www.atchoumthecat.com/my-story.html)."},{"metadata":{"_cell_guid":"94c0021c-2b40-425f-84d8-9b73cc4ab119","_uuid":"048357f52a33bb88ac6b1466611976ca1cf18d02"},"cell_type":"markdown","source":"*Try my model on random images, 1 = Dog, 0 = Cat. *"},{"metadata":{"_cell_guid":"54a14c07-8840-4f2e-9cdc-e3b725f38ca2","_uuid":"76123d138d9b7fc5c04159dec49797491b248d03","trusted":false,"collapsed":true},"cell_type":"code","source":"img = keras.preprocessing.image.load_img('../input/reshape/144.jpg')\nimg","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"3a1aec89-597d-4145-90a7-ea997a173f1d","_uuid":"ef39c1ee338932edc25d3dde72824e4dbb747bf9","trusted":false,"collapsed":true},"cell_type":"code","source":"img = image.img_to_array(img)\nimg = np.expand_dims(img, axis=0)\nimg = img/255 # Scaling image\nmodel.predict_classes(img)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"064c0ff9-df81-46d6-b55b-bee5a92ecf4a","_uuid":"c9d270b3ca3ef5d816ddccf5627e36eb7252f6ed","collapsed":true,"trusted":false},"cell_type":"code","source":"img = keras.preprocessing.image.load_img('../input/reshape/145.jpg')\nimg","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"7bd33c7f-7e20-4cde-99d8-1d94ae99deda","_uuid":"8c764e850be53b9cc5c1c51753ea592a7d63d58a","collapsed":true,"trusted":false},"cell_type":"code","source":"img = image.img_to_array(img)\nimg = np.expand_dims(img, axis=0)\nimg = img/255 # Scaling image\nmodel.predict_classes(img)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"d48afaee-8ba5-49c4-9ea5-27442e082ddd","_uuid":"059efa53d32e2577f1e90dd1efcfd56e6bd2307c","collapsed":true,"trusted":false},"cell_type":"code","source":"img = keras.preprocessing.image.load_img('../input/reshape/146.jpg')\nimg","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"56ec531e-f607-406e-be41-e5e422f6e77c","_uuid":"7a0da03655d1cf4a53d00de05f88bfbca2b8a0bd","collapsed":true,"trusted":false},"cell_type":"code","source":"img = image.img_to_array(img)\nimg = np.expand_dims(img, axis=0)\nimg = img/255 # Scaling image\nmodel.predict_classes(img)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"62e271ed-f2fb-4822-9e36-9c72a295b624","_uuid":"cd965a45ea7b1503e18c245b0d8a1ec63cd554f1"},"cell_type":"markdown","source":"Feel free to put an thumbs up if this notebook interested you.\n\nRaphaël"},{"metadata":{"_cell_guid":"95ff2766-7b6b-43d3-8e7d-d915e3c07554","_uuid":"d630aefc1894518e4574a3f8833e2ddd359298bd","collapsed":true,"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"pygments_lexer":"ipython3","codemirror_mode":{"version":3,"name":"ipython"},"name":"python","version":"3.6.5","mimetype":"text/x-python","file_extension":".py","nbconvert_exporter":"python"}},"nbformat":4,"nbformat_minor":1}