{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"09cc9d1c-e909-0225-0e47-029bfbf9b101"},"source":"This notebook does the following:\n 1. Extracts the data from the files, \n 2. Gives some insight on image size,\n 3. Resizes the images so they can be used as input for NNs, \n 4. Saves the results and drinks a coffee"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"2f1dd20b-8d57-3d3d-0b2b-71af49035049"},"outputs":[],"source":"import os\nimport numpy as np\nimport tensorflow as tf\nfrom matplotlib import pyplot as plt\n%matplotlib inline\nimport pickle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"23fc1d42-4d76-f620-3acb-560284707421"},"outputs":[],"source":"TRAIN_DIR = '../input/train/'\nTEST_DIR = '../input/test/'"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"339d456a-be0f-2a89-a1d3-7067399cbc64"},"outputs":[],"source":"# On the kaggle notebook\n# we only take the first 2000 from the training set\n# and only the first 1000 from the test set\n# REMOVE [0:2000] and [0:1000] when running locally\ntrain_image_file_names = [TRAIN_DIR+i for i in os.listdir(TRAIN_DIR)][0:2000] \ntest_image_file_names = [TEST_DIR+i for i in os.listdir(TEST_DIR)][0:1000]"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"0916524d-a1ed-9700-5c3a-11f6eb8b2611"},"outputs":[],"source":"# Slow, yet simple implementation with tensorflow\n# could be rewritten to be much faster\n# (which is not really needed as it takes less than 5 minutes on my laptop)\ndef decode_image(image_file_names, resize_func=None):\n    \n    images = []\n    \n    graph = tf.Graph()\n    with graph.as_default():\n        file_name = tf.placeholder(dtype=tf.string)\n        file = tf.read_file(file_name)\n        image = tf.image.decode_jpeg(file)\n        if resize_func != None:\n            image = resize_func(image)\n    \n    with tf.Session(graph=graph) as session:\n        tf.initialize_all_variables().run()   \n        for i in range(len(image_file_names)):\n            images.append(session.run(image, feed_dict={file_name: image_file_names[i]}))\n            if (i+1) % 1000 == 0:\n                print('Images processed: ',i+1)\n        \n        session.close()\n    \n    return images"},{"cell_type":"markdown","metadata":{"_cell_guid":"5d23d821-4e63-e60a-783a-7b6e116e5a5e"},"source":"**Having aspect ratio vary so much is not good :( , lets take a closer look...**"},{"cell_type":"markdown","metadata":{"_cell_guid":"311b745a-58d8-f1f3-bd0a-b955cc66a192"},"source":"**Some images are horizontally stretched, others are vertically stretched.** If you are to crop them to particular size WxH, then W/H should be around 1.15 ( the mean aspect ratio). Yet, I decided to go for the long shot and not crop at all..."},{"cell_type":"markdown","metadata":{"_cell_guid":"b068d10a-bad1-7c45-951b-a03131776501"},"source":"If you use all the data locally, you will see there are only two images in the training set that are bigger than 500x500..."},{"cell_type":"markdown","metadata":{"_cell_guid":"d225f52f-ce9e-0c93-2595-8d0e6c2ddf6e"},"source":"**Instead of cropping I will use padding and resize all images to 500x500... Which will make models run much slower (then cropping to i.e. 64x64), yet be more accurate, as no information is lost or distorted...**"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"bdcb40fa-168b-7a1f-0397-db5c9364adf9"},"outputs":[],"source":"WIDTH=100\nHEIGHT=100\ndef reduce_image(org_image):\n    tf.image.resize_image_with_crop_or_pad(org_image, 500, 500 )\n    #tf.image.resize_images(image, [WIDTH, HEIGHT])"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"bc2dde51-507d-6b96-5e43-2dcb2a0d7467"},"outputs":[],"source":"processed_train_images = decode_image(train_image_file_names, resize_func=reduce_image)\nprocessed_test_images = decode_image(test_image_file_names, resize_func=reduce_image)"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"b32b4ca8-ede0-75be-f460-0b3eed7ea60f"},"outputs":[],"source":"# Chech the shapes\nprint(np.shape(processed_train_images))\nprint(np.shape(processed_test_images))"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"c6de215a-2053-5a92-00a0-f545aa903dd8"},"outputs":[],"source":"# Let's check how the images look like\nfor i in range(10):\n    plt.imshow(processed_train_images[i])\n    plt.show()"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"a5f7407f-6324-385f-3fe2-7b0542b37127"},"outputs":[],"source":"def create_batch(data, label, batch_size):\n    i = 0\n    while i*batch_size <= len(data):\n        with open(label+ '_' + str(i) +'.pickle', 'wb') as handle:\n            content = data[(i * batch_size):((i+1) * batch_size)]\n            pickle.dump(content, handle)\n            print('Saved',label,'part #' + str(i), 'with', len(content),'entries.')\n        i += 1"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"ff98058d-2239-afee-a6fc-9c9830591d66"},"outputs":[],"source":"# Create one hot encoding for labels\nlabels = [[1., 0.] if 'dog' in name else [0., 1.] for name in train_image_file_names]"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"f83adafd-df23-3802-c444-802218b66912"},"outputs":[],"source":"# TO EXPORT DATA WHEN RUNNING LOCALLY - UNCOMMENT THIS LINES\n# a batch with 5000 images has a size of around 3.5 GB\n#create_batch(labels, 'data/train_labels', 5000)\n#create_batch(processed_train_images, 'data/train_images', 5000)\n#create_batch(processed_test_images, 'data/test_images', 5000)"},{"cell_type":"markdown","metadata":{"_cell_guid":"931e10a0-52fc-f655-18b8-1169b1b2179c"},"source":"**All you need now is powerful cluster of servers that can handle so much data :)**"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"c96a2533-24b6-e631-e23b-a8bd938d9103"},"outputs":[],"source":""}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":0}