{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"7220b6ff-6e89-7e10-1171-f1e0b8cf68f4"},"source":"With an interesting playground competition, Kaggle challenges us to classify cats and dogs given a training set of 25000 images and a test set of 12500 images.\n\nFor this task, I chose to proceed as follow:\n\n- Load image and augment the dataset using skimage, scipy and numpy\n- Split the dataset using sklearn\n- Create a convolutional network with lasagne \n- Feed and train the model using lasagne and theano\n- Test the model and create a submission file with predicted class probabilities using pandas\n\nThe network is trained on GPU with a g2.2xlarge AWS instance (about 220 seconds per epoch).\n\nFirst, we import our toolbox:"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"c8cd25cb-7a48-7eb9-b930-c2a802f79388"},"outputs":[],"source":"import sys\nimport os\nimport time\nfrom skimage import color, io\nfrom scipy.misc import imresize\nfrom scipy.ndimage.filters import gaussian_filter\nfrom glob import glob\nimport numpy as np\nimport theano\nimport theano.tensor as T\nfrom sklearn.cross_validation import train_test_split\nimport lasagne\nimport pandas as pd\n\ncurrent_path = '../Data/'\nfiles_path = '../Data/train/'\ntest_path = '../input/train/'"},{"cell_type":"markdown","metadata":{"_cell_guid":"ce3a3577-3fb2-51e2-ad13-8ffcc2f07c92"},"source":"Let's define a good size for image to feed to the network: Even if a small size hardly reduce the time per epoch during training, we need to be careful to don't loose too much pixel information. After several random displays, 127x127 seems to be a good trade-off and fit with convolutional network architecture below.  \n\nOn the other hand, convolutional layers need to be fed by a 4D tensor such as :\n\n- The first dimension refers to the size of the mini-batch\n- The second is the number of channels, here we only focus on grayscale because the colors are not very important to distinguish cats and dogs. It also reduce the computational cost needed to analyse image patches.\n- The third and the last are the spatial size of images. \n\nTo increase the number of samples, we augment the dataset with two methods :\n- Gaussian filter : it applies weights to the image according to a gaussian distribution\n- Flipping : first column become the last one and so on : image looks like mirrored.\n\nHence, we obtain a 75000 images dataset that will scale with the capacity of the network (too much capacity and few data could drive to a strong overfitting). Other existing methods are efficient such as random crops, rotation or ZCA whitening.  "},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"5315d9d9-5b5a-5bfc-92de-87734b672455"},"outputs":[],"source":"def Dense2OneHot(y_set):\n    y_out = np.zeros(shape = [y_set.shape[0], 2])\n    for i in range(y_set.shape[0]):\n        if y_set[i] == 1.:\n            y_out[i] = [1., 0.]\n        else: \n            y_out[i] = [0., 1.]\n    return y_out\n\ndef load_img(path, size_image):\n    cat_files_path = os.path.join(path, 'cat*.jpg')\n    dog_files_path = os.path.join(path, 'dog*.jpg')\n\n    cat_files = sorted(glob(cat_files_path))\n    dog_files = sorted(glob(dog_files_path))\n\n    n_files = len(cat_files) + len(dog_files)\n\n    #Initialisation of images arrays\n    X = np.zeros((n_files, 1, size_image, size_image), dtype='float32')\n    X_Gauss = np.zeros((n_files, 1, size_image, size_image), dtype='float32')\n    X_Flip = np.zeros((n_files, 1, size_image, size_image), dtype='float32')\n    \n    #Initialisation of labels arrays\n    y = np.zeros(n_files)\n    y_Gauss = np.zeros(n_files)\n    y_Flip = np.zeros(n_files)\n    \n    count = 0\n    \n    print(\"Loading Cats files...\")\n    for f in cat_files:\n        try:\n            img = io.imread(f, as_grey=True)\n            new_img = imresize(img, (size_image, size_image))\n            X[count] = np.array(new_img)\n            X_Gauss[count] = gaussian_filter(X[count, 0], sigma=1)\n            X_Flip[count] = np.fliplr(X[count, 0])\n            y[count] = 0\n            y_Gauss[count] = 0\n            y_Flip[count] = 0\n            count += 1\n        except:\n            continue\n    \n    print(\"Loading Dogs files...\")\n    for f in dog_files:\n        try:\n            img = io.imread(f, as_grey=True)\n            new_img = imresize(img, (size_image, size_image))\n            X[count] = np.array(new_img)\n            X_Gauss[count] = gaussian_filter(X[count, 0], sigma=1)\n            X_Flip[count] = np.fliplr(X[count, 0])\n            y[count] = 1\n            y_Gauss[count] = 1\n            y_Flip[count] = 1\n            count += 1\n        except:\n            continue\n\n    X = np.vstack((X,X_Gauss))\n    X = np.vstack((X,X_Flip))\n    y = np.concatenate((y, y_Gauss), axis=0)\n    y = np.concatenate((y, y_Flip), axis=0)\n    \n    #Rescale Data\n    for i in range(X.shape[0]):\n        X[i, 0] = X[i, 0] / 255\n    \n    y = Dense2OneHot(y)\n    X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=1260)     \n\n    return X_train, X_val, y_train, y_val"},{"cell_type":"markdown","metadata":{"_cell_guid":"c5f6212e-2a4d-cb12-c97e-e37e588b33c5"},"source":"Now we'll define a convolutional network architecture with 5 convolutional layers and 2 max pooling layers to reduce the spatial size of the output. For a good understanding, I will detail the input and output size of each layer. For more details, I strongly recommend to read this course from Stanford team : http://cs231n.github.io/\n\nFor this problem, I found reasonnable to define my own architecture instead of choosing an existing one (AlexNet, VGGNet, etc.) because it makes more easy to tune parameters and test quickly how well suited is my model. Another reason is that Lasagne does not provide existing trained networks.  "},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"b852f964-0d3e-5bef-cc4a-0610ee9782b5"},"outputs":[],"source":"def Convolutional_network():\n    #Input: Mini-batch of images (with None value because it is not known at the moment) of size (127, 127) with one channel\n    network = lasagne.layers.InputLayer((None, 1, size_image, size_image))\n    \n    #Input: W1, H1, D1 = (127, 127, 1)\n    network = lasagne.layers.Conv2DLayer(network, num_filters=64, \n                                         filter_size=5, nonlinearity=lasagne.nonlinearities.rectify, \n                                         stride=3, pad=2, W=lasagne.init.HeNormal(gain='relu'))\n    \n    #Output: W2 = H1 = (W1 - filter_size + 2 * pad) / stride + 1 = 43 ; D2 = num_filters = 64\n    # => Output size: (43, 43, 64)\n    \n    #Apply drop out to shut down neurons with probability p \n    network = lasagne.layers.DropoutLayer(network, p=0.5)\n\n    #Input: (43, 43, 64)\n    network = lasagne.layers.Conv2DLayer(network, num_filters=128, filter_size=3, \n                                         nonlinearity=lasagne.nonlinearities.rectify,\n                                         stride=2, pad=1, W=lasagne.init.HeNormal(gain='relu'))\n    \n    #Output: (22, 22, 256)\n    \n    network = lasagne.layers.DropoutLayer(network, p=0.4)\n    \n    #Input: (22, 22, 256)\n    network = lasagne.layers.Conv2DLayer(network, num_filters=128, filter_size=3, \n                                         nonlinearity=lasagne.nonlinearities.rectify,\n                                         stride=1, pad=0, W=lasagne.init.HeNormal(gain='relu'))\n    \n    #Output: (20, 20, 256)\n    \n    network = lasagne.layers.DropoutLayer(network, p=0.3)\n    \n    #Apply Max Pooling to reduce the spatial size (and hence the amount of hyperparameters)\n    #Input: (20, 20, 256)\n    network = lasagne.layers.MaxPool2DLayer(network, pool_size=(2, 2))\n\n    #Output: (10, 10, 256) (same rule as convolutionnal layer without padding)\n    \n    network = lasagne.layers.DropoutLayer(network, p=0.3)\n    \n    #Input: (10, 10, 256)\n    network = lasagne.layers.Conv2DLayer(network, num_filters=256, filter_size=3, \n                                         nonlinearity=lasagne.nonlinearities.rectify,\n                                         stride=1, pad=0, W=lasagne.init.HeNormal(gain='relu'))\n    #Output: (8, 8, 256)\n    \n    #Input: (8, 8, 256)\n    network = lasagne.layers.Conv2DLayer(network, num_filters=256, filter_size=3, \n                                         nonlinearity=lasagne.nonlinearities.rectify,\n                                         stride=1, pad=0, W=lasagne.init.HeNormal(gain='relu'))\n    #Output: (6, 6, 256)\n    \n    #Input: (6, 6, 256)\n    network = lasagne.layers.MaxPool2DLayer(network, pool_size=(2, 2))\n        \n    #Output: (3, 3, 256)     \n    \n    #Add a Dense Layer to perform classification\n    network = lasagne.layers.DenseLayer(network,\n                                      num_units=1024,\n                                      nonlinearity=lasagne.nonlinearities.rectify)\n    \n    #Network's output layer. It is important to set a sigmoid activation to avoid \"nan\" values during gradient descent\n    network = lasagne.layers.DenseLayer(network,\n                                      num_units=2,\n                                      nonlinearity=lasagne.nonlinearities.sigmoid)        \n    \n    return network"},{"cell_type":"markdown","metadata":{"_cell_guid":"c581551f-56ba-768a-ee52-b9ff8664ffd9"},"source":"For this example, the program will run on CPU. I recommend to use GPU to make the training faster. Even though Lasagne is a Theano library and support GPU transparently, the use of CUDNN layers is more efficient (at least with a NVIDIA K520, never tried on other devices). To set it, proceed as following:\n\n- Import following libraries : \nimport lasagne.layers.cuda_convnet\nimport pylearn2\nfrom pylearn2.sandbox.cuda_convnet.filter_acts import FilterActs\n- Replace \"lasagne.layers.Conv2DLayer\" by \"lasagne.layers.cuda_convnet.Conv2DCCLayer\"\n    and \"lasagne.layers.MaxPool2DLayer\" by \"lasagne.layers.cuda_convnet.MaxPool2DCCLayer\" "},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"42519690-7e49-4972-0e4f-cc01873fea99"},"outputs":[],"source":"#Generate mini-batch of samples :\ndef batch_gen(X, y, N):\n    while True:\n        idx = np.random.choice(len(y), N)\n        yield X[idx].astype('float32'), y[idx].astype('int32')"},{"cell_type":"markdown","metadata":{"_cell_guid":"861d88fb-f153-b550-1024-7493172e56d5"},"source":"We now could create a training function that load the entire dataset, define some Theano functions to compute loss, accuracy and perform the gradient descent. Several update rules and objective functions are already implemented on Lasagne. For this problem, we chose Adamax algorithm (a infinity norm based variant of Adam, see Kingma et al. for more explanations) with a learning rate of 0.008."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"dcd9c53e-5f38-276d-fc2c-ffd46d51c722"},"outputs":[],"source":"def Train_model(BATCH_SIZE, number_of_epochs, lr):\n    \n    network = Convolutional_network()\n    #To load existing model :\n    #with np.load('model.npz') as f:\n         #param_values = [f['arr_%d' % i] for i in range(len(f.files))]                     \n    #lasagne.layers.set_all_param_values(network, param_values)\n    \n    X_train, X_val, y_train, y_val = load_img(files_path, size_image)\n    \n    X_sym = T.tensor4()\n    y_sym = T.matrix()\n\n    output = lasagne.layers.get_output(network, X_sym)\n    pred = (output > 0.5)\n    #Compute binary crossentropy loss\n    loss = T.mean(lasagne.objectives.binary_crossentropy(output, y_sym))\n    #Compute accuracy\n    acc = T.mean(T.eq(pred, y_sym))\n\n    params = lasagne.layers.get_all_params(network)\n    #Compute the gradient of the loss function according to the parameters\n    grad = T.grad(loss, params)\n    updates = lasagne.updates.adamax(grad, params, learning_rate=lr)\n\n    f_train = theano.function([X_sym, y_sym], [loss, acc], updates=updates)\n    f_val = theano.function([X_sym, y_sym], [loss, acc])\n\n    N_BATCHES = len(X_train) // BATCH_SIZE\n    N_VAL_BATCHES = len(X_val) // BATCH_SIZE\n    train_batches = batch_gen(X_train, y_train, BATCH_SIZE)\n    val_batches = batch_gen(X_val, y_val, BATCH_SIZE)\n    #Store learning state after each epoch into a list\n    loss_per_epoch = []\n\n    print(\"Start training\")\n    for epoch in range(number_of_epochs):\n        train_loss = 0\n        train_acc = 0\n        start_time = time.time()\n        for _ in range(N_BATCHES):\n            X, y = next(train_batches)\n            loss, acc = f_train(X, y)\n            train_loss += loss\n            train_acc += acc\n        train_loss /= N_BATCHES\n        train_acc /= N_BATCHES\n            \n        val_loss = 0\n        val_acc = 0\n        for _ in range(N_VAL_BATCHES):\n            X, y = next(val_batches)\n            loss, acc = f_val(X, y)\n            val_loss += loss\n            val_acc += acc\n        val_loss /= N_VAL_BATCHES\n        val_acc /= N_VAL_BATCHES\n        print(\"Epoch {} of {} took {:.3f}s\".format(\n              epoch + 1, number_of_epochs, time.time() - start_time))\n        print('  Train loss: {:.03f} - Validation Loss: {:.03f}'.format(\n              train_loss, val_loss))\n        print('  Train accuracy: {:.03f}'.format(train_acc))\n        print('  Validation accuracy: {:.03f}'.format(val_acc))\n        loss_per_epoch.append([epoch+1, train_loss, val_loss])\n    #Save the learning states in a file to analyze it later\n    np.savetxt(\"Learning.csv\", loss_per_epoch, delimiter=',')\n\n    np.savez('model.npz', *lasagne.layers.get_all_param_values(network)) \n    \n    return network"},{"cell_type":"markdown","metadata":{"_cell_guid":"9b2993c8-9799-374b-d947-bf287ed017c7"},"source":"Above function will return the trained network.\nNow we need to test the model. This task could be long because it proceeds image per image and the test set is quite large. However, it is not possible to test the entire set at a time because it will consume too much memory and return a nice memory_error."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"a60aa041-0f14-7f0c-7467-e783b445dceb"},"outputs":[],"source":"def TestModel(network):\n    \n    X_sym = T.tensor4()\n\n    X_test = load_testset()\n    probas = np.zeros((12500, 2))\n    \n    output = lasagne.layers.get_output(network, X_sym, deterministic=True)\n    f_predict_probas = theano.function([X_sym], output)\n    \n    print(\"Predicting labels\")\n    for i in range(X_test.shape[0]):\n        probas[i] = f_predict_probas(X_test[i:i+1])\n    print(\"Done\")    \n    #If you are using an interactive environment, it could be interesting to play with predicted probabilities\n    # and see which image is the most unbalanced. For it, add the following line to the program :\n    # global Dogs_preds\n    Dogs_preds = probas[:,0]\n    return Dogs_preds"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"5b9d6615-cb87-8991-f116-106d1ee86a55"},"outputs":[],"source":"def Submit(predictions):\n    \n    Submission = pd.DataFrame({'id': [i for i in range(1, len(predictions)+1)],\n                                      'label': predictions})\n    \n    Submission.to_csv(os.path.join(current_path,r'Submission.csv'), header=True, index=False)        \n    print(\"CSV submission file generated on working directory\")"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"e80e0422-2564-34d3-a7cc-208e179f37ca"},"outputs":[],"source":"if __name__ == \"__main__\":\n    #network = Train_model(BATCH_SIZE=128, number_of_epochs=100, lr=0.008)\n    #Dogs_preds = TestModel(network)\n    #Submit(Dogs_preds)"},{"cell_type":"markdown","metadata":{"_cell_guid":"ee32bf22-9838-f2c2-980f-6926fe4a2bc4"},"source":"Thanks for reading ! \n\nEven if lasagne seems more tricky than Keras or sknn, it offers a lot of possibilities and is a good introduction to theano symbolic computations. \nPlease, let me know if you have questions about the code or some tips to improve it."}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.0"}},"nbformat":4,"nbformat_minor":0}