{"cells":[{"metadata":{},"cell_type":"markdown","source":"
\n
\n**“You don't have to be great to start, but you have to start to be great.”** ~ Zig Ziglar\n
\n
\n\nHey friends,
\nthis is a gentle introduction into Image Classification with the Python library fast.ai.\n\nWhat do you need to get started?\n- basic coding skills in Python\n\n[click if you don't know how to code at all](https://www.codecademy.com/learn/learn-python-3)
\n[click if you know some coding, but not Python](https://developers.google.com/edu/python/)
\n[click if you don't know the fast.ai course](https://course.fast.ai/videos/?lesson=1)\n\nFeel free to fork and tweak constants to improve accuracy. Just by doing so, I was able to get a score of 99.32% easily.\n\nCheck out the [Q&A section](#Questions-and-Answers) of this notebook. If you have any questions or certain explanations weren't completely clear, let me know in the comments. I'm happy to help everyone :)\n\n1. [Preparation](#Preparation)
\n a.) [Explore Kaggle competitions page](#Explore-Kaggle-competitions-page)
\n b.) [Setup environment](#Setup-environment)
\n c.) [Explore data](#Explore-data)
\n d.) [Data wrangling](#Data-wrangling)
\n e.) [Display images](#Display-images)
\n f.) [Load data into DataBunch](#Load-data-into-DataBunch)\n3. [Training](#Training)\n4. [Evaluation](#Evaluation)\n5. [Prediction](#Prediction)\n6. [Questions and Answers](#Questions-and-Answers)\n\n[click for the fast.ai documentation](https://docs.fast.ai)\n\n"},{"metadata":{},"cell_type":"markdown","source":"Be aware that you don't see all the code immediately. Some code is hidden! This is on purpose, we're focusing on the the most important aspects and you don't have to understand everything in detail yet.\nNevertheless you can show the code by clicking on the **Code** button.\n"},{"metadata":{},"cell_type":"markdown","source":"# Preparation"},{"metadata":{},"cell_type":"markdown","source":"## Explore Kaggle competitions page\nBefore you start coding anything, it is really important to get a thorough understanding of the competition. Kaggle already provides the most important information on the competition page ([https://www.kaggle.com/c/digit-recognizer](https://www.kaggle.com/c/digit-recognizer)). So the first thing you do when you start a new competition, is to read through the entire page and every tab within.\n\n**I will give you a quick overview of all the important information:**\n\n### Goal of the competition\n\"The goal in this competition is to take an image of a handwritten single digit, and determine what that digit is.\"\n\n### Evaluation\n\"This competition is evaluated on the categorization accuracy of your predictions\"
\nSo the used metric is: Correctly classified Images / Total number of images\n\n### Data ([read more here](https://www.kaggle.com/c/digit-recognizer/data))\n\n\nThere are 3 different files: train.csv, test.csv, sample_submission.csv\n\n#### train.csv\n- greyscale images from zero through nine\n- file contains all necessary information for training the model\n- each row is one image\n- first row of each image is the label. It tells us which digit is shown.\n- other 784 rows are the pixels for each digit, they should be read like this\n\n`000 001 002 003 ... 026 027\n028 029 030 031 ... 054 055\n056 057 058 059 ... 082 083\n | | | | ... | |\n728 729 730 731 ... 754 755\n756 757 758 759 ... 782 783`\n\n#### test.csv\n- greyscale images from zero through nine\n- structure is the same as in train.csv, but there are no labels\n- these 28000 images are used later to test how good our model is\n\n#### sample_submission.csv\n- show us, how to structure our prediction results to submit them to the competition\n- we need two columns: ImageId and Label\n- the rows don't need to be ordered\n- the submission file should look like this:

\n `ImageId, Label`
\n `1, 3`
\n `2, 4`
\n `3, 9`
\n `4, 1`
\n `5, 7`
\n `(27995 more lines)`\n"},{"metadata":{},"cell_type":"markdown","source":"## Setup environment\nFirst of all make sure you enabled GPU so the model trains faster

\nThen import the fast.ai library"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# the following three lines are suggested by the fast.ai course\n%reload_ext autoreload\n%autoreload 2\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# hide warnings\nimport warnings\nwarnings.simplefilter('ignore')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# the fast.ai library, used to easily build neural networks and train them\nfrom fastai import *\nfrom fastai.vision import *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Explore Data\nI already summarized all the data we're given for the competition, but let's still check out the files:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# to get all files from a directory\nimport os\n\n# to easier work with paths\nfrom pathlib import Path\n\n# to read and manipulate .csv-files\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"INPUT = Path(\"../input/digit-recognizer\")\nos.listdir(INPUT)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at 'train.csv' and 'test.csv':"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(INPUT/\"train.csv\")\ntrain_df.head(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv(INPUT/\"test.csv\")\ntest_df.head(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Okay, perfect! The data looks just as expected."},{"metadata":{},"cell_type":"markdown","source":"## Data Wrangling\n** = Getting the data into the right format**\n\nLooking at the [fast.ai documentation](https://docs.fast.ai/vision.data.html#ImageDataBunch) we can quickly see, that fast.ai only accepts image files for Computer Vision. In this competition we were not offered images, but .csv files containing the pixel values for each pixel of each image. If we want to use fast.ai we have to create images from the data we have.\n\nFast.ai accepts image data in different formats. We will use the from_folder function of the ImageDataBunch class to load in the data. To do this we need all images in the following structure:\n\n`path\\\n train\\\n 0\\\n ___.jpg\n ___.jpg\n ___.jpg\n 1\\\n ___.jpg\n ___.jpg\n 2\\\n ...\n 3\\\n ...\n ...\n test\\\n ___.jpg\n ___.jpg\n ...\n`\n\nLet's first create the folder structure!\n\n(nice to know: the input folder of Kaggle Competitions is always read-only, so if we want to add data or create folders, we have to do so outside of the input folder)"},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN = Path(\"../train\")\nTEST = Path(\"../test\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create training directory\nfor index in range(10):\n try:\n os.makedirs(TRAIN/str(index))\n except:\n pass","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Test whether creating the training directory was successful\nsorted(os.listdir(TRAIN))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Create test directory\ntry:\n os.makedirs(TEST)\nexcept:\n pass","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Okay, all folders are created! The next step is to create the images inside of the folders from 'train.csv' and 'test.csv'. We will use the Image module from PIL to do this.\n\nwe have to reshape each numpy array to have the desired dimensions of the image (28x28)\n\n`000 001 002 003 ... 026 027\n028 029 030 031 ... 054 055\n056 057 058 059 ... 082 083\n | | | | ... | |\n728 729 730 731 ... 754 755\n756 757 758 759 ... 782 783`\n\nthen we use the fromarray function to create a .jpg image from the numpy array and save it into the desired folder"},{"metadata":{"trusted":true},"cell_type":"code","source":"# import numpy to reshape array from flat (1x784) to square (28x28)\nimport numpy as np\n\n# import PIL to display images and to create images from arrays\nfrom PIL import Image\n\ndef saveDigit(digit, filepath):\n digit = digit.reshape(28,28)\n digit = digit.astype(np.uint8)\n\n img = Image.fromarray(digit)\n img.save(filepath)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# save training images\nfor index, row in train_df.iterrows():\n \n label,digit = row[0], row[1:]\n \n folder = TRAIN/str(label)\n filename = f\"{index}.jpg\"\n filepath = folder/filename\n \n digit = digit.values\n \n saveDigit(digit, filepath)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# save testing images\nfor index, digit in test_df.iterrows():\n\n folder = TEST\n filename = f\"{index}.jpg\"\n filepath = folder/filename\n \n digit = digit.values\n \n saveDigit(digit, filepath)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Display images\n\nTo check whether everything worked as expected, let's take a look at a few images from each folder."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# import matplotlib to arrange the images properly\nimport matplotlib.pyplot as plt\n\ndef displayTrainingData():\n fig = plt.figure(figsize=(5,10))\n \n for rowIndex in range(1, 10):\n subdirectory = str(rowIndex)\n path = TRAIN/subdirectory\n images = os.listdir(path)\n for sampleIndex in range(1, 6):\n randomNumber = random.randint(0, len(images)-1)\n image = Image.open(path/images[randomNumber])\n ax = fig.add_subplot(10, 5, 5*rowIndex + sampleIndex)\n ax.axis(\"off\")\n \n plt.imshow(image, cmap='gray')\n \n plt.show()\n \ndef displayTestingData():\n fig = plt.figure(figsize=(5, 10))\n \n paths = os.listdir(TEST)\n \n \n for i in range(1, 51):\n randomNumber = random.randint(0, len(paths)-1)\n image = Image.open(TEST/paths[randomNumber])\n \n ax = fig.add_subplot(10, 5, i)\n ax.axis(\"off\")\n \n plt.imshow(image, cmap='gray')\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('samples of training data')\ndisplayTrainingData()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('samples of testing data')\ndisplayTestingData()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's also look at one image in more detail:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"image_path = TEST/os.listdir(TEST)[9]\nimage = Image.open(image_path)\nimage_array = np.asarray(image)\n\n\nfig, ax = plt.subplots(figsize=(15, 15))\n\nimg = ax.imshow(image_array, cmap='gray')\n\nfor x in range(28):\n for y in range(28):\n value = round(image_array[y][x]/255.0, 2)\n color = 'black' if value > 0.5 else 'white'\n ax.annotate(s=value, xy=(x, y), ha='center', va='center', color=color)\n\nplt.axis('off')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load data into DataBunch\nNow that we have the right folder structure and images inside of the folders we can continue. Before training a model in fast.ai, we have to load the data into a [DataBunch](https://docs.fast.ai/basic_data.html#DataBunch), in this case, we use a ImageDataBunch, a special version of the DataBunch. Fast.ai offers different functions to create a DataBunch. We will use the from_folder method of the ImageDataBunch class to create the dataset.

\nThere are different hyperparameters we can tweak to make the model perform better:\n\n- [valid_pct](#What-are-Train,-Test-and-Validation-datasets?)\n- [size](#What-image-size-should-I-choose?)\n- [num_workers](#What-is-multiprocessing?)\n- [ds_tfms](#What-are-transforms-and-which-transforms-should-I-use?)\n- [bs (batch size)](#What-is-the-batch-size?)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# transforms\ntfms = get_transforms(do_flip=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = ImageDataBunch.from_folder(\n path = TRAIN,\n test = TEST,\n valid_pct = 0.2,\n bs = 16,\n size = 28,\n #num_workers = 0,\n ds_tfms = tfms\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's perform normalization to make the CNN converge faster. fast.ai already defined the variable mnist_stats, that we can use to normalize our data. Alternatively, we can call normalize() without any paramters. In this case fast.ai simply calculates the exact stats needed for the dataset at hand."},{"metadata":{"trusted":true},"cell_type":"code","source":"mnist_stats","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"data.normalize(mnist_stats)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# all the classes in data\nprint(data.classes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training"},{"metadata":{},"cell_type":"markdown","source":"**First we have to create the CNN**\n
The next step is to create a convolutional neural network, short CNN. The most important thing about you need to know about CNNs is that we feed the model images and it outputs a probability for each possible category, so in this competition the digits from 0 through 9."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"There are many different types of CNNs. For now we will only use one type of CNN, ResNets. The come in different sizes. There is **resnet18**, **resnet34** and a few more. At the moment, you just have to select the size.\n\nIn fast.ai creating a CNN is really easy. We can use the cnn_learner object to do so and just have to pass the data that we want to feed into the CNN later and which architecture *(base_arch)* we want to use:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = cnn_learner(data, base_arch=models.resnet18, metrics=accuracy, model_dir=\"/tmp/models\", callback_fns=ShowGraph)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"But as I said, there are many more architectures, you can use. Check out [How to select a CNN architecture?](#How-to-set-a-CNN-architecture?) to learn more."},{"metadata":{},"cell_type":"markdown","source":"The model that we just created is already [pretrained](#What-is-transfer-learning?). In fast.ai this is the default setting."},{"metadata":{},"cell_type":"markdown","source":"We can tweak one parameter when creating the CNN to make it perform better:\n- [ps (dropout propability)](#What-is-the-dropout-propability-and-how-high-should-I-set-it?)"},{"metadata":{},"cell_type":"markdown","source":"**Now it's time to train the neural network.**
\nWe do so in fast.ai using the fit_one_cycle() function.
\nTraining a CNN means, we feed it with data to predict which digit is shown. Then we compare the predictions with the actual results and update the CNN so that it better classifies the images later.\n\nThere are different hyperparameters we can tweak to make the model perform better:\n- [cyc_len (number of epochs)](#What-are-epochs-and-how-many-epochs-should-I-train-my-CNN?)\n- max_lr (learning rate)\n- moms (momentum)"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(cyc_len=5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluation\nCreate a ClassificationInterpretation object to evaluate your results."},{"metadata":{"trusted":true},"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plot the 9 images with the highest loss. These are the images the CNN was most sure about, but still got wrong."},{"metadata":{"trusted":true},"cell_type":"code","source":"interp.plot_top_losses(9, figsize=(7, 7))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A good way to summarize the performance of a classification algorithm is to create a confusion matrix. Confusion Matricies are used to understand which classes are most easily confused. As labeled on the axis, the x-axis shows the predicted classes and the y-axis the actual classes. So if (4/7)=10 it means that it happened 10 times that the CNN predicted a 7 but in reality if was a 4."},{"metadata":{"trusted":true},"cell_type":"code","source":"interp.plot_confusion_matrix()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prediction\nGet the predictions on the test set.
\nlearn.get_preds() returns a propability distribution over all possible classes for every given image."},{"metadata":{"trusted":true},"cell_type":"code","source":"class_score, y = learn.get_preds(DatasetType.Test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That means that for every image in the test set it predicts how likely each class is. In this case the highest value is obviously 1"},{"metadata":{"trusted":true},"cell_type":"code","source":"probabilities = class_score[0].tolist()\n[f\"{index}: {probabilities[index]}\" for index in range(len(probabilities))]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"But we want the CNN to predict only one class. The class with the highest probability. Argmax returns the index of the highest value."},{"metadata":{"trusted":true},"cell_type":"code","source":"class_score = np.argmax(class_score, axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is exactly what we want."},{"metadata":{"trusted":true},"cell_type":"code","source":"class_score[0].item()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The last step is creating the submission file.
\n\"sample_submission.csv\" is showing us the desired format"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = pd.read_csv(INPUT/\"sample_submission.csv\")\ndisplay(sample_submission.head(2))\ndisplay(sample_submission.tail(2))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Columns the submission file has to have:\n\n- ImageId: index in the test set, starting from 1, going up to 28000\n- Label: a prediction which digit the image shows"},{"metadata":{"trusted":true},"cell_type":"code","source":"# remove file extension from filename\nImageId = [os.path.splitext(path)[0] for path in os.listdir(TEST)]\n# typecast to int so that file can be sorted by ImageId\nImageId = [int(path) for path in ImageId]\n# +1 because index starts at 1 in the submission file\nImageId = [ID+1 for ID in ImageId]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame({\n \"ImageId\": ImageId,\n \"Label\": class_score\n})\n# submission.sort_values(by=[\"ImageId\"], inplace = True)\nsubmission.to_csv(\"submission.csv\", index=False)\ndisplay(submission.head(3))\ndisplay(submission.tail(3))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Questions and Answers\n\n- [What is MNIST?](#What-is-MNIST?)\n- [How to submit?](#How-to-submit?)\n- [What are Train, Test and Validation datasets?](#What-are-Train,-Test-and-Validation-datasets?)\n- [What is the batch size?](#What-is-the-batch-size?)\n- [What image size should I choose?](#What-image-size-should-I-choose?)\n- [What is multiprocessing?](#What-is-multiprocessing?)\n- [What are transforms and which transforms should I use?](#What-are-transforms-and-which-transforms-should-I-use?)\n- [What is transfer learning?](#What-is-transfer-learning?)\n- [How to improve your score?](#How-to-improve-your-score?)\n- [How to set a CNN architecture?](#How-to-set-a-CNN-architecture?)\n- [What is the best CNN architecture?](#What-is-the-best-CNN-architecture?)"},{"metadata":{},"cell_type":"markdown","source":"## What is MNIST?\nMNIST is the perfect dataset to get started learning more about pattern recognition and machine learning. That is why people call it the \"Hello World of machine learning\".\nIt's a large database of handwritten digits. There are a total of 70.000 grayscale images, each is 28x28 pixels. They show 10 different classes representing the numbers from 0 to 9. The dataset is split into 60.000 images in the training set and 10.000 in the test set.\nThis competition is based on the MNIST dataset. However, the train-test distribution is different. Here, there are 42.000 images in the training set and 28.000 images in the test set.\nMNIST was published by the godfather of CNNs, Yann LeCun. You can find the original dataset [here](http://yann.lecun.com/exdb/mnist/)"},{"metadata":{},"cell_type":"markdown","source":"## How to train a CNN?\nYou need an image and the label (=Which number is this?). For example an image of a nine and a nine.
\nYou feed in the image. The CNN predicts a number. You compare the predicted number to the actual number (the label). Then you update the CNN so that it better predicts this image the next time."},{"metadata":{},"cell_type":"markdown","source":"## What is the difference between the train and the test dataset?\n\nWhen you do machine learning often you have two different groups of data. You have a training dataset and a testing dataset.\n\nLet's explain this concept with the MNIST dataset! In the training dataset you have images for which you already know the label.\n\n\n\nWith the training dataset you teach the model how a nine (and every other digit) looks like. Once you're done teaching your model how each digit looks like, you take the other dataset, the test dataset. You use the test dataset to check how good your model really is, by letting it predict data it has never seen before. To do so, you only feed the images to the CNN, not the labels. Then the CNN predicts the labels. Now you calculate the accuracy: Simply compare the predicted labels with the actual labels.\n\n#### Remember:\n\nWhenever we train a CNN we need to split the data into 2 parts:\n- training set: used to teach the model how each class looks like\n- test set: test accuracy on never before seen data"},{"metadata":{},"cell_type":"markdown","source":"## What is the validation set?\n\n\n\nIn this kernel we only have a training and a test set. That's why we split the test set to get a validation set. We do this with the 'valid_pct' parameter. This is one of the parameters you could tune to increase the accuracy. \n\nTo learn more about this read [this stackoverflow post](https://stackoverflow.com/a/13623707)"},{"metadata":{},"cell_type":"markdown","source":"## What to keep in mind when you split data into train, validation and test set?\n- all three datasets should have the same distribution over classes. In MNIST this would mean that when 5% of images in the training set are showing the digit two, also 5% of image in the test and validation set should show a two. But not that all classes should exist the same number of times."},{"metadata":{},"cell_type":"markdown","source":"## What is the batch size?\nThe batch size refers to the number of images in one batch. Everytime an entire batch of images is passed through the neural network the weights of the neural network are updated\n\nWhy would you increase the batch size:\n- to improve accuracy of the model\n\nWhy would you decrease the batch size:\n- to train the network faster\n- to reduce the memory used\n\nA good value for the batch size is 16"},{"metadata":{},"cell_type":"markdown","source":"### What are epochs and how many epochs should I train my CNN?\nHow often we feed the entire data to our CNN. Train the CNN as long as the accuracy is improving"},{"metadata":{},"cell_type":"markdown","source":"### What is the dropout propability and how high should I set it?\nIn dropout you randomly deactivate nodes. That way the CNN has to learn redundant representations. This works well to reduce overfitting."},{"metadata":{},"cell_type":"markdown","source":"## What image size should I choose?\n(Most) CNNs need images of the same size. By setting the size parameter we tell fast.ai to make all images that size.\n\nBigger images, result in more calculations and thus slower speed, but the accuracy improves. Smaller images on the other hand reduce accuracy but improve speed. Don't make the trainig images bigger than the original images, as this would only be a waste of time.\n\nOur data is already of shape 28x28.\n\nWe could make the images smaller than 28x28. This would decrease the training time, but also decrease the accuracy. Because our CNN trains in a reasonable amount of time there is no reason to decrease the image size\n\nNever make the training image bigger than the original image."},{"metadata":{},"cell_type":"markdown","source":"## What is multiprocessing?\nIn Computer Science there is this thing called multiprocessing. This means that we have two or more things happending at the same time. A computer typically has multiple CPUs and we make use of exactly that. Every CPU can run one process at a time. Ususally when we do multiprocessing every CPU gets its own task. Exactly that is the default for ImageDataBunch: number workers = number of CPUs. This works fine for Linux, but makes a lot of problems in Windows. If you're on Windows, set num_workers to 0, if you're on Linux, don't set anything, then it defaults to the number of CPUs\n\nIf you use a cloud solution and are not sure which operating system is used, execute the following code"},{"metadata":{"trusted":true},"cell_type":"code","source":"import platform; platform.system()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## What are transforms and which transforms should I use?\nTo make models generalize better we can use so called transforms. They randomly change the image slightly. For example a bit of zoom or rotation. Fortunately, we don't really have to deal with transforms a lot in fast.ai. The package offers a convenient function called get_transforms() that returns pretty good values for transformations. In the case of digit recognition we want to tranform the data as much as possible so that it generalizes better, but only so much that the image would still be recognized by a human being.\n\nOne parameter we definitely have to change for that reason is do_flip. If do_flip is set to True, random horizontal flips (with a probability of 0.5) would be applied to the images. This would result in images like this:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"flip_tfm = RandTransform(tfm=TfmPixel (flip_lr), kwargs={}, p=1, resolved={}, do_run=True, is_random=True, use_on_y=True)\nfolder = TRAIN/\"3\"\nfilename = os.listdir(folder)[0]\nimg = open_image(TRAIN/folder/filename)\ndisplay(img)\ndisplay(img.apply_tfms(flip_tfm))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We don't want that! This would confuse our CNN. Therefore we set do_flip to False"},{"metadata":{"trusted":true},"cell_type":"code","source":"tfms = get_transforms(do_flip=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## What is normalization?\n\nNormalization is a technique use to make neural network converge faster. It sppeds up learning and makes the network converge faster."},{"metadata":{},"cell_type":"markdown","source":"## What is transfer learning?\nTransfer learning is a popular method in Deep Learning. It allows you to build accurate models fast.\nWhen you train a neural network, it (hopefully) gets better and better at finding patterns in data.\nResearchers found out that patterns a neural network learned in one problem can be helpful when solving another problem.\nIn computer vision, the most common form of transfer learning is using **pretrained CNNs**. You simply take a CNN from one problem, with all its weights and biases, and use it as a baseline for another problem.\nThen you only have to [finetune the model](https://www.kaggle.com/christianwallenwein/beginners-guide-to-mnist-with-fast-ai#What-is-fine-tuning?). This works surprisingly good.\nMost **pretrained CNN's** for classification problems have been trained on ImageNet.\n\nCheck out the [How to set a CNN architecture](http://)-section to learn how to use **pretrained models** for our problem."},{"metadata":{},"cell_type":"markdown","source":"## What is fine-tuning?\nFinetuning is a technique used in [transfer learning](#What-is-transfer learning?). It is used to optimize a pretrained model for the new task given.\n\nTips for fine-tuning:\n\n### Replace the last layer\nMost pretrained CNNs were trained on ImageNet. Imagenet has 1000 categories and therefore outputs probabilities for 1000 classes. Our task is to predict only 10 digits in MNIST. Therefore we have to replace the last layer, so that it only outputs 10 probabilities, one for each digit.\n\n### Use a smaller learning rate\nWhen we train a CNN, at the beginning the weights are completely random. Compare these to pretrained weights and you will find that the pretrained \nweights are much better. Because the weights are already so good, we don't have to change the model that much.\nIt is common to set a learning rate 10 times smaller than used for training usual, non-pretrained, models.\n\n### Freeze weights of the first few layers\nThe first few convolutional layers of a CNN capture universal features. These features could be edges or curves. They are so basic, that every model used for object classification needs to understand them. We don't want to change those low-level features, but rather focus on high-level features that are specific to the problem at hand. Therefore you want to freeze the weights for the first few layers and only train on the last layers."},{"metadata":{},"cell_type":"markdown","source":"## How to improve your score?\n**99.5%**: Good CNN + pooling layers + data augmentation + dropout+ batch normalization + decaying learning rate + advanced optimization\n\n**99.7%**: When you train the same CNN multiple times, you will not get the same results. So if you build a great CNN, train it 10 times and submit the results 10 times, you will probably beat 99.7%. Another approach is to build an ensemble of (great) CNNs. This means that you train multiple good, but different CNNs and combine their predictions.\n\n**99.75%**: This is the **best score reached inside of Kaggle Kernels**. Check out [this kernel](https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist).\n\n**99.79%**: This is the **best score ever achieved** . It was obtained by Yann LeCun and colleagues from NYU in [this paper](http://yann.lecun.com/exdb/publis/pdf/wan-icml-13.pdf).\n\n**99.9% - 100.0%**: To achieve a perfect or nearly **perfect score**, you have to train your CNN on the entire MNIST dataset (70.000 images). That way the CNN already \"knows\" the correct label for each image and simply has to remember it.\n\nThe information here is an excerpt from [this post](https://www.kaggle.com/c/digit-recognizer/discussion/61480)."},{"metadata":{},"cell_type":"markdown","source":"## How to set a CNN architecture?\nThere are different opportunities. You could\n- [use a predefined architecture from fast.ai](#How-to-use-a-predefined-architecture-from-fast.ai?)\n- [use a predefined architecture from PyTorch](#How-to-use-a-predefined-architecture-from-PyTorch?)\n- [use a custom PyTorch architecture](#How-to-use-a-custom-PyTorch-architecture?)"},{"metadata":{},"cell_type":"markdown","source":"### How to use a predefined architecture from fast.ai?\nAt the time of writing, the following architectures are available:\n\n- resnet18, resnet34, resnet50, resnet101, resnet152\n- squeezenet1_0, squeezenet1_1\n- densenet121, densenet169, densenet201, densenet161\n- vgg16_bn, vgg19_bn\n- alexnet\n\nTo change the architecture just change the base_arch parameter in cnn_learner to the desired architecture."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = cnn_learner(data, base_arch=models.densenet169, metrics=accuracy, model_dir=\"/tmp/models\", callback_fns=ShowGraph)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In fast.ai, the default is to use pretrained models. If you don't want to use a model pretrained on ImageNet, pass **pretrained=False** to the architecture like this:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = cnn_learner(data, base_arch=models.densenet169, pretrained=False, metrics=accuracy, model_dir=\"/tmp/models\", callback_fns=ShowGraph)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check out the [fast.ai Computer Vision models zoo](https://docs.fast.ai/vision.models.html#Computer-Vision-models-zoo). Maybe there are new architectures available."},{"metadata":{},"cell_type":"markdown","source":"### How to use a predefined architecture from PyTorch?\n\nTo do so, you first have to import the necessary submodule from PyTorch like this:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torchvision.models","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"At the time of writing, the following architectures are available:\n\n- resnet18\n- alexnet\n- vgg16\n- squeezenet1_0\n- densenet161\n- **inception_v3** (requires a tensor of size N x 3 x 299 x 299)\n- **googlenet**\n- **shufflenet_v2_x1_0**\n- **mobilenet_v2**\n- **resnext50_32x4d**\n- **wide_resnet50_2**\n- **mnasnet1_0**\n\nWhen we want to use an architecture from PyTorch, we can't simply use a cnn_learner object. Use the following code and replace google_net() with the desired architecture:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = Learner(data, torchvision.models.googlenet(), metrics=accuracy, model_dir=\"/tmp/models\", callback_fns=ShowGraph)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To use a model pretrained on ImageNet, pass **pretrained=True** to the architecture like this:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = Learner(data, torchvision.models.googlenet(pretrained=True), metrics=accuracy, model_dir=\"/tmp/models\", callback_fns=ShowGraph)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check out the [torchvision.models documentation](https://pytorch.org/docs/stable/torchvision/models.html). Maybe there are new architectures available."},{"metadata":{},"cell_type":"markdown","source":"### How to use a custom PyTorch architecture?\nTo use a custom PyTorch architecture, you first have to define it. I will not go into detail about how to do it. But there are wonderful tutorials that can show you how to do it. The skeleton of the class looks like this:"},{"metadata":{"trusted":true},"cell_type":"code","source":"class CNN(nn.Module):\n def __init__(self):\n super(CNN, self).__init__()\n # here you instantiate all the layers of the neural network and the activation function\n \n def forward(self, x):\n # here you define the forward propagation\n return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When you are finished, the CNN could look something like this:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# set the batch size\nbatch_size = 16\n\nclass CNN(nn.Module):\n def __init__(self):\n super(CNN, self).__init__()\n # input is 28 pixels x 28 pixels x 3 channels\n # our original data was grayscale, so only one channel, but fast.ai automatically loads in the data as RGB\n self.conv1 = nn.Conv2d(3,16, 3, padding=1)\n self.conv2 = nn.Conv2d(16,32, 3, padding=1)\n self.pool = nn.MaxPool2d(2, 2)\n self.fc1 = nn.Linear(7*7*32, 500)\n self.fc2 = nn.Linear(500, 10)\n self.relu = nn.ReLU()\n\n def forward(self, x):\n print (x.size())\n # x (28x28x3)\n x = self.conv1(x)\n # x (28x28x16)\n x = self.pool(x)\n # x (14x14x16)\n x = self.relu(x)\n \n x = self.conv2(x)\n # x (14x14x32)\n x = self.pool(x)\n # x (7x7x32)\n x = self.relu(x)\n\n # flatten images in batch\n print(x.size())\n x = x.view(-1,7*7*32)\n print(x.size())\n x = self.fc1(x)\n x = self.relu(x)\n \n x = self.fc2(x)\n x = self.relu(x)\n \n return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And to try the created CNN use the following code:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = Learner(data, CNN(), metrics=accuracy, model_dir=\"/tmp/models\", callback_fns=ShowGraph)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## What is the best CNN architecture?\nAccording to [this post](https://www.kaggle.com/cdeotte/how-to-choose-cnn-architecture-mnist) by [Chris Deotte](https://www.kaggle.com/cdeotte) the best architecture is:\n\n784 input nodes\n\n> [32C3-32C3-32C5S2]\n\n- 2x (convolutional layer with 32 feature maps, 3x3 filter and stride 1)\n- convolutional layer with 32 feature maps, 5x5 filter and stride 2\n> [64C3-64C3-64C5S2]\n\n- 2x (convolutional layer with 64 feature maps, 3x3 filter and stride 1)\n- convolutional layer with 64 feature maps, 5x5 filter and stride 2\n\n128 fully connected dense layers\n\n10 output nodes\n\nwith 40% dropout, batch normalization, and data augmentation added"},{"metadata":{},"cell_type":"markdown","source":"## How to submit?\nNow we're throught the entire process of how to create a CNN with fast.ai to recognize digits. To submit a file to the the competition, you have two different options.\n\nCode in Kaggle Kernel\n1. go to your kernel\n2. commit the kernel\n3. go back to all of your kernels\n4. select the kernel again\n5. scroll down to Output\n6. click on 'Submit to Competition'\n\n\nCode locally on PC\n1. go to [the competition](https://www.kaggle.com/c/digit-recognizer)\n2. click on ['Submit Predictions'](https://www.kaggle.com/c/digit-recognizer/submit)\n3. upload your submission file\n4. add a description\n5. click on 'Make Submission'\n\n"},{"metadata":{},"cell_type":"markdown","source":"Fun Fact\n![](https://images.squarespace-cdn.com/content/v1/5c293b5d55b02c783a5d8747/1551704286537-JG06FA61IJM7JJ94TAKX/ke17ZwdGBToddI8pDm48kPx25wW2-RVvoRgxIT6HShBZw-zPPgdn4jUwVcJE1ZvWQUxwkmyExglNqGp0IvTJZUJFbgE-7XRK3dMEBRBhUpxp8SK2yZDC8sLtVprOyP9xilpMqmr5qElvqElFYURlwnSzTaHLYeySc8Xfr8nFWbQ/Butterflies-taste-with-their-fet.gif?format=1000w)\n[source](https://www.learnsomethingeveryday.co.uk/#/4-march-2019/)"},{"metadata":{},"cell_type":"markdown","source":"TODO:\n\n- image augmentation https://www.kaggle.com/anisayari/generate-more-training-data-fastai\n- created images to zip file https://www.kaggle.com/anisayari/generate-more-training-data-fastai\n- better explanations\n- explain CNNs better\n- explain normalization\n- change channels from RGB to Grayscale in images to improve computation time\n- explain feed forward and backpropagation\n- explain how to make the model train faster\n- group Q&A questions: general questions, training, etc.\n- check spelling\n- visualize concepts\n- make tutorial simpler\n- explain dropout better\n- explain train and test set\n- expain train and validation set\n- supervised vs unsupervised learning"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}