{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-08-25T12:02:25.505183Z","iopub.execute_input":"2023-08-25T12:02:25.505654Z","iopub.status.idle":"2023-08-25T12:02:25.519009Z","shell.execute_reply.started":"2023-08-25T12:02:25.505619Z","shell.execute_reply":"2023-08-25T12:02:25.517766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Hello everyone! This notebook is my submission to the Digit Recognizer competition and also a quick introduction to FastAI and PyTorch. PyTorch is one of the best Python libraries used for developing deep learning systems. FastAI just adds some high-level functionalities on top of PyTorch. In this notebook, I am going to provide some context into how a neural network can be easily implemented using these tools. Moreover, every line of code from this notebook is going to be explained in order to provide more insights into how you can apply these notions yourself. That being said, let's get started and good luck with your code!\n\nIf you want to find out more about the capabilities of the FastAI library, I strongly recommend checking out https://course.fast.ai/?ref=blog.paperspace.com, a course designed by the co-founder of fast.ai, Jeremy Howard.","metadata":{}},{"cell_type":"markdown","source":"> Here are the official documentations for these two libraries:\n> * PyTorch: https://pytorch.org/docs/stable/index.html\n> * FastAI: https://docs.fast.ai/\n","metadata":{}},{"cell_type":"markdown","source":"First, we import the libraries:","metadata":{}},{"cell_type":"code","source":"import fastai\nfrom fastai.vision.all import *\nimport torch","metadata":{"execution":{"iopub.status.busy":"2023-08-25T12:02:30.243579Z","iopub.execute_input":"2023-08-25T12:02:30.244022Z","iopub.status.idle":"2023-08-25T12:02:36.897504Z","shell.execute_reply.started":"2023-08-25T12:02:30.243987Z","shell.execute_reply":"2023-08-25T12:02:36.896014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Processing","metadata":{}},{"cell_type":"code","source":"# Get the training data\ntrain_data = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\n\n# Get the test data\ntest_data = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-08-25T12:02:36.899910Z","iopub.execute_input":"2023-08-25T12:02:36.900359Z","iopub.status.idle":"2023-08-25T12:02:44.426886Z","shell.execute_reply.started":"2023-08-25T12:02:36.900324Z","shell.execute_reply":"2023-08-25T12:02:44.425500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create two paths for accessing the training and test data:","metadata":{}},{"cell_type":"code","source":"TRAIN = Path(\"../train\")\nTEST = Path(\"../test\")","metadata":{"execution":{"iopub.status.busy":"2023-08-25T12:02:44.428849Z","iopub.execute_input":"2023-08-25T12:02:44.429838Z","iopub.status.idle":"2023-08-25T12:02:44.437827Z","shell.execute_reply.started":"2023-08-25T12:02:44.429764Z","shell.execute_reply":"2023-08-25T12:02:44.435615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We create 10 directories that represent every class that an image can be assigned to (the digits from 0 to 9):","metadata":{}},{"cell_type":"code","source":"# 'makedirs' creates a directory recursively\nfor digit in range(10):\n    os.makedirs(TRAIN/str(digit))","metadata":{"execution":{"iopub.status.busy":"2023-08-25T12:02:44.442235Z","iopub.execute_input":"2023-08-25T12:02:44.442816Z","iopub.status.idle":"2023-08-25T12:02:44.464456Z","shell.execute_reply.started":"2023-08-25T12:02:44.442763Z","shell.execute_reply":"2023-08-25T12:02:44.458652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Here we can see the name of each director\nsorted(os.listdir(TRAIN))","metadata":{"execution":{"iopub.status.busy":"2023-08-25T12:02:44.466226Z","iopub.execute_input":"2023-08-25T12:02:44.466670Z","iopub.status.idle":"2023-08-25T12:02:44.486709Z","shell.execute_reply.started":"2023-08-25T12:02:44.466635Z","shell.execute_reply":"2023-08-25T12:02:44.484877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 'makedirs' also creates all intermediate-level directories needed to contain the leaf directory\nos.makedirs(TEST)","metadata":{"execution":{"iopub.status.busy":"2023-08-25T12:02:44.489716Z","iopub.execute_input":"2023-08-25T12:02:44.492181Z","iopub.status.idle":"2023-08-25T12:02:44.501773Z","shell.execute_reply.started":"2023-08-25T12:02:44.492121Z","shell.execute_reply":"2023-08-25T12:02:44.500626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we create a function that helps us save digits as images not just as arrays:","metadata":{}},{"cell_type":"code","source":"# PIL = Python Imaging Library\nfrom PIL import Image \n\ndef save_digit(path, digit):\n    digit = digit.reshape(28, 28) # 'reshape' changes the shape of an array \n    digit = digit.astype(np.uint8) # 'astype' changes the data type of the numbers inside an array \n    img = Image.fromarray(digit) # 'fromarray' creates an image object from a given array\n    img.save(path) # save the image into the specified directory","metadata":{"execution":{"iopub.status.busy":"2023-08-25T12:02:44.502999Z","iopub.execute_input":"2023-08-25T12:02:44.503493Z","iopub.status.idle":"2023-08-25T12:02:44.523873Z","shell.execute_reply.started":"2023-08-25T12:02:44.503447Z","shell.execute_reply":"2023-08-25T12:02:44.521513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we want to put the training images (in jpg format) in the director named after their label:","metadata":{}},{"cell_type":"code","source":"for index, row in train_data.iterrows():\n    label, digit = row[0], row[1:]\n    \n    director = TRAIN / str(label)\n    fname = f\"{index}.jpg\"\n    path = director / fname\n    \n    digit = digit.values\n    \n    save_digit(path, digit)","metadata":{"execution":{"iopub.status.busy":"2023-08-25T12:02:44.526412Z","iopub.execute_input":"2023-08-25T12:02:44.527044Z","iopub.status.idle":"2023-08-25T12:03:02.934231Z","shell.execute_reply.started":"2023-08-25T12:02:44.526972Z","shell.execute_reply":"2023-08-25T12:03:02.933280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The test images are put in the TEST directory in jpg format:","metadata":{}},{"cell_type":"code","source":"for index, digit in test_data.iterrows():\n    director = TEST\n    fname = f\"{index}.jpg\"\n    path = director / fname\n    \n    digit = digit.values\n    \n    save_digit(path, digit)","metadata":{"execution":{"iopub.status.busy":"2023-08-25T12:03:02.935584Z","iopub.execute_input":"2023-08-25T12:03:02.936716Z","iopub.status.idle":"2023-08-25T12:03:12.572600Z","shell.execute_reply.started":"2023-08-25T12:03:02.936674Z","shell.execute_reply":"2023-08-25T12:03:12.571391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we want to construct a DataBlock. In order to train a neural network in FastAI, the data needs to be stored in DataBlock. We use just a few parameters for building ours. If you want to explore all the capabilities of a DataBlock please check out the official documentation.","metadata":{}},{"cell_type":"markdown","source":"The 'blocks' parameter asks for one or more types of blocks in order to know how to organize the data. I used ImageBlock for the pictures and CategoryBlock for their labels. 'get_items' represents the function used to get the items into the image block. 'splitter' is used to split the data into training and validation data; if you use a seed the random split will be the same all the time you'll run the cell. 'get_y' refers to a function used to get the labels for each picture (category block). We defined the 'get_label' function to do this exact thing for us.","metadata":{}},{"cell_type":"code","source":"def get_label(fname):\n    return fname.parent.name    ","metadata":{"execution":{"iopub.status.busy":"2023-08-25T12:03:12.577153Z","iopub.execute_input":"2023-08-25T12:03:12.578129Z","iopub.status.idle":"2023-08-25T12:03:12.583551Z","shell.execute_reply.started":"2023-08-25T12:03:12.578090Z","shell.execute_reply":"2023-08-25T12:03:12.582283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"digits = DataBlock(blocks = (ImageBlock, CategoryBlock),\n                  get_items = get_image_files,\n                  splitter = RandomSplitter(seed = 42),\n                  get_y = get_label)\n\n# We use dataloaders function to load the training data\ndls = digits.dataloaders(TRAIN)","metadata":{"execution":{"iopub.status.busy":"2023-08-25T12:03:12.585630Z","iopub.execute_input":"2023-08-25T12:03:12.586594Z","iopub.status.idle":"2023-08-25T12:03:16.600474Z","shell.execute_reply.started":"2023-08-25T12:03:12.586533Z","shell.execute_reply":"2023-08-25T12:03:16.599437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Training ","metadata":{}},{"cell_type":"markdown","source":"Here we create our vision learner. \nFastAI includes some pretrained models for computer vision such as resnet34, densenet169, vgg16_bn, etc. Our first try is to fine tune one of these models on our training data. ","metadata":{}},{"cell_type":"code","source":"# The vision learner takes as parameters a dataloader, a model and a metric in order to measure the performance of the model.\nlearn = vision_learner(dls, resnet34, metrics = error_rate)\nlearn.fine_tune(2) # Here we specify the number of epochs for which we do the fine-tuning ","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:17:34.579581Z","iopub.execute_input":"2023-08-25T14:17:34.580005Z","iopub.status.idle":"2023-08-25T14:40:13.983788Z","shell.execute_reply.started":"2023-08-25T14:17:34.579974Z","shell.execute_reply":"2023-08-25T14:40:13.981910Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We can use 'one_batch' method to see the images from a batch and their labels \nx, y = dls.one_batch()","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:40:13.989137Z","iopub.execute_input":"2023-08-25T14:40:13.989614Z","iopub.status.idle":"2023-08-25T14:40:14.132134Z","shell.execute_reply.started":"2023-08-25T14:40:13.989567Z","shell.execute_reply":"2023-08-25T14:40:14.131185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here are the labels:","metadata":{}},{"cell_type":"code","source":"y","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:40:14.133464Z","iopub.execute_input":"2023-08-25T14:40:14.134224Z","iopub.status.idle":"2023-08-25T14:40:14.142219Z","shell.execute_reply.started":"2023-08-25T14:40:14.134189Z","shell.execute_reply":"2023-08-25T14:40:14.141353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We use 'get_preds' to get predictions regarding the label of every picture. The method returns a list of probabilities for each one of them. These probabilities represent how sure the model is about what digit is in the image.","metadata":{}},{"cell_type":"code","source":"preds, _ = learn.get_preds(dl=[(x,y)])\npreds[0]","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:40:14.144663Z","iopub.execute_input":"2023-08-25T14:40:14.145461Z","iopub.status.idle":"2023-08-25T14:40:14.356420Z","shell.execute_reply.started":"2023-08-25T14:40:14.145427Z","shell.execute_reply":"2023-08-25T14:40:14.354945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In our case, we get 10 probabilities, because there are 10 classes (an image can represent a digit from 0 to 9). All those probabilities sum up to 1.","metadata":{}},{"cell_type":"code","source":"len(preds[0]), preds[0].sum()","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:40:14.358935Z","iopub.execute_input":"2023-08-25T14:40:14.359455Z","iopub.status.idle":"2023-08-25T14:40:14.368298Z","shell.execute_reply.started":"2023-08-25T14:40:14.359405Z","shell.execute_reply":"2023-08-25T14:40:14.367193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can plot a confusion matrix by using ClassificationInterpretation class provided by FastAI:","metadata":{}},{"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)\ninterp.plot_confusion_matrix(figsize = (8, 8), dpi = 60)","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:40:14.370658Z","iopub.execute_input":"2023-08-25T14:40:14.371090Z","iopub.status.idle":"2023-08-25T14:41:11.529820Z","shell.execute_reply.started":"2023-08-25T14:40:14.371054Z","shell.execute_reply":"2023-08-25T14:41:11.528311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can observe how many images were classified correctly by our model on the main diagonal of the matrix and how many were misclassified anywhere else.","metadata":{}},{"cell_type":"code","source":"# We can also see what digits were the most confused \ninterp.most_confused(min_val = 5)","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:41:11.531661Z","iopub.execute_input":"2023-08-25T14:41:11.532100Z","iopub.status.idle":"2023-08-25T14:41:39.727294Z","shell.execute_reply.started":"2023-08-25T14:41:11.532063Z","shell.execute_reply":"2023-08-25T14:41:39.725884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we would want to improve our model by selecting a different learning rate. FastAI offers us a tool to do just that: the learning rate finder. For more information about how this works check out the official documentation.\n\nYou can find the research that inspired the creation of this method here: https://arxiv.org/pdf/1506.01186.pdf.","metadata":{}},{"cell_type":"code","source":"learn = vision_learner(dls, resnet34, metrics = error_rate)\nlr_min, lr_steep = learn.lr_find(suggest_funcs = (minimum, steep))","metadata":{"execution":{"iopub.status.busy":"2023-08-25T15:40:04.401042Z","iopub.execute_input":"2023-08-25T15:40:04.401545Z","iopub.status.idle":"2023-08-25T15:40:52.590341Z","shell.execute_reply.started":"2023-08-25T15:40:04.401489Z","shell.execute_reply":"2023-08-25T15:40:52.588899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lr_min, lr_steep","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:42:44.077920Z","iopub.execute_input":"2023-08-25T14:42:44.078403Z","iopub.status.idle":"2023-08-25T14:42:44.086535Z","shell.execute_reply.started":"2023-08-25T14:42:44.078368Z","shell.execute_reply":"2023-08-25T14:42:44.085243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We want to respect one of the following criteria when we choose a learning rate:\n* to be one order of magnitude smaller than the value where the minimum value of the loss was achieved (this means 10 times smaller than the value where we found the minimum)\n* to be the last point where the loss was clearly decreasing ","metadata":{}},{"cell_type":"code","source":"learn = vision_learner(dls, resnet34, metrics = error_rate)\nlearn.fine_tune(2, base_lr = 3e-2)","metadata":{"execution":{"iopub.status.busy":"2023-08-25T15:43:27.244378Z","iopub.execute_input":"2023-08-25T15:43:27.245035Z","iopub.status.idle":"2023-08-25T16:15:09.789253Z","shell.execute_reply.started":"2023-08-25T15:43:27.244981Z","shell.execute_reply":"2023-08-25T16:15:09.788037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Making Predictions","metadata":{}},{"cell_type":"markdown","source":"Now we want to make predictions on our test data. The 'glob' function is used to match all the paths that contain files that end in '.jpg'; it returns an object generator, so in order to work with the paths we created a list. \n\nIn order to make predictions on new data we have to create the test dataloader. That's what the second line of code does. In order to make the predictions on this test dataloader we have to specify it when we call 'get_preds' (we did this by writing \"dl = test_dl\"). As we explained before, 'get_preds' returns a list of probabilities for each image. That's why we use 'argmax' to get the index where the biggest probability can be found. The indexes range from 0 to 9 and match the names of our classes perfectly. ","metadata":{}},{"cell_type":"code","source":"test_images = list(TEST.glob(\"*.jpg\"))\ntest_dl = learn.dls.test_dl(test_images)\n\npredictions, _ = learn.get_preds(dl = test_dl)\npredictions = np.argmax(predictions, axis = 1)","metadata":{"execution":{"iopub.status.busy":"2023-08-25T16:40:20.832743Z","iopub.execute_input":"2023-08-25T16:40:20.833481Z","iopub.status.idle":"2023-08-25T16:42:09.976357Z","shell.execute_reply.started":"2023-08-25T16:40:20.833439Z","shell.execute_reply":"2023-08-25T16:42:09.974861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The 'stem' method extracts the name of a file without its extension. We used it to get the ids of all the images and then match them with the labels computed by the model.","metadata":{}},{"cell_type":"code","source":"ids = [int(path.stem) + 1 for path in test_images]\nsubmission_data = pd.DataFrame({\"ImageId\": ids, \"Label\": predictions})","metadata":{"execution":{"iopub.status.busy":"2023-08-25T16:43:14.352485Z","iopub.execute_input":"2023-08-25T16:43:14.353274Z","iopub.status.idle":"2023-08-25T16:43:14.441966Z","shell.execute_reply.started":"2023-08-25T16:43:14.353217Z","shell.execute_reply":"2023-08-25T16:43:14.440682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, we create a csv file with our predictions using 'to_csv' method:","metadata":{}},{"cell_type":"code","source":"submission_data.to_csv('submission.csv', index = False)","metadata":{"execution":{"iopub.status.busy":"2023-08-25T16:43:16.847759Z","iopub.execute_input":"2023-08-25T16:43:16.849039Z","iopub.status.idle":"2023-08-25T16:43:16.927450Z","shell.execute_reply.started":"2023-08-25T16:43:16.848993Z","shell.execute_reply":"2023-08-25T16:43:16.926198Z"},"trusted":true},"execution_count":null,"outputs":[]}]}