{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"!pip install --upgrade fastai\n\nfrom fastai.vision.all import *\nfrom fastai.text.all import *\nfrom fastai.collab import *\nfrom fastai.tabular.all import *","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-03-16T14:37:56.542366Z","iopub.execute_input":"2023-03-16T14:37:56.542654Z","iopub.status.idle":"2023-03-16T14:38:18.926323Z","shell.execute_reply.started":"2023-03-16T14:37:56.542625Z","shell.execute_reply":"2023-03-16T14:38:18.925081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Read train.csv into a `DataFrame`.","metadata":{}},{"cell_type":"code","source":"trainData = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ntrainData.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:38:18.929834Z","iopub.execute_input":"2023-03-16T14:38:18.930765Z","iopub.status.idle":"2023-03-16T14:38:22.485116Z","shell.execute_reply.started":"2023-03-16T14:38:18.930697Z","shell.execute_reply":"2023-03-16T14:38:22.484011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets drop the first column, which is the label, so that we just have the pixel data.","metadata":{}},{"cell_type":"code","source":"imageData = trainData.drop([\"label\"], axis=1)\nimageData.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:38:22.486784Z","iopub.execute_input":"2023-03-16T14:38:22.487567Z","iopub.status.idle":"2023-03-16T14:38:22.589383Z","shell.execute_reply.started":"2023-03-16T14:38:22.487528Z","shell.execute_reply":"2023-03-16T14:38:22.588078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I'm going to start by figuring out how to transform the data so that we can see each row as an image.\n\nCreating a tensor for each row.","metadata":{}},{"cell_type":"code","source":"imageTensor = torch.tensor(imageData.values)\nimageTensor.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:38:22.592731Z","iopub.execute_input":"2023-03-16T14:38:22.593252Z","iopub.status.idle":"2023-03-16T14:38:22.832435Z","shell.execute_reply.started":"2023-03-16T14:38:22.593213Z","shell.execute_reply":"2023-03-16T14:38:22.831114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So this tensor has 42000 rows (rank 42000) with 784 columns each. Each column representing a pixel. Each row representing a 28 pixels wide by 28 pixels tall (28*28=784) image. To make images, we'll want to transform the tensor so that each row has a shape of 28 by 28.","metadata":{}},{"cell_type":"code","source":"imagesReshaped = torch.reshape(imageTensor, (42000, 28, 28))\nimagesReshaped.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:38:22.834219Z","iopub.execute_input":"2023-03-16T14:38:22.834742Z","iopub.status.idle":"2023-03-16T14:38:22.850387Z","shell.execute_reply.started":"2023-03-16T14:38:22.834701Z","shell.execute_reply":"2023-03-16T14:38:22.849296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can check if our data looks correct. The output should look like the number 4.","metadata":{}},{"cell_type":"code","source":"torch.set_printoptions(linewidth=200)\nimagesReshaped[3]","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:38:22.852015Z","iopub.execute_input":"2023-03-16T14:38:22.852470Z","iopub.status.idle":"2023-03-16T14:38:22.901995Z","shell.execute_reply.started":"2023-03-16T14:38:22.852433Z","shell.execute_reply":"2023-03-16T14:38:22.900749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I had to use `set_printoptions` to get the printout to look correct. Lets now set it back to the default.","metadata":{}},{"cell_type":"code","source":"torch.set_printoptions(profile='default')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:38:22.906700Z","iopub.execute_input":"2023-03-16T14:38:22.908652Z","iopub.status.idle":"2023-03-16T14:38:22.915211Z","shell.execute_reply.started":"2023-03-16T14:38:22.908606Z","shell.execute_reply":"2023-03-16T14:38:22.913943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can take our list of 28 by 28 tensors show one as an image. To do this, we'll use `matplotlib`.","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:38:22.916670Z","iopub.execute_input":"2023-03-16T14:38:22.917338Z","iopub.status.idle":"2023-03-16T14:38:22.933389Z","shell.execute_reply.started":"2023-03-16T14:38:22.917299Z","shell.execute_reply":"2023-03-16T14:38:22.932200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(imagesReshaped[3], cmap=\"gray\")","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:38:22.935668Z","iopub.execute_input":"2023-03-16T14:38:22.936757Z","iopub.status.idle":"2023-03-16T14:38:23.292291Z","shell.execute_reply.started":"2023-03-16T14:38:22.936716Z","shell.execute_reply":"2023-03-16T14:38:23.291127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So now I'm going to take what I learned and try to save all the training images to disk. There is a `save_image` function I can use that takes a tensor as input. However, [the tensor needs to be normalized](https://discuss.pytorch.org/t/save-image-not-working-runtimeerror-result-type-float-cant-be-cast-to-the-desired-output-type-byte/128537), meaning each of the pixel values need to be between 0 and 1, so I'll divide each pixel value by 255.","metadata":{}},{"cell_type":"code","source":"from torchvision.utils import save_image","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:38:23.299605Z","iopub.execute_input":"2023-03-16T14:38:23.302079Z","iopub.status.idle":"2023-03-16T14:38:23.314557Z","shell.execute_reply.started":"2023-03-16T14:38:23.302015Z","shell.execute_reply":"2023-03-16T14:38:23.313184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Path('train').mkdir(exist_ok=True)\n    \nfor index, row in trainData.iterrows():\n    label = row[\"label\"]\n    pixels = row.drop([\"label\"])\n    imageTensor = torch.reshape(torch.tensor(pixels.values), (28, 28))\n    normalizedImageTensor = imageTensor / 255\n    folder = f'train/{label}'\n    Path(folder).mkdir(exist_ok=True)\n    save_image(normalizedImageTensor, f'{folder}/{index}.png')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:38:23.319594Z","iopub.execute_input":"2023-03-16T14:38:23.319966Z","iopub.status.idle":"2023-03-16T14:39:16.636355Z","shell.execute_reply.started":"2023-03-16T14:38:23.319927Z","shell.execute_reply":"2023-03-16T14:39:16.635131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now lets create the model. We'll use the `ImageDataLoaders.from_folder` function. [Docs for it are here](https://docs.fast.ai/vision.data.html#imagedataloaders.from_folder). I'll explain my parameter choices.\n\n* I specified `valid_ptc=0.2` so it uses 20% of the training data for a validation set.\n* So I ran it and it requires `path` to, even though the default value in the docs (\".\") is fine. So I specified that too.\n* `batch_tfms` is set to do some data augmentation. Since these are digits I don't want them to be flipped or rotated too much, as that could alter their meaning.","metadata":{}},{"cell_type":"code","source":"dataloader = ImageDataLoaders.from_folder(path=\".\", valid_pct=0.2, batch_tfms=aug_transforms(do_flip=False, max_rotate=5, max_zoom=1.2))\ndataloader.show_batch(max_n=12)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:39:16.641328Z","iopub.execute_input":"2023-03-16T14:39:16.643618Z","iopub.status.idle":"2023-03-16T14:39:30.616162Z","shell.execute_reply.started":"2023-03-16T14:39:16.643578Z","shell.execute_reply":"2023-03-16T14:39:30.615017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now lets create the learner that uses the dataloader.","metadata":{}},{"cell_type":"code","source":"learner = vision_learner(dataloader, resnet34, metrics=accuracy)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:39:30.619110Z","iopub.execute_input":"2023-03-16T14:39:30.619428Z","iopub.status.idle":"2023-03-16T14:39:31.928147Z","shell.execute_reply.started":"2023-03-16T14:39:30.619398Z","shell.execute_reply":"2023-03-16T14:39:31.927079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before we tell the learner to start learning, we should find an appropriate learning rate.","metadata":{}},{"cell_type":"code","source":"learningRate = learner.lr_find(suggest_funcs=(valley, slide))\nlearningRate","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:39:31.929523Z","iopub.execute_input":"2023-03-16T14:39:31.929916Z","iopub.status.idle":"2023-03-16T14:39:47.910351Z","shell.execute_reply.started":"2023-03-16T14:39:31.929876Z","shell.execute_reply":"2023-03-16T14:39:47.909045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now lets finally train the learner. In my first version I only passed 2 into `fine_tune` but this time I decided to pass 10. For the learning rate, we can see that 10^-2 should work well.","metadata":{}},{"cell_type":"code","source":"learner.fine_tune(10, .01)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:39:47.912519Z","iopub.execute_input":"2023-03-16T14:39:47.912942Z","iopub.status.idle":"2023-03-16T14:53:35.706794Z","shell.execute_reply.started":"2023-03-16T14:39:47.912901Z","shell.execute_reply":"2023-03-16T14:53:35.705441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the first version with just 2 passes I still got about 98% accuracy. Now with 10 passes it's over 99%. Seems pretty good! Lets see the results.","metadata":{}},{"cell_type":"code","source":"learner.show_results()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:53:35.712171Z","iopub.execute_input":"2023-03-16T14:53:35.715672Z","iopub.status.idle":"2023-03-16T14:53:36.452370Z","shell.execute_reply.started":"2023-03-16T14:53:35.715625Z","shell.execute_reply":"2023-03-16T14:53:36.451106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets see the results it had the most trouble with.","metadata":{}},{"cell_type":"code","source":"interp = Interpretation.from_learner(learner)\ninterp.plot_top_losses(9, figsize=(15,10))","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:53:36.457868Z","iopub.execute_input":"2023-03-16T14:53:36.460962Z","iopub.status.idle":"2023-03-16T14:53:52.960308Z","shell.execute_reply.started":"2023-03-16T14:53:36.460916Z","shell.execute_reply":"2023-03-16T14:53:52.959108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"classificationInterp = ClassificationInterpretation.from_learner(learner)\nclassificationInterp.plot_confusion_matrix()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:53:52.965497Z","iopub.execute_input":"2023-03-16T14:53:52.968561Z","iopub.status.idle":"2023-03-16T14:54:25.088582Z","shell.execute_reply.started":"2023-03-16T14:53:52.968516Z","shell.execute_reply":"2023-03-16T14:54:25.087323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I don't know if even I could figure out what those digits are supposed to be.\n\nLets now try using test time agumentation (TTA) to improve the accuracy. First, lets see the accuracy without TTA, then with.","metadata":{}},{"cell_type":"code","source":"probs, targs = learner.get_preds(dl=dataloader.valid)\naccuracy(probs, targs)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:10:48.231890Z","iopub.execute_input":"2023-03-16T15:10:48.232394Z","iopub.status.idle":"2023-03-16T15:11:01.550942Z","shell.execute_reply.started":"2023-03-16T15:10:48.232350Z","shell.execute_reply":"2023-03-16T15:11:01.549768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"probs, targs = learner.tta(dl=dataloader.valid)\naccuracy(probs, targs)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:11:24.117474Z","iopub.execute_input":"2023-03-16T15:11:24.118408Z","iopub.status.idle":"2023-03-16T15:12:38.202474Z","shell.execute_reply.started":"2023-03-16T15:11:24.118357Z","shell.execute_reply":"2023-03-16T15:12:38.200519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A little better!","metadata":{}},{"cell_type":"markdown","source":"Now that we've tried the model with our training data, I think we can test it against the test set, formulate the results into what Kaggle wants, then upload the results to the leaderboard.\n\nLets start by creating image files for each row in test.csv.","metadata":{}},{"cell_type":"code","source":"testData = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")\n\nPath(\"test\").mkdir(exist_ok=True)\n    \nfor index, row in testData.iterrows():\n    imageTensor = torch.reshape(torch.tensor(row.values), (28, 28))\n    normalizedImageTensor = imageTensor / 255\n    save_image(normalizedImageTensor, f'test/{index + 1}.png')","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:54:25.094227Z","iopub.execute_input":"2023-03-16T14:54:25.097274Z","iopub.status.idle":"2023-03-16T14:54:56.321200Z","shell.execute_reply.started":"2023-03-16T14:54:25.097222Z","shell.execute_reply":"2023-03-16T14:54:56.319872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now lets load these images into a dataloader and then pass them to the learner to make predictions.","metadata":{}},{"cell_type":"code","source":"testImages = list(Path(\"./test\").glob('**/*'))\ntestSetDataLoader = dataloader.test_dl(testImages)\ntestSetDataLoader.show_batch()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T14:54:56.323286Z","iopub.execute_input":"2023-03-16T14:54:56.324134Z","iopub.status.idle":"2023-03-16T14:54:57.112697Z","shell.execute_reply.started":"2023-03-16T14:54:56.324088Z","shell.execute_reply":"2023-03-16T14:54:57.111491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"predictions = learner.tta(dl=testSetDataLoader)\npredictions","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:13:52.464895Z","iopub.execute_input":"2023-03-16T15:13:52.465415Z","iopub.status.idle":"2023-03-16T15:16:58.959247Z","shell.execute_reply.started":"2023-03-16T15:13:52.465372Z","shell.execute_reply":"2023-03-16T15:16:58.957893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"decodedPredictions = learner.dls.vocab[np.argmax(predictions[0], axis=1)]\ndecodedPredictions","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:25:13.971213Z","iopub.execute_input":"2023-03-16T15:25:13.971713Z","iopub.status.idle":"2023-03-16T15:25:14.731020Z","shell.execute_reply.started":"2023-03-16T15:25:13.971670Z","shell.execute_reply":"2023-03-16T15:25:14.729755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets look at the sample submission file to see the format it's in.","metadata":{}},{"cell_type":"code","source":"sampleSubmission = pd.read_csv(\"/kaggle/input/digit-recognizer/sample_submission.csv\")\nsampleSubmission.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:25:30.260265Z","iopub.execute_input":"2023-03-16T15:25:30.260840Z","iopub.status.idle":"2023-03-16T15:25:30.304743Z","shell.execute_reply.started":"2023-03-16T15:25:30.260794Z","shell.execute_reply":"2023-03-16T15:25:30.303712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets create our own submission file.","metadata":{}},{"cell_type":"code","source":"testImageIds = [int(img.stem) for img in testImages]\nsubmission = pd.DataFrame(testImageIds, columns=['ImageId'])\nsubmission['Label'] = decodedPredictions\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:25:41.574820Z","iopub.execute_input":"2023-03-16T15:25:41.575693Z","iopub.status.idle":"2023-03-16T15:25:41.660844Z","shell.execute_reply.started":"2023-03-16T15:25:41.575628Z","shell.execute_reply":"2023-03-16T15:25:41.659747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv(\"submission.csv\",index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-16T15:25:47.989607Z","iopub.execute_input":"2023-03-16T15:25:47.990129Z","iopub.status.idle":"2023-03-16T15:25:48.048883Z","shell.execute_reply.started":"2023-03-16T15:25:47.990048Z","shell.execute_reply":"2023-03-16T15:25:48.047761Z"},"trusted":true},"execution_count":null,"outputs":[]}]}