{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# A fastai implementation for the famous MNIST handwritten digit dataset","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"In this knowledge contest, we are trying to predict the number of the handwritten digit.","metadata":{}},{"cell_type":"markdown","source":"**Version History**\n* *Version 1* - Getting the notebook setup, importing basic modules, reading the dataset, building a simple model, evaluate results, no submission\n* *Version 2* - Adding submisson, and callbacks\n* *Version 3* - removed MixUp callback\n* *Version 4* - cleaned up code, CrossEntropyLoss\n* *Version 5* - removed training and testing images from output\n* *Version 6* - Increased training epochs\n* *Version 7* - Adding Mixed presision callbacks (Final submission?)\n* *Version 8* - Editting and updating","metadata":{}},{"cell_type":"markdown","source":"## Background\n\nI've tried to pull together a lot of the techniques I've used and seen other use.\n\nI feel like if I ran this notebook enough times, I'd eventually find a model that would get perfect accuracy but obviously I'm just overfitting to score super high on the test set\n\n### For beginners forking this notebook\n\nI highly suggest for anyone interested in solving problems like this to look into the course by Sylvain Gugger and Jeremy Howard. They spent a lot of time building `fastai` into a great library for accessing the latest deep learning techniques.\n\n[Practical Deep Learning for Coders](https://course.fast.ai/)","metadata":{}},{"cell_type":"markdown","source":"# Setup","metadata":{}},{"cell_type":"code","source":"%reload_ext autoreload\n%autoreload 2\n%matplotlib inline","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%time\nfrom fastai.vision.all import *\nfrom fastprogress.fastprogress import progress_bar\nimport shutil","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"INPUT_DIR = Path(\"../input/digit-recognizer\")\nINPUT_DIR.ls()\nWORKING_DIR = Path(\"./\")\nTRAINING_DATA = WORKING_DIR / 'train'\nTESTING_DATA = WORKING_DIR / 'test'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv(INPUT_DIR / 'train.csv')\ntest_df = pd.read_csv(INPUT_DIR / 'test.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Convert CSV to images","metadata":{}},{"cell_type":"code","source":"Y_train = train_df[\"label\"]\nX_train = train_df.drop(\"label\", axis=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train = X_train.values.reshape(-1, 28, 28)\nX_test = test_df.values.reshape(-1, 28, 28)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(X_train[0], cmap=\"gray\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Move files into directory to make using fastai factory methods easier","metadata":{}},{"cell_type":"code","source":"filenames = list(Y_train.index)\ncategory = list(Y_train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for name, digit, image in progress_bar(list(zip(list(Y_train.index), category, X_train))):\n    file_dir = TRAINING_DATA / str(digit)\n    file_dir.mkdir(parents=True, exist_ok=True)\n    file_name = f'{name}.png'\n    file_path = file_dir / file_name\n#     print(image.shape)\n    plt.imsave(file_path, image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for name, image in progress_bar(list(zip(list(test_df.index), X_test))):\n    file_dir = TESTING_DATA\n    file_dir.mkdir(parents=True, exist_ok=True)\n    file_name = f'{name}.png'\n    file_path = file_dir / file_name\n#     print(image.shape)\n    plt.imsave(file_path, image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model","metadata":{}},{"cell_type":"code","source":"batch_tfms = [*aug_transforms(do_flip=False,), Normalize.from_stats(*imagenet_stats)]\nmetrics = [accuracy]\nbs = 128","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dls = ImageDataLoaders.from_folder(TRAINING_DATA, valid_pct=0.2, batch_tfms=batch_tfms, bs=bs)\ndls.show_batch()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn = cnn_learner(dls, resnet34, metrics=metrics, loss_func=CrossEntropyLossFlat())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lrs = learn.lr_find()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lrs","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cbs = [\n    GradientAccumulation(),\n    MixedPrecision(),\n    SaveModelCallback(monitor='accuracy', comp=np.greater, min_delta=0.001),\n    ReduceLROnPlateau(monitor='accuracy', comp=np.greater, min_delta=0.001, patience=2),\n#     MixUp(0.4),\n    EarlyStoppingCallback(monitor='accuracy', comp=np.greater, min_delta=0.001, patience=3),\n      ]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.fine_tune(15, lrs.lr_min, cbs=cbs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluate Results","metadata":{}},{"cell_type":"code","source":"%time\ninterp = ClassificationInterpretation.from_learner(learn)\ninterp.plot_confusion_matrix()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%time\ninterp.plot_top_losses(8, nrows=2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"SAMPLE_SUBMISSION = pd.read_csv(INPUT_DIR / 'sample_submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SAMPLE_SUBMISSION.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_images = list(Path(r'./test').glob('**/*'))\ntest_dl = dls.test_dl(test_images)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds = learn.get_preds(dl=test_dl, with_decoded=True)\npreds = preds[2].tolist()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ids = [int(img.stem) + 1 for img in test_images]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame(ids, columns=['ImageId'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission['Label'] = preds","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv(\"submission.csv\",index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Remove Images\n\nFor some reason I can't make a submission from the notebook without deleting all of the saved training and test data files first.","metadata":{}},{"cell_type":"code","source":"shutil.rmtree(TRAINING_DATA)\nshutil.rmtree(TESTING_DATA)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}