{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Version History\n\n* V6\n    - Resnet with Stage 1 (only head with 4 epochs) and Stage 2 (full train with 8 epochs) + More aggresive transformation applied to see if generalization can be improved.\n    - <font color = red>Public Score: TBD Rank: TBD</font>\n* V5\n    - Reverting back to single kernel for train and prediction\n    - Resnet with Stage 1 (only head with 4 epochs) and Stage 2 (full train with 4 epochs) + More aggresive transformation applied to see if generalization can be improved.\n    - <font color = red>Public Score: 0.9680 Rank: Did not improve</font>\n* V4 \n    - Scoring is slow. Is it running the entire notebook again?\n    - **Idea**: Try separating the prediction from the training process so that scoring is faster (training does not have to be run again while scoring).\n    - However, this does not seem to work. Scoring is equally slow even after separating the training and prediction into separate notebooks.\n* V3 \n    - Resnet with Stage 1 (only head with 4 epochs) and Stage 2 (full train with 4 epochs) \n    - <font color = green>**Public Score: 0.9742 Rank: 416/603 at time of submission**</font>\n* V2\n    - Resnet with Stage 1 (only head with 4 epochs) and Stage 2 (full train with 1 epochs) \n    - <font color = red>Public Score: 0.9580 Rank: Did not improve</font>\n* V1\n    - Resnet with Stage 1 (only head with 4 epochs) and Stage 2 (full train with 2 epochs)\n    - <font color = green>Public Score: 0.9708 Rank: 453/603 at time of submission</font>\n\n\n* **Summary:**\n    - Getting a validation accuracy of more than 99% consistently.\n    - In general Stage 2 can be trained for some more epochs since the train error is still more than the validation error.\n    - Since the public score (accuracy) is lower, try to train with more severe transformations so that it generalizes a little better."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n\n# Input data files are available in the \"../input/\" directory.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Import Libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport os\nimport imageio\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import fastai\nfrom fastai.vision import *\nfrom fastai.metrics import error_rate\nprint(fastai.__version__) # Checking version","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"verbose = 0\ntrain = True  ## DO NOT CHANGE 'train' to False - See NOTE below ##\n# Idea was to mark 'train' as False before submission but this deoes not help to speed up submission.\n# Hence not using 'train' anymore, but not deleting for now (set always to True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Read Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"root = '/kaggle/input/'\nparent = \"Kannada-MNIST/\"\n\nroot_plus_parent = root + parent\n\nif train:\n    if verbose >= 1:\n        print(root_plus_parent)\n\n    if verbose >= 1:\n        for dirname, _, filenames in os.walk(root):\n            for filename in filenames:\n                print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    df_train = pd.read_csv(root_plus_parent + \"train.csv\")\n    \nif train & verbose >= 1:\n    print(df_train.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = pd.read_csv(root_plus_parent + \"test.csv\")\nif train & verbose >= 1:\n    print(df_test.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train & verbose >= 1:\n    print(df_train.shape, df_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    train_x = df_train.iloc[:,1:].values.reshape(-1,28,28)\n    \nif train & verbose >= 1:\n    print(train_x.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    train_y = df_train['label'].values\n\nif train & verbose >= 1:\n    print(train_y.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_x = df_test.iloc[:,1:].values.reshape(-1,28,28)\nif train & verbose >= 1:\n    print(test_x.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train & verbose >= 1:\n    plt.imshow(train_x[0,:], cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Save data in right format for fastai"},{"metadata":{"trusted":true},"cell_type":"code","source":"# This function has been taken from another kernel: https://www.kaggle.com/demonplus/kannada-mnist-with-fast-ai-and-resnet\ndef save_imgs(path:Path, data, labels=[]):\n    path.mkdir(parents=True,exist_ok=True)\n    for label in np.unique(labels):\n        (path/str(label)).mkdir(parents=True,exist_ok=True)\n    for i in range(len(data)):\n        if(len(labels)!=0):\n            imageio.imsave( str( path/str(labels[i])/(str(i)+'.jpg') ), data[i] )\n        else:\n            imageio.imsave( str( path/(str(i)+'.jpg') ), data[i] )  # For test data which does not have a label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = \"../input/data\"\ntrain_path = Path('../input/data/train')\ntest_path = Path('../input/data/test')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    save_imgs(train_path,train_x,train_y)\n\nif train & verbose >= 1:\n    print((train_path).ls())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"save_imgs(test_path,test_x)\n\nif train & verbose >= 1:\n    print((test_path).ls())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create DataBunch object"},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    np.random.seed(42)  # Make sure your validation set is reproducible\n    # tfms = get_transforms(do_flip=False)  # We dont want to flip the images since numbers are not written flipped\n    tfms = get_transforms(do_flip=False, max_rotate=30, max_zoom=1.2, max_lighting=0.4,\n                          max_warp=0.3)  # We dont want to flip the images since numbers are not written flipped + more aggressive augmentation\n\n    data = ImageDataBunch.from_folder(path=path,  \n                                      valid_pct=0.2,\n                                      ds_tfms=tfms,\n                                      train='train',\n                                      test='test',\n                                      size=28,\n                                      bs=64).normalize(imagenet_stats)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train & verbose >= 1:\n    print(len(data.train_ds))\n    print(len(data.valid_ds))\n    print(len(data.test_ds))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train & verbose >= 1:\n    data.show_batch(rows=6, figsize=(12,12))\n\n# Subtle differences between the following which might be hard to detect\n# - 9 and 6\n# - 1 and 0\n# - 7 and 3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train & verbose >= 1:\n    print(data.classes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create Learner"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Kaggle comes with internet off. So have to copy over model to the location where fastai would have downloaded it.\n# https://forums.fast.ai/t/how-can-i-load-a-pretrained-model-on-kaggle-using-fastai/13941/23\n\n!mkdir -p /tmp/.cache/torch/checkpoints\n!cp ../input/resnet50/resnet50.pth /tmp/.cache/torch/checkpoints/resnet50-19c8e357.pth","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    learn = cnn_learner(data, models.resnet50, metrics=[error_rate, accuracy])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train & verbose >= 1:\n    print(learn.model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    learn.fit_one_cycle(4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    learn.recorder.plot_losses()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    learn.save('resnet50-stage-1')  # Saves it in the directory where the images exist (under a folder called 'models' in there)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fine tune model"},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    learn.lr_find()\n    learn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    learn.unfreeze()  # Train all layers\n    learn.fit_one_cycle(4, max_lr=slice(1e-5,1e-3))  # As per lr_finder (initial layers trained on smaller learning rate)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    learn.recorder.plot_losses()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    learn.save('resnet50-stage-2a')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<font color = red>**Train Loss is still more than the validation loss, so we could train for some more time. --> Tried this with 4 more epochs, but did not seem to reduce train loss below the validation loss.**</font>[](http://)"},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    learn.fit_one_cycle(4, max_lr=slice(1e-5,1e-3))  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    learn.recorder.plot_losses()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    learn.save('resnet50-stage-2b')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<font color = red>**Train Loss is still more than validation loss, but we will use this for now (though we could train for some more time to make sure train error is below the validation error).**</font>"},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    learn.export(file = Path(\"/kaggle/working/export.pkl\"))  # Not needed unless you want to download the model and then upload as a dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualise Results and Metrics"},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    interp = ClassificationInterpretation.from_learner(learn)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    interp.plot_confusion_matrix()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    print(interp.most_confused())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if train:\n    interp.plot_top_losses(9, figsize=(10,10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://docs.fast.ai/tutorial.inference.html#A-classification-problem\nif train:\n    deployed_path = \"/kaggle/working/\"\nelse:\n    deployed_path = \"../input/kannada-mnist-resnet50/\"  # If using a deployed model (not using right now)\n    \nprint(deployed_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = load_learner(deployed_path, test=ImageList.from_folder(test_path))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Adapted from another kernel: https://www.kaggle.com/demonplus/kannada-mnist-with-fast-ai-and-resnet\ntest_preds_probs, _ = learn.get_preds(DatasetType.Test)\ntest_preds = torch.argmax(test_preds_probs, dim=1)\nif verbose >= 1:\n    print(test_preds_probs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num = len(learn.data.test_ds)\nindexes = {}\n\nfor i in range(num):\n    filename = str(learn.data.test_ds.items[i]).split('/')[-1]\n    filename = filename[:-4] # get rid of .jpg\n    indexes[(int)(filename)] = i","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame({'id': range(0, num), 'label': [test_preds[indexes[x]].item() for x in range(0, num)] })\nprint(submission)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv(path_or_buf =\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}