{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Fastai/Pytorch - Implementing 'Best' Original MNIST Architecture"},{"metadata":{},"cell_type":"markdown","source":"This notebook contains an implementation of the 'best' original MNIST architecture found in [this awesome notebook](https://www.kaggle.com/cdeotte/how-to-choose-cnn-architecture-mnist) (but no data augmentation). Here, I've implemented the 'best' bare bones model with Fastai/Pytorch (original experiments in Keras). \n\nThis is a version 3 notebook. Changes from version 1 include normalizing the image pixel values from 0..225 to 0..1, and running the model for 20 epochs rather than 15 epochs, and adding some leaky-ness to the RELUs. Some text documentation cleaned up for more clarity."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"from fastai import *\nfrom fastai.vision import *\nDATAPATH = Path('/kaggle/input/Kannada-MNIST/')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# The Goal"},{"metadata":{},"cell_type":"markdown","source":"The goal of this challenge is to to decet Kannada digits. Kannada is a language spoken predominantly by people of Karnataka in southwestern India. We are given a .csv file containing pixel data and labels. After transforming the data, we train a network to label the numbers from _omdu_ (1) to _hattu_ (10)."},{"metadata":{},"cell_type":"markdown","source":"# Data processing"},{"metadata":{},"cell_type":"markdown","source":"The data is transformed from a .csv format into two numpy arrays. In the .csv, the first column contains the image label/id, and the rest of the columns contain the image pixel values in grayscale (0...225). The first numpy array contains the image data in the following shape: num_images x num_channels X image_height X image_witdth, and the other contains the label/id in the following shape: num_images. This is accomplised by: \n- reading the csv into a pandas dataframe\n- extracting the label/id\n- removing the label/id column, changing to a numpy float array\n- dividing pixel values by 255 and reshaping the image data into a 28x28 square\n- giving the resulting array an extra dimension to indicate the images are in grayscale"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_images_and_labels(csv,label):\n    fileraw = pd.read_csv(csv)\n    labels = fileraw[label].to_numpy()\n    data = fileraw.drop([label],axis=1).to_numpy(dtype=np.float32)\n    data = np.true_divide(data,255.).reshape((fileraw.shape[0],28,28))\n    data = np.expand_dims(data, axis=1)\n    return data, labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Process the training, testing and 'other' datasets, and then check to ensure the arrays look reasonable."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data, train_labels = get_images_and_labels(DATAPATH/'train.csv','label')\ntest_data, test_labels = get_images_and_labels(DATAPATH/'test.csv','id')\nother_data, other_labels = get_images_and_labels(DATAPATH/'Dig-MNIST.csv','label')\n\nprint(f' Train:\\tdata shape {train_data.shape}\\tlabel shape {train_labels.shape}\\n \\\nTest:\\tdata shape {test_data.shape}\\tlabel shape {test_labels.shape}\\n \\\nOther:\\tdata shape {other_data.shape}\\tlabel shape {other_labels.shape}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"raw","source":"The resulting data arrays look reasonable, and the size of the labels is the same. Let's display a labelled image:"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.title(f'Training Label: {train_labels[4]}')\nplt.imshow(train_data[4,0],cmap='gray');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Before moving forward, I need to create a validation set from the training data. Here, I create an array with random number, and assign 'True' if the number is less then 0.1, which then allows me to separate my training and validation set."},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(42)\nran_10_pct_idx = (np.random.random_sample(train_labels.shape)) < .1\n\ntrain_90_labels = train_labels[np.invert(ran_10_pct_idx)]\ntrain_90_data = train_data[np.invert(ran_10_pct_idx)]\n\nvalid_10_labels = train_labels[ran_10_pct_idx]\nvalid_10_data = train_data[ran_10_pct_idx]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Creating a Fastai Databunch"},{"metadata":{},"cell_type":"markdown","source":"Because Fastai does not have an API for directly adding numpy arrays into a databunch (_as far as I know, please leave a comment if you know a way!_), I created a bare-bones Torch Dataset class [based on this example](https://docs.fast.ai/basic_data.html) to allow me to create a `DataBunch`."},{"metadata":{"trusted":true},"cell_type":"code","source":"class ArrayDataset(Dataset):\n    \"Dataset for numpy arrays based on fastai example: \"\n    def __init__(self, x, y):\n        self.x, self.y = x, y\n        self.c = len(np.unique(y))\n    \n    def __len__(self):\n        return len(self.x)\n    \n    def __getitem__(self, i):\n        return self.x[i], self.y[i]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ds = ArrayDataset(train_90_data,train_90_labels)\nvalid_ds = ArrayDataset(valid_10_data,valid_10_labels)\nother_ds = ArrayDataset(other_data, other_labels)\ntest_ds = ArrayDataset(test_data, test_labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, I can create a Databunch, which contains, my training, validation and test sets, along with the batch size. I do not use the 'other' set, but it can be used for further model selection."},{"metadata":{"trusted":true},"cell_type":"code","source":"bs = 128\ndatabunch = DataBunch.create(train_ds, valid_ds, test_ds=test_ds, bs=bs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Architecture: use the 'best' "},{"metadata":{},"cell_type":"markdown","source":"Below is an implementation of the 'best' original MNIST architecture found [here](https://www.kaggle.com/cdeotte/how-to-choose-cnn-architecture-mnist). The tutorial was created using Keras, and here I've re-implemented it using a combination of Pytorch and Fastai. The Fastai `conv_layer` function returns a sequence of convolutional, ReLU and batchnorm layers. "},{"metadata":{"trusted":true},"cell_type":"code","source":"leak = 0.15\n\nbest_architecture = nn.Sequential(\n    \n    conv_layer(1,32,stride=1,ks=3,leaky=leak),\n    conv_layer(32,32,stride=1,ks=3,leaky=leak),\n    conv_layer(32,32,stride=2,ks=5,leaky=leak),\n    nn.Dropout(0.4),\n    \n    conv_layer(32,64,stride=1,ks=3,leaky=leak),\n    conv_layer(64,64,stride=1,ks=3,leaky=leak),\n    conv_layer(64,64,stride=2,ks=5,leaky=leak),\n    nn.Dropout(0.4),\n    \n    Flatten(),\n    nn.Linear(3136, 128),\n    relu(inplace=True),\n    nn.BatchNorm1d(128),\n    nn.Dropout(0.4),\n    nn.Linear(128,10)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This 'learner' in Fastai holds the data, model, loss function, and metric of interest. Note that a lot of the Fastai examples you will see use `cnn_learner`. I don't use that here because my model is not pretrained. "},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = Learner(databunch, best_architecture, loss_func = nn.CrossEntropyLoss(), metrics=[accuracy] )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The model is trained using a [one cycle policy](https://docs.fast.ai/callbacks.one_cycle.html), which from what I understand is not implemented in [this notebook](https://www.kaggle.com/cdeotte/how-to-choose-cnn-architecture-mnist). For best their best results, they train for 30 epochs. Here, I'm running for 20 epochs."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predictions for the Test data set"},{"metadata":{},"cell_type":"markdown","source":"Now, we can get the predictions for the test set. "},{"metadata":{"trusted":true},"cell_type":"code","source":"preds, ids = learn.get_preds(DatasetType.Test)\ny = torch.argmax(preds, dim=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame({ 'id': ids,'label': y })\nsubmission.to_csv(path_or_buf =\"submission.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"}},"nbformat":4,"nbformat_minor":1}