{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Fastai/Pytorch - Implementation of 6 Layer Convnet"},{"metadata":{},"cell_type":"markdown","source":"This notebook contains an architecture similar to [Anshuman Narayan's](https://www.kaggle.com/anshumandec94/6-layer-conv-nn-using-adam) and [Jinbao's](https://www.kaggle.com/jinbao/kannada-mnist-baseline) notebooks, but implemented in Fastai/Pytorch (original experiments in Keras). "},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"from fastai import *\nfrom fastai.vision import *\n\nDATAPATH = Path('/kaggle/input/Kannada-MNIST/')\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data processing"},{"metadata":{},"cell_type":"markdown","source":"The data is transformed from a .csv format. The first column contains the image label/id, and the rest of the columns contain the image pixel values in grayscale (0...225). They are processed into two numpy arrays: one containg the data in the following shape: num_images x num_channels X image_height X image_witdth, and the other containing the label/id in the following shape: num_images. The fucntion uses the following process: \n- read the csv into a pandas dataframe\n- extract the label/id\n- remove the label/id column, change to a numpy float array\n- divide pixel values by 255 and reshape the image data into a 28x28 square\n- give the resulting array an extra dimension to indicate the images are in grayscale"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_images_and_labels(csv,label):\n    fileraw = pd.read_csv(csv)\n    labels = fileraw[label].to_numpy()\n    data = fileraw.drop([label],axis=1).to_numpy(dtype=np.float32)\n    data = np.true_divide(data,255.).reshape((fileraw.shape[0],28,28))\n    data = np.expand_dims(data, axis=1)\n    return data, labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Process the training, testing and 'other' datasets, and then check to ensure the arrays look reasonable."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data, train_labels = get_images_and_labels(DATAPATH/'train.csv','label')\ntest_data, test_labels = get_images_and_labels(DATAPATH/'test.csv','id')\nother_data, other_labels = get_images_and_labels(DATAPATH/'Dig-MNIST.csv','label')\n\nprint(f' Train:\\tdata shape {train_data.shape}\\tlabel shape {train_labels.shape}\\n \\\nTest:\\tdata shape {test_data.shape}\\tlabel shape {test_labels.shape}\\n \\\nOther:\\tdata shape {other_data.shape}\\tlabel shape {other_labels.shape}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Display a labelled image:"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.title(f'Training Label: {train_labels[4]}')\nplt.imshow(train_data[4,0],cmap='gray');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Creating a Fastai Databunch"},{"metadata":{},"cell_type":"markdown","source":"Before creating a `Databunch`, I need to create a validation set from my training data."},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(42)\nran_10_pct_idx = (np.random.random_sample(train_labels.shape)) < .1\n\ntrain_90_labels = train_labels[np.invert(ran_10_pct_idx)]\ntrain_90_data = train_data[np.invert(ran_10_pct_idx)]\n\nvalid_10_labels = train_labels[ran_10_pct_idx]\nvalid_10_data = train_data[ran_10_pct_idx]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Because Fastai does not have an API for directly adding numpy arrays into a databunch (_as far as I know, please leave a comment if you know a way!_), I created a bare-bones Torch Dataset class [based on this example](https://docs.fast.ai/basic_data.html) to allow me to create a `DataBunch`."},{"metadata":{"trusted":true},"cell_type":"code","source":"class ArrayDataset(Dataset):\n    \"Dataset for numpy arrays based on fastai example: \"\n    def __init__(self, x, y):\n        self.x, self.y = x, y\n        self.c = len(np.unique(y))\n    \n    def __len__(self):\n        return len(self.x)\n    \n    def __getitem__(self, i):\n        return self.x[i], self.y[i]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ds = ArrayDataset(train_90_data,train_90_labels)\nvalid_ds = ArrayDataset(valid_10_data,valid_10_labels)\nother_ds = ArrayDataset(other_data, other_labels)\ntest_ds = ArrayDataset(test_data, test_labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, I can create a Databunch, which contains, my training, validation and test sets, along with the batch size. I do not use the 'other' set, but it can be used for further model selection."},{"metadata":{"trusted":true},"cell_type":"code","source":"bs = 128\ndatabunch = DataBunch.create(train_ds, valid_ds, test_ds=test_ds, bs=bs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Architecture"},{"metadata":{},"cell_type":"markdown","source":"Here I've created an architecture based on [Anshuman Narayan's](https://www.kaggle.com/anshumandec94/6-layer-conv-nn-using-adam) and [Jinbao's](https://www.kaggle.com/jinbao/kannada-mnist-baseline) notebooks. Changes that I made to those models:\n- used Fastai/Pytorch instead of Keras\n- used a different order to the components of a convolutional layer\n    - they used conv, batchnorm, relu\n    - I used conv, relu, batchnorm\n- used leaky relus\n- used AdaptiveConcatPooling instead of MaxPooling, which made my filters grow in this layer rather than a convolution"},{"metadata":{"trusted":true},"cell_type":"code","source":"leak = 0.25\nconv_drop = 0.35\nlin_drop = 0.08\n\nsix_conv_architecture = nn.Sequential(\n    \n    conv_layer(1,16,stride=1,ks=3,leaky=leak),\n    nn.Dropout(conv_drop),\n    \n    conv_layer(16,32,stride=1,ks=3,leaky=leak),\n    nn.Dropout(conv_drop),\n    AdaptiveConcatPool2d(14), \n    # return twice the number of filters \n    \n    conv_layer(64,64,stride=1,ks=5,leaky=leak),\n    nn.Dropout(conv_drop),\n    AdaptiveConcatPool2d(sz=7),\n    # return twice the number of filters \n    \n    conv_layer(128,128,stride=1,ks=5,leaky=leak),\n    nn.Dropout(conv_drop),\n    \n    conv_layer(128,64,stride=1,ks=3,leaky=leak),\n    nn.Dropout(conv_drop),\n    \n    conv_layer(64,32,stride=1,ks=3,leaky=leak),\n    nn.Dropout(conv_drop),\n    \n    Flatten(),\n    nn.Linear(1568, 50),\n    relu(inplace=True,leaky=leak),\n    nn.Dropout(lin_drop),\n    nn.Linear(50,25),\n    relu(inplace=True,leaky=leak),\n    nn.Dropout(lin_drop),\n    nn.Linear(25,10)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This 'learner' in Fastai holds the data, model, loss function, and metric of interest."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = Learner(databunch, six_conv_architecture, loss_func = nn.CrossEntropyLoss(), metrics=[accuracy] )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The model is trained using a [one cycle policy](https://docs.fast.ai/callbacks.one_cycle.html), for an arbitrary 15 epochs. "},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(15)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Test data set"},{"metadata":{},"cell_type":"markdown","source":"Now, we can get the predictions for the test set. "},{"metadata":{"trusted":true},"cell_type":"code","source":"preds, ids = learn.get_preds(DatasetType.Test)\ny = torch.argmax(preds, dim=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame({ 'id': ids,'label': y })","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv(path_or_buf =\"submission4.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"}},"nbformat":4,"nbformat_minor":1}