{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About\nWhen I'm coding, I'm led by intuition.\n\nSometimes it’s good to dig a bit into the libraries to get a better feeling about what’s going on inside.\n\nEven though there are endless articles, videos or books I won’t get a good intuition unless I investigate a bit by myself.\n\nIn this kernel I simply wan’t to see how an input Image changes through kernel convolution inside a neural net. As said it’s done many times before but the notebook might be a source for somebody trying similar things on this dataset or with fastai. \n\nAnd as expected facing all traps and hurdles which come up along the path enhance the understanding of the framework additionaly."},{"metadata":{},"cell_type":"markdown","source":"# Setup Data and Model"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.animation as animation\n\nfrom fastai import *\nfrom fastai.vision import *\nfrom fastai.callbacks import *\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"\ndf_train = pd.read_csv(dirname+'/train.csv')\ndf_train['fn'] = df_train.index\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check label distribution:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['label'].hist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Unique distribution. Build an uniquely distributed validation set:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def valid_split(df, valid_ratio = 0.1):\n    valid_idx = []\n    for i in df['label'].unique():\n        valid_idx+=list(df[df['label']==i].sample(frac=valid_ratio, random_state=2020)['fn'].values)\n    return valid_idx\n\nvalid_idx = valid_split(df_train, valid_ratio = 0.2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load the entire dataset in memory to speedup training (see https://www.kaggle.com/joatom/kannada-mnist-speed-up-fastai-image-processing). Load one channel only (gray scale).\n\nSince we want to investigat in convolution we add a pixel (value 255, later scaled to 1) one pix apart from the let top corner (1,1) for debug purpose. So we can later easily check if a 3x3 kernel convolves as expexted."},{"metadata":{"trusted":true},"cell_type":"code","source":"class PixelImageItemList(ImageList):\n    \n    def __init__(self, myimages = {}, *args, **kwargs):\n        super().__init__(*args, **kwargs)\n        self.myimages = myimages \n    \n    def open(self,fn):\n        return self.myimages.get(fn)\n    \n    @classmethod\n    def from_df(cls, df:DataFrame, path:PathOrStr, cols:IntsOrStrs=0, folder:PathOrStr=None, suffix:str='', **kwargs)->'ItemList':\n        \"Get the filenames in `cols` of `df` with `folder` in front of them, `suffix` at the end.\"\n        res = super().from_df(df, path=path, cols=cols, **kwargs)\n        \n        # FULL LOAD of all images\n        for i, row in df.drop(labels=['label','fn'],axis=1).iterrows():\n            # Numpy to Image conversion from\n            # https://www.kaggle.com/heye0507/fastai-1-0-with-customized-itemlist\n            img_pixel = row.values.reshape(28,28)\n            img_pixel = np.stack((img_pixel,)*1,axis=-1)\n            ## mark for convolution test\n            img_pixel[1,1]=255\n            \n            res.myimages[res.items[i]]=vision.Image(pil2tensor(img_pixel,np.float32).div_(255))\n\n        return res","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"%%time\ndata = (PixelImageItemList.from_df(df=df_train,path='.',cols='fn')\n        .split_by_idx(valid_idx=valid_idx) #.split_by_rand_pct()\n        .label_from_df(cols='label')\n        .databunch(bs=128))\ndata.dataset[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.show_batch(rows=3, figsize=(5,5), cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We build a customized conv-layer since the one from fastai uses the ReLU with the parameter inplace = True. Which overrides the activations of the preciding conv layer. We set leaky to 0 so we can easily see the changes of the activations from conv2d-Layer to ReLU-layer."},{"metadata":{"trusted":true},"cell_type":"code","source":"#https://github.com/fastai/fastai/blob/master/fastai/layers.py#L111\ndef myconv_layer(ni:int, nf:int, ks:int=3, stride:int=1):\n    layers = [init_default(nn.Conv2d(ni,nf,stride=stride,kernel_size=ks,padding=1, bias=False),nn.init.kaiming_normal_)]\n    \n    # make sure the ReLU doesn't override the conv2d activations, so we keep more details for later\n    layers.append(nn.ReLU(inplace=False))\n    layers.append(nn.BatchNorm2d(nf))\n    \n    return nn.Sequential(*layers)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Adapted from https://www.kaggle.com/melissarajaram/fastai-pytorch-with-best-original-mnist-arch\n\nleak = 0.15\nmodel = nn.Sequential(\n    \n    myconv_layer(1,32),\n    conv_layer(32,32,stride=1,ks=3,leaky=leak),\n    conv_layer(32,32,stride=2,ks=5,leaky=leak),\n    nn.Dropout(0.4),\n    \n    conv_layer(32,64,stride=1,ks=3,leaky=leak),\n    conv_layer(64,64,stride=1,ks=3,leaky=leak),\n    conv_layer(64,64,stride=2,ks=5,leaky=leak),\n    nn.Dropout(0.4),\n    \n    Flatten(),\n    nn.Linear(3136, 128),\n    relu(inplace=True),\n    nn.BatchNorm1d(128),\n    nn.Dropout(0.4),\n    nn.Linear(128,10)\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = Learner(data, model, loss_func = nn.CrossEntropyLoss() , metrics=[accuracy])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(3, 0.5e-3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)\ninterp.plot_top_losses(12, figsize=(7,6))\ninterp.plot_confusion_matrix()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Look inside the model"},{"metadata":{},"cell_type":"markdown","source":"## Kernels"},{"metadata":{},"cell_type":"markdown","source":"There are several ways to get the kernels. Here are the 32 kernels of the first convolution."},{"metadata":{"trusted":true},"cell_type":"code","source":"conv_kernels=list(learn.model.parameters())[0] # alternativly: learn.model[0][0].weight.data\n# convert to numpy\nconv_kernels=conv_kernels.cpu().detach().numpy()\n# example:\nprint('Shape kernels first con layer:', conv_kernels.shape)\nprint('First kernel:\\n', conv_kernels[0,0,:,:])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see the kernels of layer one (be aware that the values are internaly scaled to gray scale):"},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(conv_kernels.shape[0]):\n    ax = plt.subplot(conv_kernels.shape[0]/8, 8, i+1)\n    ax.set_xticks([])\n    ax.set_yticks([])\n    conv_kernel = conv_kernels[i,0,:,:]\n    plt.imshow(conv_kernels[i,0,:,:], cmap='gray')\n        \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Activations"},{"metadata":{},"cell_type":"markdown","source":"The activations are the outputs of a layer when an image is passed through the net. The activations can by grabed with hooks in fastai as done [here](https://github.com/fastai/fastai/blob/master/tests/test_callbacks_hooks.py#L74).\n\nLets check the first few layers of the model:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.model[0:2]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now get an image example for the test:"},{"metadata":{"trusted":true},"cell_type":"code","source":"example_img = 0\n\n#\nm = learn.model.eval()\nx,y = data.train_ds[example_img]\nxb,_ = data.one_item(x)\nxb = xb.cuda()\n\ndata.train_ds.get(example_img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Apply the hook to get the activation of the first layer (conv2d):"},{"metadata":{"trusted":true},"cell_type":"code","source":"mblock = 0 # first Sequential\ninv_layer = m[mblock][0] # conv2d layer in first block\n\ndef hooked():\n    with hook_output(inv_layer) as hook_forward:\n        preds = m(xb)\n    return hook_forward\n\nacts = hooked().stored[0].cpu()\n\ninv_layer","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now lets compare the **activation** of the debug pixel [[0,0,0,..],[0,1,0,..],[0,0,0,..]] in the left corner, "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"acts[0:5][0,:5,:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"with the first kernel"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"conv_kernels[0,0,:,:]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks good!"},{"metadata":{},"cell_type":"markdown","source":"Next we build a function to get all activations:"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef img_activations(m:nn.Module, img_id:Image, data = data, ds=data.train_ds):\n    # create batch with one image\n    xb,_ = data.one_item(x)\n    xb = xb.cuda()\n    \n    # flatten to get activations of children\n    with hook_outputs(flatten_model(m)) as hook_forward:\n        preds=m.eval()(xb)\n    \n    return [i.cpu() for i in hook_forward.stored[:]]\n\nm = learn.model.eval()\nx,_ = data.train_ds[0]\nacts = img_activations(m, x,data)\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We do a check on the function if expected layers were hooked.\nTherefore we compare activation of debug area (left upper corner) for first layer (Conv2d(1,1)) and second layer (ReLU(leaky=0)):"},{"metadata":{"trusted":true},"cell_type":"code","source":"# apply ReLU to first layer and compare to second layer\ntorch.max(torch.zeros(3,3),acts[0][0,0,:3,:3]) == acts[1][0,0,:3,:3]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finaly we print the activations of all 32 kernels of the first (conv2d) and second (Relu) layer next to the Input image. See how the Relu cuts off some (the negative) activations. Be aware that the activations are scaled internaly to fit gray scale."},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\n\ndef scale_color(im,mn=None,mx=None):\n    if mn == None:\n        mn = im.min()\n    if mx == None:\n        mx = im.max()\n    return (im-mn)/(mx-mn)\n\nfig = plt.figure(figsize=(12, 6))\nims=[]\nfor i in range(32):#acts[0]):\n    \n    krnl = scale_color(np.copy(conv_kernels[i,0,:,:]))\n    org_im = x.data.numpy()[0,:,:] \n    layer_1_im = acts[0][0,i,:,:].numpy()\n    layer_2_im = acts[1][0,i,:,:].numpy()\n    \n    org_im[:3,:3] = krnl\n    layer_1_im = scale_color(layer_1_im) \n    layer_1_im[:3,:3] = krnl #[:3,:3] \n    layer_2_im = scale_color(layer_2_im) \n    layer_2_im[:3,:3] = krnl #[:3,:3] \n    \n    #layer_1_im=np.concatenate((org_im,layer_1_im,layer_2_im), axis=1)\n    \n    ax1 = plt.subplot(131, frameon=False)\n    ax1.set_title('(0) Input')\n    ax1.set_xticks([])\n    ax1.set_yticks([])\n    im1 = plt.imshow(org_im,animated=True, cmap='gray')\n    \n    ax2 = plt.subplot(132, frameon=False)\n    ax2.set_title('(1) Conv2d(1,32)')\n    ax2.set_xticks([])\n    ax2.set_yticks([])\n    im2 = plt.imshow(layer_1_im,animated=True, cmap='gray')\n    \n    ax3 = plt.subplot(133, frameon=False)\n    ax3.set_title('(2) ReLU(32)')\n    ax3.set_xticks([])\n    ax3.set_yticks([])\n    im3 = plt.imshow(layer_2_im,animated=True, cmap='gray')\n      \n    ims.append([im1,im2, im3]) #im1,\n\nani = animation.ArtistAnimation(fig, ims, interval=500, blit=False, repeat_delay=1000)\n\nfrom IPython.display import HTML\nHTML(ani.to_jshtml())  \n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"More to be done ..."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}