{"cells":[{"metadata":{},"cell_type":"markdown","source":"## FastAI Vision CNN learner on tabular data"},{"metadata":{},"cell_type":"markdown","source":"Even though the digit recognizer dataset is organized as a table, it is indeed pictures of handwritten digits. We usually treat it as any other tabular classification problem, but it may be worth transforming it into a set of pictures and applying fastai.vision library on it. Fastai Vision is pretty good, let's see how it holds up against Tabular in its home turf.\n\nWe are going to literally turn rows into pictures. To do that we need to make our own Imagelist class to change how the data loads into our databunch. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom fastai.vision import *\nfrom fastai.tabular import *\nimport matplotlib.pyplot as plt\nimport re","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path=Path('/kaggle/working/')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We are adding the index number for now for easier manipulation. We'll be removing it later."},{"metadata":{"trusted":true},"cell_type":"code","source":"train=pd.read_csv('/kaggle/input/train.csv')\ntrain['index'] = train.index\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating Custom ImageList\n\nWe need an ImageList to chuck into the cnn. So we should make a sub-class of the ImageList and change how it loads the data."},{"metadata":{"trusted":true},"cell_type":"code","source":"class PixelList(ImageList):\n    def open(self,index):\n        regex = re.compile(r'\\d+')\n        fn = re.findall(regex,index)\n        df_fn = self.inner_df[self.inner_df.index.values == int(fn[0])]\n        img_pixel = df_fn.drop(labels=['label','index'],axis=1).values\n        img_pixel = img_pixel.reshape(28,28)\n        img_pixel = np.stack((img_pixel,)*3,axis=-1)\n        return vision.Image(pil2tensor(img_pixel,np.float32).div_(255))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Step By Step :**\n\n1. Declaring our own PixelList class, subclass of ImageList so that it inherits everything.\n2. While subclassing an ItemList we can modify three methods and \"open\" is one of them. We are going to override this method because it is called when ImageList tries to open an image.\n        For more details check [here](https://docs.fast.ai/tutorial.itemlist.html#Creating-a-custom-ItemList-subclass).\n3. Defining a regular expression to find integers and compile it.\n4. Finding and save the rows matching index number.\n5. Picking a dataframe of only the index we have already chosen to read as an image.\n6. Droping index and label, they are not parts of the picture and taking an array of values.\n7. Reshaping 784x1 to 28x28 array.\n8. Blowing the image up 3 times.\n9. Creating an Image from the array using vision.image library.\n        For more details check [here](https://docs.fast.ai/vision.image.html#pil2tensor)."},{"metadata":{},"cell_type":"markdown","source":"Now we call it just like our vanilla ImageList. We have to pass cols='index' because PixelList \"open\" expects that argument."},{"metadata":{"trusted":true},"cell_type":"code","source":"src = (PixelList.from_df(train,'.',cols='index')\n      .split_by_rand_pct(0.1)\n      .label_from_df(cols='label'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Getting transforms is a bit tricky. These are not ordinary images, so flipping, zooming and cropping will yield weird instances. Also padding needs to be zeros, we can not expect reflection on the paddings."},{"metadata":{"trusted":true},"cell_type":"code","source":"tfms=get_transforms(rand_pad(padding=5,size=28,mode='zeros'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally we get them as size 128 and normalize them. We're not using imagenet_stats for the same reason."},{"metadata":{"trusted":true},"cell_type":"code","source":"data = (src.transform(tfms,size=128)\n       .databunch(num_workers=5,bs=48)\n       .normalize())\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at the data now."},{"metadata":{"trusted":true},"cell_type":"code","source":"data.show_batch(rows=3,figsize=(10,7))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training Resnet\n\nThis part is pretty self-explanatory. We get a Resnet101, train for a bit and see the results."},{"metadata":{"trusted":true},"cell_type":"code","source":"learner=cnn_learner(data,models.resnet101,metrics=[FBeta(),accuracy,error_rate])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.lr_find()\nlearner.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lr=9e-3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.fit_one_cycle(5,slice(lr))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.save('Resnet1')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.lr_find()\nlearner.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's unfreeze all layers except first two and train some more."},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.freeze_to(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.fit_one_cycle(4,slice(5e-6,lr/50))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.save('Resnetfinal')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training Tabular Learner\n\nWe just create a databunch, get a tabular learner and train it.\nTabular learner will consider this a classfication problem by default. All 784 columns are going to become features, so we have to be careful not to overfit."},{"metadata":{"trusted":true},"cell_type":"code","source":"train=pd.read_csv('/kaggle/input/train.csv')\ndata=TabularDataBunch.from_df(path,train,dep_var='label',valid_idx=range(4000,6000))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tablearner=tabular_learner(data,layers=[200,100],ps=[0.001,0.01],emb_drop=0.004,metrics=accuracy)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tablearner.lr_find()\ntablearner.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lr=2.2e-2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tablearner.fit_one_cycle(10,slice(lr),wd=0.1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tablearner.lr_find()\ntablearner.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tablearner.fit_one_cycle(4,4e-6,wd=0.1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Verdict [spoilers: Resnet wins]\n\nAfter some trial and error, I've found tabular learner to work best under the params I've chosen above. With no input pre-processing, Tabulars best accuracy is around 97~98%.\nWhere Resnet gives <99% results every time. It seems resnet has asserted its dominance again."},{"metadata":{},"cell_type":"markdown","source":"## What About DenseNet?\n\nI have experimented a bit with densenet as well. But as it takes an obscene amount of time to train, I've decided not to include it this round.\nDenseNet201 gave >98%, but not quite like resnet; not doing justice to the time it took to train.\n"},{"metadata":{},"cell_type":"markdown","source":"## Submit!\n\nSo we have our winner.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = pd.read_csv('../input/test.csv')\ndf_test['label'] = 0\ndf_test['index'] = df_test.index\ndf_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.load('Resnetfinal')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learner.data.add_test(PixelList.from_df(df_test,path='.',cols='index'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_test = learner.get_preds(ds_type=DatasetType.Test)\ntest_result = torch.argmax(pred_test[0],dim=1)\nresult = test_result.numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final = pd.Series(result,name='Label')\nsubmission = pd.concat([pd.Series(range(1,28001),name='ImageId'),final],axis=1)\nsubmission.to_csv('submit.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}