{"cells":[{"metadata":{},"cell_type":"markdown","source":"# FastAI.tabular for Cat in the Dat\n\nThe purpose of this notebook is to show off the simple effectiveness of FastAI's tabular model, feel free to fork this notebook and give it a try. For further details on FasAI.Tabular see lesson 4 of FastAI's Practical Deep Learning for Coders course https://course.fast.ai/videos/?lesson=4\n\nPart 1 of this notebook is the simple model which will produce a reasonable score, using only a basic FastAI tabular model with minimal hyperparameter tuning. The next two parts show more complex methods that don't score better, and the final section is about blending the results to get a better result. This is a subsection of my current best submission for the Cat in the Dat competition. The notebook should score around .781 (a good score but nothing incredible).\n\n\nWhile not directly involved with this notebook, Peter Spangler https://www.linkedin.com/in/peterspangler/ and Cody Nault https://www.linkedin.com/in/codynault/ for the collaboration on one of the precursors to this notebook "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from fastai.tabular import *","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dat = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv', index_col='id')\ntest_dat = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv', index_col='id')\nout = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/sample_submission.csv', index_col='id')\ndat.head(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"FastAI won't be able to handle continuous values that aren't already integers or floats, so below i define all values, other than the target class, as being continuous"},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_names = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4', 'nom_0', 'nom_1', 'nom_2', \n             'nom_3', 'nom_4', 'nom_5', 'nom_6', 'nom_7', 'nom_8', 'nom_9',\n            'ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4', 'ord_5', 'day', 'month']\ncont_names = []","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dep_var = ['target']\nprocs = [FillMissing, Categorify, Normalize]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"FillMissing.FillStrategy='MEAN'\n\nPATH = Path('/kaggle/input/cat-in-the-dat-ii/')\ntest = TabularList.from_df(test_dat, path=PATH, cat_names=cat_names, cont_names=cont_names)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = (TabularList.from_df(dat, path=PATH,\n                            cat_names=cat_names, \n                            cont_names=cont_names,\n                            procs=procs)\n                           .split_by_idx(valid_idx = range(len(dat)-50000, len(dat)))\n                           .label_from_df(cols=dep_var)\n                           .add_test(test)\n                           .databunch())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.show_batch(rows=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = tabular_learner(data, layers=[200,100], metrics=[accuracy, FBeta(average='weighted')], ps=0.15)\nModel_Path = Path('/kaggle/working/cat-in-dat/')\nlearn.model_dir = Model_Path","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit(1, lr=1e-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.save('vinilla')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## From Classification to Regression\n\nUp till now the model has been a classification model, but the competition expects the input values to fall between 0 and 1 to indicate likelihood of the target class. When we call the predict function on our data FastAI will actually provide more info than just the predicted class. If you look at the cell below you will notice that we can specifically select the confidence of a positive case from the output.  If we take these confidence scores we can use them to make our submission entry."},{"metadata":{"trusted":true},"cell_type":"code","source":"row = dat.iloc[9]\nq = learn.predict(row)\nrow = dat.iloc[0]\nv = learn.predict(row)\nprint('Positive Case:')\nprint(q)\nprint(float(q[2][1]),'\\n\\n')\nprint('Negaitive Case:')\nprint(v)\nprint(float(v[2][1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = learn.get_preds(ds_type=DatasetType.Test)[0][:,1].numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_1 = pd.DataFrame({'id': out.index, 'target': preds})\nsubmission_1.to_csv('/kaggle/working/cat-in-dat/submission_1.csv', header=True, index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below we can see a rough idea of what the submission value distribution looks like. Assuming the test data mirrors the training, we should have about 20% of cases be positive so the mean predicted value should be around 0.2 "},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_1.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Label encoding\n\nFor this second approach I  hand performed the label encoding to change the values of each entry to numbers instead of using the inbuilt fastAI method. Label encoding is the term for changing string values to number values so that they are machine interpretable. The advantage of doing the label encoding by hand is that we can encode ordinal values in a way that keeps the values properly ordered. For instance , we can translate the hexadecimal values from nom_5-9 into integer values so that they can be translated to proper continuous values for this model. The hidden cell below shows off the label encoding methods I was using. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Resetting everything for round two\nlearn.destroy\ndat = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv', index_col='id')\ntest_dat = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv', index_col='id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_dat = dat.copy()\ndef int_it(num, x):\n    try:\n        return int(num, x)\n    except:\n        return np.nan\n\ntrain_dat['nom_5'] = pd.Series([int_it(x,16) for x in train_dat.nom_5], index=train_dat.index)\ntest_dat['nom_5'] = pd.Series([int_it(x,16) for x in test_dat.nom_5], index=test_dat.index)\n\ntrain_dat['nom_6'] = pd.Series([int_it(x,16) for x in train_dat.nom_6], index=train_dat.index)\ntest_dat['nom_6'] = pd.Series([int_it(x,16) for x in test_dat.nom_6], index=test_dat.index)\n\ntrain_dat['nom_7'] = pd.Series([int_it(x,16) for x in train_dat.nom_7], index=train_dat.index)\ntest_dat['nom_7'] = pd.Series([int_it(x,16) for x in test_dat.nom_7], index=test_dat.index)\n\ntrain_dat['nom_8'] = pd.Series([int_it(x,16) for x in train_dat.nom_8], index=train_dat.index)\ntest_dat['nom_8'] = pd.Series([int_it(x,16) for x in test_dat.nom_8], index=test_dat.index)\n\ntrain_dat['nom_9'] = pd.Series([int_it(x,16) for x in train_dat.nom_9], index=train_dat.index)\ntest_dat['nom_9'] = pd.Series([int_it(x,16) for x in test_dat.nom_9], index=test_dat.index)\n\ntrain_dat['ord_1'] = train_dat['ord_1'].fillna(0)\nord_1_map = {0 : np.nan, 'Novice': 1, 'Contributor': 2,'Expert': 3 , 'Master' : 4, 'Grandmaster': 5}\ntrain_dat['ord_1'] = pd.Series([ord_1_map[x] for x in train_dat.ord_1], index=train_dat.index)\ntest_dat['ord_1'] = test_dat['ord_1'].fillna(0)\ntest_dat['ord_1'] = pd.Series([ord_1_map[x] for x in test_dat.ord_1], index=test_dat.index)\n\ntrain_dat['ord_2'] = train_dat['ord_2'].fillna(0)\nord_2_map = {0 : np.nan, 'Freezing': 1, 'Cold': 2,'Warm': 3 , 'Hot' : 4, 'Boiling Hot': 5, 'Lava Hot' : 6}\ntrain_dat['ord_2'] = pd.Series([ord_2_map[x] for x in train_dat.ord_2], index=train_dat.index)\ntest_dat['ord_2'] = test_dat['ord_2'].fillna(0)\ntest_dat['ord_2'] = pd.Series([ord_2_map[x] for x in test_dat.ord_2], index=test_dat.index)\n\ntrain_dat['ord_3'] = train_dat['ord_3'].fillna(0)\nord_3_map = {0:np.nan,'a':1, 'b':2,'c':3,'d':4, 'e': 5, 'f' : 6, 'g' : 7, 'h' : 8, 'i' : 9, 'j' : 10,\n             'k' : 11, 'l' : 12, 'm' : 13, 'n' : 14, 'o' : 15, 'p' : 16, 'q' : 17, 'r' : 18, 's' : 19,\n             't' : 20, 'u' : 21, 'v' : 22, 'w': 23, 'x' : 24, 'y' : 25, 'z' : 26}\ntrain_dat['ord_3'] = pd.Series([ord_3_map[x] for x in train_dat.ord_3], index=train_dat.index)\ntest_dat['ord_3'] = test_dat['ord_3'].fillna(0)\ntest_dat['ord_3'] = pd.Series([ord_3_map[x] for x in test_dat.ord_3], index=test_dat.index)\n\ntrain_dat['ord_4'] = train_dat['ord_4'].fillna(0)\nord_4_map = {0:np.nan,'A':1, 'B':2,'C':3,'D':4, 'E': 5, 'F' : 6, 'G' : 7, 'H' : 8, 'I' : 9, 'J' : 10,\n             'K' : 11, 'L' : 12, 'M' : 13, 'N' : 14, 'O' : 15, 'P' : 16, 'Q' : 17, 'R' : 18, 'S' : 19,\n             'T' : 20, 'U' : 21, 'V' : 22, 'W': 23, 'X' : 24, 'Y' : 25, 'Z' : 26}\ntrain_dat['ord_4'] = pd.Series([ord_4_map[x] for x in train_dat.ord_4], index=train_dat.index)\ntest_dat['ord_4'] = test_dat['ord_4'].fillna(0)\ntest_dat['ord_4'] = pd.Series([ord_4_map[x] for x in test_dat.ord_4], index=test_dat.index)\n\ndef ord_alph(val):\n    val = val.lower()\n    if val == 'a':\n        return 1\n    if val == 'b':\n        return 2\n    if val == 'c':\n        return 3\n    if val == 'd':\n        return 4\n    if val == 'e':\n        return 5\n    if val == 'f':\n        return 6\n    if val == 'g':\n        return 7\n    if val == 'h':\n        return 8\n    if val == 'i':\n        return 9\n    if val == 'j':\n        return 10\n    if val == 'k':\n        return 11\n    if val == 'l':\n        return 12\n    if val == 'm':\n        return 13\n    if val == 'n':\n        return 14\n    if val == 'o':\n        return 15\n    if val == 'p':\n        return 16\n    if val == 'q':\n        return 17\n    if val == 'r':\n        return 18\n    if val == 's':\n        return 19\n    if val == 't':\n        return 20\n    if val == 'u':\n        return 21\n    if val == 'v':\n        return 22\n    if val == 'w':\n        return 23\n    if val == 'x':\n        return 24\n    if val == 'y':\n        return 25\n    if val == 'z':\n        return 26\n    else:\n        return 0\n\ndef ord_five_b(val):\n    try:\n        if val[0] == val[0].lower():\n            val_0 = ord_alph(val[0]) + 26\n        else:\n            val_0 = ord_alph(val[0])\n        if val[1] == val[1].lower():\n            val_1 = ord_alph(val[1]) + 26\n        else:\n            val_1 = ord_alph(val[1])\n        val_1 = val_1/100\n        return(val_0 + val_1)\n    except:\n        return np.nan\n\ntrain_dat['ord_5'] = pd.Series([ord_five_b(x) for x in train_dat.ord_5], index=train_dat.index)\ntest_dat['ord_5'] = pd.Series([ord_five_b(x) for x in test_dat.ord_5], index=test_dat.index)\n\ndat = train_dat.copy()\n# use these definitions when running this cell\ncat_names = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4', 'nom_0', 'nom_1', 'nom_2', \n             'nom_3', 'nom_4', 'nom_5', 'nom_6', 'nom_7', 'nom_8', 'nom_9']\ncont_names = ['ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4', 'ord_5', 'day', 'month']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH = Path('/kaggle/input/cat-in-the-dat-ii/')\ntest = TabularList.from_df(test_dat, path=PATH, cat_names=cat_names, cont_names=cont_names)\ndata = (TabularList.from_df(dat, path=PATH,\n                            cat_names=cat_names, \n                            cont_names=cont_names,\n                            procs=procs)\n                           .split_by_idx(valid_idx = range(len(dat)-50000, len(dat)))\n                           .label_from_df(cols=dep_var)\n                           .add_test(test)\n                           .databunch())\nlearn = tabular_learner(data, layers=[200,100], metrics=[accuracy, FBeta(average='weighted')], ps=0.15)\nModel_Path = Path('/kaggle/working/cat-in-dat/')\nlearn.model_dir = Model_Path","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit(1, lr=1e-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.save('label_encoded')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interestingly enough, the score for this model is actually lower than the vanilla FastAI model."},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = learn.get_preds(ds_type=DatasetType.Test)[0][:,1].numpy()\nsubmission_2 = pd.DataFrame({'id': out.index, 'target': preds})\nsubmission_2.to_csv('/kaggle/working/cat-in-dat/submission_2.csv', header=True, index=False)\nsubmission_2.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Label Encoding with iterative imputation\n\nThe iterative imputer is one of the newer features of SKlearn, which creates a model to predict missing values. Currently I'm using linear regression for the iteritive imputer, which produces fairly good results. I meant to try taking a smaller subset of the dataset to experiment with more complex models (if you try to use KNN or a random forest you will run out of memory before it completes), but ran out of time for this competition. "},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"# Resetting everything for round three\nlearn.destroy\ndat = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv', index_col='id')\ntest_dat = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv', index_col='id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_dat = dat.copy()\n\ntrain_dat['bin_3'] = train_dat['bin_3'].fillna(0) ## fills in the missing values for a column\nbin_3_map = {0 : np.nan, 'F': 1, 'T': 2} ## this dictionary will be used to transform the data to distinct integers\ntrain_dat['bin_3'] = pd.Series([bin_3_map[x] for x in train_dat.bin_3], index=train_dat.index) ## This replaces the original values\ntest_dat['bin_3'] = test_dat['bin_3'].fillna(0)\ntest_dat['bin_3'] = pd.Series([bin_3_map[x] for x in test_dat.bin_3], index=test_dat.index)\n\ntrain_dat['bin_4'] = train_dat['bin_4'].fillna(0)\nbin_4_map = {0 : np.nan, 'N': 1, 'Y': 2}\ntrain_dat['bin_4'] = pd.Series([bin_4_map[x] for x in train_dat.bin_4], index=train_dat.index)\ntest_dat['bin_4'] = test_dat['bin_4'].fillna(0)\ntest_dat['bin_4'] = pd.Series([bin_4_map[x] for x in test_dat.bin_4], index=test_dat.index)\n\ntrain_dat['nom_0'] = train_dat['nom_0'].fillna(0)\nnom_0_map = {0 : np.nan, 'Red': 1, 'Blue': 2, 'Green': 3}\ntrain_dat['nom_0'] = pd.Series([nom_0_map[x] for x in train_dat.nom_0], index=train_dat.index)\ntest_dat['nom_0'] = test_dat['nom_0'].fillna(0)\ntest_dat['nom_0'] = pd.Series([nom_0_map[x] for x in test_dat.nom_0], index=test_dat.index)\n\ntrain_dat['nom_1'] = train_dat['nom_1'].fillna(0)\nnom_1_map = {0 : np.nan, 'Circle': 1, 'Triangle': 2,'Square': 3 , 'Trapezoid' : 4, 'Star': 5, 'Polygon': 6}\ntrain_dat['nom_1'] = pd.Series([nom_1_map[x] for x in train_dat.nom_1], index=train_dat.index)\n\ntest_dat['nom_1'] = test_dat['nom_1'].fillna(0)\ntest_dat['nom_1'] = pd.Series([nom_1_map[x] for x in test_dat.nom_1], index=test_dat.index)\n\ntrain_dat['nom_2'] = train_dat['nom_2'].fillna(0)\nnom_2_map = {0 : np.nan, 'Hamster': 1, 'Axolotl': 2,'Lion': 3 , 'Dog' : 4, 'Cat': 5, 'Snake': 6}\ntrain_dat['nom_2'] = pd.Series([nom_2_map[x] for x in train_dat.nom_2], index=train_dat.index)\ntest_dat['nom_2'] = test_dat['nom_2'].fillna(0)\ntest_dat['nom_2'] = pd.Series([nom_2_map[x] for x in test_dat.nom_2], index=test_dat.index)\n\ntrain_dat['nom_3'] = train_dat['nom_3'].fillna(0)\nnom_3_map = {0 : np.nan, 'Finland': 1, 'Russia': 2,'Costa Rica': 3 , 'India' : 4, 'China': 5, 'Canada': 6}\ntrain_dat['nom_3'] = pd.Series([nom_3_map[x] for x in train_dat.nom_3], index=train_dat.index)\ntest_dat['nom_3'] = test_dat['nom_3'].fillna(0)\ntest_dat['nom_3'] = pd.Series([nom_3_map[x] for x in test_dat.nom_3], index=test_dat.index)\n\ntrain_dat['nom_4'] = train_dat['nom_4'].fillna(0)\nnom_4_map = {0 : np.nan, 'Piano': 1, 'Bassoon': 2,'Theremin': 3 , 'Oboe' : 4}\ntrain_dat['nom_4'] = pd.Series([nom_4_map[x] for x in train_dat.nom_4], index=train_dat.index)\ntest_dat['nom_4'] = test_dat['nom_4'].fillna(0)\ntest_dat['nom_4'] = pd.Series([nom_4_map[x] for x in test_dat.nom_4], index=test_dat.index)\n\ndef int_it(num, x):\n    try:\n        return int(num, x)\n    except:\n        return np.nan\n\ntrain_dat['nom_5'] = pd.Series([int_it(x,16) for x in train_dat.nom_5], index=train_dat.index)\ntest_dat['nom_5'] = pd.Series([int_it(x,16) for x in test_dat.nom_5], index=test_dat.index)\n\ntrain_dat['nom_6'] = pd.Series([int_it(x,16) for x in train_dat.nom_6], index=train_dat.index)\ntest_dat['nom_6'] = pd.Series([int_it(x,16) for x in test_dat.nom_6], index=test_dat.index)\n\ntrain_dat['nom_7'] = pd.Series([int_it(x,16) for x in train_dat.nom_7], index=train_dat.index)\ntest_dat['nom_7'] = pd.Series([int_it(x,16) for x in test_dat.nom_7], index=test_dat.index)\n\ntrain_dat['nom_8'] = pd.Series([int_it(x,16) for x in train_dat.nom_8], index=train_dat.index)\ntest_dat['nom_8'] = pd.Series([int_it(x,16) for x in test_dat.nom_8], index=test_dat.index)\n\ntrain_dat['nom_9'] = pd.Series([int_it(x,16) for x in train_dat.nom_9], index=train_dat.index)\ntest_dat['nom_9'] = pd.Series([int_it(x,16) for x in test_dat.nom_9], index=test_dat.index)\n\ntrain_dat['ord_1'] = train_dat['ord_1'].fillna(0)\nord_1_map = {0 : np.nan, 'Novice': 1, 'Contributor': 2,'Expert': 3 , 'Master' : 4, 'Grandmaster': 5}\ntrain_dat['ord_1'] = pd.Series([ord_1_map[x] for x in train_dat.ord_1], index=train_dat.index)\ntest_dat['ord_1'] = test_dat['ord_1'].fillna(0)\ntest_dat['ord_1'] = pd.Series([ord_1_map[x] for x in test_dat.ord_1], index=test_dat.index)\n\ntrain_dat['ord_2'] = train_dat['ord_2'].fillna(0)\nord_2_map = {0 : np.nan, 'Freezing': 1, 'Cold': 2,'Warm': 3 , 'Hot' : 4, 'Boiling Hot': 5, 'Lava Hot' : 6}\ntrain_dat['ord_2'] = pd.Series([ord_2_map[x] for x in train_dat.ord_2], index=train_dat.index)\ntest_dat['ord_2'] = test_dat['ord_2'].fillna(0)\ntest_dat['ord_2'] = pd.Series([ord_2_map[x] for x in test_dat.ord_2], index=test_dat.index)\n\ntrain_dat['ord_3'] = train_dat['ord_3'].fillna(0)\nord_3_map = {0:np.nan,'a':1, 'b':2,'c':3,'d':4, 'e': 5, 'f' : 6, 'g' : 7, 'h' : 8, 'i' : 9, 'j' : 10,\n             'k' : 11, 'l' : 12, 'm' : 13, 'n' : 14, 'o' : 15, 'p' : 16, 'q' : 17, 'r' : 18, 's' : 19,\n             't' : 20, 'u' : 21, 'v' : 22, 'w': 23, 'x' : 24, 'y' : 25, 'z' : 26}\ntrain_dat['ord_3'] = pd.Series([ord_3_map[x] for x in train_dat.ord_3], index=train_dat.index)\ntest_dat['ord_3'] = test_dat['ord_3'].fillna(0)\ntest_dat['ord_3'] = pd.Series([ord_3_map[x] for x in test_dat.ord_3], index=test_dat.index)\n\ntrain_dat['ord_4'] = train_dat['ord_4'].fillna(0)\nord_4_map = {0:np.nan,'A':1, 'B':2,'C':3,'D':4, 'E': 5, 'F' : 6, 'G' : 7, 'H' : 8, 'I' : 9, 'J' : 10,\n             'K' : 11, 'L' : 12, 'M' : 13, 'N' : 14, 'O' : 15, 'P' : 16, 'Q' : 17, 'R' : 18, 'S' : 19,\n             'T' : 20, 'U' : 21, 'V' : 22, 'W': 23, 'X' : 24, 'Y' : 25, 'Z' : 26}\ntrain_dat['ord_4'] = pd.Series([ord_4_map[x] for x in train_dat.ord_4], index=train_dat.index)\ntest_dat['ord_4'] = test_dat['ord_4'].fillna(0)\ntest_dat['ord_4'] = pd.Series([ord_4_map[x] for x in test_dat.ord_4], index=test_dat.index)\n\ndef ord_alph(val):\n    val = val.lower()\n    if val == 'a':\n        return 1\n    if val == 'b':\n        return 2\n    if val == 'c':\n        return 3\n    if val == 'd':\n        return 4\n    if val == 'e':\n        return 5\n    if val == 'f':\n        return 6\n    if val == 'g':\n        return 7\n    if val == 'h':\n        return 8\n    if val == 'i':\n        return 9\n    if val == 'j':\n        return 10\n    if val == 'k':\n        return 11\n    if val == 'l':\n        return 12\n    if val == 'm':\n        return 13\n    if val == 'n':\n        return 14\n    if val == 'o':\n        return 15\n    if val == 'p':\n        return 16\n    if val == 'q':\n        return 17\n    if val == 'r':\n        return 18\n    if val == 's':\n        return 19\n    if val == 't':\n        return 20\n    if val == 'u':\n        return 21\n    if val == 'v':\n        return 22\n    if val == 'w':\n        return 23\n    if val == 'x':\n        return 24\n    if val == 'y':\n        return 25\n    if val == 'z':\n        return 26\n    else:\n        return 0\n\ndef ord_five_b(val):\n    try:\n        if val[0] == val[0].lower():\n            val_0 = ord_alph(val[0]) + 26\n        else:\n            val_0 = ord_alph(val[0])\n        if val[1] == val[1].lower():\n            val_1 = ord_alph(val[1]) + 26\n        else:\n            val_1 = ord_alph(val[1])\n        val_1 = val_1/100\n        return(val_0 + val_1)\n    except:\n        return np.nan\n\ntrain_dat['ord_5'] = pd.Series([ord_five_b(x) for x in train_dat.ord_5], index=train_dat.index)\ntest_dat['ord_5'] = pd.Series([ord_five_b(x) for x in test_dat.ord_5], index=test_dat.index)\n\ndat = train_dat.copy()\n# use these definitions when running this cell\ncat_names = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4', 'nom_0', 'nom_1', 'nom_2', \n             'nom_3', 'nom_4', 'nom_5', 'nom_6', 'nom_7', 'nom_8', 'nom_9']\ncont_names = ['ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4', 'ord_5', 'day', 'month']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\nfrom sklearn.experimental import enable_iterative_imputer\nfrom sklearn.impute import IterativeImputer\ntd = dat.copy()\ntest_dat_copy = test_dat.copy()\nfeatures = list(td.columns)\nfeatures.remove('target')\nX = td[features]\nX_plus = pd.merge(X, test_dat_copy, how='outer')\niterimp = IterativeImputer(estimator=LinearRegression(),max_iter=300, add_indicator=False,tol=2e-4, random_state=0)#add_indicator=True,\niterimp.fit(X_plus)\ndat[features] = iterimp.transform(dat[features])#[features]\ntest_dat[features] = iterimp.transform(test_dat[features])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH = Path('/kaggle/input/cat-in-the-dat-ii/')\ntest = TabularList.from_df(test_dat, path=PATH, cat_names=cat_names, cont_names=cont_names)\ndata = (TabularList.from_df(dat, path=PATH,\n                            cat_names=cat_names, \n                            cont_names=cont_names,\n                            procs=procs)\n                           .split_by_idx(valid_idx = range(len(dat)-50000, len(dat)))\n                           .label_from_df(cols=dep_var)\n                           .add_test(test)\n                           .databunch())\nlearn = tabular_learner(data, layers=[200,100], metrics=[accuracy, FBeta(average='weighted')], ps=0.15)\nModel_Path = Path('/kaggle/working/cat-in-dat/')\nlearn.model_dir = Model_Path","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit(1, lr=1e-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.save('Iteritive Imputation')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Once again, this model will score below the vanilla FastAI model (but by a smaller margin)."},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = learn.get_preds(ds_type=DatasetType.Test)[0][:,1].numpy()\nsubmission_3 = pd.DataFrame({'id': out.index, 'target': preds})\nsubmission_3.to_csv('/kaggle/working/cat-in-dat/submission_3.csv', header=True, index=False)\nsubmission_3.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Blend the results"},{"metadata":{"trusted":true},"cell_type":"code","source":"final = out.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#need to test this section when time permits\nfinal['target_1'] = pd.Series([x for x in submission_1.target], index=out.index)\nfinal['target_2'] = pd.Series([x for x in submission_2.target], index=out.index)\nfinal['target_3'] = pd.Series([x for x in submission_3.target], index=out.index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final = final.drop('target', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final['target'] = final.mean(axis=1)\nfinal = final.drop('target_1', axis=1)\nfinal = final.drop('target_2', axis=1)\nfinal = final.drop('target_3', axis=1)\nfinal.index = out.index","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final.to_csv('/kaggle/working/final.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Final thoughts and observations\n\nInterestingly enough, taking the mean of these three models predictions will produce the highest score. I would guess this is caused by each model capturing different parts of the signal but still missing something. I wish I had more time to experiment with this before the deadline for the competition, but I'm about to leave for a camping trip and won't be back until after the deadline.\n\nfeel free to leave constructive feedback in the comment section or send me a message directly. If you found this notebook helpful please give it an upvote and possibly share it with a friend, if you didn't then share it with an enemy. "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}