{"cells":[{"metadata":{},"cell_type":"markdown","source":"![catboost](https://pbs.twimg.com/profile_images/894733787288014848/vX7FuXaA.jpg)\n\n# Introduction\n\nHello all, in this dataset we are given a bunch of categorical features like the previous one; except that this one has missing values. So we are going to approach the problem in 3 steps. First one is to fill these missing values,then we're going to encode them. Finally, we will fit a model. In this dataset, I chose to use Logistic Regression with regularization. We have binary, nominal and ordinal features. Lets begin."},{"metadata":{},"cell_type":"markdown","source":"## A look"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/cat-in-the-dat-ii/train.csv\")\ntest = pd.read_csv(\"../input/cat-in-the-dat-ii/test.csv\")\n\n# Subset\ntarget = train['target']\ntrain_id = train['id']\ntest_id = test['id']\ntrain.drop(['id'], axis=1, inplace=True)\ntest.drop('id', axis=1, inplace=True)\n\n#Null values\nnull_df = pd.DataFrame({'Percentile':train.isnull().sum()/len(train), 'Count':train.isnull().sum()})\nprint(null_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All variables has missing values around 3%."},{"metadata":{"trusted":true},"cell_type":"code","source":"my_df = pd.concat([train, test])\nmy_df.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preprocessing"},{"metadata":{},"cell_type":"markdown","source":"Start with binary ones."},{"metadata":{"trusted":true},"cell_type":"code","source":"my_df[\"bin_3\"] = my_df[\"bin_3\"].apply(lambda x: 1 if x=='T' else (0 if x=='F' else None))\nmy_df[\"bin_4\"] = my_df[\"bin_4\"].apply(lambda x: 1 if x=='Y' else (0 if x=='N' else None))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In nominal and ordinal ones, first we have to convert to string by .astype. But when we do this, NaN values are converted to string \"nan\". So we will write them None afterwards."},{"metadata":{"trusted":true},"cell_type":"code","source":"for enc in [\"nom_0\",\"nom_1\",\"nom_2\",\"nom_3\",\"nom_4\",\"day\",\"month\",\"ord_3\"]:#,\"ord_4\",\"ord_5\"]:\n    my_df[enc] = my_df[enc].astype(\"str\")\n    my_df[enc] = my_df[enc].apply(lambda x: x.lower())\n    my_df[enc] = my_df[enc].apply(lambda x: None if x=='nan' else x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for enc in [\"ord_4\",\"ord_5\"]:\n    my_df[enc] = my_df[enc].astype(\"str\")\n    my_df[enc] = my_df[enc].apply(lambda x: None if x=='nan' else x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"ord_1 and ord_2 can be encoded like the following. They include ordering such as Warm>Cold>Freezing etc."},{"metadata":{"trusted":true},"cell_type":"code","source":"my_df[\"ord_1\"] = my_df[\"ord_1\"].apply(lambda x: 1 if x=='Novice' else (2 if x=='Contributor' else (3 if x=='Expert' else (4 if x=='Master' else (5 if x=='Grandmaster' else None)))))\nmy_df[\"ord_2\"] = my_df[\"ord_2\"].apply(lambda x: 1 if x=='Freezing' else (2 if x=='Cold' else (3 if x=='Warm' else (4 if x=='Hot' else (5 if x=='Boiling Hot' else (6 if x=='Lava Hot' else None))))))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now here comes the important part. As Bojan mentioned [here](https://www.kaggle.com/tunguz/adversicat-ii) , the most important features are the weird ones, which are nom_5, nom_6, nom_7, nom_8 and nom_9. I filled the missing values of these with mode and used these features to impute the missing in other variables with similar values. The reason that I filled with mode is, when we use pandas' group by function, if one of the grouped variables are missing, then it returns NaN. So they have to be complete."},{"metadata":{"trusted":true},"cell_type":"code","source":"for col in [\"nom_5\",\"nom_6\",\"nom_7\",\"nom_8\",\"nom_9\"]:\n    mode = my_df[col].mode()[0]\n    my_df[col] = my_df[col].astype(str)\n    my_df[col] = my_df[col].apply(lambda x: mode if x=='nan' else x)\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Get the other variables list. And fill wtih similar values."},{"metadata":{"trusted":true},"cell_type":"code","source":"columns = list(my_df.columns)\nfor col in [\"target\",\"nom_5\",\"nom_6\",\"nom_7\",\"nom_8\",\"nom_9\"]:\n    columns.remove(col)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I used nom_7 to group only since it looks like the most important variable according to the feature importances from the link I mentioned above. Check this pic:\n\n![LGBM feature importances](https://www.kaggleusercontent.com/kf/26171048/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..eK-Q-tCnbzLMUn7Pf5ZIqg.RQSRdWSqNROLYjt92VnNsyVzkfweEVGtelMOl2IFGewy5ZuiPJIYKafjFFLzHL9azM6VxIrwqcp3r-Bu9XTzmVe4CB6Z7lYqVJPEk4PioBP5McLkwu9KN28tbB6ypiAKZ-3nY8JyYnQWprOmKj5kUg.bzUiCf8_7zXqNLnlwPJPQg/__results___files/__results___10_0.png)"},{"metadata":{},"cell_type":"markdown","source":"The reason we didn't use more variables to group by is, grouping with high-dimensional features is not efficient. Sometimes it creates bias and result in overfitting. This way we are kinda safe. After grouping, we will fill with the mode value of that group."},{"metadata":{"trusted":true},"cell_type":"code","source":"for col in columns:\n    my_df[col] = my_df.groupby([\"nom_7\"])[col].transform(lambda x: x.fillna(x.mode()[0]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now check the missing values."},{"metadata":{"trusted":true},"cell_type":"code","source":"my_df.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"No missing values left except for test set's target. Now, lets encode the remaining variables. I will convert ordinal ones with ordinal encoder. It will assign 1,2,3... according to alphabetic order of string variables. Note that ord_4 and ord_5 include both upper and lower case letters. They might include information. So lets use ordinal encoder on them too.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import OrdinalEncoder\noencoder = OrdinalEncoder(dtype=np.int16)\nfor enc in [\"ord_3\",\"ord_4\",\"ord_5\"]:\n    my_df[enc] = oencoder.fit_transform(np.array(my_df[enc]).reshape(-1,1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can get dummies for the rest of the categoric variables. For high dimensional ones, we will only create dummies for 2 of them. "},{"metadata":{"trusted":true},"cell_type":"code","source":"for category in [\"nom_5\",\"nom_6\",\"nom_7\",\"nom_8\",\"nom_9\"]:\n    print(\"{} has {} unique values\".format(category,len(np.unique(my_df[category]))))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"nom_7 and nom_8 has lower dimension. We can use pandas' dummy encoder on them. If we use it on all, it will increase the width of our train set too much and it will create bugs/errors on direct model fitting. This way still performs well. We will use another encoding for high-dim variables later."},{"metadata":{"trusted":true},"cell_type":"code","source":"for enc in [\"nom_0\",\"nom_1\",\"nom_2\",\"nom_3\",\"nom_4\",\"day\",\"month\",\"nom_7\",\"nom_8\"]:\n    enc1 = pd.get_dummies(my_df[enc], prefix=enc)\n    my_df.drop(columns=enc, inplace=True)\n    my_df = pd.concat([my_df,enc1], axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Scale the ordinal and binary variables (-1,1). This can help linear models."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import MinMaxScaler\n\nscaler = MinMaxScaler((-1,1))\nfor feat in [\"ord_0\",\"ord_1\",\"ord_2\",\"ord_3\",\"ord_4\",\"ord_5\"]:\n    my_df[feat] = scaler.fit_transform(np.array(my_df[feat]).reshape(-1,1))\n\n\nfor feat in [\"bin_0\",\"bin_1\",\"bin_2\",\"bin_3\",\"bin_4\"]:\n    my_df[feat] = scaler.fit_transform(np.array(my_df[feat]).reshape(-1,1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Create train and test."},{"metadata":{"trusted":true},"cell_type":"code","source":"test = my_df[my_df[\"target\"].isnull()]\ntest.drop(columns='target', inplace=True)\n\ntrain = my_df[my_df[\"target\"].isnull()==False]\ntarget = train[\"target\"]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Before dropping target column from train set, I will use LeaveOneEncoder on high dimensional nominal variables."},{"metadata":{"trusted":true},"cell_type":"code","source":"from category_encoders import  LeaveOneOutEncoder\nleaveOneOut_encoder = LeaveOneOutEncoder()\nfor nom in [\"nom_5\",\"nom_6\",\"nom_9\"]:\n    train[nom] = leaveOneOut_encoder.fit_transform(train[nom], train[\"target\"])\n    test[nom] = leaveOneOut_encoder.transform(test[nom])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now lets drop the target and reset indexes."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.drop(columns='target', inplace=True)\n\ntrain.reset_index(drop=True, inplace=True)\ntest.reset_index(drop=True, inplace=True)\ntarget.reset_index(drop=True, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Modelling"},{"metadata":{},"cell_type":"markdown","source":"Import the libraries we gonna need."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score as auc\nfrom sklearn.linear_model import LogisticRegression","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_cv_model(train, test, target, model_fn, params={}, eval_fn=None, label='model'):\n    kf = KFold(n_splits=5)\n    fold_splits = kf.split(train, target)\n    cv_scores = []\n    pred_full_test = 0\n    pred_train = np.zeros((train.shape[0]))\n    i = 1\n    for dev_index, val_index in fold_splits:\n        print('Started ' + label + ' fold ' + str(i) + '/5')\n        dev_X, val_X = train.loc[dev_index], train.loc[val_index]\n        dev_y, val_y = target.loc[dev_index], target.loc[val_index]\n        params2 = params.copy()\n        pred_val_y, pred_test_y = model_fn(dev_X, dev_y, val_X, val_y, test, params2)\n        pred_full_test = pred_full_test + pred_test_y\n        pred_train[val_index] = pred_val_y\n        if eval_fn is not None:\n            cv_score = eval_fn(val_y, pred_val_y)\n            cv_scores.append(cv_score)\n            print(label + ' cv score {}: {}'.format(i, cv_score))\n        i += 1\n    print('{} cv scores : {}'.format(label, cv_scores))\n    print('{} cv mean score : {}'.format(label, np.mean(cv_scores)))\n    print('{} cv std score : {}'.format(label, np.std(cv_scores)))\n    pred_full_test = pred_full_test / 5.0\n    results = {'label': label,\n              'train': pred_train, 'test': pred_full_test,\n              'cv': cv_scores}\n    return results\n\n\n\ndef runLR(train_X, train_y, test_X, test_y, test_X2, params):\n    print('Train LR')\n    model = LogisticRegression(**params)\n    model.fit(train_X, train_y)\n    print('Predict 1/2')\n    pred_test_y = model.predict_proba(test_X)[:, 1]\n    print('Predict 2/2')\n    pred_test_y2 = model.predict_proba(test_X2)[:, 1]\n    return pred_test_y, pred_test_y2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Set inverse regularization parameter C=0.1 and initialize."},{"metadata":{"trusted":true},"cell_type":"code","source":"lr_params = {'solver': 'lbfgs', 'C':0.1, 'max_iter':500}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results = run_cv_model(train, test, target, runLR, lr_params, auc, 'lr')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our mean CV value is 78,5. For this much feature engineering, it is sufficient I suppose :) Let's submit the results."},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame({'id': test_id, 'target': results['test']})\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}