{"cells":[{"metadata":{"id":"ZrxLwK16As9_","colab_type":"text"},"cell_type":"markdown","source":"![](https://cdn-images-1.medium.com/max/1200/1*x7P7gqjo8k2_bj2rTQWAfg.jpeg)\n\n# Welcome to my kernel!\n- This is my 'naive' approach for the [Categorical Feature Encoding Challenge II](https://www.kaggle.com/c/cat-in-the-dat-ii) and also one of my first steps in Kaggle platform.\n- This notebook has been implemented during my day-off because of the [Coronavirus](https://www.kaggle.com/sudalairajkumar/novel-corona-virus-2019-dataset) spread (2019-nCoV) \n- And as usual, if my work can make you feel excited, help me to <font color='red' size=3>upvote this kernel </font>on the right corner 💖💖\n\n\nP/s: I come from Vietnam, so please ignore my English grammar mistakes through out this notebook 😊😊\n"},{"metadata":{"id":"wCIe--osE79n","colab_type":"text"},"cell_type":"markdown","source":"## My work\n- Have some feature distribution observations and give several conclusions\n- Take some simple feature engineering\n- Fine-tune a Logistic Regression\n"},{"metadata":{"id":"8d75DdyYEJzJ","colab_type":"text"},"cell_type":"markdown","source":"# Preparation\n\nImport necessary libraries "},{"metadata":{"id":"ALSXksI4jr3a","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.impute import SimpleImputer\nimport scipy\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import auc, roc_curve\nfrom sklearn.model_selection import StratifiedKFold, GridSearchCV\nfrom tqdm import tqdm_notebook","execution_count":null,"outputs":[]},{"metadata":{"id":"tkLNgc7AFxL4","colab_type":"text"},"cell_type":"markdown","source":"# Data\nLet's load the data and take some observations"},{"metadata":{"id":"8bUNAOmE6orZ","colab_type":"code","outputId":"5886391e-e585-4f8d-aa7e-b0e66ba2f0d4","executionInfo":{"status":"ok","timestamp":1581079489634,"user_tz":-420,"elapsed":8709,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":34},"trusted":true},"cell_type":"code","source":"raw_train = pd.read_csv('../input/cat-in-the-dat-ii/train.csv', index_col='id')\nraw_test = pd.read_csv('../input/cat-in-the-dat-ii/test.csv', index_col='id')\n\nprint(raw_train.shape, raw_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"id":"QlOHi7xljtSF","colab_type":"code","outputId":"3afd1518-f799-4835-d4f0-665fa7353984","executionInfo":{"status":"ok","timestamp":1581079489635,"user_tz":-420,"elapsed":8582,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":295},"trusted":true},"cell_type":"code","source":"raw_test.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"fnBGOaTtjwqg","colab_type":"code","outputId":"7f697da4-5042-4da1-c40b-02877afb24a6","executionInfo":{"status":"ok","timestamp":1581079489636,"user_tz":-420,"elapsed":8456,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":278},"trusted":true},"cell_type":"code","source":"raw_train.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"KHJ6Bwr-GCPS","colab_type":"text"},"cell_type":"markdown","source":"# Explore Data Analysis"},{"metadata":{"id":"tVLI2MABFpk4","colab_type":"text"},"cell_type":"markdown","source":"## Missing value"},{"metadata":{"colab_type":"code","id":"rjjcD3WQAkU7","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_missing_values(df):\n\n    cols = df.columns\n    count = [df[col].isnull().sum() for col in cols]\n    percent = [i/len(df) for i in count]\n    missing = pd.DataFrame({'number':count, 'proportion': percent}, index=cols)\n    \n    fig, ax = plt.subplots(1,2, figsize=(20,7))\n    for i, col in enumerate(missing.columns):\n\n        plt.subplot(1,2,i+1)\n        plt.title(f'Missing values on each columns({col})')\n        sns.barplot(missing[col], missing.index)\n        mean = np.mean(missing[col])\n        std = np.std(missing[col])\n        plt.ylabel('Columns')\n        plt.plot([], [], ' ', label=f'Average {col} of missing values: {mean:.2f} \\u00B1 {std:.2f}')\n        plt.legend()\n    plt.show()\n    return missing.sort_values(by='number', ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"id":"bqZZUax-0ZZQ","colab_type":"code","outputId":"f7868393-b1b8-4193-b3b6-316546bbd773","executionInfo":{"status":"ok","timestamp":1581079490665,"user_tz":-420,"elapsed":9011,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":657},"trusted":true},"cell_type":"code","source":"missing_train = plot_missing_values(raw_train)\nmissing_train.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"j71pnl-s0Zbl","colab_type":"code","outputId":"c3b31469-0d7f-4717-dcc7-8f6db356c1cd","executionInfo":{"status":"ok","timestamp":1581079491717,"user_tz":-420,"elapsed":9930,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":657},"trusted":true},"cell_type":"code","source":"missing_test = plot_missing_values(raw_test)\nmissing_test.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"fzulmj8lvTws","colab_type":"text"},"cell_type":"markdown","source":"There are misssing value on almost all columns in both train and test set. However this figure accounts for a small proportion"},{"metadata":{"id":"eFqjUpgSGHwl","colab_type":"text"},"cell_type":"markdown","source":"## Target distribution"},{"metadata":{"id":"IhDNVqNuj3NG","colab_type":"code","outputId":"d0b70d15-23f8-4d3c-87c1-658eed7382df","executionInfo":{"status":"ok","timestamp":1581079491718,"user_tz":-420,"elapsed":9524,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":388},"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(6,6))\nax = sns.countplot(raw_train.target)\n\nheight = sum([p.get_height() for p in ax.patches])\nfor p in ax.patches:\n        ax.annotate(f'{100*p.get_height()/height:.2f} %', (p.get_x()+0.3, p.get_height()+5000),animated=True)","execution_count":null,"outputs":[]},{"metadata":{"id":"D9EZBatJv3DU","colab_type":"text"},"cell_type":"markdown","source":"One can easily find that, this compettition is an imbalanced binary classification task with 81.28% of 0's and 18.72% of 1's."},{"metadata":{"id":"p443VeXlGv9B","colab_type":"text"},"cell_type":"markdown","source":"## Heatmap\nLet's plot now the train data (all the data) using a heatmap"},{"metadata":{"id":"13zCcw0R0Im_","colab_type":"code","outputId":"c6f78399-3ce1-4961-df43-b8b92b59ba3d","executionInfo":{"status":"ok","timestamp":1581079492724,"user_tz":-420,"elapsed":10140,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":450},"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,7))\nnum_cols = raw_train.select_dtypes(exclude=['object']).columns\ncorr = raw_train[num_cols].corr()\nsns.heatmap(corr, \n            xticklabels=corr.columns.values,\n            yticklabels=corr.columns.values)","execution_count":null,"outputs":[]},{"metadata":{"id":"o1KOZQYtz3vZ","colab_type":"text"},"cell_type":"markdown","source":"There're no special correlation between these features, except from the pairs of ( ord_0 - target) and ( month - target)"},{"metadata":{"id":"Ja_CZ-2RG4VW","colab_type":"text"},"cell_type":"markdown","source":"Take a deeper look on different target samples, I will plot separate heatmap for target values 0 and 1."},{"metadata":{"id":"mfZUTuhT0tpu","colab_type":"code","outputId":"71fbf90b-fa28-450d-e728-220220fdc019","executionInfo":{"status":"ok","timestamp":1581079492725,"user_tz":-420,"elapsed":9710,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":450},"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,7))\n\ncorr_0 = raw_train[num_cols][raw_train.target==0].corr()\nsns.heatmap(corr_0, \n            xticklabels=corr_0.columns.values,\n            yticklabels=corr_0.columns.values)","execution_count":null,"outputs":[]},{"metadata":{"id":"Zyh8Cext1X-O","colab_type":"code","outputId":"af0bcb72-ff0e-4014-d38e-49e2c51717db","executionInfo":{"status":"ok","timestamp":1581079492725,"user_tz":-420,"elapsed":9537,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":450},"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,7))\n\ncorr_1 = raw_train[num_cols][raw_train.target==1].corr()\nsns.heatmap(corr_1, \n            xticklabels=corr_1.columns.values,\n            yticklabels=corr_1.columns.values)","execution_count":null,"outputs":[]},{"metadata":{"id":"jav5KpT_1CT7","colab_type":"text"},"cell_type":"markdown","source":"We get nothing special in these maps"},{"metadata":{"id":"Qa_pedpEHDto","colab_type":"text"},"cell_type":"markdown","source":"## Density plot of numeric features\n\nNow, I want to investigate numeric features first, using seaborn's distribution plot module"},{"metadata":{"id":"SaMr5jJxqwC5","colab_type":"code","outputId":"5b020102-bad7-423a-eb41-f0bbb6f9915f","executionInfo":{"status":"ok","timestamp":1581079493944,"user_tz":-420,"elapsed":10355,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":461},"trusted":true},"cell_type":"code","source":"num_cols = raw_test.select_dtypes(exclude=['object']).columns\nfig, ax = plt.subplots(2,3,figsize=(22,7))\nfor i, col in enumerate(num_cols):\n    plt.subplot(2,3,i+1)\n    plt.xlabel(col, fontsize=9)\n    sns.kdeplot(raw_train[col].values, bw=0.5,label='Train')\n    sns.kdeplot(raw_test[col].values, bw=0.5,label='Test')\n   \nplt.show() ","execution_count":null,"outputs":[]},{"metadata":{"id":"cNwIpuYl13cP","colab_type":"text"},"cell_type":"markdown","source":"Wow, the patterns in both train and test set are almost the same in 5 numeric features, which mean that if I have a good enough model ( good performance on the training set) it seem not to find difficulty in generalization ( i.e archives relatively good performance in the test set)\n\n> The train and test data are generated by a probability distribution over datasets called the data generating process. We typically make a set of assumptions known collectively as the i.i.d. assumptions. These assumptions are that the examples in each dataset are independent from each other, and that the train set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption allows us to describe the data generating  process with a probability distribution over a single example. The same\ndistribution is then used to generate every train example and every test example ([Ian Goodfellow](http://www.deeplearningbook.org/))"},{"metadata":{"id":"MmSQDcTvHKs0","colab_type":"text"},"cell_type":"markdown","source":"We will have greater insight with others plot for separate distribution visualization for samples with target value 0 and 1."},{"metadata":{"id":"9lIoRxpVmzlp","colab_type":"code","outputId":"64e3e5e8-3ddc-4d15-de50-daf53c1aa225","executionInfo":{"status":"ok","timestamp":1581079496186,"user_tz":-420,"elapsed":11864,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":463},"trusted":true},"cell_type":"code","source":"target0 = raw_train.loc[raw_train['target'] == 0]\ntarget1 = raw_train.loc[raw_train['target'] == 1]\n\nfig, ax = plt.subplots(2,3,figsize=(22,7))\nfor i, col in enumerate(num_cols):\n    plt.subplot(2,3,i+1)\n    plt.xlabel(col, fontsize=9)\n    sns.kdeplot(target0[col].values, bw=0.5,label='Target: 0')\n    sns.kdeplot(target1[col].values, bw=0.5,label='Target: 1')\n    sns.kdeplot(raw_test[col].values, bw=0.5,label='Test')\n    \nplt.show() ","execution_count":null,"outputs":[]},{"metadata":{"id":"Q__ACnrb6UsJ","colab_type":"text"},"cell_type":"markdown","source":"These 3 patterns are almost the same, except from the case of ord_1 and month features, the 1-target class has a slightly different distribution. That means, similar to the train data set, there will be an overwhelming number of 1-target samples compare to the rest"},{"metadata":{"id":"JkkRCWDAHsuS","colab_type":"text"},"cell_type":"markdown","source":"## Binary features\nLet take some small visualizations with these bool features"},{"metadata":{"id":"0EXVCjOf0Zdp","colab_type":"code","outputId":"891bb997-f2fc-463c-f975-90252c3438e9","executionInfo":{"status":"ok","timestamp":1581079496186,"user_tz":-420,"elapsed":11380,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":413},"trusted":true},"cell_type":"code","source":"bin_cols = [f'bin_{i}' for i in range(5)]\n\nfig, ax = plt.subplots(1,5, figsize=(22, 5))\n\nfor i, col in enumerate(bin_cols):\n     ax0 = plt.subplot(1,5,i+1)\n     raw_train[col].value_counts().plot.bar(color='pink')\n     height = sum([p.get_height() for p in ax0.patches])\n\n     for p in ax0.patches:\n         ax0.text(p.get_x()+p.get_width()/2., p.get_height()+4000, f'{100*p.get_height()/height:.2f} %', ha='center')\n     plt.xlabel(f'{col}')\nplt.suptitle('Distribution over binary feature of train data')\n","execution_count":null,"outputs":[]},{"metadata":{"id":"G5c4EYPDXC3g","colab_type":"code","outputId":"9f8c5879-a520-4c3a-a8ca-3080eac3f746","executionInfo":{"status":"ok","timestamp":1581079497543,"user_tz":-420,"elapsed":12578,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":413},"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,5, figsize=(22, 5))\n\nfor i, col in enumerate(bin_cols):\n     ax0 = plt.subplot(1,5,i+1)\n     raw_test[col].value_counts().plot.bar(color='lime')\n     height = sum([p.get_height() for p in ax0.patches])\n\n     for p in ax0.patches:\n         ax0.text(p.get_x()+p.get_width()/2., p.get_height()+4000, f'{100*p.get_height()/height:.2f} %', ha='center')\n     plt.xlabel(f'{col}')\nplt.suptitle('Distribution over binary feature of test data')","execution_count":null,"outputs":[]},{"metadata":{"id":"Q-34mTTpFouY","colab_type":"text"},"cell_type":"markdown","source":"The pattern of binary features is very similar in both train and test set, these figures are extremely close, compare to the [Categorical Feature Encoding Challenge I](https://www.kaggle.com/c/cat-in-the-dat/) dataset (in which there still a gap between the train and test set) "},{"metadata":{"id":"cwr-WXaDHw75","colab_type":"text"},"cell_type":"markdown","source":"## Time features\n\nTime features are forgotten in many machine learnning compettions. But in my personal view, time series are extremely important in statistics and  pattern recognition because many prediction problems that involve a time component\n![](https://miro.medium.com/max/343/1*70cevmU8wNggGJEdLam1lw.png)"},{"metadata":{"id":"ENSI2fqFAV2X","colab_type":"text"},"cell_type":"markdown","source":"First, consider these figures as categorical features. I will plot some simple bar charts using seaborn's countplot"},{"metadata":{"id":"yin33gAbmn1X","colab_type":"code","outputId":"66154641-5f2c-49a6-defc-7d8af0c46445","executionInfo":{"status":"ok","timestamp":1581079497544,"user_tz":-420,"elapsed":11984,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":424},"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(22,6))\nplt.title('Day distribution')\nax = sns.countplot(raw_train.day, hue=raw_train.target)\nfor p in ax.patches:\n    ax.text(p.get_x()+p.get_width()/2., p.get_height()+1000, f'{100*p.get_height()/height:.2f} %',ha='center')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"kwd9X_qxXC71","colab_type":"code","outputId":"a4af4671-312d-41fc-abe7-7978a56ab4b1","executionInfo":{"status":"ok","timestamp":1581084904823,"user_tz":-420,"elapsed":2234,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":424},"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(22,6))\nplt.title('Month distribution')\nax = sns.countplot(raw_train.month, hue=raw_train.target)\nfor p in ax.patches:\n    ax.text(p.get_x()+p.get_width()/2., p.get_height()+1000, f'{100*p.get_height()/height:.2f} %', ha='center')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"LG6jUGFV96l8","colab_type":"text"},"cell_type":"markdown","source":"It is more likely a sample will has the True target if it happens in August (16.54 %), March (15.43 %) and get the False target in December (3.98 %)"},{"metadata":{"id":"6IL0VRlYdh8D","colab_type":"text"},"cell_type":"markdown","source":"But we only see the affections of time seies data when it is plotted via line charts\n\n"},{"metadata":{"id":"4LPn7MNtBXaD","colab_type":"text"},"cell_type":"markdown","source":"### Plot trend"},{"metadata":{"id":"U8x_i-hjEFsK","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"df_train = raw_train.dropna(subset=['month', 'day'])[['day', 'month', 'target']]\ndf_test = raw_test.dropna(subset=['month', 'day'])[['day', 'month']]\ndf0 = df_train[df_train.target == 0]\ndf1 = df_train[df_train.target == 1]\n\ndef number2datetime(df):\n    time_col = '2019/' + df.month.astype(int).astype(str) + '/' + df.day.astype(int).astype(str)\n    df['time'] = pd.to_datetime(time_col , format = '%Y/%m/%d')\n    df = df.drop(columns=['day', 'month'])\n    return df\ndf0 = number2datetime(df0)\ndf1 = number2datetime(df1)\ndf_test = number2datetime(df_test)","execution_count":null,"outputs":[]},{"metadata":{"id":"J3H1W4lMVc8S","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"count0 = df0.time.value_counts()/len(df0)\ncount0 = count0.sort_index()\ncount1 = df1.time.value_counts()/len(df1)\ncount1 = count1.sort_index()\ncount_test = df_test.time.value_counts(normalize=True)","execution_count":null,"outputs":[]},{"metadata":{"id":"GKpR2DmmXGhW","colab_type":"code","outputId":"49e8c008-6f3b-4674-d842-6a22e0baf898","executionInfo":{"status":"ok","timestamp":1581079500673,"user_tz":-420,"elapsed":14252,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":520},"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,8))\nsns.lineplot(count0.index, count0.values, label='Target:0')\nsns.lineplot(count1.index, count1.values, label='Target:1')\nsns.lineplot(count_test.index, count_test.values, label='Test')\nplt.legend(loc='upper left')","execution_count":null,"outputs":[]},{"metadata":{"id":"pUBxqvLQBa3K","colab_type":"text"},"cell_type":"markdown","source":"As being shown on the heatmap and numeric features density plots, the test set and the 0-target class in the train data set have the similar patterns whereas the 1-target samples are generated by a little bit different distribution"},{"metadata":{"id":"jqnZd6TzI4UQ","colab_type":"text"},"cell_type":"markdown","source":"## Nominal features\nI will divide the nominal features into 2 groups: low- (less than 10 different values) and high-cardinality nominal features (the rest)\n"},{"metadata":{"id":"wpgspiSjJBwg","colab_type":"text"},"cell_type":"markdown","source":"### Low cardinality features"},{"metadata":{"id":"DXMipYlrDSfx","colab_type":"text"},"cell_type":"markdown","source":"#### Bar charts"},{"metadata":{"id":"4j-3fJufrYmi","colab_type":"code","outputId":"4946bb7f-70d1-4771-8b95-ae2c84aa775f","executionInfo":{"status":"ok","timestamp":1581079502256,"user_tz":-420,"elapsed":15129,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":409},"trusted":true},"cell_type":"code","source":"nom_cols = [f'nom_{i}' for i in range(5)]\nfig, ax = plt.subplots(1,5, figsize=(22, 6))\nfor i, col in enumerate(nom_cols):\n    plt.subplot(1,5,i+1)\n    sns.countplot(f'nom_{i}', hue='target', data= raw_train)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"3qMB3OPr4sC9","colab_type":"code","outputId":"24e82844-a87d-4f0d-dcf3-4370c70c5017","executionInfo":{"status":"ok","timestamp":1581079504526,"user_tz":-420,"elapsed":17230,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":669},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(17, 35)) \nfig, ax = plt.subplots(2,3,figsize=(22,10))\n\nfor i, col in enumerate(raw_train[nom_cols]): \n    tmp = pd.crosstab(raw_train[col], raw_train['target'], normalize='index') * 100\n    tmp = tmp.reset_index()\n    tmp.rename(columns={0:'No',1:'Yes'}, inplace=True)\n\n    ax = plt.subplot(2,3,i+1)\n    sns.countplot(x=col, data=raw_train, order=list(tmp[col].values) , palette='Set2') \n    ax.set_ylabel('Count', fontsize=15) # y axis label\n    ax.set_title(f'{col} Distribution by Target', fontsize=18) # title label\n    ax.set_xlabel(f'{col} values', fontsize=15) # x axis label\n\n    # twinX - to build a second yaxis\n    gt = ax.twinx()\n    gt = sns.pointplot(x=col, y='Yes', data=tmp,\n                           order=list(tmp[col].values),\n                           color='black', legend=False)\n    gt.set_ylim(tmp['Yes'].min()-5,tmp['Yes'].max()*1.1)\n    gt.set_ylabel(\"Target %True(1)\", fontsize=16)\n    sizes=[] # Get highest values in y\n    total = sum([p.get_height() for p in ax.patches])\n    for p in ax.patches: # loop to all objects\n        height = p.get_height()\n        sizes.append(height)\n        ax.text(p.get_x()+p.get_width()/2.,\n                    height + 2000,\n                    '{:1.2f}%'.format(height/total*100),\n                    ha=\"center\") \n    ax.set_ylim(0, max(sizes) * 1.15) # set y limit based on highest heights\n\n\nplt.subplots_adjust(hspace = 0.5, wspace=.3)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"QqpX46x2HP3Z","colab_type":"text"},"cell_type":"markdown","source":" Beautiful charts and provide useful information for me, too. Let get some:\n\n- nom_0 - Blue (~20%) value have the highest % of positive values in the target\n- nom_1 - Trapezoid (~23%) value have the highest % of positive values in the target\n- nom_2 - Lion (~21%) value have the highest % of positive values in the target\n- nom_3 - Russia (~22%) value have the highest % of positive values in the target\n- nom_4 - Bassoon (~21%) value have the highest % of positive values in the target\n\nAny special recognition?\nYes, all the values with highest % of True values on target, are not category's with highest frequency on the nominal category's\nThis is the same property between this and the previous [Cat in the Dat](https://www.kaggle.com/c/cat-in-the-dat/overview) competition"},{"metadata":{"id":"SnTTK6oDJOtf","colab_type":"text"},"cell_type":"markdown","source":"Now, let plot them seperately"},{"metadata":{"id":"00mC2Sf1LPjw","colab_type":"text"},"cell_type":"markdown","source":"#### Pie charts"},{"metadata":{"id":"MCmc8KYG3ybw","colab_type":"code","outputId":"1003fe24-0ed4-4b24-9002-40f52737f8e9","executionInfo":{"status":"ok","timestamp":1581079508116,"user_tz":-420,"elapsed":20266,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":1000},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"for col in nom_cols:\n    fig, ax = plt.subplots(1,3,figsize=(22,6))\n    ax[0].set_title(f'Target 0 data {col}')\n    ax[1].set_title(f'Target 1 data {col}')\n    ax[2].set_title(f'Test data {col}')\n\n    explode = np.zeros(raw_train[col].nunique()+1)\n    explode[1] = 0.05   \n    target0_count = target0[col].value_counts(dropna=False)\n    target1_count = target1[col].value_counts(dropna=False)    \n    test_count = raw_test[col].value_counts(dropna=False)\n\n    ax[0].pie(target0_count, labels=target0_count.index, autopct='%1.1f%%', explode=explode, shadow=True, startangle=90)\n    ax[0].legend(labels=target0_count.index,loc=3)\n    ax[1].pie(target1_count, labels=target1_count.index, autopct='%1.1f%%', explode=explode, shadow=True, startangle=90)\n    ax[1].legend(labels=target1_count.index,loc=3)    \n    ax[2].pie(test_count, labels=test_count.index, autopct='%1.1f%%', explode=explode, shadow=False, startangle=90)\n    ax[2].legend(labels=test_count.index,loc=3)","execution_count":null,"outputs":[]},{"metadata":{"id":"Ag5shoDOJpHA","colab_type":"text"},"cell_type":"markdown","source":"As usual, the patterns of 0 target class and test set are similar whereas the 1 class samples's distribution are slightly unalike "},{"metadata":{"id":"7KQUu4fAISbu","colab_type":"text"},"cell_type":"markdown","source":"### High cardinality features\n"},{"metadata":{"id":"kCcqARdf3qhg","colab_type":"code","outputId":"62547c2a-4588-4835-be08-538284456010","executionInfo":{"status":"ok","timestamp":1581079562930,"user_tz":-420,"elapsed":74649,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":1000},"trusted":true},"cell_type":"code","source":"nom_cols = [f'nom_{i}' for i in range(5,10)]\nfig, ax = plt.subplots(5,1, figsize=(22,17))\nfor i,col in enumerate(nom_cols):\n    plt.subplot(5,1,i+1)\n    sns.countplot(raw_train[col])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"mtOu5UA0DyJl","colab_type":"text"},"cell_type":"markdown","source":"It is impossible for any conclusion when plotting high cardinality features, but I still leave them there, just because they are colorful and impressive (at least for me 😊😊😊)\n"},{"metadata":{"id":"DrZJU67rJXdo","colab_type":"text"},"cell_type":"markdown","source":"## Ordinal features\nAgain, I will divide them into groups"},{"metadata":{"id":"0N4BWNLAJg7K","colab_type":"text"},"cell_type":"markdown","source":"### Low cardinality features"},{"metadata":{"id":"Vc-YVd9E9wbc","colab_type":"code","outputId":"497746e3-8cb7-4e46-e3bc-4c4d73cd952f","executionInfo":{"status":"ok","timestamp":1581080204841,"user_tz":-420,"elapsed":4612,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":921},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"ord_cols = [f'ord_{i}' for i in range(3)]\nplt.figure(figsize=(17, 35)) \nfig, ax = plt.subplots(3,1,figsize=(15,15))\n\nfor i, col in enumerate(raw_train[ord_cols]): \n    tmp = pd.crosstab(raw_train[col], raw_train['target'], normalize='index') * 100\n    tmp = tmp.reset_index()\n    tmp.rename(columns={0:'No',1:'Yes'}, inplace=True)\n\n    ax = plt.subplot(3,1,i+1)\n    sns.countplot(x=col, data=raw_train, order=list(tmp[col].values) , palette='Set2') \n    ax.set_ylabel('Count', fontsize=15) # y axis label\n    ax.set_title(f'{col} Distribution by Target', fontsize=18) # title label\n    ax.set_xlabel(f'{col} values', fontsize=15) # x axis label\n\n    gt = ax.twinx()\n    gt = sns.pointplot(x=col, y='Yes', data=tmp,\n                           order=list(tmp[col].values),\n                           color='black', legend=False)\n    gt.set_ylim(tmp['Yes'].min()-5,tmp['Yes'].max()*1.1)\n    gt.set_ylabel(\"Target %True(1)\", fontsize=16)\n    sizes=[] # Get highest values in y\n    total = sum([p.get_height() for p in ax.patches])\n    for p in ax.patches: # loop to all objects\n        height = p.get_height()\n        sizes.append(height)\n        ax.text(p.get_x()+p.get_width()/2.,\n                    height + 2000,\n                    '{:1.2f}%'.format(height/total*100),\n                    ha=\"center\") \n    ax.set_ylim(0, max(sizes) * 1.15) # set y limit based on highest heights\n\nplt.subplots_adjust(hspace = 0.5, wspace=.3)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"4dzZOeSo7O8e","colab_type":"text"},"cell_type":"markdown","source":"Grand Master accounts for a small proportion(16,47 %) but has the highest percentage of positive values in the target\nMaybe data was taken from another Kaggle compettion 😍???\n"},{"metadata":{"id":"vyXzo0FmK-Hy","colab_type":"text"},"cell_type":"markdown","source":"### High cardinality features"},{"metadata":{"id":"nYlhQPLy_Vez","colab_type":"code","outputId":"658e76df-8651-44e0-d585-716cffea94b7","executionInfo":{"status":"ok","timestamp":1581080225061,"user_tz":-420,"elapsed":2741,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":669},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"ord_cols = ['ord_3', 'ord_4']\nplt.figure(figsize=(17, 35)) \nfig, ax = plt.subplots(2,1,figsize=(22,10))\n\nfor i, col in enumerate(raw_train[ord_cols]): \n    tmp = pd.crosstab(raw_train[col], raw_train['target'], normalize='index') * 100\n    tmp = tmp.reset_index()\n    tmp.rename(columns={0:'No',1:'Yes'}, inplace=True)\n\n    ax = plt.subplot(2,1,i+1)\n    sns.countplot(x=col, data=raw_train, order=list(tmp[col].values) , palette='Set2') \n    ax.set_ylabel('Count', fontsize=15) # y axis label\n    ax.set_title(f'{col} Distribution by Target', fontsize=18) # title label\n    ax.set_xlabel(f'{col} values', fontsize=15) # x axis label\n\n    # twinX - to build a second yaxis\n    gt = ax.twinx()\n    gt = sns.pointplot(x=col, y='Yes', data=tmp,\n                           order=list(tmp[col].values),\n                           color='black', legend=False)\n    gt.set_ylim(tmp['Yes'].min()-5,tmp['Yes'].max()*1.1)\n    gt.set_ylabel(\"Target %True(1)\", fontsize=16)\n    sizes=[] # Get highest values in y\n    total = sum([p.get_height() for p in ax.patches])\n    for p in ax.patches: # loop to all objects\n        height = p.get_height()\n        sizes.append(height)\n        ax.text(p.get_x()+p.get_width()/2.,\n                    height + 2000,\n                    '{:1.2f}%'.format(height/total*100),\n                    ha=\"center\") \n    ax.set_ylim(0, max(sizes) * 1.15) # set y limit based on highest heights\n\n\nplt.subplots_adjust(hspace = 0.5, wspace=.3)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"7zeZvBkjKVhC","colab_type":"text"},"cell_type":"markdown","source":"Great, the frequency are varied  but look at the True values on target proportion, this line graph is covariates (i.e we can see a clearly linear relationship) which means the higher in character order, the more likely they will appear in the 1 target class"},{"metadata":{"id":"Spj_KwZgKIn-","colab_type":"code","outputId":"5cc59ac2-b867-4435-df2e-8f2fc04b2b0b","executionInfo":{"status":"ok","timestamp":1581079567290,"user_tz":-420,"elapsed":77401,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":489},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"tmp = pd.crosstab(raw_train['ord_5'], raw_train['target'], normalize='index') * 100\ntmp = tmp.reset_index()\ntmp.rename(columns={0:'No',1:'Yes'}, inplace=True)\nplt.figure(figsize=(18,7))\n\nplt.subplot()\nax = sns.countplot(x='ord_5', data=raw_train, order=list(tmp['ord_5'].values) , color='chocolate') \nax.set_ylabel('Count', fontsize=17) # y axis label\nax.set_title('ord_5 Distribution', fontsize=20) # title label\nax.set_xlabel('ord_5 values', fontsize=17) # x axis label","execution_count":null,"outputs":[]},{"metadata":{"id":"fv9eM_9tD_7t","colab_type":"text"},"cell_type":"markdown","source":"That is enough for me to explore and understand the data. If you have any others useful pattern recognition, please leave some comments at the end of my kernel\n\nP/s: There charts make me feel like they come from a IELTS writing test 😁😁😁"},{"metadata":{"id":"AB099bLuefKz","colab_type":"text"},"cell_type":"markdown","source":"# Feature Engineer\nI will try to extract information from these features which hopefully can be used to improve the performance of my Logistic Regression model"},{"metadata":{"id":"9DL9A_HqKrL0","colab_type":"code","outputId":"ec138415-1e45-4ebb-cc82-6d27a76526fd","executionInfo":{"status":"ok","timestamp":1581080991677,"user_tz":-420,"elapsed":2361,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":34},"trusted":true},"cell_type":"code","source":"full_data = pd.concat([raw_train, raw_test], sort=False).drop(columns='target')\nfull_data.shape","execution_count":null,"outputs":[]},{"metadata":{"id":"_Wx0eLupijcd","colab_type":"text"},"cell_type":"markdown","source":"## XOR values"},{"metadata":{"id":"lH-X2YXFexK8","colab_type":"code","outputId":"88aa1ecc-3c52-4aa6-b40f-2278e8f6c8ac","executionInfo":{"status":"ok","timestamp":1581080992227,"user_tz":-420,"elapsed":2893,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":168},"trusted":true},"cell_type":"code","source":"# Replace values which doesnt appear in both train and test set with another special value ('xor')\ncate_columns = full_data.select_dtypes(include=['object']).columns\nfor col in cate_columns:\n    train_values = set(raw_train[col].unique())\n    test_values = set(raw_test[col].unique())\n\n    xor_values = test_values - train_values \n    if xor_values:\n        print(f'Replace {len(xor_values)} in {col} column')\n        print('They are: ', xor_values)\n        print()\n        full_data.loc[full_data[col].isin(xor_values), col] = 'xor'","execution_count":null,"outputs":[]},{"metadata":{"id":"cPQ-7iZEinbY","colab_type":"text"},"cell_type":"markdown","source":"## Ordinal features mapping\nOrdinal encoding uses a single column of integers to represent the classes. An optional mapping dict can be passed in; in this case, we use the knowledge that there is some true order to the classes themselves"},{"metadata":{"id":"jPfWOxyte31f","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"map_ord1 = {'Novice':1, \n            'Contributor':2, \n            'Expert':4, \n            'Master':5, \n            'Grandmaster':6}\nfull_data.ord_1 = full_data.ord_1.map(map_ord1)","execution_count":null,"outputs":[]},{"metadata":{"id":"Y2pCGIgHe7X2","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"map_ord2 = {'Freezing':1, \n            'Cold':2, \n            'Warm':3, \n            'Hot':4, \n            'Boiling Hot':5, \n            'Lava Hot':6}\nfull_data.ord_2 = full_data.ord_2.map(map_ord2)","execution_count":null,"outputs":[]},{"metadata":{"id":"UiHVZe8ge-pY","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"# Replace a character with its ASCII value\nfull_data['ord_3_by_ord'] = full_data.ord_3.map(ord, na_action='ignore')\nmap_ord3 = {key:value for value,key in enumerate(sorted(full_data.ord_3.dropna().unique()))}\nfull_data.ord_3 = full_data.ord_3.map(map_ord3)","execution_count":null,"outputs":[]},{"metadata":{"id":"uEjAaavqfAUg","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"full_data['ord_4_by_ord'] = full_data.ord_4.map(ord, na_action='ignore')\nmap_ord4 = {key:value for value,key in enumerate(sorted(full_data.ord_4.dropna().unique()))}\nfull_data.ord_4 = full_data.ord_4.map(map_ord4)","execution_count":null,"outputs":[]},{"metadata":{"id":"c1ZePuItfDUn","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"# ord_5 is a little bit more special(2-characters-string)\n# We divide it into 2 pieces of character but also keep the orgin string and convert to categorical features by Label Encoder\n\nfull_data['ord_5_1'] = full_data.ord_5.map(lambda string: ord(string[0]), na_action='ignore')\nfull_data['ord_5_2'] = full_data.ord_5.map(lambda string: ord(string[1]), na_action='ignore')\n\nmap_ord5 = {key:value for value,key in enumerate(sorted(full_data.ord_5.dropna().unique()))} \nfull_data.ord_5 = full_data.ord_5.map(map_ord5)","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"bcMs3rhys-0W"},"cell_type":"markdown","source":"## Missing values indicator\n\nSome columns will be imputed below so that I have to keep track of which values were missing, it really makes sense that my imputation would perform better"},{"metadata":{"id":"M5XimavafHLt","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"num_columns = full_data.select_dtypes(exclude=['object']).columns.drop(['bin_0', 'bin_1', 'bin_2'])\ncate_columns = full_data.columns.drop(num_columns)","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"code","id":"Oh_uB-O0tCKn","colab":{},"trusted":true},"cell_type":"code","source":"missing_num_columns = [col for col in num_columns if any(full_data[col].isnull())]\nfor col in missing_num_columns:\n    full_data[col+'_is_missing'] = full_data[col].isnull().astype(int)","execution_count":null,"outputs":[]},{"metadata":{"id":"K_8gXRKBujIK","colab_type":"text"},"cell_type":"markdown","source":"## Simple imputer and features encoding"},{"metadata":{"id":"JOBYVq_-5QyE","colab_type":"text"},"cell_type":"markdown","source":"I tried to encode cyclic feature with trigonometric functions and expect this technique can be useful to understand and extract insights from samples based upon the patterns and behaviors of the data points over a specific time period"},{"metadata":{"id":"py0GlsHIhVL7","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"time_cols = ['day', 'month']\n\nfor col in time_cols:\n    full_data[col+'_sin'] = np.sin(2*np.pi*full_data[col]/7)\n    full_data[col+'_cos'] = np.cos(2*np.pi*full_data[col]/12)\nfull_data = full_data.drop(columns=time_cols)","execution_count":null,"outputs":[]},{"metadata":{"id":"34rbeucs56iu","colab_type":"text"},"cell_type":"markdown","source":"I will divide features in to 2 groups, the first will be one hot encoded, and the other will be retained"},{"metadata":{"id":"VtA5eaM1ouq0","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"retain_cols = [f'ord_{i}' for i in range(6)] + ['day_sin', 'day_cos', 'month_sin', 'month_cos']\nOH_cols = full_data.columns#.drop(retain_cols)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![](https://miro.medium.com/max/878/1*WXpoiS7HXRC-uwJPYsy1Dg.png)\n\nOne-hot encoding is a process of binarizing the categorical variable. This is done by transforming a categorical variable with n unique values into n unique columns in the datasets while keeping the number of rows the same\n\n"},{"metadata":{"id":"Zt1OI52ZsIKG","colab_type":"code","outputId":"06460c78-1919-4158-ee27-365d229592f7","executionInfo":{"status":"ok","timestamp":1581081010169,"user_tz":-420,"elapsed":18545,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":34},"trusted":true},"cell_type":"code","source":"print(f\"One-Hot encoding {len(OH_cols)} columns\")\n\nOH_full = pd.get_dummies(\n    full_data,\n    columns=OH_cols,\n    drop_first=True,\n    dummy_na=True,\n    sparse=True,\n).sparse.to_coo()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This produces output as a pandas dataframe.Alternatively we can use *OneHotEncoder()* method available in* sklearn* to convert out data to on-hot encoded data.\nBut this method produces a sparse metrix.The advantage of this methos is that is uses very less memory/cpu resourses, if one try to encode all of these features with sklearn's *OneHotEncoder()*, it pottentially cause a crash session"},{"metadata":{"id":"wtcMMDbjwNSo","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"# Impute numeric features with mean value and normalize afterward \nimputer = SimpleImputer(strategy='mean')\nretain_full  = pd.DataFrame(imputer.fit_transform(full_data[retain_cols]), columns=retain_cols)\nretain_full = retain_full/retain_full.max()","execution_count":null,"outputs":[]},{"metadata":{"id":"4eqlT0a5vNc-","colab_type":"text"},"cell_type":"markdown","source":"Combine these encoded dataframe and notice that I've added a term (retain_full$^2$) which will help me to archive higher score (but also reduce the training speed)"},{"metadata":{"id":"MYsCOigexPfp","colab_type":"code","outputId":"7f337d8c-5b8d-4fed-8ad2-f5d310021c88","executionInfo":{"status":"ok","timestamp":1581082495571,"user_tz":-420,"elapsed":4127,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"colab":{"base_uri":"https://localhost:8080/","height":34},"trusted":true},"cell_type":"code","source":"encoded_full = scipy.sparse.hstack([OH_full, retain_full, retain_full**2]).tocsr()\nprint(encoded_full.shape)\n\nencoded_train = encoded_full[:len(raw_train)]\nencoded_test = encoded_full[len(raw_train):]","execution_count":null,"outputs":[]},{"metadata":{"id":"XNrbYLtiiahn","colab_type":"text"},"cell_type":"markdown","source":"# Model\n- In this kernel, I will implement a sklearn's Logistic Regression model\n\n- You can also take a look at my first (and also very simple) XGBoost model for this competitions [here](https://www.kaggle.com/warkingleo2000/xgboost-with-onehot-and-label-encoding)"},{"metadata":{"id":"S0MnFBMPd2AM","colab_type":"text"},"cell_type":"markdown","source":"## Vanilla Logistic Regression\nIt's actually not a vanilla Logistic Regression, the C parameter (inverse of regularization strength) has been set to .03"},{"metadata":{"id":"aOT2S4ngl4R5","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"model = LogisticRegression(C=0.03, max_iter=300)","execution_count":null,"outputs":[]},{"metadata":{"id":"gP4e1HZ14lrG","colab_type":"text"},"cell_type":"markdown","source":"Evaluate my model using 5-fold cross validation and plot the ROC curve"},{"metadata":{"id":"a9ojqQnpl-m0","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":312},"outputId":"2fb3d4ab-809c-4038-9a91-749ab4340b95","executionInfo":{"status":"ok","timestamp":1581081526393,"user_tz":-420,"elapsed":86643,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(8,5))\naucs = []\ncv = StratifiedKFold(n_splits=5, shuffle=True)\n\nfor i, (train,valid) in tqdm_notebook(enumerate(cv.split(encoded_train, raw_train.target))):\n    \n    model.fit(encoded_train[train], raw_train.target[train])\n    valid_pred = model.predict_proba(encoded_train[valid])[:, 1]\n    \n    fpr, tpr, threshold = roc_curve(raw_train.target[valid], valid_pred)\n    roc_auc = auc(fpr, tpr)\n    plt.plot(fpr, tpr, label = f'Folf number {i+1} (AUC = {roc_auc:.4f})')\n    aucs.append(roc_auc)\n\nax.plot([0,1], [0,1], label='Luck', linestyle='--', color='r')  \nmean_auc = np.mean(aucs)\nstd_auc = np.std(aucs)\nax.plot(mean_auc, label=f'Average AUC score: {mean_auc:.4f} $\\pm$ {std_auc:.4f}')\nax.legend(loc=\"lower right\")\nax.set(xlim=[-.1, 1.1], ylim=[-.1, 1.1], title='Logistic Regression')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"eR6MrDwleAdO","colab_type":"text"},"cell_type":"markdown","source":"## Parameter Tuning\nParameter estimation using grid search with cross-validation and taking the best model"},{"metadata":{"id":"tWO8EsMo9f7P","colab_type":"code","colab":{},"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"\"\"\"\nmodel = LogisticRegression()\nparam_grid = {'C' : np.logspace(-4, 4, 20), 'penalty' : ['l1', 'l2']}\n\n# Create grid search object\n\nclf = GridSearchCV(LogisticRegression(), scoring='roc_auc', param_grid = param_grid, cv = 5, verbose=True, n_jobs=-1)\n\n# Fit on data\n\nclf.fit(encoded_train, raw_train.target)\n\nprint(\"tuned hpyerparameters :(best parameters) \",clf.best_params_)\nprint(\"Accuracy :\",clf.best_score_)\"\"\"","execution_count":null,"outputs":[]},{"metadata":{"id":"WP_ZO7NC1aow","colab_type":"text"},"cell_type":"markdown","source":"My GridCV requires total: 40 (candidates) x 5 (folds) = 200 (fits)\n\n So I comment these codes in order to avoid time consuming\n After have the best parameter combination, you can try to refit the LR to encoded data (code below)"},{"metadata":{"id":"q2zFiy2neGxT","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"\"\"\"\nmodel = LogisticRegression(**clf.best_params_)\nfig, ax = plt.subplots(figsize=(8,5))\naucs = []\ncv = StratifiedKFold(n_splits=5, shuffle=True)\n\nfor i, (train,valid) in tqdm_notebook(enumerate(cv.split(encoded_train, raw_train.target))):\n    \n    model.fit(encoded_train[train], raw_train.target[train])\n    valid_pred = model.predict_proba(encoded_train[valid])[:, 1]\n    \n    fpr, tpr, threshold = roc_curve(raw_train.target[valid], valid_pred)\n    roc_auc = auc(fpr, tpr)\n    plt.plot(fpr, tpr, label = f'Folf number {i} (AUC = {roc_auc:.4f})')\n    aucs.append(roc_auc)\n\nax.plot([0,1], [0,1], label='Luck', linestyle='--', color='r')  \nmean_auc = np.mean(aucs)\nstd_auc = np.std(aucs)\nax.plot(mean_auc, label=f'Average AUC score: {mean_auc:.4f} $\\pm$ {std_auc:.4f}')\nax.legend(loc=\"lower right\")\nax.set(xlim=[-.1, 1.1], ylim=[-.1, 1.1], title='Logistic Regression')\nplt.show()\"\"\"","execution_count":null,"outputs":[]},{"metadata":{"id":"0hHN4r9286l8","colab_type":"text"},"cell_type":"markdown","source":"# Predict test set and make submission"},{"metadata":{"id":"gLKWRrW79W7J","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":50},"outputId":"cf3bcf1d-b434-4b7c-e708-9fc659d96041","executionInfo":{"status":"ok","timestamp":1581085506799,"user_tz":-420,"elapsed":177710,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"trusted":true},"cell_type":"code","source":"%%time\nmodel = LogisticRegression(C=0.03, max_iter=300)\nmodel.fit(encoded_train, raw_train.target)\ntest_pred = model.predict_proba(encoded_test)[:, 1]","execution_count":null,"outputs":[]},{"metadata":{"id":"2lHXdK3V9Adu","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":50},"outputId":"d02d7a9b-cfc0-477e-af9d-b789b1b2c8b0","executionInfo":{"status":"ok","timestamp":1581085507346,"user_tz":-420,"elapsed":177886,"user":{"displayName":"Khoa Lâm","photoUrl":"","userId":"16579191490282353088"}},"trusted":true},"cell_type":"code","source":"%%time\nsubmiss = pd.DataFrame({\"id\": raw_test.index, \"target\": test_pred})\nsubmiss.to_csv('Phan_Viet_Hoang.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<p><font size=\"3\" color=\"green\">Last update on 14/2/2020</font></p> \n<p><font size=\"5\" color=\"yellow\">What's next?</font></p>\nI will try to impove my model, try others esemble methods as well as feature engineering technique and update them in this kernel\n\n<p><font size=\"3\" color=\"red\">Thank you</font> for spending time on my kernel!</p>\n"}],"metadata":{"colab":{"name":"EDA.ipynb","provenance":[],"collapsed_sections":[],"toc_visible":true,"authorship_tag":"ABX9TyOY4zAnL1JW86Ql34H8GBT4"},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"nbformat":4,"nbformat_minor":1}