{"cells":[{"metadata":{},"cell_type":"markdown","source":"![](https://media.boingboing.net/wp-content/uploads/2019/01/giphy-3.gif)"},{"metadata":{},"cell_type":"markdown","source":"# Welcome to my EDA and Modeling kernel !\nIt's a very cool opportunity to practice and learn from other kagglers about interesting feature encoding techniques and modelling;\n\nI hope you enjoy my work and bring me your feedback. If this kernel is useful for you, please don't forget to upvote the kernel"},{"metadata":{},"cell_type":"markdown","source":"# Competition description:\nCan you find more cat in your dat?\n\nWe loved the participation and engagement with the first Cat in the Dat competition.\n\nBecause this is such a common task and important skill to master, we've put together a dataset that contains only categorical features, and includes:\n\nbinary features\nlow- and high-cardinality nominal features\nlow- and high-cardinality ordinal features\n(potentially) cyclical features\nThis follow-up competition offers an even more challenging dataset so that you can continue to build your skills with the common machine learning task of encoding categorical variables. This challenge adds the additional complexity of feature interactions, as well as missing data.\n\nThis Playground competition will give you the opportunity to try different encoding schemes for different algorithms to compare how they perform. We encourage you to share what you find with the community.\n\nIf you're not sure how to get started, you can check out the Categorical Variables section of Kaggle's Intermediate Machine Learning course.\n\n\nHave Fun!"},{"metadata":{},"cell_type":"markdown","source":"# Objective:\nI want to do complete exploration to understand the data and after it I will build a Machine Learning Model."},{"metadata":{},"cell_type":"markdown","source":"#### Source links:\nEncoding Challenge I Notebook--> https://www.kaggle.com/vikassingh1996/handling-categorical-variables-encoding-modeling\n\nModel---> https://www.kaggle.com/peterhurford/why-not-logistic-regression\n\nVisualization-->https://www.kaggle.com/kabure/extensive-eda-and-modeling-xgb-hyperopt"},{"metadata":{},"cell_type":"markdown","source":"# Importing Packages and Collecting Data"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"'''Import basic modules'''\nimport pandas as pd\nimport numpy as np\nimport string\nfrom sklearn.model_selection import KFold, StratifiedKFold, cross_val_score\nfrom sklearn.metrics import roc_auc_score as auc\nfrom sklearn.linear_model import LogisticRegression\n\n'''import visualization'''\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set_style(\"ticks\")\n%matplotlib inline\n\n'''Plotly visualization .'''\nimport plotly.offline as py\nimport plotly.graph_objs as go\nimport plotly.tools as tls\npy.init_notebook_mode(connected=True)\n\n'''Display markdown formatted output like bold, italic bold etc.'''\nfrom IPython.display import Markdown\ndef bold(string):\n    display(Markdown(string))\n\n'''Ignore deprecation and future, and user warnings.'''\nimport warnings as wrn\nwrn.filterwarnings('ignore', category = DeprecationWarning) \nwrn.filterwarnings('ignore', category = FutureWarning) \nwrn.filterwarnings('ignore', category = UserWarning) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%%time\n\n# Load data\ntrain = pd.read_csv('../input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('../input/cat-in-the-dat-ii/test.csv')\n\nprint(train.shape)\nprint(test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Variable Description'''\ndef description(df):\n    print(f\"Dataset Shape: {df.shape}\")\n    summary = pd.DataFrame(df.dtypes,columns=['dtypes'])\n    summary = summary.reset_index()\n    summary['Name'] = summary['index']\n    summary = summary[['Name','dtypes']]\n    summary['Missing'] = df.isnull().sum().values    \n    summary['Uniques'] = df.nunique().values\n    summary['First Value'] = df.iloc[0].values\n    summary['Second Value'] = df.iloc[1].values\n    summary['Third Value'] = df.iloc[2].values\n    return summary\nbold('**Variable Description of  train Data:**')\ndescription(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def replace_nan(data):\n    for column in data.columns:\n        if data[column].isna().sum() > 0:\n            data[column] = data[column].fillna(data[column].mode()[0])\n\n\nreplace_nan(train)\nreplace_nan(test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Target Feature\n   * Let's see the distribution and if we can identify what is the nature of this feature"},{"metadata":{"trusted":true},"cell_type":"code","source":"total = len(train)\nplt.figure(figsize=(10,6))\n\ng = sns.countplot(x='target', data=train, palette='coolwarm')\ng.set_title(\"TARGET DISTRIBUTION\", fontsize = 20)\ng.set_xlabel(\"Target Vaues\", fontsize = 15)\ng.set_ylabel(\"Count\", fontsize = 15)\nsizes=[] # Get highest values in y\nfor p in g.patches:\n    height = p.get_height()\n    sizes.append(height)\n    g.text(p.get_x()+p.get_width()/2.,\n            height + 3,\n            '{:1.2f}%'.format(height/total*100),\n            ha=\"center\", fontsize=14) \ng.set_ylim(0, max(sizes) * 1.15) # set y limit based on highest heights\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that data is imbalanced with 81.28% of 0's and 18.72% of 1's."},{"metadata":{},"cell_type":"markdown","source":"## Binary Features"},{"metadata":{"trusted":true},"cell_type":"code","source":"bin_cols = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4']\n\nimport matplotlib.gridspec as gridspec # to do the grid of plots\ngrid = gridspec.GridSpec(3, 2) # The grid of chart\nplt.figure(figsize=(16,20)) # size of figure\n\n# loop to get column and the count of plots\nfor n, col in enumerate(train[bin_cols]): \n    ax = plt.subplot(grid[n]) # feeding the figure of grid\n    sns.countplot(x=col, data=train, hue='target', palette='Set2') \n    ax.set_ylabel('Count', fontsize=15) # y axis label\n    ax.set_title(f'{col} Distribution by Target', fontsize=18) # title label\n    ax.set_xlabel(f'{col} values', fontsize=15) # x axis label\n    sizes=[] # Get highest values in y\n    for p in ax.patches: # loop to all objects\n        height = p.get_height()\n        sizes.append(height)\n        ax.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(height/total*100),\n                ha=\"center\", fontsize=14) \n    ax.set_ylim(0, max(sizes) * 1.15) #set y limit based on highest heights\n    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In binary variables, it seens that only bin_0 is imbalanced."},{"metadata":{},"cell_type":"markdown","source":"# Nominal Features (with more than 2 and less than 15 values)\n* Let's see the distribution of the feature and target Ratio for each value in nominal features"},{"metadata":{"trusted":true},"cell_type":"code","source":"nom_cols = ['nom_0', 'nom_1', 'nom_2', 'nom_3', 'nom_4']\n\n\ndef ploting_cat_fet(df, cols, vis_row=5, vis_col=2):\n    \n    grid = gridspec.GridSpec(vis_row,vis_col) # The grid of chart\n    plt.figure(figsize=(17, 35)) # size of figure\n\n    # loop to get column and the count of plots\n    for n, col in enumerate(train[cols]): \n        tmp = pd.crosstab(train[col], train['target'], normalize='index') * 100\n        tmp = tmp.reset_index()\n        tmp.rename(columns={0:'No',1:'Yes'}, inplace=True)\n\n        ax = plt.subplot(grid[n]) # feeding the figure of grid\n        sns.countplot(x=col, data=train, order=list(tmp[col].values) , palette='Set1') \n        ax.set_ylabel('Count', fontsize=15) # y axis label\n        ax.set_title(f'{col} Distribution by Target', fontsize=18) # title label\n        ax.set_xlabel(f'{col} values', fontsize=15) # x axis label\n\n        # twinX - to build a second yaxis\n        gt = ax.twinx()\n        gt = sns.pointplot(x=col, y='Yes', data=tmp,\n                           order=list(tmp[col].values),\n                           color='black', legend=False)\n        gt.set_ylim(tmp['Yes'].min()-5,tmp['Yes'].max()*1.1)\n        gt.set_ylabel(\"Target %True(1)\", fontsize=16)\n        sizes=[] # Get highest values in y\n        for p in ax.patches: # loop to all objects\n            height = p.get_height()\n            sizes.append(height)\n            ax.text(p.get_x()+p.get_width()/2.,\n                    height + 3,\n                    '{:1.2f}%'.format(height/total*100),\n                    ha=\"center\", fontsize=14) \n        ax.set_ylim(0, max(sizes) * 1.15) # set y limit based on highest heights\n\n\n    plt.subplots_adjust(hspace = 0.5, wspace=.3)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ploting_cat_fet(train, nom_cols, vis_row=5, vis_col=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Ordinal Features (with more than 2 and less than 15 values)"},{"metadata":{"trusted":true},"cell_type":"code","source":"ord_cols = ['ord_0', 'ord_1', 'ord_2', 'ord_3']\n\n#Ploting\nploting_cat_fet(train, ord_cols, vis_row=5, vis_col=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Ord_4 and ord_5 (More than 15 Values)"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['ord_5_ot'] = 'Others'\ntrain.loc[train['ord_5'].isin(train['ord_5'].value_counts()[:25].sort_index().index), 'ord_5_ot'] = train['ord_5']","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tmp = pd.crosstab(train['ord_4'], train['target'], normalize='index') * 100\ntmp = tmp.reset_index()\ntmp.rename(columns={0:'No',1:'Yes'}, inplace=True)\nplt.figure(figsize=(15,12))\n\nplt.subplot(211)\nax = sns.countplot(x='ord_4', data=train, order=list(tmp['ord_4'].values) , palette='Set1') \nax.set_ylabel('Count', fontsize=17) # y axis label\nax.set_title('ord_4 Distribution with Target %ratio', fontsize=20) # title label\nax.set_xlabel('ord_4 values', fontsize=17) # x axis label\n# twinX - to build a second yaxis\ngt = ax.twinx()\ngt = sns.pointplot(x='ord_4', y='Yes', data=tmp,\n                   order=list(tmp['ord_4'].values),\n                   color='black', legend=False)\ngt.set_ylim(tmp['Yes'].min()-5,tmp['Yes'].max()*1.1)\ngt.set_ylabel(\"Target %True(1)\", fontsize=16)\n\ntmp = pd.crosstab(train['ord_5_ot'], train['target'], normalize='index') * 100\ntmp = tmp.reset_index()\ntmp.rename(columns={0:'No',1:'Yes'}, inplace=True)\n\nplt.subplot(212)\nax1 = sns.countplot(x='ord_5_ot', data=train,\n                   order=list(train['ord_5_ot'].value_counts().sort_index().index) ,\n                   palette='Set1') \nax1.set_ylabel('Count', fontsize=17) # y axis label\nax1.set_title('TOP 25 ord_5 and \"others\" Distribution with Target %ratio', fontsize=20) # title label\nax1.set_xlabel('ord_5 values', fontsize=17) # x axis label\n# twinX - to build a second yaxis\ngt = ax1.twinx()\ngt = sns.pointplot(x='ord_5_ot', y='Yes', data=tmp,\n                   order=list(train['ord_5_ot'].value_counts().sort_index().index),\n                   color='black', legend=False)\ngt.set_ylim(tmp['Yes'].min()-5,tmp['Yes'].max()*1.1)\ngt.set_ylabel(\"Target %True(1)\", fontsize=16)\n\nplt.subplots_adjust(hspace = 0.4, wspace=.3)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Date Features"},{"metadata":{"trusted":true},"cell_type":"code","source":"date_cols = ['day', 'month']\n\n# Calling the plot function with date columns\nploting_cat_fet(train, date_cols, vis_row=5, vis_col=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Feature Engineering"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n'''Subset'''\ntarget = train['target']\ntrain_id = train['id']\ntest_id = test['id']\ntrain.drop(['target', 'id','ord_5_ot'], axis=1, inplace=True)\ntest.drop('id', axis=1, inplace=True)\n\nprint(train.shape)\nprint(test.shape)\nprint(target.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n'''One Hot Encode'''\ntraintest = pd.concat([train, test])\ndummies = pd.get_dummies(traintest, columns=traintest.columns, drop_first=True, sparse=True)\ntrain_ohe = dummies.iloc[:train.shape[0], :]\ntest_ohe = dummies.iloc[train.shape[0]:, :]\n\nprint(train_ohe.shape)\nprint(test_ohe.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n'''Covert dataframe to spare matrix'''\ntrain_ohe = train_ohe.sparse.to_coo().tocsr()\ntest_ohe = test_ohe.sparse.to_coo().tocsr()\ntype(train_ohe)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Model\ndef run_cv_model(train, test, target, model_fn, params={}, eval_fn=None, label='model'):\n    kf = KFold(n_splits=5)\n    fold_splits = kf.split(train, target)\n    cv_scores = []\n    pred_full_test = 0\n    pred_train = np.zeros((train.shape[0]))\n    i = 1\n    for dev_index, val_index in fold_splits:\n        print('Started ' + label + ' fold ' + str(i) + '/5')\n        dev_X, val_X = train[dev_index], train[val_index]\n        dev_y, val_y = target[dev_index], target[val_index]\n        params2 = params.copy()\n        pred_val_y, pred_test_y = model_fn(dev_X, dev_y, val_X, val_y, test, params2)\n        pred_full_test = pred_full_test + pred_test_y\n        pred_train[val_index] = pred_val_y\n        if eval_fn is not None:\n            cv_score = eval_fn(val_y, pred_val_y)\n            cv_scores.append(cv_score)\n            print(label + ' cv score {}: {}'.format(i, cv_score))\n        i += 1\n    print('{} cv scores : {}'.format(label, cv_scores))\n    print('{} cv mean score : {}'.format(label, np.mean(cv_scores)))\n    print('{} cv std score : {}'.format(label, np.std(cv_scores)))\n    pred_full_test = pred_full_test / 5.0\n    results = {'label': label,\n              'train': pred_train, 'test': pred_full_test,\n              'cv': cv_scores}\n    return results\n\n\ndef runLR(train_X, train_y, test_X, test_y, test_X2, params):\n    print('Train LR')\n    model = LogisticRegression(**params)\n    model.fit(train_X, train_y)\n    print('Predict 1/2')\n    pred_test_y = model.predict_proba(test_X)[:, 1]\n    print('Predict 2/2')\n    pred_test_y2 = model.predict_proba(test_X2)[:, 1]\n    return pred_test_y, pred_test_y2\n\n\nlr_params = {'solver': 'liblinear', 'C':  0.1, 'max_iter': 1000}\nresults = run_cv_model(train_ohe, test_ohe, target, runLR, lr_params, auc, 'lr')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nsubmission = pd.DataFrame({'id': test_id, 'target': results['test']})\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"import pandas as pd\nsample_submission = pd.read_csv(\"../input/cat-in-the-dat-ii/sample_submission.csv\")\ntest = pd.read_csv(\"../input/cat-in-the-dat-ii/test.csv\")\ntrain = pd.read_csv(\"../input/cat-in-the-dat-ii/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Give me your feedback and if you find my kernel helpful please UPVOTE will be appreciated."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}