{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About competition \n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Can you find more cat in your dat?\n\nWe loved the participation and engagement with the first Cat in the Dat competition.\n\nBecause this is such a common task and important skill to master, we've put together a dataset that contains only categorical features, and includes:\n\n* binary features\n* low- and high-cardinality nominal features\n* low- and high-cardinality ordinal features\n* (potentially) cyclical features\nThis follow-up competition offers an even more challenging dataset so that you can continue to build your skills with the common machine learning task of encoding categorical variables. This challenge adds the additional complexity of feature interactions, as well as missing data.\n\nThis Playground competition will give you the opportunity to try different encoding schemes for different algorithms to compare how they perform. We encourage you to share what you find with the community.\n\nIf you're not sure how to get started, you can check out the Categorical Variables section of Kaggle's Intermediate Machine Learning course.\n\n\nHave Fun!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Importing packages ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Import basic python modules'''\nimport pandas as pd\nimport numpy as np\nimport string\nfrom sklearn.model_selection import KFold, StratifiedKFold, cross_val_score\nfrom sklearn.metrics import roc_auc_score as auc\nfrom sklearn.linear_model import LogisticRegression\n\n'''import visualization libraries'''\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set_style(\"ticks\")\n%matplotlib inline\n\n'''Plotly visualization library .'''\nimport plotly.offline as py\nimport plotly.graph_objs as go\nimport plotly.tools as tls\npy.init_notebook_mode(connected=True)\n\n# markdown display formatted output\nfrom IPython.display import Markdown\ndef bold(string):\n    display(Markdown(string))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Load data\ntrain = pd.read_csv('../input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('../input/cat-in-the-dat-ii/test.csv')\n\nprint(train.shape)\nprint(test.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train.sample(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Variable description\ndef description(df):\n    print(f\"Dataset Shape: {df.shape}\")\n    summary = pd.DataFrame(df.dtypes,columns=['dtypes'])\n    summary = summary.reset_index()\n    summary['Name'] = summary['index']\n    summary = summary[['Name','dtypes']]\n    summary['Missing'] = df.isnull().sum().values    \n    summary['Uniques'] = df.nunique().values\n    summary['First Value'] = df.iloc[0].values\n    summary['Second Value'] = df.iloc[1].values\n    summary['Third Value'] = df.iloc[2].values\n    return summary\nbold('**Variable Description of  train Data:**')\ndescription(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def replace_nan(data):\n    for column in data.columns:\n        if data[column].isna().sum() > 0:\n            data[column] = data[column].fillna(data[column].mode()[0])\n\n\nreplace_nan(train)\nreplace_nan(test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Target feature","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now we will see the distribution of the target feature and try to find its nature ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"total = len(train)\nplt.figure(figsize=(10,6))\n\ng = sns.countplot(x='target', data=train, palette='coolwarm')\ng.set_title(\"TARGET DISTRIBUTION\", fontsize = 20)\ng.set_xlabel(\"Target Vaues\", fontsize = 15)\ng.set_ylabel(\"Count\", fontsize = 15)\nsizes=[] # Get highest values in y\nfor p in g.patches:\n    height = p.get_height()\n    sizes.append(height)\n    g.text(p.get_x()+p.get_width()/2.,\n            height + 3,\n            '{:1.2f}%'.format(height/total*100),\n            ha=\"center\", fontsize=14) \ng.set_ylim(0, max(sizes) * 1.15) # set y limit based on highest heights\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Clearly the data is imbalanced ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Let us see binary features","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"bin_cols = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4']\n\nimport matplotlib.gridspec as gridspec # to do the grid of plots\ngrid = gridspec.GridSpec(3, 2) # The grid of chart\nplt.figure(figsize=(16,20)) # size of figure\n\n# loop to get column and the count of plots\n\nfor n, col in enumerate(train[bin_cols]): \n    ax = plt.subplot(grid[n]) # feeding the figure of grid\n    sns.countplot(x=col, data=train, hue='target', palette='Set1') \n    ax.set_ylabel('Count', fontsize=15) # y axis label\n    ax.set_title(f'{col} Distribution by Target', fontsize=18) # title label\n    ax.set_xlabel(f'{col} values', fontsize=15) # x axis label\n    sizes=[] # Get highest values in y\n    for p in ax.patches: # loop to all objects\n        height = p.get_height()\n        sizes.append(height)\n        ax.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(height/total*100),\n                ha=\"center\", fontsize=14) \n    ax.set_ylim(0, max(sizes) * 1.15) #set y limit based on highest heights\n    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Nominal features","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now we will see the distribution of the feature and target Ratio for each value in nominal features","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"nom_cols = ['nom_0', 'nom_1', 'nom_2', 'nom_3', 'nom_4']\n\n\ndef ploting_cat_fet(df, cols, vis_row=5, vis_col=2):\n    \n    grid = gridspec.GridSpec(vis_row,vis_col) # The grid of chart\n    plt.figure(figsize=(17, 35)) # size of figure\n\n    # loop to get column and the count of plots\n    for n, col in enumerate(train[cols]): \n        tmp = pd.crosstab(train[col], train['target'], normalize='index') * 100\n        tmp = tmp.reset_index()\n        tmp.rename(columns={0:'No',1:'Yes'}, inplace=True)\n\n        ax = plt.subplot(grid[n]) # feeding the figure of grid\n        sns.countplot(x=col, data=train, order=list(tmp[col].values) , palette='Set3') \n        ax.set_ylabel('Count', fontsize=15) # y axis label\n        ax.set_title(f'{col} Distribution by Target', fontsize=18) # title label\n        ax.set_xlabel(f'{col} values', fontsize=15) # x axis label\n\n        # twinX - to build a second yaxis\n        gt = ax.twinx()\n        gt = sns.pointplot(x=col, y='Yes', data=tmp,\n                           order=list(tmp[col].values),\n                           color='black', legend=False)\n        gt.set_ylim(tmp['Yes'].min()-5,tmp['Yes'].max()*1.1)\n        gt.set_ylabel(\"Target %True(1)\", fontsize=16)\n        sizes=[] # Get highest values in y\n        for p in ax.patches: # loop to all objects\n            height = p.get_height()\n            sizes.append(height)\n            ax.text(p.get_x()+p.get_width()/2.,\n                    height + 3,\n                    '{:1.2f}%'.format(height/total*100),\n                    ha=\"center\", fontsize=14) \n        ax.set_ylim(0, max(sizes) * 1.15) # set y limit based on highest heights\n\n\n    plt.subplots_adjust(hspace = 0.5, wspace=.3)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ploting_cat_fet(train, nom_cols, vis_row=5, vis_col=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Ordinal features with more then 2 and less then 15 values","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ord_cols = ['ord_0', 'ord_1', 'ord_2', 'ord_3']\n\n#Ploting\nploting_cat_fet(train, ord_cols, vis_row=5, vis_col=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Date features","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"date_cols = ['day', 'month']\n\n# Calling the plot function with date columns\nploting_cat_fet(train, date_cols, vis_row=5, vis_col=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['ord_5_ot'] = 'Others'\ntrain.loc[train['ord_5'].isin(train['ord_5'].value_counts()[:25].sort_index().index), 'ord_5_ot'] = train['ord_5']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature engineering","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"target = train['target']\ntrain_id = train['id']\ntest_id = test['id']\ntrain.drop(['target', 'id','ord_5_ot'], axis=1, inplace=True)\ntest.drop('id', axis=1, inplace=True)\n\nprint(train.shape)\nprint(test.shape)\nprint(target.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### One Hot Encoding","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"traintest = pd.concat([train, test])\ndummies = pd.get_dummies(traintest, columns=traintest.columns, drop_first=True, sparse=True)\ntrain_ohe = dummies.iloc[:train.shape[0], :]\ntest_ohe = dummies.iloc[train.shape[0]:, :]\n\nprint(train_ohe.shape)\nprint(test_ohe.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Converting dataframe into sparse matrix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n'''Covert dataframe to spare matrix'''\ntrain_ohe = train_ohe.sparse.to_coo().tocsr()\ntest_ohe = test_ohe.sparse.to_coo().tocsr()\ntype(train_ohe)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Model\ndef run_cv_model(train, test, target, model_fn, params={}, eval_fn=None, label='model'):\n    kf = KFold(n_splits=5)\n    fold_splits = kf.split(train, target)\n    cv_scores = []\n    pred_full_test = 0\n    pred_train = np.zeros((train.shape[0]))\n    i = 1\n    for dev_index, val_index in fold_splits:\n        print('Started ' + label + ' fold ' + str(i) + '/5')\n        dev_X, val_X = train[dev_index], train[val_index]\n        dev_y, val_y = target[dev_index], target[val_index]\n        params2 = params.copy()\n        pred_val_y, pred_test_y = model_fn(dev_X, dev_y, val_X, val_y, test, params2)\n        pred_full_test = pred_full_test + pred_test_y\n        pred_train[val_index] = pred_val_y\n        if eval_fn is not None:\n            cv_score = eval_fn(val_y, pred_val_y)\n            cv_scores.append(cv_score)\n            print(label + ' cv score {}: {}'.format(i, cv_score))\n        i += 1\n    print('{} cv scores : {}'.format(label, cv_scores))\n    print('{} cv mean score : {}'.format(label, np.mean(cv_scores)))\n    print('{} cv std score : {}'.format(label, np.std(cv_scores)))\n    pred_full_test = pred_full_test / 5.0\n    results = {'label': label,\n              'train': pred_train, 'test': pred_full_test,\n              'cv': cv_scores}\n    return results","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def runLR(train_X, train_y, test_X, test_y, test_X2, params):\n    print('Train LR')\n    model = LogisticRegression(**params)\n    model.fit(train_X, train_y)\n    print('Predict 1/2')\n    pred_test_y = model.predict_proba(test_X)[:, 1]\n    print('Predict 2/2')\n    pred_test_y2 = model.predict_proba(test_X2)[:, 1]\n    return pred_test_y, pred_test_y2\n\n\nlr_params = {'solver': 'liblinear', 'C':  0.1, 'max_iter': 1000}\nresults = run_cv_model(train_ohe, test_ohe, target, runLR, lr_params, auc, 'lr')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Creating submission file","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nsubmission = pd.DataFrame({'id': test_id, 'target': results['test']})\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Thankyou for going through my kernel if you found it good please upvote!!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Your suggetion are most welcome :) ","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}