{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"One of the most important things to establish for every Kaggle competition is whether there is a significatn differnece in distributions of the train and test sets. So far the CV validation scores for kernels and for public LB have been pretty close, but the local CV seems to be consistently about 0.01 better than the LB scores. It would be interesting, and potentially very valuable, to find out in a more quantitative and specific way how do these distributions compare. For that purpose we'll build an adverserial validation scheme - we'll run a CV classifier that tries to predict if any given question belongs to the train or the test set.\n\nFirts, we'll have to deal with data gregation adn building of the combined train and test datasets. This work has already been doen in many of the kernels, and in this kernel we'll realy on [Rahul Bamola](https://www.kaggle.com/bamola)'s excelent [kernel](https://www.kaggle.com/bamola/elo-eda-and-modelling-lgbm)."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb\nfrom sklearn.model_selection import KFold\nfrom sklearn import model_selection, preprocessing, metrics\nimport warnings\nimport datetime\nwarnings.filterwarnings(\"ignore\")\nimport os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ad8accff26409d09c65bfeedede56bbf795774cb"},"cell_type":"code","source":"#Loading Train and Test Data\ndf_train = pd.read_csv(\"../input/train.csv\", parse_dates=[\"first_active_month\"])\ndf_test = pd.read_csv(\"../input/test.csv\", parse_dates=[\"first_active_month\"])\nprint(\"{} observations and {} features in train set.\".format(df_train.shape[0],df_train.shape[1]))\nprint(\"{} observations and {} features in test set.\".format(df_test.shape[0],df_test.shape[1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1dddd226e4065e337b998cb6017c9e6f5a28e4cd"},"cell_type":"code","source":"# Let's explore New and Old Merchants\ndf_new_trans = pd.read_csv(\"../input/new_merchant_transactions.csv\")\ndf_hist_trans = pd.read_csv(\"../input/historical_transactions.csv\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8c34ce7e0e4bb4aa72cca8ac61358ad301b1b649"},"cell_type":"code","source":"df_h = df_hist_trans.groupby(\"card_id\").size().reset_index().rename({0:'transactions'},axis=1)\ndf_n = df_new_trans.groupby(\"card_id\").size().reset_index().rename({0:'transactions'},axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2f880ca770747af469f9af280745edcf5d1dcc58"},"cell_type":"code","source":"print(\"Historic Transactions ---->   Average transactions per card : {:.0f}, Maximum transactions : {}.\\nNew Transactions ---->   Average transactions per card : {:.0f}, Maximum transactions : {}. \".format(df_h['transactions'].mean(),df_h['transactions'].max(),df_n['transactions'].mean(),df_n['transactions'].max()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f1ff9521610a97552a56ac04575a02d747610d65"},"cell_type":"code","source":"#Comparting Historic Transactions vs New Transactions for Top 50 Merchants \nm_df_h = df_hist_trans.groupby(\"merchant_id\").size().reset_index().rename({0:'transactions'},axis=1)\nm_df_n = df_new_trans.groupby(\"merchant_id\").size().reset_index().rename({0:'transactions'},axis=1)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a3bdb501c3618821e97b17973c2edd44b950c197"},"cell_type":"code","source":"print(\"Historic Transactions ---->   Average transactions per merchant : {:.0f}, Maximum transactions : {}.\\nNew Transactions ---->   Average transactions per merchant : {:.0f}, Maximum transactions : {}. \".format(m_df_h['transactions'].mean(),m_df_h['transactions'].max(),m_df_n['transactions'].mean(),m_df_n['transactions'].max()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fe1e25a482d2c8909fc644f4e4a24d8b7bc27c4a"},"cell_type":"code","source":"df_train[\"year\"] = df_train[\"first_active_month\"].dt.year\ndf_test[\"year\"] = df_test[\"first_active_month\"].dt.year\ndf_train[\"month\"] = df_train[\"first_active_month\"].dt.month\ndf_test[\"month\"] = df_test[\"first_active_month\"].dt.month\ndf_train['elapsed_time'] = (datetime.date(2018, 2, 1) - df_train['first_active_month'].dt.date).dt.days\ndf_test['elapsed_time'] = (datetime.date(2018, 2, 1) - df_test['first_active_month'].dt.date).dt.days\n\ndf_hist_trans['authorized_flag'] = df_hist_trans['authorized_flag'].map({'Y':1, 'N':0})\ndf_new_trans['authorized_flag'] = df_new_trans['authorized_flag'].map({'Y':1, 'N':0})\n#df_merch_trans = pd.concat([df_hist_trans,df_new_trans])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4399ed6dea52db56333a046382de976c499ea629"},"cell_type":"code","source":"def aggregate_hist_transactions(trans):\n    \n    trans.loc[:, 'purchase_date'] = pd.DatetimeIndex(trans['purchase_date']).\\\n                                      astype(np.int64) * 1e-9\n    \n    agg_func = {\n        'authorized_flag': ['sum', 'mean'],\n        'merchant_id': ['nunique'],\n        'city_id': ['nunique'],\n        'purchase_amount': ['sum', 'median', 'max', 'min', 'std'],\n        'installments': ['sum', 'median', 'max', 'min', 'std'],\n        'purchase_date': [np.ptp],\n        'month_lag': ['min', 'max']\n        }\n    agg_trans = trans.groupby(['card_id']).agg(agg_func)\n    agg_trans.columns = ['hist_' + '_'.join(col).strip() \n                           for col in agg_trans.columns.values]\n    agg_trans.reset_index(inplace=True)\n    \n    df = (trans.groupby('card_id')\n          .size()\n          .reset_index(name='transactions_count'))\n    \n    agg_trans = pd.merge(df, agg_trans, on='card_id', how='left')\n    \n    return agg_trans\n\ndef aggregate_new_transactions(trans):\n    \n    trans.loc[:, 'purchase_date'] = pd.DatetimeIndex(trans['purchase_date']).\\\n                                      astype(np.int64) * 1e-9\n    \n    agg_func = {\n        'authorized_flag': ['sum', 'mean'],\n        'merchant_id': ['nunique'],\n        'city_id': ['nunique'],\n        'purchase_amount': ['sum', 'median', 'max', 'min', 'std'],\n        'installments': ['sum', 'median', 'max', 'min', 'std'],\n        'purchase_date': [np.ptp],\n        'month_lag': ['min', 'max']\n        }\n    agg_trans = trans.groupby(['card_id']).agg(agg_func)\n    agg_trans.columns = ['new_' + '_'.join(col).strip() \n                           for col in agg_trans.columns.values]\n    agg_trans.reset_index(inplace=True)\n    \n    df = (trans.groupby('card_id')\n          .size()\n          .reset_index(name='transactions_count'))\n    \n    agg_trans = pd.merge(df, agg_trans, on='card_id', how='left')\n    \n    return agg_trans","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c22bb7399bbd2a19fc2d239dd98ccabe4a1a8557"},"cell_type":"code","source":"merch_hist = aggregate_hist_transactions(df_hist_trans)\nmerch_new = aggregate_new_transactions(df_new_trans)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"18363849b452537f6b801893cb3ed14b9e294511"},"cell_type":"code","source":"#Merging history with training and test data\ndf_train = pd.merge(df_train, merch_hist, on='card_id',how='left')\ndf_test = pd.merge(df_test, merch_hist, on='card_id',how='left')\n\ndf_train = pd.merge(df_train, merch_new, on='card_id',how='left')\ndf_test = pd.merge(df_test, merch_new, on='card_id',how='left')\n\ntarget = df_train['target']\n\nuse_cols  = [c for c in df_train.columns if c not in ['card_id', 'first_active_month', 'target']]\n\nfeatures = list(df_train[use_cols].columns)\ncategorical_features = [col for col in features if 'feature_' in col]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d7379cfce054cc722e936b57b7a271ac2da051fb"},"cell_type":"code","source":"df_train = df_train[use_cols]\ndf_test = df_test[use_cols]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0e995f5286057b2728b7ae645f51a774e447cc32"},"cell_type":"code","source":"df_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1954b439b12a9cc587bc2d0376b9159de1822263"},"cell_type":"code","source":"df_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a5eb444de4e2a423c643e428f729157c6839472d"},"cell_type":"code","source":"df_train['target'] = 0\ndf_test['target'] = 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6fa1d19b21f3ddbf713b45115c16c43c1f1eb48a"},"cell_type":"code","source":"train_test = pd.concat([df_train, df_test], axis =0)\n\ntarget = train_test['target'].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ca972598fc4734662a39e40a4d285df397876c91"},"cell_type":"code","source":"param = {'num_leaves': 50,\n         'min_data_in_leaf': 30, \n         'objective':'binary',\n         'max_depth': 5,\n         'learning_rate': 0.001,\n         \"min_child_samples\": 20,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9 ,\n         \"bagging_seed\": 17,\n         \"metric\": 'auc',\n         \"lambda_l1\": 0.1,\n         \"verbosity\": -1}\n\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(len(train_test))\n\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_test.values, target)):\n    print(\"fold n°{}\".format(fold_))\n    trn_data = lgb.Dataset(train_test.iloc[trn_idx][features], label=target[trn_idx], categorical_feature=categorical_features)\n    val_data = lgb.Dataset(train_test.iloc[val_idx][features], label=target[val_idx], categorical_feature=categorical_features)\n\n    num_round = 10000\n    clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=100, early_stopping_rounds = 100)\n    oof[val_idx] = clf.predict(train_test.iloc[val_idx][features], num_iteration=clf.best_iteration)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4317fb053013a4d7ec3ad3bf25445fabdbcf7928"},"cell_type":"markdown","source":"We see that the oof AUC is **VERY** close to 0.5, so these two datasets seem very statistically similar. In other words, relying on your local validation should work very well for this competition. "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}