{"cells":[{"metadata":{"_uuid":"b05307c6e9faf65e975fe38dbf6f133e6a68af8d"},"cell_type":"markdown","source":"## Overview\n\nThe purpose of this kernel is to take a look at the data, come up with some insights, and attempt to create a predictive model or two. This notebook is still **very** raw. I will work on it as my very limited time permits, and hope to expend it in the upcoming days and weeks.\n\nNB: Most of the feature engineering and some of the modeling is based on [Peter Hurford's excellent kernel.](https://www.kaggle.com/peterhurford/you-re-going-to-want-more-categories-lb-3-737/notebook) \n\nInspired From : https://www.kaggle.com/tunguz/eloda-with-feature-engineering-and-stacking\n\n## Packages\n\nFirst, let's load a few useful Python packages. This section will keep growing in subsequent versions of this EDA."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb\nfrom sklearn.linear_model import Ridge\n\nfrom sklearn import preprocessing\nimport warnings\nimport datetime\nwarnings.filterwarnings(\"ignore\")\nimport gc\n\nfrom scipy.stats import describe\n%matplotlib inline\n\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_error\nimport xgboost as xgb\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cb7c6188e659ba49b11168b1154092d040f3fc93"},"cell_type":"markdown","source":"Let's see what files we have in the input directory:"},{"metadata":{"trusted":true,"_uuid":"fc756f5a07aa306835fe5e24e933a8c6b21e7d7c"},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"186afdd8cefab16d2a7fb3ffc2b178ef7b2c87a8"},"cell_type":"markdown","source":"We see that in addition to the usual,`train`, `test` and `sample_submission` files, we also have `merchants`, `historical_transactions`, `new_merchant_transactions`, and even one (**HORROR!!!**) excel file - `Data_Ditionary`. The names of the files are pretty self-explanatory, but we'll take a look at them and explore them. First, let's look at the `train` and `test` files."},{"metadata":{"trusted":true,"_uuid":"0c0c9c2f5f90bf0fc21aa8d664d0209ceb1e03fe"},"cell_type":"code","source":"%%time\n#Loading Train and Test Data\ntrain = pd.read_csv(\"../input/train.csv\", parse_dates=[\"first_active_month\"])\ntest = pd.read_csv(\"../input/test.csv\", parse_dates=[\"first_active_month\"])\nprint(\"{} observations and {} features in train set.\".format(train.shape[0],train.shape[1]))\nprint(\"{} observations and {} features in test set.\".format(test.shape[0],test.shape[1]))\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c1531fe28a8484caeb387ff3d485ad4bff98b270"},"cell_type":"code","source":"%time train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a38ec0dbedbf2163ff7b1bec3ffa27346d5c1921"},"cell_type":"code","source":"%time test.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fb61542f287afe3180e2af13dd8279dd4a2f3a42"},"cell_type":"markdown","source":"Seems farily straightforward - just ID, first active months, three anonimous features, and target firld for train set.\n\nLet's take a look at the target variable:\n"},{"metadata":{"trusted":true,"_uuid":"a5d7d3007de2483606a8b518f097917c18fd3e0e"},"cell_type":"code","source":"%time train.target.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"898f72ebac71a0cdb75d2a88376d3373766620ca"},"cell_type":"markdown","source":"Seems like a very wide range of values, relatively spaking. Let's take a look at the graph of the distribution:"},{"metadata":{"trusted":true,"_uuid":"bb28a8fd8e4293af7fbd7acc24f232c8ee311157"},"cell_type":"code","source":"%%time\nplt.figure(figsize=(12, 5))\nplt.hist(train.target.values, bins=200)\nplt.title('Histogram target counts')\nplt.xlabel('Count')\nplt.ylabel('Target')\nplt.show()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cf99081474f63fbc24b477687f59c38d1a7fac85"},"cell_type":"markdown","source":"Seems like a pretty nice normal-looking distribution, except for the few anomalous elements at teh far left. They will have to be dealt with separately.\n\nLet's look at the \"violin\" version of the same plot. "},{"metadata":{"trusted":true,"_uuid":"259aca59eb9007cd05b480e9d5b857fe7168c89e"},"cell_type":"code","source":"%%time\nsns.set_style(\"whitegrid\")\nax = sns.violinplot(x=train.target.values)\nplt.show()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"107dd4730d422fc66b115c7e2a8044c82da06683"},"cell_type":"markdown","source":"Yup, there is that little bump on the far left again."},{"metadata":{"_uuid":"9459fdba4a9c005326dd8b7b63039fabf1865b0e"},"cell_type":"markdown","source":"Let's now look at the distributions of various \"features\""},{"metadata":{"trusted":true,"_uuid":"44433fb6a105bca6bcd6ca47773ddbf23dbaccaa"},"cell_type":"code","source":"%%time\nplt.figure(figsize=(12, 5))\nplt.hist(train.feature_1.values, bins=50, color = 'orange')\nplt.title('Histogram feature_1 counts')\nplt.xlabel('Count')\nplt.ylabel('Target')\nplt.show()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5a27d2b4773131d41f2a4642cfa24392966eaf86"},"cell_type":"code","source":"plt.figure(figsize=(12, 5))\nplt.hist(train.feature_2.values, bins=50, color=\"green\")\nplt.title('Histogram feature_2 counts')\nplt.xlabel('Count')\nplt.ylabel('Target')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c6688a59f2f563f716ddcfb33ad4df12539d75b1"},"cell_type":"code","source":"plt.figure(figsize=(12, 5))\nplt.hist(train.feature_3.values, bins=50, color = \"red\")\nplt.title('Histogram feature_3 counts')\nplt.xlabel('Count')\nplt.ylabel('Target')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"59cd40c338ec080eee3816fd3301bdf8815a40e6"},"cell_type":"markdown","source":"For now I am not including plots for the test set, as they at first approsimation look very similar.\n\nA couple of things that stand out are:\n\n1. There are only a handful of values for each of the three features.\n2. They are discrete\n3. They are relatively eavenly distributed.\n\nAll of this suggests that these features are categorical and have been label-encoded. "},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"Here is a gratuitous embedding of YouTube video of 'The Girl From Ipanema'. For no good reason."},{"metadata":{"trusted":true,"_uuid":"05209ca8d97f761026845453145c3a5cd26b5b5c"},"cell_type":"markdown","source":"# Feature Engineering"},{"metadata":{"trusted":true,"_uuid":"0592adb0399adfb5725cfe9abfd73b9fe01ccf1a"},"cell_type":"code","source":"%%time\ntrain[\"month\"] = train[\"first_active_month\"].dt.month\ntest[\"month\"] = test[\"first_active_month\"].dt.month\ntrain[\"year\"] = train[\"first_active_month\"].dt.year\ntest[\"year\"] = test[\"first_active_month\"].dt.year\ntrain['elapsed_time'] = (datetime.date(2018, 2, 1) - train['first_active_month'].dt.date).dt.days\ntest['elapsed_time'] = (datetime.date(2018, 2, 1) - test['first_active_month'].dt.date).dt.days\ntrain.head()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"319baa73a6fb69ffcaaa99d1a822626eddd086ae"},"cell_type":"code","source":"%%time\ntrain = pd.get_dummies(train, columns=['feature_1', 'feature_2'])\ntest = pd.get_dummies(test, columns=['feature_1', 'feature_2'])\ntrain.head()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5cf61d938b3cd4455e03c2aef646f60a87cad825"},"cell_type":"code","source":"%%time\nhist_trans = pd.read_csv(\"../input/historical_transactions.csv\")\nhist_trans.head()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"71c22f0abc1b90c285a78812acc5b5c6c57eb7c1"},"cell_type":"code","source":"%%time\nhist_trans = pd.get_dummies(hist_trans, columns=['category_2', 'category_3'])\nhist_trans['authorized_flag'] = hist_trans['authorized_flag'].map({'Y': 1, 'N': 0})\nhist_trans['category_1'] = hist_trans['category_1'].map({'Y': 1, 'N': 0})\nhist_trans.head()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"73e7ce787e9e4884b309346ab22e82dce23a314b"},"cell_type":"code","source":"%%time\ndef aggregate_transactions(trans, prefix):  \n    trans.loc[:, 'purchase_date'] = pd.DatetimeIndex(trans['purchase_date']).\\\n                                      astype(np.int64) * 1e-9\n    \n    agg_func = {\n        'authorized_flag': ['sum', 'mean'],\n        'category_1': ['mean'],\n        'category_2_1.0': ['mean'],\n        'category_2_2.0': ['mean'],\n        'category_2_3.0': ['mean'],\n        'category_2_4.0': ['mean'],\n        'category_2_5.0': ['mean'],\n        'category_3_A': ['mean'],\n        'category_3_B': ['mean'],\n        'category_3_C': ['mean'],\n        'merchant_id': ['nunique'],\n        'purchase_amount': ['sum', 'mean', 'max', 'min', 'std'],\n        'installments': ['sum', 'mean', 'max', 'min', 'std'],\n        'purchase_date': [np.ptp],\n        'month_lag': ['min', 'max']\n    }\n    agg_trans = trans.groupby(['card_id']).agg(agg_func)\n    agg_trans.columns = [prefix + '_'.join(col).strip() \n                           for col in agg_trans.columns.values]\n    agg_trans.reset_index(inplace=True)\n    \n    df = (trans.groupby('card_id')\n          .size()\n          .reset_index(name='{}transactions_count'.format(prefix)))\n    \n    agg_trans = pd.merge(df, agg_trans, on='card_id', how='left')\n    \n    return agg_trans\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"31956c30b5a2ebadee300181dad44f702a8df644"},"cell_type":"code","source":"%%time\nimport gc\nmerch_hist = aggregate_transactions(hist_trans, prefix='hist_')\ndel hist_trans\ngc.collect()\ntrain = pd.merge(train, merch_hist, on='card_id',how='left')\ntest = pd.merge(test, merch_hist, on='card_id',how='left')\ndel merch_hist\ngc.collect()\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8de5b5e1a8c113cc8db49cbe1fcc50a96ff6e296"},"cell_type":"code","source":"%%time\nnew_trans = pd.read_csv(\"../input/new_merchant_transactions.csv\")\nnew_trans.head()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5bb03bbf6d0eb2ca17e2520324abca00a6fe0f31"},"cell_type":"code","source":"%%time\nnew_trans = pd.get_dummies(new_trans, columns=['category_2', 'category_3'])\nnew_trans['authorized_flag'] = new_trans['authorized_flag'].map({'Y': 1, 'N': 0})\nnew_trans['category_1'] = new_trans['category_1'].map({'Y': 1, 'N': 0})\nnew_trans.head()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4e8452298c476c8f42e811b13152158136f13518"},"cell_type":"code","source":"%%time\nmerch_new = aggregate_transactions(new_trans, prefix='new_')\ndel new_trans\ngc.collect()\ntrain = pd.merge(train, merch_new, on='card_id',how='left')\ntest = pd.merge(test, merch_new, on='card_id',how='left')\ndel merch_new\ngc.collect()\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9cf73722d0a9a6bcdae516feae7d3fa2ce81f611"},"cell_type":"code","source":"%%time\ntarget = train['target']\ndrops = ['card_id', 'first_active_month', 'target']\nuse_cols = [c for c in train.columns if c not in drops]\nfeatures = list(train[use_cols].columns)\ntrain[features].head()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"db3ca3347698bc09c43717ae4b61aa68d9497321"},"cell_type":"code","source":"print(train[features].shape)\nprint(test[features].shape)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"678bd414b0ced69455a241c0e59fc9d6dc075347"},"cell_type":"code","source":"%%time\ntrain[features+['target']].to_csv('new_train.csv', index=False)\ntest[features].to_csv('new_test.csv', index=False)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"09a3eb12540e359250a85c570e7aedf6b87a648b"},"cell_type":"markdown","source":"# Modeling\n\nNow let's do some of what everyone is here for - modeling. We'll start with a simple Ridge regression model. "},{"metadata":{"trusted":true,"_uuid":"97579d5a922792beaf9f60e63d7af5e17db3abb6"},"cell_type":"code","source":"%%time\nfolds = KFold(n_splits=7, shuffle=True, random_state=15)\noof_ridge = np.zeros(train.shape[0])\npredictions_ridge = np.zeros(test.shape[0])\n\ntst_data = test.copy()\ntst_data.fillna((tst_data.mean()), inplace=True)\n\ntst_data = tst_data[features].values\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train, target)):\n    print(\"fold n°{}\".format(fold_+1))\n    trn_data, trn_y = train.iloc[trn_idx][features], target.iloc[trn_idx].values\n    val_data, val_y = train.iloc[val_idx][features], target.iloc[val_idx].values\n    \n    trn_data.fillna((trn_data.mean()), inplace=True)\n    val_data.fillna((val_data.mean()), inplace=True)\n    \n    trn_data = trn_data.values\n    val_data = val_data.values\n\n    clf = Ridge(alpha=100)\n    clf.fit(trn_data, trn_y)\n    \n    oof_ridge[val_idx] = clf.predict(val_data)\n    predictions_ridge += clf.predict(tst_data) / folds.n_splits\n\nnp.save('oof_ridge', oof_ridge)\nnp.save('predictions_ridge', predictions_ridge)\nnp.sqrt(mean_squared_error(target.values, oof_ridge))\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b8f360fbd758914d4f6061331fbc336c95684b41"},"cell_type":"markdown","source":"3.83 CV is not bad, but it's far from what the best models can do in this competition. Let's take a look at a few non-linear models. We'll start with LightGBM."},{"metadata":{"trusted":true,"_uuid":"98b9d3e3c675e410c3681fc3750d45f80544492c"},"cell_type":"code","source":"%%time\nparam = {'num_leaves': 50,\n         'min_data_in_leaf': 30, \n         'objective':'regression',\n         'max_depth': -1,\n         'learning_rate': 0.005,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 0.1,\n         \"verbosity\": -1}\n\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof_lgb = np.zeros(len(train))\npredictions_lgb = np.zeros(len(test))\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train.values, target.values)):\n    print('-')\n    print(\"Fold {}\".format(fold_ + 1))\n    trn_data = lgb.Dataset(train.iloc[trn_idx][features], label=target.iloc[trn_idx])\n    val_data = lgb.Dataset(train.iloc[val_idx][features], label=target.iloc[val_idx])\n\n    num_round = 10000\n    clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=100, early_stopping_rounds=100)\n    oof_lgb[val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n    predictions_lgb += clf.predict(test[features], num_iteration=clf.best_iteration) / folds.n_splits\n    \nnp.save('oof_lgb', oof_lgb)\nnp.save('predictions_lgb', predictions_lgb)\nnp.sqrt(mean_squared_error(target.values, oof_lgb))\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"65147cbc994a4fc474eac2b73f4db55756ab68b5"},"cell_type":"markdown","source":"Now soem XGBoost:"},{"metadata":{"trusted":true,"_uuid":"252970e8696dcb0dc1193fb222f3d6173b30d4af"},"cell_type":"code","source":"%%time\nxgb_params = {'eta': 0.005, 'max_depth': 10, 'subsample': 0.8, 'colsample_bytree': 0.8, \n          'objective': 'reg:linear', 'eval_metric': 'rmse', 'silent': True}\n\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof_xgb = np.zeros(len(train))\npredictions_xgb = np.zeros(len(test))\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train.values, target.values)):\n    print('-')\n    print(\"Fold {}\".format(fold_ + 1))\n    trn_data = xgb.DMatrix(data=train.iloc[trn_idx][features], label=target.iloc[trn_idx])\n    val_data = xgb.DMatrix(data=train.iloc[val_idx][features], label=target.iloc[val_idx])\n    watchlist = [(trn_data, 'train'), (val_data, 'valid')]\n    print(\"xgb \" + str(fold_) + \"-\" * 50)\n    num_round = 10000\n    xgb_model = xgb.train(xgb_params, trn_data, num_round, watchlist, early_stopping_rounds=50, verbose_eval=1000)\n    oof_xgb[val_idx] = xgb_model.predict(xgb.DMatrix(train.iloc[val_idx][features]), ntree_limit=xgb_model.best_ntree_limit+50)\n\n    predictions_xgb += xgb_model.predict(xgb.DMatrix(test[features]), ntree_limit=xgb_model.best_ntree_limit+50) / folds.n_splits\n    \nnp.save('oof_xgb', oof_xgb)\nnp.save('predictions_xgb', predictions_xgb)\nnp.sqrt(mean_squared_error(target.values, oof_xgb))\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"35c03107285a38e0a6e922981955c9561c7e3730"},"cell_type":"markdown","source":"# Second Set of Features and Models"},{"metadata":{"_uuid":"79bd92bb0ac1af6bc1a8bea752d4fb1d82b97453"},"cell_type":"markdown","source":"A great thing about stackign is that you can not only use different set of models, but also create the same models with a different set of features. Here we'll use features from the wonderful [Elo world kernel](https://www.kaggle.com/fabiendaniel/elo-world) by Fabien Daniel:"},{"metadata":{"trusted":true,"_uuid":"b488889ef8e93f2c3fe9d0fa7b1bb00d23bd1689"},"cell_type":"code","source":"%%time\ndel train, test\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"793663bd6548291d911ae9c8c0a2a2d69ae82a28"},"cell_type":"code","source":"new_transactions = pd.read_csv('../input/new_merchant_transactions.csv', parse_dates=['purchase_date'])\nhistorical_transactions = pd.read_csv('../input/historical_transactions.csv', parse_dates=['purchase_date'])\n\ndef binarize(df):\n    for col in ['authorized_flag', 'category_1']:\n        df[col] = df[col].map({'Y':1, 'N':0})\n    return df\n\nhistorical_transactions = binarize(historical_transactions)\nnew_transactions = binarize(new_transactions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"19a32b2ee3e55eaecee39ce4e543c89f8920cf8d"},"cell_type":"code","source":"def read_data(input_file):\n    df = pd.read_csv(input_file)\n    df['first_active_month'] = pd.to_datetime(df['first_active_month'])\n    df['elapsed_time'] = (datetime.date(2018, 2, 1) - df['first_active_month'].dt.date).dt.days\n    return df\n#_________________________________________\ntrain = read_data('../input/train.csv')\ntest = read_data('../input/test.csv')\n\ntarget = train['target']\ndel train['target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f877e97eb056fc7209a1ea5fed3a827674e4e8f0"},"cell_type":"code","source":"historical_transactions = pd.get_dummies(historical_transactions, columns=['category_2', 'category_3'])\nnew_transactions = pd.get_dummies(new_transactions, columns=['category_2', 'category_3'])\n\nhistorical_transactions = reduce_mem_usage(historical_transactions)\nnew_transactions = reduce_mem_usage(new_transactions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0b6a5d45a038e5b0a275f43e3366e1cc80f33065"},"cell_type":"code","source":"historical_transactions['purchase_month'] = historical_transactions['purchase_date'].dt.month\nnew_transactions['purchase_month'] = new_transactions['purchase_date'].dt.month\n\ndef aggregate_transactions(history):\n    \n    history.loc[:, 'purchase_date'] = pd.DatetimeIndex(history['purchase_date']).\\\n                                      astype(np.int64) * 1e-9\n    \n    agg_func = {\n        'authorized_flag': ['sum', 'mean'],\n        'category_1': ['sum', 'mean'],\n        'category_2_1.0': ['mean'],\n        'category_2_2.0': ['mean'],\n        'category_2_3.0': ['mean'],\n        'category_2_4.0': ['mean'],\n        'category_2_5.0': ['mean'],\n        'category_3_A': ['mean'],\n        'category_3_B': ['mean'],\n        'category_3_C': ['mean'],\n        'merchant_id': ['nunique'],\n        'merchant_category_id': ['nunique'],\n        'state_id': ['nunique'],\n        'city_id': ['nunique'],\n        'subsector_id': ['nunique'],\n        'purchase_amount': ['sum', 'mean', 'max', 'min', 'std'],\n        'installments': ['sum', 'mean', 'max', 'min', 'std'],\n        'purchase_month': ['mean', 'max', 'min', 'std'],\n        'purchase_date': [np.ptp],\n        'month_lag': ['min', 'max']\n        }\n    \n    agg_history = history.groupby(['card_id']).agg(agg_func)\n    agg_history.columns = ['_'.join(col).strip() for col in agg_history.columns.values]\n    agg_history.reset_index(inplace=True)\n    \n    df = (history.groupby('card_id')\n          .size()\n          .reset_index(name='transactions_count'))\n    \n    agg_history = pd.merge(df, agg_history, on='card_id', how='left')\n    \n    return agg_history\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ac351f80ecf8a0bade99831a1a065e3b863f1b4b"},"cell_type":"code","source":"history = aggregate_transactions(historical_transactions)\nhistory.columns = ['hist_' + c if c != 'card_id' else c for c in history.columns]\nhistory[:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"000c010057df8afa6e07a59f2bf8961b897e6875"},"cell_type":"code","source":"new = aggregate_transactions(new_transactions)\nnew.columns = ['new_' + c if c != 'card_id' else c for c in new.columns]\nnew[:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"842b8f1fb1d900487a17b79220760dc0805b5bb3"},"cell_type":"code","source":"def aggregate_per_month(history):\n    grouped = history.groupby(['card_id', 'month_lag'])\n\n    agg_func = {\n            'purchase_amount': ['count', 'sum', 'mean', 'min', 'max', 'std'],\n            'installments': ['count', 'sum', 'mean', 'min', 'max', 'std'],\n            }\n\n    intermediate_group = grouped.agg(agg_func)\n    intermediate_group.columns = ['_'.join(col).strip() for col in intermediate_group.columns.values]\n    intermediate_group.reset_index(inplace=True)\n\n    final_group = intermediate_group.groupby('card_id').agg(['mean', 'std'])\n    final_group.columns = ['_'.join(col).strip() for col in final_group.columns.values]\n    final_group.reset_index(inplace=True)\n    \n    return final_group\n#___________________________________________________________\nfinal_group =  aggregate_per_month(historical_transactions) \nfinal_group[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"17713cb3da3825e06eb4f2235f8ccdc2395c3a2c"},"cell_type":"code","source":"train = pd.merge(train, history, on='card_id', how='left')\ntest = pd.merge(test, history, on='card_id', how='left')\n\ntrain = pd.merge(train, new, on='card_id', how='left')\ntest = pd.merge(test, new, on='card_id', how='left')\n\ntrain = pd.merge(train, final_group, on='card_id')\ntest = pd.merge(test, final_group, on='card_id')\n\nfeatures = [c for c in train.columns if c not in ['card_id', 'first_active_month']]\ncategorical_feats = [c for c in features if 'feature_' in c]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0c45ac2382a09c0c6eca21d42fcf42797873d4bb"},"cell_type":"code","source":"train[features+['target']].to_csv('new_train_2.csv', index=False)\ntest[features].to_csv('new_test_2.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1e71d688a44bea39da2154d5384f3b98ff3646f8"},"cell_type":"code","source":"param = {'num_leaves': 50,\n         'min_data_in_leaf': 30, \n         'objective':'regression',\n         'max_depth': -1,\n         'learning_rate': 0.005,\n         \"min_child_samples\": 20,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9 ,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 0.1,\n         \"verbosity\": -1}\n\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof_lgb_2 = np.zeros(len(train))\npredictions_lgb_2 = np.zeros(len(test))\nstart = time.time()\nfeature_importance_df = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train.values, target.values)):\n    print(\"fold n°{}\".format(fold_))\n    trn_data = lgb.Dataset(train.iloc[trn_idx][features], label=target.iloc[trn_idx], categorical_feature=categorical_feats)\n    val_data = lgb.Dataset(train.iloc[val_idx][features], label=target.iloc[val_idx], categorical_feature=categorical_feats)\n\n    num_round = 10000\n    clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=100, early_stopping_rounds = 200)\n    oof_lgb_2[val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n    \n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"feature\"] = features\n    fold_importance_df[\"importance\"] = clf.feature_importance()\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n    \n    predictions_lgb_2 += clf.predict(test[features], num_iteration=clf.best_iteration) / folds.n_splits\n\nnp.save('oof_lgb_2', oof_lgb_2)\nnp.save('predictions_lgb_2', predictions_lgb_2)\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(oof_lgb_2, target)**0.5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3959dd88dd6894a6fb913c627596debd315b4e83"},"cell_type":"code","source":"'''xgb_params = {'eta': 0.005, 'max_depth': 10, 'subsample': 0.8, 'colsample_bytree': 0.8, \n          'objective': 'reg:linear', 'eval_metric': 'rmse', 'silent': True}\n\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof_xgb_2 = np.zeros(len(train))\npredictions_xgb_2 = np.zeros(len(test))\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train.values, target.values)):\n    print('-')\n    print(\"Fold {}\".format(fold_ + 1))\n    trn_data = xgb.DMatrix(data=train.iloc[trn_idx][features], label=target.iloc[trn_idx])\n    val_data = xgb.DMatrix(data=train.iloc[val_idx][features], label=target.iloc[val_idx])\n    watchlist = [(trn_data, 'train'), (val_data, 'valid')]\n    print(\"xgb \" + str(fold_) + \"-\" * 50)\n    num_round = 10000\n    xgb_model = xgb.train(xgb_params, trn_data, num_round, watchlist, early_stopping_rounds=50, verbose_eval=1000)\n    oof_xgb_2[val_idx] = xgb_model.predict(xgb.DMatrix(train.iloc[val_idx][features]), ntree_limit=xgb_model.best_ntree_limit+50)\n\n    predictions_xgb_2 += xgb_model.predict(xgb.DMatrix(test[features]), ntree_limit=xgb_model.best_ntree_limit+50) / folds.n_splits\n    \nnp.save('oof_xgb_2', oof_xgb_2)\nnp.save('predictions_xgb_2', predictions_xgb_2)\nnp.sqrt(mean_squared_error(target.values, oof_xgb_2))'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ed9741e07adc6996765a3af75f0fedc0056e72bd"},"cell_type":"markdown","source":"Finally, we'll stack them all together:"},{"metadata":{"trusted":true,"_uuid":"c52e2b713e889385923667369646329f0f8e0bc7"},"cell_type":"code","source":"train_stack = np.vstack([oof_ridge, oof_lgb, oof_xgb, oof_lgb_2]).transpose()\ntest_stack = np.vstack([predictions_ridge, predictions_lgb, predictions_xgb, \n                        predictions_lgb_2]).transpose()\n\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(train_stack.shape[0])\npredictions = np.zeros(test_stack.shape[0])\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_stack, target)):\n    print(\"fold n°{}\".format(fold_))\n    trn_data, trn_y = train_stack[trn_idx], target.iloc[trn_idx].values\n    val_data, val_y = train_stack[val_idx], target.iloc[val_idx].values\n\n    clf = Ridge(alpha=1)\n    clf.fit(trn_data, trn_y)\n    \n    oof[val_idx] = clf.predict(val_data)\n    predictions += clf.predict(test_stack) / folds.n_splits\n\n\nnp.sqrt(mean_squared_error(target.values, oof))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5b25bb8b74f5e469439647a97f093e475b472dfd"},"cell_type":"code","source":"sample_submission = pd.read_csv('../input/sample_submission.csv')\nsample_submission['target'] = predictions\nsample_submission.to_csv('stacker_2.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}