{"cells":[{"metadata":{"_uuid":"b05307c6e9faf65e975fe38dbf6f133e6a68af8d"},"cell_type":"markdown","source":"## Overview\n\nThe purpose of this kernel is to take a look at the data, come up with some insights, and attempt to create a predictive model or two. This notebook is still **very** raw. I will work on it as my very limited time permits, and hope to expend it in the upcoming days and weeks.\n\nNB: Most of the feature engineering and some of the modeling is based on [Peter Hurford's excellent kernel](https://www.kaggle.com/peterhurford/you-re-going-to-want-more-categories-lb-3-737/notebook) , and [another one by Konrad Banachevitz.](https://www.kaggle.com/konradb/lgb-fe-lb-3-707)\n\n## Packages\n\nFirst, let's load a few useful Python packages. This section will keep growing in subsequent versions of this EDA."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb\nfrom sklearn.linear_model import Ridge\nimport time\nfrom sklearn import preprocessing\nimport warnings\nimport datetime\nwarnings.filterwarnings(\"ignore\")\nimport gc\nfrom tqdm import tqdm\n\nfrom scipy.stats import describe\n%matplotlib inline\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import mean_squared_error\nimport xgboost as xgb\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d95f16e2adc3092f173f84db8ae8e5e9741be587"},"cell_type":"code","source":"def reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cb7c6188e659ba49b11168b1154092d040f3fc93"},"cell_type":"markdown","source":"Let's see what files we have in the input directory:"},{"metadata":{"trusted":true,"_uuid":"fc756f5a07aa306835fe5e24e933a8c6b21e7d7c"},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"186afdd8cefab16d2a7fb3ffc2b178ef7b2c87a8"},"cell_type":"markdown","source":"We see that in addition to the usual,`train`, `test` and `sample_submission` files, we also have `merchants`, `historical_transactions`, `new_merchant_transactions`, and even one (**HORROR!!!**) excel file - `Data_Dictionary`. The names of the files are pretty self-explanatory, but we'll take a look at them and explore them. First, let's look at the `train` and `test` files."},{"metadata":{"trusted":true,"_uuid":"0c0c9c2f5f90bf0fc21aa8d664d0209ceb1e03fe"},"cell_type":"code","source":"#Loading Train and Test Data\ntrain = pd.read_csv(\"../input/train.csv\", parse_dates=[\"first_active_month\"])\ntest = pd.read_csv(\"../input/test.csv\", parse_dates=[\"first_active_month\"])\nprint(\"{} observations and {} features in train set.\".format(train.shape[0],train.shape[1]))\nprint(\"{} observations and {} features in test set.\".format(test.shape[0],test.shape[1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c1531fe28a8484caeb387ff3d485ad4bff98b270"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a38ec0dbedbf2163ff7b1bec3ffa27346d5c1921"},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fb61542f287afe3180e2af13dd8279dd4a2f3a42"},"cell_type":"markdown","source":"Seems farily straightforward - just ID, first active months, three anonimous features, and target firld for train set.\n\nLet's take a look at the target variable:\n"},{"metadata":{"trusted":true,"_uuid":"a5d7d3007de2483606a8b518f097917c18fd3e0e"},"cell_type":"code","source":"train.target.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"898f72ebac71a0cdb75d2a88376d3373766620ca"},"cell_type":"markdown","source":"Seems like a very wide range of values, relatively spaking. Let's take a look at the graph of the distribution:"},{"metadata":{"trusted":true,"_uuid":"bb28a8fd8e4293af7fbd7acc24f232c8ee311157"},"cell_type":"code","source":"plt.figure(figsize=(12, 5))\nplt.hist(train.target.values, bins=200)\nplt.title('Histogram target counts')\nplt.xlabel('Count')\nplt.ylabel('Target')\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cf99081474f63fbc24b477687f59c38d1a7fac85"},"cell_type":"markdown","source":"Seems like a pretty nice normal-looking distribution, except for the few anomalous elements at teh far left. They will have to be dealt with separately.\n\nLet's look at the \"violin\" version of the same plot. "},{"metadata":{"trusted":true,"_uuid":"259aca59eb9007cd05b480e9d5b857fe7168c89e"},"cell_type":"code","source":"sns.set_style(\"whitegrid\")\nax = sns.violinplot(x=train.target.values)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"107dd4730d422fc66b115c7e2a8044c82da06683"},"cell_type":"markdown","source":"Yup, there is that little bump on the far left again."},{"metadata":{"_uuid":"9459fdba4a9c005326dd8b7b63039fabf1865b0e"},"cell_type":"markdown","source":"Let's now look at the distributions of various \"features\""},{"metadata":{"trusted":true,"_uuid":"44433fb6a105bca6bcd6ca47773ddbf23dbaccaa"},"cell_type":"code","source":"plt.figure(figsize=(12, 5))\nplt.hist(train.feature_1.values, bins=200)\nplt.title('Histogram feature_1 counts')\nplt.xlabel('Count')\nplt.ylabel('Target')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5a27d2b4773131d41f2a4642cfa24392966eaf86"},"cell_type":"code","source":"plt.figure(figsize=(12, 5))\nplt.hist(train.feature_2.values, bins=200)\nplt.title('Histogram feature_2 counts')\nplt.xlabel('Count')\nplt.ylabel('Target')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c6688a59f2f563f716ddcfb33ad4df12539d75b1"},"cell_type":"code","source":"plt.figure(figsize=(12, 5))\nplt.hist(train.feature_3.values, bins=200)\nplt.title('Histogram feature_3 counts')\nplt.xlabel('Count')\nplt.ylabel('Target')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"59cd40c338ec080eee3816fd3301bdf8815a40e6"},"cell_type":"markdown","source":"For now I am not including plots for the test set, as they at first approsimation look very similar.\n\nA couple of things that stand out are:\n\n1. There are only a handful of values for each of the three features.\n2. They are discrete\n3. They are relatively eavenly distributed.\n\nAll of this suggests that these features are categorical and have been label-encoded. "},{"metadata":{"_uuid":"6aad5ddd0c91b3fca4da8f0f0070de38d3819b60"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"Here is a gratuitous embedding of YouTube video of 'The Girl From Ipanema'. For no good reason."},{"metadata":{"trusted":true,"_uuid":"ed75166ba8e25eb6a16c673ea8cee3f543166221"},"cell_type":"code","source":"from IPython.display import YouTubeVideo\nYouTubeVideo('UJkxFhFRFDA')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"05209ca8d97f761026845453145c3a5cd26b5b5c"},"cell_type":"markdown","source":"# Feature Engineering"},{"metadata":{"_uuid":"4c717c69db1c781442ad94c474a72694a60c7972"},"cell_type":"markdown","source":"This version uses outlier labeling and stratification from [this kernel](https://www.kaggle.com/chauhuynh/my-first-kernel-3-699)."},{"metadata":{"trusted":true,"_uuid":"0592adb0399adfb5725cfe9abfd73b9fe01ccf1a"},"cell_type":"code","source":"'''train['first_active_month'] = pd.to_datetime(train['first_active_month'])\ntest['first_active_month'] = pd.to_datetime(test['first_active_month'])\ntrain['elapsed_time'] = (datetime.date(2018, 2, 1) - train['first_active_month'].dt.date).dt.days\ntest['elapsed_time'] = (datetime.date(2018, 2, 1) - test['first_active_month'].dt.date).dt.days\n\ntrain['outliers'] = 0\ntrain.loc[train['target'] < -30, 'outliers'] = 1\n\n\n\ntarget = train['target']\ndel train['target']\n\ntrain.head()'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5cf61d938b3cd4455e03c2aef646f60a87cad825"},"cell_type":"code","source":"'''new_transactions = pd.read_csv('../input/new_merchant_transactions.csv', parse_dates=['purchase_date'])\nhistorical_transactions = pd.read_csv('../input/historical_transactions.csv', parse_dates=['purchase_date'])\n\ndef binarize(df):\n    for col in ['authorized_flag', 'category_1']:\n        df[col] = df[col].map({'Y':1, 'N':0})\n    return df\n\nhistorical_transactions = binarize(historical_transactions)\nnew_transactions = binarize(new_transactions)'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0b3bf9325c2c7a2ec89d9f782029e5c2aacbf81e"},"cell_type":"code","source":"'''historical_transactions = pd.get_dummies(historical_transactions, columns=['category_2', 'category_3'])\nnew_transactions = pd.get_dummies(new_transactions, columns=['category_2', 'category_3'])\n\nhistorical_transactions = reduce_mem_usage(historical_transactions)\nnew_transactions = reduce_mem_usage(new_transactions)\n\nagg_fun = {'authorized_flag': ['sum', 'mean']}\nauth_mean = historical_transactions.groupby(['card_id']).agg(agg_fun)\nauth_mean.columns = ['_'.join(col).strip() for col in auth_mean.columns.values]\nauth_mean.reset_index(inplace=True)\n\nauthorized_transactions = historical_transactions[historical_transactions['authorized_flag'] == 1]\nhistorical_transactions = historical_transactions[historical_transactions['authorized_flag'] == 0]'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"71c22f0abc1b90c285a78812acc5b5c6c57eb7c1"},"cell_type":"code","source":"'''historical_transactions['purchase_month'] = historical_transactions['purchase_date'].dt.month\nauthorized_transactions['purchase_month'] = authorized_transactions['purchase_date'].dt.month\nnew_transactions['purchase_month'] = new_transactions['purchase_date'].dt.month'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"73e7ce787e9e4884b309346ab22e82dce23a314b"},"cell_type":"code","source":"'''def aggregate_transactions(history):\n    \n    history.loc[:, 'purchase_date'] = pd.DatetimeIndex(history['purchase_date']).\\\n                                      astype(np.int64) * 1e-9\n    \n    agg_func = {\n        'category_1': ['sum', 'mean'],\n        'category_2_1.0': ['mean'],\n        'category_2_2.0': ['mean'],\n        'category_2_3.0': ['mean'],\n        'category_2_4.0': ['mean'],\n        'category_2_5.0': ['mean'],\n        'category_3_A': ['mean'],\n        'category_3_B': ['mean'],\n        'category_3_C': ['mean'],\n        'merchant_id': ['nunique'],\n        'merchant_category_id': ['nunique'],\n        'state_id': ['nunique'],\n        'city_id': ['nunique'],\n        'subsector_id': ['nunique'],\n        'purchase_amount': ['sum', 'mean', 'max', 'min', 'std'],\n        'installments': ['sum', 'mean', 'max', 'min', 'std'],\n        'purchase_month': ['mean', 'max', 'min', 'std'],\n        'purchase_date': [np.ptp, 'min', 'max'],\n        'month_lag': ['min', 'max']\n        }\n    \n    agg_history = history.groupby(['card_id']).agg(agg_func)\n    agg_history.columns = ['_'.join(col).strip() for col in agg_history.columns.values]\n    agg_history.reset_index(inplace=True)\n    \n    df = (history.groupby('card_id')\n          .size()\n          .reset_index(name='transactions_count'))\n    \n    agg_history = pd.merge(df, agg_history, on='card_id', how='left')\n    \n    return agg_history'''\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"31956c30b5a2ebadee300181dad44f702a8df644"},"cell_type":"code","source":"'''history = aggregate_transactions(historical_transactions)\nhistory.columns = ['hist_' + c if c != 'card_id' else c for c in history.columns]\nhistory[:5]'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8de5b5e1a8c113cc8db49cbe1fcc50a96ff6e296"},"cell_type":"code","source":"'''authorized = aggregate_transactions(authorized_transactions)\nauthorized.columns = ['auth_' + c if c != 'card_id' else c for c in authorized.columns]\nauthorized[:5]'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5bb03bbf6d0eb2ca17e2520324abca00a6fe0f31"},"cell_type":"code","source":"'''new = aggregate_transactions(new_transactions)\nnew.columns = ['new_' + c if c != 'card_id' else c for c in new.columns]\nnew[:5]'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cb92205c33911133947d4ff192b8e8d40e3732bd"},"cell_type":"code","source":"'''def aggregate_per_month(history):\n    grouped = history.groupby(['card_id', 'month_lag'])\n\n    agg_func = {\n            'purchase_amount': ['count', 'sum', 'mean', 'min', 'max', 'std'],\n            'installments': ['count', 'sum', 'mean', 'min', 'max', 'std'],\n            }\n\n    intermediate_group = grouped.agg(agg_func)\n    intermediate_group.columns = ['_'.join(col).strip() for col in intermediate_group.columns.values]\n    intermediate_group.reset_index(inplace=True)\n\n    final_group = intermediate_group.groupby('card_id').agg(['mean', 'std'])\n    final_group.columns = ['_'.join(col).strip() for col in final_group.columns.values]\n    final_group.reset_index(inplace=True)\n    \n    return final_group\n#___________________________________________________________\nfinal_group =  aggregate_per_month(historical_transactions) \nfinal_group[:10]'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4e8452298c476c8f42e811b13152158136f13518"},"cell_type":"code","source":"'''train = pd.merge(train, history, on='card_id', how='left')\ntest = pd.merge(test, history, on='card_id', how='left')\n\ntrain = pd.merge(train, authorized, on='card_id', how='left')\ntest = pd.merge(test, authorized, on='card_id', how='left')\n\ntrain = pd.merge(train, new, on='card_id', how='left')\ntest = pd.merge(test, new, on='card_id', how='left')\n\ntrain = pd.merge(train, final_group, on='card_id', how='left')\ntest = pd.merge(test, final_group, on='card_id', how='left')\n\ntrain = pd.merge(train, auth_mean, on='card_id', how='left')\ntest = pd.merge(test, auth_mean, on='card_id', how='left')'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3255c7adbc4c505636744d3cab82995c0082cafe"},"cell_type":"code","source":"'''del history, new, final_group, auth_mean, authorized\ngc.collect()\ngc.collect()'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9cf73722d0a9a6bcdae516feae7d3fa2ce81f611"},"cell_type":"code","source":"'''features = [c for c in train.columns if c not in ['card_id', 'first_active_month', 'outliers']]\ncategorical_feats = [c for c in features if 'feature_' in c]'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"411bf5f213870ebc8de3cb05d41d91bbd96e39f5"},"cell_type":"code","source":"'''print(train.shape)\nprint(test.shape)'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"678bd414b0ced69455a241c0e59fc9d6dc075347"},"cell_type":"code","source":"'''train.to_csv('train_1.csv', index=False)\ntest.to_csv('test_1.csv', index=False)\ntarget.to_csv('target.csv', index=False)'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"09a3eb12540e359250a85c570e7aedf6b87a648b"},"cell_type":"markdown","source":"# Modeling\n\nNow let's do some of what everyone is here for - modeling. We'll start with a simple Ridge regression model. "},{"metadata":{"trusted":true,"_uuid":"97579d5a922792beaf9f60e63d7af5e17db3abb6"},"cell_type":"code","source":"'''oof_ridge = np.zeros(train.shape[0])\npredictions_ridge = np.zeros(test.shape[0])\n\ntst_data = test.copy()\ntst_data.fillna((tst_data.mean()), inplace=True)\n\ntst_data = tst_data[features].values\n\nfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=15)\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train,train['outliers'].values)):\n    print(\"fold n°{}\".format(fold_+1))\n    trn_data, trn_y = train.iloc[trn_idx][features], target.iloc[trn_idx].values\n    val_data, val_y = train.iloc[val_idx][features], target.iloc[val_idx].values\n    \n    trn_data.fillna((trn_data.mean()), inplace=True)\n    val_data.fillna((val_data.mean()), inplace=True)\n    \n    trn_data = trn_data.values\n    val_data = val_data.values\n\n    clf = Ridge(alpha=100)\n    clf.fit(trn_data, trn_y)\n    \n    oof_ridge[val_idx] = clf.predict(val_data)\n    predictions_ridge += clf.predict(tst_data) / folds.n_splits\n\nnp.save('oof_ridge', oof_ridge)\nnp.save('predictions_ridge', predictions_ridge)\nnp.sqrt(mean_squared_error(target.values, oof_ridge))'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1efccf5394de86a9d09d23d26e0b629b5c46d265"},"cell_type":"code","source":"'''del tst_data\ngc.collect()'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b8f360fbd758914d4f6061331fbc336c95684b41"},"cell_type":"markdown","source":"3.78 CV is not bad, but it's far from what the best models can do in this competition. Let's take a look at a few non-linear models. We'll start with LightGBM."},{"metadata":{"trusted":true,"_uuid":"98b9d3e3c675e410c3681fc3750d45f80544492c"},"cell_type":"code","source":"'''param = {'num_leaves': 120,\n         'min_data_in_leaf': 30, \n         'objective':'regression',\n         'max_depth': -1,\n         'learning_rate': 0.005,\n         \"min_child_samples\": 30,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9 ,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 0.1,\n         \"verbosity\": -1}\n\noof_lgb = np.zeros(len(train))\npredictions_lgb = np.zeros(len(test))\n\nfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=15)\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train,train['outliers'].values)):    \n    print('-')\n    print(\"Fold {}\".format(fold_ + 1))\n    trn_data = lgb.Dataset(train.iloc[trn_idx][features], label=target.iloc[trn_idx])\n    val_data = lgb.Dataset(train.iloc[val_idx][features], label=target.iloc[val_idx])\n\n    num_round = 10000\n    clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=100, early_stopping_rounds=200)\n    oof_lgb[val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n    predictions_lgb += clf.predict(test[features], num_iteration=clf.best_iteration) / folds.n_splits\n    \nnp.save('oof_lgb', oof_lgb)\nnp.save('predictions_lgb', predictions_lgb)\nnp.sqrt(mean_squared_error(target.values, oof_lgb))'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"65147cbc994a4fc474eac2b73f4db55756ab68b5"},"cell_type":"markdown","source":"Now some XGBoost:"},{"metadata":{"trusted":true,"_uuid":"252970e8696dcb0dc1193fb222f3d6173b30d4af"},"cell_type":"code","source":"'''xgb_params = {'eta': 0.005, 'max_depth': 10, 'subsample': 0.8, 'colsample_bytree': 0.8, \n          'objective': 'reg:linear', 'eval_metric': 'rmse', 'silent': True}\n\noof_xgb = np.zeros(len(train))\npredictions_xgb = np.zeros(len(test))\n\nfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=15)\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train,train['outliers'].values)):\n    print('-')\n    print(\"Fold {}\".format(fold_ + 1))\n    trn_data = xgb.DMatrix(data=train.iloc[trn_idx][features], label=target.iloc[trn_idx])\n    val_data = xgb.DMatrix(data=train.iloc[val_idx][features], label=target.iloc[val_idx])\n    watchlist = [(trn_data, 'train'), (val_data, 'valid')]\n    print(\"xgb \" + str(fold_) + \"-\" * 50)\n    num_round = 10000\n    xgb_model = xgb.train(xgb_params, trn_data, num_round, watchlist, early_stopping_rounds=50, verbose_eval=1000)\n    oof_xgb[val_idx] = xgb_model.predict(xgb.DMatrix(train.iloc[val_idx][features]), ntree_limit=xgb_model.best_ntree_limit+50)\n\n    predictions_xgb += xgb_model.predict(xgb.DMatrix(test[features]), ntree_limit=xgb_model.best_ntree_limit+50) / folds.n_splits\n    \nnp.save('oof_xgb', oof_xgb)\nnp.save('predictions_xgb', predictions_xgb)\nnp.sqrt(mean_squared_error(target.values, oof_xgb))'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"35c03107285a38e0a6e922981955c9561c7e3730"},"cell_type":"markdown","source":"# Second Set of Features and Models"},{"metadata":{"_uuid":"79bd92bb0ac1af6bc1a8bea752d4fb1d82b97453"},"cell_type":"markdown","source":"A great thing about stackign is that you can not only use different set of models, but also create the same models with a different set of features. Here we'll use features from the wonderful [Elo world kernel](https://www.kaggle.com/fabiendaniel/elo-world) by Fabien Daniel:"},{"metadata":{"trusted":true,"_uuid":"b488889ef8e93f2c3fe9d0fa7b1bb00d23bd1689"},"cell_type":"code","source":"'''del train, test\ngc.collect()'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"793663bd6548291d911ae9c8c0a2a2d69ae82a28"},"cell_type":"code","source":"'''new_transactions = pd.read_csv('../input/new_merchant_transactions.csv', parse_dates=['purchase_date'])\nhistorical_transactions = pd.read_csv('../input/historical_transactions.csv', parse_dates=['purchase_date'])\n\ndef binarize(df):\n    for col in ['authorized_flag', 'category_1']:\n        df[col] = df[col].map({'Y':1, 'N':0})\n    return df\n\nhistorical_transactions = binarize(historical_transactions)\nnew_transactions = binarize(new_transactions)'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"19a32b2ee3e55eaecee39ce4e543c89f8920cf8d"},"cell_type":"code","source":"'''def read_data(input_file):\n    df = pd.read_csv(input_file)\n    df['first_active_month'] = pd.to_datetime(df['first_active_month'])\n    df['elapsed_time'] = (datetime.date(2018, 2, 1) - df['first_active_month'].dt.date).dt.days\n    return df\n#_________________________________________\ntrain = read_data('../input/train.csv')\ntest = read_data('../input/test.csv')\n\ntrain['outliers'] = 0\ntrain.loc[train['target'] < -30, 'outliers'] = 1\n\ntarget = train['target']\ndel train['target']'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f877e97eb056fc7209a1ea5fed3a827674e4e8f0"},"cell_type":"code","source":"'''historical_transactions = pd.get_dummies(historical_transactions, columns=['category_2', 'category_3'])\nnew_transactions = pd.get_dummies(new_transactions, columns=['category_2', 'category_3'])\n\nhistorical_transactions = reduce_mem_usage(historical_transactions)\nnew_transactions = reduce_mem_usage(new_transactions)'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0b6a5d45a038e5b0a275f43e3366e1cc80f33065"},"cell_type":"code","source":"'''historical_transactions['purchase_month'] = historical_transactions['purchase_date'].dt.month\nnew_transactions['purchase_month'] = new_transactions['purchase_date'].dt.month\n\ndef aggregate_transactions(history):\n    \n    history.loc[:, 'purchase_date'] = pd.DatetimeIndex(history['purchase_date']).\\\n                                      astype(np.int64) * 1e-9\n    \n    agg_func = {\n        'authorized_flag': ['sum', 'mean'],\n        'category_1': ['sum', 'mean'],\n        'category_2_1.0': ['mean'],\n        'category_2_2.0': ['mean'],\n        'category_2_3.0': ['mean'],\n        'category_2_4.0': ['mean'],\n        'category_2_5.0': ['mean'],\n        'category_3_A': ['mean'],\n        'category_3_B': ['mean'],\n        'category_3_C': ['mean'],\n        'merchant_id': ['nunique'],\n        'merchant_category_id': ['nunique'],\n        'state_id': ['nunique'],\n        'city_id': ['nunique'],\n        'subsector_id': ['nunique'],\n        'purchase_amount': ['sum', 'mean', 'max', 'min', 'std'],\n        'installments': ['sum', 'mean', 'max', 'min', 'std'],\n        'purchase_month': ['mean', 'max', 'min', 'std'],\n        'purchase_date': [np.ptp],\n        'month_lag': ['min', 'max']\n        }\n    \n    agg_history = history.groupby(['card_id']).agg(agg_func)\n    agg_history.columns = ['_'.join(col).strip() for col in agg_history.columns.values]\n    agg_history.reset_index(inplace=True)\n    \n    df = (history.groupby('card_id')\n          .size()\n          .reset_index(name='transactions_count'))\n    \n    agg_history = pd.merge(df, agg_history, on='card_id', how='left')\n    \n    return agg_history\n'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ac351f80ecf8a0bade99831a1a065e3b863f1b4b"},"cell_type":"code","source":"'''history = aggregate_transactions(historical_transactions)\nhistory.columns = ['hist_' + c if c != 'card_id' else c for c in history.columns]\nhistory[:5]'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"000c010057df8afa6e07a59f2bf8961b897e6875"},"cell_type":"code","source":"'''new = aggregate_transactions(new_transactions)\nnew.columns = ['new_' + c if c != 'card_id' else c for c in new.columns]\nnew[:5]'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"842b8f1fb1d900487a17b79220760dc0805b5bb3"},"cell_type":"code","source":"'''def aggregate_per_month(history):\n    grouped = history.groupby(['card_id', 'month_lag'])\n\n    agg_func = {\n            'purchase_amount': ['count', 'sum', 'mean', 'min', 'max', 'std'],\n            'installments': ['count', 'sum', 'mean', 'min', 'max', 'std'],\n            }\n\n    intermediate_group = grouped.agg(agg_func)\n    intermediate_group.columns = ['_'.join(col).strip() for col in intermediate_group.columns.values]\n    intermediate_group.reset_index(inplace=True)\n\n    final_group = intermediate_group.groupby('card_id').agg(['mean', 'std'])\n    final_group.columns = ['_'.join(col).strip() for col in final_group.columns.values]\n    final_group.reset_index(inplace=True)\n    \n    return final_group\n#___________________________________________________________\nfinal_group =  aggregate_per_month(historical_transactions) \nfinal_group[:10]'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"17713cb3da3825e06eb4f2235f8ccdc2395c3a2c"},"cell_type":"code","source":"'''train = pd.merge(train, history, on='card_id', how='left')\ntest = pd.merge(test, history, on='card_id', how='left')\n\ntrain = pd.merge(train, new, on='card_id', how='left')\ntest = pd.merge(test, new, on='card_id', how='left')\n\ntrain = pd.merge(train, final_group, on='card_id')\ntest = pd.merge(test, final_group, on='card_id')\n\nfeatures = [c for c in train.columns if c not in ['card_id', 'first_active_month', 'outliers']]\ncategorical_feats = [c for c in features if 'feature_' in c]'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4e8372df679b2b4f8502123526110d6fee5982ed"},"cell_type":"code","source":"'''del history, new, final_group\ngc.collect()\ngc.collect()'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0c45ac2382a09c0c6eca21d42fcf42797873d4bb"},"cell_type":"code","source":"'''train[features].to_csv('train_2.csv', index=False)\ntest[features].to_csv('test_2.csv', index=False)'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1e71d688a44bea39da2154d5384f3b98ff3646f8"},"cell_type":"code","source":"'''param = {'num_leaves': 50,\n         'min_data_in_leaf': 30, \n         'objective':'regression',\n         'max_depth': -1,\n         'learning_rate': 0.005,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 0.1,\n         \"verbosity\": -1}\n\noof_lgb_2 = np.zeros(len(train))\npredictions_lgb_2 = np.zeros(len(test))\nstart = time.time()\n\nfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=15)\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train,train['outliers'].values)):    \n    print(\"fold n°{}\".format(fold_))\n    trn_data = lgb.Dataset(train.iloc[trn_idx][features], label=target.iloc[trn_idx], categorical_feature=categorical_feats)\n    val_data = lgb.Dataset(train.iloc[val_idx][features], label=target.iloc[val_idx], categorical_feature=categorical_feats)\n\n    num_round = 10000\n    clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=100, early_stopping_rounds = 200)\n    oof_lgb_2[val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n    \n    predictions_lgb_2 += clf.predict(test[features], num_iteration=clf.best_iteration) / folds.n_splits\n\nnp.save('oof_lgb_2', oof_lgb_2)\nnp.save('predictions_lgb_2', predictions_lgb_2)\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(oof_lgb_2, target)**0.5))'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3959dd88dd6894a6fb913c627596debd315b4e83"},"cell_type":"code","source":"'''xgb_params = {'eta': 0.005, 'max_depth': 10, 'subsample': 0.8, 'colsample_bytree': 0.8, \n          'objective': 'reg:linear', 'eval_metric': 'rmse', 'silent': True}\n\noof_xgb_2 = np.zeros(len(train))\npredictions_xgb_2 = np.zeros(len(test))\n\nfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=15)\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train,train['outliers'].values)):    \n    print('-')\n    print(\"Fold {}\".format(fold_ + 1))\n    trn_data = xgb.DMatrix(data=train.iloc[trn_idx][features], label=target.iloc[trn_idx])\n    val_data = xgb.DMatrix(data=train.iloc[val_idx][features], label=target.iloc[val_idx])\n    watchlist = [(trn_data, 'train'), (val_data, 'valid')]\n    print(\"xgb \" + str(fold_) + \"-\" * 50)\n    num_round = 10000\n    xgb_model = xgb.train(xgb_params, trn_data, num_round, watchlist, early_stopping_rounds=50, verbose_eval=1000)\n    oof_xgb_2[val_idx] = xgb_model.predict(xgb.DMatrix(train.iloc[val_idx][features]), ntree_limit=xgb_model.best_ntree_limit+50)\n\n    predictions_xgb_2 += xgb_model.predict(xgb.DMatrix(test[features]), ntree_limit=xgb_model.best_ntree_limit+50) / folds.n_splits\n    \nnp.save('oof_xgb_2', oof_xgb_2)\nnp.save('predictions_xgb_2', predictions_xgb_2)\nnp.sqrt(mean_squared_error(target.values, oof_xgb_2))'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"54db36d566926c2dc3991d3a68dc0f5196299b87"},"cell_type":"code","source":"'''del train, test\ngc.collect()\ngc.collect()'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cafa167b2306711da153ebd2a603f6c96a574cb1"},"cell_type":"markdown","source":"Now we add another set of features from [this kernel](https://www.kaggle.com/chauhuynh/my-first-kernel-3-699)."},{"metadata":{"trusted":true,"_uuid":"7430584e38ad37941e7bf75bc4daaf0450aae1c7"},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')\nhist_trans = pd.read_csv('../input/historical_transactions.csv')\nnew_merchant_trans = pd.read_csv('../input/new_merchant_transactions.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"02fa9c91bc5cc35e1a3ed33f9a26c4bb37068f72"},"cell_type":"code","source":"for df in [hist_trans,new_merchant_trans]:\n    df['category_2'].fillna(1.0,inplace=True)\n    df['category_3'].fillna('A',inplace=True)\n    df['merchant_id'].fillna('M_ID_00a6ca8a8a',inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4d5541999852590388fc0ce0ef557bc151539ec1"},"cell_type":"code","source":"def get_new_columns(name,aggs):\n    return [name + '_' + k + '_' + agg for k in aggs.keys() for agg in aggs[k]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ee3c781ca91b67f5b4b61ce527e9e3265fe80146"},"cell_type":"code","source":"for df in [hist_trans,new_merchant_trans]:\n    df['purchase_date'] = pd.to_datetime(df['purchase_date'])\n    df['year'] = df['purchase_date'].dt.year\n    df['weekofyear'] = df['purchase_date'].dt.weekofyear\n    df['month'] = df['purchase_date'].dt.month\n    df['dayofweek'] = df['purchase_date'].dt.dayofweek\n    df['weekend'] = (df.purchase_date.dt.weekday >=5).astype(int)\n    df['hour'] = df['purchase_date'].dt.hour\n    df['authorized_flag'] = df['authorized_flag'].map({'Y':1, 'N':0})\n    df['category_1'] = df['category_1'].map({'Y':1, 'N':0}) \n    #https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73244\n    df['month_diff'] = ((datetime.datetime.today() - df['purchase_date']).dt.days)//30\n    df['month_diff'] += df['month_lag']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f4ec6c138f63e29462988a38ab64c0f04078f9c8"},"cell_type":"code","source":"aggs = {}\nfor col in ['month','hour','weekofyear','dayofweek','year','subsector_id','merchant_id','merchant_category_id']:\n    aggs[col] = ['nunique']\n\naggs['purchase_amount'] = ['sum','max','min','mean','var']\naggs['installments'] = ['sum','max','min','mean','var']\naggs['purchase_date'] = ['max','min']\naggs['month_lag'] = ['max','min','mean','var']\naggs['month_diff'] = ['mean']\naggs['authorized_flag'] = ['sum', 'mean']\naggs['weekend'] = ['sum', 'mean']\naggs['category_1'] = ['sum', 'mean']\naggs['card_id'] = ['size']\n\nfor col in ['category_2','category_3']:\n    hist_trans[col+'_mean'] = hist_trans.groupby([col])['purchase_amount'].transform('mean')\n    aggs[col+'_mean'] = ['mean']    \n\nnew_columns = get_new_columns('hist',aggs)\nhist_trans_group = hist_trans.groupby('card_id').agg(aggs)\nhist_trans_group.columns = new_columns\nhist_trans_group.reset_index(drop=False,inplace=True)\nhist_trans_group['hist_purchase_date_diff'] = (hist_trans_group['hist_purchase_date_max'] - hist_trans_group['hist_purchase_date_min']).dt.days\nhist_trans_group['hist_purchase_date_average'] = hist_trans_group['hist_purchase_date_diff']/hist_trans_group['hist_card_id_size']\nhist_trans_group['hist_purchase_date_uptonow'] = (datetime.datetime.today() - hist_trans_group['hist_purchase_date_max']).dt.days\ntrain = train.merge(hist_trans_group,on='card_id',how='left')\ntest = test.merge(hist_trans_group,on='card_id',how='left')\ndel hist_trans_group;gc.collect();gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6bc9390f943cbca04970321ce46410b40faba55d"},"cell_type":"code","source":"aggs = {}\nfor col in ['month','hour','weekofyear','dayofweek','year','subsector_id','merchant_id','merchant_category_id']:\n    aggs[col] = ['nunique']\naggs['purchase_amount'] = ['sum','max','min','mean','var']\naggs['installments'] = ['sum','max','min','mean','var']\naggs['purchase_date'] = ['max','min']\naggs['month_lag'] = ['max','min','mean','var']\naggs['month_diff'] = ['mean']\naggs['weekend'] = ['sum', 'mean']\naggs['category_1'] = ['sum', 'mean']\naggs['card_id'] = ['size']\n\nfor col in ['category_2','category_3']:\n    new_merchant_trans[col+'_mean'] = new_merchant_trans.groupby([col])['purchase_amount'].transform('mean')\n    aggs[col+'_mean'] = ['mean']\n    \nnew_columns = get_new_columns('new_hist',aggs)\nhist_trans_group = new_merchant_trans.groupby('card_id').agg(aggs)\nhist_trans_group.columns = new_columns\nhist_trans_group.reset_index(drop=False,inplace=True)\nhist_trans_group['new_hist_purchase_date_diff'] = (hist_trans_group['new_hist_purchase_date_max'] - hist_trans_group['new_hist_purchase_date_min']).dt.days\nhist_trans_group['new_hist_purchase_date_average'] = hist_trans_group['new_hist_purchase_date_diff']/hist_trans_group['new_hist_card_id_size']\nhist_trans_group['new_hist_purchase_date_uptonow'] = (datetime.datetime.today() - hist_trans_group['new_hist_purchase_date_max']).dt.days\ntrain = train.merge(hist_trans_group,on='card_id',how='left')\ntest = test.merge(hist_trans_group,on='card_id',how='left')\ndel hist_trans_group;gc.collect();gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d9ad3aa4b454cd524b5f604c480bbf512985a075"},"cell_type":"code","source":"del hist_trans;gc.collect()\ndel new_merchant_trans;gc.collect()\ntrain.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c5ef8962152b1ebe370c898f7afc6a86e6cc7d49"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a45875cc3bbff04b8ab710ef481f653cc2b77692"},"cell_type":"code","source":"train['outliers'] = 0\ntrain.loc[train['target'] < -30, 'outliers'] = 1\ntrain['outliers'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"25fc96558b7524e0da8448118df6355507cee057"},"cell_type":"code","source":"for df in [train,test]:\n    df['first_active_month'] = pd.to_datetime(df['first_active_month'])\n    df['dayofweek'] = df['first_active_month'].dt.dayofweek\n    df['weekofyear'] = df['first_active_month'].dt.weekofyear\n    df['month'] = df['first_active_month'].dt.month\n    df['elapsed_time'] = (datetime.datetime.today() - df['first_active_month']).dt.days\n    df['hist_first_buy'] = (df['hist_purchase_date_min'] - df['first_active_month']).dt.days\n    df['new_hist_first_buy'] = (df['new_hist_purchase_date_min'] - df['first_active_month']).dt.days\n    for f in ['hist_purchase_date_max','hist_purchase_date_min','new_hist_purchase_date_max',\\\n                     'new_hist_purchase_date_min']:\n        df[f] = df[f].astype(np.int64) * 1e-9\n    df['card_id_total'] = df['new_hist_card_id_size']+df['hist_card_id_size']\n    df['purchase_amount_total'] = df['new_hist_purchase_amount_sum']+df['hist_purchase_amount_sum']\n\nfor f in ['feature_1','feature_2','feature_3']:\n    order_label = train.groupby([f])['outliers'].mean()\n    train[f] = train[f].map(order_label)\n    test[f] = test[f].map(order_label)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0714a5c08053c6b16b0e63113c761fdc9c1ea3c0"},"cell_type":"code","source":"train_columns = [c for c in train.columns if c not in ['card_id', 'first_active_month','target','outliers']]\ntarget = train['target']\ndel train['target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7eb32658a1e730643733f99da8d52d45d4a559a5"},"cell_type":"code","source":"train[train_columns].to_csv('train_3.csv', index=False)\ntest[train_columns].to_csv('test_3.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4d3d4bae8142601d6e6df99cb657df66c8951196"},"cell_type":"code","source":"param = {'num_leaves': 31,\n         'min_data_in_leaf': 30, \n         'objective':'regression',\n         'max_depth': -1,\n         'learning_rate': 0.01,\n         \"min_child_samples\": 20,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9 ,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 0.1,\n         \"verbosity\": -1,\n         \"nthread\": 4,\n         \"random_state\": 4590}\n\noof_lgb_3 = np.zeros(len(train))\npredictions_lgb_3 = np.zeros(len(test))\nstart = time.time()\n\nfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=15)\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train,train['outliers'].values)):    \n    print(\"fold n°{}\".format(fold_))\n    trn_data = lgb.Dataset(train.iloc[trn_idx][train_columns], label=target.iloc[trn_idx])\n    val_data = lgb.Dataset(train.iloc[val_idx][train_columns], label=target.iloc[val_idx])\n\n    num_round = 10000\n    clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=100, early_stopping_rounds = 200)\n    oof_lgb_3[val_idx] = clf.predict(train.iloc[val_idx][train_columns], num_iteration=clf.best_iteration)\n    \n    predictions_lgb_3 += clf.predict(test[train_columns], num_iteration=clf.best_iteration) / folds.n_splits\n\nnp.save('oof_lgb_3', oof_lgb_3)\nnp.save('predictions_lgb_3', predictions_lgb_3)\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(oof_lgb_3, target)**0.5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"50353a20df2642a7c13b3314c4a131401c6d6ac7"},"cell_type":"code","source":"xgb_params = {'eta': 0.005, 'max_depth': 10, 'subsample': 0.8, 'colsample_bytree': 0.8, \n          'objective': 'reg:linear', 'eval_metric': 'rmse', 'silent': True}\n\noof_xgb_3 = np.zeros(len(train))\npredictions_xgb_3 = np.zeros(len(test))\n\nfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=15)\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train,train['outliers'].values)):    \n    print('-')\n    print(\"Fold {}\".format(fold_ + 1))\n    trn_data = xgb.DMatrix(data=train.iloc[trn_idx][train_columns], label=target.iloc[trn_idx])\n    val_data = xgb.DMatrix(data=train.iloc[val_idx][train_columns], label=target.iloc[val_idx])\n    watchlist = [(trn_data, 'train'), (val_data, 'valid')]\n    print(\"xgb \" + str(fold_) + \"-\" * 50)\n    num_round = 10000\n    xgb_model = xgb.train(xgb_params, trn_data, num_round, watchlist, early_stopping_rounds=50, verbose_eval=1000)\n    oof_xgb_3[val_idx] = xgb_model.predict(xgb.DMatrix(train.iloc[val_idx][train_columns]), ntree_limit=xgb_model.best_ntree_limit+50)\n\n    predictions_xgb_3 += xgb_model.predict(xgb.DMatrix(test[train_columns]), ntree_limit=xgb_model.best_ntree_limit+50) / folds.n_splits\n    \nnp.save('oof_xgb_3', oof_xgb_3)\nnp.save('predictions_xgb_3', predictions_xgb_3)\nnp.sqrt(mean_squared_error(target.values, oof_xgb_3))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ed9741e07adc6996765a3af75f0fedc0056e72bd"},"cell_type":"markdown","source":"Finally, we'll stack them all together:"},{"metadata":{"trusted":true,"_uuid":"c52e2b713e889385923667369646329f0f8e0bc7"},"cell_type":"code","source":"train_stack = np.vstack([oof_lgb_3, oof_xgb_3]).transpose()\ntest_stack = np.vstack([predictions_lgb_3, predictions_xgb_3]).transpose()\n\nfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(train_stack.shape[0])\npredictions = np.zeros(test_stack.shape[0])\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_stack, train['outliers'].values)):\n    print(\"fold n°{}\".format(fold_))\n    trn_data, trn_y = train_stack[trn_idx], target.iloc[trn_idx].values\n    val_data, val_y = train_stack[val_idx], target.iloc[val_idx].values\n\n    clf = Ridge(alpha=1)\n    clf.fit(trn_data, trn_y)\n    \n    oof[val_idx] = clf.predict(val_data)\n    predictions += clf.predict(test_stack) / folds.n_splits\n\n\nnp.sqrt(mean_squared_error(target.values, oof))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5b25bb8b74f5e469439647a97f093e475b472dfd"},"cell_type":"code","source":"sample_submission = pd.read_csv('../input/sample_submission.csv')\nsample_submission['target'] = predictions\nsample_submission.to_csv('stacker_4.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ae49760e6db0fa8cdc10ccbd59f564d1691f84f3"},"cell_type":"markdown","source":"To be continued ..."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}