{"cells":[{"metadata":{"_uuid":"3bce64ad51f654d35fcc98c7200691ab47983318"},"cell_type":"markdown","source":"This kernel is assignment of jiyelee teacher.  \n- Version1 : Baseline kernel \n- Version3 : Add some public kernel features \n    - https://www.kaggle.com/mfjwr1/simple-lightgbm-without-blending\n    - https://www.kaggle.com/roydatascience/elo-stack-with-goss-boosting"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import gc\nimport logging\nimport datetime\nimport warnings\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport lightgbm as lgb\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import StratifiedKFold, KFold\nfrom sklearn.metrics import mean_squared_error\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"377c68e8ceb410856ed22807109670276c338fd6"},"cell_type":"markdown","source":"The data size is big. So we use effectively RAM for running in kernel. \n- Reduce memory by chaning types ( e.g float64 -> float16 ). \n- Using Debug mode. "},{"metadata":{"trusted":true,"_uuid":"853222035a35943b9dea24d9f4d4e9d4280adffa"},"cell_type":"code","source":"# reduce memory\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2\n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)\n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n\n    return df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"86b1496b8d9b67495a039fe973aaac2a9bf1dc1f"},"cell_type":"markdown","source":"> Note: All data is simulated and fictitious, and is not real customer data\n- I think that this is important information. if you find some simulation secret then you got a medal. "},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"Debug = False\nif Debug:\n    train_df = reduce_mem_usage(pd.read_csv('../input/train.csv',nrows=100000))\n    test_df = reduce_mem_usage(pd.read_csv('../input/test.csv'))\n    historical_trans_df = reduce_mem_usage(pd.read_csv('../input/historical_transactions.csv',nrows=1000000))\n    new_merchant_trans_df = reduce_mem_usage(pd.read_csv('../input/new_merchant_transactions.csv',nrows=1000000))\nelse:\n    train_df = reduce_mem_usage(pd.read_csv('../input/train.csv'))\n    test_df = reduce_mem_usage(pd.read_csv('../input/test.csv'))\n    historical_trans_df = reduce_mem_usage(pd.read_csv('../input/historical_transactions.csv'))\n    new_merchant_trans_df = reduce_mem_usage(pd.read_csv('../input/new_merchant_transactions.csv'))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"901679cde5657ac7e051faccb4c7647243f7fe49"},"cell_type":"markdown","source":"why don't use `merchant.csv` ? this is 2 reasons. \n- 1. don't using in kaggle kernel. because of RAM Memory. \n- 2. don't improve many score. ( but someone using merchant.csv then improve 0.001 - https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77987 )\n\nnow find some insights and features. "},{"metadata":{"_uuid":"e2ae235568d319226cea30809ecd83458cec3408"},"cell_type":"markdown","source":"## Data Exploration"},{"metadata":{"trusted":true,"_uuid":"d2426196f9c7628c3236a090176f87910e04ceb0"},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"541829ea8cfa8958db7cae277d25e0ede46ecd69"},"cell_type":"code","source":"train_df.columns.tolist()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0c2dc5dcc4fdbdabb4238f6f8dbb42f5899d9e2a"},"cell_type":"markdown","source":"The main data `train` has 6 values. 'first_active_month', 'card_id', 'feature_1', 'feature_2', 'feature_3', 'target'. \n- first_active_month : This is `active_month` for card_id. it is not mean that first buy in historical_merchant.csv. \n- feature_1,2,3 : it is key important but hidden meaning. \n- target : Loyalty numerical score calculated 2 months after `historical` and `evaluation period` \n\n**You remember target calculated 2 months after `historical` and `evaluation period` **"},{"metadata":{"trusted":true,"_uuid":"9a568515dab52d9c0dfda4ddeb478d030a7c0888"},"cell_type":"code","source":"#histogram\nf, ax = plt.subplots(figsize=(14, 6))\nsns.distplot(train_df['target'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba2389ad99c638530a603add63165f5280b2e487"},"cell_type":"markdown","source":"Target has normal distribution but has many **outliers**. So it is issue that \n- Q1. Why -33 calculated? \n- Q2. How to deal this values ?\n\nQ1. Why -33 calculated ? \nA. I don't know the reason. Do you know it?\n\nQ2. How to deal this values ?\nA. Someone deal with it. \n- 1. waitingli : Post - processing (https://www.kaggle.com/waitingli/combining-your-model-with-a-model-without-outlier - Very nice kernel. \n- 2. Aleksandr Kosolapov\n : Sort values - (https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/78903#469047)"},{"metadata":{"trusted":true,"_uuid":"e087b0c4db3ca512c7ca531241a908f579478171"},"cell_type":"code","source":"train_df.groupby(['feature_1','feature_2','feature_3'])['target'].agg({'min','mean','max','std','skew'})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9bbf9addbd500491f88c916554918e830af8843"},"cell_type":"markdown","source":"Hmm.. I don't know the relationship of features_. but feature1 - feature 3 has relationship. ( if feature_1 are 1,2,3 then feature_3 is 0. else if feature_1 are 2,4 then feature_3 is 1 ) Someone try FE with relationship of features like below. or Using FFM. "},{"metadata":{"_uuid":"13dbb48cfa41ece85d9ab4b337f95d74caa5720d"},"cell_type":"markdown","source":"    #https://www.kaggle.com/mfjwr1/simple-lightgbm-without-blending\n    df['elapsed_time'] = (datetime.datetime.today() - df['first_active_month']).dt.days\n\n    df['days_feature1'] = df['elapsed_time'] * df['feature_1']\n    df['days_feature2'] = df['elapsed_time'] * df['feature_2']\n    df['days_feature3'] = df['elapsed_time'] * df['feature_3']\n\n    df['days_feature1_ratio'] = df['feature_1'] / df['elapsed_time']\n    df['days_feature2_ratio'] = df['feature_2'] / df['elapsed_time']\n    df['days_feature3_ratio'] = df['feature_3'] / df['elapsed_time']\n\n    # one hot encoding\n    df, cols = one_hot_encoder(df, nan_as_category=False)\n\n    for f in ['feature_1','feature_2','feature_3']:\n        order_label = df.groupby([f])['outliers'].mean()\n        df[f] = df[f].map(order_label)\n\n    df['feature_sum'] = df['feature_1'] + df['feature_2'] + df['feature_3']\n    df['feature_mean'] = df['feature_sum']/3\n    df['feature_max'] = df[['feature_1', 'feature_2', 'feature_3']].max(axis=1)\n    df['feature_min'] = df[['feature_1', 'feature_2', 'feature_3']].min(axis=1)\n    df['feature_var'] = df[['feature_1', 'feature_2', 'feature_3']].std(axis=1)"},{"metadata":{"_uuid":"5b3b08117afc1fa786c02ddfd92c609895f4eb34"},"cell_type":"markdown","source":"Now, let's see the historical_transactions.csv and new_merchant_transactions.csv. In the data description, \n```\nThe historical_transactions.csv and new_merchant_transactions.csv files contain information about each card's transactions. historical_transactions.csv contains up to 3 months' worth of transactions for every card at any of the provided merchant_ids. new_merchant_transactions.csv contains the transactions at new merchants (merchant_ids that this particular card_id has not yet visited) over a period of two months.\n``` \n\nBut new_merchant_transactions also has old merchant_id. (don't important)"},{"metadata":{"trusted":true,"_uuid":"3374eba6ca0b2c4d6a63b11db4b7387f53ba2e59"},"cell_type":"code","source":"historical_trans_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9e6c86a8fd8bc7889f2c17bb9e30000e98597aca"},"cell_type":"markdown","source":"- authorized_flag : Y/N - 'Y' if approved, 'N' if denied ( new_merchant_transactions has only Y , people deal with 1. divied authorized_flag - Y / N data frame 2. don't divied it )\n- city_id, state_id : the information of location. ( recruta42 : https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/76579#latest-465748 )\n- category_1,2,3 : the hidden features. but someone try conveal it. ( kyakovlev : https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/78839#latest-469163)\n- merchant_id,merchant_category_id, subsector_id : the information of merchant. ( merchant.csv has more information of merchant )\n- month_lag : month lag to reference date\n- purchase_amount : Normalized purchase amount. ( It is normalized value. so they has minus value. Also you can see the real number in this kernel. raddar : https://www.kaggle.com/raddar/towards-de-anonymizing-the-data-some-insights)\n- purchase_date : Purchase date"},{"metadata":{"trusted":true,"_uuid":"339f54c89fa5178b172965db55ce631334701430"},"cell_type":"code","source":"# new_merchant also same\nhistorical_trans_df.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a96b7a0a2dc83ce6f27cf6fff9f6b138bb00953"},"cell_type":"markdown","source":"## Data preprocessing \n- Based kernel : gpreda https://www.kaggle.com/gpreda/elo-world-high-score-without-blending"},{"metadata":{"trusted":true,"_uuid":"d818144ab146d23f57d24b74e4d2f9f4bffd70b9"},"cell_type":"code","source":"#process NA2 for transactions\nfor df in [historical_trans_df, new_merchant_trans_df]:\n    df['category_2'].fillna(1.0,inplace=True)\n    df['category_3'].fillna('A',inplace=True)\n    df['merchant_id'].fillna('M_ID_00a6ca8a8a',inplace=True)\n    \n#define function for aggregation\ndef create_new_columns(name,aggs):\n    return [name + '_' + k + '_' + agg for k in aggs.keys() for agg in aggs[k]]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b8fcc3fb83592c9f529c836ea96c0d6dde833b21"},"cell_type":"markdown","source":"## Feature Engineering \n- gpreda : https://www.kaggle.com/gpreda/elo-world-high-score-without-blending\n- mfjwr1 : https://www.kaggle.com/mfjwr1/simple-lightgbm-without-blending"},{"metadata":{"trusted":true,"_uuid":"34f63c0370c8ecf49fb0863a1598435d87c21dbb"},"cell_type":"code","source":"# The trick deal with outliers - Aleksandr Kosolapov (Rank 1st)\ntrain_df['rounded_target'] = train_df['target'].round(0)\ntrain_df = train_df.sort_values('rounded_target').reset_index(drop=True)\nvc = train_df['rounded_target'].value_counts()\nvc = dict(sorted(vc.items()))\ndf = pd.DataFrame()\ntrain_df['indexcol'],i = 0,1\nfor k,v in vc.items():\n    step = train_df.shape[0]/v\n    indent = train_df.shape[0]/(v+1)\n    df2 = train_df[train_df['rounded_target'] == k].sample(v, random_state=120).reset_index(drop=True)\n    for j in range(0, v):\n        df2.at[j, 'indexcol'] = indent + j*step + 0.000001*i\n    df = pd.concat([df2,df])\n    i+=1\ntrain_df = df.sort_values('indexcol', ascending=True).reset_index(drop=True)\ndel train_df['indexcol'], train_df['rounded_target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"99fc4db8d289b45b97817c36ca93fc3e098fc965"},"cell_type":"code","source":"for df in [historical_trans_df, new_merchant_trans_df]:\n    df['purchase_date'] = pd.to_datetime(df['purchase_date'])\n    df['year'] = df['purchase_date'].dt.year\n    df['weekofyear'] = df['purchase_date'].dt.weekofyear\n    df['month'] = df['purchase_date'].dt.month\n    df['dayofweek'] = df['purchase_date'].dt.dayofweek\n    df['weekend'] = (df.purchase_date.dt.weekday >=5).astype(int)\n    df['hour'] = df['purchase_date'].dt.hour\n    df['authorized_flag'] = df['authorized_flag'].map({'Y':1, 'N':0})\n    df['category_1'] = df['category_1'].map({'Y':1, 'N':0}) \n    df['category_3'] = df['category_3'].map({'A':0, 'B':1, 'C':2}) \n    df['month_diff'] = ((datetime.datetime.today() - df['purchase_date']).dt.days)//30\n    df['month_diff'] += df['month_lag']\n    df['price'] = df['purchase_amount'] / df['installments']\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4032c40378fe7d332c1f96f8d7b97c6e5a2c3379"},"cell_type":"code","source":"aggs = {}\n\nfor col in ['subsector_id','merchant_id','merchant_category_id', 'state_id', 'city_id']:\n    aggs[col] = ['nunique']\nfor col in ['month', 'hour', 'weekofyear', 'dayofweek']:\n    aggs[col] = ['nunique', 'mean', 'min', 'max']\n    \n   \naggs['purchase_amount'] = ['sum','max','min','mean','var', 'std']\naggs['installments'] = ['sum','max','min','mean','var', 'std']\naggs['purchase_date'] = ['max','min', 'nunique']\naggs['month_lag'] = ['max','min','mean','var','nunique']\naggs['month_diff'] = ['mean', 'min', 'max', 'var','nunique']\naggs['authorized_flag'] = ['sum', 'mean', 'nunique']\naggs['weekend'] = ['sum', 'mean', 'nunique']\naggs['year'] = ['nunique', 'mean']\naggs['category_1'] = ['sum', 'mean', 'min', 'max', 'nunique', 'std']\naggs['category_2'] = ['sum', 'mean', 'min', 'nunique', 'std']\naggs['category_3'] = ['sum', 'mean', 'min', 'nunique', 'std']\naggs['card_id'] = ['size', 'count']\naggs['price'] = ['min','mean','std','max']\n\n\nfor col in ['category_2','category_3']:\n    historical_trans_df[col+'_mean'] = historical_trans_df.groupby([col])['purchase_amount'].transform('mean')\n    historical_trans_df[col+'_min'] = historical_trans_df.groupby([col])['purchase_amount'].transform('min')\n    historical_trans_df[col+'_max'] = historical_trans_df.groupby([col])['purchase_amount'].transform('max')\n    historical_trans_df[col+'_sum'] = historical_trans_df.groupby([col])['purchase_amount'].transform('sum')\n    historical_trans_df[col+'_std'] = historical_trans_df.groupby([col])['purchase_amount'].transform('std')\n    aggs[col+'_mean'] = ['mean']    \n\nnew_columns = create_new_columns('hist',aggs)\nhistorical_trans_group_df = historical_trans_df.groupby('card_id').agg(aggs)\nhistorical_trans_group_df.columns = new_columns\nhistorical_trans_group_df.reset_index(drop=False,inplace=True)\nhistorical_trans_group_df['hist_purchase_date_diff'] = (historical_trans_group_df['hist_purchase_date_max'] - historical_trans_group_df['hist_purchase_date_min']).dt.days\nhistorical_trans_group_df['hist_purchase_date_average'] = historical_trans_group_df['hist_purchase_date_diff']/historical_trans_group_df['hist_card_id_size']\nhistorical_trans_group_df['hist_purchase_date_uptonow'] = (datetime.datetime.today() - historical_trans_group_df['hist_purchase_date_max']).dt.days\nhistorical_trans_group_df['hist_purchase_date_uptomin'] = (datetime.datetime.today() - historical_trans_group_df['hist_purchase_date_min']).dt.days\n\n#merge with train, test\ntrain_df = train_df.merge(historical_trans_group_df,on='card_id',how='left')\ntest_df = test_df.merge(historical_trans_group_df,on='card_id',how='left')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a9dbba4e608bd8154c3c8cb55175ae5b26c1347c"},"cell_type":"code","source":"del historical_trans_group_df; gc.collect()\n\n#define aggregations with new_merchant_trans_df \naggs = {}\nfor col in ['subsector_id','merchant_id','merchant_category_id','state_id', 'city_id']:\n    aggs[col] = ['nunique']\nfor col in ['month', 'hour', 'weekofyear', 'dayofweek']:\n    aggs[col] = ['nunique', 'mean', 'min', 'max']\n\n    \naggs['purchase_amount'] = ['sum','max','min','mean','var','std']\naggs['installments'] = ['sum','max','min','mean','var','std']\naggs['purchase_date'] = ['max','min', 'nunique']\naggs['month_lag'] = ['max','min','mean','var', 'nunique']\naggs['month_diff'] = ['mean', 'max', 'min', 'var','nunique']\naggs['weekend'] = ['sum', 'mean', 'nunique']\naggs['year'] = ['nunique', 'mean']\naggs['category_1'] = ['sum', 'mean', 'min', 'nunique']\naggs['category_2'] = ['sum', 'mean', 'min', 'nunique']\naggs['category_3'] = ['sum', 'mean', 'min', 'nunique']\naggs['card_id'] = ['size', 'count']\naggs['price'] = ['min','mean','std','max']\n\n\nfor col in ['category_2','category_3']:\n    new_merchant_trans_df[col+'_mean'] = new_merchant_trans_df.groupby([col])['purchase_amount'].transform('mean')\n    new_merchant_trans_df[col+'_min'] = new_merchant_trans_df.groupby([col])['purchase_amount'].transform('min')\n    new_merchant_trans_df[col+'_max'] = new_merchant_trans_df.groupby([col])['purchase_amount'].transform('max')\n    new_merchant_trans_df[col+'_sum'] = new_merchant_trans_df.groupby([col])['purchase_amount'].transform('sum')\n    new_merchant_trans_df[col+'_std'] = new_merchant_trans_df.groupby([col])['purchase_amount'].transform('std')\n    aggs[col+'_mean'] = ['mean']\n\nnew_columns = create_new_columns('new_hist',aggs)\nnew_merchant_trans_group_df = new_merchant_trans_df.groupby('card_id').agg(aggs)\nnew_merchant_trans_group_df.columns = new_columns\nnew_merchant_trans_group_df.reset_index(drop=False,inplace=True)\nnew_merchant_trans_group_df['new_hist_purchase_date_diff'] = (new_merchant_trans_group_df['new_hist_purchase_date_max'] - new_merchant_trans_group_df['new_hist_purchase_date_min']).dt.days\nnew_merchant_trans_group_df['new_hist_purchase_date_average'] = new_merchant_trans_group_df['new_hist_purchase_date_diff']/new_merchant_trans_group_df['new_hist_card_id_size']\nnew_merchant_trans_group_df['new_hist_purchase_date_uptonow'] = (datetime.datetime.today() - new_merchant_trans_group_df['new_hist_purchase_date_max']).dt.days\nnew_merchant_trans_group_df['new_hist_purchase_date_uptomin'] = (datetime.datetime.today() - new_merchant_trans_group_df['new_hist_purchase_date_min']).dt.days\n#merge with train, test\ntrain_df = train_df.merge(new_merchant_trans_group_df,on='card_id',how='left')\ntest_df = test_df.merge(new_merchant_trans_group_df,on='card_id',how='left')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"da5462682ee369550af93d1b7ea5ebb728a467d2"},"cell_type":"code","source":"#clean-up memory\ndel new_merchant_trans_group_df; gc.collect()\ndel historical_trans_df; gc.collect()\ndel new_merchant_trans_df; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"da75e178214a68ce8b0fb2b23a0fb3494cd0df1f"},"cell_type":"code","source":"train_df['outliers'] = 0\ntrain_df.loc[train_df['target'] < -30, 'outliers'] = 1\noutls = train_df['outliers'].value_counts()\nprint(\"Outliers: {}\".format(outls))\n\n## process both train and test\nfor df in [train_df, test_df]:\n    df['first_active_month'] = pd.to_datetime(df['first_active_month'])\n    df['dayofweek'] = df['first_active_month'].dt.dayofweek\n    df['weekofyear'] = df['first_active_month'].dt.weekofyear\n    df['dayofyear'] = df['first_active_month'].dt.dayofyear\n    df['quarter'] = df['first_active_month'].dt.quarter\n    df['is_month_start'] = df['first_active_month'].dt.is_month_start\n    df['month'] = df['first_active_month'].dt.month\n    df['elapsed_time'] = (datetime.datetime.today() - df['first_active_month']).dt.days\n    df['hist_first_buy'] = (df['hist_purchase_date_min'] - df['first_active_month']).dt.days\n    df['hist_last_buy'] = (df['hist_purchase_date_max'] - df['first_active_month']).dt.days\n    df['new_hist_first_buy'] = (df['new_hist_purchase_date_min'] - df['first_active_month']).dt.days\n    df['new_hist_last_buy'] = (df['new_hist_purchase_date_max'] - df['first_active_month']).dt.days\n    \n    for f in ['hist_purchase_date_max','hist_purchase_date_min','new_hist_purchase_date_max',\\\n                     'new_hist_purchase_date_min']:\n        df[f] = df[f].astype(np.int64) * 1e-9\n        \n    df['card_id_total'] = df['new_hist_card_id_size']+df['hist_card_id_size']\n    df['card_id_cnt_total'] = df['new_hist_card_id_count']+df['hist_card_id_count']\n    df['purchase_amount_total'] = df['new_hist_purchase_amount_sum']+df['hist_purchase_amount_sum']\n    df['purchase_amount_mean'] = df['new_hist_purchase_amount_mean']+df['hist_purchase_amount_mean']\n    df['purchase_amount_max'] = df['new_hist_purchase_amount_max']+df['hist_purchase_amount_max']\n    \n    df['days_feature1'] = df['elapsed_time'] * df['feature_1']\n    df['days_feature2'] = df['elapsed_time'] * df['feature_2']\n    df['days_feature3'] = df['elapsed_time'] * df['feature_3']\n\n    df['days_feature1_ratio'] = df['feature_1'] / df['elapsed_time']\n    df['days_feature2_ratio'] = df['feature_2'] / df['elapsed_time']\n    df['days_feature3_ratio'] = df['feature_3'] / df['elapsed_time']\n\nfor f in ['feature_1','feature_2','feature_3']:\n    order_label = train_df.groupby([f])['outliers'].mean()\n    train_df[f] = train_df[f].map(order_label)\n    test_df[f] = test_df[f].map(order_label)\n    \nfor df in [train_df,test_df]:\n    df['feature_sum'] = df['feature_1'] + df['feature_2'] + df['feature_3']\n    df['feature_mean'] = df['feature_sum']/3\n    df['feature_max'] = df[['feature_1', 'feature_2', 'feature_3']].max(axis=1)\n    df['feature_min'] = df[['feature_1', 'feature_2', 'feature_3']].min(axis=1)\n    df['feature_var'] = df[['feature_1', 'feature_2', 'feature_3']].std(axis=1)\n    \n##\ntrain_columns = [c for c in train_df.columns if c not in ['card_id', 'first_active_month','target','outliers']]\ntarget = train_df['target']\ndel train_df['target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"361c1f6404697901326dc155429795317db1c64e"},"cell_type":"code","source":"##model params\nparam = {'num_leaves': 51,\n         'min_data_in_leaf': 35, \n         'objective':'regression',\n         'max_depth': -1,\n         'learning_rate': 0.008,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.85,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.82,\n         \"bagging_seed\": 42,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 0.11,\n         \"verbosity\": -1,\n         \"nthread\": 4,\n         \"random_state\": 2019}\n\n#prepare fit model with cross-validation\nfolds = KFold(n_splits=9, shuffle=True, random_state=2019)\noof = np.zeros(len(train_df))\npredictions = np.zeros(len(test_df))\nfeature_importance_df = pd.DataFrame()\n#run model\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_df)):\n    strLog = \"fold {}\".format(fold_)\n    print(strLog)\n    trn_data = lgb.Dataset(train_df.iloc[trn_idx][train_columns], label=target.iloc[trn_idx])#, categorical_feature=categorical_feats)\n    val_data = lgb.Dataset(train_df.iloc[val_idx][train_columns], label=target.iloc[val_idx])#, categorical_feature=categorical_feats)\n\n    num_round = 10000\n    clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=200, early_stopping_rounds = 150)\n    oof[val_idx] = clf.predict(train_df.iloc[val_idx][train_columns], num_iteration=clf.best_iteration)\n    #feature importance\n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"Feature\"] = train_columns\n    fold_importance_df[\"importance\"] = clf.feature_importance()\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n    #predictions\n    predictions += clf.predict(test_df[train_columns], num_iteration=clf.best_iteration) / folds.n_splits\n    logger.info(strLog)\n    \nstrRMSE = \"\".format(np.sqrt(mean_squared_error(oof, target)))\nprint(strRMSE)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3a1fd100dc5da936901872fa241f77355408aa35"},"cell_type":"code","source":"##plot the feature importance\ncols = (feature_importance_df[[\"Feature\", \"importance\"]]\n        .groupby(\"Feature\")\n        .mean()\n        .sort_values(by=\"importance\", ascending=False)[:1000].index)\nbest_features = feature_importance_df.loc[feature_importance_df.Feature.isin(cols)]\n\nplt.figure(figsize=(14,26))\nsns.barplot(x=\"importance\", y=\"Feature\", data=best_features.sort_values(by=\"importance\",ascending=False))\nplt.title('LightGBM Features (averaged over folds)')\nplt.tight_layout()\nplt.savefig('lgbm_importances.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"043a15ebca1ba485a2ea967ca23c22921305519a"},"cell_type":"code","source":"sub_df = pd.DataFrame({\"card_id\":test_df[\"card_id\"].values})\nsub_df[\"target\"] = predictions\nsub_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be552f9ed8534979d43a6b96c0d96b13d113c7d2"},"cell_type":"markdown","source":"Next, You read a discussion and make a feature.\n\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75935#latest-468296 (Best Discussion. Thanks Yifan xie)\n- FFM : https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/76480\n- Counter Vectorizer : https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75034#latest-468611\n- Feature selection : https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73937#latest-459610\n- NN\n- Post Processing : https://www.kaggle.com/waitingli/combining-your-model-with-a-model-without-outlier"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}