{"cells":[{"metadata":{"_uuid":"2267e6060da09f816f32e1050dbb4ebca5eb70b2"},"cell_type":"markdown","source":"## General information\nThis kernel constists many usefull features (in our opinion). \nAlso this kernel need more then 16gb RAM while processing dataset (Didn't work at Kaggle). \nPart of the final solution was based on the following kernels:\n- [Elo EDA and models](https://www.kaggle.com/artgor/elo-eda-and-models/)\n- [Elo world](https://www.kaggle.com/fabiendaniel/elo-world)\n- [Simple Exploration Notebook - Elo](https://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-elo) \n![](https://storage.googleapis.com/kaggle-competitions/kaggle/10445/logos/thumb76_76.png?t=2018-10-24-17-14-05)"},{"metadata":{"trusted":false,"_uuid":"48c24b0b214713e4443521eef1ee497dc1a5ff92"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nfrom sklearn.preprocessing import LabelEncoder\nimport gc\nimport os\n\nimport re\n\nfrom sklearn.model_selection import KFold, RepeatedKFold\n\nimport time\nimport lightgbm as lgb\n\nfrom sklearn.metrics import mean_squared_error,roc_auc_score\nfrom sklearn.ensemble import ExtraTreesRegressor, RandomForestRegressor\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import OneHotEncoder,StandardScaler,MinMaxScaler,RobustScaler\nfrom sklearn.model_selection import GridSearchCV\nfrom catboost import CatBoostRegressor, Pool, CatBoost\nfrom sklearn.linear_model import LinearRegression,BayesianRidge\nfrom boruta import BorutaPy\n#from keras.utils import to_categorical\n\nimport matplotlib.pyplot as plt \nimport seaborn as sns \nfrom pylab import rcParams\n\nimport datetime\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nplt.style.use('ggplot')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"259cb87b8d38bae487777b654aa55d319dec5e48"},"cell_type":"markdown","source":"## Functions\nHere you can see functions, that we used to change data format and clar/prepare it.\n- model_lgb - simple lgbm model function\n- rmse - RMSE metric calculation function\n- replace_id - function for card_id replacement to number - it's usefull for the first iteration.\n- replace_month - function to make some time related features like month_id and elapsed_time\n- dist_holiday - get from [this kernel](https://www.kaggle.com/cttsai/simple-xgboost-cv-bagging)"},{"metadata":{"trusted":false,"_uuid":"53de53680b6df19f2906d43816a826efda4c95f8"},"cell_type":"code","source":"def rmse(y_true,y_pred):\n return np.sqrt(np.power(y_true-y_pred,2).sum()/len(y_true))\n\ndef replace_id(df,dictionary = False):\n \n df['card_id2'] = df.card_id.str.replace('C','')\n df['card_id2'] = df.card_id2.str.replace('_','')\n df['card_id2'] = df.card_id2.str.replace('I','')\n df['card_id2'] = df.card_id2.str.replace('D','')\n df['card_id2'] = df.card_id2.str.replace('a','10')\n df['card_id2'] = df.card_id2.str.replace('b','11')\n df['card_id2'] = df.card_id2.str.replace('c','12')\n df['card_id2'] = df.card_id2.str.replace('d','13')\n df['card_id2'] = df.card_id2.str.replace('e','14')\n df['card_id2'] = df.card_id2.str.replace('f','15').astype(np.float64) \n \n if dictionary:\n d = df[['card_id','card_id2']]\n df.drop('card_id',axis=1,inplace=True)\n df = df.rename(columns = {'card_id2':'card_id'})\n return df,d\n df.drop('card_id',axis=1,inplace=True)\n df = df.rename(columns = {'card_id2':'card_id'})\n return df\n \n\ndef replace_month(df):\n df['month_id'] = df.first_active_month.str.replace(\"-\",'').fillna(201802).astype(int)\n \n df['first_active_month'] = pd.to_datetime(df['first_active_month'])\n df['elapsed_time'] = (datetime.date(2018, 8, 1) - df['first_active_month'].dt.date).dt.days\n return df\n\ndef reduce_mem_usage(df, verbose=True):\n numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n start_mem = df.memory_usage().sum() / 1024**2 \n for col in df.columns:\n col_type = df[col].dtypes\n if col_type in numerics:\n c_min = df[col].min()\n c_max = df[col].max()\n if str(col_type)[:3] == 'int':\n if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int8)\n elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n else:\n if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n df[col] = df[col].astype(np.float16)\n elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n df[col] = df[col].astype(np.float32)\n else:\n df[col] = df[col].astype(np.float64) \n end_mem = df.memory_usage().sum() / 1024**2\n if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n return df\n\ndef dist_holiday(df, col_name, date_holiday, date_ref, period=100):\n df[col_name] = np.maximum(np.minimum((pd.to_datetime(date_holiday) - df[date_ref]).dt.days, period), 0)\n \ndef submit(df,name,score):\n arr = []\n path = '../'\n for a in os.listdir(path):\n arr.append(a)\n arr = [re.sub(\"[^0-9_]\",'',a) for a in arr ]\n arr = [a.split('_') for a in arr if a not in ['']]\n arr = [int(a[0]) for a in arr if a[0] not in ['']]\n pre = str(max(arr))\n str_dat = str(time.localtime().tm_year)+'_'+str(time.localtime().tm_mon)+'_'+str(time.localtime().tm_mday)\n str_dat\n df[['card_id','target']].to_csv(path+pre+'_'+str_dat+'_CV_'+str(score)+'.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7e89f81367dfa8f38ef099657ab8c4086884abe4"},"cell_type":"markdown","source":"### Deanon features\n[This kernel](https://www.kaggle.com/raddar/towards-de-anonymizing-the-data-some-insights)"},{"metadata":{"trusted":false,"_uuid":"48bb638d955dd0917be39d3fbabd1b5edd32a8cf"},"cell_type":"code","source":"\ndef deanon_purchase(purchase):\n return np.round(purchase/0.00150265118 + 497.06,8)\n\ndef deanon_target(target):\n return np.exp2(target)\n\ndef anon_target(target):\n return np.log2(target)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"088b860300f24ef85209042fb2ef60baf5c8b0b9"},"cell_type":"markdown","source":"## Load data"},{"metadata":{"trusted":false,"_uuid":"716148cdb5821e8a763c632a703c6169b3eb565b"},"cell_type":"code","source":"%%time\ntrain = pd.read_csv('../inpt/train.csv')\ntest_final = pd.read_csv('../input/test.csv')\nhist = pd.read_csv('../input/historical_transactions.csv',parse_dates=[\"purchase_date\"])\nmerch = pd.read_csv('../input/merchants.csv')\nnew_t = pd.read_csv('../input/new_merchant_transactions.csv',parse_dates=[\"purchase_date\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"c016c33af36f9fcaae067c10f4ef360b03d12c07"},"cell_type":"code","source":"train = reduce_mem_usage(train)\ntest_final = reduce_mem_usage(test_final)\nhist = reduce_mem_usage(hist)\nmerch = reduce_mem_usage(merch)\nnew_t = reduce_mem_usage(new_t)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"91045aca4cc82e0b1ded1b0bf564214d3fa3f78b"},"cell_type":"code","source":"plt.hist(train['target']);","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"12d0c579277e12e4b4085495916ae977ccccaf21"},"cell_type":"code","source":"print('Sample with target less then -33: ',train[train.target < -33.].card_id.nunique(),'rows')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"da3978f9e17006b57e1ec7715c118ccc7006a84e"},"cell_type":"markdown","source":"Here we can see, that we have many target near zero and some strange target with value -33.21928095. \nFor RMSE this values can be very important. \nI take this card_id into separate variable to use this in a future"},{"metadata":{"trusted":false,"_uuid":"28c45c108ea0b033ee92a6a4f109e219bea8984d"},"cell_type":"code","source":"card_with_large_negative = train[train.target < -33.].card_id.unique()\nprint('Cards with large negative:',train[train.target < -33.].card_id.nunique())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"07e1c629fa0b6bc47cc1649a4d87c958c682db94"},"cell_type":"markdown","source":"## Feature engineering"},{"metadata":{"trusted":false,"_uuid":"8ac39d6e36b540bf58dcaa21834851114ec2c3ea"},"cell_type":"code","source":"%%time\ntrain = replace_month(train)\ntest_final = replace_month(test_final)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4c9414030a3214824c3364935cc482a7988d2ce4"},"cell_type":"markdown","source":"### Merchant.csv\nIn many kernel we can see this file is useless, but let's try to get some features from it."},{"metadata":{"trusted":false,"_uuid":"8152fb1455a4e1733e097f3f311075f284e2b72e"},"cell_type":"code","source":"merch.category_1 = merch.category_1.replace({'Y':'1','N':'0'}).astype(int)\nmerch.category_2.fillna(1,inplace=True)\nmerch.category_4 = merch.category_4.replace({'Y':'1','N':'0'}).astype(int)\nmerch.most_recent_sales_range = merch.most_recent_sales_range.replace({'A':'1','B':'2','C':'3','D':'4','E':'5'}).astype(int)\nmerch.most_recent_purchases_range = merch.most_recent_purchases_range.replace({'A':'1','B':'2','C':'3','D':'4','E':'5'}).astype(int)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"900ca1be84236273596e4c0358e59b8fad20c11f"},"cell_type":"markdown","source":"Deanon numerical and purchase"},{"metadata":{"trusted":false,"_uuid":"0e3c8f0d74adc4aee1e6637907eb836fe03f4a7e"},"cell_type":"code","source":"merch['numerical_1'] = np.round(merch['numerical_1'] / 0.009914905 + 5.79639, 0)\nmerch['numerical_2'] = np.round(merch['numerical_2'] / 0.009914905 + 5.79639, 0)\nmerch['avg_purchases_lag3' ] = deanon_purchase(merch['avg_purchases_lag3' ])\nmerch['avg_purchases_lag6' ] = deanon_purchase(merch['avg_purchases_lag6' ])\nmerch['avg_purchases_lag12'] = deanon_purchase(merch['avg_purchases_lag12'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"6597f3404db8523a7f34d927b57e191a4a35198f"},"cell_type":"code","source":"merch.describe().T","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"adae2ec5c7bc6d6352c2fbc572fee8bd50ab2802"},"cell_type":"markdown","source":"Here we can see, that we have inf values in avg_purchases_lag3, avg_purchases_lag6, avg_purchases_lag12. Lets replace this on mean values."},{"metadata":{"trusted":false,"_uuid":"305469dc4847e6d5d48071799ad572884afe26be"},"cell_type":"code","source":"merch.loc[(merch.avg_purchases_lag3 == np.inf),'avg_purchases_lag3'] = merch[(merch.avg_purchases_lag3 != np.inf)]['avg_purchases_lag3'].mean()\nmerch.loc[(merch.avg_purchases_lag6 == np.inf),'avg_purchases_lag6'] = merch[(merch.avg_purchases_lag6 != np.inf)]['avg_purchases_lag6'].mean()\nmerch.loc[(merch.avg_purchases_lag12 == np.inf),'avg_purchases_lag12'] = merch[(merch.avg_purchases_lag12 != np.inf)]['avg_purchases_lag12'].mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8fdbe659681c965e119efc69dcde6c93b52b0393"},"cell_type":"code","source":"print(len(merch['merchant_id'].unique()))\nprint(merch['merchant_id'].count())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a6c89241b6b8c1bcedc94ced800afa1805c4898c"},"cell_type":"markdown","source":"Also we can see, that some merchants have more then one record. \nOnly 63 merchants have this problem, its less then 0.1% of data, and i just get mean for this samples"},{"metadata":{"trusted":false,"_uuid":"ce207ef5965087521e17b9c19684d7f5d8c5bf35"},"cell_type":"code","source":"merch = merch.groupby('merchant_id').mean().reset_index()\nmerch.columns = ['merch_'+a for a in merch.columns]\nmerch = merch.rename(columns={'merch_merchant_id':'merchant_id'})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f21703c40d901bdf22d3300e02c2ce0f60b97231"},"cell_type":"markdown","source":"After many test I get only this features, that may be good in feature importance"},{"metadata":{"trusted":false,"_uuid":"498a7fc8928757aac69adc4768c391327c08fcb9"},"cell_type":"code","source":"merch = merch[['merchant_id','merch_avg_sales_lag3',\n 'merch_avg_purchases_lag3','merch_numerical_1','merch_numerical_2']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"9690ec6126df826160eff2263313ba8ae8ceee11"},"cell_type":"code","source":"hist = pd.merge(hist ,merch,on='merchant_id',how='left')\nnew_t = pd.merge(new_t,merch,on='merchant_id',how='left')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"c1d4720434623cfe283f8498d28555071efae233"},"cell_type":"code","source":"del merch;\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1ed83184f2693c2caa70182977f2c7580620f982"},"cell_type":"markdown","source":"### Historical and new transactions"},{"metadata":{"_uuid":"59148511da1019c66ddb1f27ed30eac799055585"},"cell_type":"markdown","source":"We split history data on two parts, because we can see in new transactions data only authorized purchases. \nAlso i see on this kernel: [Simple Exploration Notebook - Elo](https://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-elo) and think this is a good idea. \nplus in separate dataset i get card_ids, that have target with -33.21928095."},{"metadata":{"trusted":false,"_uuid":"64b0eae61103d8bda2ca6c173b11678ed371c94c"},"cell_type":"code","source":"auth = hist[hist['authorized_flag']== 'Y']\nnon_auth = hist[hist['authorized_flag']== 'N']\ncard_ln = hist[hist.card_id.isin(card_with_large_negative)]\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d94234d120c8f0bf07b3e5105e2b7d7e0391b673"},"cell_type":"markdown","source":"#### This is the main function, that changes format historical_transaction and new_merchant_transactions \n"},{"metadata":{"trusted":false,"_uuid":"d9a6db6cbeb765e75d448d756cfaba82af80ca80"},"cell_type":"code","source":"%%time\ndef format_transaction(added_1,pre=''):\n print('Starting formating for',pre)\n added_1['merchant_id'].fillna('M_ID_00a6ca8a8a',inplace=True) #one missing merchant\n orig = added_1.copy()\n #replace object data\n print('replace object data',end=' - ')\n added_1.authorized_flag = added_1.authorized_flag.replace({'Y':'1','N':'0'}).astype(int)\n added_1.category_1 = added_1.category_1.replace({'Y':'1','N':'0'}).astype(int)\n added_1.category_3 = added_1.category_3.replace({'A':'1','B':'2','C':'3',np.nan:'1'}).astype(int)\n print('ready') \n #circled days, weeks and month\n print('Circle dates',end=' - ')\n added_1['cos_doy' ] = np.cos(added_1.purchase_date.dt.dayofyear*2*np.pi/365)\n added_1['sin_doy' ] = np.cos(added_1.purchase_date.dt.dayofyear*2*np.pi/365) \n added_1['cos_dow' ] = np.cos(added_1.purchase_date.dt.dayofweek*2*np.pi/7)\n added_1['sin_dow' ] = np.cos(added_1.purchase_date.dt.dayofweek*2*np.pi/7) \n added_1['cos_month'] = np.cos(added_1.purchase_date.dt.month*2*np.pi/12)\n added_1['sin_month'] = np.cos(added_1.purchase_date.dt.month*2*np.pi/12) \n added_1['cos_week' ] = np.cos(added_1.purchase_date.dt.week*2*np.pi/52)\n added_1['sin_week' ] = np.cos(added_1.purchase_date.dt.week*2*np.pi/52)\n print('ready')\n #added year, month, and yearmonth\n print('added year, month, and yearmonth',end=' - ')\n added_1['year'] = added_1.purchase_date.dt.year.astype(str)\n added_1['weekofyear'] = added_1.purchase_date.dt.weekofyear\n added_1['dayofweek'] = added_1.purchase_date.dt.dayofweek\n added_1['month'] = added_1['purchase_date'].dt.month\n added_1['weekend'] = (added_1.purchase_date.dt.weekday >=5).astype(int)\n added_1['not_weekend'] = (added_1.purchase_date.dt.weekday <5).astype(int)\n added_1['hour'] = added_1.purchase_date.dt.hour \n added_1.loc[added_1['month']<10,'month'] = '0'+added_1.loc[added_1['month']<10,'month'].astype(str)\n added_1['month_id'] = added_1['year'].astype(str)+added_1['month'].astype(str)\n \n added_1['month_diff'] = ((datetime.datetime.today() - added_1['purchase_date']).dt.days)//30\n added_1['month_diff'] += added_1['month_lag'] \n added_1['day'] = added_1['purchase_date'].dt.day\n added_1['dayofyear'] = added_1['purchase_date'].dt.dayofyear\n added_1['quarter'] = added_1['purchase_date'].dt.quarter\n \n added_1['price'] = added_1['purchase_amount'] / added_1['installments']\n added_1['duration'] = added_1['purchase_amount'] * added_1['month_diff']\n added_1['amount_month_ratio'] = added_1['purchase_amount'] / added_1['month_diff']\n \n print('ready')\n #add holidays \n holidays = [\n ('Christmas_Day_2017', '2017-12-25'), # Christmas: December 25 2017\n ('Mothers_Day_2017', '2017-06-04'), # Mothers Day: May 14 2017\n ('fathers_day_2017', '2017-08-13'), # fathers day: August 13 2017\n ('Children_day_2017', '2017-10-12'), # Childrens day: October 12 2017\n ('Valentine_Day_2017', '2017-06-12'), # Valentine's Day : 12th June, 2017\n ('Black_Friday_2017', '2017-11-24'), # Black Friday: 24th November 2017\n ('Mothers_Day_2018', '2018-05-13'),\n ]\n \n for d_name, d_day in holidays:\n dist_holiday(added_1, d_name, d_day, 'purchase_date')\n \n #deanon purchase\n print('deanon purchase',end=' - ')\n added_1['purchase_amount'] = deanon_purchase(added_1['purchase_amount'])\n print('ready')\n #add category pairs\n print('add category pairs',end=' - ')\n added_1['category_12'] = added_1['category_1']*10 + added_1['category_2']\n added_1['category_23'] = added_1['category_2']*10 + added_1['category_3']\n added_1['category_13'] = added_1['category_1']*10 + added_1['category_3'] \n \n added_1['category_all'] = added_1['category_1']*100 + added_1['category_2']*10 + added_1['category_3'] \n \n added_1['month_diff'] = ((datetime.datetime.today() - added_1['purchase_date']).dt.days)//30\n added_1['month_diff'] += added_1['month_lag']\n \n added_1['month_diff_days'] = ((datetime.datetime.today() - added_1['purchase_date']).dt.days)//1\n added_1['month_diff_days'] += added_1['month_lag']*30.5\n \n added_1 = pd.get_dummies(added_1, columns=['category_3','category_2'])\n \n added_1.loc[:, 'purchase_date'] = pd.DatetimeIndex(added_1['purchase_date']).\\\n astype(np.int64) * 1e-9 \n \n \n print('ready') \n \n aggregate = {'installments':['sum','mean','median','std','var','nunique'],\n 'merchant_category_id':['nunique'],\n 'merchant_id':['nunique'],\n 'city_id':['nunique'],\n 'state_id':['nunique'],\n 'subsector_id':['nunique'],\n \n 'purchase_amount': ['sum', 'mean','median', 'max', 'min', 'std','var'],\n 'authorized_flag':['sum'],\n \n 'month_id':['nunique'],\n 'purchase_date':[np.ptp,'min','max'],\n 'month_lag':['max','min','mean','median','sum','std','var'],\n 'weekofyear':['max','min','mean','median','sum','std','var'],\n 'dayofweek':['max','min','mean','median','sum','std','var'],\n 'weekend':['sum','mean','median'],\n 'day':['sum','mean','median'],\n 'dayofyear':['sum','mean','median'],\n 'quarter':['sum','mean','median'],\n 'not_weekend':['sum','mean','median'],\n 'hour':['mean','median','sum'],\n 'month_diff':['mean','median','var','min','max'], \n 'cos_doy' :['mean','median','var','min','max'], \n 'sin_doy' :['mean','median','var','min','max'], \n 'cos_dow' :['mean','median','var','min','max'], \n 'sin_dow' :['mean','median','var','min','max'], \n 'cos_month':['mean','median','var','min','max'], \n 'sin_month':['mean','median','var','min','max'], \n 'cos_week' :['mean','median','var','min','max'], \n 'sin_week' :['mean','median','var','min','max'], \n \n 'category_1':['sum'], \n 'category_2_1.0': ['mean','median'],\n 'category_2_2.0': ['mean','median'],\n 'category_2_3.0': ['mean','median'],\n 'category_2_4.0': ['mean','median'],\n 'category_2_5.0': ['mean','median'], \n 'category_3_1': ['mean','median'],\n 'category_3_2': ['mean','median'],\n 'category_3_3': ['mean','median'], \n 'category_12':['mean','median'],\n 'category_23':['mean','median'],\n 'category_13':['mean','median'], \n 'category_all':['mean','median'],\n 'price': ['sum', 'mean','median', 'std','var'], \n 'duration': ['sum', 'mean','median', 'std','var'], \n 'amount_month_ratio': ['sum', 'mean','median', 'std','var'],\n \n 'merch_avg_sales_lag3':['sum', 'mean','median', 'std','var'],\n 'merch_avg_purchases_lag3':['sum', 'mean','median', 'std','var'],\n 'merch_numerical_1':['sum', 'mean','median', 'std','var'],\n 'merch_numerical_2':['sum', 'mean','median', 'std','var'],\n \n 'Christmas_Day_2017': ['mean', 'sum'],\n 'Mothers_Day_2017': ['mean', 'sum'],\n 'fathers_day_2017': ['mean', 'sum'],\n 'Children_day_2017': ['mean', 'sum'],\n 'Valentine_Day_2017': ['mean', 'sum'],\n 'Black_Friday_2017': ['mean', 'sum'],\n 'Mothers_Day_2018': ['mean', 'sum']\n }\n\n agg_month = {\n 'purchase_amount': ['sum', 'mean','median', 'min', 'max', 'std','var'],\n 'installments': ['sum','mean','median','nunique'], \n 'merch_avg_sales_lag3':['sum', 'mean','median', 'std'],\n 'merch_avg_purchases_lag3':['sum', 'mean','median', 'std'],\n 'merch_numerical_1':['sum', 'mean','median', 'std','var'],\n 'merch_numerical_2':['sum', 'mean','median', 'std','var'],\n 'weekofyear':['max','min','mean','median','sum','std','var'],\n 'dayofweek':['max','min','mean','median','sum','std','var'],\n 'weekend':['sum','mean','median'],\n 'not_weekend':['sum','mean','median'],\n 'hour':['mean','median','sum'],\n 'day':['mean','median','sum']\n }\n \n agg_month2 = {\n 'purchase_amount': ['sum', 'mean','median', 'min', 'max', 'std','var'],\n 'installments': ['sum','mean','median','nunique'], \n 'merch_avg_sales_lag3':['sum', 'mean','median', 'std'],\n 'merch_avg_purchases_lag3':['sum', 'mean','median', 'std'], \n 'merch_numerical_1':['sum', 'mean','median', 'std','var'],\n 'merch_numerical_2':['sum', 'mean','median', 'std','var'],\n 'weekofyear':['max','min','mean','median','sum','std','var'],\n 'dayofweek':['max','min','mean','median','sum','std','var'],\n 'weekend':['sum','mean','median'],\n 'not_weekend':['sum','mean','median'],\n 'hour':['mean','median','sum'],\n 'month_diff':['mean','median'],\n 'day':['mean','median','sum']\n }\n \n print('Start_aggr_month',end=' - ')\n #----\n #dont forget do it in cicle\n agg_month_1 = added_1.groupby(['card_id','month_lag']).agg(agg_month)\n agg_month_1.columns = ['ad1_'.join(col).strip() for col in agg_month_1.columns.values]\n agg_month_1.reset_index(inplace=True)\n\n agg_month_1 = agg_month_1.groupby(['card_id']).agg(['mean','median'])\n agg_month_1.columns = ['ad1_'.join(col).strip() for col in agg_month_1.columns.values]\n agg_month_1.reset_index(inplace=True)\n #----\n agg_month_2 = added_1.groupby(['card_id','month_diff']).agg(agg_month)\n agg_month_2.columns = ['ad2_'.join(col).strip() for col in agg_month_2.columns.values]\n agg_month_2.reset_index(inplace=True)\n\n agg_month_2 = agg_month_2.groupby(['card_id']).agg(['mean','median'])\n agg_month_2.columns = ['ad2_'.join(col).strip() for col in agg_month_2.columns.values]\n agg_month_2.reset_index(inplace=True)\n #----\n agg_month_3 = added_1.groupby(['card_id','installments']).agg(agg_month2)\n agg_month_3.columns = ['ad3_'.join(col).strip() for col in agg_month_3.columns.values]\n agg_month_3.reset_index(inplace=True)\n\n agg_month_3 = agg_month_3.groupby(['card_id']).agg(['mean','median'])\n agg_month_3.columns = ['ad3_'.join(col).strip() for col in agg_month_3.columns.values]\n agg_month_3.reset_index(inplace=True)\n \n #----\n agg_month_4 = added_1.groupby(['card_id','state_id']).agg(agg_month)\n agg_month_4.columns = ['ad4_'.join(col).strip() for col in agg_month_4.columns.values]\n agg_month_4.reset_index(inplace=True)\n\n agg_month_4 = agg_month_4.groupby(['card_id']).agg(['mean','median'])\n agg_month_4.columns = ['ad4_'.join(col).strip() for col in agg_month_4.columns.values]\n agg_month_4.reset_index(inplace=True)\n \n #----\n agg_month_5 = added_1.groupby(['card_id','category_all']).agg(agg_month)\n agg_month_5.columns = ['ad5_'.join(col).strip() for col in agg_month_5.columns.values]\n agg_month_5.reset_index(inplace=True)\n\n agg_month_5 = agg_month_5.groupby(['card_id']).agg(['mean','median'])\n agg_month_5.columns = ['ad5_'.join(col).strip() for col in agg_month_5.columns.values]\n agg_month_5.reset_index(inplace=True)\n print ('ready')\n \n print (added_1.columns)\n print('Start_aggr',end=' - ')\n added_1 = added_1.groupby(['card_id']).agg(aggregate)\n print ('ready')\n added_1.columns = [pre+'_'.join(col).strip() for col in added_1.columns.values]\n added_1.reset_index(inplace=True)\n print('Start merge',end=' - ')\n #added_1 = pd.merge(added_1,dummys_to_card,on='card_id',how='left')\n print ('ready')\n\n print('Start merge month',end=' - ')\n added_1 = pd.merge(added_1,agg_month_1,on='card_id',how='left')\n added_1 = pd.merge(added_1,agg_month_2,on='card_id',how='left')\n added_1 = pd.merge(added_1,agg_month_3,on='card_id',how='left')\n added_1 = pd.merge(added_1,agg_month_4,on='card_id',how='left')\n added_1 = pd.merge(added_1,agg_month_5,on='card_id',how='left')\n print ('ready')\n \n del agg_month_1,agg_month_2,agg_month_3,agg_month_4,agg_month_5;\n gc.collect();\n \n agr_feat = ['category_1','installments','state_id']\n \n #orig['category_all'] = orig['category_1']*100 + orig['category_2']*10 + orig['category_3'] \n \n for a in agr_feat:\n print('Start gen features from '+a,end=' - ')\n agr = orig.groupby(['card_id',a])['purchase_amount'].mean()\n agr = pd.DataFrame(agr).reset_index().groupby('card_id')['purchase_amount'].agg(['mean','median'])\n agr.columns = [a+'_purchase_amount_'+col for col in agr.columns.values]\n agr.reset_index(inplace=True)\n added_1 = pd.merge(added_1,agr,on='card_id', how='left')\n print('ready')\n \n agr_feat = ['category_1','state_id']\n for a in agr_feat:\n print('Start gen features from '+a,end=' - ')\n agr = orig.groupby(['card_id',a])['installments'].mean()\n agr = pd.DataFrame(agr).reset_index().groupby('card_id')['installments'].agg(['mean','median'])\n agr.columns = [a+'_installments_'+col for col in agr.columns.values]\n agr.reset_index(inplace=True)\n added_1 = pd.merge(added_1,agr,on='card_id', how='left')\n print('ready')\n del orig;\n return added_1","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"263e863778c0e53ec0e05bf63cc3adb78b873fb4"},"cell_type":"markdown","source":"### Formating all datasets"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"8f894dfa774a427858ec32b67142998f8fc07e09"},"cell_type":"code","source":"%%time\nadded_2 = format_transaction(auth.copy(),'auth')\nadded_3 = format_transaction(non_auth.copy(),'non_auth')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"424cecd80627753e3ae037112adc403d03cea82c"},"cell_type":"markdown","source":"Do the same with new merch"},{"metadata":{"trusted":false,"_uuid":"fd3eac3592ed8c7f5c3128b6fd54dfd08088cf32"},"cell_type":"code","source":"%%time\nadded_4 = format_transaction(new_t.copy(),'new_')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"426a5aeca1d6a4bfd9e3d8cc111c883090145605"},"cell_type":"markdown","source":"![](http://)For card, that have target -33 we make simple features, just like borders."},{"metadata":{"trusted":false,"_uuid":"3981fb4aca99879ec7352a017477fbd34e952da0"},"cell_type":"code","source":"added_5 = card_ln.copy()\nadded_5.authorized_flag = added_5.authorized_flag.replace({'Y':'1','N':'0'}).astype(int)\nadded_5.category_1 = added_5.category_1.replace({'Y':'1','N':'0'}).astype(int)\nadded_5.category_3 = added_5.category_3.replace({'A':'1','B':'2','C':'3',np.nan:'4'}).astype(int)\n\nadded_5 = added_5.groupby(['card_id']).agg({'authorized_flag':'mean',\n 'city_id':'nunique',\n 'category_1':'mean',\n 'installments':'mean',\n 'category_3':'mean',\n 'merchant_category_id':'nunique',\n 'merchant_id':'nunique',\n 'month_lag':'mean',\n 'purchase_amount':'mean',\n 'category_2':'mean',\n 'state_id':'nunique',\n 'subsector_id':'nunique'\n \n}).reset_index()\nadded_5 = added_5.mean()\ncolumns = added_5.reset_index().T.head(1).values[0]\nadded_5 = added_5.reset_index().T\nadded_5.columns = columns\n# added_5 = added_5[added_5.card_id!='card_id']\nadded_5.columns = ['MEAN_LN_'+col for col in added_5.columns.values]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"754d7bc4fd6a8d633cb225ab8e275aaf5740932e"},"cell_type":"code","source":"LN_col = added_5.columns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"778c5aa04d0fb015e57927b56bc2705d7d7b7004"},"cell_type":"markdown","source":"### Lets join all feature to train and test"},{"metadata":{"trusted":false,"_uuid":"2cc7c10416939e25efb5538133b6b530adf364c5"},"cell_type":"code","source":"train = pd.merge(train,added_2,on=['card_id'],how='left')\ntest_final = pd.merge(test_final,added_2,on=['card_id'],how='left')\n\ntrain = pd.merge(train,added_3,on=['card_id'],how='left')\ntest_final = pd.merge(test_final,added_3,on=['card_id'],how='left')\n\ntrain = pd.merge(train,added_4,on=['card_id'],how='left')\ntest_final = pd.merge(test_final,added_4,on=['card_id'],how='left')\n\n\ntrain.fillna(0,inplace=True)\ntest_final.fillna(0,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"23dbe80641ae2fe8d181c08a5319080ecdf226aa"},"cell_type":"markdown","source":"## Something intresting about ELO\nIn Discussions we can found some intrestings thinks about ELO. ([This discussion](https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75034)) \n[ELO](https://en.wikipedia.org/wiki/Elo_rating_system) - is a method for calculating the relative skill levels of players in zero-sum games such as chess. \nAlso we can find formula, to calculate ELO rating: \n$$E_A=\\frac{1}{1+10^{(R_B-R_A)/400}}$$ \nAs we can see from description of this formula, we need 2 veriables, that mean ratings of 2 parametrs. \n\nWe try calculate this between this samples:\n- authorized purchase and non authorized purchase\n- month_diff and month_lag as most important features \n\nBut I'm not sure that these variables are what we need."},{"metadata":{"trusted":false,"_uuid":"18d8134b0a881b8bdba49092d6ee0f73473dbf12"},"cell_type":"code","source":"train['ELO_auth_sum'] = 1/(1+pow(10,(train.authpurchase_amount_sum-train.non_authpurchase_amount_sum)/400))\ntrain['ELO_purchase_sum'] = 1/(1+pow(10,(train.authpurchase_amount_sum+train.non_authpurchase_amount_sum-abs(train.non_authpurchase_amount_sum))/400))\n\ntrain['ELO_auth_mean'] = 1/(1+pow(10,(train.authpurchase_amount_mean-train.non_authpurchase_amount_mean)/400))\ntrain['ELO_purchase_mean'] = 1/(1+pow(10,(train.authpurchase_amount_mean+train.non_authpurchase_amount_mean-abs(train.non_authpurchase_amount_mean))/400))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"126a6302d9c42ca537d1b9eb02518601657bb67e"},"cell_type":"code","source":"test_final['ELO_auth_sum'] = 1/(1+pow(10,(test_final.authpurchase_amount_sum-test_final.non_authpurchase_amount_sum)/400))\ntest_final['ELO_purchase_sum'] = 1/(1+pow(10,(test_final.authpurchase_amount_sum+test_final.non_authpurchase_amount_sum-abs(test_final.non_authpurchase_amount_sum))/400))\n\ntest_final['ELO_auth_mean'] = 1/(1+pow(10,(test_final.authpurchase_amount_mean-test_final.non_authpurchase_amount_mean)/400))\ntest_final['ELO_purchase_mean'] = 1/(1+pow(10,(test_final.authpurchase_amount_mean+test_final.non_authpurchase_amount_mean-abs(test_final.non_authpurchase_amount_mean))/400))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"28d7d1ec274a02b8ce55f981fa6ebc4f057c24c7"},"cell_type":"code","source":"train['ELO_auth_month_diff'] = 1/(1+pow(10,(train.authmonth_diff_mean-train.non_authmonth_diff_mean)/400))\ntrain['ELO_auth_new_month_diff'] = 1/(1+pow(10,(train.authmonth_diff_mean-train.new_month_diff_mean)/400))\n\ntrain['ELO_auth_month_lag'] = 1/(1+pow(10,(train.authmonth_lag_mean-train.non_authmonth_lag_mean)/400))\ntrain['ELO_auth_new_month_lag'] = 1/(1+pow(10,(train.authmonth_lag_mean-train.new_month_lag_mean)/400))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"3043f14e3a84a06a03c08d41eba428a7360fb619"},"cell_type":"code","source":"test_final['ELO_auth_month_diff'] = 1/(1+pow(10,(test_final.authmonth_diff_mean-test_final.non_authmonth_diff_mean)/400))\ntest_final['ELO_auth_new_month_diff'] = 1/(1+pow(10,(test_final.authmonth_diff_mean-test_final.new_month_diff_mean)/400))\n\ntest_final['ELO_auth_month_lag'] = 1/(1+pow(10,(test_final.authmonth_lag_mean-test_final.non_authmonth_lag_mean)/400))\ntest_final['ELO_auth_new_month_lag'] = 1/(1+pow(10,(test_final.authmonth_lag_mean-test_final.new_month_lag_mean)/400))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"44ec5240a7542d713f6fc093d3e02d3bc1952ee5"},"cell_type":"markdown","source":"### Calculating rating"},{"metadata":{"_uuid":"88d3e704c7859c6f9c4fe15a9860748cd6b6a54f"},"cell_type":"markdown","source":"(added) \nAfter few days of thinking and discussing all the things we decided that variables we really need are rating of card and border of loyality (positive or negative). \nBecause if we read description of ELO we see, that rating of player (in this competition card) increase after win by fromula:\n$$R'_A=R_A+K*(S_A-E_A)$$\nwhere $K$ - koef for top 10 =10, for middle = 20, for first 30 games = 40, $S_A$ - points(1 for win, 0.5 for draw, 0 for loose) \nWe have authorized transaction, nonauthorized transaction and positive or negative. \nWe split it like this:\n- positive authorized transaction like 1\n- all nonauthorized transaction like 0 \nFor $K$ i get only 20 and 40 koef\nAfter that i get feature like rating of card"},{"metadata":{"trusted":false,"_uuid":"f5151888b40b7284f329bb59056e03b664f8f599"},"cell_type":"code","source":"rating = pd.concat((hist[['card_id','merchant_id','purchase_amount','purchase_date','month_lag','authorized_flag']],new_t[['card_id','merchant_id','purchase_amount','purchase_date','month_lag','authorized_flag']]))\nrating['month_diff'] = ((datetime.datetime.today() - rating['purchase_date']).dt.days)//30\nrating['month_diff'] += rating['month_lag']\nrating.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"a5b1fbf546784de5a5752e615a53f0f624266971"},"cell_type":"code","source":"rating = rating.sort_values(['card_id','purchase_date'])\nrating['num_of_purchase'] = rating.groupby(['card_id']).cumcount()+1\n\nrating['K_koef'] = 0\nrating.loc[(rating['authorized_flag']=='Y')&(rating['num_of_purchase']<=30),'K_koef'] = 40\nrating.loc[(rating['authorized_flag']=='Y')&(rating['num_of_purchase']>30),'K_koef'] = 20\n\nrating['S_koef'] = 0\nrating.loc[(rating['authorized_flag']=='Y'),'S_koef'] = 1\n\nrating['change_rating'] = rating['K_koef'] * (rating['S_koef'] - rating['purchase_amount'])\nrating['rating_of_card'] = rating.groupby(['card_id'])['change_rating'].cumsum()\n\nrating['change_rating2'] = rating['K_koef'] * (rating['S_koef'] - rating['month_diff'])\nrating['rating_of_card2'] = rating.groupby(['card_id'])['change_rating2'].cumsum()\n\nrating['change_rating3'] = rating['K_koef'] * (rating['S_koef'] - abs(rating['purchase_amount']))\nrating['rating_of_card3'] = rating.groupby(['card_id'])['change_rating3'].cumsum()\n\nrating['change_rating4'] = rating['K_koef'] * (rating['S_koef'] - abs(rating['month_diff']))\nrating['rating_of_card4'] = rating.groupby(['card_id'])['change_rating4'].cumsum()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2b21583ec977c05fdb00fb897a41c106a457cb9c"},"cell_type":"markdown","source":"It will be rating of card. But we need second rating. We think it would be merchant rating. \nAnd ELO we can calculate with usage card rating vs merchant rating. "},{"metadata":{"trusted":false,"_uuid":"6d7d39d1b434b1a92722c50245944b3862fc00fd"},"cell_type":"code","source":"rating = rating.sort_values(['merchant_id','purchase_date'])\nrating['num_of_purchase'] = rating.groupby(['merchant_id']).cumcount()+1\n\nrating['K_koef'] = 0\nrating.loc[(rating['authorized_flag']=='Y')&(rating['num_of_purchase']<=30),'K_koef'] = 40\nrating.loc[(rating['authorized_flag']=='Y')&(rating['num_of_purchase']>30),'K_koef'] = 20\n\nrating['S_koef'] = 0\nrating.loc[(rating['authorized_flag']=='Y'),'S_koef'] = 1\n\nrating['change_rating_merch'] = rating['K_koef'] * (rating['S_koef'] - (rating['purchase_amount']*(-1)))\nrating['rating_of_merch'] = rating.groupby(['merchant_id'])['change_rating_merch'].cumsum()\n\nrating['change_rating_merch2'] = rating['K_koef'] * (rating['S_koef'] - rating['month_diff'])\nrating['rating_of_merch2'] = rating.groupby(['merchant_id'])['change_rating_merch2'].cumsum()\n\nrating['change_rating_merch3'] = rating['K_koef'] * (rating['S_koef'] - abs(rating['purchase_amount']))\nrating['rating_of_merch3'] = rating.groupby(['merchant_id'])['change_rating_merch3'].cumsum()\n\nrating['change_rating_merch4'] = rating['K_koef'] * (rating['S_koef'] - abs(rating['month_diff']))\nrating['rating_of_merch4'] = rating.groupby(['merchant_id'])['change_rating_merch4'].cumsum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"590624e4cff4e2fbff252eddca0d922cd44d5f4c"},"cell_type":"markdown","source":"### Calculating ELO"},{"metadata":{"trusted":false,"_uuid":"bb942d69eaa863f10b767e83ed69b781bb59d362"},"cell_type":"code","source":"rating['ELO_1'] = 1/(1+pow(10,(rating.rating_of_card -rating.rating_of_merch )/400))\nrating['ELO_2'] = 1/(1+pow(10,(rating.rating_of_card2-rating.rating_of_merch2)/400))\nrating['ELO_3'] = 1/(1+pow(10,(rating.rating_of_card3-rating.rating_of_merch3)/400))\nrating['ELO_4'] = 1/(1+pow(10,(rating.rating_of_card4-rating.rating_of_merch4)/400))\n\nrating['ELO_11'] = 1/(1+pow(10,(rating.rating_of_card -rating.rating_of_merch )/400))\nrating['ELO_21'] = 1/(1+pow(10,(rating.rating_of_card2-rating.rating_of_merch2)/400))\nrating['ELO_31'] = 1/(1+pow(10,(rating.rating_of_card3-rating.rating_of_merch3)/400))\nrating['ELO_41'] = 1/(1+pow(10,(rating.rating_of_card4-rating.rating_of_merch4)/400))\n\nrating['ELO_12'] = 1/(1+pow(10,(rating.rating_of_card -rating.rating_of_merch )/400))\nrating['ELO_22'] = 1/(1+pow(10,(rating.rating_of_card2-rating.rating_of_merch2)/400))\nrating['ELO_32'] = 1/(1+pow(10,(rating.rating_of_card3-rating.rating_of_merch3)/400))\nrating['ELO_42'] = 1/(1+pow(10,(rating.rating_of_card4-rating.rating_of_merch4)/400))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"5a18e19280a8ab86c006773a893f5252815433df"},"cell_type":"code","source":"#rating = replace_id(rating)\nrating_agg = {'S_koef':'mean',\n 'change_rating' :'mean',\n 'rating_of_card' :'max',\n 'rating_of_card2':'max',\n 'rating_of_card3':'max',\n 'rating_of_card4':'max',\n 'rating_of_merch':'max',\n 'rating_of_merch2':'max',\n 'rating_of_merch3':'max',\n 'rating_of_merch4':'max',\n 'ELO_1':'mean',\n 'ELO_2':'mean',\n 'ELO_3':'mean',\n 'ELO_4':'mean',\n 'ELO_11':'max',\n 'ELO_21':'max',\n 'ELO_31':'max',\n 'ELO_41':'max',\n 'ELO_12':'sum',\n 'ELO_22':'sum',\n 'ELO_32':'sum',\n 'ELO_42':'sum'\n }\nrating_card = rating.groupby('card_id').agg(rating_agg).reset_index()\nrating_card.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"515a63e002e778441f947f81afcf302129fbaa7a"},"cell_type":"code","source":"train = pd.merge(train,rating_card,on='card_id',how='left')\ntest_final = pd.merge(test_final,rating_card,on='card_id',how='left')\n\ntrain.fillna(0,inplace=True)\ntest_final.fillna(0,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2202aa7f45106d21caeab9f72af9a4bbce0543aa"},"cell_type":"markdown","source":"> And finaly add features, that calculate credit facility of each card"},{"metadata":{"trusted":false,"_uuid":"51d3f1415e4c98df64300923444185940539c05e"},"cell_type":"code","source":"union = pd.concat((hist,new_t))\nunion[union.installments==-1]\nunion = union[union.installments != 0].sort_values(['purchase_date'])\nunion['installments_lag'] = union.month_lag+union.installments\nunion['installments_lag2'] = union.month_lag+union.installments\nunion.loc[union.installments_lag<0,'installments_lag'] = 0\nunion['credit_now'] = union.installments_lag*union.purchase_amount\nunion['credit_all'] = union.installments*union.purchase_amount\nunion2 = union.groupby('card_id').agg({'installments_lag':['min','max','sum','mean','var','median']\n ,'installments_lag2':['min','max','sum','mean','var','median']\n ,'credit_now':['min','max','sum','mean','var','median']\n ,'credit_all':['min','max','sum','mean','var','median']})\nunion2.columns = ['install_'.join(col).strip() for col in union2.columns.values]\nunion2.reset_index(inplace=True)\nunion2.fillna(0,inplace=True)\n\ntrain = pd.merge(train,union2,on='card_id',how='left')\ntest_final = pd.merge(test_final,union2,on='card_id',how='left')\ntrain.fillna(0,inplace=True)\ntest_final.fillna(0,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cda9c65931ef922f17b5b48c411eda39bd339771"},"cell_type":"markdown","source":"Saving produced dataset on disk, for clear memory and work only with data"},{"metadata":{"trusted":false,"_uuid":"c4336afa37a77b269e1c130ec6643d0b5c50f888"},"cell_type":"code","source":"%%time\ntrain.to_csv(r'F:\\Work\\RepoSVNwn\\Data\\train.csv')\ntest_final.to_csv(r'F:\\Work\\RepoSVNwn\\Data\\test_final.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"41da014b7fd701bbad85397ae6b9b1f7b04a8f37"},"cell_type":"code","source":"%%time\ntrain = pd.read_csv(r'F:\\Work\\RepoSVNwn\\Data\\train.csv')\ntest_final = pd.read_csv(r'F:\\Work\\RepoSVNwn\\Data\\test_final.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"869370f2774674d102abdf7ca45279f36806c121"},"cell_type":"markdown","source":"### Drop all high-correlated features"},{"metadata":{"_uuid":"dfdc6e76a666813b4f0535aa29771d4cf8716040"},"cell_type":"markdown","source":"We have really many features (something about 2100) \nLets cut features, that have hight correlation (not good method, because some features may correlate, but have diffrent information) \nIMPORTANT \nWe get TWO train dataframe, one with this block (5.3) and one without this block. After that a just get mean values of two submit"},{"metadata":{"trusted":false,"_uuid":"6301e7b349a78fa21fe045e50042c5abf13665b7"},"cell_type":"code","source":"%%time\nprint('Search correlations - ',end='')\nfeatures = [a for a in train.columns if a not in ['target','first_active_month','card_id']] \ncorr_matrix = train[features].corr()\nprint('ready')\ni = 1\nar = []\nfor a in corr_matrix.index:\n ar.append(i)\n i+=1\ncorr_matrix['rank'] = ar","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"7da89c7eda34d92bf8ba66a59513ae0c8348e49f"},"cell_type":"code","source":"fe_no_corr = []\ncorr_coef = 0.90\nfor a in corr_matrix.columns:\n if a == 'rank':\n continue\n r1 = corr_matrix[(corr_matrix.index ==a)]['rank'].values[0]\n try:\n r2 = corr_matrix[(abs(corr_matrix[a]) >= corr_coef)&(corr_matrix.index !=a)]['rank'].values[0]\n except:\n r2 = 100000\n #print(a,'with rank',corr_matrix[(corr_matrix.index ==a)]['rank'].values[0])\n #print('Correlation with',corr_matrix[(corr_matrix[a] >= corr_coef)&(corr_matrix.index !=a)].index.values,\n # 'with ranks:',corr_matrix[(corr_matrix[a] >= corr_coef)&(corr_matrix.index !=a)]['rank'].values)\n if r1+ auth_transaction | | merch_data |\n| | | | | | | |\n+--------+--------+ +--------+---------+ +----+-------------+ +--------+-------+\n | | | |\n | v | |\n | +--------+---------+ | |\n | | | | |\n | |noauth_transaction| | |\n | | | | |\n | +--------+---------+ v |\n | | +----------+------+ |\n | +------>+ | |\n | | train_data +<------------------------------+\n +---------------------------->+ ~ 2500 features |\n +---------+-------+\n |\n |\n v\n +------+------+\n | |\n | lgbm |\n | |\n +------+------+\n |\n v\n +----------------+-----------------+\n | |\n | FI best [50,100,150,200,300,400] |\n | |\n +----------------+-----------------+\n |\n |\n |\n |\n +---------------+---------------+--------+--------+---------------+---------------+\n | | | | | |\n v v v v v v\n +------+------+ +------+------+ +------+------+ +------+------+ +------+------+ +------+------+\n | K-fold 10 | | K-fold 10 | | K-fold 10 | | K-fold 10 | | K-fold 10 | | K-fold 10 |\n | goss lgbm | | gbdt lgbm | | dart lgbm | | lgbm tuned | | lgbm tuned | | lgbm tuned |\n | | | | | | | goss | | dart | | gbrt |\n +------+------+ +------+------+ +----------+--+ +--+----------+ +------+------+ +------+------+\n | | | | | |\n | | | | | |\n | | v v | |\n | | +---+---------+----+ | |\n | +-------------->+ +<-------------+ |\n | | ByesianRidge | |\n +------------------------------>+ +<-----------------------------+\n +--------+---------+\n |\n v\n +--------+---------+\n | Submit |\n | param:outlier if |\n | target <-15 |\n +------------------+\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7ac78f531128faae8076f01ada0ffb5d86b254b0"},"cell_type":"markdown","source":"*block for load data if notebook was shutdown or reload*"},{"metadata":{"trusted":false,"_uuid":"18c265edfcb4b24c4b8181c19bc9dba56341cd46"},"cell_type":"code","source":"%%time\ntrain = pd.read_csv('../input/train.csv')\ntrain_cards = train['card_id']\ntrain = pd.read_csv('../input/train_clear.csv')\ntrain['card_id'] = train_cards\ntest_final = pd.read_csv('../input/test_final_clear.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"987b1995f3e240cb1dc47c8b81ad34045a2df4bf"},"cell_type":"markdown","source":"*end load block*"},{"metadata":{"trusted":false,"_uuid":"8ae4f6f8c58071e2643f7afd3ee9ddec2b1e8531"},"cell_type":"code","source":"cards = test_final[['card_id']]\ntrain['outlier'] = 0\ntrain.loc[train.target<-22.,'outlier'] = 1\noutlier = train['outlier']","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"db88fc50c7a8fb7400df81cac2e4f7661b8d356e"},"cell_type":"code","source":"#train.target = deanon_target(train.target)\ntarget = train.target","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c86156d66d375c28d58ecdec9bb21caae1acbcf4"},"cell_type":"markdown","source":"![](http://)![](http://)Little restruct features."},{"metadata":{"trusted":false,"_uuid":"dd9fbc3f11d47e995657b6e67d1d39aa7b2300b0"},"cell_type":"code","source":"features = [a for a in train.columns if a not in ['target','first_active_month','card_id']] \npurchase_column = [a for a in features if 'purchase_amount' in a]\nfor a in [a for a in features if 'rating' in a]:\n purchase_column.append(a)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"ee341da880acbe248a217c7e33b3280c8550466b"},"cell_type":"code","source":"for a in purchase_column:\n train['anon_'+a] = anon_target(train[a])\n test_final['anon_'+a] = anon_target(test_final[a])\nprint('Done')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"257ed9dce01b326a39fd6644dba34b2af7499b9d"},"cell_type":"markdown","source":"### Zero model: Predict on all/no correlate features\nFirst of all I train model to take good features, that can be usefull. I take 400 best features. \nAlso we save prediction, to use it in future mix."},{"metadata":{"trusted":false,"_uuid":"665de422acaa01d060360b2524bb038c786d6f0c"},"cell_type":"code","source":"features = [a for a in train.columns if a not in ['card_id','target','first_active_month','outlier','p1','p1_5','p2_5','pred_valid','pred_valid2','pred_valid3']]","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"547c3e12fee3da627ff1cacf5b3d0f77bba4302b"},"cell_type":"code","source":"%%time \ncat_features = ['feature_1', 'feature_2', 'feature_3']\n\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(len(train))\npredictions = np.zeros(len(test_final))\nstart = time.time()\n\nprint('ready')\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train[features].values, target.values)):\n param = {'task': 'train',\n 'boosting': 'gbdt',\n 'objective': 'regression',\n 'metric': 'rmse',\n 'learning_rate': 0.01,\n 'subsample': 0.9855232997390695,\n 'max_depth': 7,\n 'top_rate': 0.9064148448434349,\n 'num_leaves': 20,\n 'min_child_weight': 41.9612869171337,\n 'other_rate': 0.0721768246018207,\n 'reg_alpha': 9.677537745007898,\n 'colsample_bytree': 0.5665320670155495,\n 'min_split_gain': 9.820197773625843,\n 'reg_lambda': 8.2532317400459,\n 'min_data_in_leaf': 21,\n 'verbose': -1,\n 'seed':int(2**fold_*2),\n 'bagging_seed':int(2**fold_*2),\n 'drop_seed':int(2**fold_*2),\n 'lambda_l2':5,\n 'lambda_l1':5,\n \"feature_fraction\": 0.85,\n \"bagging_freq\": 1,\n \"bagging_fraction\": 0.9 ,\n \"bagging_seed\": int(2**fold_*2),\n \"max_bin\":4,\n \"n_jobs\":6\n }\n print(\"fold n°{}\".format(fold_))\n trn_data = lgb.Dataset(train.iloc[trn_idx][features],\n label=target.iloc[trn_idx],\n categorical_feature=cat_features\n )\n val_data = lgb.Dataset(train.iloc[val_idx][features],\n label=target.iloc[val_idx],\n categorical_feature=cat_features\n )\n print('Data ready')\n num_round = 10000\n clf = lgb.train(param,\n trn_data,\n num_round,\n valid_sets = [trn_data, val_data],\n verbose_eval=100,\n early_stopping_rounds = 200)\n \n oof[val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n \n predictions += clf.predict(test_final[features], num_iteration=clf.best_iteration) / folds.n_splits\n\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(oof, target)**0.5))\n#print(\"CV score: {:<8.5f}\".format(mean_squared_error(anon_target(np.where(oof<0,0.0000001,oof)), anon_target(np.where(target<0,0.0000001,target)))**0.5))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b52c61f8826a180ab96100e1aa9d27f7633ed136"},"cell_type":"markdown","source":"[](http://)Saving this predict"},{"metadata":{"trusted":false,"_uuid":"96cc4c98b25e0c09609a81d692cfd023dec5d121"},"cell_type":"code","source":"dictionary = cards\ndictionary['target'] = predictions\n#dictionary.loc[dictionary['target']<-12,'target'] = -33.21928095\ndictionary[['card_id','target']].to_csv('/all_feature_210219.csv',index=False)\ndictionary[['card_id','target']].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"70b5cd7c32d2a34b3f0d539f998155882f44b2c8"},"cell_type":"code","source":"fi2 = pd.DataFrame(clf.feature_importance())\nfi2.columns = ['importance']\nfi2['feature'] = features\n\nplt.figure(figsize=(14,25))\nsns.barplot(x=\"importance\",\n y=\"feature\",\n data=fi2.sort_values(by=\"importance\",\n ascending=False)[:100])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a35c9d0866b1f60ad73df9dbf156cc3b7ae474fa"},"cell_type":"markdown","source":"#### Analysis predict"},{"metadata":{"_uuid":"925bff298db0fed2a2baac7bc8880d17842c1774"},"cell_type":"markdown","source":"Lets see, what of features can be category features"},{"metadata":{"trusted":false,"_uuid":"60bee4368a4d665e8e7b2b6b8868efcc7694d332"},"cell_type":"code","source":"%%time\ncat_col = []\nfor a in train.columns:\n if train[a].nunique() < 5:\n if a != 'outlier':\n cat_col.append(a)\ncat_col","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"98b2bf52af7f77ee2bc22acce590e7c527b170d8"},"cell_type":"markdown","source":"### First model: LBGM on goss"},{"metadata":{"_uuid":"aaea1b1d0f4aa1d3e972528707d7aac9b48e7f8d"},"cell_type":"markdown","source":"Fit main model on all data \nUsing K-fold with 8 splits and enabled shuffle + using diffrent count of features, selected by feature importance"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"c1ba4cc249c693702d07b8ae933887427448fbe6"},"cell_type":"code","source":"%%time\noof = []\npredictions = []\n\n\nfor a in [50,100,150,200,300,400]:\n oof.append(np.zeros(len(train)))\n predictions.append(np.zeros(len(test_final)))\n \ni=0\nfor a in [50,100,150,200,300,400]: \n features = fi2.sort_values(by=\"importance\",ascending=False)['feature'][:a]\n cat_features = []\n\n folds = KFold(n_splits=8, shuffle=True, random_state=a*2)\n start = time.time()\n print('ready')\n\n for fold_, (trn_idx, val_idx) in enumerate(folds.split(train[features].values, target.values)):\n print(\"fold n°{}\".format(fold_))\n param = {'task': 'train',\n 'boosting': 'goss',\n 'objective': 'regression',\n 'metric': 'rmse',\n 'learning_rate': 0.005,\n 'subsample': 0.9855232997390695,\n 'max_depth': 7,\n 'top_rate': 0.9064148448434349,\n 'num_leaves': 63,\n 'min_child_weight': 41.9612869171337,\n 'other_rate': 0.0721768246018207,\n 'reg_alpha': 9.677537745007898,\n 'colsample_bytree': 0.5665320670155495,\n 'min_split_gain': 9.820197773625843,\n 'reg_lambda': 8.2532317400459,\n 'min_data_in_leaf': 21,\n 'verbose': -1,\n 'seed':int(2**fold_*a),\n 'bagging_seed':int(2**fold_/a),\n 'drop_seed':int(2**fold_+a),\n \"n_jobs\":20,\n \"lambda_l1\": 0.05,\n }\n trn_data = lgb.Dataset(train.iloc[trn_idx][features],\n label=target.iloc[trn_idx],\n categorical_feature=cat_features\n )\n val_data = lgb.Dataset(train.iloc[val_idx][features],\n label=target.iloc[val_idx],\n categorical_feature=cat_features\n )\n print('Data ready')\n num_round = 10000\n clf = lgb.train(param,\n trn_data,\n num_round,\n valid_sets = [trn_data, val_data],\n verbose_eval=100,\n #feval = minowski_dist,\n early_stopping_rounds = 200)\n\n oof[i][val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n\n predictions[i] += clf.predict(test_final[features], num_iteration=clf.best_iteration) / folds.n_splits\n\n print(\"CV score: {:<8.5f}\".format(mean_squared_error(oof[i], target)**0.5))\n train['p1_'+str(a)] = oof[i]\n i+=1\nresult = np.zeros(len(train))\nfor a in range(0,len(oof)):\n result += oof[a]/len(oof)\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(result, target)**0.5))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c26e3c7b438ced38af5969e0fb5867bb48e4dc8a"},"cell_type":"markdown","source":"### Second model: Dart\nFit main model on all data \nUsing K-fold with 3 splits and enabled shuffle + using diffrent count of features, selected by feature importance \nGet only 3 k-fold just because dart calculate can spend many time"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"ef33e80c42949eb07d43bc8ea8c55610f8fefbf5"},"cell_type":"code","source":"%%time\noof2 = []\npredictions2 = []\n\nfor a in [50,100,150,200,300,400]:\n oof2.append(np.zeros(len(train)))\n predictions2.append(np.zeros(len(test_final)))\n \ni=0\nfor a in [50,100,150,200,300,400]: \n features = fi2.sort_values(by=\"importance\",ascending=False)['feature'][:a]\n cat_features = []\n\n folds = KFold(n_splits = 3,shuffle = True, random_state = 12*a)\n start = time.time()\n\n param = {'num_leaves': 31,\n 'min_data_in_leaf': 32, \n 'objective':'regression',\n 'max_depth': -1,\n 'learning_rate': 0.004,\n \"min_child_samples\": 20,\n \"boosting\": \"dart\",\n \"feature_fraction\": 0.9,\n \"bagging_freq\": 1,\n \"bagging_fraction\": 0.9 ,\n \"bagging_seed\": 11,\n \"metric\": 'rmse',\n \"lambda_l1\": 0.18,\n \"nthread\": 24,\n \"verbosity\": -1}\n print('ready')\n for fold_, (trn_idx, val_idx) in enumerate(folds.split(train[features].values, target.values)):\n print(\"fold n°{}\".format(fold_))\n trn_data = lgb.Dataset(train.iloc[trn_idx][features],\n label=target.iloc[trn_idx],\n categorical_feature=cat_features\n )\n val_data = lgb.Dataset(train.iloc[val_idx][features],\n label=target.iloc[val_idx],\n categorical_feature=cat_features\n )\n print('Data ready')\n num_round = 10000\n clf = lgb.train(param,\n trn_data,\n num_round,\n valid_sets = [trn_data, val_data],\n verbose_eval=100,\n #feval = minowski_dist,\n early_stopping_rounds = 200)\n\n oof2[i][val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n\n predictions2[i] += clf.predict(test_final[features], num_iteration=clf.best_iteration) / folds.n_splits\n\n print(\"CV score: {:<8.5f}\".format(mean_squared_error(oof2[i], target)**0.5))\n train['p2_'+str(a)] = oof2[i]\n i+=1\n \nresult2 = np.zeros(len(train))\nfor a in range(0,len(oof2)):\n result2 += oof2[a]/len(oof2)\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(result2, target)**0.5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"678a7ce15e418d84e8da37f681b1bd0c5b72bc04"},"cell_type":"code","source":"dictionary = cards\ndictionary['target'] = predictions\ndictionary[['card_id','target']].to_csv('/dart_150219_cv_3.65983.csv',index=False)\ndictionary[['card_id','target']].head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"59dc3ea5be1bdf04fcaca7fdcf0526b833672df4"},"cell_type":"markdown","source":"### Third model: gbdt\nFit main model on all data \nUsing K-fold with 3 splits and enabled shuffle + using diffrent count of features, selected by feature importance "},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"0a808b017eeffe55d423c61b00a535bbc2f8d1fc"},"cell_type":"code","source":"%%time\noof3 = []\npredictions3 = []\n\nfor a in [50,100,150,200,300,400]:\n oof3.append(np.zeros(len(train)))\n predictions3.append(np.zeros(len(test_final)))\n \ni=0\nfor a in [50,100,150,200,300,400]: \n features = fi2.sort_values(by=\"importance\",ascending=False)['feature'][:a]\n cat_features = []\n\n folds = KFold(n_splits = 7,shuffle = True, random_state = 12*a)\n start = time.time()\n\n print('ready')\n\n for fold_, (trn_idx, val_idx) in enumerate(folds.split(train[features].values, target.values)):\n print(\"fold n°{}\".format(fold_))\n param = {'num_leaves': 10,\n 'min_data_in_leaf': 32, \n 'objective':'regression',\n 'max_depth': -1,\n 'learning_rate': 0.005,\n \"min_child_samples\": 20,\n \"boosting\": \"gbdt\",\n \"feature_fraction\": 0.9,\n \"bagging_freq\": 1,\n \"bagging_fraction\": 0.9 ,\n \"bagging_seed\": int(2**fold_*a),\n \"metric\": 'rmse',\n \"lambda_l1\": 0.11,\n \"nthread\": 10,\n \"verbosity\": -1}\n trn_data = lgb.Dataset(train.iloc[trn_idx][features],\n label=target.iloc[trn_idx],\n categorical_feature=cat_features\n )\n val_data = lgb.Dataset(train.iloc[val_idx][features],\n label=target.iloc[val_idx],\n categorical_feature=cat_features\n )\n print('Data ready')\n num_round = 10000\n clf = lgb.train(param,\n trn_data,\n num_round,\n valid_sets = [trn_data, val_data],\n verbose_eval=100,\n #feval = minowski_dist,\n early_stopping_rounds = 200)\n\n oof3[i][val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n\n predictions3[i] += clf.predict(test_final[features], num_iteration=clf.best_iteration) / folds.n_splits\n\n print(\"CV score: {:<8.5f}\".format(mean_squared_error(oof3[i], target)**0.5))\n train['p3_'+str(a)] = oof3[i]\n i+=1\nresult3 = np.zeros(len(train))\nfor a in range(0,len(oof3)):\n result3 += oof3[a]/len(oof3)\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(result3, target)**0.5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"bde2317c22ff69fce8ff700df1c68710cb0cb28d"},"cell_type":"code","source":"dictionary = cards\ndictionary['target'] = predictions\ndictionary[['card_id','target']].to_csv('/gbdt_150219_cv_3.65867.csv',index=False)\ndictionary[['card_id','target']].head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c2a111cbea7c63d228b11f26fe230b2e486ac7d5"},"cell_type":"markdown","source":"### Fourth model: Tuned LGBM gbrt ([Dmitry Kravchuk](https://www.kaggle.com/dishkakrauch))\nUsing K-fold with 5 splits and enabled shuffle + using diffrent count of features, selected by feature importance "},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"04dd2d10e7281371c280845717ba7e38dc3fa0a4"},"cell_type":"code","source":"%%time\noof5 = []\npredictions5 = []\n\nfor a in [50,100,150,200,300,400]:\n oof5.append(np.zeros(len(train)))\n predictions5.append(np.zeros(len(test_final)))\n \ni=0\nfor a in [50,100,150,200,300,400]: \n features = fi2.sort_values(by=\"importance\",ascending=False)['feature'][:a]\n cat_features = []\n\n folds = KFold(n_splits=5, shuffle=True, random_state=a*2)\n start = time.time()\n print('ready')\n\n for fold_, (trn_idx, val_idx) in enumerate(folds.split(train[features].values, target.values)):\n print(\"fold n°{}\".format(fold_))\n param = {\n #Core Parameters\n 'task': 'train',\n 'objective': 'regression',\n 'boosting': 'gbrt',\n 'learning_rate': .01,\n 'num_leaves': 114, #2^max_depth*70%\n 'num_threads': 24,\n 'device_type': 'cpu',\n 'seed': int(2**a*fold_),\n #Learning Control Parameters\n 'max_depth': 3,\n 'min_data_in_leaf': 20,\n 'min_sum_hessian_in_leaf': 1e-3,\n 'bagging_fraction': .75,\n 'bagging_freq': 2,\n 'bagging_seed': int(2**a*fold_),\n 'feature_fraction': .75,\n 'feature_fraction_seed': int(2**a*fold_),\n 'max_delta_step': 0.0,\n 'lambda_l1': 170.,\n 'lambda_l2': 227.,\n 'min_gain_to_split': 0.0,\n 'min_data_per_group': 100,\n 'max_cat_threshold': 32,\n 'cat_l2': 10.0,\n 'cat_smooth': 10.0,\n 'max_cat_to_onehot': 4,\n 'monotone_constraints': None,\n 'feature_contri': None,\n 'forcedsplits_filename': '',\n 'refit_decay_rate': .9,\n 'verbosity': 1,\n 'max_bin': 171,\n 'min_data_in_bin': 3,\n 'bin_construct_sample_cnt': 200000,\n 'histogram_pool_size': -1.0,\n 'data_random_seed': int(2**a*fold_),\n 'snapshot_freq': -1,\n 'iniscore_filename': '',\n 'valid_data_iniscore': '',\n 'pre_partition': False,\n 'enable_bundle': True,\n 'max_conflict': 0.0,\n 'is_enable_sparse': True,\n 'sparse_threshold': .8,\n 'use_missing': True,\n 'zero_as_missing': True,\n 'two_round': False,\n 'save_binary': False,\n 'enable_load_from_binary_file': True,\n 'header': False,\n 'label_column': '',\n 'weight_columns': '',\n 'group_column': '',\n 'ignore_columns': '',\n 'boost_from_average': True,\n 'reg_sqrt': False,\n 'alpha': .9,\n 'fair_c': 1.0,\n 'poisson_max_delta_step': .7,\n 'tweedie_variance_power': 1.5,\n #Metric Parameters\n 'metric': 'l2_root',\n 'metric_freq': 1,\n 'is_provide_training_metric': False\n }\n trn_data = lgb.Dataset(train.iloc[trn_idx][features],\n label=target.iloc[trn_idx],\n categorical_feature=cat_features\n )\n val_data = lgb.Dataset(train.iloc[val_idx][features],\n label=target.iloc[val_idx],\n categorical_feature=cat_features\n )\n print('Data ready')\n num_round = 10000\n clf = lgb.train(param,\n trn_data,\n num_round,\n valid_sets = [trn_data, val_data],\n verbose_eval=100,\n #feval = minowski_dist,\n early_stopping_rounds = 200)\n\n oof5[i][val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n\n predictions5[i] += clf.predict(test_final[features], num_iteration=clf.best_iteration) / folds.n_splits\n\n print(\"CV score: {:<8.5f}\".format(mean_squared_error(oof5[i], target)**0.5))\n train['p5_'+str(a)] = oof5[i]\n i+=1\nresult = np.zeros(len(train))\nfor a in range(0,len(oof5)):\n result += oof5[a]/len(oof5)\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(result, target)**0.5))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d9ee9cc4488dba3b063360efebaeea21e3e97ff5"},"cell_type":"markdown","source":"### Fifth model: Tuned LGBM goss (Krauchuk_D)\nUsing K-fold with 5 splits and enabled shuffle + using diffrent count of features, selected by feature importance "},{"metadata":{"trusted":false,"_uuid":"103e37c394ecd762d4e3f51fcf74ff6ce2237470"},"cell_type":"code","source":"%%time\noof6 = []\npredicions6 = []\n\nfor a in [50,100,150,200,300,400]:\n oof6.append(np.zeros(len(train)))\n predicions6.append(np.zeros(len(test_final)))\n \ni=0\nfor a in [50,100,150,200,300,400]: \n features = fi2.sort_values(by=\"importance\",ascending=False)['feature'][:a]\n cat_features = []\n\n folds = KFold(n_splits=5, shuffle=True, random_state=a*2)\n start = time.time()\n print('ready')\n\n for fold_, (trn_idx, val_idx) in enumerate(folds.split(train[features].values, target.values)):\n print(\"fold n°{}\".format(fold_))\n param = {\n #Core Parameters\n 'task': 'train',\n 'objective': 'regression',\n 'boosting': 'goss',\n 'learning_rate': .01,\n 'num_leaves': 171, #2^max_depth*70%\n 'num_threads': 24,\n 'device_type': 'cpu',\n 'seed': int(2**a*fold_),\n #Learning Control Parameters\n 'max_depth': 3,\n 'min_data_in_leaf': 20,\n 'min_sum_hessian_in_leaf': 1e-3,\n 'feature_fraction': .75,\n 'feature_fraction_seed': int(2**a*fold_),\n 'max_delta_step': 0.0,\n 'lambda_l1': 114.,\n 'lambda_l2': 199.,\n 'min_gain_to_split': 0.0,\n 'top_rate': .4, #goss\n 'oter_rate': .4, #goss\n 'min_data_per_group': 100,\n 'max_cat_threshold': 32,\n 'cat_l2': 10.0,\n 'cat_smooth': 10.0,\n 'max_cat_to_onehot': 4,\n 'monotone_constraints': None,\n 'feature_contri': None,\n 'forcedsplits_filename': '',\n 'refit_decay_rate': .9,\n #IO Parameters\n 'verbosity': 1,\n 'max_bin': 171,\n 'min_data_in_bin': 3,\n 'bin_construct_sample_cnt': 200000,\n 'histogram_pool_size': -1.0,\n 'data_random_seed': int(2**a*fold_),\n 'snapshot_freq': -1,\n 'iniscore_filename': '',\n 'valid_data_iniscore': '',\n 'pre_partition': False,\n 'enable_bundle': True,\n 'max_conflict': 0.0,\n 'is_enable_sparse': True,\n 'sparse_threshold': .8,\n 'use_missing': True,\n 'zero_as_missing': True,\n 'two_round': False,\n 'save_binary': False,\n 'enable_load_from_binary_file': True,\n 'header': False,\n 'label_column': '',\n 'weight_columns': '',\n 'group_column': '',\n 'ignore_columns': '',\n 'boost_from_average': True,\n 'reg_sqrt': False,\n 'alpha': .9,\n 'fair_c': 1.0,\n 'poisson_max_delta_step': .7,\n 'tweedie_variance_power': 1.5,\n #Metric Parameters\n 'metric': 'l2_root',\n 'metric_freq': 1,\n 'is_provide_training_metric': False,\n #Network Parameters\n 'num_machines': 1,\n 'local_listen_port': 12400,\n 'time-out': 120,\n 'machine_list_filename': '',\n 'machines': '',\n #GPU Parameters\n 'gpu_platform_id': 0,\n 'gpu_device_id': 0,\n 'gpu_use_dp': True\n }\n \n trn_data = lgb.Dataset(train.iloc[trn_idx][features],\n label=target.iloc[trn_idx],\n categorical_feature=cat_features\n )\n val_data = lgb.Dataset(train.iloc[val_idx][features],\n label=target.iloc[val_idx],\n categorical_feature=cat_features\n )\n print('Data ready')\n num_round = 10000\n clf = lgb.train(param,\n trn_data,\n num_round,\n valid_sets = [trn_data, val_data],\n verbose_eval=100,\n #feval = minowski_dist,\n early_stopping_rounds = 200)\n\n oof6[i][val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n\n predicions6[i] += clf.predict(test_final[features], num_iteration=clf.best_iteration) / folds.n_splits\n\n print(\"CV score: {:<8.5f}\".format(mean_squared_error(oof6[i], target)**0.5))\n train['p6_'+str(a)] = oof6[i]\n i+=1\nresult = np.zeros(len(train))\nfor a in range(0,len(oof6)):\n result += oof6[a]/len(oof6)\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(result, target)**0.5))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"894e7ac8762f6add315f8d042b722e83e03b5482"},"cell_type":"markdown","source":"### Sixth model: Tuned LGBM dart ([Dmitry Kravchuk](https://www.kaggle.com/dishkakrauch))\nUsing K-fold with 5 splits and enabled shuffle + using diffrent count of features, selected by feature importance "},{"metadata":{"trusted":false,"_uuid":"85905fa111c803dbb71728566527505b3eb53838"},"cell_type":"code","source":"%%time\noof7 = []\npredicions7 = []\n\nfor a in [50,100,150,200,300,400]:\n oof7.append(np.zeros(len(train)))\n predicions7.append(np.zeros(len(test_final)))\n \ni=0\nfor a in [50,100,150,200,300,400]: \n features = fi2.sort_values(by=\"importance\",ascending=False)['feature'][:a]\n cat_features = []\n\n folds = KFold(n_splits=5, shuffle=True, random_state=a*2)\n start = time.time()\n print('ready')\n\n for fold_, (trn_idx, val_idx) in enumerate(folds.split(train[features].values, target.values)):\n print(\"fold n°{}\".format(fold_))\n param = {\n #Core Parameters\n 'task': 'train',\n 'objective': 'regression',\n 'boosting': 'dart',\n 'learning_rate': .1,\n 'num_leaves': 114, #2^max_depth*70%\n 'num_threads': 24,\n 'device_type': 'cpu',\n 'seed': int(2**a*fold_),\n #Learning Control Parameters\n 'max_depth': 3,\n 'min_data_in_leaf': 20,\n 'min_sum_hessian_in_leaf': 1e-3,\n 'bagging_fraction': .75,\n 'bagging_freq': 2,\n 'bagging_seed': int(2**a*fold_),\n 'feature_fraction': .75,\n 'feature_fraction_seed': int(2**a*fold_),\n 'max_delta_step': 0.0,\n 'lambda_l1': 0.,\n 'lambda_l2': 114.,\n 'min_gain_to_split': 0.0,\n 'drop_rate': .55, #dart\n 'max_dop': 64, #dart\n 'skip_drop': .1, #dart\n 'xgboost_dart_mode': False, #dart\n 'uniform_drop': True, #dart\n 'drop_seed': int(2**a*fold_), #dart\n 'min_data_per_group': 100,\n 'max_cat_threshold': 32,\n 'cat_l2': 10.0,\n 'cat_smooth': 10.0,\n 'max_cat_to_onehot': 4,\n 'monotone_constraints': None,\n 'feature_contri': None,\n 'forcedsplits_filename': '',\n 'refit_decay_rate': .9,\n #IO Parameters\n 'verbosity': 1,\n 'max_bin': 255,\n 'min_data_in_bin': 3,\n 'bin_construct_sample_cnt': 200000,\n 'histogram_pool_size': -1.0,\n 'data_random_seed': int(2**a*fold_),\n 'snapshot_freq': -1,\n 'iniscore_filename': '',\n 'valid_data_iniscore': '',\n 'pre_partition': False,\n 'enable_bundle': True,\n 'max_conflict': 0.0,\n 'is_enable_sparse': True,\n 'sparse_threshold': .8,\n 'use_missing': True,\n 'zero_as_missing': True,\n 'two_round': False,\n 'save_binary': False,\n 'enable_load_from_binary_file': True,\n 'header': False,\n 'label_column': '',\n 'weight_columns': '',\n 'group_column': '',\n 'ignore_columns': '',\n 'boost_from_average': True,\n 'reg_sqrt': False,\n 'alpha': .9,\n 'fair_c': 1.0,\n 'poisson_max_delta_step': .7,\n 'tweedie_variance_power': 1.5,\n #Metric Parameters\n 'metric': 'l2_root',\n 'metric_freq': 1,\n 'is_provide_training_metric': False,\n #Network Parameters\n 'num_machines': 1,\n 'local_listen_port': 12400,\n 'time-out': 120,\n 'machine_list_filename': '',\n 'machines': '',\n #GPU Parameters\n 'gpu_platform_id': 0,\n 'gpu_device_id': 0,\n 'gpu_use_dp': True\n }\n trn_data = lgb.Dataset(train.iloc[trn_idx][features],\n label=target.iloc[trn_idx],\n categorical_feature=cat_features\n )\n val_data = lgb.Dataset(train.iloc[val_idx][features],\n label=target.iloc[val_idx],\n categorical_feature=cat_features\n )\n print('Data ready')\n num_round = 10000\n clf = lgb.train(param,\n trn_data,\n num_round,\n valid_sets = [trn_data, val_data],\n verbose_eval=100,\n #feval = minowski_dist,\n early_stopping_rounds = 200)\n\n oof7[i][val_idx] = clf.predict(train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n\n predicions7[i] += clf.predict(test_final[features], num_iteration=clf.best_iteration) / folds.n_splits\n\n print(\"CV score: {:<8.5f}\".format(mean_squared_error(oof7[i], target)**0.5))\n train['p1_'+str(a)] = oof7[i]\n i+=1\nresult = np.zeros(len(train))\nfor a in range(0,len(oof7)):\n result += oof7[a]/len(oof7)\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(result, target)**0.5))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7e8ed79e522ca85675df21c5fe050ec9c74a71ec"},"cell_type":"markdown","source":"### Mix with BayesianRidge\nFor the best mix we use BayesianRidge and ensemble out models"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"a6e8a19143e706520ed09b419887552ace80e5b2"},"cell_type":"code","source":"train_stack = []\ntest_stack = []\nfor a in range(0,len(oof)):\n train_stack.append(oof[a])\n train_stack.append(oof2[a])\n train_stack.append(oof3[a])\n \n train_stack.append(oof5[a])\n train_stack.append(oof6[a])\n train_stack.append(oof7[a])\n \n test_stack.append(predictions[a])\n test_stack.append(predictions2[a])\n test_stack.append(predictions3[a])\n \n test_stack.append(predictions5[a])\n test_stack.append(predicions6[a])\n test_stack.append(predicions7[a])\n#train_stack.append(oof4)\n#test_stack.append(predictions4)\n\ntrain_stack = np.vstack(train_stack).transpose()\ntest_stack = np.vstack(test_stack).transpose()\n\nfolds = KFold(n_splits=20,shuffle=True,random_state=4520)\noof_stack = np.zeros(train_stack.shape[0])\npredictions_stack = np.zeros(test_stack.shape[0])\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_stack, target)):\n print(\"fold n°{}\".format(fold_))\n trn_data, trn_y = train_stack[trn_idx], target.iloc[trn_idx].values\n val_data, val_y = train_stack[val_idx], target.iloc[val_idx].values\n\n print(\"-\" * 10 + \"Stacking \" + str(fold_) + \"-\" * 10)\n clf = BayesianRidge()\n clf.fit(trn_data, trn_y)\n \n oof_stack[val_idx] = clf.predict(val_data)\n print(\"CV score fold: {:<8.5f}\".format(mean_squared_error(oof_stack[val_idx], target[val_idx])**0.5))\n predictions_stack += clf.predict(test_stack) / folds.n_splits\n\nprint(\"CV score summary:\",np.sqrt(mean_squared_error(target.values, oof_stack)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"6ae3ec4a360a1061793ecbd02967bd81dd10c580"},"cell_type":"code","source":"dictionary = cards\ndictionary['target'] = predictions_stack\nborder = -14.5\nprint('Count outliers:',len(dictionary[dictionary.target