{"cells":[{"metadata":{"_uuid":"39ac5a6369b5f1fab58357d3caef03d30965dcfc"},"cell_type":"markdown","source":"# **Intro**\nI used this competition/kernel to just add to my portfolio a bit while also working on my skills. I didn't at all look at what others were doing for this kernel, since I wanted to work on this as a real world project where one doesn't have all this external help.  I am now going to look at what others did to learn from them, but I will still leave this as is.  The main idea for me here was just to practice and learn.  You will notice that I only used a subset of features.  I did this to try to get a simple model working without spending a ton of time.  I wasn't worried about the score as much as building something from start to finish independently.\nThanks for checking it out, and please feel free to comment."},{"metadata":{"_uuid":"27c08143f8e50e5749e1a92bb9fc1f31c7110896"},"cell_type":"markdown","source":"# **Imports**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os, matplotlib, math, copy\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nprint(os.listdir(\"../input\"))\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\n\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.preprocessing import QuantileTransformer\nfrom sklearn.preprocessing import PowerTransformer","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e38184bd5c216faf4129b7b2caf82dc99b3bbb19"},"cell_type":"markdown","source":"# Data Downloads"},{"metadata":{"trusted":true,"_uuid":"8fc72b440eb76774552cb14f28085eb6d4077fe7"},"cell_type":"code","source":"new_merchant_transactions = pd.read_csv('../input/new_merchant_transactions.csv')\ntrain = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')\nmerchants = pd.read_csv('../input/merchants.csv')\nsample_submission = pd.read_csv('../input/sample_submission.csv')\nhistorical_transactions = pd.read_csv('../input/historical_transactions.csv')\nData_Dictionary = pd.read_excel('../input/Data_Dictionary.xlsx')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2eeebd96c52395226473497e3af9eed398003562"},"cell_type":"markdown","source":"## **Functions**"},{"metadata":{"trusted":true,"_uuid":"4e0842b817fc4fd5b982946c6ab4cd4529e0b574"},"cell_type":"code","source":"def tf(pd_series):\n    return np.array(pd_series).reshape(-1,1)\n\ndef transform_viz(pd_series,transform_list = [QuantileTransformer(), MinMaxScaler(), StandardScaler(),PowerTransformer()]):\n    transform_list = [QuantileTransformer(), MinMaxScaler(), StandardScaler(),PowerTransformer()]\n    fig, ax = plt.subplots(1,len(transform_list)+1,figsize = (18,3))\n    sns.distplot(pd_series.dropna().sample(1000),ax = ax[0]);\n    for i, transformer in enumerate(transform_list):\n        sns.distplot(transformer.fit_transform(tf(pd_series.dropna().sample(1000))),ax = ax[i+1]);\n\ndef plot_details(pd_series):\n    plt.figure(figsize = (3,3))\n    std, mean, skew = pd_series.std(), pd_series.mean(), pd_series.skew()\n    print('shape: {}, mean: {:.2f}, std: {:.2f}, skew: {:.2f}'.format(pd_series.shape,mean,std,skew))\n    sns.distplot(pd_series)\n    plt.show();\n    return std, mean, skew\n\ndef remove_outliers_and_skew(pd_series,skew_threshold = 1, remove_outliers = True, deviations = 2.15):\n    \"\"\"\n    args:\n        pd-series: required\n        skew_threshold: skew calc above which series with be log-transformed\n        remove_outliers: will return pd_series with ...\n        deviations: ...\n    \"\"\"\n    print('original distribution')\n    std, mean, skew = plot_details(pd_series)\n    if skew > 1:\n        print('log transform original')\n        new_series = np.log1p(pd_series)\n        if remove_outliers:\n            std, mean, skew = plot_details(new_series)\n            print('update size')\n            new_series = new_series[new_series.between(mean - (std * deviations),mean + (std * deviations))]\n    else:\n        if remove_outliers:\n            print('update size')\n            new_series = new_series[new_series.between(mean - (std * deviations),mean + (std * deviations))]\n        else:\n            new_series = pd_series\n    plot_details(new_series)\n    return new_series","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c2d7c0b43d1ac856676787f13b3e3277abb98029"},"cell_type":"markdown","source":"# **EDA - Exploratory Data Analysis**"},{"metadata":{"trusted":true,"_uuid":"785eb84a212df6ca0f06b210b87336e38b50e101"},"cell_type":"code","source":"sample_submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"144f32882d3bb462e12451df276d8293473c46f8"},"cell_type":"code","source":"sns.distplot(train['target']);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"52f648160059d88c0b8b36512245b94f8034ef93"},"cell_type":"code","source":"print(new_merchant_transactions.shape)\nnew_merchant_transactions.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b5c8d61d2cee608d4587dca46e059259d032dc7e"},"cell_type":"code","source":"print(train.shape)\ntrain.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"99c723bf31fa6a9ceee685378e8832f2fd879c75"},"cell_type":"code","source":"train.nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"edb5c49c083612e47aacba1f75b034d21cbf8255"},"cell_type":"code","source":"print(test.shape)\ntest.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5d524c77de17d8ded95517ee0e6bcacb21cb064f"},"cell_type":"code","source":"print(merchants.shape)\nmerchants.head(2)\n#merchants.nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"04e8da48afcc99eb21e4cec896fb1692aaa96b3e"},"cell_type":"code","source":"print(historical_transactions.shape)\nhistorical_transactions.head(2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"23727404774df493a5d2b3bab10c33da0ac885f5"},"cell_type":"markdown","source":"Maybe it's already obvious... but we're looking for unique cardholders and just checking on the counts here"},{"metadata":{"trusted":true,"_uuid":"55aa05e257cf2b53a600abb7ad652170327aa334"},"cell_type":"code","source":"print(historical_transactions[['card_id','merchant_id']].nunique())\nprint(train.shape[0])\nprint(test.shape[0])\nprint((train.shape[0])+(test.shape[0]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5a352c51dbf69ac194cd865c35229ae39fed09b6"},"cell_type":"code","source":"Data_Dictionary","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9d7893caeae7df1d968b88a5a615916851614030"},"cell_type":"markdown","source":"## Data Types and Basic Visuals (for historicals_transactions)"},{"metadata":{"trusted":true,"_uuid":"d723e0b1818b6f23f1b1598ca2d868ec8e71f559"},"cell_type":"markdown","source":"### Unique Counts\nTaking a quick sample of 100,000 to choose which non-numeric features can be dummy variable based on unique counts"},{"metadata":{"trusted":true,"_uuid":"d723e0b1818b6f23f1b1598ca2d868ec8e71f559"},"cell_type":"code","source":"historical_trans_sample = historical_transactions[:100000].nunique()\ndata_types = historical_transactions.dtypes\nfeatures = pd.DataFrame({'unique':historical_trans_sample,'dtypes':data_types})\nfeatures","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8441583b87f21326a5ba58bfc2c5373e07ea4074"},"cell_type":"code","source":"features_numeric = features[features['dtypes']!=object].index.tolist()\nprint(features_numeric)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"79bc28a734dd25495e71f7700a83a7807bd16d2e"},"cell_type":"code","source":"features_dummy = features[(features['dtypes']==object)&(features['unique']<30)].index.tolist()\nfeatures_dummy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f202b163f47ae98151b65a32bbafd7e744417d0d"},"cell_type":"code","source":"historical_transactions[features_dummy].sample(100000).nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d26d480738a8ccfe58c1719efc1c2e762749c0df"},"cell_type":"code","source":"#note that this just pulls a sample from each\ncolumns = 3\nrows = math.ceil(len(features_numeric)/columns)\n\nfig, ax = plt.subplots(rows,columns,figsize = (16,8))\nfor i, feature in list(enumerate(features_numeric)):\n    sns.distplot(historical_transactions[feature].dropna().sample(20000),ax= ax[math.floor(i/columns),i%columns],axlabel=feature)\n    plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=1.0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"64d6b81e2dfb7e0b5d59dbaca43bba42500bdd1b"},"cell_type":"markdown","source":"switching some features from numeric to dummy"},{"metadata":{"trusted":true,"_uuid":"9a1a86071a06489f2513cf5ae3ac3451654f712c"},"cell_type":"code","source":"for feature in ['category_2','state_id']:\n    features_numeric.remove(feature)\n    features_dummy.append(feature)\n    \nprint('numeric features: {}'.format(features_numeric))\nprint('dummy features: {}'.format(features_dummy))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5879d6bc135b011eb5459e8a4ac752d7e773de67"},"cell_type":"markdown","source":"# **Feature Engineering**"},{"metadata":{"_uuid":"ee8a2003a99f1c43c2186e4d28ba5fdec33b9ce4"},"cell_type":"markdown","source":"## **features_numeric**"},{"metadata":{"_uuid":"e60b486861e1c4da1efbb1e57bb0f364c12c5942"},"cell_type":"markdown","source":"### purchase_amount"},{"metadata":{"_uuid":"feab8eaa504e6d1c997c57d51f8634aba710c30d"},"cell_type":"markdown","source":"Will pull customers' total purchase amount"},{"metadata":{"trusted":true,"_uuid":"44739ca2c7fd4cb058abb9aa73c80e434b4e68bb"},"cell_type":"markdown","source":"from datetime import date, datetime\nd = date.today()\ndatetime.strptime(d.strftime('%Y%m%d'), '%Y%m%d')"},{"metadata":{"trusted":true,"_uuid":"ae0baef5b2d355347dbd8a68d2eef254f8eb2b69"},"cell_type":"code","source":"historical_transactions['purchase_amount'] = historical_transactions['purchase_amount'] - historical_transactions['purchase_amount'].min()\npurchase_amount_sum = historical_transactions[['purchase_amount','card_id']].groupby('card_id').sum()['purchase_amount']\npurchase_amount_sum_transformed = remove_outliers_and_skew(purchase_amount_sum,1,False,6) ##########\ndel purchase_amount_sum","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d7b93257652c69ac1971d2ee3eff7f7aba268216"},"cell_type":"code","source":"sns.distplot(historical_transactions['month_lag'].abs().sample(20000))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"4b673e4f71ae72a1243e9b2c7a5ee7d002b04687"},"cell_type":"code","source":"historical_transactions['month_lag'] = historical_transactions['month_lag'].abs()\nmonth_lag_abs_mean = historical_transactions[['month_lag','card_id']].groupby('card_id').mean()['month_lag'] ##########\nmonth_lag_abs_mean = remove_outliers_and_skew(month_lag_abs_mean,1,False,6)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66f21ffe1f8929e3194e98d3930e39c486735251"},"cell_type":"markdown","source":"## features_dummy"},{"metadata":{"trusted":true,"_uuid":"74a2749f951701252aadf952a2f035354e8f9908"},"cell_type":"code","source":"dummy_data = historical_transactions[['card_id']].drop_duplicates()\nprint(dummy_data.shape)\ndummy_data.set_index('card_id', inplace = True)\n#dummy_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"717139e0020493b6037f3423a5cafcdb708094f9"},"cell_type":"code","source":"# Creating dummies of historical transactions from state_id is too large, 29m x 15\n# Keeping only those with higher counts\nstate_id_counts = historical_transactions[['state_id','card_id']].groupby('state_id').count() / historical_transactions.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"65a0d9a8232cefb3f25cee657486bf44ab24ee9d"},"cell_type":"code","source":"state_id_counts","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f107aaa5f45995f8a3e139a78fe97d7af959c6a3"},"cell_type":"code","source":"state_id_counts[state_id_counts['card_id']>.03]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3b5b1f81884bf8c12ea0b0c653207a4e04746bc9"},"cell_type":"code","source":"null_states = list(state_id_counts[state_id_counts['card_id']<=.03].index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7ec23408b818eec6de36f6a2535494436ca75d25"},"cell_type":"code","source":"#before\nhistorical_transactions[['state_id','card_id']].groupby('state_id').count().head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9b64104e846c5eb66a679edf649ee47377e00a3a"},"cell_type":"code","source":"#historical_transactions['state_id'].isin(null_states),0\n#df.loc[df.my_channel > 20000, 'my_channel'] = 0\nhistorical_transactions.state_id[historical_transactions['state_id'].isin(null_states)] = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"73e63e4b0c2cab0826b7f3d789cf3394a456ecd0"},"cell_type":"code","source":"#after\nhistorical_transactions[['state_id','card_id']].groupby('state_id').count().head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a799a729236a1c7a917e2357337b92b9b01b973a"},"cell_type":"code","source":"#after\nhistorical_transactions['state_id'].nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5a36feee6c0505e4ddc3b3d5ca9b8db0cce0a895"},"cell_type":"code","source":"features_dummy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6da2cde068e067c76ba7e1a3def9dea09ef6cfd1"},"cell_type":"code","source":"for feature in features_dummy:\n    dummies = pd.get_dummies(historical_transactions[feature].astype(str),prefix=feature)\n    dummies = pd.merge(dummies,historical_transactions[['card_id']],left_index = True, right_index = True)\n    dummies = dummies.groupby('card_id').sum()\n    dummy_data = pd.merge(dummy_data, dummies, left_index = True, right_index = True, how = 'inner')\n    print(dummy_data.columns)\n\ndel dummies    \ndummy_data.head(3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a43d502bda9a8f22851db52aafc67348c1590897"},"cell_type":"markdown","source":"# **Merge Data for Train and Test**"},{"metadata":{"trusted":true,"_uuid":"f35cd6cc594dbc92cd610ccbb6c9b50342b8d368","scrolled":true},"cell_type":"code","source":"card_data = pd.merge(month_lag_abs_mean.to_frame(), purchase_amount_sum_transformed.to_frame(),left_index = True, right_index = True,how = 'inner')\nprint(card_data.shape)\ncard_data.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"070774604eb53c3f7213f9930bca411d1875fa7f"},"cell_type":"code","source":"print(card_data.shape)\ncard_data = pd.merge(card_data,dummy_data, left_index = True, right_index = True, how = 'inner')\nprint(card_data.shape)\ndel dummy_data\ncard_data.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"77b44cf55e5960ced3017dfdd5d059c86f308c97"},"cell_type":"code","source":"columns = 2\n\ntrain_numeric = list(train.dtypes[train.dtypes != object].index)\nrows = math.ceil(len(train_numeric)/columns)\n\nfig, ax = plt.subplots(rows,columns,figsize = (8,4))\nfor i, feature in list(enumerate(train_numeric)):\n    #sns.distplot(train[feature].dropna().sample(20000),ax= ax[math.floor(i/columns),i%columns],axlabel=feature)\n    sns.distplot(train[feature].dropna().sample(20000),ax= ax[math.floor(i/columns),i%columns])\n    plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=1.0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fa2a5f1d6998cc603cc7a38eead9b085fd03df85"},"cell_type":"code","source":"results = pd.concat([train,test])\nresults = results.drop(['first_active_month'],axis = 1)\nprint(results.shape)\nresults.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"56d533a19f1829b1fb3c3adb1ac101f8bf1088b9"},"cell_type":"code","source":"card_data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fc8f52eb8568838f3a6e533e599b61faf256899e"},"cell_type":"code","source":"results[results['target'].isnull()].shape[0] + results.dropna().shape[0] - results.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f2e523359c236ff28e8daca7499660678589225d"},"cell_type":"code","source":"all_data = pd.merge(card_data, results, left_index = True, right_on = 'card_id', how = 'inner')\nall_data.set_index('card_id', inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1d1b7ccacb03d21bef540f0ca4be667781f46e94"},"cell_type":"code","source":"results_train = all_data.dropna()\nresults_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0450e0f1b7337d4289b74f1081649a2e9b0bb575"},"cell_type":"code","source":"results_test = all_data[all_data['target'].isnull()]\nresults_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b29a10d9538b96168191e0eafba06d6fd4e67e92"},"cell_type":"code","source":"results_train.shape[0] + results_test.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d4a02dc9b84ebf1335fc205807b7019852aa7a6c"},"cell_type":"code","source":"results_test.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a22f456323b7102a7a29fb2cfafb08156d382cd8"},"cell_type":"code","source":"results_test = results_test.drop('target',axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"917814bfef777be01a45418686554c7f953876ef"},"cell_type":"code","source":"X, y = results_train.drop(['target'],axis = 1), results_train['target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"17bd85833db5006df64eef6a5f0546d12b063960"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d5415ed17c04fc1d0043ae971e9e180b426b5595"},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"44f2413d095a7b685d9e29a8ea6af38568f45a88"},"cell_type":"markdown","source":"# Cleanup"},{"metadata":{"trusted":true,"_uuid":"897e1992699a00bc7a4dfab820b3e060dec59a37"},"cell_type":"code","source":"#del card_data\n#del results_train\n#del results","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"827a863070f6373d472e52ced4b277001007d11d"},"cell_type":"markdown","source":"# **Model Testing (new)**"},{"metadata":{"trusted":true,"_uuid":"6255b27183f7057b5f0ca0dc4f4fce62c33ef449"},"cell_type":"code","source":"from sklearn.pipeline import Pipeline\nfrom sklearn.model_selection import GridSearchCV","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0a9ac9aea34f2bda1af2433163c6f3e4a74cf2b4"},"cell_type":"code","source":"from sklearn.svm import SVC\nfrom sklearn.svm import LinearSVC\n\nfrom sklearn.linear_model import Ridge\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.linear_model import Lasso\nfrom sklearn.linear_model import ElasticNet\nfrom sklearn.linear_model import LinearRegression\n\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.ensemble import AdaBoostRegressor\nfrom sklearn.ensemble import BaggingRegressor\nfrom sklearn.ensemble import ExtraTreesRegressor\nfrom sklearn.ensemble import GradientBoostingRegressor","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5d95a780c8a67a8ba57ad7d076e663e523351fa"},"cell_type":"markdown","source":"from sklearn.metrics import SCORERS\nsorted(SCORERS.keys())"},{"metadata":{"trusted":true,"_uuid":"6b0656266b9cad8ae84366b1057110fd4dc803d6"},"cell_type":"code","source":"def best_score(model, param_grid):\n    grid = GridSearchCV(model, param_grid=param_grid,scoring='neg_mean_squared_error', cv=5)\n    grid.fit(X = X_train,y = y_train)\n    print(grid.best_score_, grid.best_params_)\n    return grid.best_score_, grid.best_params_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"096d92443f665a4071c874b79a7d2c6ba2fa4f85"},"cell_type":"code","source":"import os\nos.environ['KMP_DUPLICATE_LIB_OK']='True'\nfrom xgboost import XGBRegressor\n#from xgboost import XGBClassifier","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"675f4e0bfe2cac5f92343eff64f490cb277df313"},"cell_type":"code","source":"#best_score(XGBRegressor(), {'max_depth': [3,6],'n_estimators':[50,300],'learning_rate':[0.05]})\nbest_score(XGBRegressor(), {'max_depth': [3],'n_estimators':[300],'learning_rate':[0.05]})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0b02eaef0023ea473e931a9cc2ddc88c680d498a"},"cell_type":"code","source":"grid = GridSearchCV(Ridge(), param_grid={'alpha': [1, 10]},scoring='neg_mean_squared_error', cv=5)\ngrid.fit(X = X_train,y = y_train)\ngrid.best_score_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"59d31d2ab4e64cfd78b51f2d6dc3bd5868c457f9"},"cell_type":"code","source":"best_score(Ridge(), {'alpha': [1,10]})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8716bc520749599358e7a6c17ada78e7ef884ac2"},"cell_type":"markdown","source":"best_score(Ridge(), {'alpha': [1,2,4,6,10,20,30,100,10000]})"},{"metadata":{"trusted":true,"_uuid":"594546ad99995dcb555869b22f928d1c94af207e"},"cell_type":"code","source":"best_score(Lasso(), {'alpha': [1,2,4,6,10,20,30,100]})\nbest_score(Lasso(), {'alpha': [1,2,10, 30,100]})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"174ca2a19b15eaffecb300d9a698b2376b390b3a"},"cell_type":"code","source":"#best_score(ElasticNet(), {'alpha': [1,2,4,6,10,20,30,100],'l1_ratio':[.2, .5, .8]})\nbest_score(ElasticNet(), {'alpha': [1],'l1_ratio':[.2]})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"90e027df993b285abcaed2850a82ad84f953d556","collapsed":true},"cell_type":"markdown","source":"best_score(RandomForestRegressor(), {'min_samples_leaf': [1,3,5],'min_samples_split':[2,3,5,10]})"},{"metadata":{"trusted":true,"_uuid":"ec8c4601716bd5777e2b58fb10f5475a6ade8e56"},"cell_type":"markdown","source":"best_score(ExtraTreesRegressor(), {'min_samples_leaf': [1],'min_samples_split':[2]})"},{"metadata":{"trusted":true,"_uuid":"863acf8d2c831bd8bedf8fb3095d76defe7dd89a"},"cell_type":"markdown","source":"best_score(AdaBoostRegressor(), {'base_estimator': [Ridge(alpha = 10),LinearRegression()]})"},{"metadata":{"trusted":true,"_uuid":"b99ea34a7c5bc813c9ac8e2e95a90c92cf561a58"},"cell_type":"markdown","source":"best_score(GradientBoostingRegressor(), {'loss': ['ls']})"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"83f27cf28b55a0a084463f3864c91f8db1d48424"},"cell_type":"markdown","source":"best_score(BaggingRegressor(), {'base_estimator': [Ridge(alpha = 10),LinearRegression()]})"},{"metadata":{"trusted":true,"_uuid":"1358eba29516c78e13e38d73dba6d51ed9df99b9"},"cell_type":"markdown","source":"best_score(LinearRegression(), {'fit_intercept': [True]})"},{"metadata":{"_uuid":"827a863070f6373d472e52ced4b277001007d11d"},"cell_type":"markdown","source":"#**Model Testing**"},{"metadata":{"trusted":true,"_uuid":"89b3f499c2cf1875390cdad43eaba0e685a49132"},"cell_type":"markdown","source":"#from sklearn.linear_model import Ridge\nfrom sklearn.metrics import mean_squared_error\n\ndef get_mse(model,alpha):\n    run_model = model(alpha = alpha)\n    run_model.fit(X_train,y_train)\n    return mean_squared_error(y_pred=run_model.predict(X_test),y_true=y_test)\n\nalphas = [.01,.05,0.1,0.5,1,2,3,5,10,20,40,100,1000,5000]\nridge_mses = [get_mse(Ridge,x) for x in alphas]\nplt.plot(alphas,ridge_mses)"},{"metadata":{"trusted":true,"_uuid":"3710a96b510259ffce5ca400fc37dd7d0391e873"},"cell_type":"markdown","source":"from sklearn.ensemble import RandomForestRegressor\nmodel = RandomForestRegressor()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)"},{"metadata":{"trusted":true,"_uuid":"b255e14641e415b7221922b0567a7fc8c3d7be32"},"cell_type":"markdown","source":"from sklearn.ensemble import AdaBoostRegressor\nmodel = AdaBoostRegressor()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)"},{"metadata":{"trusted":true,"_uuid":"710cba9812e74033f955399de3aa981573221351"},"cell_type":"markdown","source":"from sklearn.ensemble import BaggingRegressor\nmodel = BaggingRegressor()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)"},{"metadata":{"trusted":true,"_uuid":"d637a6a8f6e7da45b3a8f15795eaa7d8ff7ae5b8"},"cell_type":"markdown","source":"from sklearn.linear_model import Lasso\nmodel = Lasso()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)"},{"metadata":{"trusted":true,"_uuid":"5ba6e7ea4d78ef3801047bf47ecffbf380de8189"},"cell_type":"markdown","source":"from sklearn.linear_model import ElasticNet\nmodel = ElasticNet()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)"},{"metadata":{"trusted":true,"_uuid":"522114850480d9a8d18ea6e1a741f6e95160b645"},"cell_type":"markdown","source":"from sklearn.linear_model import LinearRegression\nmodel = LinearRegression()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)"},{"metadata":{"trusted":true,"_uuid":"98ad3a548fac9ba7ba51b14d829997225c189d36"},"cell_type":"markdown","source":"from sklearn.ensemble import ExtraTreesRegressor\nmodel = ExtraTreesRegressor()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)"},{"metadata":{"trusted":true,"_uuid":"a9244908c6f15fe788ba348d297bad3ffe00b20e"},"cell_type":"markdown","source":"from sklearn.ensemble import GradientBoostingRegressor\nmodel = GradientBoostingRegressor()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)"},{"metadata":{"_uuid":"292498e7f09ad3bd1c4ae9fa59abde9e1dbe5d06"},"cell_type":"markdown","source":"# **Train on Full Dataset**"},{"metadata":{"trusted":true,"_uuid":"a9244908c6f15fe788ba348d297bad3ffe00b20e"},"cell_type":"markdown","source":"model = GradientBoostingRegressor()\nmodel = GradientBoostingRegressor()"},{"metadata":{"trusted":true,"_uuid":"a9244908c6f15fe788ba348d297bad3ffe00b20e"},"cell_type":"code","source":"model = XGBRegressor(max_depth = 3, n_estimators = 300,learning_rate = 0.05)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a9244908c6f15fe788ba348d297bad3ffe00b20e"},"cell_type":"code","source":"#model.fit(X_train,y_train)\nmodel.fit(X,y)\n#mean_squared_error(y_pred=model.predict(X_test),y_true=y_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"af4da8c154b159484eaf3a852f2c3b85717512e7"},"cell_type":"markdown","source":"# **Prepare Submission**"},{"metadata":{"trusted":true,"_uuid":"c5c6e7a771408fa8f9b74c76616ae59af81640ad"},"cell_type":"code","source":"sample_submission.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ccdffb71e88d733e293da040c6fc73d77373113e"},"cell_type":"code","source":"results_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d42b347689462edb86b74f88e3b1b0954fb96bc9"},"cell_type":"code","source":"predictions = model.predict(results_test)\nsub = pd.DataFrame({'card_id':results_test.index, 'target':predictions})\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ca461dafb34e099a74f030529991507d4ff79e82"},"cell_type":"code","source":"sns.distplot(sub['target'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c955d47afe886e3a453914db52cd4a319971254b"},"cell_type":"code","source":"sns.distplot(train['target'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d76e6db49375b512fb4bf3aa257188a981dfa16c"},"cell_type":"code","source":"sub.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0352b824019a3654b7c8aa54d0749fd148262bf3"},"cell_type":"markdown","source":"# **Slag**"},{"metadata":{"trusted":true,"_uuid":"6da2cde068e067c76ba7e1a3def9dea09ef6cfd1"},"cell_type":"markdown","source":"dummies = pd.get_dummies(historical_transactions[features_dummy].astype(str),prefix='dummy')\ndummies = pd.merge(dummies,historical_transactions[['card_id']],left_index = True, right_index = True)\ndummies = dummies.groupby('card_id').sum()\nprint(dummies.columns)\ndummies.head(3)\n\nfeature = 'category_2'\ndummies = pd.get_dummies(historical_transactions[feature].astype(str),prefix=feature)\ndummies = pd.merge(dummies,historical_transactions[['card_id']],left_index = True, right_index = True)\ndummies = dummies.groupby('card_id').sum()\ndummies.head()\n\ncategory_2_dummies = pd.get_dummies(historical_transactions['category_2'],prefix='cat_2')\ncategory_2_dummies = pd.merge(category_2_dummies,historical_transactions[['card_id']],left_index = True, right_index = True)\ncategory_2_dummies = category_2_dummies.groupby('card_id').sum()\n\ncategory_1_dummies = pd.get_dummies(historical_transactions['category_1'],prefix='cat_1')\ncategory_1_dummies = pd.merge(category_1_dummies,historical_transactions[['card_id']],left_index = True, right_index = True)\ncategory_1_dummies = category_1_dummies.groupby('card_id').sum()\n\npd.get_dummies(historical_transactions[features_dummy].sample(10),prefix='asdf_')\n\ndummies = pd.merge(category_1_dummies, category_2_dummies, left_index = True, right_index = True, how = 'inner')\ndel category_1_dummies, category_2_dummies\nprint(dummies.shape)\ndummies.head(2)\n\ncard_data = pd.merge(card_data,dummies, left_index = True, right_index = True, how = 'inner')\ncard_data.head(2)"},{"metadata":{"trusted":true,"_uuid":"405bfb94ba95f70d4eba3f18da47405f531eea61"},"cell_type":"markdown","source":"best_score(GradientBoostingRegressor(), {'alpha': [1,2,4,8,16]})\nbest_score(Lasso(), {'alpha': [1,2,4,6,10,20,30,100]})\n\nbest_score(LogisticRegression(), {'C': [1,10]})\n\nbest_score(LinearSVC(), {'C': [0.5, 1.0]})"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}