{"cells":[{"metadata":{"_uuid":"39ac5a6369b5f1fab58357d3caef03d30965dcfc"},"cell_type":"markdown","source":"# **Intro**\nI used this competition/kernel to just add to my portfolio a bit while also working on my skills. I didn't at all look at what others were doing for this kernel, since I wanted to work on this as a real world project where one doesn't have all this external help.  I am now going to look at what others did to learn from them, but I will still leave this as is.  The main idea for me here was just to practice and learn.  You will notice that I only used a subset of features.  I did this to try to get a simple model working without spending a ton of time.  I wasn't worried about the score as much as building something from start to finish independently.\nThanks for checking it out, and please feel free to comment."},{"metadata":{"_uuid":"27c08143f8e50e5749e1a92bb9fc1f31c7110896"},"cell_type":"markdown","source":"# **Imports**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os, matplotlib, math, copy\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nprint(os.listdir(\"../input\"))\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\n\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.preprocessing import QuantileTransformer\nfrom sklearn.preprocessing import PowerTransformer","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e38184bd5c216faf4129b7b2caf82dc99b3bbb19"},"cell_type":"markdown","source":"# Data Downloads"},{"metadata":{"trusted":true,"_uuid":"8fc72b440eb76774552cb14f28085eb6d4077fe7"},"cell_type":"code","source":"new_merchant_transactions = pd.read_csv('../input/new_merchant_transactions.csv')\ntrain = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')\nmerchants = pd.read_csv('../input/merchants.csv')\nsample_submission = pd.read_csv('../input/sample_submission.csv')\nhistorical_transactions = pd.read_csv('../input/historical_transactions.csv')\nData_Dictionary = pd.read_excel('../input/Data_Dictionary.xlsx')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c2d7c0b43d1ac856676787f13b3e3277abb98029"},"cell_type":"markdown","source":"# **EDA - Exploratory Data Analysis**"},{"metadata":{"trusted":true,"_uuid":"785eb84a212df6ca0f06b210b87336e38b50e101"},"cell_type":"code","source":"sample_submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"144f32882d3bb462e12451df276d8293473c46f8"},"cell_type":"code","source":"sns.distplot(train['target']);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"52f648160059d88c0b8b36512245b94f8034ef93"},"cell_type":"code","source":"print(new_merchant_transactions.shape)\nnew_merchant_transactions.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b5c8d61d2cee608d4587dca46e059259d032dc7e"},"cell_type":"code","source":"print(train.shape)\ntrain.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"99c723bf31fa6a9ceee685378e8832f2fd879c75"},"cell_type":"code","source":"train.nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"edb5c49c083612e47aacba1f75b034d21cbf8255"},"cell_type":"code","source":"print(test.shape)\ntest.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5d524c77de17d8ded95517ee0e6bcacb21cb064f"},"cell_type":"code","source":"print(merchants.shape)\nmerchants.head(2)\n#merchants.nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"04e8da48afcc99eb21e4cec896fb1692aaa96b3e"},"cell_type":"code","source":"print(historical_transactions.shape)\nhistorical_transactions.head(2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"23727404774df493a5d2b3bab10c33da0ac885f5"},"cell_type":"markdown","source":"Maybe it's already obvious... but we're looking for unique cardholders and just checking on the counts here"},{"metadata":{"trusted":true,"_uuid":"55aa05e257cf2b53a600abb7ad652170327aa334"},"cell_type":"code","source":"print(historical_transactions[['card_id','merchant_id']].nunique())\nprint(train.shape[0])\nprint(test.shape[0])\nprint((train.shape[0])+(test.shape[0]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5a352c51dbf69ac194cd865c35229ae39fed09b6"},"cell_type":"code","source":"Data_Dictionary","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9d7893caeae7df1d968b88a5a615916851614030"},"cell_type":"markdown","source":"## Data Types and Basic Visuals (for historicals_transactions)"},{"metadata":{"trusted":true,"_uuid":"d723e0b1818b6f23f1b1598ca2d868ec8e71f559"},"cell_type":"markdown","source":"### Unique Counts\nTaking a quick sample of 100,000 to choose which non-numeric features can be dummy variable based on unique counts"},{"metadata":{"trusted":true,"_uuid":"d723e0b1818b6f23f1b1598ca2d868ec8e71f559"},"cell_type":"code","source":"historical_trans_sample = historical_transactions[:100000].nunique()\ndata_types = historical_transactions.dtypes\nfeatures = pd.DataFrame({'unique':historical_trans_sample,'dtypes':data_types})\nfeatures","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8441583b87f21326a5ba58bfc2c5373e07ea4074"},"cell_type":"code","source":"features_numeric = features[features['dtypes']!=object].index.tolist()\nprint(features_numeric)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"79bc28a734dd25495e71f7700a83a7807bd16d2e"},"cell_type":"code","source":"features_dummy = features[(features['dtypes']==object)&(features['unique']<30)].index.tolist()\nfeatures_dummy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d26d480738a8ccfe58c1719efc1c2e762749c0df"},"cell_type":"code","source":"#note that this just pulls a sample from each\ncolumns = 3\nrows = math.ceil(len(features_numeric)/columns)\n\nfig, ax = plt.subplots(rows,columns,figsize = (16,8))\nfor i, feature in list(enumerate(features_numeric)):\n    sns.distplot(historical_transactions[feature].dropna().sample(20000),ax= ax[math.floor(i/columns),i%columns],axlabel=feature)\n    plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=1.0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5879d6bc135b011eb5459e8a4ac752d7e773de67"},"cell_type":"markdown","source":"# **Feature Engineering**"},{"metadata":{"_uuid":"e60b486861e1c4da1efbb1e57bb0f364c12c5942"},"cell_type":"markdown","source":"### purchase_amount"},{"metadata":{"trusted":true,"_uuid":"4e0842b817fc4fd5b982946c6ab4cd4529e0b574"},"cell_type":"code","source":"def tf(pd_series):\n    return np.array(pd_series).reshape(-1,1)\n\ndef transform_viz(pd_series,transform_list = [QuantileTransformer(), MinMaxScaler(), StandardScaler(),PowerTransformer()]):\n    transform_list = [QuantileTransformer(), MinMaxScaler(), StandardScaler(),PowerTransformer()]\n    fig, ax = plt.subplots(1,len(transform_list)+1,figsize = (18,3))\n    sns.distplot(pd_series.dropna().sample(1000),ax = ax[0]);\n    for i, transformer in enumerate(transform_list):\n        sns.distplot(transformer.fit_transform(tf(pd_series.dropna().sample(1000))),ax = ax[i+1]);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6ebb11283732d532ae90bbbd193a7f159314b849"},"cell_type":"code","source":"def plot_details(pd_series):\n    plt.figure(figsize = (3,3))\n    std, mean, skew = pd_series.std(), pd_series.mean(), pd_series.skew()\n    print('shape: {}, mean: {:.2f}, std: {:.2f}, skew: {:.2f}'.format(pd_series.shape,mean,std,skew))\n    sns.distplot(pd_series)\n    plt.show();\n    return std, mean, skew","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cc3458758fdb19e7f9051dfb60b34b76fd07123d"},"cell_type":"code","source":"def remove_outliers_and_skew(pd_series,skew_threshold = 1, remove_outliers = True, deviations = 2.15):\n    \"\"\"\n    args:\n        pd-series: required\n        skew_threshold: skew calc above which series with be log-transformed\n        remove_outliers: will return pd_series with ...\n        deviations: ...\n    \"\"\"\n    print('original distribution')\n    std, mean, skew = plot_details(pd_series)\n    if skew > 1:\n        print('log transform original')\n        new_series = np.log1p(pd_series)\n        if remove_outliers:\n            std, mean, skew = plot_details(new_series)\n            print('update size')\n            new_series = new_series[new_series.between(mean - (std * deviations),mean + (std * deviations))]\n    else:\n        if remove_outliers:\n            print('update size')\n            new_series = new_series[new_series.between(mean - (std * deviations),mean + (std * deviations))]\n        else:\n            new_series = pd_series\n    plot_details(new_series)\n    return new_series","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"feab8eaa504e6d1c997c57d51f8634aba710c30d"},"cell_type":"markdown","source":"Will pull customers' total purchase amount"},{"metadata":{"trusted":true,"_uuid":"ae0baef5b2d355347dbd8a68d2eef254f8eb2b69"},"cell_type":"code","source":"historical_transactions['purchase_amount'] = historical_transactions['purchase_amount'] - historical_transactions['purchase_amount'].min()\npurchase_amount_sum = historical_transactions[['purchase_amount','card_id']].groupby('card_id').sum()['purchase_amount']\npurchase_amount_sum_transformed = remove_outliers_and_skew(purchase_amount_sum,1,False,6) ##########\ndel purchase_amount_sum","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d7b93257652c69ac1971d2ee3eff7f7aba268216"},"cell_type":"code","source":"sns.distplot(historical_transactions['month_lag'].abs().sample(20000))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"4b673e4f71ae72a1243e9b2c7a5ee7d002b04687"},"cell_type":"code","source":"historical_transactions['month_lag'] = historical_transactions['month_lag'].abs()\nmonth_lag_abs_mean = historical_transactions[['month_lag','card_id']].groupby('card_id').mean()['month_lag'] ##########\nmonth_lag_abs_mean = remove_outliers_and_skew(month_lag_abs_mean,1,False,6)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"77a115d13752d42dace21edc5a18c82848ea803f"},"cell_type":"code","source":"category_2_dummies = pd.get_dummies(historical_transactions['category_2'],prefix='cat_2')\ncategory_2_dummies = pd.merge(category_2_dummies,historical_transactions[['card_id']],left_index = True, right_index = True)\ncategory_2_dummies = category_2_dummies.groupby('card_id').sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"77a115d13752d42dace21edc5a18c82848ea803f"},"cell_type":"code","source":"category_1_dummies = pd.get_dummies(historical_transactions['category_1'],prefix='cat_1')\ncategory_1_dummies = pd.merge(category_1_dummies,historical_transactions[['card_id']],left_index = True, right_index = True)\ncategory_1_dummies = category_1_dummies.groupby('card_id').sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a43d502bda9a8f22851db52aafc67348c1590897"},"cell_type":"markdown","source":"# **Merge Data for Train and Test**"},{"metadata":{"trusted":true,"_uuid":"f35cd6cc594dbc92cd610ccbb6c9b50342b8d368"},"cell_type":"code","source":"card_data = pd.merge(month_lag_abs_mean.to_frame(), purchase_amount_sum_transformed.to_frame(),left_index = True, right_index = True,how = 'inner')\nprint(card_data.shape)\ncard_data.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"598b8690628686dc42a8603e1e7989a203cc2c34"},"cell_type":"code","source":"dummies = pd.merge(category_1_dummies, category_2_dummies, left_index = True, right_index = True, how = 'inner')\ndel category_1_dummies, category_2_dummies\nprint(dummies.shape)\ndummies.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"070774604eb53c3f7213f9930bca411d1875fa7f"},"cell_type":"code","source":"card_data = pd.merge(card_data,dummies, left_index = True, right_index = True, how = 'inner')\ncard_data.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"77b44cf55e5960ced3017dfdd5d059c86f308c97"},"cell_type":"code","source":"columns = 2\n\ntrain_numeric = list(train.dtypes[train.dtypes != object].index)\nrows = math.ceil(len(train_numeric)/columns)\n\nfig, ax = plt.subplots(rows,columns,figsize = (8,4))\nfor i, feature in list(enumerate(train_numeric)):\n    #sns.distplot(train[feature].dropna().sample(20000),ax= ax[math.floor(i/columns),i%columns],axlabel=feature)\n    sns.distplot(train[feature].dropna().sample(20000),ax= ax[math.floor(i/columns),i%columns])\n    plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=1.0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fa2a5f1d6998cc603cc7a38eead9b085fd03df85"},"cell_type":"code","source":"results = pd.concat([train,test])\nresults = results.drop(['first_active_month'],axis = 1)\nprint(results.shape)\nresults.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"56d533a19f1829b1fb3c3adb1ac101f8bf1088b9"},"cell_type":"code","source":"card_data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fc8f52eb8568838f3a6e533e599b61faf256899e"},"cell_type":"code","source":"results[results['target'].isnull()].shape[0] + results.dropna().shape[0] - results.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f2e523359c236ff28e8daca7499660678589225d"},"cell_type":"code","source":"all_data = pd.merge(card_data, results, left_index = True, right_on = 'card_id', how = 'inner')\nall_data.set_index('card_id', inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1d1b7ccacb03d21bef540f0ca4be667781f46e94"},"cell_type":"code","source":"results_train = all_data.dropna()\nresults_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0450e0f1b7337d4289b74f1081649a2e9b0bb575"},"cell_type":"code","source":"results_test = all_data[all_data['target'].isnull()]\nresults_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b29a10d9538b96168191e0eafba06d6fd4e67e92"},"cell_type":"code","source":"results_train.shape[0] + results_test.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d4a02dc9b84ebf1335fc205807b7019852aa7a6c"},"cell_type":"code","source":"results_test.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a22f456323b7102a7a29fb2cfafb08156d382cd8"},"cell_type":"code","source":"results_test = results_test.drop('target',axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"917814bfef777be01a45418686554c7f953876ef"},"cell_type":"code","source":"X, y = results_train.drop(['target'],axis = 1), results_train['target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"17bd85833db5006df64eef6a5f0546d12b063960"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d5415ed17c04fc1d0043ae971e9e180b426b5595"},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"827a863070f6373d472e52ced4b277001007d11d"},"cell_type":"markdown","source":"# **Model**"},{"metadata":{"trusted":true,"_uuid":"50c0a8d0ece643e270980dfe3e2d4ce190b6e1de"},"cell_type":"code","source":"from sklearn.linear_model import Ridge\nfrom sklearn.metrics import mean_squared_error\n\ndef get_mse(model,alpha):\n    run_model = model(alpha = alpha)\n    run_model.fit(X_train,y_train)\n    return mean_squared_error(y_pred=run_model.predict(X_test),y_true=y_test)\n\nalphas = [.01,.05,0.1,0.5,1,2,3,5,10,20,40,100,1000,5000]\nridge_mses = [get_mse(Ridge,x) for x in alphas]\nplt.plot(alphas,ridge_mses)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3710a96b510259ffce5ca400fc37dd7d0391e873"},"cell_type":"code","source":"from sklearn.ensemble import RandomForestRegressor\nmodel = RandomForestRegressor()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b255e14641e415b7221922b0567a7fc8c3d7be32"},"cell_type":"code","source":"from sklearn.ensemble import AdaBoostRegressor\nmodel = AdaBoostRegressor()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"710cba9812e74033f955399de3aa981573221351"},"cell_type":"code","source":"from sklearn.ensemble import BaggingRegressor\nmodel = BaggingRegressor()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d637a6a8f6e7da45b3a8f15795eaa7d8ff7ae5b8"},"cell_type":"code","source":"from sklearn.linear_model import Lasso\nmodel = Lasso()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5ba6e7ea4d78ef3801047bf47ecffbf380de8189"},"cell_type":"code","source":"from sklearn.linear_model import ElasticNet\nmodel = ElasticNet()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"522114850480d9a8d18ea6e1a741f6e95160b645"},"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\nmodel = LinearRegression()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"98ad3a548fac9ba7ba51b14d829997225c189d36"},"cell_type":"code","source":"from sklearn.ensemble import ExtraTreesRegressor\nmodel = ExtraTreesRegressor()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f98c439f2bd19f70d74b0accdd628edce5a3c78e"},"cell_type":"code","source":"from sklearn.linear_model import \nmodel = ExtraTreesRegressor()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a9244908c6f15fe788ba348d297bad3ffe00b20e"},"cell_type":"code","source":"from sklearn.ensemble import GradientBoostingRegressor\nmodel = GradientBoostingRegressor()\nmodel.fit(X_train,y_train)\nmean_squared_error(y_pred=model.predict(X_test),y_true=y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c5c6e7a771408fa8f9b74c76616ae59af81640ad"},"cell_type":"code","source":"sample_submission.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ccdffb71e88d733e293da040c6fc73d77373113e"},"cell_type":"code","source":"results_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d42b347689462edb86b74f88e3b1b0954fb96bc9"},"cell_type":"code","source":"predictions = model.predict(results_test)\nsub = pd.DataFrame({'card_id':results_test.index, 'target':predictions})\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ca461dafb34e099a74f030529991507d4ff79e82"},"cell_type":"code","source":"sns.distplot(sub['target'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c955d47afe886e3a453914db52cd4a319971254b"},"cell_type":"code","source":"sns.distplot(train['target'])`","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d76e6db49375b512fb4bf3aa257188a981dfa16c"},"cell_type":"code","source":"sub.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}