{"cells":[{"metadata":{"_uuid":"cb7281970e7c44017ff87d82a98828d023f03f45"},"cell_type":"markdown","source":"# Starter Notebook XGBoost + EDA of Elo Merchant Data"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport matplotlib.pylab as plt\nimport seaborn as sns\nimport sklearn\nimport xgboost as xgb\nplt.style.use('ggplot') # Lets make our plots pretty","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# Read in the dataframes\ntrain = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')\nmerch = pd.read_csv('../input/merchants.csv')\nht = pd.read_csv('../input/historical_transactions.csv')\nss = pd.read_csv('../input/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1341842455acc4eb06f7790d520e35d59b5c6625"},"cell_type":"code","source":"# Print data shapes\nprint('train shape', train.shape)\nprint('test shape', test.shape)\nprint('merchants shape', merch.shape)\nprint('sample submission shape', ss.shape)\nprint('historical_transactions', ht.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b8cb492042a84bb10c81c81362840595ee8db524"},"cell_type":"markdown","source":"# What files do I need?\nYou will need, at a minimum, the `train.csv` and `test.csv` files. These contain the card_ids that we'll be using for training and prediction.\n\n`train.csv` and `test.csv` contain `card_ids` and information about the card itself - the first month the card was active, etc. train.csv also contains the target."},{"metadata":{"trusted":true,"_uuid":"fea20404fa66741a76c0ac557bad3c60a87fd7b6"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b998d0604f7605e99dbc715575345e5579fa490e"},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cb6363f73ac635b62203201002cf2ecbdbf52537"},"cell_type":"markdown","source":"## Plot Features\nThere are three given features:\n- Feature 1 has 5 possible values\n- Feature 2 has 3 possible values\n- Feature 3 has 2 possible values\n\nTrain and test set appear so have similar distribution"},{"metadata":{"trusted":true,"_uuid":"a28a808065989f36dbdc76f87c813221b435471b"},"cell_type":"code","source":"features = ['feature_1', 'feature_2', 'feature_3']\nfor feature in features:\n    fig, axes = plt.subplots(nrows=1, ncols=2)\n    train.groupby(feature).count()['first_active_month'].plot(kind='bar',\n                                                              title='train {}'.format(feature),\n                                                              figsize=(15, 4),\n                                                              ax=axes[0])\n    test.groupby(feature).count()['first_active_month'].plot(kind='bar',\n                                                              title='test {}'.format(feature),\n                                                              figsize=(15, 4))\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"58ddb9cf991ca1efdff6f02980c490fe061311b3"},"cell_type":"markdown","source":"# Plot Target Variable\n- The target variable is normally distributed around zero\n- The exception being some very low values below -30\n- Possibly identify these low values and remove them from the training?\n- Still need to transform the target to make it more normally distributed ?"},{"metadata":{"trusted":true,"_uuid":"5543da0306d55ce38c0809d949e7b7b3d0297e6a"},"cell_type":"code","source":"train['target'].plot(kind='hist', bins=50, figsize=(15, 5), title='Target variable distribution')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"39b0dbececde52bce156c67cf0e4d04911450a18"},"cell_type":"code","source":"train['target_log5p'] = (train['target'] + 5).apply(np.log1p)\ntrain['target_log5p'].plot(kind='hist', bins=50, figsize=(15, 5), title='Target variable log+5 transform distribution')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9024c993cf50f23901faa8d7b4eb78071653b00"},"cell_type":"markdown","source":"# Plot the first_active_month - train and test"},{"metadata":{"trusted":true,"_uuid":"e7f34ca59c5a00d2de318131dcca483d277ffa5d"},"cell_type":"code","source":"train['first_active_month'] = pd.to_datetime(train['first_active_month'])\ntrain.groupby('first_active_month').count()['card_id'].plot(figsize=(15,5),\n                                                            title='Count of First Active Month in Train Set',\n                                                           color='r')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"20a76416e5a40b76582325b6e08d30408d2ddbd3"},"cell_type":"code","source":"test['first_active_month'] = pd.to_datetime(test['first_active_month'])\ntest.groupby('first_active_month').count()['card_id'].plot(figsize=(15,5),\n                                                           title='Count of First Active Month in Test Set',\n                                                          color='b')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c705a31a992de6b2fb2a8aa36f3df45feb34d410"},"cell_type":"markdown","source":"# Historical Transactions\n`historical_transactions.csv` contains up to 3 months' worth of transactions for every card at any of the provided merchant_ids."},{"metadata":{"trusted":true,"_uuid":"b7103723e4921b3d2ee4f0aff765c060d8fdb3b0"},"cell_type":"code","source":"ht.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"de51ddeb8cfc7db8f6be03919ccfc6a15e1a4347"},"cell_type":"markdown","source":"# Create Features and bare bones first attempt XGBoost model"},{"metadata":{"trusted":true,"_uuid":"1eedc99e8cd764b99eb4b3cb2ff1f262e2ba2a7a"},"cell_type":"code","source":"# Create features from historic transactions\ngdf = ht.groupby(\"card_id\")\ngdf = gdf[\"purchase_amount\"].agg(['sum', 'mean', 'std', 'min', 'max']).reset_index()\ngdf.columns = [\"card_id\", \"sum_hist_trans\", \"mean_hist_trans\", \"std_hist_trans\", \"min_hist_trans\", \"max_hist_trans\"]\n\ndef merge_hist_transactions(df, hist_agg_df):\n    \"\"\"Merges the historic transactions data with train or test set\"\"\"\n    return pd.merge(df, hist_agg_df, on=\"card_id\", how=\"left\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0ff9309bcc59af7750ab2fcdd9040ca9dbed37a3"},"cell_type":"code","source":"def create_features(df, target=False):\n    \"\"\"\n    Creates raw features including one hot encoding for train and test\n    \"\"\"\n    # One-hot encode features\n    feat1 = pd.get_dummies(df['feature_1'], prefix='f1_')\n    feat2 = pd.get_dummies(df['feature_2'], prefix='f2_')\n    feat3 = pd.get_dummies(df['feature_3'], prefix='f3_')\n    # Numerical representation of the first active month\n    fam_num = pd.to_timedelta(df['first_active_month']).dt.total_seconds().astype(int)\n    # Historical features\n    hist_features = df[['sum_hist_trans','mean_hist_trans','std_hist_trans','min_hist_trans','max_hist_trans']]\n    if target:\n        return pd.concat([feat1, feat2, feat3, fam_num, hist_features], axis=1, sort=False), df['target']\n    return pd.concat([feat1, feat2, feat3, fam_num, hist_features], axis=1, sort=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c9ce5b8a8bc837a62e77abf40ef595d0c2a53d81"},"cell_type":"markdown","source":"# Na Values\nWe have one NA value in our test set for the first active month"},{"metadata":{"trusted":true,"_uuid":"8cf1094ad6c23ee246fd5e29d837d410f98f2eb5"},"cell_type":"code","source":"# Fill the NA value with the last month?\ntest['first_active_month'] = pd.to_datetime(test['first_active_month'].fillna('2018-01-01'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cf9436c91d15cd4a72aeaf4d8bedcf33949702aa"},"cell_type":"code","source":"train_with_hist = merge_hist_transactions(train, gdf)\ntest_with_hist = merge_hist_transactions(test, gdf)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fd74c62c8943e9c4b2e6edfc4d7f74fac798b516"},"cell_type":"code","source":"X_train, y_train = create_features(train_with_hist, target=True)\nX_test = create_features(test_with_hist, target=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66f638460309651b2c5d6e01ed741eceac653990"},"cell_type":"markdown","source":"# Train model"},{"metadata":{"trusted":true,"_uuid":"59809e4e7ce5c268fc440340d90ad35f383c68fe"},"cell_type":"code","source":"dtrain = xgb.DMatrix(X_train, label=y_train.values)\nmodel = xgb.train(params={'silent':1}, dtrain=dtrain, verbose_eval=False, num_boost_round=100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a8a718a19db7f322812a616225466241011d89f8"},"cell_type":"code","source":"# Predict and format submission\ndtest = xgb.DMatrix(X_test)\npreds = model.predict(dtest)\n# Our submission\nsubmission = pd.concat([test['card_id'], pd.Series(preds)], axis = 1)\nsubmission = submission.rename(columns={0:'target'})\nprint('submission shape', submission.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2ac5aac54ed331aac696f5598170836b80a0eb45"},"cell_type":"code","source":"submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e4aec073b80a3f689fdf803a0af6def337033df7"},"cell_type":"code","source":"# Save our output for submission\nsubmission.to_csv('submission.csv', header=True, index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"30481cb237bace90524df538b15714af433d684b"},"cell_type":"markdown","source":"# Train model removing <-30 targets from training\nJust as a test lets try removing these values from our training"},{"metadata":{"trusted":true,"_uuid":"ef9dd4ed6fe3f2ff7833e3e66090de9ec39d3cc0"},"cell_type":"code","source":"train_remneg30 = train_with_hist.loc[train_with_hist['target'] > -30]\nX_train_rem30, y_train_rem30 = create_features(train_remneg30, target=True)\n# X_test = create_features(test, target=False)\ndtrain = xgb.DMatrix(X_train_rem30, label=y_train_rem30.values)\nmodel2 = xgb.train(params={'silent':1}, dtrain=dtrain, verbose_eval=False, num_boost_round=100)\n# Predict and format submission\ndtest = xgb.DMatrix(X_test)\npreds = model2.predict(dtest)\n# Our submission\nsubmission2 = pd.concat([test['card_id'], pd.Series(preds)], axis = 1)\nsubmission2 = submission2.rename(columns={0:'target'})\nsubmission2.to_csv('submission_remneg30.csv', header=True, index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e19de9c6aa1334416fce883acd49662e8d1116ed"},"cell_type":"markdown","source":"# Compare the two submission distributions"},{"metadata":{"trusted":true,"_uuid":"8a4e2427262cb817a8faae898f37f739ec0233de"},"cell_type":"code","source":"submission2['target'].plot(kind='hist', bins=50, title='submission removing <-30 values')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2fcefce8b85275b5fcb9f09260ea7bc1f5ae066b"},"cell_type":"code","source":"submission['target'].plot(kind='hist', bins=50, title='submission not removing <-30 values')\nplt.show()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"60d3de0999691b1b5af6904613f4c1978c7ec0a5"},"cell_type":"markdown","source":"# Still to come.....\n1. Explore the below additional data\n2. Create a more complex model using historical transactions\n3. Utilize new merchant transactions\n\n## The `historical_transactions.csv` and `new_merchant_transactions.csv` files contain information about each card's transactions.\n\n## `historical_transactions.csv` contains up to 3 months' worth of transactions for every card at any of the provided merchant_ids.\n\n## `new_merchant_transactions.csv` contains the transactions at new merchants (merchant_ids that this particular card_id has not yet visited) over a period of two months.\n\n## `merchants.csv` contains aggregate information for each merchant_id represented in the data set.\n"},{"metadata":{"trusted":true,"_uuid":"5eef4975ad5bddaa92628f3262ab7911382dc259"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}