{"cells":[{"metadata":{"_uuid":"c7e2833e14266d309c95c415429e618b9d98685c"},"cell_type":"markdown","source":"# Introduction\nThis note book should serve as a guide for setting up a basic pipeline for a kaggle competition. It walks through the basic steps of simple EDA, feature engineering, CV setup, model setup, submissions, and working in the kaggle environment."},{"metadata":{"_uuid":"5bd7e2d769375c7ca9da182cba79cf56da3c5966"},"cell_type":"markdown","source":"## Setup\nAdding competition datasources is easy! On the right toolbar, click the add data button, navigate to the 'competitions' tab, and select the relevant competition. You can also use this feature to add data from other public kernels. These files can be accessed via a relative path from your notebook/script in kaggle."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport time\nfrom datetime import datetime\nimport lightgbm as lgb\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.model_selection import KFold\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n\nimport os\nprint(os.listdir(\"../input\")) # Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"## Data Exploration\n\nFor the purpose of this kernel, let's just look at the core data set in train.csv and test.csv"},{"metadata":{"trusted":true,"_uuid":"014baa6376e19e4f5141ab014207d2b538eec899"},"cell_type":"code","source":"train_df = pd.read_csv('../input/train.csv')\ntest_df = pd.read_csv('../input/test.csv')\nsubmit_df = pd.read_csv('../input/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41780ba6e37d1621df95ea310a580ba27d073c81"},"cell_type":"code","source":"print(train_df.shape)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"036c6eb681268540d995ce1aba26ca99bf4b967a"},"cell_type":"code","source":"print(test_df.shape)\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d199e12fc7f52d16c3144ab7747ebe2887df8e1e"},"cell_type":"markdown","source":"### Some information to check\n\n* are card_ids unique?\n* what do the 'feature' distributions look like?\n* How does train differ from test?"},{"metadata":{"trusted":true,"_uuid":"64cf25942d3f792d8eeca9f539c7d53851f3c436"},"cell_type":"code","source":"# it'll be useful to combine the train and test data into a single dataframe to see how they differ\n\ndf = pd.concat([train_df, test_df], sort=False)\ndf['is_train'] = df['target'].notnull() \ndf['first_active_month'] = df['first_active_month'].apply(pd.to_datetime)\ndf.sample(4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4fc6dd1b048ee07559029c3f65d3ed69c607fec0"},"cell_type":"code","source":"# card ids are indeed unique identifiers\nprint(df.card_id.nunique() / df.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"6ec719ff2b4b42a0157928e0b54aac68c76330d2"},"cell_type":"code","source":"# lets look at the first_active_month column. I suspect that train and test differ greatly\ndef df_pivot_distribution(df, index, split='is_train', count_on='card_id'):\n    return df.groupby([index, split]).count()[count_on].reset_index().pivot(index=index, columns=split, values=count_on)#.plot.bar(figsize=(20,6), stacked=True)\n\ndf_pivot_distribution(df, 'first_active_month').plot.bar(stacked=True, figsize=(15, 4))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6268d1fa914e75435dce67b8f0ebd30cd380d74d"},"cell_type":"markdown","source":"Interestingly, the train and test set actually seem to split evenly around the dates. The majority of these dates occur in 2017."},{"metadata":{"_uuid":"19416fe47d7a4c64b125e4545a78bf064903cf76"},"cell_type":"markdown","source":"So, what do the mysterious \"features\" looks like?"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"bc6e7d7a3ddec1bbba94ed4898182cde5bd71d6d"},"cell_type":"code","source":"df_pivot_distribution(df, 'feature_1').plot.bar()\ndf_pivot_distribution(df, 'feature_2').plot.bar()\ndf_pivot_distribution(df, 'feature_3').plot.bar()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6951367a475a4aef54a4cc120b8f140c3901df6d"},"cell_type":"markdown","source":"There isn't too much to infer from this, since these features are already engineered and anonymous, but it's useful to know that the train and test set don't differ too much in their populations. "},{"metadata":{"trusted":true,"_uuid":"091cd0963ed3ae64d0fb4c8cabe566da1a9193e0"},"cell_type":"code","source":"df['target'].describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f8532fb0c79db0806cbfaac9ea3f2047d02148a0"},"cell_type":"markdown","source":"## Feature Engineering\n\nFeature engineering tends to be the most critical and time consuming process in these competitions. For this demo, since we're only using a small subset of the data, it'll be a bit simple. \n\nEssentially, the end goal of this process is to have a dataframe, indexed by card_id, with a bunch of columns of data that are expected to help predict *customer loyalty*\n\nThe role of the **model** is to **learn** from this data that has a non-null **target** field to predict the loyalty score for those in the test data"},{"metadata":{"trusted":true,"_uuid":"642c6b2824cc0604ee6895f560f527b93201f747"},"cell_type":"code","source":"# since the columns labeled 'features' are already engineered, let's try to get useful information from the date field\n\ndf['first_active_mo'] = df['first_active_month'].dt.month\ndf['first_active_yr'] = df['first_active_month'].dt.year","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"47e442ab19ba39f65d3b7fe44ee5b1978c3a3b70"},"cell_type":"code","source":"# Our 'feature' features appear to be categorical - rather than actually numeric. We may get clearer signal if we\n# don't treat them as sequential\n\ncat_columns = ['feature_1', 'feature_2', 'feature_3', 'first_active_yr']\ndf = pd.get_dummies(df, columns=cat_columns)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e54533c5eb52311f80f09078f4090fc421202b94"},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b905748b806a9729b6fbf5a0b5ddaa0b463f4493"},"cell_type":"markdown","source":"## Modeling and Cross-validation\n\nAt this point modeling can be relatively straight forward since our data is organized pretty well into train and test, and since our features have been developed. However, it is equally important to develop a technique for checking how accurate your model is at predicting the target. \n\nIn kaggle competitions we can make submissions. However public submissions are only validated against a small portion of the test set. Additionally, since this test set is static, it is easy to fall into the trap of overfitting your model to the public submission set. Lastly, kaggle submissions are limited to ~5 per day, so measuring how small changes affect your model isn't really possible. This is why we set up cross-validation as part of our pipeline.\n\nCross validation is simply a technique for using your existing data to measure the performance of your model. The most basic way of doing this is simply splitting up your training data into 80% for training and 20% into validating it. The problem with this method is that we lose 20% of our training that could've helped us improve our model. One potential solution for this is to use **K-fold** validation.\n\nThe strategy is pretty simple. We split our training data into K *folds* (in this example we split it into fifths). For each fold, we train the model on the other four folds and predict on the fifth. This allows us train and validate over the entire dataset. The test set predictions are made from the average of the five models we just made.\n\nK-Folds isn't always the best choice for cross-validation, but it's a strong place to start. One potential weakness of it is that it ignore any natural pattern that your train and test set follow. If you know that all of your test data occurs 6 months after your training data, than your K-fold tested model may not generalize well to the test set.\n\n<img src=\"https://i.stack.imgur.com/1fXzJ.png\" width=\"800px\"/> \nimage from https://www.kaggle.com/dansbecker/cross-validation\n\n\nThe code below runs through a simple 5 fold K-Fold and uses a gradient boosting model that tends to be popular in kaggle competitions. For more on the model, the documentation can ve found at https://lightgbm.readthedocs.io/en/latest/\n"},{"metadata":{"trusted":true,"_uuid":"88589d08c15a8ab5e69ba2e1d0b98cb7160847ff","scrolled":true},"cell_type":"code","source":"# params taken from public kernel https://www.kaggle.com/peterhurford/you-re-going-to-want-more-categories-lb-3-737\nparam = {'num_leaves': 50,\n         'min_data_in_leaf': 30, \n         'objective':'regression',\n         'max_depth': -1,\n         'learning_rate': 0.0041,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 0.1,\n         \"verbosity\": -1}\n\n# seperate our train, test, and targets\ntrain_df = df.loc[df['is_train'] == True].set_index('card_id')\ntest_df = df.loc[df['is_train'] == False].set_index('card_id')\ntarget = train_df['target']\n\n# we only want to pass our import numeric features to the model\nignored_columns = ['first_active_month', 'target', 'is_train']\ntrain_df = train_df[[ix for ix in train_df.columns if ix not in ignored_columns]]\ntest_df = test_df[[ix for ix in test_df.columns if ix not in ignored_columns]]\n\n#initalize our outputs and our folds\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(len(train_df))\npredictions = np.zeros(len(test_df))\nstart = time.time()\nfeatures = list(train_df.columns)\nfeature_importance_df = pd.DataFrame()\n\n# loop through the folds, train the model, and make a prediction\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_df.values, target.values)):\n    trn_data = lgb.Dataset(train_df.iloc[trn_idx].values, label=target.iloc[trn_idx].values)\n    val_data = lgb.Dataset(train_df.iloc[val_idx].values, label=target.iloc[val_idx].values)\n    \n    num_round = 10000\n    clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=100, early_stopping_rounds = 100)\n    oof[val_idx] = clf.predict(train_df.iloc[val_idx].values, num_iteration=clf.best_iteration)\n    \n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"feature\"] = features\n    fold_importance_df[\"importance\"] = clf.feature_importance()\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n    \n    predictions += clf.predict(test_df.values, num_iteration=clf.best_iteration) / folds.n_splits\n    \nprint(\"Total OOF RMSE: {}\".format(np.sqrt(mean_squared_error(target, oof))))\nprint(\"Our Baseline model (predicting all 0's) RMSE:{}\".format(np.sqrt(mean_squared_error(target, np.array([0] * len(oof))))) )","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e71d9ebae8ae997e7b39c09ba51b2e3707201dd0"},"cell_type":"markdown","source":"## Prepare Submission File\nSubmitting from a kaggle kernel is pretty simple. Simply output a csv that resembles the sample submission. In this competition the column headers should be *card_id* and *target* where target is our predicted value"},{"metadata":{"trusted":true,"_uuid":"7a4686173582eb7f41255682c5de79147c1109b9"},"cell_type":"code","source":"test_df['target'] = predictions\ntest_df.reset_index()[['card_id', 'target']].to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e28cdf814636d0807bec0a7361d55aa37d32b867"},"cell_type":"markdown","source":"## Moving Forward\n\nOverall, our model is pretty weak and barely outperforms simply guessing 0 for every target. Improvement over this score primarily comes from two sources:\n\n1) Feature engineering\n*  We only use one small data source, but this competition has tons of data that contains powerful information for predicting our target\n* Clever use of combining interactions between features will give information that the model may not learn\n\n2) Model selection and ensembling\n* We only used a simple, out of the box, solution to quickly get a prediction. It's possible that other models may perform better, or that better hyper parameters will perform better"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}