{"cells":[{"metadata":{"_uuid":"9e97164a7191d9d43ba8158c0f7e13aa455a0ce9"},"cell_type":"markdown","source":"# Combining your model with a model without outlier"},{"metadata":{"_uuid":"e0f082732acbf893c2cd4c07200904effcfde928"},"cell_type":"markdown","source":"Assuming that you have already finished your feature engineering and you have two dataset:\n\n- ***train_clean.csv***\n- ***test_clean.csv***\n\nIn train_clean.csv, there's an **'outlier' column with values 1/0. **\n\nBesides, you have your best LB submission:\n- ***3.695.csv*** (thanks  **Ashish Patel(阿希什)** My original model can't rich this score, so I try to use the idea to improve your submission to get better LB socre.）\n\nThe flows of this pipline is as follows:\n1. Training a model using a training set without outliers. (we get: **Model_1**)\n2. Training a model to classify outliers. (we get: **Model_2**)\n3. Using **Model_2** to predict whether an card_id in test set is an outliers. (we get:**Outlier_Likelyhood**)\n4. Spliting out the card_id from **Outlier_Likelyhood** with top 10% (or some other ratio) score. (we get:**Outlier_ID**)\n5. Combining your submission using your **best submission (that is, your best model)** to predict **Outlier_ID** in test set and using **Model_1** to predict the rest of the test set.\n\nThe  basic idea behind this pipline is:\n1. Training model without outliers make the model more accurate for non-outliers.\n2. A great proportion of the error is caused by outliers, so we need to use a model training with outliers to predict them. How to find them out? build a classifier!"},{"metadata":{"trusted":true,"_uuid":"7dd3ea3236069cf9706e377d6594c39b5e5a5507"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport time\nimport lightgbm as lgb\nfrom sklearn.model_selection import StratifiedKFold, KFold\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.metrics import log_loss","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fcf141a7658f4089b1de2822ba41da9646c73835"},"cell_type":"markdown","source":"# Part 1 Training Model Without Outliers"},{"metadata":{"trusted":true,"_uuid":"aef67435b765a8c55e8a7653b47ad8d9071ea11b"},"cell_type":"code","source":"%%time\ndf_train = pd.read_csv('../input/predicting-outliers-to-improve-your-score/train_clean.csv')\ndf_test = pd.read_csv('../input/predicting-outliers-to-improve-your-score/test_clean.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"18d93d315eee398fb1cff4ad8d115c21d359cd49"},"cell_type":"markdown","source":"## filtering out outliers"},{"metadata":{"trusted":true,"_uuid":"3eb5ac784a25271b3433f6371d7b46cdabb420b5"},"cell_type":"code","source":"df_train = df_train[df_train['outliers'] == 0]\ntarget = df_train['target']\ndel df_train['target']\nfeatures = [c for c in df_train.columns if c not in ['card_id', 'first_active_month','outliers']]\ncategorical_feats = [c for c in features if 'feature_' in c]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8256bfde460ac4adc45852228a12194b2be6b497"},"cell_type":"markdown","source":"## parameters"},{"metadata":{"trusted":true,"_uuid":"a0ecb4a6605d477b290275f404a508bfacd614e0"},"cell_type":"code","source":"param = {'objective':'regression',\n         'num_leaves': 31,\n         'min_data_in_leaf': 25,\n         'max_depth': 7,\n         'learning_rate': 0.01,\n         'lambda_l1':0.13,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\":0.85,\n         'bagging_freq':8,\n         \"bagging_fraction\": 0.9 ,\n         \"metric\": 'rmse',\n         \"verbosity\": -1,\n         \"random_state\": 2333}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"204231995912b388ad8d6bed1c602c6fa29ebb4a"},"cell_type":"markdown","source":"## training model"},{"metadata":{"trusted":true,"_uuid":"1f153534282a64ea904b599143b023f56bf01e05","scrolled":true},"cell_type":"code","source":"%%time\nfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=2333)\noof = np.zeros(len(df_train))\npredictions = np.zeros(len(df_test))\nfeature_importance_df = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(df_train,df_train['outliers'].values)):\n    print(\"fold {}\".format(fold_))\n    trn_data = lgb.Dataset(df_train.iloc[trn_idx][features], label=target.iloc[trn_idx])#, categorical_feature=categorical_feats)\n    val_data = lgb.Dataset(df_train.iloc[val_idx][features], label=target.iloc[val_idx])#, categorical_feature=categorical_feats)\n\n    num_round = 10000\n    clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval= 100, early_stopping_rounds = 200)\n    oof[val_idx] = clf.predict(df_train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n    \n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"Feature\"] = features\n    fold_importance_df[\"importance\"] = clf.feature_importance()\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n    \n    predictions += clf.predict(df_test[features], num_iteration=clf.best_iteration) / folds.n_splits\n\nprint(\"CV score: {:<8.5f}\".format(mean_squared_error(oof, target)**0.5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4842069eb5b3fa8617b624c1d2fd00940c75137e"},"cell_type":"code","source":"model_without_outliers = pd.DataFrame({\"card_id\":df_test[\"card_id\"].values})\nmodel_without_outliers[\"target\"] = predictions","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"60b9a679754a23f8a07a4e06bf2003abdfbe57db"},"cell_type":"markdown","source":"# Part 2 Training Model For Outliers Classification"},{"metadata":{"trusted":true,"_uuid":"3f28c4be6b16f4e9d9e66d309a1c8f00175653f1"},"cell_type":"code","source":"%%time\ndf_train = pd.read_csv('../input/predicting-outliers-to-improve-your-score/train_clean.csv')\ndf_test = pd.read_csv('../input/predicting-outliers-to-improve-your-score/test_clean.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c2410c958c49b761f042ecdf5619af54730865d3"},"cell_type":"markdown","source":"## using outliers column as labels instead of target column"},{"metadata":{"trusted":true,"_uuid":"19cf3932dbbbc42a8e7e0c50ac7ae4c74fd15f62"},"cell_type":"code","source":"target = df_train['outliers']\ndel df_train['outliers']\ndel df_train['target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a60cc387b88592ae2ae6ad78f33d6adcce22780d"},"cell_type":"code","source":"features = [c for c in df_train.columns if c not in ['card_id', 'first_active_month']]\ncategorical_feats = [c for c in features if 'feature_' in c]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8867a7ba6c07d9a6c0ee04277960ff4d9c1559a8"},"cell_type":"markdown","source":"## parameters"},{"metadata":{"trusted":true,"_uuid":"b067562422807b5f08688c9c91e22c86998b07c8"},"cell_type":"code","source":"param = {'num_leaves': 31,\n         'min_data_in_leaf': 30, \n         'objective':'binary',\n         'max_depth': 6,\n         'learning_rate': 0.01,\n         \"boosting\": \"rf\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9 ,\n         \"bagging_seed\": 11,\n         \"metric\": 'binary_logloss',\n         \"lambda_l1\": 0.1,\n         \"verbosity\": -1,\n         \"random_state\": 2333}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7664c74d5ed84d82d0a045b5f2862cf813a52fd8"},"cell_type":"markdown","source":"## training model"},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"292376354fc58ca097cc04128d31f3117c648653"},"cell_type":"code","source":"%%time\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(len(df_train))\npredictions = np.zeros(len(df_test))\nfeature_importance_df = pd.DataFrame()\n\nstart = time.time()\n\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(df_train.values, target.values)):\n    print(\"fold n°{}\".format(fold_))\n    trn_data = lgb.Dataset(df_train.iloc[trn_idx][features], label=target.iloc[trn_idx], categorical_feature=categorical_feats)\n    val_data = lgb.Dataset(df_train.iloc[val_idx][features], label=target.iloc[val_idx], categorical_feature=categorical_feats)\n\n    num_round = 10000\n    clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=100, early_stopping_rounds = 200)\n    oof[val_idx] = clf.predict(df_train.iloc[val_idx][features], num_iteration=clf.best_iteration)\n    \n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"feature\"] = features\n    fold_importance_df[\"importance\"] = clf.feature_importance()\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n    \n    predictions += clf.predict(df_test[features], num_iteration=clf.best_iteration) / folds.n_splits\n\nprint(\"CV score: {:<8.5f}\".format(log_loss(target, oof)))","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"41f9c7d0a52f8d5669704619fdf76ad1451afaa2"},"cell_type":"code","source":"### 'target' is the probability of whether an observation is an outlier\ndf_outlier_prob = pd.DataFrame({\"card_id\":df_test[\"card_id\"].values})\ndf_outlier_prob[\"target\"] = predictions\ndf_outlier_prob.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a6cdbee1c92967a2ecb3924ab58af8d86675df31"},"cell_type":"markdown","source":"# Part 3 Combining Submission:\nSo far so good !\nWe now have three dataset:\n\n1. Best Submission\n2. Prediction Using Model Without Outliers\n3. Probability of Outliers In Test set\n"},{"metadata":{"trusted":true,"_uuid":"d56e431808a9a70a102491ac7d77e4404e312a4b"},"cell_type":"code","source":"# if the test set has the same ratio of outliers as training set, \n# then the numbuer of outliers in test is about: (1.06% outliers in training set)\n123623*0.0106","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"40e683026c0e476bae7d15a759b97fa027af977d"},"cell_type":"code","source":"# In case missing some predictable outlier, we choose top 25000 with highest outliers likelyhood.\noutlier_id = pd.DataFrame(df_outlier_prob.sort_values(by='target',ascending = False).head(25000)['card_id'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e4c9575fad64e1cf9bd38e48fcf5e988b850189f"},"cell_type":"code","source":"best_submission = pd.read_csv('../input/predicting-outliers-to-improve-your-score/3.695.csv')","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"7c6f4c87221dd9fa0bf2f87a60e46abbe39f7164"},"cell_type":"code","source":"most_likely_liers = best_submission.merge(outlier_id,how='right')\nmost_likely_liers.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0f1518b58748e9dd2ef3c1d92f161d03f923bac5"},"cell_type":"code","source":"%%time\nfor card_id in most_likely_liers['card_id']:\n    model_without_outliers.loc[model_without_outliers['card_id']==card_id,'target']\\\n    = most_likely_liers.loc[most_likely_liers['card_id']==card_id,'target'].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"21e2c892dc716481293cb0b34f6b67be3a912aa9"},"cell_type":"code","source":"model_without_outliers.to_csv(\"combining_submission.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"toc":{"base_numbering":1,"nav_menu":{},"number_sections":true,"sideBar":true,"skip_h1_title":false,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":false,"toc_position":{},"toc_section_display":true,"toc_window_display":false}},"nbformat":4,"nbformat_minor":1}