{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nsns.set_theme()\nsns.set_style(\"ticks\")\nsns.despine()\n\n%matplotlib inline\n\nplt.rc('savefig', dpi=300)","metadata":{"execution":{"iopub.status.busy":"2022-11-21T09:45:57.179432Z","iopub.execute_input":"2022-11-21T09:45:57.180580Z","iopub.status.idle":"2022-11-21T09:45:58.204765Z","shell.execute_reply.started":"2022-11-21T09:45:57.180470Z","shell.execute_reply":"2022-11-21T09:45:58.203041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Experiment Log File","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"../input/home-loan-default-risk-prediction-experiments/log_file.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-11-21T09:45:58.206771Z","iopub.execute_input":"2022-11-21T09:45:58.207179Z","iopub.status.idle":"2022-11-21T09:45:58.226772Z","shell.execute_reply.started":"2022-11-21T09:45:58.207142Z","shell.execute_reply":"2022-11-21T09:45:58.225577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-21T09:45:58.228571Z","iopub.execute_input":"2022-11-21T09:45:58.229032Z","iopub.status.idle":"2022-11-21T09:45:58.260289Z","shell.execute_reply.started":"2022-11-21T09:45:58.228981Z","shell.execute_reply":"2022-11-21T09:45:58.258614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Metric by Each Strategy\n\nEach experiment is a combination of the following:\n- **Imputers** - Simple imputer using mean, simple imputer using median and iterative imputer\n- **Scalers** - Standard scaler and min-max scaler.\n- **Encoders** - One-hot encoding and label encoding. Note that in case of one-hot encoding, `handle_unknown` is set to `True` and `drop` is set to `if_binary`. That is, unknown categories in the validation set are encoded as all 0s and binary categorical features have only one column associated with them. In case of label encoding, unknown categories are mappped to `-1`.\n- **Estimators** - Logistic Regression, AdaBoost, Random Forest, XGBoost and LightGBM.\n\nThus, there are a total of 60 experiments. Each experiment measures the macro F1 score per fold for the train and validation sets, and the final average macro F1 score.\n\nBelow, we look at the average validation score according to each component of the experiment.","metadata":{}},{"cell_type":"code","source":"columns = [\"imputer\", \"scaler\", \"encoder\", \"estimator\"]","metadata":{"execution":{"iopub.status.busy":"2022-11-21T09:46:02.741914Z","iopub.execute_input":"2022-11-21T09:46:02.742373Z","iopub.status.idle":"2022-11-21T09:46:02.748569Z","shell.execute_reply.started":"2022-11-21T09:46:02.742329Z","shell.execute_reply":"2022-11-21T09:46:02.746973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We note that:\n- The iterative imputer beats the simple imputer.\n- The standard scaler slightly outperforms the minmax scaler.\n- One-hot encoding performs better than label encoding.\n- XGBoost comes out to be the best model.\n\nFrom the graphs, we can also conclude that the model is the dominant factor in deciding the performance of the overall pipeline. The other experiment parameters do not affect the score by a huge margin.","metadata":{}},{"cell_type":"code","source":"f, axes = plt.subplots(2, 2, figsize=(20, 10))\n\nfor col, ax in zip(columns, axes.flat):\n    sns.barplot(data=df, x=col, y=\"avg_val_score\", ax=ax)\n    ax.bar_label(ax.containers[0], fmt=\"%.4f\")\n        \nplt.tight_layout()\n\nplt.savefig(\"metric_by_strategy.png\")","metadata":{"execution":{"iopub.status.busy":"2022-11-21T09:46:54.828717Z","iopub.execute_input":"2022-11-21T09:46:54.829078Z","iopub.status.idle":"2022-11-21T09:46:56.877485Z","shell.execute_reply.started":"2022-11-21T09:46:54.829047Z","shell.execute_reply":"2022-11-21T09:46:56.875740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Top 5 Performers\n\nWe see that XGBoost is the estimator in the top 5 experiments. The best combination is as follows:\n- Iterative imputer\n- Min-max scaler\n- One-hot Encoding\n- XGBoost estimator\n\nThe next best experiment is the same as above, except it uses label encoding. The difference between their scores is only ~0.0002. It might be that with hyperparameter tuning, label encoding beats one-hot encoding. Thus, we will train with both the experiments and use the best submission.","metadata":{}},{"cell_type":"code","source":"df.nlargest(5, \"avg_val_score\")","metadata":{"execution":{"iopub.status.busy":"2022-11-21T09:46:09.600857Z","iopub.execute_input":"2022-11-21T09:46:09.601274Z","iopub.status.idle":"2022-11-21T09:46:09.625286Z","shell.execute_reply.started":"2022-11-21T09:46:09.601239Z","shell.execute_reply":"2022-11-21T09:46:09.624149Z"},"trusted":true},"execution_count":null,"outputs":[]}]}