{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":85540,"databundleVersionId":9725394,"sourceType":"competition"}],"dockerImageVersionId":30775,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"The goal of this notebook is to help give you a jump start working with kaggle by showing you how to submit your predictions to the leaderboard. Through this guide we will make the submission that is known as \"benchmark.csv\" on the leaderboard. That way you will have more time to focus on feature engineering and model development, which should ultimately lead to a set of predictions that greatly outperform the benchmark.\n\n\nLets start by loading the relevant software packages and tabular data:","metadata":{}},{"cell_type":"code","source":"#Load Software\nfrom sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score\nfrom sklearn.model_selection import RandomizedSearchCV, GridSearchCV\nfrom sklearn.ensemble import AdaBoostRegressor\nfrom catboost import CatBoostRegressor\nimport pandas as pd, numpy as np, os, random, sklearn\n\n#Load Data\nROOT='/kaggle/input/applications-of-deep-learning-wustl-fall-2024/'\ntest=pd.read_csv(f\"{ROOT}/df-test_200x11.csv\", dtype={'homeid': str})\ntrain=pd.read_csv(f\"{ROOT}/df-train_1000x12.csv\", dtype={'homeid': str})","metadata":{"execution":{"iopub.status.busy":"2024-10-03T19:26:22.036619Z","iopub.execute_input":"2024-10-03T19:26:22.037072Z","iopub.status.idle":"2024-10-03T19:26:26.337426Z","shell.execute_reply.started":"2024-10-03T19:26:22.037027Z","shell.execute_reply":"2024-10-03T19:26:26.336243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Preprocess Data**\n\nAt this point we would derive and engineer new features to go into the predictive model, but for the sake of this example, I will only use the default features available. The code below completes the mimimum nessessary steps for model development.\n","metadata":{}},{"cell_type":"code","source":"#Training Data\nX_train=train.drop(columns=['homeid','target_homevalue'])\ny_train=train['target_homevalue']\n\n#Testing Data\nX_test=test.drop(columns=['homeid'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Prepare Grid Search**\n\nIt is not clear which predictive algorithem would be best suited for this challenge, and it is possible that a model ensembling framework might lead to the best solution. Here, I am using an Adaptive Tree Boosting algorithem with Catigorical Boosting Regressors, which is an ensemble of decision trees that is powerful, but can be prone to overfitting.\n\n\nYou can control the complexity of this type of algorithem based on its number of trees and the maximum depth that a given tree can be. The code below tunes both of these hyperparameters for the AdaBoostRegressors and CatBoostRegressors. Lastly, I have lowered the learning rate to improve covergance stability and have more percision around the most optimal model parameters.","metadata":{}},{"cell_type":"code","source":"#Define Hyper-parameter Search Space\nparameters = {\n    'estimator__n_estimators': [8, 16, 32, 64, 128, 256, 512],\n    'estimator__max_depth': [3],\n    'estimator__learning_rate': [0.05],    \n\t'n_estimators': [2, 4, 6],\n\t'learning_rate': [0.1]\n}","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Optimize Predictive Algorithem**\n\nThe code below optimizes the tree-based algorithem using a 5-fold grid search to find the most optimal hyperparamters. Prior to training the model, I defined which variables were categorical, as the CatBoostRegression is capible of incorporting categorical features without the need for one-hot or label encoding, a unique advantage of this algorithem relative to using XGBoost or DecisionTreeRegressors as the base estimitors.","metadata":{}},{"cell_type":"code","source":"#Define and Train Predictive Algorithem\ncategorical = list(X_train.select_dtypes(include=['object', 'category']).columns)\ncategorical_indices = [X_train.columns.get_loc(col) for col in categorical]\nmodel = CatBoostRegressor(cat_features=categorical_indices,verbose=0)\ngrid = GridSearchCV(\n    estimator=AdaBoostRegressor(estimator=model),\n\tparam_grid=parameters,\n\tcv=5,\n\tn_jobs=-1,\n\tscoring=\"neg_mean_absolute_error\"\n)\nmodel = grid.fit(X_train, y_train.values.ravel())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Derive Predictions on the Testing Data**\n\nThe input features of our predictive algorithem should be the same for both the training and testing data. Therefore, applying our model to the testing data is relatively easy, but to evaluate the predictive accuracy of your model on the testing data you need to submit your perdictions to the Kaggle leaderboard. To complete this all you need to do is append the **homeid** from the testing data to the predictions, as shown below:","metadata":{}},{"cell_type":"code","source":"#Generate Home Value Predictions\npredictions = pd.DataFrame({\n\t'homeid': test['homeid'], \n\t'target_homevalue': np.round(model.predict(X_test),decimals=0).astype(int),\n}); \n\npredictions.to_csv('/kaggle/working/benchmark.csv',index=False)\nprint(predictions.head(10))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Good luck and Godspeed!**\n\nThe mean absolute error across this set of predictions was **255,198 dollars**, indicating the deafult features did not derive a very robust tool that could be use to buy and sell houses. With more feature engineering and model development, I am optimistic that you can derive a  solution with a mean error under **100,000 dollars**, but I encourage you to get as close to **0** as you can.","metadata":{}},{"cell_type":"markdown","source":"**Downloading Data via Kaggle API (Optional)**\n\nYou have a number of alternative computing evironments that can be used to work on this kaggle compeition, including your local machine, google colab, and the WUSTL high performance cluster. In case, you would like to use an alternative environment, below is the bash code to help you download the data via the Kaggle API.\n\nYou can find your API key if you go to your profile, click settings in the top right corner, and scroll down to \"create new token\" under the API section. Once created your key will be downloaded as a file labeled 'kaggle.json'. This file needs to be stored in the following directory: '~/.kaggle/'. Afterward you should be able to complete the data download using the commented-out code below.","metadata":{}},{"cell_type":"code","source":"#mkdir -p ~/.kaggle\n#mv ~/Downloads/kaggle.json ~/.kaggle/\n#pip3 install kaggle -q\n#kaggle competitions download -c applications-of-gen-ai-wustl-fall-2024\n#unzip applications-of-gen-ai-wustl-fall-2024.zip","metadata":{},"execution_count":null,"outputs":[]}]}