{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🎯Hyperparameter Optimization for Machine Learning\n\nThe aim of this notebook:\n> - Discuss multiple ways to optimize hyperparameters.\n> - Understand the logic of each technique.\n> - Considerations when utilizing each technique.\n> - Master the use of Python open-source for hyperparameter tuning.\n\n---\n\n## Parameters in ML models\n> - The objective of a typical learning algorithm is to find a function `f` that minimizes a certain `loss` over a `dataset`.\n> - The learning algorithm produces `f` through the optimization of a training criteron with respect to a set of `parameters`.\n\n---\n\n## Hyperparameters in ML models\n> - Hyperparameters are parameters that are not directly learnt by the learning algorithm.\n> - Hyperparameters are specified outside of the training procedure.\n> - Hyperparameters control the capacity of the model, i.e., how flexible the model is to fit the data.\n> - Prevent over-fitting.\n> - Hyperparameters could have a big impact on the performance of the learning algorithm.\n> - Optimal hyperparameter settings often differ for different datasets. Therefore they should be optimized for each dataset.\n---\n\n## Hyperparameter Nature\n>- Some hyperparameters are discrete: Number of estimators in ensemble models.\n>- Some hyperparameters are continuous: Penalization coefficient, Number of samples per split.\n>- Some hyperparameters are categorical: Loss (deviance, exponential), Regularization (Lasso, Ridge)\n\n---\n\n## Parameters vs Hyperparameters\n\n|Parameters                  |   Hyperparameters |\n|:-------------------------|----------------------:|\n| - Intrinsic to model equation     | - Defined before training |\n| - Optimized during training | - Constrain the algorithm|\n\n> - The process of finding the best Hyperparameters for a given dataset is called `Hyperparameter Tuning` or `Hyperparameter Optimization`.\n\n---\n\n## Challenges\n>- We can't define a formula to find the hyperparameters.\n>- Try different combinations of hyperparameters and evaluate model performance. The critical step is to choose how many different combinations we are going to test.\n\nThe number of hyperparameter combination ---> the chance to get a better model ---> Computational cost\n\n>- How do we find the hyperparameter combinations to maximize performance while diminishing computational costs?\n\n---\n\n## Methods\nDifferent hyperparamete optimization strategies:\n>- Manual Search\n>- Grid Search\n>- Random Search\n>- Bayesian Optimization\n\n---\n\n## Generalization vs Over-fitting\n> Generalization is the ability of an algorithm to be effective across various inputs. The performance of the machine learning model is constant across different datasets (with the same distribution on the training data). When the model performs well on the train set, but not on new / naive data, the model over-fits to the training data.\n\n---\n\n## Training a Machine Learning Model\n> To prevent over-fitting, it is common practice to:\n> - Separate the data into a train and a test set.\n> - Train the model in the train set.\n> - Evaluate in the test set.","metadata":{}},{"cell_type":"markdown","source":"# Hyperparameter Optimization\n\n## Grid Search\n\n>- Exhaustive search through a specified subset of hyperparameters of a learning algorithm.\n>- Examines all possible combinations of the specified hyperparameters (Cartesian product of hyperparameters).\n\n### Limitations\n>- Curse of dimentionality: possible combinations grow exponentially with the number of hyperparameters.\n>- Computationally expensive.\n>- Hyperparameter values are determined manually.\n>- Not ideal for continuous hyperparameters.\n>- Does not explore the entire hyperparameter space (not feasible).\n>- It performs worse than other searches (for models with complex hyperparameter spaces).\n\n### Advantages\n>- For models with simpler hyperparameter spaces works well.\n>- It can be parallelized.\n\nGrid Search is the most expensive method in terms of total computation time. However, if run in parallel, it is fast in terms of wall clock time. Sometimes, we run a small grid, determine where the optimum lies, and then expand the grid in that direction.","metadata":{}},{"cell_type":"markdown","source":"# Tuning XGBoost with Optuna\n\nIn this example, the hyperparameters `learning_rate`, `max_depth`, `n_estimators`, and `min_child_weight` are tuned using the Optuna library. The objective function is defined to return the negative accuracy on the test set, as Optuna minimizes the objective function. The `study.optimize` function is used to run the hyperparameter tuning, with `n_trials` specifying the number of trials to run. The final performance of the tuned classifier is evaluated on the test set.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom sklearn.datasets import load_breast_cancer\nfrom sklearn.model_selection import train_test_split\nfrom xgboost import XGBClassifier\nimport optuna\n\n# Load the breast cancer dataset\ndata = load_breast_cancer()\nX = data.data\ny = data.target\n\n# Split the data into training and testing sets\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n\ndef objective(trial):\n    # Define the hyperparameters to tune\n    learning_rate = trial.suggest_float(\"learning_rate\", 1e-5, 1e-1)\n    max_depth = trial.suggest_int(\"max_depth\", 3, 7)\n    n_estimators = trial.suggest_int(\"n_estimators\", 100, 1000)\n    min_child_weight = trial.suggest_int(\"min_child_weight\", 1, 5)\n    \n    # Create an XGBoost classifier\n    clf = XGBClassifier(\n        learning_rate=learning_rate, \n        max_depth=max_depth,\n        n_estimators=n_estimators, \n        min_child_weight=min_child_weight\n    )\n    \n    # Train the classifier and calculate the accuracy on the validation set\n    clf.fit(X_train, y_train)\n    score = clf.score(X_test, y_test)\n    \n    return 1.0 - score\n\n# Use Optuna to tune the hyperparameters\nstudy = optuna.create_study()\nstudy.optimize(objective, n_trials=100)\n\n# Print the best hyperparameters and the best score\nprint(\"Best hyperparameters: \", study.best_params)\nprint(\"Best score: \", 1.0 - study.best_value)\n\n# Train the classifier with the best hyperparameters on the full training set\nbest_params = study.best_params\nclf = XGBClassifier(\n    learning_rate=best_params[\"learning_rate\"], \n    max_depth=best_params[\"max_depth\"],\n    n_estimators=best_params[\"n_estimators\"], \n    min_child_weight=best_params[\"min_child_weight\"]\n)\nclf.fit(X, y)\n\n# Evaluate the tuned classifier on the test set\nscore = clf.score(X_test, y_test)\nprint(\"Test set accuracy: \", score)","metadata":{"execution":{"iopub.status.busy":"2023-02-03T19:31:44.561441Z","iopub.execute_input":"2023-02-03T19:31:44.562541Z","iopub.status.idle":"2023-02-03T19:35:29.238252Z","shell.execute_reply.started":"2023-02-03T19:31:44.562498Z","shell.execute_reply":"2023-02-03T19:35:29.237305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Tuning Random Forest with Optuna\n\nIn this example, the hyperparameters `n_estimators`, `max_depth`, `min_samples_split`, and `min_samples_leaf` are tuned using the Optuna library. The objective function is defined to return the negative accuracy on the test set, as Optuna minimizes the objective function. The `study.optimize` function is used to run the hyperparameter tuning, with `n_trials` specifying the number of trials to run. The final performance of the tuned classifier is evaluated on","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\n# Load the breast cancer dataset\ndata = load_breast_cancer()\nX = data.data\ny = data.target\n\n# Split the data into training and testing sets\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n\ndef objective(trial):\n    # Define the hyperparameters to tune\n    n_estimators = trial.suggest_int(\"n_estimators\", 100, 1000)\n    max_depth = trial.suggest_int(\"max_depth\", 3, 7)\n    min_samples_split = trial.suggest_int(\"min_samples_split\", 2, 5)\n    min_samples_leaf = trial.suggest_int(\"min_samples_leaf\", 1, 5)\n    \n    # Create a random forest classifier\n    clf = RandomForestClassifier(\n        n_estimators=n_estimators, \n        max_depth=max_depth,\n        min_samples_split=min_samples_split, \n        min_samples_leaf=min_samples_leaf\n    )\n    \n    # Train the classifier and calculate the accuracy on the validation set\n    clf.fit(X_train, y_train)\n    score = clf.score(X_test, y_test)\n    \n    return 1.0 - score\n\n# Use Optuna to tune the hyperparameters\nstudy = optuna.create_study()\nstudy.optimize(objective, n_trials=100)\n\n# Print the best hyperparameters and the best score\nprint(\"Best hyperparameters: \", study.best_params)\nprint(\"Best score: \", 1.0 - study.best_value)\n\n# Train the classifier with the best hyperparameters on the full training set\nbest_params = study.best_params\nclf = RandomForestClassifier(\n    n_estimators=best_params[\"n_estimators\"], \n    max_depth=best_params[\"max_depth\"],\n    min_samples_split=best_params[\"min_samples_split\"], \n    min_samples_leaf=best_params[\"min_samples_leaf\"]\n)\nclf.fit(X, y)\n\n# Evaluate the tuned classifier on the test set\nscore = clf.score(X_test, y_test)\nprint(\"Test set accuracy: \", score)","metadata":{"execution":{"iopub.status.busy":"2023-02-03T19:37:21.266850Z","iopub.execute_input":"2023-02-03T19:37:21.267261Z","iopub.status.idle":"2023-02-03T19:38:32.441480Z","shell.execute_reply.started":"2023-02-03T19:37:21.267226Z","shell.execute_reply":"2023-02-03T19:38:32.440203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Random Search\n\n>- Hyperparameter values are selected by independent (random) draws from uniform distribution of the hyperparameter space. Random Search selects the combinations of hyperparameter values at random from all the possible combinations given a hyperparameter space.\n\n---\n\n## Random Search vs Grid Search\n>- Some parameters affect performance a lot and some others don't (Low Effective Dimension). \n\n|      Random Search                                                |   Grid Search                            |\n|:------------------------------------------------------------------|-----------------------------------------:|\n| Allows the exploration of more dimensions of important parameters | Waste time exploring non-important dimensions |\n| Select values from a distribution of parameter values             | Parameters are defined manually |\n| Good for continuous parameters                                    | Good for discrete parameters |\n\n---\n\n## Considerations\n>- We choose a (computational) budget independently of the number of parameters and possible values.\n>- Adding parameters that do not influence the performance does not decrease efficiency of the search (if enough iterations are allowed).\n>- Important to specify a continuous distribution of the hyperparameter to take full advantage of the randomization.","metadata":{}}]}