{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# No BS Guide to Hyperparameter Tuning With Optuna\n## Everyone is obsessed with it these days, let's find out why\n\n\n \n Photo by \n Bo Mei\n on \n Pixabay. All images are by author unless specified otherwise.\n \n","metadata":{}},{"cell_type":"markdown","source":"## Setup","metadata":{"_kg_hide-input":true}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\nimport optuna\nimport pandas as pd\nimport seaborn as sns\n\noptuna.logging.set_verbosity(optuna.logging.WARNING)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:19.829980Z","iopub.execute_input":"2021-07-30T15:59:19.830640Z","iopub.status.idle":"2021-07-30T15:59:22.022589Z","shell.execute_reply.started":"2021-07-30T15:59:19.830534Z","shell.execute_reply":"2021-07-30T15:59:22.021699Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Introduction","metadata":{}},{"cell_type":"markdown","source":"Turns out I have been living under a rock.\n\nWhile every single MOOC taught me to use GridSearch for hyperparameter tuning, Kagglers have been using Optuna almost exclusively for almost 2 years. This even predates the time I started learning data science.\n\nKaggle community is known for its brutal competitiveness, and for a package to achieve this level of domination, it needs to be damn good. After being active on the platform for the last month (and achieving [expert status](https://medium.com/r/?url=https%3A%2F%2Fwww.kaggle.com%2Fbextuychiev) in two tiers), I saw Optuna used almost everywhere and by everyone.\n\nSo, what makes Optuna so widely received by the largest machine learning community out there? We will answer this question in this kernel by getting hands-on on the framework. We will learn how it works and how it squeezes every bit of performance out of any model, including neural networks.","metadata":{}},{"cell_type":"markdown","source":"# What is Optuna?","metadata":{}},{"cell_type":"markdown","source":"\n\n \n Optuna logo\n \n","metadata":{}},{"cell_type":"markdown","source":"Optuna is a next-generation automatic hyperparameter tuning framework written completely in Python.\n\nIts most prominent features are:\n- the ability to define Pythonic search spaces using loops and conditionals.\n- Platform-agnostic API - you can tune estimators of almost any ML, DL package/framework, including Sklearn, PyTorch, TensorFlow, Keras, XGBoost, LightGBM, CatBoost, etc.\n- a large suite of optimization algorithms with early stopping and pruning features baked in.\n- Easy parallelization with little or no changes to the code.\n- Built-in support for visual exploration of search results.\n\nWe will try to validate these overly optimistic claims made in [Optuna's documentation](https://optuna.readthedocs.io/en/stable/index.html) in the coming sections.","metadata":{}},{"cell_type":"markdown","source":"# Optuna basics","metadata":{}},{"cell_type":"markdown","source":"Let's familiarize ourselves with Optuna API by tuning a simple function like $(x-1)^2 + (y+3)^2$. We know the function reaches its minimum at x=1 and y=-3. Let's see if Optuna can find these:","metadata":{}},{"cell_type":"code","source":"import optuna # pip install optuna\n\n\ndef objective(trial):\n x = trial.suggest_float(\"x\", -7, 7)\n y = trial.suggest_float(\"y\", -7, 7)\n return (x - 1) ** 2 + (y + 3) ** 2","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:22.024407Z","iopub.execute_input":"2021-07-30T15:59:22.024689Z","iopub.status.idle":"2021-07-30T15:59:22.031635Z","shell.execute_reply.started":"2021-07-30T15:59:22.024663Z","shell.execute_reply":"2021-07-30T15:59:22.030474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After importing `optuna`, we define an objective that returns the function we want to minimize.\n\nIn the body of the objective, we define the parameters to be optimized, in this case, simple `x` and `y`. The argument `trial` is a special `Trial` object of optuna, which does the optimization for each hyperparameter.\n\nAmong others, it has a `suggest_float` method that takes the name of the hyperparameter and the range to look for its optimal value. In other words,\n\n```\nx = trial.suggest_float(\"x\", -7, 7)\n```\nis almost the same as `{\"x\": np.arange(-7, 7)}` when doing GridSearch.\n\nTo start the optimization, we create a `study` object from Optuna and pass the `objective` function to its `optimize` method:","metadata":{}},{"cell_type":"code","source":"study = optuna.create_study()\nstudy.optimize(objective, n_trials=100) # number of iterations","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:22.034325Z","iopub.execute_input":"2021-07-30T15:59:22.034835Z","iopub.status.idle":"2021-07-30T15:59:22.773826Z","shell.execute_reply.started":"2021-07-30T15:59:22.034771Z","shell.execute_reply":"2021-07-30T15:59:22.773072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study.best_params","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:22.775104Z","iopub.execute_input":"2021-07-30T15:59:22.775551Z","iopub.status.idle":"2021-07-30T15:59:22.783245Z","shell.execute_reply.started":"2021-07-30T15:59:22.775520Z","shell.execute_reply":"2021-07-30T15:59:22.782228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Pretty close, but not as close as you would want. Here, we only did 100 trials, as can be seen with:","metadata":{}},{"cell_type":"code","source":"len(study.trials)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:22.784442Z","iopub.execute_input":"2021-07-30T15:59:22.784743Z","iopub.status.idle":"2021-07-30T15:59:22.811900Z","shell.execute_reply.started":"2021-07-30T15:59:22.784708Z","shell.execute_reply":"2021-07-30T15:59:22.810978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, I will introduce the first magic that comes with Optuna. We can resume the optimization even after it is finished if we are not satisfied with the results!\n\nThis is a **distinct advantage** over other similar tools because after the search is done, they completely forget the history of previous trials. Optuna does not!\n\nTo continue searching, call `optimize` again with the desired params. Here, we will run 100 more trials:","metadata":{}},{"cell_type":"code","source":"study.optimize(objective, n_trials=100)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:22.813084Z","iopub.execute_input":"2021-07-30T15:59:22.813577Z","iopub.status.idle":"2021-07-30T15:59:23.739200Z","shell.execute_reply.started":"2021-07-30T15:59:22.813547Z","shell.execute_reply":"2021-07-30T15:59:23.738496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study.best_params","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:23.740364Z","iopub.execute_input":"2021-07-30T15:59:23.740869Z","iopub.status.idle":"2021-07-30T15:59:23.746315Z","shell.execute_reply.started":"2021-07-30T15:59:23.740836Z","shell.execute_reply":"2021-07-30T15:59:23.745277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This time, the results are much closer to the optimal parameters.","metadata":{}},{"cell_type":"markdown","source":"# A note on Optuna terminology and conventions","metadata":{}},{"cell_type":"markdown","source":"In Optuna, the whole optimization process is called a study. For example, tuning XGBoost parameters with a log loss as a metric is one study:","metadata":{}},{"cell_type":"code","source":"study = optuna.create_study()\ntype(study)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:23.749262Z","iopub.execute_input":"2021-07-30T15:59:23.749643Z","iopub.status.idle":"2021-07-30T15:59:23.766140Z","shell.execute_reply.started":"2021-07-30T15:59:23.749604Z","shell.execute_reply":"2021-07-30T15:59:23.764817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A study needs a function it can optimize. Typically, this function is defined by the user, and by convention, it should be named `objective`.\n\nThe objective function is expected to have this signature:","metadata":{}},{"cell_type":"code","source":"def objective(trial: optuna.Trial):\n \"\"\"Conventional optimization function\n signature for optuna.\n \"\"\"\n custom_metric = ...\n return custom_metric","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:23.768244Z","iopub.execute_input":"2021-07-30T15:59:23.768580Z","iopub.status.idle":"2021-07-30T15:59:23.777985Z","shell.execute_reply.started":"2021-07-30T15:59:23.768549Z","shell.execute_reply":"2021-07-30T15:59:23.776768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It should accept an `optuna.Trial` object as a parameter and return the metric we want to optimize for.\n\nAs we saw in the first example, a study is a collection of trials wherein each trial, we evaluate the objective function using a single set of hyperparameters from the given search space.\n\nEach trial in the study is represented as `optuna.Trial` class. This class is key to how Optuna finds optimal values for parameters.\n\nTo start a study, we create a study object with `direction`:","metadata":{}},{"cell_type":"code","source":"study = optuna.create_study(direction=\"maximize\")","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:23.779500Z","iopub.execute_input":"2021-07-30T15:59:23.779845Z","iopub.status.idle":"2021-07-30T15:59:23.793547Z","shell.execute_reply.started":"2021-07-30T15:59:23.779805Z","shell.execute_reply":"2021-07-30T15:59:23.792104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If the metric we want to optimize is a point-performance score like ROC AUC or accuracy, we set the direction to `maximize`. Otherwise, we minimize a loss function like RMSE, RMSLE, log loss, etc. by setting direction to `minimize`.\n\nThen, we will call the optimize method of the study passing the objective function name and the number of trials we want:\n\n```python\n# Optimization with 100 trials\nstudy.optimize(objective, n_trials=100)\n```\n\nNext, we will take a closer look into creating these objective functions.","metadata":{}},{"cell_type":"markdown","source":"# Defining the search space","metadata":{}},{"cell_type":"markdown","source":"Usually, the first thing you do in an objective function is to create the search space using built-in Optuna methods:","metadata":{}},{"cell_type":"code","source":"def objective(trial):\n rf_params = {\n \"n_estimators\": trial.suggest_integer(name=\"n_estimators\", low=100, high=2000),\n \"max_depth\": trial.suggest_float(\"max_depth\", 3, 8),\n \"max_features\": trial.suggest_categorical(\n \"max_features\", choices=[\"auto\", \"sqrt\", \"log2\"]\n ),\n \"n_jobs\": -1,\n \"random_state\": 1121218,\n }\n\n rf = RandomForestRegressor(**rf_params)\n ...","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:23.795501Z","iopub.execute_input":"2021-07-30T15:59:23.795889Z","iopub.status.idle":"2021-07-30T15:59:23.806508Z","shell.execute_reply.started":"2021-07-30T15:59:23.795854Z","shell.execute_reply":"2021-07-30T15:59:23.805266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the above objective function, we are creating a small search space of Random Forest hyperparameters.\n\nThe search space is a plain-old dictionary. To create possible values to search over, you must use the trial object's `suggest_*` functions.\n\nThese functions require at least the hyperparameter name, min, and max of the range to search over or possible categories for categorical hyperparameters.\n\nTo make the space smaller, `suggest_float` and `suggest_int` have additional `step` or `log` arguments:","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import GradientBoostingRegressor\n\n\ndef objective(trial):\n params = {\n \"n_estimators\": trial.suggest_int(\"n_estimators\", 1000, 10000, step=200),\n \"learning_rate\": trial.suggest_float(\"learning_rate\", 1e-7, 0.3, log=True),\n \"max_depth\": trial.suggest_int(\"max_depth\", 3, 12, step=2),\n \"random_state\": 1121218,\n }\n boost_reg = GradientBoostingRegressor(**params)\n rmsle = ...\n return rmsle","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:23.807715Z","iopub.execute_input":"2021-07-30T15:59:23.808123Z","iopub.status.idle":"2021-07-30T15:59:23.822321Z","shell.execute_reply.started":"2021-07-30T15:59:23.808088Z","shell.execute_reply":"2021-07-30T15:59:23.821328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above, we are binning the distribution of `n_estimators` by 200-intervals to make it sparser. Also, `learning_rate` is defined at a logarithmic scale.","metadata":{}},{"cell_type":"markdown","source":"# How are possible parameters sampled?","metadata":{}},{"cell_type":"markdown","source":"Under the hood, Optuna has several classes responsible for parameter sampling. These are:\n- `GridSampler`: the same as `GridSearch` of Sklearn. Never use for large search spaces!\n- `RandomSampler`: the same as `RandomizedGridSearch` of Sklearn.\n- `TPESampler`: Tree-structured Parzen Estimator sampler - bayesian optimization using kernel fitting\n- `CmaEsSampler`: a sampler based on CMA ES algorithm (does not allow categorical hyperparameters).\n\n> I have no idea of how the last two samplers work and I don't expect this to affect any interaction I have with Optuna.\n\nTPE Sampler is used by default - it tries to sample hyperparameter candidates by improving on the last trial's scores. In other words, you can expect incremental (maybe marginal) improvements from trial to trial with this sampler.\n\nIf you ever want to switch samplers, this is how you do it:","metadata":{}},{"cell_type":"code","source":"from optuna.samplers import CmaEsSampler, RandomSampler\n\n# Study with a random sampler\nstudy = optuna.create_study(sampler=RandomSampler(seed=1121218))\n\n# Study with a CMA ES sampler\nstudy = optuna.create_study(sampler=CmaEsSampler(seed=1121218))","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:23.823583Z","iopub.execute_input":"2021-07-30T15:59:23.824094Z","iopub.status.idle":"2021-07-30T15:59:23.837918Z","shell.execute_reply.started":"2021-07-30T15:59:23.824057Z","shell.execute_reply":"2021-07-30T15:59:23.836990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# End-to-end example with GradientBoostingRegressor","metadata":{}},{"cell_type":"markdown","source":"Let's put everything we have learned into something tangible. We will be predicting penguin body weights using several numeric and categorical features.\n\nWe will establish a base score with Sklearn `GradientBoostingRegressor` and improve it by tuning with Optuna:","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\nfrom sklearn.ensemble import GradientBoostingRegressor\nfrom sklearn.model_selection import KFold, cross_validate, train_test_split\n\n# Load data\npenguins = sns.load_dataset(\"penguins\").dropna()\nX, y = penguins.drop(\"body_mass_g\", axis=1), penguins[[\"body_mass_g\"]]\n\n# OH encode categoricals\nX = pd.get_dummies(X)\n\n# Init model with defaults\ngr_reg = GradientBoostingRegressor(random_state=1121218)\n\nkf = KFold(n_splits=5, shuffle=True, random_state=1121218)\nscores = cross_validate(\n gr_reg, X, y, cv=kf, scoring=\"neg_mean_squared_log_error\", n_jobs=-1\n)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:23.839119Z","iopub.execute_input":"2021-07-30T15:59:23.839585Z","iopub.status.idle":"2021-07-30T15:59:27.328877Z","shell.execute_reply.started":"2021-07-30T15:59:23.839551Z","shell.execute_reply":"2021-07-30T15:59:27.327550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rmsle = np.sqrt(-scores[\"test_score\"].mean())\nprint(f\"Base RMSLE: {rmsle:.5f}\")","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:27.334277Z","iopub.execute_input":"2021-07-30T15:59:27.334701Z","iopub.status.idle":"2021-07-30T15:59:27.342112Z","shell.execute_reply.started":"2021-07-30T15:59:27.334662Z","shell.execute_reply":"2021-07-30T15:59:27.340924Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we will create the `objective` function and define the search space:","metadata":{}},{"cell_type":"code","source":"def objective(trial, X, y, cv, scoring):\n params = {\n \"n_estimators\": trial.suggest_int(\"n_estimators\", 100, 5000, step=100),\n \"learning_rate\": trial.suggest_float(\"learning_rate\", 1e-4, 0.3, log=True),\n \"max_depth\": trial.suggest_int(\"max_depth\", 3, 9),\n \"subsample\": trial.suggest_float(\"subsample\", 0.5, 0.9, step=0.1),\n \"max_features\": trial.suggest_categorical(\n \"max_features\", [\"auto\", \"sqrt\", \"log2\"]\n ),\n \"random_state\": 1121218,\n \"n_iter_no_change\": 50, # early stopping\n \"validation_fraction\": 0.05,\n }\n # Perform CV\n gr_reg = GradientBoostingRegressor(**params)\n scores = cross_validate(gr_reg, X, y, cv=cv, scoring=scoring, n_jobs=-1)\n # Compute RMSLE\n rmsle = np.sqrt(-scores[\"test_score\"].mean())\n\n return rmsle","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:27.344208Z","iopub.execute_input":"2021-07-30T15:59:27.344948Z","iopub.status.idle":"2021-07-30T15:59:27.355944Z","shell.execute_reply.started":"2021-07-30T15:59:27.344897Z","shell.execute_reply":"2021-07-30T15:59:27.355015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We built a grid of 5 hyperparameters with different ranges and some static ones for random seed and early stopping.\n\nThe above objective function is slightly different - it accepts additional arguments for the data sets, scoring and `cv`. That's why we have to wrap it inside another function. Generally, you do this with a `lambda` function like below:\n\n> This is the recommended syntax if you want to pass `objective` functions that accept multiple parameters.","metadata":{}},{"cell_type":"code","source":"%%time\n\n# Create study that minimizes\nstudy = optuna.create_study(direction=\"minimize\")\n\n# Wrap the objective inside a lambda with the relevant arguments\nkf = KFold(n_splits=5, shuffle=True, random_state=1121218)\n# Pass additional arguments inside another function\nfunc = lambda trial: objective(trial, X, y, cv=kf, scoring=\"neg_mean_squared_log_error\")\n\n# Start optimizing with 100 trials\nstudy.optimize(func, n_trials=100)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T15:59:27.357769Z","iopub.execute_input":"2021-07-30T15:59:27.358590Z","iopub.status.idle":"2021-07-30T16:01:42.639768Z","shell.execute_reply.started":"2021-07-30T15:59:27.358540Z","shell.execute_reply":"2021-07-30T16:01:42.638545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Base RMSLE : {rmsle:.5f}\")\nprint(f\"Optimized RMSLE: {study.best_value:.5f}\")","metadata":{"execution":{"iopub.status.busy":"2021-07-30T16:01:42.641259Z","iopub.execute_input":"2021-07-30T16:01:42.641709Z","iopub.status.idle":"2021-07-30T16:01:42.648677Z","shell.execute_reply.started":"2021-07-30T16:01:42.641671Z","shell.execute_reply":"2021-07-30T16:01:42.647316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In just under a minute, we achieved a significant score boost (in terms of log errors, 0.004 is pretty sweet). We did this with only 100 trials. Let's boldly run another 200 and see what happens:","metadata":{}},{"cell_type":"markdown","source":"> The score was higher on my local machine. Forgot to seed the `study`, rookie mistake.","metadata":{}},{"cell_type":"code","source":"%%time\n\nstudy.optimize(func, n_trials=200)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T16:01:42.650588Z","iopub.execute_input":"2021-07-30T16:01:42.651062Z","iopub.status.idle":"2021-07-30T16:04:30.017952Z","shell.execute_reply.started":"2021-07-30T16:01:42.651015Z","shell.execute_reply":"2021-07-30T16:04:30.016565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Best params:\")\nfor key, value in study.best_params.items():\n print(f\"\\t{key}: {value}\")","metadata":{"execution":{"iopub.status.busy":"2021-07-30T16:04:30.019805Z","iopub.execute_input":"2021-07-30T16:04:30.020230Z","iopub.status.idle":"2021-07-30T16:04:30.034198Z","shell.execute_reply.started":"2021-07-30T16:04:30.020188Z","shell.execute_reply":"2021-07-30T16:04:30.032628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Base RMSLE : {rmsle:.5f}\")\nprint(f\"Optimized RMSLE: {study.best_value:.5f}\")","metadata":{"execution":{"iopub.status.busy":"2021-07-30T16:04:30.036453Z","iopub.execute_input":"2021-07-30T16:04:30.036929Z","iopub.status.idle":"2021-07-30T16:04:30.049405Z","shell.execute_reply.started":"2021-07-30T16:04:30.036880Z","shell.execute_reply":"2021-07-30T16:04:30.048264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> All these didn't take that much long on my local machine. Sorry for people running this notebook...","metadata":{}},{"cell_type":"markdown","source":"The score *did* improve but marginally. It looks like we hit it close to the max in the first run!\n\nMost importantly, we achieved this score in just over 2 minutes using a search space that would probably take hours with regular GridSearch.\n\nI don't know about you, but I am sold!","metadata":{}},{"cell_type":"markdown","source":"# Using visuals for more insights and smarter tuning","metadata":{}},{"cell_type":"markdown","source":"Optuna offers a wide range of plots under its `visualization` subpackage. Here, we will discuss only 2, which I think are the most useful.\n\nFirst, let's plot the optimization history of the last `study`:","metadata":{}},{"cell_type":"code","source":"from optuna.visualization import plot_optimization_history\n\nplotly_config = {\"staticPlot\": True}\n\nfig = plot_optimization_history(study)\nfig.show(config=plotly_config)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T16:04:30.051719Z","iopub.execute_input":"2021-07-30T16:04:30.052142Z","iopub.status.idle":"2021-07-30T16:04:30.333704Z","shell.execute_reply.started":"2021-07-30T16:04:30.052101Z","shell.execute_reply":"2021-07-30T16:04:30.332736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This plot tells us that Optuna made the score converge to the minimum after only a few trials.\n\nNext, let's plot hyperparameter importances:","metadata":{}},{"cell_type":"code","source":"from optuna.visualization import plot_param_importances\n\nfig = plot_param_importances(study)\nfig.show(config=plotly_config)","metadata":{"execution":{"iopub.status.busy":"2021-07-30T16:04:30.336456Z","iopub.execute_input":"2021-07-30T16:04:30.336806Z","iopub.status.idle":"2021-07-30T16:04:45.883340Z","shell.execute_reply.started":"2021-07-30T16:04:30.336755Z","shell.execute_reply":"2021-07-30T16:04:45.882366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This plot is massively useful! It tells us several things, including:\n- `max_depth` and `learning_rate` are the most important\n- `subsample` and `max_features` are useless for minimizing the loss\n\nA plot like this comes in handy when tuning models with many hyperparameters. For example, you could take a test run of 40–50 trials and plot the parameter importances.\n\nDepending on the plot, you might decide to discard some less important parameters and give a larger search space for other ones, possibly reducing the search time and space.\n\nYou can check out [this page](https://optuna.readthedocs.io/en/stable/reference/visualization/index.html) of the documentation for more information on Optuna's supported plot types. ","metadata":{}},{"cell_type":"markdown","source":"# Summary","metadata":{}},{"cell_type":"markdown","source":"I think we can all agree that Optuna lived up to the whole hype I made in the introduction. It is awesome!\n\nThis kernel only gave you the basics you can do with Optuna. Actually, Optuna is capable of much more. Some of the critical topics we didn't cover today:\n- [Use cases of Optuna with other ML/DL frameworks](https://github.com/optuna/optuna-examples/)\n- [Choosing a pruning algorithm to immediately weed out unpromising trials](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/003_efficient_optimization_algorithms.html#activating-pruners)\n- [Parallelization](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/004_distributed.html)\n\nand the coolest of all:\n- [Using SQLite or other databases (local or remote) to run massive-scale optimization with resume/pause capabilities](https://optuna.readthedocs.io/en/stable/tutorial/20_recipes/001_rdb.html#sphx-glr-tutorial-20-recipes-001-rdb-py)\n\nDo check out the links to the relevant documentation pages. In the meantime, I will work on another kernel that shows how to use Optuna with XGBoost and choose a pruning algorithm. See you!","metadata":{}},{"cell_type":"markdown","source":"## You might also be interested...\n- [Automatic Hyperparameter Tuning with Sklearn GridSearchCV and RandomizedSearchCV](https://towardsdatascience.com/automatic-hyperparameter-tuning-with-sklearn-gridsearchcv-and-randomizedsearchcv-e94f53a518ee?source=your_stories_page-------------------------------------)\n- [11 Times Faster Hyperparameter Tuning with HalvingGridSearch](https://towardsdatascience.com/11-times-faster-hyperparameter-tuning-with-halvinggridsearch-232ed0160155?source=your_stories_page-------------------------------------)\n- [20 Burning XGBoost FAQs Answered to Use the Library Like a Pro](https://towardsdatascience.com/20-burning-xgboost-faqs-answered-to-use-the-library-like-a-pro-f8013b8df3e4?source=your_stories_page-------------------------------------)","metadata":{}}]}