A company has a limited cloud-computing budget and needs to tune a machine learning model with 10 hyperparameters.

Here is a comparison of the three hyperparameter optimization (HPO) methods given your constraints (10 hyperparameters, 3 hours per evaluation, strict cloud budget), followed by a recommendation.

---

Comparison of HPO Methods

---

Method Breakdown

1. Random Search

   • Pros: Zero modeling overhead, dead simple to set up, easily parallelized across whatever compute instances you have available.
   • Cons: It does not learn from past evaluations. When each run takes 3 hours and your budget is tight, you can only afford a small number of trials—meaning random search has a high chance of leaving performance on the table in a 10-dimensional search space.

2. Bayesian Optimization

   • Pros: Builds a probabilistic surrogate model (such as a Gaussian Process or Tree-structured Parzen Estimator) to predict which regions of the parameter space are most promising. It explicitly balances exploration and exploitation, making it the industry standard for expensive, black-box evaluations.
   • Cons: Typically sequential (each run guides the next), though modern libraries support asynchronous/batched evaluation.

3. Gradient-Based HPO (e.g., implicit differentiation or unrolled reverse-mode autodiff)

   • Pros: Extremely fast convergence for continuous hyperparameters (like learning rates and weight decays).
   • Cons: Not feasible for discrete or structural hyperparameters (such as batch size, number of layers, kernel size, or optimizer choice). It also demands substantial GPU memory for unrolled training steps and significant engineering effort to implement.

---

Recommendation: Bayesian Optimization

Why this is the best choice:

1. Maximum value per compute dollar (Sample Efficiency): With each trial costing 3 hours of compute, your biggest bottleneck is the evaluation cost. Bayesian Optimization is specifically designed for expensive objective functions—it extracts maximum information from each completed run to pick the next candidate intelligently.
2. Right dimensionality: A 10-dimensional space is in the sweet spot for Bayesian Optimization (especially using TPE / Tree-structured Parzen Estimators, as implemented in tools like Optuna). It is rich enough to overwhelm random search, but compact enough that surrogate models remain fast and reliable.
3. Hyperparameter flexibility: Unlike gradient-based methods, BO easily handles mixed parameter types (categorical, integer, log-uniform continuous) without requiring a custom, differentiable training loop.

Practical Tip to Save More Budget:

Pair Bayesian Optimization with early-stopping pruners (like Median Pruner or Hyperband/ASHA). Instead of letting every bad configuration run for the full 3 hours, the framework can kill unpromising runs after the first few epochs, allowing you to evaluate 2–4× more configurations within the exact same budget.