Human-centric Explanations for Bayesian Optimization

Modern industrial systems face complex optimization challenges where decisions must balance multiple conflicting objectives. Traditional optimization methods struggle with these real-world "black-box" systems, which are expensive to evaluate and lack accessible analytical gradients. Bayesian Optimization (BO) offers a data-efficient solution by using a probabilistic surrogate model to guide the search for optimal solutions with limited evaluations. Despite its effectiveness, BO's opaque nature creates a critical barrier to its adoption in industrial settings. Experts need to understand what the optimization has learned, why certain solutions are recommended, and where to intervene. This need for transparency is driven by the fact that optimization is rarely a purely algorithmic task; it's a collaborative process where human expertise is essential for interpreting results, refining recommendations, and accounting for practical constraints that are hard to formalize upfront. Existing Explainable Artificial Intelligence (XAI) methods are often ill-suited to BO's unique challenges, such as sparse and sequentially generated data, and the need to support an ongoing, iterative search rather than explaining a static model. The goal of this thesis is to develop and evaluate explanation methods for BO that enhance human–algorithm collaboration by improving transparency, supporting expert decision-making, and fulfilling key explainability requirements. We develop new explanation methods specifically designed to support human-algorithm collaboration in BO. We categorize this collaboration into two paradigms: post-hoc and human-in-the-loop. For the post-hoc setting, we introduce RXBO (Rule based explanations for Bayesian optimization) and TNTRules (Tune-No-Tune rules), which provide interpretable, actionable rule sets and graphs that explain BO recommendations. These methods highlight relationships between inputs and outputs, communicate uncertainty, and guide parameter refinement. For the human-in-the-loop setting, where humans provide preference feedback, we introduce MOLONE (Multi-Output LOcal Narrative Explanation), a comparative explanation framework. MOLONE uses local, dual-sided explanations to clarify why one candidate solution is preferred over another, revealing both input-level feature importance and outcome-level trade-off to help users make informed decisions. Our research demonstrates that these explanation methods significantly enhance human-algorithm collaboration. Further we conducted a large-scale user study which revealed that users with explanations consistently achieved better task performance, required fewer trials, and reported higher levels of understanding and trust in the system. Importantly, these benefits were achieved without increasing cognitive load. The findings provide empirical validation that explanations are crucial for effective human-BO collaboration, proving that interpretability is not a luxury but a necessity for building trustworthy and efficient decision-making workflows in complex industrial applications.