Lectura

In machine learning practice, efficiently finding optimal solutions when facing expensive black-box function evaluations (such as hyperparameter tuning) is a core challenge. The article "Exploring Bayesian Optimization" published by Distill breaks down complex Bayesian optimization into understandable modules through the vivid analogy of "gold mining." The article points out that modern algorithms have numerous hyperparameters, and using them effectively requires selecting good values; Bayesian optimization is precisely the toolkit used to tune these parameters.

The article first distinguishes between two related but different problems: Active Learning and Bayesian Optimization. In the gold mining analogy, the goal of active learning is to accurately estimate the distribution of gold along the entire line, so it tends to choose points with the highest uncertainty for drilling to maximize information gain. In contrast, the goal of Bayesian optimization is simply to find the location with the maximum gold content. Spending expensive costs to precisely model low-value regions just to find the maximum would clearly be wasteful. Therefore, the core question of Bayesian optimization is: "Based on what we currently know, which point should be evaluated next?"

To solve this problem, Bayesian optimization relies on two key components: the Surrogate Model and the Acquisition Function. The surrogate model typically uses Gaussian Processes (GP) because they are flexible and provide uncertainty estimates. As new data points are added, the model updates its posterior distribution according to Bayes' rule. However, the model alone is insufficient to decide the next action; we need a heuristic rule to measure the expected value of evaluating a certain point, which is the acquisition function. It is responsible for balancing "exploration" (trying unknown regions) and "exploitation" (focusing on known high-value regions).

Taking Probability of Improvement (PI) as an example, this acquisition function selects the point with the highest probability of improvement over the current maximum. The epsilon parameter in the formula controls this balance: a smaller epsilon favors exploiting known high-potential regions, while a larger epsilon encourages exploring regions with high uncertainty. Through visual demonstrations, the article shows that when epsilon is too large, the algorithm may over-explore and fail to converge to the global maximum; when epsilon is moderate, it can quickly locate peaks within a few iterations. This reveals the essence of Bayesian optimization: there is no need to build a perfect function model, only to intelligently guide the search direction.

For readers, understanding this mechanism helps in choosing appropriate tuning strategies for real-world projects. If you care about maximizing model performance rather than fully understanding the loss surface, Bayesian optimization is more efficient than grid search or random search. It is particularly suitable for scenarios with extremely high evaluation costs and lower dimensions. Mastering the intuition behind acquisition functions can help engineers debug why an optimizer gets stuck in local optima or under-explores.

Recommended reading: "Exploring Bayesian Optimization" by Apoorv Agnihotri and Nipun Batra, published in 2020. This Distill article is suitable for machine learning engineers and researchers who wish to deeply understand the principles of hyperparameter tuning. It is suggested to start reading from the "Mining Gold!" section, focusing on the comparison between active learning and Bayesian optimization, as well as the subsequent intuitive explanations of acquisition functions (particularly PI and EI).

In the 2001 letter to shareholders, Warren Buffett reaffirmed that the key standard for measuring investment success is not absolute returns, but performance relative to market benchmarks. He cited the "Ground Rules" established when he founded his partnership fund in 1956, noting that "Whether we do a good job or a poor job is to be measured against the general experience in securities." This viewpoint emphasizes that long-term investors should focus on whether they consistently outperform major index benchmarks, rather than simply looking at fluctuations in book value.

Addressing the criticism that "relative performance doesn't put food on the table," Buffett refuted this through analogy. He argued that if holding the S&P 500 Index is expected to yield reasonable returns, then the advantage of slightly beating that index each year will accumulate significantly over time. He wrote: "can you regularly feast on investment returns that beat the averages, however variable the absolute numbers may be." This mindset places short-term volatility within the framework of long-term compounding, explaining why Berkshire Hathaway was considered relatively successful even if it incurred a loss in a given year (such as the 6.2% decline in per-share book value in 2001), provided it performed better than the concurrent decline of the S&P 500.

Beyond performance benchmarks, Buffett also emphasized the deep alignment of management and shareholder interests. He pledged not to take cash compensation, restricted stock, or option grants, ensuring his economic outcomes were perfectly aligned with those of shareholders. He stated directly in the letter: "We will not take cash compensation, restricted stock or option grants that would make our results superior to yours." This arrangement aims to eliminate agency risk, encouraging managers to think like owners and avoid damaging long-term value for the sake of short-term stock prices.

For new readers, understanding these principles helps establish correct investment expectations. If an investor faces a market downturn and their portfolio declines less than the index, this aligns with Buffett's definition of doing a "good job." Conversely, blindly pursuing absolute positive returns while ignoring benchmark comparisons may lead to a misassessment of managerial capability. This perspective based on relative performance is a crucial tool for distinguishing luck from skill.

The Berkshire Hathaway Shareholder Letter 2001 was published by Warren E. Buffett on February 28, 2002. It is suitable for investors who wish to deeply understand the logic of performance evaluation in value investing and corporate governance structures. It is recommended to start with the sections on "Ground Rules" and interest alignment, reading them alongside letters from subsequent years to observe how these principles are applied in different market environments.