Reading

In the field of natural language processing, unsupervised pre-training is key to improving downstream task performance. For a long time, autoregressive (AR) language models and denoising autoencoder-based models (such as BERT) have been the two most successful approaches. However, each has distinct limitations. Traditional AR models can only encode context unidirectionally, making it difficult to capture deep bidirectional information; while BERT leverages bidirectional context, its reliance on mask tokens leads to a data distribution inconsistency between the pre-training and fine-tuning stages (the "pre-training/fine-tuning discrepancy"), and it assumes that predicted words are independent of one another, ignoring the long-range dependencies prevalent in natural language.

To address these pain points, the Carnegie Mellon University and Google AI teams proposed XLNet. This is a generalized autoregressive pre-training method designed to combine the strengths of AR models and BERT while avoiding their defects. XLNet's core innovation lies in "permutation language modeling." Unlike traditional AR models with fixed forward or backward decomposition orders, XLNet maximizes the expected log-likelihood across all possible permutation orders. This means that for every position in the sequence, the context can include words from both the left and right sides. Statistically, the model learns to utilize information from all positions, thereby achieving bidirectional context capture without compromising the autoregressive nature.

Furthermore, because XLNet maintains an autoregressive form, it does not rely on input data corruption (such as BERT's [MASK] operation), thus eliminating the discrepancy between pre-training and fine-tuning. Simultaneously, the autoregressive objective allows the use of the product rule to decompose joint probabilities, naturally modeling dependencies between words and overcoming the simplification problem of BERT's independence assumption. To further enhance performance, XLNet integrates the segment-recurrence mechanism and relative positional encoding from Transformer-XL, enabling better performance when processing long text sequences. Experiments show that under comparable settings, XLNet outperforms BERT in 20 tasks including question answering, natural language inference, sentiment analysis, and document ranking, often with significant advantages.

Recommended reading: "XLNet: Generalized Autoregressive Pretraining for Language Understanding" (2019). This paper provides an in-depth analysis of the pros and cons of BERT and traditional autoregressive models, and demonstrates how clever design through permutation language modeling enables bidirectional context learning while maintaining autoregressive characteristics. For readers hoping to understand the evolution of large-scale language model pre-training mechanisms, particularly how to balance bidirectionality and autoregressive advantages, this is a must-read classic technical literature.

In The Black Swan, published in 2007, Nassim Nicholas Taleb presents a core framework regarding uncertainty. The book defines the "Black Swan" as an event that is extremely unlikely to occur, characterized by three main features: it is unpredictable; it carries a massive impact; and after the fact, we construct explanations that make it appear less random and more predictable than it actually was. This concept challenges the traditional reliance on "known unknowns" in risk management, emphasizing extreme events that are not included in models but are sufficient to overturn the world.

Taleb points out that humans naturally tend to focus on specific details rather than general patterns, which leads to our inability to truly assess opportunities. We fall into the impulse to simplify, narrate, and categorize, and we are often intolerant of those who can imagine "impossible" things. These cognitive biases cause us to overlook the existence of Black Swan events before they happen, and then rationalize the outcomes through narrative fallacy afterward, thereby underestimating the role of randomness. The author argues that many failures on Wall Street stem from people overestimating their explanatory abilities and skills, while failing to realize that they do not understand the true mechanisms by which the world operates.

Consider an investor who constructs a portfolio based on historical data that appears highly stable, believing that market volatility follows a normal distribution. When an emerging technology company suddenly sees its market value soar due to an unexpected technological breakthrough, triggering a reconstruction of valuation logic across the entire industry, this investor suffers significant relative losses or misses out on huge gains because such "unpredictable" high-impact events were not included in their risk considerations. This is precisely the scenario warned against in The Black Swan: over-reliance on past certainty makes one vulnerable to future extreme uncertainty.

For readers seeking to deeply understand the nature of uncertainty and avoid being misled by pseudo-experts, The Black Swan provides important thinking tools. It is recommended to start with the new chapter on "Robustness vs. Fragility," combining Taleb’s interdisciplinary analysis of probability theory and cognitive psychology to re-examine blind spots in personal decision-making and investment judgment. The book was published by Nassim Nicholas Taleb in 2007 and is suitable for readers interested in risk management and cognitive science.