Technical Author Grootendorst Breaks Down Two Architectural Paths for Multimodal LLM Generation
Technical author Maarten Grootendorst explains two architectural paths for multimodal LLMs to move from merely understanding images, audio, and video to actually generating them: Late Fusion, which connects external generators via projection layers and requires training only millions of parameters; and Early Fusion, which embeds generation capabilities directly within the LLM itself, incurring higher training costs but yielding more accurate outputs. Using Stable Diffusion, AudioLM, NExT-GPT, and DeepSeek Janus as examples, the article dissects how components collaborate to perform iterative denoising from noise to image, quantization from continuous vectors to discrete tokens, and the respective conditions under which each path is applicable.
Most multimodal LLMs today are multimodal in only one direction: they can understand images, audio, and video as input, but their output remains text-only. Enabling an LLM to truly generate these modalities requires a completely different set of architectures. Technical author Maarten Grootendorst breaks this problem down into two clear paths in this illustrated guide—Late Fusion (output projection) and Early Fusion (unified generation)—and uses specific models to illustrate their mechanisms and trade-offs.
First, let's understand generation itself. To connect a generator to an LLM, we must first know how non-text modalities are generated. The core of image generation lies in diffusion models: starting from pure noise, the model iteratively predicts and removes noise until it reconstructs an image that matches the text description. Stable Diffusion works by first compressing the image into a latent space using a Variational Autoencoder (VAE), running iterative denoising on this low-dimensional tensor with a U-Net, and finally decoding it back into pixels. Text conditioning is injected through cross-attention layers of a CLIP encoder, telling the model which structures to preserve and which to treat as noise to be removed.
Audio generation follows a similar logic but with a different representation method. AudioLM uses SoundStream, a neural codec, to compress audio into discrete tokens: it first maps continuous audio to a fixed codebook using a variant of VQ-VAE called Residual Vector Quantization (RVQ) to obtain acoustic tokens; then it extracts semantic tokens using w2v-BERT to capture spoken content and intent. These two layers of tokens are generated sequentially by chained Transformers and finally decoded back into waveforms by the SoundStream decoder. The key constraint here is that Transformers require discrete representations, so continuous vectors must be quantized before entering the generation pipeline.
Video generation builds upon the image diffusion framework but adds a temporal dimension. The Latte model explored two tokenization methods: per-frame spatial patches and cross-frame tubelet compression. The authors found that the latter performed worse, speculating that compression loses spatio-temporal information. The best-performing architecture ultimately interleaved spatial and temporal attention blocks, evaluated using Fréchet Video Distance.
With these generation mechanisms in mind, let's look at the two connection methods. The core of Late Fusion is projection: the LLM outputs embeddings of text tokens, which are projected into a vector space understandable by an external generator via a trainable MLP or small Transformer. NExT-GPT is a typical example—it assigns special tokens for each modality (e.g., [IMG], [AUD], [VID]). After the LLM generates these tokens, downstream pre-trained diffusion models conditionally generate content based on the token semantics. Throughout this process, only the projection layer needs training; the base models remain frozen, reducing the number of trainable parameters from billions to millions. OmniGen2 and BLIP-3o also follow this route, decoupling autoregressive Transformers for text generation from diffusion models for image generation.
Early Fusion, on the other hand, embeds generation capabilities directly into the LLM architecture. DeepSeek's Janus uses two separate encoders to serve understanding and generation respectively: SigLIP provides high-level semantic representations for understanding, while VQ-GAN provides discrete image tokens for generation. The LLM outputs both text tokens and image tokens simultaneously, which are then decoded back into their respective modalities. Training occurs in three stages: first, training only the adapters and image head to establish conceptual connections; second, unified pre-training to teach the model both understanding and generation simultaneously; and finally, instruction fine-tuning.
The trade-offs between the two paths are clear. Late Fusion has lower training costs and allows reusing powerful independent generators, but the LLM itself is not a multimodal entity; it acts more like an agent calling tools. Early Fusion unifies all capabilities within a single architecture. It requires updating most parameters during training, making it significantly more expensive, but because all modalities are jointly optimized in the same representation space, the outputs are more accurate. Grootendorst concludes that as native multimodal LLMs become more common, agentic systems will gain stronger multimodal environmental control capabilities—but this is an inference based on architectural trends, not a quantified performance conclusion.