GenAI Beyond LLMs: Diffusion models overview

LLMs generate sequences — words, tokens, code. Diffusion models do something else entirely: they learn to generate high-dimensional geometries that the human eye recognises as images. Diffusion models power some of the most impressive advances in image, video, and multimodal generation. If we only talk about text, we miss more than half of the revolution.

What exactly is an image?

An image is a grid of pixels, a two-dimensional array of values that together form something the human eye recognises as a picture. More precisely, it is a 3D tensor of numbers with the shape height × width × channels. The spatial dimensions (height and width) define the layout of the image, while the channels encode colour information.

Each pixel is not a single value. It is a small vector of numbers, typically corresponding to the RGB (red, green, blue) colour space. Instead of storing colour as one combined concept, each channel is represented as its own 2D grid of values:

• one grid for red intensities
• one grid for green intensities
• one grid for blue intensities

When stacked together, these channels form the full image tensor. In this sense, an image is simply structured numerical data arranged in space. Diffusion models operate directly on this structure, manipulating raw pixel values with no inherent notion of the objects or scenes they represent.

How do you generate an image?

Image generation can start from many different inputs: pure noise, another image, a latent vector, a text prompt, a structural sketch, or even audio. The starting point shapes the entire architecture around it.

Diffusion models start from pure noise. This turns out to be a powerful choice: noise is simple to sample, contains no prior assumptions about the output, and gives the model complete freedom to learn the generation process from scratch.

Other architectures make different tradeoffs, suiting different use cases, but noise-based diffusion has proven the most general and the most scalable.

The early attempts at image generation

2012–2014: Autoencoders and variational autoencoders

The first serious attempts at learned image generation came via autoencoders and variational autoencoders (VAEs). The idea: compress an image into a compact latent vector, then reconstruct it from that representation. VAEs went further by modelling the latent space as a probability distribution, enabling genuine sampling of new images.

The results were structurally coherent but blurry. Something was always lost in compression, and the reconstruction couldn't fully recover it.

2014–2020: Generative adversarial networks

Ian Goodfellow's 2014 paper introduced a radically different approach: put two networks against each other. A generator produces fake images from latent noise; a discriminator tries to tell real from fake. Train them adversarially, and both keep improving.

GANs produced far sharper results than VAEs, and by the late 2010s, GAN-generated faces were hard to distinguish from photographs. But they were notoriously difficult to train, and suffered from mode collapse. The generator would converge on a narrow range of outputs rather than capturing the full diversity of real images.

2020–present: The diffusion revolution

A new paradigm emerged, loosely inspired by non-equilibrium thermodynamics. Instead of generating images in a single step, diffusion models corrupt images with noise and then learn to reverse the process.

The results were striking: higher fidelity, greater diversity, and more reliable training than anything before. Diffusion models are now the engine behind Stable Diffusion, DALL-E, Midjourney, and most of the state-of-the-art image generation tools in use today.

How diffusion models actually work

The forward process: Adding noise

The core idea is intuitive. Take a real image and gradually add Gaussian noise, step by step, until nothing recognisable remains. The amount of noise added at each step is controlled by a variance schedule, with small increments early, larger ones later.

A neat mathematical trick called a one-step shortcut makes training practical: rather than running through all the noising steps one by one, you can jump directly from the original image to any noise level in a single calculation. This dramatically speeds up training.

The reverse process: Removing noise

This is where the learning happens. Given a noisy image, the model is trained to predict the noise that was added to it, then subtract it to recover a slightly cleaner version. This step-by-step noise removal is called denoising. Do it enough times, and you go from pure noise all the way back to a coherent image.

The training objective is simple: compare the actual noise to what the model predicted, and minimise the difference. During generation, the model no longer has access to the ground truth noise. It has to predict it entirely on its own. That's exactly what it learned to do.

UNet: The neural network behind diffusion

The denoising network is a U-Net. It is an architecture originally developed in 2015 for biomedical image segmentation. Its structure is symmetric:

• An encoder progressively downsamples the input, extracting features at increasing levels of abstraction from edges to textures to object-level structures.
• A decoder progressively upsamples back to full resolution.
• Skip connections link the encoder and decoder layers, allowing the decoder to recover spatial detail that would otherwise be lost in the bottleneck.

For diffusion, the current timestep is encoded as a small vector and injected throughout the UNet, so the model always knows how noisy its input is and adjusts its denoising accordingly.

Stable diffusion: Moving into latent space

Stable Diffusion is an open-source text-to-image model released in 2022 by Stability AI, based on the Latent Diffusion Model architecture developed by Robin Rombach and his team at the University of Heidelberg. It was the first high-quality diffusion model available to run locally, which made it widely adopted by researchers, developers, and the broader creative community.

Standard diffusion operates directly on pixel values. For a 512×512 image, that's roughly 786,000 numbers to process at every denoising step, which is computationally expensive.

Latent Diffusion Models, the architecture behind Stable Diffusion, solve this by running the diffusion process in a compressed latent space instead. An encoder first compresses the image to a much smaller representation, the diffusion process runs on that, and then a decoder expands the result back into a full-resolution image.

Since latent representations are typically 8× smaller in each spatial dimension, the computational savings are substantial without a meaningful drop in quality. The encoder and decoder are trained once as a VAE and then frozen, so the diffusion model only ever needs to learn in this compact space.

This design also opens the door to conditional generation: the model can be steered by text prompts, semantic maps, reference images, or other signals injected during the denoising process.

Text-to-image: How language guides generation

Getting a diffusion model to respond to a text prompt requires bridging two very different modalities. The key enabler was CLIP (Contrastive Language–Image Pretraining, OpenAI 2021).

CLIP is trained on hundreds of millions of image–text pairs. It learns to pull matching pairs close together in a shared embedding space and push non-matching pairs apart. The result: an image of a dog and the caption "a photo of a dog" end up near each other, while unrelated pairings are pushed far apart.

For text-to-image generation, CLIP's text encoder is plugged into the UNet's denoising process via cross-attention. At every timestep, the noisy image asks: "What noise should I remove?" — and the text embedding answers: "Remove noise in the direction of this specific concept." The model learns to denoise not just toward any clean image, but toward the image described in the prompt.

Images and text reach this shared space through different encoders: convolutional networks for images and attention-based transformers for text, but CLIP ensures their representations are compatible, and cross-attention is the mechanism that brings them together at generation time.

Why diffusion models won

A few properties explain why diffusion models have displaced GANs so thoroughly:

• Training stability. Diffusion models optimise a single, well-behaved loss function — no adversarial balancing act required.
• Mode coverage. The reverse process naturally explores the full distribution, producing diverse outputs rather than collapsing to a narrow set of modes.
• Composability. The conditioning mechanism is modular. The same architecture can be steered by text, images, depth maps, or combinations of them, without fundamental changes.
• Scalability. Latent diffusion brought the compute cost into a practical range, and the architecture scales predictably with more data and more parameters.

The gap between artificial intelligence-generated images has narrowed to near-indistinguishability. The architecture will keep improving. The more interesting question now is what we build with it and how organisations integrate these capabilities into products where they create genuine value.

Find your AI solution