On Diffusion

Around 2020, back when generative AI was still a curiosity, I fell down the rabbit hole. StyleGANs, early diffusion models, latent walks, whatever I could get running. I trained some models of my own through Runway using my photography archive, mostly drone shots and a bunch of trees and flowers I shot around my neighborhood while on walks during the COVID years.

I liked the output, but I liked watching them generate even more. You'd start with something nondescript, and slowly the structure came into focus. Latent walks in particular were fun because you knew the beginning and the end, and could watch the two connect.

I was poking at language models around the same time. In late 2020, I was running GPT-2 in a Google Colab notebook on the company dime and built AndreBot 3000.

This was before AI copywriting was a product. AndreBot was a skunkworks thing I kept around because it was useful.

Things moved quickly after that. Stable Diffusion arrived in 2022, and suddenly anyone could mess with this stuff. Then LLMs ate everything.

Diffusion kept getting better, but the interface became conversational. You stopped opening a separate tool and started asking a language model. DALL-E moved into ChatGPT, and image generation became another tool a model could invoke.

That shift also made one architecture feel synonymous with AI. Most of the models in the apps we actually use generate autoregressively, which just means they build in sequence. If the model has the cat sat on the, it predicts something like mat, and then compounds from there.

Diffusion comes at it from the other direction. The model pulls the whole thing toward something coherent, almost all at once.

I came back to it because of Mercury 2, Inception's diffusion language model (not affiliated). It has the same fuzzy quality I liked in those image models years ago. As different parts come into focus at different times, you can start reading meaning into it before it's actually there.

I've watched things like:

@)Pugi"Th "I am st|K:>ngW tved to eat berries KG=wW2G

resolve into:

{"thought":"I am starving, need to eat berries nearby."}

There's a moment where your brain tries to interpret something the model hasn't finished deciding yet, which feels a bit like staring at a blurry picture of bigfoot a tree.

It's also fast, which is what made the next part practical. In my testing with concurrent agents, Mercury's instant mode came in at a 427ms p50 and 1.24s p95 end to end, with roughly 500 generated tokens per second across the whole group.

My first pass was an isometric town with two dozen hand-written residents. Names, jobs, quirks, little biographies, all living on a fixed map where the buildings were already standing. They could form opinions, reproduce, move around, build things, fight, make up, and get themselves into trouble. Like a chaotic Sims meets Truman Show. It was fun, but ultimately I had preordained everyone's fate.

A town comes with a lot of baggage. Citizens imply families, jobs and relationships; a settlement implies buildings, resources, roads and everything else that makes a town a town. So I tried again. The second version started with two nameless residents on a procedurally generated map. There were no jobs or biographies waiting for them, and the terrain itself was for them to discover, which felt like a much cleaner starting point. Then they gathered materials, built shelters, paired off, produced a child and started quarrelling over scarce resources.

Which was useful, because apparently I hadn't stripped enough out.

The map was still isometric. They were still residents. There were still resources and shelters and reproduction, so even without the authored characters I had left the shape of a town underneath.

The current version goes further. There isn't a town, and there aren't really characters anymore. It starts with one body, a handful of unnamed quantities, and an abstract substrate rendered as procedurally terraformed Unicode. The world can still have shape, variation, adjacency and movement, but a mark on the screen doesn't need to mean tree, road, shelter, food, or anything else unless something inside the system gives it that meaning.

That change has been more useful than I expected, and not just visually. I ran a controlled version of the system with human labels and explicit remedies — things like hunger and food — against an otherwise identical version where the state was just drifting quantities and substances with machine names. The unlabelled version performed better, and it was also the only one where the agents started coining new terms. Giving them my vocabulary made the system easier for me to understand while apparently making it harder for them to develop one of their own.

One of the bodies in the unlabelled experiment eventually wrote a quantified claim about its own state into its own memory without being asked to do it. That emergence interests me a lot more than an agent producing a clever monologue.

Language has been the most obvious place to watch this. There's no requirement that communication stay in clean English just because English makes the logs convenient. If some shorthand ends up being useful, I want to allow it.