wtf is retrieval-augmented-diffusion-models?

Retrieval-Augmented Diffusion Models is a research project that generates images from text prompts, but with a twist: instead of relying purely on what the model learned during training, it also pulls in real images from a reference database to guide the output. You give it a caption like "a happy bear reading a newspaper, oil on canvas" and it produces a matching picture, informed by similar images it retrieves on the fly. The key idea is that when you type a prompt, the system converts your text into a numeric representation using CLIP (a model that understands both images and text in a shared way). It then searches a pre-built database of image embeddings to find visually similar examples. Those retrieved images act as additional context for the diffusion process, which gradually builds a new image step by step. The same trained model can work with different databases, for instance, one built from ImageNet versus one from OpenImages, and swapping databases changes the visual style and capabilities of what it generates, even though the model's weights stay the same. This would appeal to AI researchers, developers exploring generative models, or anyone curious about alternative approaches to text-to-image generation. For example, if you are building an image tool and want to steer outputs toward a specific visual domain, you could prepare a custom retrieval database from your own image collection and let the model reference it at generation time. The project includes two model types: RDM for general image generation and RARM, which is demonstrated on narrower categories like dogs, mammals, or animals from ImageNet. The repo ships with pretrained models and demo notebooks so you can try inference without training anything yourself. Training your own model from scratch is supported but requires significant compute resources and large datasets. The codebase builds on existing diffusion model frameworks from OpenAI and others, so it is primarily aimed at people comfortable navigating research code rather than a polished product.