What is Stable Diffusion Trained On?

Stable Diffusion is trained on LAION (Large-scale Artificial Intelligence Open Network) datasets, primarily LAION-5B containing 5.85 billion image-text pairs scraped from the internet. The training process uses LAION-400M and LAION-2B subsets for initial stages, followed by LAION-5B for final training. These datasets include images with associated alt-text, captions, and metadata from web sources like Common Crawl. Training employs a three-stage process: autoencoder training on ImageNet, text encoder training using CLIP methodology, and diffusion model training in latent space. The model learns associations between textual descriptions and visual concepts through contrastive learning and denoising objectives. Additional fine-tuning uses curated datasets and safety filtering to reduce harmful content generation. For AI agents, understanding Stable Diffusion’s training data helps predict model capabilities, limitations, and potential biases in generated outputs.