Weak to Strong Generalization

Weak to strong generalization is an AI alignment research paradigm where weaker AI models are used to supervise and train stronger AI systems, addressing the fundamental challenge of how humans can maintain oversight over superintelligent AI. This concept, pioneered by OpenAI, explores whether weak supervisors can effectively guide more capable models to perform tasks correctly, even when the supervisor cannot fully evaluate the stronger model's outputs. The approach involves training a weak model on limited data, then using it to generate training signals for a stronger model that has access to more computational resources and data. Weak to strong generalization is crucial for AI safety because it mirrors the real-world scenario where humans (the weak supervisor) must align and control AI systems that may eventually surpass human capabilities. Research in this area investigates techniques like constitutional AI, recursive reward modeling, and scalable oversight methods. The goal is to develop robust alignment techniques that can scale with increasing AI capabilities, ensuring that advanced AI systems remain beneficial and controllable even when they exceed human-level performance in specific domains.

Related terms

• Weak-to-Strong Generalization
• Weak to strong
• Strong AI is
• Strong AI is and could be considered to have
• Strong Artificial Intelligence

Vstorm builds production systems that use Weak to Strong Generalization: Agentic AI consulting.

← Back to the AI Glossary