Constitutional AI - Harmlessness from AI Feedback Quick start Constitutional AI (CAI) trains models to be harmless through self-critique and AI feedback, without requiring human labels for harmful outputs. Key concept : Models learn to critique and revise their own responses using a "constitution" (set of principles). Two phases : 1. Supervised Learning (SL) : Self-critique + revision 2. Reinforcement Learning (RL) : RLAIF (RL from AI Feedback) Constitution example : Common workflows Workflow 1: Supervised learning phase (self-critique + revision) Step 1: Generate initial responses : Step 2:…