GRPO/RL Training with TRL Expert-level guidance for implementing Group Relative Policy Optimization (GRPO) using the Transformer Reinforcement Learning (TRL) library. This skill provides battle-tested patterns, critical insights, and production-ready workflows for fine-tuning language models with custom reward functions. When to Use This Skill Use GRPO training when you need to: - Enforce specific output formats (e.g., XML tags, JSON, structured reasoning) - Teach verifiable tasks with objective correctness metrics (math, coding, fact-checking) - Improve reasoning capabilities by rewarding ch…

\n responses = [comp[0]['content'] for comp in completions]\n matches = [re.match(pattern, r, re.DOTALL) for r in responses]\n return [0.5 if match else 0.0 for match in matches]\n\ndef soft_xml_format_reward(completions, **kwargs) -> List[float]:\n \"\"\"\n Relaxed XML format: allows whitespace variations.\n Use for: When structure matters more than exact spacing\n\n Weight: 0.5\n \"\"\"\n pattern = r'\u003creasoning>.*?\u003c/reasoning>\\s*\u003canswer>.*?\u003c/answer>'\n responses = [comp[0]['content'] for comp in completions]\n matches = [re.search(pattern, r, re.DOTALL) for r in responses]\n return [0.5 if match else 0.0 for match in matches]\n\ndef json_format_reward(completions, **kwargs) -> List[float]:\n \"\"\"\n Reward valid JSON output.\n Use for: Structured data extraction, API responses\n\n Weight: 0.5\n \"\"\"\n import json\n\n responses = [comp[0]['content'] for comp in completions]\n rewards = []\n\n for r in responses:\n try:\n json.loads(r)\n rewards.append(0.5)\n except:\n rewards.append(0.0)\n\n return rewards\n\ndef incremental_format_reward(completions, tags=['reasoning', 'answer'], **kwargs) -> List[float]:\n \"\"\"\n Partial credit for each required tag.\n Use for: Training models to gradually learn format\n\n Weight: sum(0.125 * num_tags * 2) = up to 0.5 for 2 tags\n \"\"\"\n responses = [comp[0]['content'] for comp in completions]\n rewards = []\n\n for r in responses:\n score = 0.0\n for tag in tags:\n if f'\u003c{tag}>' in r:\n score += 0.125\n if f'\u003c/{tag}>' in r:\n score += 0.125\n\n # Penalize extra content after final closing tag\n if f'\u003c/{tags[-1]}>' in r:\n extra = r.split(f'\u003c/{tags[-1]}>')[-1].strip()\n score -= len(extra) * 0.001\n\n rewards.append(score)\n\n return rewards\n\n# ==================== LENGTH REWARDS ====================\n\ndef ideal_length_reward(completions, ideal_tokens=100, **kwargs) -> List[float]:\n \"\"\"\n Reward responses near ideal length.\n Use for: Controlling verbosity\n\n Weight: 0.3\n \"\"\"\n responses = [comp[0]['content'] for comp in completions]\n rewards = []\n\n for r in responses:\n length = len(r.split())\n distance = abs(length - ideal_tokens)\n # Gaussian-like reward peaking at ideal length\n reward = 0.3 * max(0, 1 - distance / ideal_tokens)\n rewards.append(reward)\n\n return rewards\n\ndef min_length_reward(completions, min_tokens=50, **kwargs) -> List[float]:\n \"\"\"\n Penalize responses that are too short.\n Use for: Ensuring detailed explanations\n\n Weight: 0.2\n \"\"\"\n responses = [comp[0]['content'] for comp in completions]\n rewards = []\n\n for r in responses:\n length = len(r.split())\n reward = 0.2 if length >= min_tokens else -0.2\n rewards.append(reward)\n\n return rewards\n\ndef max_length_penalty(completions, max_tokens=500, **kwargs) -> List[float]:\n \"\"\"\n Penalize excessively long responses.\n Use for: Preventing rambling\n\n Weight: -0.3 when violated\n \"\"\"\n responses = [comp[0]['content'] for comp in completions]\n rewards = []\n\n for r in responses:\n length = len(r.split())\n reward = -0.3 if length > max_tokens else 0.0\n rewards.append(reward)\n\n return rewards\n\n# ==================== STYLE REWARDS ====================\n\ndef reasoning_quality_reward(completions, **kwargs) -> List[float]:\n \"\"\"\n Reward detailed reasoning with logical connectors.\n Use for: Improving chain-of-thought quality\n\n Weight: 0.3\n \"\"\"\n logical_words = ['therefore', 'thus', 'because', 'since', 'consequently',\n 'first', 'second', 'next', 'finally', 'however']\n\n responses = [comp[0]['content'] for comp in completions]\n rewards = []\n\n for r in responses:\n reasoning = extract_xml_tag(r, 'reasoning').lower()\n # Count logical connectors\n count = sum(1 for word in logical_words if word in reasoning)\n # Normalize by length\n score = min(0.3, count * 0.05)\n rewards.append(score)\n\n return rewards\n\ndef citation_reward(completions, **kwargs) -> List[float]:\n \"\"\"\n Reward responses with citations or references.\n Use for: Research tasks, fact-checking\n\n Weight: 0.2\n \"\"\"\n citation_patterns = [\n r'\\[\\d+\\]', # [1], [2]\n r'\\([A-Z][a-z]+,?\\s+\\d{4}\\)', # (Smith, 2020)\n r'according to',\n r'as stated in',\n ]\n\n responses = [comp[0]['content'] for comp in completions]\n rewards = []\n\n for r in responses:\n has_citation = any(re.search(pattern, r) for pattern in citation_patterns)\n rewards.append(0.2 if has_citation else 0.0)\n\n return rewards\n\ndef no_repetition_penalty(completions, **kwargs) -> List[float]:\n \"\"\"\n Penalize repetitive text (same phrase repeated).\n Use for: Improving output diversity\n\n Weight: -0.3 when repetitive\n \"\"\"\n responses = [comp[0]['content'] for comp in completions]\n rewards = []\n\n for r in responses:\n words = r.lower().split()\n # Check for repeated trigrams\n trigrams = [' '.join(words[i:i+3]) for i in range(len(words)-2)]\n unique_ratio = len(set(trigrams)) / max(len(trigrams), 1)\n\n reward = -0.3 if unique_ratio \u003c 0.7 else 0.0\n rewards.append(reward)\n\n return rewards\n\n# ==================== COMBINED REWARDS ====================\n\ndef math_problem_reward(prompts, completions, answer, **kwargs) -> List[float]:\n \"\"\"\n Combined reward for math problems: format + correctness.\n Automatically balances multiple objectives.\n\n Weight: 2.5 total\n \"\"\"\n format_rewards = soft_xml_format_reward(completions)\n correctness_rewards = exact_match_reward(prompts, completions, answer)\n\n return [f + c for f, c in zip(format_rewards, correctness_rewards)]\n\ndef code_generation_reward(prompts, completions, test_cases, **kwargs) -> List[float]:\n \"\"\"\n Combined reward for code: format + execution + style.\n\n Weight: 2.7 total\n \"\"\"\n code_format_rewards = code_block_format_reward(completions)\n execution_rewards = code_execution_reward(prompts, completions, test_cases)\n no_error_rewards = no_syntax_error_reward(completions)\n\n return [f + e + s for f, e, s in zip(code_format_rewards, execution_rewards, no_error_rewards)]\n\n# ==================== HELPER FUNCTIONS ====================\n\ndef extract_answer(text: str) -> str:\n \"\"\"Extract content from \u003canswer> tags.\"\"\"\n return extract_xml_tag(text, 'answer')\n\ndef extract_xml_tag(text: str, tag: str) -> str:\n \"\"\"Generic XML tag extraction.\"\"\"\n pattern = f'\u003c{tag}>(.*?)\u003c/{tag}>'\n match = re.search(pattern, text, re.DOTALL)\n return match.group(1).strip() if match else \"\"\n\ndef extract_code_block(text: str) -> str:\n \"\"\"Extract code from markdown code blocks.\"\"\"\n pattern = r'```(?:python)?\\n(.*?)\\n```'\n match = re.search(pattern, text, re.DOTALL)\n return match.group(1) if match else \"\"\n\ndef run_test_cases(code: str, test_cases: List[tuple]) -> bool:\n \"\"\"\n Execute code with test cases (MUST be sandboxed in production!).\n\n Args:\n code: Python code string\n test_cases: List of (input, expected_output) tuples\n\n Returns:\n True if all tests pass\n \"\"\"\n # WARNING: This is a simplified example\n # In production, use proper sandboxing (e.g., docker, pypy sandbox)\n try:\n exec_globals = {}\n exec(code, exec_globals)\n\n for input_val, expected in test_cases:\n result = exec_globals['solution'](input_val)\n if result != expected:\n return False\n return True\n except:\n return False\n\n# ==================== REWARD FUNCTION PRESETS ====================\n\n# Preset for math/reasoning tasks\nMATH_REASONING_REWARDS = [\n incremental_format_reward,\n soft_xml_format_reward,\n exact_match_reward,\n reasoning_quality_reward,\n]\n\n# Preset for code generation\nCODE_GENERATION_REWARDS = [\n code_block_format_reward,\n code_execution_reward,\n no_syntax_error_reward,\n]\n\n# Preset for summarization\nSUMMARIZATION_REWARDS = [\n ideal_length_reward,\n fuzzy_match_reward,\n no_repetition_penalty,\n]\n\n# Preset for Q&A\nQA_REWARDS = [\n exact_match_reward,\n min_length_reward,\n citation_reward,\n]\n","content_type":"text/x-python; charset=utf-8","language":"python","size":11568,"content_sha256":"aa2104ebb7bb38df0bd06752998cd9822e47126cf8515f57c1c19545da6364cc"},{"filename":"README.md","content":"# GRPO/RL Training Skill\n\n**Expert-level guidance for Group Relative Policy Optimization with TRL**\n\n## 📁 Skill Structure\n\n```\ngrpo-rl-training/\n├── SKILL.md # Main skill documentation (READ THIS FIRST)\n├── README.md # This file\n├── templates/\n│ └── basic_grpo_training.py # Production-ready training template\n└── examples/\n └── reward_functions_library.py # 20+ reward function examples\n```\n\n## 🚀 Quick Start\n\n1. **Read SKILL.md** - Comprehensive guide with all concepts and patterns\n2. **Copy `templates/basic_grpo_training.py`** - Start with working code\n3. **Browse `examples/reward_functions_library.py`** - Pick reward functions for your task\n4. **Modify for your use case** - Adapt dataset, rewards, and config\n\n## 💡 What's Inside\n\n### SKILL.md (Main Documentation)\n- Core GRPO concepts and algorithm fundamentals\n- Complete implementation workflow (dataset → rewards → training → deployment)\n- 10+ reward function examples with code\n- Hyperparameter tuning guide\n- Training insights (loss behavior, metrics, debugging)\n- Troubleshooting guide\n- Production best practices\n\n### Templates\n- **basic_grpo_training.py**: Minimal, production-ready training script\n - Uses Qwen 2.5 1.5B Instruct\n - 3 reward functions (format + correctness)\n - LoRA for efficient training\n - Fully documented and ready to run\n\n### Examples\n- **reward_functions_library.py**: 20+ battle-tested reward functions\n - Correctness rewards (exact match, fuzzy match, numeric, code execution)\n - Format rewards (XML, JSON, strict/soft)\n - Length rewards (ideal length, min/max)\n - Style rewards (reasoning quality, citations, repetition penalty)\n - Combined rewards (multi-objective optimization)\n - Preset collections for common tasks\n\n## 📖 Usage for Agents\n\nWhen this skill is loaded in your agent's context:\n\n1. **Always read SKILL.md first** before implementing\n2. **Start simple** - Use length-based reward to validate setup\n3. **Build incrementally** - Add one reward function at a time\n4. **Reference examples** - Copy patterns from reward_functions_library.py\n5. **Monitor training** - Watch reward metrics (not loss!)\n\n## 🎯 Common Use Cases\n\n| Task Type | Recommended Rewards | Template |\n|-----------|---------------------|----------|\n| Math reasoning | `MATH_REASONING_REWARDS` preset | basic_grpo_training.py |\n| Code generation | `CODE_GENERATION_REWARDS` preset | Modify dataset in template |\n| Summarization | `SUMMARIZATION_REWARDS` preset | Adjust prompts + rewards |\n| Q&A | `QA_REWARDS` preset | Use fuzzy match + citations |\n\n## ⚠️ Critical Reminders\n\n- **Loss goes UP during training** - This is normal (it's KL divergence)\n- **Use 3-5 reward functions** - Single rewards often fail\n- **Test rewards before training** - Debug each function independently\n- **Monitor reward_std** - Should stay > 0.1 (avoid mode collapse)\n- **Start with num_generations=4-8** - Scale up if GPU allows\n\n## 🔗 External Resources\n\n- [TRL Documentation](https://huggingface.co/docs/trl)\n- [DeepSeek R1 Paper](https://arxiv.org/abs/2501.12948)\n- [Open R1 Implementation](https://github.com/huggingface/open-r1)\n- [Unsloth (2-3x faster)](https://docs.unsloth.ai/)\n\n## 📝 Version\n\n**v1.0.0** - Initial release (January 2025)\n\n## 👨‍💻 Maintained By\n\nOrchestra Research\nFor questions or improvements, see https://orchestra.com\n\n---\n\n**License:** MIT\n**Last Updated:** January 2025\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3514,"content_sha256":"e6c726197d8f97daf8af38489bc0e33f0cf7d224d7279276339e351908abdc8b"},{"filename":"templates/basic_grpo_training.py","content":"\"\"\"\nBasic GRPO Training Template\n=============================\n\nA minimal, production-ready template for GRPO training with TRL.\nAdapt this for your specific task by modifying:\n1. Dataset loading (get_dataset function)\n2. Reward functions (reward_*_func)\n3. System prompt (SYSTEM_PROMPT)\n4. Hyperparameters (GRPOConfig)\n\"\"\"\n\nimport torch\nimport re\nfrom datasets import load_dataset, Dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom peft import LoraConfig\nfrom trl import GRPOTrainer, GRPOConfig\n\n# ==================== CONFIGURATION ====================\n\nMODEL_NAME = \"Qwen/Qwen2.5-1.5B-Instruct\"\nOUTPUT_DIR = \"outputs/grpo-model\"\nMAX_PROMPT_LENGTH = 256\nMAX_COMPLETION_LENGTH = 512\n\nSYSTEM_PROMPT = \"\"\"\nRespond in the following format:\n\u003creasoning>\n[Your step-by-step thinking]\n\u003c/reasoning>\n\u003canswer>\n[Final answer]\n\u003c/answer>\n\"\"\"\n\n# ==================== DATASET ====================\n\ndef get_dataset(split=\"train\"):\n \"\"\"\n Load and prepare your dataset.\n\n Returns: Dataset with columns:\n - 'prompt': List[Dict] with role/content\n - 'answer': str (ground truth, optional)\n \"\"\"\n # Example: GSM8K math dataset\n data = load_dataset('openai/gsm8k', 'main')[split]\n\n def process_example(x):\n # Extract ground truth answer\n answer = x['answer'].split('####')[1].strip() if '####' in x['answer'] else None\n\n return {\n 'prompt': [\n {'role': 'system', 'content': SYSTEM_PROMPT},\n {'role': 'user', 'content': x['question']}\n ],\n 'answer': answer\n }\n\n return data.map(process_example)\n\n# ==================== HELPER FUNCTIONS ====================\n\ndef extract_xml_tag(text: str, tag: str) -> str:\n \"\"\"Extract content between XML tags.\"\"\"\n pattern = f'\u003c{tag}>(.*?)\u003c/{tag}>'\n match = re.search(pattern, text, re.DOTALL)\n return match.group(1).strip() if match else \"\"\n\ndef extract_answer(text: str) -> str:\n \"\"\"Extract the final answer from structured output.\"\"\"\n return extract_xml_tag(text, 'answer')\n\n# ==================== REWARD FUNCTIONS ====================\n\ndef correctness_reward_func(prompts, completions, answer, **kwargs):\n \"\"\"\n Reward correct answers.\n Weight: 2.0 (highest priority)\n \"\"\"\n responses = [comp[0]['content'] for comp in completions]\n extracted = [extract_answer(r) for r in responses]\n return [2.0 if ans == gt else 0.0 for ans, gt in zip(extracted, answer)]\n\ndef format_reward_func(completions, **kwargs):\n \"\"\"\n Reward proper XML format.\n Weight: 0.5\n \"\"\"\n pattern = r'\u003creasoning>.*?\u003c/reasoning>\\s*\u003canswer>.*?\u003c/answer>'\n responses = [comp[0]['content'] for comp in completions]\n return [0.5 if re.search(pattern, r, re.DOTALL) else 0.0 for r in responses]\n\ndef incremental_format_reward_func(completions, **kwargs):\n \"\"\"\n Incremental reward for partial format compliance.\n Weight: up to 0.5\n \"\"\"\n responses = [comp[0]['content'] for comp in completions]\n rewards = []\n\n for r in responses:\n score = 0.0\n if '\u003creasoning>' in r:\n score += 0.125\n if '\u003c/reasoning>' in r:\n score += 0.125\n if '\u003canswer>' in r:\n score += 0.125\n if '\u003c/answer>' in r:\n score += 0.125\n\n # Penalize extra content after closing tag\n if '\u003c/answer>' in r:\n extra = r.split('\u003c/answer>')[-1].strip()\n score -= len(extra) * 0.001\n\n rewards.append(score)\n\n return rewards\n\n# ==================== MODEL SETUP ====================\n\ndef setup_model_and_tokenizer():\n \"\"\"Load model and tokenizer with optimizations.\"\"\"\n model = AutoModelForCausalLM.from_pretrained(\n MODEL_NAME,\n torch_dtype=torch.bfloat16,\n attn_implementation=\"flash_attention_2\",\n device_map=\"auto\"\n )\n\n tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)\n tokenizer.pad_token = tokenizer.eos_token\n\n return model, tokenizer\n\ndef get_peft_config():\n \"\"\"LoRA configuration for parameter-efficient training.\"\"\"\n return LoraConfig(\n r=16,\n lora_alpha=32,\n target_modules=[\n \"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n \"gate_proj\", \"up_proj\", \"down_proj\"\n ],\n task_type=\"CAUSAL_LM\",\n lora_dropout=0.05,\n )\n\n# ==================== TRAINING ====================\n\ndef main():\n \"\"\"Main training function.\"\"\"\n\n # Load data\n print(\"Loading dataset...\")\n dataset = get_dataset()\n print(f\"Dataset size: {len(dataset)}\")\n\n # Setup model\n print(\"Loading model...\")\n model, tokenizer = setup_model_and_tokenizer()\n\n # Training configuration\n training_args = GRPOConfig(\n output_dir=OUTPUT_DIR,\n run_name=\"grpo-training\",\n\n # Learning rate\n learning_rate=5e-6,\n adam_beta1=0.9,\n adam_beta2=0.99,\n weight_decay=0.1,\n warmup_ratio=0.1,\n lr_scheduler_type='cosine',\n\n # Batch settings\n per_device_train_batch_size=1,\n gradient_accumulation_steps=4,\n\n # GRPO specific\n num_generations=8,\n max_prompt_length=MAX_PROMPT_LENGTH,\n max_completion_length=MAX_COMPLETION_LENGTH,\n\n # Training duration\n num_train_epochs=1,\n\n # Optimization\n bf16=True,\n optim=\"adamw_8bit\",\n max_grad_norm=0.1,\n\n # Logging\n logging_steps=1,\n save_steps=100,\n report_to=\"wandb\", # Change to \"none\" to disable logging\n )\n\n # Initialize trainer\n trainer = GRPOTrainer(\n model=model,\n processing_class=tokenizer,\n reward_funcs=[\n incremental_format_reward_func,\n format_reward_func,\n correctness_reward_func,\n ],\n args=training_args,\n train_dataset=dataset,\n peft_config=get_peft_config(),\n )\n\n # Train\n print(\"Starting training...\")\n trainer.train()\n\n # Save final model\n print(f\"Saving model to {OUTPUT_DIR}/final\")\n trainer.save_model(f\"{OUTPUT_DIR}/final\")\n\n print(\"Training complete!\")\n\nif __name__ == \"__main__\":\n main()\n","content_type":"text/x-python; charset=utf-8","language":"python","size":6122,"content_sha256":"95abe6a3f69bea79d52c1a36301c7600a1439eb805f442b1cda304812ebced02"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"GRPO/RL Training with TRL","type":"text"}]},{"type":"paragraph","content":[{"text":"Expert-level guidance for implementing Group Relative Policy Optimization (GRPO) using the Transformer Reinforcement Learning (TRL) library. This skill provides battle-tested patterns, critical insights, and production-ready workflows for fine-tuning language models with custom reward functions.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"When to Use This Skill","type":"text"}]},{"type":"paragraph","content":[{"text":"Use GRPO training when you need to:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Enforce specific output formats","type":"text","marks":[{"type":"strong"}]},{"text":" (e.g., XML tags, JSON, structured reasoning)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Teach verifiable tasks","type":"text","marks":[{"type":"strong"}]},{"text":" with objective correctness metrics (math, coding, fact-checking)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Improve reasoning capabilities","type":"text","marks":[{"type":"strong"}]},{"text":" by rewarding chain-of-thought patterns","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Align models to domain-specific behaviors","type":"text","marks":[{"type":"strong"}]},{"text":" without labeled preference data","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Optimize for multiple objectives","type":"text","marks":[{"type":"strong"}]},{"text":" simultaneously (format + correctness + style)","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Do NOT use GRPO for:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Simple supervised fine-tuning tasks (use SFT instead)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Tasks without clear reward signals","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"When you already have high-quality preference pairs (use DPO/PPO instead)","type":"text"}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Core Concepts","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"1. GRPO Algorithm Fundamentals","type":"text"}]},{"type":"paragraph","content":[{"text":"Key Mechanism:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Generates ","type":"text"},{"text":"multiple completions","type":"text","marks":[{"type":"strong"}]},{"text":" for each prompt (group size: 4-16)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Compares completions within each group using reward functions","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Updates policy to favor higher-rewarded responses relative to the group","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Critical Difference from PPO:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"No separate reward model needed","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"More sample-efficient (learns from within-group comparisons)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Simpler to implement and debug","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Mathematical Intuition:","type":"text","marks":[{"type":"strong"}]}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"For each prompt p:\n 1. Generate N completions: {c₁, c₂, ..., cₙ}\n 2. Compute rewards: {r₁, r₂, ..., rₙ}\n 3. Learn to increase probability of high-reward completions\n relative to low-reward ones in the same group","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"2. Reward Function Design Philosophy","type":"text"}]},{"type":"paragraph","content":[{"text":"Golden Rules:","type":"text","marks":[{"type":"strong"}]}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Compose multiple reward functions","type":"text","marks":[{"type":"strong"}]},{"text":" - Each handles one aspect (format, correctness, style)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Scale rewards appropriately","type":"text","marks":[{"type":"strong"}]},{"text":" - Higher weight = stronger signal","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Use incremental rewards","type":"text","marks":[{"type":"strong"}]},{"text":" - Partial credit for partial compliance","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Test rewards independently","type":"text","marks":[{"type":"strong"}]},{"text":" - Debug each reward function in isolation","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Reward Function Types:","type":"text","marks":[{"type":"strong"}]}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Type","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use Case","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Example Weight","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Correctness","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Verifiable tasks (math, code)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"2.0 (highest)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Format","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Strict structure enforcement","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"0.5-1.0","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Length","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Encourage verbosity/conciseness","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"0.1-0.5","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Style","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Penalize unwanted patterns","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"-0.5 to 0.5","type":"text"}]}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Implementation Workflow","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 1: Dataset Preparation","type":"text"}]},{"type":"paragraph","content":[{"text":"Critical Requirements:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Prompts in chat format (list of dicts with 'role' and 'content')","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Include system prompts to set expectations","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For verifiable tasks, include ground truth answers as additional columns","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Example Structure:","type":"text","marks":[{"type":"strong"}]}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"from datasets import load_dataset, Dataset\n\nSYSTEM_PROMPT = \"\"\"\nRespond in the following format:\n\u003creasoning>\n[Your step-by-step thinking]\n\u003c/reasoning>\n\u003canswer>\n[Final answer]\n\u003c/answer>\n\"\"\"\n\ndef prepare_dataset(raw_data):\n \"\"\"\n Transform raw data into GRPO-compatible format.\n\n Returns: Dataset with columns:\n - 'prompt': List[Dict] with role/content (system + user messages)\n - 'answer': str (ground truth, optional but recommended)\n \"\"\"\n return raw_data.map(lambda x: {\n 'prompt': [\n {'role': 'system', 'content': SYSTEM_PROMPT},\n {'role': 'user', 'content': x['question']}\n ],\n 'answer': extract_answer(x['raw_answer'])\n })","type":"text"}]},{"type":"paragraph","content":[{"text":"Pro Tips:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Use one-shot or few-shot examples in system prompt for complex formats","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Keep prompts concise (max_prompt_length: 256-512 tokens)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Validate data quality before training (garbage in = garbage out)","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 2: Reward Function Implementation","type":"text"}]},{"type":"paragraph","content":[{"text":"Template Structure:","type":"text","marks":[{"type":"strong"}]}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"def reward_function_name(\n prompts, # List[List[Dict]]: Original prompts\n completions, # List[List[Dict]]: Model generations\n answer=None, # Optional: Ground truth from dataset\n **kwargs # Additional dataset columns\n) -> list[float]:\n \"\"\"\n Evaluate completions and return rewards.\n\n Returns: List of floats (one per completion)\n \"\"\"\n # Extract completion text\n responses = [comp[0]['content'] for comp in completions]\n\n # Compute rewards\n rewards = []\n for response in responses:\n score = compute_score(response)\n rewards.append(score)\n\n return rewards","type":"text"}]},{"type":"paragraph","content":[{"text":"Example 1: Correctness Reward (Math/Coding)","type":"text","marks":[{"type":"strong"}]}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"def correctness_reward(prompts, completions, answer, **kwargs):\n \"\"\"Reward correct answers with high score.\"\"\"\n responses = [comp[0]['content'] for comp in completions]\n extracted = [extract_final_answer(r) for r in responses]\n return [2.0 if ans == gt else 0.0\n for ans, gt in zip(extracted, answer)]","type":"text"}]},{"type":"paragraph","content":[{"text":"Example 2: Format Reward (Structured Output)","type":"text","marks":[{"type":"strong"}]}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"import re\n\ndef format_reward(completions, **kwargs):\n \"\"\"Reward XML-like structured format.\"\"\"\n pattern = r'\u003creasoning>.*?\u003c/reasoning>\\s*\u003canswer>.*?\u003c/answer>'\n responses = [comp[0]['content'] for comp in completions]\n return [1.0 if re.search(pattern, r, re.DOTALL) else 0.0\n for r in responses]","type":"text"}]},{"type":"paragraph","content":[{"text":"Example 3: Incremental Format Reward (Partial Credit)","type":"text","marks":[{"type":"strong"}]}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"def incremental_format_reward(completions, **kwargs):\n \"\"\"Award partial credit for format compliance.\"\"\"\n responses = [comp[0]['content'] for comp in completions]\n rewards = []\n\n for r in responses:\n score = 0.0\n if '\u003creasoning>' in r:\n score += 0.25\n if '\u003c/reasoning>' in r:\n score += 0.25\n if '\u003canswer>' in r:\n score += 0.25\n if '\u003c/answer>' in r:\n score += 0.25\n # Penalize extra text after closing tag\n if r.count('\u003c/answer>') == 1:\n extra_text = r.split('\u003c/answer>')[-1].strip()\n score -= len(extra_text) * 0.001\n rewards.append(score)\n\n return rewards","type":"text"}]},{"type":"paragraph","content":[{"text":"Critical Insight:","type":"text","marks":[{"type":"strong"}]},{"text":" Combine 3-5 reward functions for robust training. Order matters less than diversity of signals.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 3: Training Configuration","type":"text"}]},{"type":"paragraph","content":[{"text":"Memory-Optimized Config (Small GPU)","type":"text","marks":[{"type":"strong"}]}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"from trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n output_dir=\"outputs/grpo-model\",\n\n # Learning rate\n learning_rate=5e-6, # Lower = more stable\n adam_beta1=0.9,\n adam_beta2=0.99,\n weight_decay=0.1,\n warmup_ratio=0.1,\n lr_scheduler_type='cosine',\n\n # Batch settings\n per_device_train_batch_size=1,\n gradient_accumulation_steps=4, # Effective batch = 4\n\n # GRPO-specific\n num_generations=8, # Group size: 8-16 recommended\n max_prompt_length=256,\n max_completion_length=512,\n\n # Training duration\n num_train_epochs=1,\n max_steps=None, # Or set fixed steps (e.g., 500)\n\n # Optimization\n bf16=True, # Faster on A100/H100\n optim=\"adamw_8bit\", # Memory-efficient optimizer\n max_grad_norm=0.1,\n\n # Logging\n logging_steps=1,\n save_steps=100,\n report_to=\"wandb\", # Or \"none\" for no logging\n)","type":"text"}]},{"type":"paragraph","content":[{"text":"High-Performance Config (Large GPU)","type":"text","marks":[{"type":"strong"}]}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"training_args = GRPOConfig(\n output_dir=\"outputs/grpo-model\",\n learning_rate=1e-5,\n per_device_train_batch_size=4,\n gradient_accumulation_steps=2,\n num_generations=16, # Larger groups = better signal\n max_prompt_length=512,\n max_completion_length=1024,\n num_train_epochs=1,\n bf16=True,\n use_vllm=True, # Fast generation with vLLM\n logging_steps=10,\n)","type":"text"}]},{"type":"paragraph","content":[{"text":"Critical Hyperparameters:","type":"text","marks":[{"type":"strong"}]}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Parameter","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Impact","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Tuning Advice","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"num_generations","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Group size for comparison","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Start with 8, increase to 16 if GPU allows","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"learning_rate","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Convergence speed/stability","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"5e-6 (safe), 1e-5 (faster, riskier)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"max_completion_length","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Output verbosity","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Match your task (512 for reasoning, 256 for short answers)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"gradient_accumulation_steps","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Effective batch size","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Increase if GPU memory limited","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 4: Model Setup and Training","type":"text"}]},{"type":"paragraph","content":[{"text":"Standard Setup (Transformers)","type":"text","marks":[{"type":"strong"}]}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"import torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom peft import LoraConfig\nfrom trl import GRPOTrainer\n\n# Load model\nmodel_name = \"Qwen/Qwen2.5-1.5B-Instruct\"\nmodel = AutoModelForCausalLM.from_pretrained(\n model_name,\n torch_dtype=torch.bfloat16,\n attn_implementation=\"flash_attention_2\", # 2-3x faster\n device_map=\"auto\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\ntokenizer.pad_token = tokenizer.eos_token\n\n# Optional: LoRA for parameter-efficient training\npeft_config = LoraConfig(\n r=16, # Rank (higher = more capacity)\n lora_alpha=32, # Scaling factor (typically 2*r)\n target_modules=[\n \"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n \"gate_proj\", \"up_proj\", \"down_proj\"\n ],\n task_type=\"CAUSAL_LM\",\n lora_dropout=0.05,\n)\n\n# Initialize trainer\ntrainer = GRPOTrainer(\n model=model,\n processing_class=tokenizer,\n reward_funcs=[\n incremental_format_reward,\n format_reward,\n correctness_reward,\n ],\n args=training_args,\n train_dataset=dataset,\n peft_config=peft_config, # Remove for full fine-tuning\n)\n\n# Train\ntrainer.train()\n\n# Save\ntrainer.save_model(\"final_model\")","type":"text"}]},{"type":"paragraph","content":[{"text":"Unsloth Setup (2-3x Faster)","type":"text","marks":[{"type":"strong"}]}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"from unsloth import FastLanguageModel\n\nmodel, tokenizer = FastLanguageModel.from_pretrained(\n model_name=\"google/gemma-3-1b-it\",\n max_seq_length=1024,\n load_in_4bit=True,\n fast_inference=True,\n max_lora_rank=32,\n)\n\nmodel = FastLanguageModel.get_peft_model(\n model,\n r=32,\n target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n \"gate_proj\", \"up_proj\", \"down_proj\"],\n lora_alpha=32,\n use_gradient_checkpointing=\"unsloth\",\n)\n\n# Rest is identical to standard setup\ntrainer = GRPOTrainer(model=model, ...)\ntrainer.train()","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Critical Training Insights","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"1. Loss Behavior (EXPECTED PATTERN)","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Loss starts near 0 and INCREASES during training","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"This is CORRECT - loss measures KL divergence from initial policy","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Model is learning (diverging from original behavior to optimize rewards)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Monitor reward metrics instead of loss for progress","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"2. Reward Tracking","type":"text"}]},{"type":"paragraph","content":[{"text":"Key metrics to watch:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"reward","type":"text","marks":[{"type":"code_inline"}]},{"text":": Average across all completions","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"reward_std","type":"text","marks":[{"type":"code_inline"}]},{"text":": Diversity within groups (should remain > 0)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"kl","type":"text","marks":[{"type":"code_inline"}]},{"text":": KL divergence from reference (should grow moderately)","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Healthy Training Pattern:","type":"text","marks":[{"type":"strong"}]}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Step Reward Reward_Std KL\n100 0.5 0.3 0.02\n200 0.8 0.25 0.05\n300 1.2 0.2 0.08 ← Good progression\n400 1.5 0.15 0.12","type":"text"}]},{"type":"paragraph","content":[{"text":"Warning Signs:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Reward std → 0 (model collapsing to single response)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"KL exploding (> 0.5) (diverging too much, reduce LR)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Reward stuck (reward functions too harsh or model capacity issue)","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"3. Common Pitfalls and Solutions","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Problem","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Symptom","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Solution","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Mode collapse","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"All completions identical","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Increase ","type":"text"},{"text":"num_generations","type":"text","marks":[{"type":"code_inline"}]},{"text":", add diversity penalty","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"No learning","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Flat rewards","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Check reward function logic, increase LR","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"OOM errors","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"GPU memory exceeded","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Reduce ","type":"text"},{"text":"num_generations","type":"text","marks":[{"type":"code_inline"}]},{"text":", enable gradient checkpointing","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Slow training","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"\u003c 1 it/s","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Enable ","type":"text"},{"text":"use_vllm=True","type":"text","marks":[{"type":"code_inline"}]},{"text":", use Unsloth, reduce seq length","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Format ignored","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Model doesn't follow structure","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Increase format reward weight, add incremental rewards","type":"text"}]}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Advanced Patterns","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"1. Multi-Stage Training","type":"text"}]},{"type":"paragraph","content":[{"text":"For complex tasks, train in stages:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"# Stage 1: Format compliance (epochs=1)\ntrainer_stage1 = GRPOTrainer(\n model=model,\n reward_funcs=[incremental_format_reward, format_reward],\n ...\n)\ntrainer_stage1.train()\n\n# Stage 2: Correctness (epochs=1)\ntrainer_stage2 = GRPOTrainer(\n model=model,\n reward_funcs=[format_reward, correctness_reward],\n ...\n)\ntrainer_stage2.train()","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"2. Adaptive Reward Scaling","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"class AdaptiveReward:\n def __init__(self, base_reward_func, initial_weight=1.0):\n self.func = base_reward_func\n self.weight = initial_weight\n\n def __call__(self, *args, **kwargs):\n rewards = self.func(*args, **kwargs)\n return [r * self.weight for r in rewards]\n\n def adjust_weight(self, success_rate):\n \"\"\"Increase weight if model struggling, decrease if succeeding.\"\"\"\n if success_rate \u003c 0.3:\n self.weight *= 1.2\n elif success_rate > 0.8:\n self.weight *= 0.9","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"3. Custom Dataset Integration","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"def load_custom_knowledge_base(csv_path):\n \"\"\"Example: School communication platform docs.\"\"\"\n import pandas as pd\n df = pd.read_csv(csv_path)\n\n dataset = Dataset.from_pandas(df).map(lambda x: {\n 'prompt': [\n {'role': 'system', 'content': CUSTOM_SYSTEM_PROMPT},\n {'role': 'user', 'content': x['question']}\n ],\n 'answer': x['expert_answer']\n })\n return dataset","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Deployment and Inference","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Save and Merge LoRA","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"# Merge LoRA adapters into base model\nif hasattr(trainer.model, 'merge_and_unload'):\n merged_model = trainer.model.merge_and_unload()\n merged_model.save_pretrained(\"production_model\")\n tokenizer.save_pretrained(\"production_model\")","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Inference Example","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"from transformers import pipeline\n\ngenerator = pipeline(\n \"text-generation\",\n model=\"production_model\",\n tokenizer=tokenizer\n)\n\nresult = generator(\n [\n {'role': 'system', 'content': SYSTEM_PROMPT},\n {'role': 'user', 'content': \"What is 15 + 27?\"}\n ],\n max_new_tokens=256,\n do_sample=True,\n temperature=0.7,\n top_p=0.9\n)\nprint(result[0]['generated_text'])","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Best Practices Checklist","type":"text"}]},{"type":"paragraph","content":[{"text":"Before Training:","type":"text","marks":[{"type":"strong"}]}]},{"type":"checkbox_list","attrs":{"id":null},"content":[{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Validate dataset format (prompts as List[Dict])","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Test reward functions on sample data","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Calculate expected max_prompt_length from data","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Choose appropriate num_generations based on GPU memory","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Set up logging (wandb recommended)","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"During Training:","type":"text","marks":[{"type":"strong"}]}]},{"type":"checkbox_list","attrs":{"id":null},"content":[{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Monitor reward progression (should increase)","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Check reward_std (should stay > 0.1)","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Watch for OOM errors (reduce batch size if needed)","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Sample generations every 50-100 steps","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Validate format compliance on holdout set","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"After Training:","type":"text","marks":[{"type":"strong"}]}]},{"type":"checkbox_list","attrs":{"id":null},"content":[{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Merge LoRA weights if using PEFT","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Test on diverse prompts","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Compare to baseline model","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Document reward weights and hyperparameters","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Save reproducibility config","type":"text"}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Troubleshooting Guide","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Debugging Workflow","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Isolate reward functions","type":"text","marks":[{"type":"strong"}]},{"text":" - Test each independently","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Check data distribution","type":"text","marks":[{"type":"strong"}]},{"text":" - Ensure diversity in prompts","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Reduce complexity","type":"text","marks":[{"type":"strong"}]},{"text":" - Start with single reward, add gradually","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Monitor generations","type":"text","marks":[{"type":"strong"}]},{"text":" - Print samples every N steps","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Validate extraction logic","type":"text","marks":[{"type":"strong"}]},{"text":" - Ensure answer parsing works","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Quick Fixes","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"# Debug reward function\ndef debug_reward(completions, **kwargs):\n responses = [comp[0]['content'] for comp in completions]\n for i, r in enumerate(responses[:2]): # Print first 2\n print(f\"Response {i}: {r[:200]}...\")\n return [1.0] * len(responses) # Dummy rewards\n\n# Test without training\ntrainer = GRPOTrainer(..., reward_funcs=[debug_reward])\ntrainer.generate_completions(dataset[:1]) # Generate without updating","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"References and Resources","type":"text"}]},{"type":"paragraph","content":[{"text":"Official Documentation:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"TRL GRPO Trainer: https://huggingface.co/docs/trl/grpo_trainer","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"DeepSeek R1 Paper: https://arxiv.org/abs/2501.12948","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Unsloth Docs: https://docs.unsloth.ai/","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Example Repositories:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Open R1 Implementation: https://github.com/huggingface/open-r1","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"TRL Examples: https://github.com/huggingface/trl/tree/main/examples","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Recommended Reading:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Progressive Disclosure Pattern for agent instructions","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Reward shaping in RL (Ng et al.)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"LoRA paper (Hu et al., 2021)","type":"text"}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Usage Instructions for Agents","type":"text"}]},{"type":"paragraph","content":[{"text":"When this skill is loaded:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Read this entire file","type":"text","marks":[{"type":"strong"}]},{"text":" before implementing GRPO training","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Start with the simplest reward function","type":"text","marks":[{"type":"strong"}]},{"text":" (e.g., length-based) to validate setup","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Use the templates","type":"text","marks":[{"type":"strong"}]},{"text":" in ","type":"text"},{"text":"templates/","type":"text","marks":[{"type":"code_inline"}]},{"text":" directory as starting points","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Reference examples","type":"text","marks":[{"type":"strong"}]},{"text":" in ","type":"text"},{"text":"examples/","type":"text","marks":[{"type":"code_inline"}]},{"text":" for task-specific implementations","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Follow the workflow","type":"text","marks":[{"type":"strong"}]},{"text":" sequentially (don't skip steps)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Debug incrementally","type":"text","marks":[{"type":"strong"}]},{"text":" - add one reward function at a time","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Critical Reminders:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Always use multiple reward functions (3-5 is optimal)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Monitor reward metrics, not loss","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Test reward functions before training","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Start small (num_generations=4), scale up gradually","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Save checkpoints frequently (every 100 steps)","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"This skill is designed for ","type":"text"},{"text":"expert-level implementation","type":"text","marks":[{"type":"strong"}]},{"text":". Beginners should start with supervised fine-tuning before attempting GRPO.","type":"text"}]}]},"metadata":{"date":"2026-11-02","name":"grpo-rl-training","tags":["Post-Training","Reinforcement Learning","GRPO","TRL","RLHF","Reward Modeling","Reasoning","DPO","PPO","Structured Output"],"author":"Orchestra Research","source":{"stars":14202,"repo_name":"AI-Research-SKILLs","origin_url":"https://github.com/Orchestra-Research/AI-Research-SKILLs/blob/HEAD/06-post-training/grpo-rl-training/SKILL.md","repo_owner":"Orchestra-Research","body_sha256":"c9b771c280840f12d0e98749250c35df4a83a21f95465c41e8ae13f0229ecb40","cluster_key":"f162a23bb192edda6b97e6dfb98169c9d010fb7bba529e07e9152473f93045b8","clean_bundle":{"format":"clean-skill-bundle-v1","source":"Orchestra-Research/AI-Research-SKILLs/06-post-training/grpo-rl-training/SKILL.md","text_bytes":21204,"source_commit":"773a52944ba4747a18bd4ae9ade53fff041adcbc","skills_sh_rank":13125,"attachment_count":3,"text_attachments":3,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"06-post-training/grpo-rl-training/SKILL.md","import_metadata":{"date":"2026-11-02","author":"Orchestra Research","version":"1.0.0","category":"ai-agent-development","category_label":"AI"},"exact_dupes_collapsed_into_this":0},"license":"MIT","version":"1.0.0","category":"ai-agent-development","import_tag":"skills-sh-top20k-2026-11","description":"Expert guidance for GRPO/RL fine-tuning with TRL for reasoning and task-specific model training","dependencies":["transformers>=4.47.0","trl>=0.14.0","datasets>=3.2.0","peft>=0.14.0","torch"]}},"renderedAt":1793666431472}

GRPO/RL Training with TRL Expert-level guidance for implementing Group Relative Policy Optimization (GRPO) using the Transformer Reinforcement Learning (TRL) library. This skill provides battle-tested patterns, critical insights, and production-ready workflows for fine-tuning language models with custom reward functions. When to Use This Skill Use GRPO training when you need to: - Enforce specific output formats (e.g., XML tags, JSON, structured reasoning) - Teach verifiable tasks with objective correctness metrics (math, coding, fact-checking) - Improve reasoning capabilities by rewarding ch…