Senior Prompt Engineer Eval-driven prompt engineering, RAG quality measurement, and agent workflow validation. Everything here is model-agnostic by design : techniques are framed by what they do, not by which model generation they were observed on, and the tools never hardcode model IDs or pricing — you supply your provider's current rates when you want dollar figures. Operating Rules 1. Never change a prompt without a baseline. Capture metrics first ( ), then compare every iteration against it. 2. Eval set before optimization. 10–20 representative cases with expected outputs minimum. If the…

)]\n if missing:\n tool_status[tool.name] = f\"WARN: Missing config: {missing}\"\n else:\n tool_status[tool.name] = \"OK\"\n else:\n tool_status[tool.name] = \"OK - No config needed\"\n\n # Check description\n if not tool.description:\n warnings.append(f\"Tool '{tool.name}' has no description\")\n\n # Validate pattern-specific requirements\n if config.pattern == AgentPattern.MULTI_AGENT:\n if len(config.tools) \u003c 2:\n warnings.append(\"Multi-agent pattern typically requires 2+ specialized tools\")\n\n # Check for potential infinite loops\n potential_loop = config.max_iterations > 50\n\n # Estimate tokens\n base_tokens = len(config.system_prompt.split()) * 1.3 if config.system_prompt else 200\n tool_tokens = sum(t.estimated_tokens for t in config.tools)\n\n min_tokens = int(base_tokens + tool_tokens)\n max_tokens = int((base_tokens + tool_tokens * 2) * config.max_iterations)\n\n return ValidationResult(\n is_valid=len(errors) == 0,\n errors=errors,\n warnings=warnings,\n tool_status=tool_status,\n estimated_tokens_per_run=(min_tokens, max_tokens),\n potential_infinite_loop=potential_loop,\n max_depth=config.max_iterations\n )\n\n\ndef generate_ascii_diagram(config: AgentConfig) -> str:\n \"\"\"Generate ASCII workflow diagram\"\"\"\n lines = []\n\n # Header\n width = max(40, len(config.name) + 10)\n lines.append(\"┌\" + \"─\" * width + \"┐\")\n lines.append(\"│\" + config.name.center(width) + \"│\")\n lines.append(\"│\" + f\"({config.pattern.value} Pattern)\".center(width) + \"│\")\n lines.append(\"└\" + \"─\" * (width // 2 - 1) + \"┬\" + \"─\" * (width // 2) + \"┘\")\n lines.append(\" \" * (width // 2) + \"│\")\n\n # User Query\n lines.append(\" \" * (width // 2 - 8) + \"┌───────────────┐\")\n lines.append(\" \" * (width // 2 - 8) + \"│ User Query │\")\n lines.append(\" \" * (width // 2 - 8) + \"└───────┬───────┘\")\n lines.append(\" \" * (width // 2) + \"│\")\n\n if config.pattern == AgentPattern.REACT:\n # ReAct loop\n lines.append(\" \" * (width // 2 - 8) + \"┌───────────────┐\")\n lines.append(\" \" * (width // 2 - 8) + \"│ Think │◄──────┐\")\n lines.append(\" \" * (width // 2 - 8) + \"└───────┬───────┘ │\")\n lines.append(\" \" * (width // 2) + \"│ │\")\n lines.append(\" \" * (width // 2 - 8) + \"┌───────────────┐ │\")\n lines.append(\" \" * (width // 2 - 8) + \"│ Select Tool │ │\")\n lines.append(\" \" * (width // 2 - 8) + \"└───────┬───────┘ │\")\n lines.append(\" \" * (width // 2) + \"│ │\")\n\n # Tools\n if config.tools:\n tool_line = \" \".join([f\"[{t.name}]\" for t in config.tools[:4]])\n if len(config.tools) > 4:\n tool_line += \" ...\"\n lines.append(\" \" * 4 + tool_line)\n lines.append(\" \" * (width // 2) + \"│ │\")\n\n lines.append(\" \" * (width // 2 - 8) + \"┌───────────────┐ │\")\n lines.append(\" \" * (width // 2 - 8) + \"│ Observe │───────┘\")\n lines.append(\" \" * (width // 2 - 8) + \"└───────┬───────┘\")\n\n elif config.pattern == AgentPattern.PLAN_EXECUTE:\n # Plan phase\n lines.append(\" \" * (width // 2 - 8) + \"┌───────────────┐\")\n lines.append(\" \" * (width // 2 - 8) + \"│ Create Plan │\")\n lines.append(\" \" * (width // 2 - 8) + \"└───────┬───────┘\")\n lines.append(\" \" * (width // 2) + \"│\")\n\n # Execute loop\n lines.append(\" \" * (width // 2 - 8) + \"┌───────────────┐\")\n lines.append(\" \" * (width // 2 - 8) + \"│ Execute Step │◄──────┐\")\n lines.append(\" \" * (width // 2 - 8) + \"└───────┬───────┘ │\")\n lines.append(\" \" * (width // 2) + \"│ │\")\n\n if config.tools:\n tool_line = \" \".join([f\"[{t.name}]\" for t in config.tools[:4]])\n lines.append(\" \" * 4 + tool_line)\n lines.append(\" \" * (width // 2) + \"│ │\")\n\n lines.append(\" \" * (width // 2 - 8) + \"┌───────────────┐ │\")\n lines.append(\" \" * (width // 2 - 8) + \"│ Check Done? │───────┘\")\n lines.append(\" \" * (width // 2 - 8) + \"└───────┬───────┘\")\n\n else:\n # Generic tool use\n lines.append(\" \" * (width // 2 - 8) + \"┌───────────────┐\")\n lines.append(\" \" * (width // 2 - 8) + \"│ Process Query │\")\n lines.append(\" \" * (width // 2 - 8) + \"└───────┬───────┘\")\n lines.append(\" \" * (width // 2) + \"│\")\n\n if config.tools:\n for tool in config.tools[:6]:\n lines.append(\" \" * (width // 2 - 8) + f\"├──▶ [{tool.name}]\")\n if len(config.tools) > 6:\n lines.append(\" \" * (width // 2 - 8) + \"├──▶ [...]\")\n\n # Final answer\n lines.append(\" \" * (width // 2) + \"│\")\n lines.append(\" \" * (width // 2 - 8) + \"┌───────────────┐\")\n lines.append(\" \" * (width // 2 - 8) + \"│ Final Answer │\")\n lines.append(\" \" * (width // 2 - 8) + \"└───────────────┘\")\n\n return '\\n'.join(lines)\n\n\ndef generate_mermaid_diagram(config: AgentConfig) -> str:\n \"\"\"Generate Mermaid flowchart\"\"\"\n lines = [\"```mermaid\", \"flowchart TD\"]\n\n # Start and query\n lines.append(f\" subgraph {config.name}[{config.name}]\")\n lines.append(\" direction TB\")\n lines.append(\" A[User Query] --> B{Process}\")\n\n if config.pattern == AgentPattern.REACT:\n lines.append(\" B --> C[Think]\")\n lines.append(\" C --> D{Select Tool}\")\n\n for i, tool in enumerate(config.tools[:6]):\n lines.append(f\" D -->|{tool.name}| T{i}[{tool.name}]\")\n lines.append(f\" T{i} --> E[Observe]\")\n\n lines.append(\" E -->|Continue| C\")\n lines.append(\" E -->|Done| F[Final Answer]\")\n\n elif config.pattern == AgentPattern.PLAN_EXECUTE:\n lines.append(\" B --> P[Create Plan]\")\n lines.append(\" P --> X{Execute Step}\")\n\n for i, tool in enumerate(config.tools[:6]):\n lines.append(f\" X -->|{tool.name}| T{i}[{tool.name}]\")\n lines.append(f\" T{i} --> R[Review]\")\n\n lines.append(\" R -->|More Steps| X\")\n lines.append(\" R -->|Complete| F[Final Answer]\")\n\n else:\n for i, tool in enumerate(config.tools[:6]):\n lines.append(f\" B -->|use| T{i}[{tool.name}]\")\n lines.append(f\" T{i} --> F[Final Answer]\")\n\n lines.append(\" end\")\n lines.append(\"```\")\n\n return '\\n'.join(lines)\n\n\ndef estimate_cost(config: AgentConfig, runs: int = 100,\n input_price_per_mtok: Optional[float] = None,\n output_price_per_mtok: Optional[float] = None) -> Dict[str, Any]:\n \"\"\"Estimate token usage (always) and dollar costs (only with user-supplied prices).\n\n Model pricing changes too often to hardcode. Look up your provider's current\n rates and pass --input-price-per-mtok / --output-price-per-mtok; without them\n the tool reports token estimates only.\n \"\"\"\n validation = validate_agent(config)\n min_tokens, max_tokens = validation.estimated_tokens_per_run\n\n result: Dict[str, Any] = {\n 'model': config.model,\n 'tokens_per_run': {'min': min_tokens, 'max': max_tokens},\n 'cost_per_run': None,\n 'estimated_monthly': {'runs': runs * 30, 'cost_min': None, 'cost_max': None},\n }\n\n if input_price_per_mtok is None or output_price_per_mtok is None:\n return result\n\n # Assume 60% input, 40% output\n def run_cost(tokens: int) -> float:\n return (tokens * 0.6 / 1_000_000 * input_price_per_mtok +\n tokens * 0.4 / 1_000_000 * output_price_per_mtok)\n\n cost_per_run_min = run_cost(min_tokens)\n cost_per_run_max = run_cost(max_tokens)\n\n result['cost_per_run'] = {'min': round(cost_per_run_min, 4), 'max': round(cost_per_run_max, 4)}\n result['estimated_monthly'] = {\n 'runs': runs * 30,\n 'cost_min': round(cost_per_run_min * runs * 30, 2),\n 'cost_max': round(cost_per_run_max * runs * 30, 2),\n }\n return result\n\n\ndef format_validation_report(config: AgentConfig, result: ValidationResult) -> str:\n \"\"\"Format validation result as human-readable report\"\"\"\n lines = []\n lines.append(\"=\" * 50)\n lines.append(\"AGENT VALIDATION REPORT\")\n lines.append(\"=\" * 50)\n lines.append(\"\")\n\n lines.append(f\"📋 AGENT INFO\")\n lines.append(f\" Name: {config.name}\")\n lines.append(f\" Pattern: {config.pattern.value}\")\n lines.append(f\" Model: {config.model}\")\n lines.append(\"\")\n\n lines.append(f\"🔧 TOOLS ({len(config.tools)} registered)\")\n for tool in config.tools:\n status = result.tool_status.get(tool.name, \"Unknown\")\n emoji = \"✅\" if status.startswith(\"OK\") else \"⚠️\"\n lines.append(f\" {emoji} {tool.name} - {status}\")\n lines.append(\"\")\n\n lines.append(\"📊 FLOW ANALYSIS\")\n lines.append(f\" Max iterations: {result.max_depth}\")\n lines.append(f\" Estimated tokens: {result.estimated_tokens_per_run[0]:,} - {result.estimated_tokens_per_run[1]:,}\")\n lines.append(f\" Potential loop: {'⚠️ Yes' if result.potential_infinite_loop else '✅ No'}\")\n lines.append(\"\")\n\n if result.errors:\n lines.append(f\"❌ ERRORS ({len(result.errors)})\")\n for error in result.errors:\n lines.append(f\" • {error}\")\n lines.append(\"\")\n\n if result.warnings:\n lines.append(f\"⚠️ WARNINGS ({len(result.warnings)})\")\n for warning in result.warnings:\n lines.append(f\" • {warning}\")\n lines.append(\"\")\n\n # Overall status\n if result.is_valid:\n lines.append(\"✅ VALIDATION PASSED\")\n else:\n lines.append(\"❌ VALIDATION FAILED\")\n\n lines.append(\"\")\n lines.append(\"=\" * 50)\n\n return '\\n'.join(lines)\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"Agent Orchestrator - Design and validate agent workflows\",\n formatter_class=argparse.RawDescriptionHelpFormatter,\n epilog=\"\"\"\nExamples:\n %(prog)s agent.yaml --validate\n %(prog)s agent.yaml --visualize\n %(prog)s agent.yaml --visualize --format mermaid\n %(prog)s agent.yaml --estimate-cost --runs 100\n\nAgent config format (YAML):\n\nname: research_assistant\npattern: react\nmodel: any-model-id # optional, informational only\nmax_iterations: 10\ntools:\n - name: web_search\n description: Search the web\n required_config: [api_key]\n - name: calculator\n description: Evaluate math expressions\n \"\"\"\n )\n\n parser.add_argument('config', help='Agent configuration file (YAML or JSON)')\n parser.add_argument('--validate', '-V', action='store_true', help='Validate agent configuration')\n parser.add_argument('--visualize', '-v', action='store_true', help='Visualize agent workflow')\n parser.add_argument('--format', '-f', choices=['ascii', 'mermaid'], default='ascii',\n help='Visualization format (default: ascii)')\n parser.add_argument('--estimate-cost', '-e', action='store_true',\n help='Estimate token usage (and dollar cost if prices supplied)')\n parser.add_argument('--runs', '-r', type=int, default=100, help='Daily runs for cost estimation')\n parser.add_argument('--input-price-per-mtok', type=float, default=None,\n help='Input price in USD per million tokens (your provider\\'s current rate)')\n parser.add_argument('--output-price-per-mtok', type=float, default=None,\n help='Output price in USD per million tokens (your provider\\'s current rate)')\n parser.add_argument('--output', '-o', help='Output file path')\n parser.add_argument('--json', '-j', action='store_true', help='Output as JSON')\n\n args = parser.parse_args()\n\n # Load config\n config_path = Path(args.config)\n if not config_path.exists():\n print(f\"Error: Config file not found: {args.config}\", file=sys.stderr)\n sys.exit(1)\n\n try:\n config = load_config(config_path)\n except Exception as e:\n print(f\"Error parsing config: {e}\", file=sys.stderr)\n sys.exit(1)\n\n # Default to validate if no action specified\n if not any([args.validate, args.visualize, args.estimate_cost]):\n args.validate = True\n\n output_parts = []\n\n # Validate\n if args.validate:\n result = validate_agent(config)\n if args.json:\n output_parts.append(json.dumps(asdict(result), indent=2))\n else:\n output_parts.append(format_validation_report(config, result))\n\n # Visualize\n if args.visualize:\n if args.format == 'mermaid':\n diagram = generate_mermaid_diagram(config)\n else:\n diagram = generate_ascii_diagram(config)\n output_parts.append(diagram)\n\n # Cost estimation\n if args.estimate_cost:\n costs = estimate_cost(config, args.runs,\n args.input_price_per_mtok, args.output_price_per_mtok)\n if args.json:\n output_parts.append(json.dumps(costs, indent=2))\n else:\n output_parts.append(\"\")\n output_parts.append(\"💰 COST ESTIMATION\")\n output_parts.append(f\" Model: {costs['model']}\")\n output_parts.append(f\" Tokens per run: {costs['tokens_per_run']['min']:,} - {costs['tokens_per_run']['max']:,}\")\n if costs['cost_per_run'] is not None:\n output_parts.append(f\" Cost per run: ${costs['cost_per_run']['min']:.4f} - ${costs['cost_per_run']['max']:.4f}\")\n output_parts.append(f\" Monthly ({costs['estimated_monthly']['runs']:,} runs):\")\n output_parts.append(f\" Min: ${costs['estimated_monthly']['cost_min']:.2f}\")\n output_parts.append(f\" Max: ${costs['estimated_monthly']['cost_max']:.2f}\")\n else:\n output_parts.append(\" Dollar cost: n/a — pass --input-price-per-mtok and \"\n \"--output-price-per-mtok with your provider's current rates\")\n\n # Output\n output = '\\n'.join(output_parts)\n print(output)\n\n if args.output:\n Path(args.output).write_text(output)\n print(f\"\\nOutput saved to {args.output}\")\n\n\nif __name__ == '__main__':\n main()\n","content_type":"text/x-python; charset=utf-8","language":"python","size":21450,"content_sha256":"5db98fffb31dfa384a20981e6ad1a7938607194cfd3b49169e47808805c9cffe"},{"filename":"scripts/prompt_optimizer.py","content":"#!/usr/bin/env python3\n\"\"\"\nPrompt Optimizer - Static analysis tool for prompt engineering\n\nFeatures:\n- Token estimation (model-agnostic chars-per-token approximation)\n- Prompt structure analysis\n- Clarity scoring\n- Few-shot example extraction and management\n- Optimization suggestions\n\nModel-agnostic by design: pass any model name with --model (only the tokenizer\nfamily is inferred from it), and supply current pricing yourself with\n--price-per-mtok if you want cost estimates. No model IDs or prices are\nhardcoded, so this tool does not rot as the model landscape changes.\n\nUsage:\n python prompt_optimizer.py prompt.txt --analyze\n python prompt_optimizer.py prompt.txt --tokens\n python prompt_optimizer.py prompt.txt --tokens --model claude --price-per-mtok 3.00\n python prompt_optimizer.py prompt.txt --optimize --output optimized.txt\n python prompt_optimizer.py prompt.txt --extract-examples --output examples.json\n\"\"\"\n\nimport argparse\nimport json\nimport re\nimport sys\nfrom pathlib import Path\nfrom typing import Dict, List, Optional, Tuple\nfrom dataclasses import dataclass, asdict\n\n\n# Token estimation ratios (chars per token approximation), inferred from the\n# tokenizer FAMILY in the --model string. Any model name is accepted.\nTOKEN_RATIOS = {\n 'claude': 3.5, # Anthropic tokenizer tends to be denser\n 'default': 4.0, # common BPE-family approximation\n}\n\n\n@dataclass\nclass PromptAnalysis:\n \"\"\"Results of prompt analysis\"\"\"\n token_count: int\n estimated_cost: Optional[float]\n model: str\n clarity_score: int\n structure_score: int\n issues: List[Dict[str, str]]\n suggestions: List[str]\n sections: List[Dict[str, any]]\n has_examples: bool\n example_count: int\n has_output_format: bool\n word_count: int\n line_count: int\n\n\n@dataclass\nclass FewShotExample:\n \"\"\"A single few-shot example\"\"\"\n input_text: str\n output_text: str\n index: int\n\n\ndef estimate_tokens(text: str, model: str = 'default') -> int:\n \"\"\"Estimate token count based on character ratio (family inferred from model name)\"\"\"\n family = 'claude' if 'claude' in model.lower() else 'default'\n ratio = TOKEN_RATIOS[family]\n return int(len(text) / ratio)\n\n\ndef estimate_cost(token_count: int, price_per_mtok: Optional[float] = None) -> Optional[float]:\n \"\"\"Estimate input cost from a user-supplied price (USD per million tokens).\n\n Returns None when no price is supplied — pricing changes too often to hardcode;\n look up your provider's current rate and pass --price-per-mtok.\n \"\"\"\n if price_per_mtok is None:\n return None\n return round((token_count / 1_000_000) * price_per_mtok, 6)\n\n\ndef find_ambiguous_instructions(text: str) -> List[Dict[str, str]]:\n \"\"\"Find vague or ambiguous instructions\"\"\"\n issues = []\n\n # Vague verbs that need specificity\n vague_patterns = [\n (r'\\b(analyze|process|handle|deal with)\\b', 'Vague verb - specify the exact action'),\n (r'\\b(good|nice|appropriate|suitable)\\b', 'Subjective term - define specific criteria'),\n (r'\\b(etc\\.|and so on|and more)\\b', 'Open-ended list - enumerate all items explicitly'),\n (r'\\b(if needed|as necessary|when appropriate)\\b', 'Conditional without criteria - specify when'),\n (r'\\b(some|several|many|few|various)\\b', 'Vague quantity - use specific numbers'),\n ]\n\n lines = text.split('\\n')\n for i, line in enumerate(lines, 1):\n for pattern, message in vague_patterns:\n matches = re.finditer(pattern, line, re.IGNORECASE)\n for match in matches:\n issues.append({\n 'type': 'ambiguity',\n 'line': i,\n 'text': match.group(),\n 'message': message,\n 'context': line.strip()[:80]\n })\n\n return issues\n\n\ndef find_redundant_content(text: str) -> List[Dict[str, str]]:\n \"\"\"Find potentially redundant content\"\"\"\n issues = []\n lines = text.split('\\n')\n\n # Check for repeated phrases (3+ words)\n seen_phrases = {}\n for i, line in enumerate(lines, 1):\n words = line.split()\n for j in range(len(words) - 2):\n phrase = ' '.join(words[j:j+3]).lower()\n phrase = re.sub(r'[^\\w\\s]', '', phrase)\n if phrase and len(phrase) > 10:\n if phrase in seen_phrases:\n issues.append({\n 'type': 'redundancy',\n 'line': i,\n 'text': phrase,\n 'message': f'Phrase repeated from line {seen_phrases[phrase]}',\n 'context': line.strip()[:80]\n })\n else:\n seen_phrases[phrase] = i\n\n return issues\n\n\ndef check_output_format(text: str) -> Tuple[bool, List[str]]:\n \"\"\"Check if prompt specifies output format\"\"\"\n suggestions = []\n\n format_indicators = [\n r'respond\\s+(in|with)\\s+(json|xml|csv|markdown)',\n r'output\\s+format',\n r'return\\s+(only|just)',\n r'format:\\s*\\n',\n r'\\{[\"\\']?\\w+[\"\\']?\\s*:', # JSON-like structure\n r'```\\w*\\n', # Code block\n ]\n\n has_format = any(re.search(p, text, re.IGNORECASE) for p in format_indicators)\n\n if not has_format:\n suggestions.append('Add explicit output format specification (e.g., \"Respond in JSON with keys: ...\")')\n\n return has_format, suggestions\n\n\ndef extract_sections(text: str) -> List[Dict[str, any]]:\n \"\"\"Extract logical sections from prompt\"\"\"\n sections = []\n\n # Common section patterns\n section_patterns = [\n r'^#+\\s+(.+)

Senior Prompt Engineer Eval-driven prompt engineering, RAG quality measurement, and agent workflow validation. Everything here is model-agnostic by design : techniques are framed by what they do, not by which model generation they were observed on, and the tools never hardcode model IDs or pricing — you supply your provider's current rates when you want dollar figures. Operating Rules 1. Never change a prompt without a baseline. Capture metrics first ( ), then compare every iteration against it. 2. Eval set before optimization. 10–20 representative cases with expected outputs minimum. If the…

, # Markdown headers\n r'^([A-Z][A-Za-z\\s]+):\\s*

Senior Prompt Engineer Eval-driven prompt engineering, RAG quality measurement, and agent workflow validation. Everything here is model-agnostic by design : techniques are framed by what they do, not by which model generation they were observed on, and the tools never hardcode model IDs or pricing — you supply your provider's current rates when you want dollar figures. Operating Rules 1. Never change a prompt without a baseline. Capture metrics first ( ), then compare every iteration against it. 2. Eval set before optimization. 10–20 representative cases with expected outputs minimum. If the…

, # Title Case Label:\n r'^(Instructions|Context|Examples?|Input|Output|Task|Role|Format)[:.]',\n ]\n\n lines = text.split('\\n')\n current_section = {'name': 'Introduction', 'start': 1, 'content': []}\n\n for i, line in enumerate(lines, 1):\n is_header = False\n for pattern in section_patterns:\n match = re.match(pattern, line.strip(), re.IGNORECASE)\n if match:\n if current_section['content']:\n current_section['end'] = i - 1\n current_section['line_count'] = len(current_section['content'])\n sections.append(current_section)\n current_section = {\n 'name': match.group(1).strip() if match.groups() else line.strip(),\n 'start': i,\n 'content': []\n }\n is_header = True\n break\n\n if not is_header:\n current_section['content'].append(line)\n\n # Add last section\n if current_section['content']:\n current_section['end'] = len(lines)\n current_section['line_count'] = len(current_section['content'])\n sections.append(current_section)\n\n return sections\n\n\ndef extract_few_shot_examples(text: str) -> List[FewShotExample]:\n \"\"\"Extract few-shot examples from prompt\"\"\"\n examples = []\n\n # Pattern 1: \"Example N:\" or \"Example:\" blocks\n example_pattern = r'Example\\s*\\d*:\\s*\\n(Input:\\s*(.+?)\\n(?:Output:\\s*(.+?)(?=\\n\\nExample|\\n\\n[A-Z]|\\Z)))'\n\n matches = re.finditer(example_pattern, text, re.DOTALL | re.IGNORECASE)\n for i, match in enumerate(matches, 1):\n examples.append(FewShotExample(\n input_text=match.group(2).strip() if match.group(2) else '',\n output_text=match.group(3).strip() if match.group(3) else '',\n index=i\n ))\n\n # Pattern 2: Input/Output pairs without \"Example\" label\n if not examples:\n io_pattern = r'Input:\\s*[\"\\']?(.+?)[\"\\']?\\s*\\nOutput:\\s*(.+?)(?=\\nInput:|\\Z)'\n matches = re.finditer(io_pattern, text, re.DOTALL)\n for i, match in enumerate(matches, 1):\n examples.append(FewShotExample(\n input_text=match.group(1).strip(),\n output_text=match.group(2).strip(),\n index=i\n ))\n\n return examples\n\n\ndef calculate_clarity_score(text: str, issues: List[Dict]) -> int:\n \"\"\"Calculate clarity score (0-100)\"\"\"\n score = 100\n\n # Deduct for issues\n score -= len([i for i in issues if i['type'] == 'ambiguity']) * 5\n score -= len([i for i in issues if i['type'] == 'redundancy']) * 3\n\n # Check for structure\n if not re.search(r'^#+\\s|^[A-Z][a-z]+:', text, re.MULTILINE):\n score -= 10 # No clear sections\n\n # Check for instruction clarity\n if not re.search(r'(you (should|must|will)|please|your task)', text, re.IGNORECASE):\n score -= 5 # No clear directives\n\n return max(0, min(100, score))\n\n\ndef calculate_structure_score(sections: List[Dict], has_format: bool, has_examples: bool) -> int:\n \"\"\"Calculate structure score (0-100)\"\"\"\n score = 50 # Base score\n\n # Bonus for clear sections\n if len(sections) >= 2:\n score += 15\n if len(sections) >= 4:\n score += 10\n\n # Bonus for output format\n if has_format:\n score += 15\n\n # Bonus for examples\n if has_examples:\n score += 10\n\n return min(100, score)\n\n\ndef generate_suggestions(analysis: PromptAnalysis) -> List[str]:\n \"\"\"Generate optimization suggestions\"\"\"\n suggestions = []\n\n if not analysis.has_output_format:\n suggestions.append('Add explicit output format: \"Respond in JSON with keys: ...\"')\n\n if analysis.example_count == 0:\n suggestions.append('Consider adding 2-3 few-shot examples for consistent outputs')\n elif analysis.example_count == 1:\n suggestions.append('Add 1-2 more examples to improve consistency')\n elif analysis.example_count > 5:\n suggestions.append(f'Consider reducing examples from {analysis.example_count} to 3-5 to save tokens')\n\n if analysis.clarity_score \u003c 70:\n suggestions.append('Improve clarity: replace vague terms with specific instructions')\n\n if analysis.token_count > 2000:\n suggestions.append(f'Prompt is {analysis.token_count} tokens - consider condensing for cost efficiency')\n\n # Check for role prompting\n if not re.search(r'you are|act as|as a\\s+\\w+', analysis.sections[0].get('content', [''])[0] if analysis.sections else '', re.IGNORECASE):\n suggestions.append('Consider adding role context: \"You are an expert...\"')\n\n return suggestions\n\n\ndef analyze_prompt(text: str, model: str = 'default',\n price_per_mtok: Optional[float] = None) -> PromptAnalysis:\n \"\"\"Perform comprehensive prompt analysis\"\"\"\n\n # Basic metrics\n token_count = estimate_tokens(text, model)\n cost = estimate_cost(token_count, price_per_mtok)\n word_count = len(text.split())\n line_count = len(text.split('\\n'))\n\n # Find issues\n ambiguity_issues = find_ambiguous_instructions(text)\n redundancy_issues = find_redundant_content(text)\n all_issues = ambiguity_issues + redundancy_issues\n\n # Extract structure\n sections = extract_sections(text)\n examples = extract_few_shot_examples(text)\n has_format, format_suggestions = check_output_format(text)\n\n # Calculate scores\n clarity_score = calculate_clarity_score(text, all_issues)\n structure_score = calculate_structure_score(sections, has_format, len(examples) > 0)\n\n analysis = PromptAnalysis(\n token_count=token_count,\n estimated_cost=cost,\n model=model,\n clarity_score=clarity_score,\n structure_score=structure_score,\n issues=all_issues,\n suggestions=[],\n sections=[{'name': s['name'], 'lines': f\"{s['start']}-{s.get('end', s['start'])}\"} for s in sections],\n has_examples=len(examples) > 0,\n example_count=len(examples),\n has_output_format=has_format,\n word_count=word_count,\n line_count=line_count\n )\n\n analysis.suggestions = generate_suggestions(analysis) + format_suggestions\n\n return analysis\n\n\ndef optimize_prompt(text: str) -> str:\n \"\"\"Generate optimized version of prompt\"\"\"\n optimized = text\n\n # Remove redundant whitespace\n optimized = re.sub(r'\\n{3,}', '\\n\\n', optimized)\n optimized = re.sub(r' {2,}', ' ', optimized)\n\n # Trim lines\n lines = [line.rstrip() for line in optimized.split('\\n')]\n optimized = '\\n'.join(lines)\n\n return optimized.strip()\n\n\ndef format_report(analysis: PromptAnalysis) -> str:\n \"\"\"Format analysis as human-readable report\"\"\"\n report = []\n report.append(\"=\" * 50)\n report.append(\"PROMPT ANALYSIS REPORT\")\n report.append(\"=\" * 50)\n report.append(\"\")\n\n report.append(\"📊 METRICS\")\n report.append(f\" Token count: {analysis.token_count:,}\")\n if analysis.estimated_cost is not None:\n report.append(f\" Estimated cost: ${analysis.estimated_cost:.4f} ({analysis.model}, user-supplied price)\")\n else:\n report.append(\" Estimated cost: n/a (pass --price-per-mtok with your provider's current rate)\")\n report.append(f\" Word count: {analysis.word_count:,}\")\n report.append(f\" Line count: {analysis.line_count}\")\n report.append(\"\")\n\n report.append(\"📈 SCORES\")\n report.append(f\" Clarity: {analysis.clarity_score}/100 {'✅' if analysis.clarity_score >= 70 else '⚠️'}\")\n report.append(f\" Structure: {analysis.structure_score}/100 {'✅' if analysis.structure_score >= 70 else '⚠️'}\")\n report.append(\"\")\n\n report.append(\"📋 STRUCTURE\")\n report.append(f\" Sections: {len(analysis.sections)}\")\n report.append(f\" Examples: {analysis.example_count} {'✅' if analysis.has_examples else '❌'}\")\n report.append(f\" Output format: {'✅ Specified' if analysis.has_output_format else '❌ Missing'}\")\n report.append(\"\")\n\n if analysis.sections:\n report.append(\" Detected sections:\")\n for section in analysis.sections:\n report.append(f\" - {section['name']} (lines {section['lines']})\")\n report.append(\"\")\n\n if analysis.issues:\n report.append(f\"⚠️ ISSUES FOUND ({len(analysis.issues)})\")\n for issue in analysis.issues[:10]: # Limit to first 10\n report.append(f\" Line {issue['line']}: {issue['message']}\")\n report.append(f\" Found: \\\"{issue['text']}\\\"\")\n if len(analysis.issues) > 10:\n report.append(f\" ... and {len(analysis.issues) - 10} more issues\")\n report.append(\"\")\n\n if analysis.suggestions:\n report.append(\"💡 SUGGESTIONS\")\n for i, suggestion in enumerate(analysis.suggestions, 1):\n report.append(f\" {i}. {suggestion}\")\n report.append(\"\")\n\n report.append(\"=\" * 50)\n\n return '\\n'.join(report)\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"Prompt Optimizer - Analyze and optimize prompts\",\n formatter_class=argparse.RawDescriptionHelpFormatter,\n epilog=\"\"\"\nExamples:\n %(prog)s prompt.txt --analyze\n %(prog)s prompt.txt --tokens --model claude --price-per-mtok 3.00\n %(prog)s prompt.txt --optimize --output optimized.txt\n %(prog)s prompt.txt --extract-examples --output examples.json\n \"\"\"\n )\n\n parser.add_argument('prompt', help='Prompt file to analyze')\n parser.add_argument('--analyze', '-a', action='store_true', help='Run full analysis')\n parser.add_argument('--tokens', '-t', action='store_true', help='Count tokens only')\n parser.add_argument('--optimize', '-O', action='store_true', help='Generate optimized version')\n parser.add_argument('--extract-examples', '-e', action='store_true', help='Extract few-shot examples')\n parser.add_argument('--model', '-m', default='default',\n help='Model name (any string; only the tokenizer family is inferred: '\n 'names containing \"claude\" use 3.5 chars/token, otherwise 4.0)')\n parser.add_argument('--price-per-mtok', type=float, default=None,\n help='Input price in USD per million tokens (look up your provider\\'s '\n 'current rate); omit to skip cost estimation')\n parser.add_argument('--output', '-o', help='Output file path')\n parser.add_argument('--json', '-j', action='store_true', help='Output as JSON')\n parser.add_argument('--compare', '-c', help='Compare with baseline analysis JSON')\n\n args = parser.parse_args()\n\n # Read prompt file\n prompt_path = Path(args.prompt)\n if not prompt_path.exists():\n print(f\"Error: File not found: {args.prompt}\", file=sys.stderr)\n sys.exit(1)\n\n text = prompt_path.read_text(encoding='utf-8')\n\n # Tokens only\n if args.tokens:\n token_count = estimate_tokens(text, args.model)\n cost = estimate_cost(token_count, args.price_per_mtok)\n if args.json:\n print(json.dumps({\n 'tokens': token_count,\n 'cost': cost,\n 'model': args.model\n }, indent=2))\n else:\n print(f\"Tokens: {token_count:,}\")\n if cost is not None:\n print(f\"Estimated cost: ${cost:.4f} ({args.model}, user-supplied price)\")\n sys.exit(0)\n\n # Extract examples\n if args.extract_examples:\n examples = extract_few_shot_examples(text)\n output = [asdict(ex) for ex in examples]\n\n if args.output:\n Path(args.output).write_text(json.dumps(output, indent=2))\n print(f\"Extracted {len(examples)} examples to {args.output}\")\n else:\n print(json.dumps(output, indent=2))\n sys.exit(0)\n\n # Optimize\n if args.optimize:\n optimized = optimize_prompt(text)\n\n if args.output:\n Path(args.output).write_text(optimized)\n print(f\"Optimized prompt written to {args.output}\")\n\n # Show comparison\n orig_tokens = estimate_tokens(text, args.model)\n new_tokens = estimate_tokens(optimized, args.model)\n saved = orig_tokens - new_tokens\n print(f\"Tokens: {orig_tokens:,} -> {new_tokens:,} (saved {saved:,})\")\n else:\n print(optimized)\n sys.exit(0)\n\n # Default: full analysis\n analysis = analyze_prompt(text, args.model, args.price_per_mtok)\n\n # Compare with baseline\n if args.compare:\n baseline_path = Path(args.compare)\n if baseline_path.exists():\n baseline = json.loads(baseline_path.read_text())\n print(\"\\n📊 COMPARISON WITH BASELINE\")\n print(f\" Tokens: {baseline.get('token_count', 0):,} -> {analysis.token_count:,}\")\n print(f\" Clarity: {baseline.get('clarity_score', 0)} -> {analysis.clarity_score}\")\n print(f\" Issues: {len(baseline.get('issues', []))} -> {len(analysis.issues)}\")\n print()\n\n if args.json:\n print(json.dumps(asdict(analysis), indent=2))\n else:\n print(format_report(analysis))\n\n # Write to output file\n if args.output:\n output_data = asdict(analysis)\n Path(args.output).write_text(json.dumps(output_data, indent=2))\n print(f\"\\nAnalysis saved to {args.output}\")\n\n\nif __name__ == '__main__':\n main()\n","content_type":"text/x-python; charset=utf-8","language":"python","size":19039,"content_sha256":"cbf0beead6af85424ce3f750a12110f56b06b2996713fc7f0d2aaa12b72c6eab"},{"filename":"scripts/rag_evaluator.py","content":"#!/usr/bin/env python3\n\"\"\"\nRAG Evaluator - Evaluation tool for Retrieval-Augmented Generation systems\n\nFeatures:\n- Context relevance scoring (lexical overlap)\n- Answer faithfulness checking\n- Retrieval metrics (Precision@K, Recall@K, MRR)\n- Coverage analysis\n- Quality report generation\n\nUsage:\n python rag_evaluator.py --contexts contexts.json --questions questions.json\n python rag_evaluator.py --contexts ctx.json --questions q.json --metrics relevance,faithfulness\n python rag_evaluator.py --contexts ctx.json --questions q.json --output report.json --verbose\n\"\"\"\n\nimport argparse\nimport json\nimport re\nimport sys\nfrom pathlib import Path\nfrom typing import Dict, List, Optional, Set, Tuple\nfrom dataclasses import dataclass, asdict, field\nfrom collections import Counter\nimport math\n\n\n@dataclass\nclass RetrievalMetrics:\n \"\"\"Retrieval quality metrics\"\"\"\n precision_at_k: float\n recall_at_k: float\n mrr: float # Mean Reciprocal Rank\n ndcg_at_k: float\n k: int\n\n\n@dataclass\nclass ContextEvaluation:\n \"\"\"Evaluation of a single context\"\"\"\n context_id: str\n relevance_score: float\n token_overlap: float\n key_terms_covered: List[str]\n missing_terms: List[str]\n\n\n@dataclass\nclass AnswerEvaluation:\n \"\"\"Evaluation of an answer against context\"\"\"\n question_id: str\n faithfulness_score: float\n groundedness_score: float\n claims: List[Dict[str, any]]\n unsupported_claims: List[str]\n context_used: List[str]\n\n\n@dataclass\nclass RAGEvaluationReport:\n \"\"\"Complete RAG evaluation report\"\"\"\n total_questions: int\n avg_context_relevance: float\n avg_faithfulness: float\n avg_groundedness: float\n retrieval_metrics: Dict[str, float]\n coverage: float\n issues: List[Dict[str, str]]\n recommendations: List[str]\n question_details: List[Dict[str, any]] = field(default_factory=list)\n\n\ndef tokenize(text: str) -> List[str]:\n \"\"\"Simple tokenization for text comparison\"\"\"\n # Lowercase and split on non-alphanumeric\n text = text.lower()\n tokens = re.findall(r'\\b\\w+\\b', text)\n # Remove common stopwords\n stopwords = {'the', 'a', 'an', 'is', 'are', 'was', 'were', 'be', 'been',\n 'being', 'have', 'has', 'had', 'do', 'does', 'did', 'will',\n 'would', 'could', 'should', 'may', 'might', 'must', 'shall',\n 'can', 'to', 'of', 'in', 'for', 'on', 'with', 'at', 'by',\n 'from', 'as', 'into', 'through', 'during', 'before', 'after',\n 'above', 'below', 'up', 'down', 'out', 'off', 'over', 'under',\n 'again', 'further', 'then', 'once', 'here', 'there', 'when',\n 'where', 'why', 'how', 'all', 'each', 'few', 'more', 'most',\n 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own',\n 'same', 'so', 'than', 'too', 'very', 'just', 'and', 'but',\n 'if', 'or', 'because', 'until', 'while', 'it', 'this', 'that',\n 'these', 'those', 'i', 'you', 'he', 'she', 'we', 'they'}\n return [t for t in tokens if t not in stopwords and len(t) > 2]\n\n\ndef extract_key_terms(text: str, top_n: int = 10) -> List[str]:\n \"\"\"Extract key terms from text based on frequency\"\"\"\n tokens = tokenize(text)\n freq = Counter(tokens)\n return [term for term, _ in freq.most_common(top_n)]\n\n\ndef calculate_token_overlap(text1: str, text2: str) -> float:\n \"\"\"Calculate Jaccard similarity between two texts\"\"\"\n tokens1 = set(tokenize(text1))\n tokens2 = set(tokenize(text2))\n\n if not tokens1 or not tokens2:\n return 0.0\n\n intersection = tokens1 & tokens2\n union = tokens1 | tokens2\n\n return len(intersection) / len(union) if union else 0.0\n\n\ndef calculate_rouge_l(reference: str, candidate: str) -> float:\n \"\"\"Calculate ROUGE-L score (Longest Common Subsequence)\"\"\"\n ref_tokens = tokenize(reference)\n cand_tokens = tokenize(candidate)\n\n if not ref_tokens or not cand_tokens:\n return 0.0\n\n # LCS using dynamic programming\n m, n = len(ref_tokens), len(cand_tokens)\n dp = [[0] * (n + 1) for _ in range(m + 1)]\n\n for i in range(1, m + 1):\n for j in range(1, n + 1):\n if ref_tokens[i-1] == cand_tokens[j-1]:\n dp[i][j] = dp[i-1][j-1] + 1\n else:\n dp[i][j] = max(dp[i-1][j], dp[i][j-1])\n\n lcs_length = dp[m][n]\n\n # F1-like score\n precision = lcs_length / n if n > 0 else 0\n recall = lcs_length / m if m > 0 else 0\n\n if precision + recall == 0:\n return 0.0\n\n return 2 * precision * recall / (precision + recall)\n\n\ndef evaluate_context_relevance(question: str, context: str, context_id: str = \"\") -> ContextEvaluation:\n \"\"\"Evaluate how relevant a context is to a question\"\"\"\n question_terms = set(extract_key_terms(question, 15))\n context_terms = set(extract_key_terms(context, 30))\n\n covered = question_terms & context_terms\n missing = question_terms - context_terms\n\n # Calculate relevance based on term coverage and overlap\n term_coverage = len(covered) / len(question_terms) if question_terms else 0\n token_overlap = calculate_token_overlap(question, context)\n\n # Combined relevance score\n relevance = 0.6 * term_coverage + 0.4 * token_overlap\n\n return ContextEvaluation(\n context_id=context_id,\n relevance_score=round(relevance, 3),\n token_overlap=round(token_overlap, 3),\n key_terms_covered=list(covered),\n missing_terms=list(missing)\n )\n\n\ndef extract_claims(answer: str) -> List[str]:\n \"\"\"Extract individual claims from an answer\"\"\"\n # Split on sentence boundaries\n sentences = re.split(r'[.!?]+', answer)\n claims = []\n\n for sentence in sentences:\n sentence = sentence.strip()\n if len(sentence) > 10: # Filter out very short fragments\n claims.append(sentence)\n\n return claims\n\n\ndef check_claim_support(claim: str, context: str) -> Tuple[bool, float]:\n \"\"\"Check if a claim is supported by the context\"\"\"\n claim_terms = set(tokenize(claim))\n context_terms = set(tokenize(context))\n\n if not claim_terms:\n return True, 1.0 # Empty claim is \"supported\"\n\n # Check term overlap\n overlap = claim_terms & context_terms\n support_ratio = len(overlap) / len(claim_terms)\n\n # Also check for ROUGE-L style matching\n rouge_score = calculate_rouge_l(context, claim)\n\n # Combined support score\n support_score = 0.5 * support_ratio + 0.5 * rouge_score\n\n return support_score > 0.3, support_score\n\n\ndef evaluate_answer_faithfulness(\n question: str,\n answer: str,\n contexts: List[str],\n question_id: str = \"\"\n) -> AnswerEvaluation:\n \"\"\"Evaluate if answer is faithful to the provided contexts\"\"\"\n claims = extract_claims(answer)\n combined_context = ' '.join(contexts)\n\n claim_evaluations = []\n supported_claims = 0\n unsupported = []\n context_used = []\n\n for claim in claims:\n is_supported, score = check_claim_support(claim, combined_context)\n\n claim_eval = {\n 'claim': claim[:100] + '...' if len(claim) > 100 else claim,\n 'supported': is_supported,\n 'score': round(score, 3)\n }\n\n # Track which contexts support this claim\n for i, ctx in enumerate(contexts):\n _, ctx_score = check_claim_support(claim, ctx)\n if ctx_score > 0.3:\n claim_eval[f'context_{i}'] = round(ctx_score, 3)\n if f'context_{i}' not in context_used:\n context_used.append(f'context_{i}')\n\n claim_evaluations.append(claim_eval)\n\n if is_supported:\n supported_claims += 1\n else:\n unsupported.append(claim[:100])\n\n # Faithfulness = % of claims supported\n faithfulness = supported_claims / len(claims) if claims else 1.0\n\n # Groundedness = average support score\n avg_score = sum(c['score'] for c in claim_evaluations) / len(claim_evaluations) if claim_evaluations else 1.0\n\n return AnswerEvaluation(\n question_id=question_id,\n faithfulness_score=round(faithfulness, 3),\n groundedness_score=round(avg_score, 3),\n claims=claim_evaluations,\n unsupported_claims=unsupported,\n context_used=context_used\n )\n\n\ndef calculate_retrieval_metrics(\n retrieved: List[str],\n relevant: Set[str],\n k: int = 5\n) -> RetrievalMetrics:\n \"\"\"Calculate standard retrieval metrics\"\"\"\n retrieved_k = retrieved[:k]\n\n # Precision@K\n relevant_in_k = sum(1 for doc in retrieved_k if doc in relevant)\n precision = relevant_in_k / k if k > 0 else 0\n\n # Recall@K\n recall = relevant_in_k / len(relevant) if relevant else 0\n\n # MRR (Mean Reciprocal Rank)\n mrr = 0.0\n for i, doc in enumerate(retrieved):\n if doc in relevant:\n mrr = 1.0 / (i + 1)\n break\n\n # NDCG@K\n dcg = 0.0\n for i, doc in enumerate(retrieved_k):\n rel = 1 if doc in relevant else 0\n dcg += rel / math.log2(i + 2)\n\n # Ideal DCG (all relevant at top)\n idcg = sum(1 / math.log2(i + 2) for i in range(min(len(relevant), k)))\n ndcg = dcg / idcg if idcg > 0 else 0\n\n return RetrievalMetrics(\n precision_at_k=round(precision, 3),\n recall_at_k=round(recall, 3),\n mrr=round(mrr, 3),\n ndcg_at_k=round(ndcg, 3),\n k=k\n )\n\n\ndef generate_recommendations(report: RAGEvaluationReport) -> List[str]:\n \"\"\"Generate actionable recommendations based on evaluation\"\"\"\n recommendations = []\n\n if report.avg_context_relevance \u003c 0.8:\n recommendations.append(\n f\"Context relevance ({report.avg_context_relevance:.2f}) is below target (0.80). \"\n \"Consider: improving chunking strategy, adding metadata filtering, or using hybrid search.\"\n )\n\n if report.avg_faithfulness \u003c 0.95:\n recommendations.append(\n f\"Faithfulness ({report.avg_faithfulness:.2f}) is below target (0.95). \"\n \"Consider: adding source citations, implementing fact-checking, or adjusting temperature.\"\n )\n\n if report.avg_groundedness \u003c 0.85:\n recommendations.append(\n f\"Groundedness ({report.avg_groundedness:.2f}) is below target (0.85). \"\n \"Consider: using more restrictive prompts, adding 'only use provided context' instructions.\"\n )\n\n if report.coverage \u003c 0.9:\n recommendations.append(\n f\"Coverage ({report.coverage:.2f}) indicates some questions lack relevant context. \"\n \"Consider: expanding document corpus, improving embedding model, or adding fallback responses.\"\n )\n\n retrieval = report.retrieval_metrics\n if retrieval.get('precision_at_k', 0) \u003c 0.7:\n recommendations.append(\n \"Retrieval precision is low. Consider: re-ranking retrieved documents, \"\n \"using cross-encoder for reranking, or adjusting similarity threshold.\"\n )\n\n if not recommendations:\n recommendations.append(\"All metrics meet targets. Consider A/B testing new improvements.\")\n\n return recommendations\n\n\ndef evaluate_rag_system(\n questions: List[Dict],\n contexts: List[Dict],\n k: int = 5,\n verbose: bool = False\n) -> RAGEvaluationReport:\n \"\"\"Comprehensive RAG system evaluation\"\"\"\n\n all_context_scores = []\n all_faithfulness_scores = []\n all_groundedness_scores = []\n issues = []\n question_details = []\n\n questions_with_context = 0\n\n for q_data in questions:\n question = q_data.get('question', q_data.get('query', ''))\n question_id = q_data.get('id', str(questions.index(q_data)))\n answer = q_data.get('answer', q_data.get('response', ''))\n expected = q_data.get('expected', q_data.get('ground_truth', ''))\n\n # Find contexts for this question\n q_contexts = []\n for ctx in contexts:\n if ctx.get('question_id') == question_id or ctx.get('query_id') == question_id:\n q_contexts.append(ctx.get('content', ctx.get('text', '')))\n\n # If no specific contexts, use all contexts (for simple datasets)\n if not q_contexts:\n q_contexts = [ctx.get('content', ctx.get('text', ''))\n for ctx in contexts[:k]]\n\n if q_contexts:\n questions_with_context += 1\n\n # Evaluate context relevance\n context_evals = []\n for i, ctx in enumerate(q_contexts[:k]):\n eval_result = evaluate_context_relevance(question, ctx, f\"ctx_{i}\")\n context_evals.append(eval_result)\n all_context_scores.append(eval_result.relevance_score)\n\n # Evaluate answer faithfulness\n if answer and q_contexts:\n answer_eval = evaluate_answer_faithfulness(question, answer, q_contexts, question_id)\n all_faithfulness_scores.append(answer_eval.faithfulness_score)\n all_groundedness_scores.append(answer_eval.groundedness_score)\n\n # Track issues\n if answer_eval.unsupported_claims:\n issues.append({\n 'type': 'unsupported_claim',\n 'question_id': question_id,\n 'claims': answer_eval.unsupported_claims[:3]\n })\n\n # Check for low relevance contexts\n low_relevance = [e for e in context_evals if e.relevance_score \u003c 0.5]\n if low_relevance:\n issues.append({\n 'type': 'low_relevance',\n 'question_id': question_id,\n 'contexts': [e.context_id for e in low_relevance]\n })\n\n if verbose:\n question_details.append({\n 'question_id': question_id,\n 'question': question[:100],\n 'context_scores': [asdict(e) for e in context_evals],\n 'answer_faithfulness': all_faithfulness_scores[-1] if all_faithfulness_scores else None\n })\n\n # Calculate aggregates\n avg_context_relevance = sum(all_context_scores) / len(all_context_scores) if all_context_scores else 0\n avg_faithfulness = sum(all_faithfulness_scores) / len(all_faithfulness_scores) if all_faithfulness_scores else 0\n avg_groundedness = sum(all_groundedness_scores) / len(all_groundedness_scores) if all_groundedness_scores else 0\n coverage = questions_with_context / len(questions) if questions else 0\n\n # Simulated retrieval metrics (based on relevance scores)\n high_relevance = sum(1 for s in all_context_scores if s > 0.5)\n retrieval_metrics = {\n 'precision_at_k': round(high_relevance / len(all_context_scores) if all_context_scores else 0, 3),\n 'estimated_recall': round(coverage, 3),\n 'k': k\n }\n\n report = RAGEvaluationReport(\n total_questions=len(questions),\n avg_context_relevance=round(avg_context_relevance, 3),\n avg_faithfulness=round(avg_faithfulness, 3),\n avg_groundedness=round(avg_groundedness, 3),\n retrieval_metrics=retrieval_metrics,\n coverage=round(coverage, 3),\n issues=issues[:20], # Limit to 20 issues\n recommendations=[],\n question_details=question_details if verbose else []\n )\n\n report.recommendations = generate_recommendations(report)\n\n return report\n\n\ndef format_report(report: RAGEvaluationReport) -> str:\n \"\"\"Format report as human-readable text\"\"\"\n lines = []\n lines.append(\"=\" * 60)\n lines.append(\"RAG EVALUATION REPORT\")\n lines.append(\"=\" * 60)\n lines.append(\"\")\n\n lines.append(f\"📊 SUMMARY\")\n lines.append(f\" Questions evaluated: {report.total_questions}\")\n lines.append(f\" Coverage: {report.coverage:.1%}\")\n lines.append(\"\")\n\n lines.append(\"📈 RETRIEVAL METRICS\")\n lines.append(f\" Context Relevance: {report.avg_context_relevance:.2f} {'✅' if report.avg_context_relevance >= 0.8 else '⚠️'} (target: >0.80)\")\n lines.append(f\" Precision@{report.retrieval_metrics.get('k', 5)}: {report.retrieval_metrics.get('precision_at_k', 0):.2f}\")\n lines.append(\"\")\n\n lines.append(\"📝 GENERATION METRICS\")\n lines.append(f\" Answer Faithfulness: {report.avg_faithfulness:.2f} {'✅' if report.avg_faithfulness >= 0.95 else '⚠️'} (target: >0.95)\")\n lines.append(f\" Groundedness: {report.avg_groundedness:.2f} {'✅' if report.avg_groundedness >= 0.85 else '⚠️'} (target: >0.85)\")\n lines.append(\"\")\n\n if report.issues:\n lines.append(f\"⚠️ ISSUES FOUND ({len(report.issues)})\")\n for issue in report.issues[:10]:\n if issue['type'] == 'unsupported_claim':\n lines.append(f\" Q{issue['question_id']}: {len(issue.get('claims', []))} unsupported claim(s)\")\n elif issue['type'] == 'low_relevance':\n lines.append(f\" Q{issue['question_id']}: Low relevance contexts: {issue.get('contexts', [])}\")\n if len(report.issues) > 10:\n lines.append(f\" ... and {len(report.issues) - 10} more issues\")\n lines.append(\"\")\n\n lines.append(\"💡 RECOMMENDATIONS\")\n for i, rec in enumerate(report.recommendations, 1):\n lines.append(f\" {i}. {rec}\")\n lines.append(\"\")\n\n lines.append(\"=\" * 60)\n\n return '\\n'.join(lines)\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"RAG Evaluator - Evaluate Retrieval-Augmented Generation systems\",\n formatter_class=argparse.RawDescriptionHelpFormatter,\n epilog=\"\"\"\nExamples:\n %(prog)s --contexts contexts.json --questions questions.json\n %(prog)s --contexts ctx.json --questions q.json --k 10\n %(prog)s --contexts ctx.json --questions q.json --output report.json --verbose\n\nInput file formats:\n\nquestions.json:\n[\n {\"id\": \"q1\", \"question\": \"What is X?\", \"answer\": \"X is...\"},\n {\"id\": \"q2\", \"question\": \"How does Y work?\", \"answer\": \"Y works by...\"}\n]\n\ncontexts.json:\n[\n {\"question_id\": \"q1\", \"content\": \"Retrieved context text...\"},\n {\"question_id\": \"q2\", \"content\": \"Another context...\"}\n]\n \"\"\"\n )\n\n parser.add_argument('--contexts', '-c', required=True, help='JSON file with retrieved contexts')\n parser.add_argument('--questions', '-q', required=True, help='JSON file with questions and answers')\n parser.add_argument('--k', type=int, default=5, help='Number of top contexts to evaluate (default: 5)')\n parser.add_argument('--output', '-o', help='Output file for detailed report (JSON)')\n parser.add_argument('--json', '-j', action='store_true', help='Output as JSON instead of text')\n parser.add_argument('--verbose', '-v', action='store_true', help='Include per-question details')\n parser.add_argument('--compare', help='Compare with baseline report JSON')\n\n args = parser.parse_args()\n\n # Load input files\n contexts_path = Path(args.contexts)\n questions_path = Path(args.questions)\n\n if not contexts_path.exists():\n print(f\"Error: Contexts file not found: {args.contexts}\", file=sys.stderr)\n sys.exit(1)\n\n if not questions_path.exists():\n print(f\"Error: Questions file not found: {args.questions}\", file=sys.stderr)\n sys.exit(1)\n\n try:\n contexts = json.loads(contexts_path.read_text(encoding='utf-8'))\n questions = json.loads(questions_path.read_text(encoding='utf-8'))\n except json.JSONDecodeError as e:\n print(f\"Error: Invalid JSON format: {e}\", file=sys.stderr)\n sys.exit(1)\n\n # Run evaluation\n report = evaluate_rag_system(questions, contexts, k=args.k, verbose=args.verbose)\n\n # Compare with baseline\n if args.compare:\n baseline_path = Path(args.compare)\n if baseline_path.exists():\n baseline = json.loads(baseline_path.read_text())\n print(\"\\n📊 COMPARISON WITH BASELINE\")\n print(f\" Relevance: {baseline.get('avg_context_relevance', 0):.2f} -> {report.avg_context_relevance:.2f}\")\n print(f\" Faithfulness: {baseline.get('avg_faithfulness', 0):.2f} -> {report.avg_faithfulness:.2f}\")\n print(f\" Groundedness: {baseline.get('avg_groundedness', 0):.2f} -> {report.avg_groundedness:.2f}\")\n print()\n\n # Output\n if args.json:\n print(json.dumps(asdict(report), indent=2))\n else:\n print(format_report(report))\n\n # Save to file\n if args.output:\n Path(args.output).write_text(json.dumps(asdict(report), indent=2))\n print(f\"\\nDetailed report saved to {args.output}\")\n\n\nif __name__ == '__main__':\n main()\n","content_type":"text/x-python; charset=utf-8","language":"python","size":20362,"content_sha256":"bf4593989f2b805221d421d5f4f3dfe843a90d5bbad997a05e28777469e87089"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"Senior Prompt Engineer","type":"text"}]},{"type":"paragraph","content":[{"text":"Eval-driven prompt engineering, RAG quality measurement, and agent workflow validation. Everything here is ","type":"text"},{"text":"model-agnostic by design","type":"text","marks":[{"type":"strong"}]},{"text":": techniques are framed by what they do, not by which model generation they were observed on, and the tools never hardcode model IDs or pricing — you supply your provider's current rates when you want dollar figures.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Operating Rules","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Never change a prompt without a baseline.","type":"text","marks":[{"type":"strong"}]},{"text":" Capture metrics first (","type":"text"},{"text":"--analyze --output baseline.json","type":"text","marks":[{"type":"code_inline"}]},{"text":"), then compare every iteration against it.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Eval set before optimization.","type":"text","marks":[{"type":"strong"}]},{"text":" 10–20 representative cases with expected outputs minimum. If the user has no eval set, build one with them before touching the prompt — optimizing against vibes is the #1 failure mode.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Prefer platform features over prompt hacks.","type":"text","marks":[{"type":"strong"}]},{"text":" If the provider offers native structured outputs / JSON schema enforcement, tool-use APIs, or prompt caching, use those instead of \"respond ONLY with JSON\" incantations. Prompt-level format enforcement is the fallback, not the default.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Current-generation models need less scaffolding.","type":"text","marks":[{"type":"strong"}]},{"text":" Don't add chain-of-thought boilerplate, role framing, or few-shot examples reflexively — frontier models often do worse with redundant scaffolding. Add each element only when the eval set shows it helps.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Cost numbers are always user-supplied.","type":"text","marks":[{"type":"strong"}]},{"text":" Look up the provider's current per-Mtok pricing and pass it via ","type":"text"},{"text":"--price-per-mtok","type":"text","marks":[{"type":"code_inline"}]},{"text":" (never trust a cached price table — including any you remember).","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Tools (exact CLIs, all stdlib)","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"1. Prompt Optimizer — ","type":"text"},{"text":"scripts/prompt_optimizer.py","type":"text","marks":[{"type":"code_inline"}]}]},{"type":"paragraph","content":[{"text":"Static analysis: token estimate, clarity/structure scores (0–100), ambiguity + redundancy detection, few-shot example extraction.","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Full analysis (human-readable report)\npython3 scripts/prompt_optimizer.py prompt.txt --analyze\n\n# Save machine-readable baseline for later comparison\npython3 scripts/prompt_optimizer.py prompt.txt --analyze --json --output baseline.json\n\n# Token estimate; cost only if you supply your provider's current rate\npython3 scripts/prompt_optimizer.py prompt.txt --tokens --model claude --price-per-mtok 3.00\n\n# Whitespace/redundancy-trimmed version\npython3 scripts/prompt_optimizer.py prompt.txt --optimize --output optimized.txt\n\n# Extract Input/Output few-shot pairs to JSON\npython3 scripts/prompt_optimizer.py prompt.txt --extract-examples --output examples.json\n\n# Compare a revision against the saved baseline\npython3 scripts/prompt_optimizer.py optimized.txt --analyze --compare baseline.json","type":"text"}]},{"type":"paragraph","content":[{"text":"--model","type":"text","marks":[{"type":"code_inline"}]},{"text":" accepts any string; only the tokenizer family is inferred (names containing \"claude\" → 3.5 chars/token, otherwise 4.0). Exit 0 on success, 1 on missing file.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"2. RAG Evaluator — ","type":"text"},{"text":"scripts/rag_evaluator.py","type":"text","marks":[{"type":"code_inline"}]}]},{"type":"paragraph","content":[{"text":"Measures retrieval and grounding quality from two JSON files (formats printed in ","type":"text"},{"text":"--help","type":"text","marks":[{"type":"code_inline"}]},{"text":").","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json\npython3 scripts/rag_evaluator.py --contexts ctx.json --questions q.json --k 10 --json\npython3 scripts/rag_evaluator.py --contexts ctx.json --questions q.json --output report.json --verbose\npython3 scripts/rag_evaluator.py --contexts ctx.json --questions q.json --compare baseline_report.json","type":"text"}]},{"type":"paragraph","content":[{"text":"Reports context relevance, precision@k, coverage, answer faithfulness, groundedness. Treat relevance \u003c 0.80 as a retrieval problem (chunking/embedding/filtering), not a prompt problem — fix retrieval before rewriting the generation prompt.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"3. Agent Orchestrator — ","type":"text"},{"text":"scripts/agent_orchestrator.py","type":"text","marks":[{"type":"code_inline"}]}]},{"type":"paragraph","content":[{"text":"Validates agent configs (YAML/JSON): tool wiring, missing required config, loop risk, token estimates.","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/agent_orchestrator.py agent.yaml --validate\npython3 scripts/agent_orchestrator.py agent.yaml --visualize --format mermaid\npython3 scripts/agent_orchestrator.py agent.yaml --estimate-cost --runs 100 \\\n --input-price-per-mtok 3.00 --output-price-per-mtok 15.00","type":"text"}]},{"type":"paragraph","content":[{"text":"Without the two price flags, ","type":"text"},{"text":"--estimate-cost","type":"text","marks":[{"type":"code_inline"}]},{"text":" reports token estimates only. The ","type":"text"},{"text":"model:","type":"text","marks":[{"type":"code_inline"}]},{"text":" field in the config is informational — any model name is accepted.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Workflows","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Prompt Optimization (eval-gated)","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Baseline:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"python3 scripts/prompt_optimizer.py current_prompt.txt --analyze --json --output baseline.json","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Diagnose","type":"text","marks":[{"type":"strong"}]},{"text":" from the report: ambiguous verbs (\"analyze\", \"handle\"), redundant blocks, missing output contract, token waste.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Apply one change at a time","type":"text","marks":[{"type":"strong"}]},{"text":", in this order of leverage:","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Symptom","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Fix","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Malformed/unparseable output","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Native structured outputs / JSON schema if the API supports it; explicit schema-in-prompt otherwise","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Inconsistent answers across runs","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Tighten instructions + add 2–3 contrastive examples (one near-miss showing what NOT to do)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Misses edge cases","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Enumerate the edge cases explicitly; add a \"when uncertain, do X\" rule","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Token bloat on repeated calls","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Move stable prefix (system rules, examples) first so prompt caching applies; trim redundancy","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Wrong reasoning on hard cases","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Ask for stepwise reasoning ","type":"text"},{"text":"in a scratch field the consumer ignores","type":"text","marks":[{"type":"em"}]},{"text":", or use the provider's extended-thinking mode","type":"text"}]}]}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Re-analyze and compare:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"python3 scripts/prompt_optimizer.py revised.txt --analyze --compare baseline.json","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Eval gate (must pass before shipping):","type":"text","marks":[{"type":"strong"}]},{"text":" run the revised prompt over the eval set, write per-case pass/fail to ","type":"text"},{"text":"eval_results.json","type":"text","marks":[{"type":"code_inline"}]},{"text":", then assert:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/prompt_optimizer.py revised.txt --analyze --json --output revised.json \\\n && python3 -c \"\nimport json, sys\nr = json.load(open('revised.json')); b = json.load(open('baseline.json'))\nok = r['clarity_score'] >= b['clarity_score'] and r['token_count'] \u003c= b['token_count'] * 1.10\nsys.exit(0 if ok else 1)\"\necho \"gate exit=$?\" # 0 = ship; 1 = regression, iterate again","type":"text"}]},{"type":"paragraph","content":[{"text":"Pair this structural gate with your task-level eval: the revision must not lose any previously-passing eval case (no-regression rule).","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Few-Shot Example Design","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Define the task contract first (input shape, output shape, edge-case policy).","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Start with ","type":"text"},{"text":"zero examples","type":"text","marks":[{"type":"strong"}]},{"text":" and measure — current models often need none. Add examples only for failure clusters the eval reveals.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"When adding: 3–5 max, ordered simple → edge → negative (what NOT to extract), formatted identically to the real output contract.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Validate consistency: ","type":"text"},{"text":"python3 scripts/prompt_optimizer.py prompt_with_examples.txt --extract-examples --output examples.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" and inspect that every extracted pair parses against your schema.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Re-run the eval set; if a case passes only because it resembles an example, add a held-out variant to the eval set.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Structured Output Design","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Write the JSON Schema first (types, enums, required, maxLength).","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Prefer API-native enforcement","type":"text","marks":[{"type":"strong"}]},{"text":": structured-outputs / response-schema / tool-call parameters guarantee shape; prompt text cannot.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Fallback (API without schema support): include the schema rendered as field-by-field rules + one valid example, and instruct \"output only the JSON object\".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Gate: pipe 10 eval outputs through a schema validator (","type":"text"},{"text":"python3 -c \"import json,sys; [json.loads(l) for l in sys.stdin]\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" at minimum); 10/10 must parse, else return to step 2.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"RAG Tuning Loop","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Build ","type":"text"},{"text":"questions.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" (id, question, reference answer) and capture current retrievals to ","type":"text"},{"text":"contexts.json","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"python3 scripts/rag_evaluator.py --contexts contexts.json --questions questions.json --output rag_baseline.json","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Fix the ","type":"text"},{"text":"lowest metric first","type":"text","marks":[{"type":"strong"}]},{"text":": relevance → chunking/embeddings/metadata filters; faithfulness → grounding instructions + \"answer only from context\" + citation requirement; coverage → retrieval k / query expansion.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Gate: ","type":"text"},{"text":"python3 scripts/rag_evaluator.py --contexts new_contexts.json --questions questions.json --compare rag_baseline.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" — every metric must be ≥ baseline; any regression blocks the change.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Agent Config Review","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"python3 scripts/agent_orchestrator.py agent.yaml --validate","type":"text","marks":[{"type":"code_inline"}]},{"text":" — must exit with VALIDATION PASSED; fix every error and warning (missing tool config, unbounded iterations, loop risk).","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Check context discipline: each tool description ≤ 1–2 sentences, tool count minimal for the job, stable system prompt placed first (cache-friendly), iteration cap + early-exit condition present.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Budget: ","type":"text"},{"text":"--estimate-cost --runs N","type":"text","marks":[{"type":"code_inline"}]},{"text":" with your current prices; if cost/run exceeds budget, cut tools or context before downgrading the model.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"References","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"File","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Contains","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Load when user asks about","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/prompt_engineering_patterns.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"10 prompt patterns with input/output examples","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"\"which pattern?\", few-shot design, decomposition, meta-prompting","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/llm_evaluation_frameworks.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Eval metrics, scoring methods, A/B testing","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"\"how to evaluate?\", \"measure quality\", \"compare prompts\"","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/agentic_system_design.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Agent architectures (ReAct, Plan-Execute, Tool Use)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"\"build agent\", \"tool calling\", \"multi-agent\"","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Related Skills","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"engineering-team/skills/senior-ml-engineer","type":"text","marks":[{"type":"code_inline"}]},{"text":" — model deployment and serving (this skill stops at the prompt/eval layer)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"engineering/rag-architect","type":"text","marks":[{"type":"code_inline"}]},{"text":" — RAG system architecture (this skill measures RAG quality; that one designs the pipeline)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"engineering/agent-designer","type":"text","marks":[{"type":"code_inline"}]},{"text":" — full agent system design (this skill validates configs; that one designs the architecture)","type":"text"}]}]}]}]},"metadata":{"date":"2026-11-02","name":"senior-prompt-engineer","author":"@skillopedia","source":{"stars":29164,"repo_name":"claude-skills","origin_url":"https://github.com/alirezarezvani/claude-skills/blob/HEAD/engineering-team/skills/senior-prompt-engineer/SKILL.md","repo_owner":"alirezarezvani","body_sha256":"9716b4a303b58062eebf6773e0da1f05686b5c72bfadba89c7d5112d6f49c8cb","cluster_key":"ede161bb478d99db20a43dcd3a048ba31c7f1934cb6602a9d1234249c6e05e5b","clean_bundle":{"format":"clean-skill-bundle-v1","source":"alirezarezvani/claude-skills/engineering-team/skills/senior-prompt-engineer/SKILL.md","text_bytes":109529,"source_commit":"9ef0c3c213932e645cc5cbeeb3c35c3a762a2437","skills_sh_rank":13305,"attachment_count":6,"text_attachments":6,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"engineering-team/skills/senior-prompt-engineer/SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"v1","category":"ai-agent-development","category_label":"AI"},"exact_dupes_collapsed_into_this":0},"version":"v1","category":"ai-agent-development","import_tag":"skills-sh-top20k-2026-11","description":"Use when optimizing a prompt, building an eval set, measuring RAG retrieval quality or validating an agent's tool config. Iterates with prompt_optimizer.py, scores RAG with rag_evaluator.py and checks agents with agent_orchestrator.py."}},"renderedAt":1793666470160}

Senior Prompt Engineer Eval-driven prompt engineering, RAG quality measurement, and agent workflow validation. Everything here is model-agnostic by design : techniques are framed by what they do, not by which model generation they were observed on, and the tools never hardcode model IDs or pricing — you supply your provider's current rates when you want dollar figures. Operating Rules 1. Never change a prompt without a baseline. Capture metrics first ( ), then compare every iteration against it. 2. Eval set before optimization. 10–20 representative cases with expected outputs minimum. If the…