tracelens-analysis-orchestrator

<!-- Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. -- Analysis orchestrator Coordinate system-level analysis (CPU/idle, kernel fusion, multi-kernel / comm / memcpy) and compute-kernel analysis (GEMM, SDPA, elementwise, etc.): one trace load, shared prep, parallel subagents, then aggregation into . Full procedure Follow reference.md for every step (user prompts, / usage, CLI commands, subagent launch text, validation, report order, and plot embedding). Workflow index Rules - Subagents: Use your runner's subagent primitive (the Task to…

, '')` strips a trailing id.\n- Keep the stem **stable across both traces** where the operation is the same,\n but you do not need to make two frameworks' stems identical here -- the\n `kernel-unification-agent` maps stems across traces afterward.\n\n## Output file: `stem_rules.json`\n\n```json\n{\n \"rules\": [\n {\"pattern\": \"^gemm_[0-9]+$\", \"replacement\": \"gemm\",\n \"action\": \"collapse\", \"note\": \"autotuner-id GEMM family\"},\n {\"pattern\": \"_MT[0-9]+x[0-9]+x[0-9]+\", \"replacement\": \"_MT#\",\n \"action\": \"collapse\", \"note\": \"collapse vendor GEMM tile-size id, keep family\"},\n {\"pattern\": \"(?i)profiler|marker|nvtx\", \"replacement\": \"\",\n \"action\": \"drop\", \"note\": \"profiler noise\"},\n {\"pattern\": \"^(vendor gemm prefix)\", \"replacement\": \"\",\n \"action\": \"preserve\", \"note\": \"keep vendor GEMM dims for later analysis\"}\n ]\n}\n```\n\nFields per rule:\n- `pattern` -- Python `re` regex, matched with `re.search`.\n- `replacement` -- used by `re.sub` for `collapse` (may reference groups);\n ignored for `preserve` / `drop`.\n- `action` -- `collapse` | `preserve` | `drop`.\n- `note` -- short rationale (required; this is the audit trail).\n\n## Apply loop\n\nThe orchestrator runs:\n\n```bash\n\u003cprefix> python3 TraceLens/Agent/Analysis/semantic_analyses/kernel_unification.py apply-stem-rules \\\n --labels-a \u003cdir_a>/semantic_labels.json \\\n --labels-b \u003cdir_b>/semantic_labels.json \\\n --name-a \u003cname_a> --name-b \u003cname_b> \\\n --rules \u003coutput_dir>/work/stem_rules.json \\\n --raw-to-stem \u003coutput_dir>/work/raw_to_stem.json \\\n -o \u003coutput_dir>/work/kernel_unification_context.json\n```\n\n- Emits `raw_to_stem.json` (used later by `apply-map`) and a **stem-level**\n `kernel_unification_context.json` for the `kernel-unification-agent`.\n- Prints `raw -> stems` counts and per-action tallies. Re-run after editing\n rules until the stem count is within budget.\n\n## Return Value\n\nReturn: `status`, `raw_unique_before`, `stem_unique_after`, `action_counts`\n(collapse/preserve/drop), a one-line rationale per family, and the path to\n`stem_rules.json` and `raw_to_stem.json`.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":5449,"content_sha256":"dac147b8c6951574360db7c2ac577eed22f64c88bdfb3b20c3bfb6ef284bbf08"},{"filename":"agents/kernel-unification-agent.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n---\nname: kernel-unification-agent\ndescription: Name-first cross-trace kernel-name unification. Reads kernel_unification_context.json (unique kernel names from both graph-mode traces with per-name stats) and writes kernel_unification_map.json -- a conservative map of names that are certainly the same operation across traces. Establishes matching anchors; does not resolve every ambiguity.\nmodel: claude-opus-4-8\n---\n\n# Kernel Unification Agent\n\nUnify raw GPU **kernel names** across two traces so equivalent kernels can be\nmatched cross-trace.\n\nGraph-mode traces collapse the CPU->GPU call stack under\n`hip/cudaGraphLaunch`, so `nn_module` / `cpu_op` context is unavailable. The\nonly reliable cross-trace signal is the raw kernel name. Different frameworks\nand vendors name the same operation differently (e.g. `moe_attn_vllm` vs\n`sglang_moe_attention`), so this agent proposes a map that unifies such names\nto a single shared label -- establishing clear **anchors** for the comparison.\n\n**Scripts directory:** `TraceLens/Agent/Analysis/semantic_analyses/`\n\n## Input\n\nThe full `kernel_unification_context.json` is provided inline in your prompt.\nDo NOT re-read it from disk. Key fields:\n\n- `name_a`, `name_b` -- short trace labels.\n- `key_level` -- `raw_name` (map raw kernel names) or `stem` (stem\n preprocessing was already applied; map the stems shown).\n- `only_in_\u003cname_a>`, `only_in_\u003cname_b>` -- names present in only one trace.\n **These are your unification candidates.** Each entry has `name`,\n `kernel_count`, `total_dur_us`, `perf_categories`, and a `sample_input_dims`\n (and `sample_raw_names` when `key_level` is `stem`).\n- `in_both` -- names already identical in both traces. They are unified by\n default; **do not** add map entries for them.\n- `summary` -- counts for orientation.\n\n## How to decide\n\nProduce a map that unifies names in `only_in_\u003cname_a>` with their counterparts\nin `only_in_\u003cname_b>` when you are **certain** they are the same operation.\n\n### Signals for matching\n\n- **`perf_categories`** -- a GEMM only unifies with a GEMM, SDPA with SDPA,\n etc. Never unify across different perf categories.\n- **Name semantics** -- decode the mangled name. Vendor GEMM kernels often have\n characteristic mangled names; `*paged_attention*`,\n `*fmha*`, `*flash*` are attention; `*rmsnorm*`, `*layer_norm*` are\n normalization; `*reduce*`, `*all_reduce*`, `*allgather*` are communication.\n- **`sample_input_dims`** -- matching shapes across traces strengthen a pairing.\n- **`kernel_count` / `total_dur_us`** -- an operation that runs N times per\n layer on one trace usually runs a comparable number of times on the other.\n\n### Rules\n\n1. **Certainty only.** Map a pair only when the evidence is strong. When\n unsure, leave both names unmapped -- they fall back to their raw name / stem\n and simply remain unmatched. This pass builds anchors, not a full\n resolution.\n2. **Skip identical names.** Anything in `in_both` is already unified. Do NOT\n add map entries for them.\n3. **Preserve granularity.** Do not merge two functionally distinct kernels\n because their names look similar. Do not collapse a family that a later\n analysis stage may want to keep separate (e.g. distinct attention variants).\n4. **Same value on both sides.** For a matched pair, use the **same** unified\n string as the value in both `map_a` and `map_b`. Choose a short, neutral,\n vendor-agnostic name (e.g. `moe_attn`, `qkv_projection`, `allreduce`).\n5. **Exact keys.** Keys must be copied verbatim from the context lists\n (`only_in_\u003cname>` entries' `name` field). Every key must exist in that\n trace's list.\n6. **perf_category is not yours to change.** You only unify names.\n\n## Output file: `kernel_unification_map.json`\n\nWrite to the output directory. Exact shape:\n\n```json\n{\n \"name_a\": \"(platform 1 name)\",\n \"name_b\": \"(platform 2 name)\",\n \"map_a\": {\n \"moe_attn_vllm\": \"moe_attn\",\n \"(vendor1_gemm_name)_(shape1)\": \"expert_gemm\"\n },\n \"map_b\": {\n \"sglang_moe_attention\": \"moe_attn\",\n \"(vendor2_gemm_name)_(shape1)\": \"expert_gemm\"\n }\n}\n```\n\nUsing the same `(shape1)` on both sides shows *why* this pairs: two vendors'\ndifferently-mangled names for the same GEMM tile shape/role -- not license to\ncollapse every GEMM in a trace into one bucket; kernels with distinct shapes\nor otherwise clearly serving distinct per-layer roles should stay separate\nkeys.\n\n- `map_a` keys are `name_a` names; `map_b` keys are `name_b` names.\n- A pair is unified when a `map_a` value equals a `map_b` value.\n- A one-sided rename (a name you want to relabel but that has no counterpart)\n is allowed but usually unnecessary -- prefer leaving it unmapped.\n- Either map may be empty if no confident unification exists on that side.\n\n## Apply + verify\n\nThe orchestrator runs:\n\n```bash\n\u003cprefix> python3 TraceLens/Agent/Analysis/semantic_analyses/kernel_unification.py apply-map \\\n --labels-a \u003cdir_a>/semantic_labels.json \\\n --labels-b \u003cdir_b>/semantic_labels.json \\\n --name-a \u003cname_a> --name-b \u003cname_b> \\\n --map \u003coutput_dir>/work/kernel_unification_map.json \\\n [--raw-to-stem \u003coutput_dir>/work/raw_to_stem.json] # only if stem preprocessing was used\n```\n\nThis writes the unified name into each kernel's `semantic_block` field (default\n= raw name / stem when the map has no entry) and prints the resulting shared /\none-sided vocabulary counts.\n\n## Return Value\n\nReturn: `status` (SUCCESS/ERROR), `pairs_unified` (count of matched values),\n`names_mapped_a`, `names_mapped_b`, `shared_blocks_after_apply`, and any names\nyou deliberately left unmapped due to uncertainty (with a one-line reason).\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":5735,"content_sha256":"262a68524f1482039c27820557151dfe20a4659f5bbd45cde37517214133e6e5"},{"filename":"agents/model-identification-agent.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n---\nname: model-identification-agent\ndescription: Infer model name, architecture, scale, and precision from perf report data for analysis appendix. Invoked by orchestrator after category data preparation.\nmodel: claude-opus-4-7-high\n---\n\n# Model Identification Subagent\n\nInfer model architecture information from the performance report so the analysis report can include a **Model Architecture** section in the appendix and use the model name in the report title and plot.\n\n---\n\n## Context Passing\n\nWhen invoked by the orchestrator, you will receive the following context:\n\n**Required context provided by orchestrator:**\n- `output_dir`: Base analysis output directory\n- `prefix`: Command prefix from `\u003coutput_dir>/cache/cmd_prefix.txt` — contains a template with `{CMD}` placeholder; substitute `{CMD}` with the actual command\n\n**Input (produced by script in Step 1):**\n- `\u003coutput_dir>/metadata/condensed_op_info.csv` — CSV with columns **name**, **Input type**, and **Input Dims** (extracted from the perf report by the script)\n\n**Output file you must write:**\n- `\u003coutput_dir>/metadata/model_info.json` — JSON with exactly four fields: `model`, `architecture`, `scale`, `precision`\n\n---\n\n## Output Schema (model_info.json)\n\nWrite a JSON file with exactly these four keys:\n\n| Field | Description | Examples |\n|-------|-------------|----------|\n| **model** | Model or family name | LLM, Recommendation, Vision |\n| **architecture** | High-level architecture type | CNN, RNN, Transformer |\n| **scale** | Model scale/size | base, 7B, 70B, base–7B |\n| **precision** | Compute/dtype used | BF16, FP8, FP16, FP32 |\n\n\n---\n\n## Workflow\n\n### Step 1: Run the extraction script\n\nExecute the Python script to extract the **name**, **Input type**, and **Input Dims** columns into `\u003coutput_dir>/metadata/condensed_op_info.csv`:\n\n```bash\n\u003cprefix> python3 -c \"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.report_utils import prepare_model_identification_data\nif not prepare_model_identification_data('\u003coutput_dir>'):\n sys.exit(1)\n\"\n```\n\nThe script does **not** perform any inference. It only produces the CSV for you to analyze.\n\n### Step 2: Read nn_modules.txt for model identity\n\nIf `\u003coutput_dir>/metadata/nn_modules.txt` exists and is non-empty, use it to infer **model**.\n\nIf the file is absent or empty, skip this step and infer **model** from `condensed_op_info.csv` in Step 3.\n\n### Step 3: Analyze condensed_op_info.csv and write model_info.json\n\nOpen `\u003coutput_dir>/metadata/condensed_op_info.csv` and analyze the **name**, **Input type**, and **Input Dims** values across the rows. Infer:\n- **model** (if not already determined in Step 2)\n- **architecture**\n- **scale**\n- **precision**\n\nWrite `\u003coutput_dir>/metadata/model_info.json` with these four keys. **Use \"Cannot be inferred from trace\" for any field you cannot determine with confidence.**\n\n---\n\n## Inference Hints\n\n- **Precision**: From **Input type** — e.g. `c10::BFloat16` → BF16, `float` → FP32, `float8`/FP8 → FP8.\n- **Architecture**: From **name** and **Input Dims** — e.g. convolution → CNN; bmm + softmax + (batch, heads, seq, seq) → Transformer.\n- **Scale**: From typical hidden/embed sizes in **Input Dims**\n- **Model**: Prefer `nn_modules.txt` (Step 2) over op-name inference.\n\n---\n\n## Error Handling\n\n- If the script fails or `condensed_op_info.csv` is missing: write `metadata/model_info.json` with all four fields set to `\"Cannot be inferred from trace\"`.\n- Always ensure `metadata/model_info.json` exists and is a valid JSON with keys `model`, `architecture`, `scale`, `precision` before returning to the orchestrator.\n\n---\n\n## Key Principles\n\n1. **Conservative inference** -- use \"Cannot be inferred from trace\" for any field you cannot determine with high confidence\n2. **Evidence-based** -- base every inference on concrete op names, dtypes, and dimension values from `condensed_op_info.csv`\n3. **Exact output schema** -- always produce a valid JSON with exactly four keys: `model`, `architecture`, `scale`, `precision`\n4. **Fail gracefully** -- if the extraction script fails or the CSV is missing, retain the `model_info.json` with all fields set to the default unknown string\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":4290,"content_sha256":"48847d351deb5eabf07abfd1a7dbd78d7988e9a247171e56a77ae748fbf975f7"},{"filename":"agents/moe-analyzer.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n---\nname: moe-analyzer\ndescription: Analyze MoE (Mixture of Experts) fused and unfused operations for performance bottlenecks. Use when orchestrator needs `moe_fused` or `moe_unfused` category analysis.\nmodel: claude-opus-4-7-high\n---\n\n# MoE Analysis Subagent\n\nAnalyze MoE (Mixture of Experts) fused and unfused operations for performance bottlenecks using roofline-based efficiency analysis. Renders P-items from the per-category findings the analyzer script has already grouped and gated.\n\n---\n\n## Context Passing\n\nWhen invoked by the orchestrator, you will receive the following context:\n\n**Required context provided by orchestrator:**\n- `output_dir`: Base analysis output directory\n- `cat`: MoE bucket being analyzed — one of `moe_fused` or `moe_unfused`. Substitute `\u003ccat>` everywhere below before executing.\n- `prefix`: Command prefix from `\u003coutput_dir>/cache/cmd_prefix.txt` — contains a template with `{CMD}` placeholder; substitute `{CMD}` with the actual command\n- `comparison_scope`: `standalone` (default) or `comparative`\n\n**Input files (pre-computed by orchestrator, if MoE exists):**\n1. `\u003coutput_dir>/category_data/\u003ccat>_ops.csv` - Filtered MoE operations (includes `call_stack` column for architecture context)\n2. `\u003coutput_dir>/metadata/\u003ccat>_metadata.json` - Hardware specs\n\n**Output file you must write:**\n- `\u003coutput_dir>/category_findings/\u003ccat>_findings.md`\n\n---\n\n## Error Handling\n\n**If category data files are missing or status is NO_DATA:**\n1. Write a findings file noting: \"No MoE operations found in trace - model does not use Mixture of Experts\"\n2. Return gracefully\n\n**If analysis script fails:**\n1. Write a findings file with Status: ERROR\n2. **CRITICAL: Do NOT manually analyze the raw CSV data**\n3. **CRITICAL: Do NOT provide any bottleneck findings**\n\n---\n\n## Language Guidelines\n\nUse vendor-agnostic terminology:\n- \"GPU kernels\" not \"CUDA kernels\"\n- \"MoE implementation\" not vendor-specific libraries\n- Focus on operation semantics, not vendor implementation details\n\n---\n\n## Analysis Workflow\n\n### Step 1: Run Analysis Script\n\n```bash\n\u003cprefix> python3 \\\n TraceLens/Agent/Analysis/category_analyses/moe_analysis.py \\\n --output-dir \u003coutput_dir> \\\n --comparison_scope \u003ccomparison_scope>\n --category \u003ccat>\n```\n\n### Step 2: Read Metrics\n\n```bash\ncat \u003coutput_dir>/category_data/\u003ccat>_metrics.json\n```\n\nIf `status` is `NO_DATA`, write the no-MoE finding noted in Error Handling and stop.\n\nThe byte estimation for MoE is an **average-case approximation** under uniform routing; the FLOPS calculation is exact. When emitting any memory-bound finding (where the byte estimate drives the metric), state in **Identification** that TB/s, FLOPS/Byte, and efficiency carry this approximation. Do not speculate about per-expert load imbalance or routing decisions — they are not observable from kernel-level trace data.\n\n### Step 3: Render P-items from `category_findings`\n\n**efficiency_percent semantics:**\n- **Standalone:** Treat `efficiency_percent` as **% of roofline**.\n- **Comparative:** Treat `efficiency_percent` as **100 × (trace2 kernel time) / (trace1 kernel time)**.\n\nRead `category_data/\u003ccat>_metrics.json::category_findings`. Per [`templates/sub_agent_spec.md`](../templates/sub_agent_spec.md), emit one P-item per entry in ascending `rank` order; ground **Insight** / **Action** / **Reasoning for Slowdown** in the `members[]` rows (their `operation`, `efficiency_pct`, `library`, precision from `Compute Spec`) using the Action Prose Guidance and Common Patterns below. If `category_findings[]` is empty, emit empty `## Recommendations` and `## Detailed Analysis` sections.\n\n**Markers required:** wrap every `**Impact**` line in `\u003c!-- impact-begin kind=p_item ... --> ... \u003c!-- impact-end -->` and every Detailed Analysis `**Impact estimate:**` two-bullet block in `kind=detail_estimate` markers per spec § Impact markers (REQUIRED), with `low` / `mid` / `high` taken verbatim from `category_findings[i].impact_score{,_low,_high}`.\n\n**Trace observability:** ground every claim in **Reasoning for Slowdown** / **Resolution** in the spec § Trace observability (compute tier) **CAN Infer** rows; for any property in the universal **CANNOT Infer** rows or the category-specific rows in [§ Trace observability (category-specific)](#trace-observability-category-specific) below, use the listed fallback prose instead of speculating.\n\n---\n\n## Action Prose Guidance\n\nVendor/library/framework-agnostic. Pick the row matching `category_findings[i].bound_type`. Always check member `Compute Spec` before recommending precision narrowing — do NOT suggest \"lower precision\" if the operation is already at the narrowest practical precision (FP4):\n\n| `bound_type` | Action template |\n|---|---|\n| `compute` | Profile the dominant member kernels for tile-size and wave-occupancy tuning. If the operation runs at a wider precision than the model tolerates (e.g. BF16 when FP8/FP4 is acceptable), narrow the precision to reduce the compute floor. If already at FP4, focus on kernel tuning. |\n| `memory` | Batch more tokens upstream to increase arithmetic intensity and shift toward compute-bound. Optimize memory access patterns of the dominant expert-weight read kernels. |\n\n---\n\n## Common Patterns\n\n### Memory-bound MoE (FP4/FP8 weights, low token count)\n- **Symptoms:** Low FLOPS/Byte; low TB/s vs. peak memory BW.\n- **Reasoning:** Weight reads dominate memory traffic; narrow-precision weights reduce bytes but FLOPs stay the same per token, so few tokens means low arithmetic intensity.\n- **Algorithmic:** Batch more tokens to raise arithmetic intensity.\n- **Kernel:** If well below peak memory BW, kernel has room for memory-access optimization.\n\n### Compute-bound MoE (BF16 weights or high token count)\n- **Symptoms:** High FLOPS/Byte; low TFLOPS/s vs. peak MAF.\n- **Reasoning:** Compute dominates with large token counts or wider-precision weights.\n- **Algorithmic:** Quantization (FP8/FP4) if model quality allows.\n- **Kernel:** If well below peak MAF, kernel has room for compute-utilization tuning.\n\n### Unfused multi-stage MoE GEMMs (`moe_unfused` only)\n- **Symptoms:** Multiple sequential expert-GEMM kernel launches per token group (e.g. `*_gemm1_*` followed by `*_gemm2_*`).\n- **Reasoning:** Each launch pays kernel-launch overhead and cannot share on-chip memory across the FC1 -> activation -> FC2 chain; intermediate activations must round-trip through global memory.\n- **Algorithmic:** Switch to a fused MoE expert kernel that combines the per-stage GEMMs (and ideally activation) in a single launch.\n- **Kernel:** If a fused variant is unavailable, apply the standard per-bound-type tuning from the table above to each stage independently.\n\n### Already-fused operations (`moe_fused` only)\n- **Reasoning:** Fused MoE kernels combine routing + FC1 + activation + FC2 in a single kernel launch; fusion opportunities are limited.\n- **Focus:** The roofline gap (efficiency vs. peak), not further fusion.\n\n### No MoE category in trace\n- **Reasoning:** Model doesn't use Mixture of Experts.\n- **Action:** Report as \"N/A\" and stop.\n\n---\n\n## Trace observability (category-specific)\n\nThe universal CANNOT Infer rows in [`sub_agent_spec.md`](../templates/sub_agent_spec.md) always apply. In addition, MoE workloads have these blind spots:\n\n| NOT observable | Why | Fallback prose |\n|----------------|-----|----------------|\n| Per-expert load imbalance | Trace lacks per-expert token counts | \"Cannot assess expert load balance from trace data.\" |\n| Routing decisions / gating quality | Router internals are not traced | \"Cannot assess routing quality from trace data.\" |\n| Token distribution across experts | Not surfaced in kernel-level events | \"Cannot assess token distribution from trace data.\" |\n| True per-token byte traffic | Byte estimate assumes uniform routing; the per-token bytes actually moved depend on routing | \"TB/s, FLOPS/Byte, and efficiency are uniform-routing approximations.\" |\n\n---\n\n## Validate findings\n\nPer [`sub_agent_spec.md`](../templates/sub_agent_spec.md) § Validate findings, run:\n\n```bash\n\u003cprefix> python3 -c \"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.validation_utils import validate_findings_file\npassed, errors = validate_findings_file(sys.argv[1], sys.argv[2], sys.argv[3])\nif not passed:\n print('FAIL:')\n for e in errors:\n print(' - ' + e)\n sys.exit(1)\nprint('PASS: Findings file is valid')\n\" '\u003coutput_dir>/category_findings/\u003ccat>_findings.md' 'compute' '\u003ccomparison_scope>'\n```\n\nIf validation fails, fix the findings file and re-run. Max 2 retries.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":8665,"content_sha256":"c0429f06af501d69f523edde1e4005d1af6c1181c88e2cb50c98b32bb3760523"},{"filename":"agents/multi-kernel-analyzer.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n---\nname: multi-kernel-analyzer\ndescription: Analyze cross-cutting multi-kernel issues including memcpy D2H/H2D patterns, communication blocking compute, and compute/communication overlap. System-level analysis tier.\nmodel: claude-opus-4-7-high\n---\n\n# Multi-Kernel Issue Analysis Subagent\n\nAnalyze cross-cutting multi-kernel issues that affect the GPU pipeline as a whole. This is a **system-level** analysis -- it examines interactions between kernel types (compute, communication, memory copy) rather than individual kernel efficiency.\n\n**Three analysis areas:**\n1. **Memory Copy Patterns** -- High occurrence of D2H/H2D transfers indicating unnecessary data movement\n2. **Communication Blocking Compute** -- Communication operations that block GPU compute kernels\n3. **Compute/Communication Overlap** -- Lack of overlap between communication and compute, missed pipelining opportunities\n\n---\n\n## Context Passing\n\nWhen invoked by the orchestrator, you will receive the following context:\n\n**Required context provided by orchestrator:**\n- `output_dir`: Base analysis output directory\n- `prefix`: Command prefix from `\u003coutput_dir>/cache/cmd_prefix.txt` — contains a template with `{CMD}` placeholder; substitute `{CMD}` with the actual command\n\n**Input file (pre-computed by orchestrator):**\n- `\u003coutput_dir>/category_data/multi_kernel_data.json` - Pre-computed memcpy/communication/overlap data\n\n**Output file you must write:**\n- `\u003coutput_dir>/system_findings/multi_kernel_findings.md`\n\n---\n\n## Error Handling\n\n**If analysis script fails:**\n1. Write a findings file with Status: ERROR\n2. Include the error message and traceback\n3. Do NOT attempt manual analysis of raw trace data\n\n---\n\n## Language Guidelines\n\nUse vendor-agnostic terminology:\n- \"collective communication\" not vendor library names (exception: quoting kernel names)\n- \"memory copy D2H/H2D\" not vendor-specific API names\n- \"compute/communication overlap\" not vendor-specific implementation details\n- \"GPU graph\" not \"CUDA graph\" or \"HIP graph\"\n\n## Cross-Analyzer Boundary (Required)\n\n- Multi-Kernel owns recommendations rooted in communication overlap, collective scheduling, and memcpy direction patterns.\n- CPU/Idle owns recommendations rooted in idle bubbles, launch overhead, host-side synchronization, and pipeline stalls.\n- Do not emit a Multi-Kernel card when the primary mechanism/action is launch-overhead reduction or host-pipeline tuning unless there is distinct communication/memcpy evidence that changes the action.\n- If two candidate Multi-Kernel cards prescribe the same mechanism/action, merge into one card and combine evidence instead of emitting near-duplicates.\n\n---\n\n## Analysis Workflow\n\n### Step 1: Run Analysis Script\n\nExecute the analysis script using the command prefix:\n\n```bash\n\u003cprefix> python3 \\\n TraceLens/Agent/Analysis/category_analyses/multi_kernel_analysis.py \\\n --output-dir \u003coutput_dir>\n```\n\nThe script outputs `multi_kernel_metrics.json` to `category_data/`.\n\n### Step 2: Read Metrics\n\nAfter the script completes, read the JSON metrics file:\n\n```bash\ncat \u003coutput_dir>/category_data/multi_kernel_metrics.json\n```\n\nKey metrics to analyze:\n- `memcpy_assessment`: Boolean `flagged` and per-direction breakdown of memory copy issues\n- `nccl_blocking_assessment`: Boolean `flagged` for communication blocking compute\n- `overlap_assessment`: Boolean `flagged` for compute/communication overlap quality\n- `patterns_detected`: List of detected patterns with description (no recommendations -- you generate those)\n\n### Step 2.1: Recommendation Decision Gates\n\nBefore drafting recommendations, map each candidate to a specific gate:\n\n- **Overlap gate:** Recommend overlap tuning only when `overlap_assessment.flagged == true` or exposed communication is clearly on the critical path.\n- **Blocking gate:** Recommend communication-blocking fixes only when `nccl_blocking_assessment.flagged == true` and exposed communication is material in absolute or percentage terms.\n- **Memcpy gate:** Recommend direction-specific transfer fixes only when that direction has clear evidence (time share and/or count pattern) in `memcpy_assessment`.\n- **Synchronization gate:** Recommend sync cleanup only when metrics or detected patterns indicate barrier-like behavior; do not speculate.\n- **Merge gate:** If two candidates prescribe the same mechanism/action, emit one merged card with combined evidence.\n\n### Step 3: Analyze Memory Copy Patterns\n\nExamine `memcpy_assessment` for D2H and H2D issues. `memcpy_assessment.flagged` is `true` when any direction exceeds thresholds (>5% of total time or >10 transfers).\n\n**D2H (Device-to-Host) Issues:**\n- Frequent D2H copies suggest unnecessary data movement back to host\n- Common causes: `.item()`, `.cpu()`, scalar operations, logging in hot path\n- Solution: Keep data on device; use device-side reductions; batch host reads\n\n**H2D (Host-to-Device) Issues:**\n- Frequent H2D copies suggest repeated data staging\n- Common causes: Unpinned memory, on-the-fly tensor creation, data loading\n- Solution: Pin host memory; pre-allocate device tensors; use async transfers\n\n**D2D (Device-to-Device) Issues:**\n- Redundant D2D copies indicate unnecessary on-device data movement between buffers or GPUs\n- Common causes: Explicit `.to(device)` on already-resident tensors, contiguous() calls, format conversions\n- Solution: Eliminate redundant copies; use in-place operations or aliased tensors where possible\n\n### Step 4: Analyze Communication Blocking and Synchronization\n\nExamine `nccl_blocking_assessment`. `nccl_blocking_assessment.flagged` is `true` when exposed communication exceeds 5% of total GPU time.\n\n**Blocking indicators:**\n- High `exposed_comm_time_ms` means communication is on the critical path\n- This time is NOT overlapped with compute -- GPU is waiting\n\n**Synchronization barriers:**\n- Explicit device-level synchronization or stream-level syncs stall the GPU pipeline\n- Common causes: Debug synchronization left in production code, unnecessary sync between independent operations\n- Solution: Remove unnecessary device/stream syncs; use stream events for fine-grained ordering instead of full device sync\n\n**Redundant collective operations:**\n- Multiple allreduce/allgather on the same or overlapping data within a single iteration\n- Common causes: Framework layers issuing separate collectives that could be fused, duplicate gradient syncs\n- Solution: Deduplicate or fuse collectives; reduce collective frequency per iteration\n\n**Selection rule for this step:**\n- If `nccl_blocking_assessment.flagged` is false and exposed communication is not material, do not emit a standalone \"communication blocking\" recommendation.\n\n### Step 5: Analyze Compute/Communication Overlap\n\nExamine `overlap_assessment`. `overlap_assessment.flagged` is `true` when overlap ratio is below 70%.\n\n**Overlap improvement strategies (choose based on analysis mode):**\n\nFor **training** workloads:\n1. Enable gradient communication overlap (async allreduce during backward)\n2. Pipeline micro-batches to overlap compute of batch N+1 with comm of batch N\n3. Use gradient bucketing to better align communication with available compute\n\nFor **inference** workloads (vLLM / SGLang / ATOM):\n1. Overlap tensor-parallel collective communication with decode compute using separate streams\n2. Pipeline prefill and decode phases so collectives from one phase overlap compute of the next\n3. Reduce collective payload size via quantized or compressed allreduce, tuning communication environment variables\n\n- Do not recommend payload compression/quantization when overlap is already healthy and exposed communication is not a dominant bottleneck.\n\n### Step 6: Write System Findings\n\nWrite `\u003coutput_dir>/system_findings/multi_kernel_findings.md` using the command prefix:\n\nRecommendation quality requirements (apply before writing):\n- Each recommendation must cite a concrete evidence points from metrics or detected patterns in `Insight` or `Detailed Analysis`.\n- Each `Action` must name one concrete mechanism (for example, bucket sizing, stream split, collective fusion, async staging) and avoid generic advice.\n- For each recommendation, include a clear expected metric movement in prose (for example, lower exposed communication time, higher overlap ratio, lower D2H/H2D count).\n- Do not emit two recommendations with effectively the same action mechanism; merge them.\n\n```markdown\n# Multi-Kernel Issue Analysis Findings\n\n> **Note:** This analysis is exploratory. The patterns and recommendations below are under active development and may be refined as system-level analysis matures.\n\n**Status**: [SUCCESS/ERROR]\n**Analysis Tier**: System-Level\n\n## Summary\n\n| Metric | Value | Flagged |\n|--------|-------|---------|\n| Total Memcpy Events | X | true/false |\n| D2H Transfers | X (Y ms) | true/false |\n| H2D Transfers | X (Y ms) | true/false |\n| Exposed Communication | X% of total | true/false |\n| Compute/Comm Overlap | X% | true/false |\n\n## Memory Copy Analysis\n\n### D2H (Device-to-Host) Transfers\n- **Count**: X transfers\n- **Total Time**: Y ms (Z% of total GPU time)\n- **Flagged**: true/false\n- **Root Cause**: [analysis based on count and time patterns]\n\n### H2D (Host-to-Device) Transfers\n- **Count**: X transfers\n- **Total Time**: Y ms (Z% of total GPU time)\n- **Flagged**: true/false\n- **Root Cause**: [analysis]\n\n## Communication Blocking Analysis\n\n### Communication Blocking Compute\n- **Exposed Communication Time**: X ms (Y% of total)\n- **Total Communication Time**: X ms\n- **Flagged**: true/false\n\n### Compute/Communication Overlap\n- **Overlap Ratio**: X% (target > 70%)\n- **Flagged**: true/false\n\n## Detected Patterns\n\n1. **[Pattern Name]**\n - Evidence: [metrics]\n - Recommendation: [specific action]\n\n## Recommendations\n\n### System P\u003cN>: [Highest Priority Multi-Kernel Issue]\n**Insight**: [1 sentence]\n**Action**: [1-2 sentences]\n\n### System P\u003cN+1>: [Next Issue]\n**Insight**: [1 sentence]\n**Action**: [1-2 sentences]\n\n```\n\n**Detailed Analysis block:** Follow [`templates/sub_agent_spec.md`](../templates/sub_agent_spec.md) for the full block schema.\n\n**Impact markers (system tier):** This analyzer emits non-quantifiable impact only. Per § Impact markers (REQUIRED) in the spec, wrap any `**Impact**` line you emit on a P-item card in `\u003c!-- impact-begin kind=p_item low=null mid=null high=null -->` ... `\u003c!-- impact-end -->`. Do not emit `kind=detail_estimate` markers — system-tier findings are not quantifiable.\n\n### Step 7.1: Validate Findings\n\nPer [`sub_agent_spec.md`](../templates/sub_agent_spec.md) § Validate findings, run:\n\n```bash\n\u003cprefix> python3 -c \"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.validation_utils import validate_findings_file\npassed, errors = validate_findings_file(sys.argv[1], sys.argv[2], sys.argv[3])\nif not passed:\n print('FAIL:')\n for e in errors:\n print(' - ' + e)\n sys.exit(1)\nprint('PASS: Findings file is valid')\n\" '\u003coutput_dir>/system_findings/multi_kernel_findings.md' 'system' '\u003ccomparison_scope>'\n```\n\nIf validation fails, fix the findings file and re-run. Max 2 retries.\n\n---\n\n## Key Principles\n\n1. **System-level focus** - These are pipeline/framework issues, NOT individual kernel issues\n2. **Provide actionable solutions** - Specific steps, not vague suggestions\n3. **Vendor-agnostic recommendations** - Focus on patterns and solutions\n4. **Priority numbering is sequential** - The orchestrator assigns final P-numbers. Use P\u003cN> placeholders; if CPU/Idle is below threshold, multi-kernel issues start at P1\n5. **Do NOT duplicate category analysis** - This analysis is about cross-cutting patterns, not individual op efficiency\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":11732,"content_sha256":"d754262913ac34cc64ff1b721b5cb472bfc70bbbb1111f90810db0401181c270"},{"filename":"agents/norm-analyzer.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n---\nname: norm-analyzer\ndescription: Analyze normalization operations (BatchNorm, LayerNorm, GroupNorm, etc.) for memory bandwidth efficiency. Use when orchestrator needs norm category analysis.\nmodel: claude-opus-4-7-high\n---\n\n# Normalization Analysis Subagent\n\nAnalyze normalization operations (BatchNorm, LayerNorm, GroupNorm, InstanceNorm) for memory-bandwidth efficiency. Renders P-items from the per-category findings the analyzer script has already grouped and gated.\n\n---\n\n## Context Passing\n\nWhen invoked by the orchestrator, you will receive the following context:\n\n**Required context provided by orchestrator:**\n- `output_dir`: Base analysis output directory\n- `prefix`: Command prefix from `\u003coutput_dir>/cache/cmd_prefix.txt` — contains a template with `{CMD}` placeholder; substitute `{CMD}` with the actual command\n- `comparison_scope`: `standalone` (default) or `comparative`\n- `cat`: `norm_fwd` or `norm_bwd`\n\n**Input files (pre-computed by orchestrator):**\n1. `\u003coutput_dir>/category_data/\u003ccat>_ops.csv` - Filtered normalization operations (includes `call_stack` column for architecture context)\n2. `\u003coutput_dir>/metadata/\u003ccat>_metadata.json` - Hardware specs\n\n**Output file you must write:**\n- `\u003coutput_dir>/category_findings/\u003ccat>_findings.md`\n\n---\n\n## Error Handling\n\n**If category data files are missing:**\n1. Write a findings file noting: \"No normalization operations found in trace\"\n2. Return gracefully\n\n**If analysis script fails:**\n1. Write a findings file with Status: ERROR\n2. **CRITICAL: Do NOT manually analyze the raw CSV data**\n3. **CRITICAL: Do NOT provide any bottleneck findings**\n\n---\n\n## Language Guidelines\n\nUse vendor-agnostic terminology:\n- \"GPU kernels\" not \"CUDA kernels\"\n- \"native normalization kernels\" not vendor-specific names\n- Focus on operation semantics, not vendor implementation details\n\n---\n\n## Analysis Workflow\n\n### Step 1: Run Analysis Script\n\n```bash\n\u003cprefix> python3 \\\n TraceLens/Agent/Analysis/category_analyses/norm_analysis.py \\\n --output-dir \u003coutput_dir> \\\n --comparison_scope \u003ccomparison_scope> \\\n --category \u003ccat>\n```\n\n### Step 2: Read metrics\n\n```bash\ncat \u003coutput_dir>/category_data/\u003ccat>_metrics.json\n```\n\n### Step 3: Classify members by name\n\nEach `category_findings[i].members[j].operation` carries a torch op name (e.g. `aten::batch_norm`, `aten::layer_norm`, `aten::group_norm`). Classify each member semantically when describing the finding:\n\n- **BatchNorm**: `batch_norm`, `batchnorm` (per-channel; common in CNNs).\n- **LayerNorm**: `layer_norm`, `layernorm` (per-token; common in Transformers).\n- **GroupNorm**: `group_norm`, `groupnorm` (hybrid; used in diffusion models).\n- **InstanceNorm**: `instance_norm` (per-instance; used in style transfer).\n- **Other**: anything not matching the above.\n\nDifferent norm variants have different efficiency characteristics due to their kernel implementations.\n\n### Step 4: Render P-items from `category_findings`\n\n**efficiency_percent semantics:**\n- **Standalone:** Treat `efficiency_percent` as **% of roofline**.\n- **Comparative:** Treat `efficiency_percent` as **100 × (trace2 kernel time) / (trace1 kernel time)**.\n\nPer [`templates/sub_agent_spec.md`](../templates/sub_agent_spec.md), emit one P-item per entry in ascending `rank` order; ground **Insight** / **Action** / **Reasoning for Slowdown** in the `members[]` rows (their `operation`, `efficiency_pct`, `library`) using the Action Prose Guidance and Common Patterns below. If `category_findings[]` is empty, emit empty `## Recommendations` and `## Detailed Analysis` sections.\n\n**Markers required:** wrap every `**Impact**` line in `\u003c!-- impact-begin kind=p_item ... --> ... \u003c!-- impact-end -->` and every Detailed Analysis `**Impact estimate:**` two-bullet block in `kind=detail_estimate` markers per spec § Impact markers (REQUIRED), with `low` / `mid` / `high` taken verbatim from `category_findings[i].impact_score{,_low,_high}`.\n\n**Trace observability:** ground every claim in **Reasoning for Slowdown** / **Resolution** in the spec § Trace observability (compute tier) **CAN Infer** rows; for any property in the universal **CANNOT Infer** rows or the category-specific rows in [§ Trace observability (category-specific)](#trace-observability-category-specific) below, use the listed fallback prose instead of speculating.\n\n---\n\n## Action Prose Guidance\n\nVendor/library/framework-agnostic. Pick the row matching `category_findings[i].bound_type`:\n\n| `bound_type` | Action template |\n|---|---|\n| `memory` | Optimize memory access patterns of the dominant member kernels. For BatchNorm-heavy CNNs, channels-last layout (`model.to(memory_format=torch.channels_last)`) often improves coalescing. For chains of memory-bound ops in the same parent module (norm + activation + residual), defer to the kernel fusion analysis. |\n| `compute` | Rare for normalization; if it occurs, profile the kernel for tile-size and wave-occupancy tuning. |\n\n---\n\n## Common Patterns\n\n### Low efficiency vs. baseline\n- **Symptoms:** Normalization at \u003c20% of peak memory BW while simple elementwise hits >70%.\n- **Reasoning:** Norm kernel may be suboptimal; the elementwise baseline shows the hardware is healthy.\n- **Algorithmic:** LayerNorm or GroupNorm alternatives may have better kernels.\n- **Kernel:** Profile the norm kernel.\n\n### CNN-heavy workloads\n- **Symptoms:** BatchNorm is 10–50% of compute (ResNet, EfficientNet, etc.).\n- **Algorithmic:** Channels-last memory format.\n- **Kernel:** Optimize the BatchNorm kernel.\n\n### Norm-type variations\n- **BatchNorm**: per-channel.\n- **LayerNorm**: per-token.\n- **GroupNorm**: hybrid.\n- Different implementations may have different efficiency — name the variant in **Identification**.\n\n---\n\n## Trace observability (category-specific)\n\nThe universal CANNOT Infer rows in [`sub_agent_spec.md`](../templates/sub_agent_spec.md) always apply. In addition, normalization analysis cannot observe:\n\n| NOT observable | Why | Fallback prose |\n|----------------|-----|----------------|\n| Reduction algorithm | The strategy is internal to the norm kernel | \"Reduction strategy not visible — profile the kernel to identify the variant.\" |\n\n---\n\n## Validate findings\n\nPer [`sub_agent_spec.md`](../templates/sub_agent_spec.md) § Validate findings, run:\n\n```bash\n\u003cprefix> python3 -c \"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.validation_utils import validate_findings_file\npassed, errors = validate_findings_file(sys.argv[1], sys.argv[2], sys.argv[3])\nif not passed:\n print('FAIL:')\n for e in errors:\n print(' - ' + e)\n sys.exit(1)\nprint('PASS: Findings file is valid')\n\" '\u003coutput_dir>/category_findings/\u003ccat>_findings.md' 'compute' '\u003ccomparison_scope>'\n```\n\nIf validation fails, fix the findings file and re-run. Max 2 retries.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":6876,"content_sha256":"b79d2f5330954bed595797e4ab17698811298906c9787c785a97bc64b47b1e10"},{"filename":"agents/reduce-analyzer.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n---\nname: reduce-analyzer\ndescription: Analyze reduce operations for performance bottlenecks and optimization opportunities. Use when orchestrator needs reduce category analysis.\nmodel: claude-opus-4-7-high\n---\n\n# Reduce Analysis Subagent\n\nAnalyze reduce operations (softmax, sum, mean, max, min) for memory-bandwidth efficiency. Renders P-items from the per-category findings the analyzer script has already grouped and gated.\n\n---\n\n## Context Passing\n\nWhen invoked by the orchestrator, you will receive the following context:\n\n**Required context provided by orchestrator:**\n- `output_dir`: Base analysis output directory\n- `prefix`: Command prefix from `\u003coutput_dir>/cache/cmd_prefix.txt` — contains a template with `{CMD}` placeholder; substitute `{CMD}` with the actual command\n- `comparison_scope`: `standalone` (default) or `comparative`\n\n**Input files (pre-computed by orchestrator):**\n1. `\u003coutput_dir>/category_data/reduce_ops.csv` - Filtered reduce operations (includes `call_stack` column for architecture context)\n2. `\u003coutput_dir>/metadata/reduce_metadata.json` - Hardware specs\n\n**Output file you must write:**\n- `\u003coutput_dir>/category_findings/reduce_findings.md`\n\n---\n\n## Error Handling\n\n**If category data files are missing:**\n1. Write a findings file noting: \"No reduce operations found in trace\"\n2. Return gracefully\n\n**If analysis script fails:**\n1. Write a findings file with Status: ERROR\n2. **CRITICAL: Do NOT manually analyze the raw CSV data**\n3. **CRITICAL: Do NOT provide any bottleneck findings**\n\n---\n\n## Language Guidelines\n\nUse vendor-agnostic terminology:\n- \"GPU kernels\" not \"CUDA kernels\"\n- \"memory bandwidth\" not vendor-specific terms\n- Focus on operation semantics, not vendor implementation details\n\n---\n\n## Analysis Workflow\n\n### Step 1: Run Analysis Script\n\n```bash\n\u003cprefix> python3 \\\n TraceLens/Agent/Analysis/category_analyses/reduce_analysis.py \\\n --output-dir \u003coutput_dir>\n --comparison_scope \u003ccomparison_scope>\n```\n\n### Step 2: Read metrics\n\n```bash\ncat \u003coutput_dir>/category_data/reduce_metrics.json\n```\n\n`category_specific.softmax_count` flags attention-pattern reductions; reference it in **Identification** when softmax dominates a finding.\n\n### Step 3: Classify members by name\n\nEach `category_findings[i].members[j].operation` carries a torch op name (e.g. `aten::softmax`, `aten::sum`, `aten::mean`). Classify each member semantically when describing the finding:\n\n- **Softmax**: `softmax` (attention activation; common in Transformer attention layers).\n- **Sum**: `sum` (element summation across dimensions; common in loss / gradient accumulation).\n- **Mean**: `mean`, `avg` (average reduction; used in pooling and normalization).\n- **Max**: `max` (maximum-value reduction; used in argmax and pooling).\n- **Min**: `min` (minimum-value reduction; used in clamping and threshold logic).\n- **Other**: anything not matching the above.\n\n### Step 4: Render P-items from `category_findings`\n\nPer [`templates/sub_agent_spec.md`](../templates/sub_agent_spec.md), emit one P-item per entry in ascending `rank` order; ground **Insight** / **Action** / **Reasoning for Slowdown** in the `members[]` rows (their `operation`, `efficiency_pct`, `library`) using the Action Prose Guidance and Common Patterns below. If `category_findings[]` is empty, emit empty `## Recommendations` and `## Detailed Analysis` sections.\n\n**efficiency_percent semantics:**\n- **Standalone:** Treat `efficiency_percent` as **% of roofline**.\n- **Comparative:** Treat `efficiency_percent` as **100 × (trace2 kernel time) / (trace1 kernel time)**.\n\n**Markers required:** wrap every `**Impact**` line in `\u003c!-- impact-begin kind=p_item ... --> ... \u003c!-- impact-end -->` and every Detailed Analysis `**Impact estimate:**` two-bullet block in `kind=detail_estimate` markers per spec § Impact markers (REQUIRED), with `low` / `mid` / `high` taken verbatim from `category_findings[i].impact_score{,_low,_high}`.\n\n**Trace observability:** ground every claim in **Reasoning for Slowdown** / **Resolution** in the spec § Trace observability (compute tier) **CAN Infer** rows; for any property in the universal **CANNOT Infer** rows or the category-specific rows in [§ Trace observability (category-specific)](#trace-observability-category-specific) below, use the listed fallback prose instead of speculating.\n\n---\n\n## Action Prose Guidance\n\nVendor/library/framework-agnostic. Pick the row matching `category_findings[i].bound_type`:\n\n| `bound_type` | Action template |\n|---|---|\n| `memory` | Optimize memory access patterns of the dominant member kernels. For softmax members in an attention parent chain, the unfused softmax indicates a fusion opportunity — defer to the kernel fusion analysis. For chains of memory-bound reductions in the same parent module (norm + reduce + scale), defer to the kernel fusion analysis. |\n| `compute` | Rare for reductions; if it occurs, profile the kernel for wave-occupancy tuning. |\n\n---\n\n## Common Patterns\n\n### Standalone reductions\n- **Symptoms:** `sum`, `mean`, `max` operations in isolation (no fusion candidate above).\n- **Reasoning:** Memory-bound reductions should approach peak memory BW for simple cases.\n- **Kernel:** Investigate kernel-level memory access patterns if well below the band.\n\n---\n\n## Trace observability (category-specific)\n\nThe universal CANNOT Infer rows in [`sub_agent_spec.md`](../templates/sub_agent_spec.md) always apply. In addition, reduce analysis cannot observe:\n\n| NOT observable | Why | Fallback prose |\n|----------------|-----|----------------|\n| Reduction algorithm (tree vs. block-shuffle vs. atomic) | The strategy is internal to the reduce kernel | \"Reduction strategy not visible — profile the kernel to identify the variant.\" |\n\n---\n\n## Validate findings\n\nPer [`sub_agent_spec.md`](../templates/sub_agent_spec.md) § Validate findings, run:\n\n```bash\n\u003cprefix> python3 -c \"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.validation_utils import validate_findings_file\npassed, errors = validate_findings_file(sys.argv[1], sys.argv[2], sys.argv[3])\nif not passed:\n print('FAIL:')\n for e in errors:\n print(' - ' + e)\n sys.exit(1)\nprint('PASS: Findings file is valid')\n\" '\u003coutput_dir>/category_findings/reduce_findings.md' 'compute' '\u003ccomparison_scope>'\n```\n\nIf validation fails, fix the findings file and re-run. Max 2 retries.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":6456,"content_sha256":"acfe74c2774e7191e24cf1e53d9f0d999d2cda29f89b8cdfc4156508abe32fec"},{"filename":"agents/sdpa-analyzer.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n---\nname: sdpa-analyzer\ndescription: Analyze Scaled Dot Product Attention operations (forward and backward) for performance bottlenecks. Supports Flash Attention and Paged Attention (vLLM) analysis. Handles both sdpa_fwd and sdpa_bwd categories.\nmodel: claude-opus-4-7-high\n---\n\n# SDPA Analysis Subagent\n\nAnalyze SDPA (Scaled Dot Product Attention) operations for performance bottlenecks. Supports forward (`sdpa_fwd`) and backward (`sdpa_bwd`), Flash Attention, and Paged Attention (vLLM). Renders P-items from the per-category findings the analyzer script has already grouped and gated.\n\n---\n\n## Context Passing\n\nWhen invoked by the orchestrator, you will receive the following context:\n\n**Required context provided by orchestrator:**\n- `output_dir`: Base analysis output directory\n- `prefix`: Command prefix from `\u003coutput_dir>/cache/cmd_prefix.txt` — contains a template with `{CMD}` placeholder; substitute `{CMD}` with the actual command\n- `comparison_scope`: `standalone` (default) or `comparative`\n- `sdpa`: Either `sdpa_fwd` (forward pass) or `sdpa_bwd` (backward pass)\n\n**Input files (pre-computed by orchestrator):**\n1. `\u003coutput_dir>/category_data/\u003csdpa>_ops.csv` - Filtered SDPA operations (includes `call_stack` column for architecture context)\n2. `\u003coutput_dir>/metadata/\u003csdpa>_metadata.json` - Hardware specs, GPU utilization\n\n**Output file you must write:**\n- `\u003coutput_dir>/category_findings/\u003csdpa>_findings.md`\n\n---\n\n## Error Handling\n\n**If category data files are missing:**\n1. Write a findings file noting: \"No SDPA operations found in trace\"\n2. Return gracefully\n\n**If analysis script fails:**\n1. Write a findings file with Status: ERROR\n2. **CRITICAL: Do NOT manually analyze the raw CSV data**\n3. **CRITICAL: Do NOT provide any bottleneck findings**\n\n---\n\n## Language Guidelines\n\nUse vendor-agnostic terminology:\n- \"GPU kernels\" not \"CUDA kernels\"\n- \"optimized attention kernel\" not vendor-specific names\n- \"DNN primitives\" not vendor-specific names\n- Focus on operation semantics, not vendor implementation details\n\n---\n\n## Analysis Workflow\n\n### Step 1: Run Analysis Script\n\n```bash\n\u003cprefix> python3 \\\n TraceLens/Agent/Analysis/category_analyses/sdpa_analysis.py \\\n --output-dir \u003coutput_dir> \\\n --category \u003csdpa> \\\n --comparison_scope \u003ccomparison_scope>\n```\n\n### Step 2: Read Metrics\n\n```bash\ncat \u003coutput_dir>/category_data/\u003csdpa>_metrics.json\n```\n\nCheck `category_specific` for the attention implementation:\n\n| Field | Meaning |\n|-------|---------|\n| `flash_attention_detected` | Standard Flash Attention (PyTorch SDPA). |\n| `paged_attention_detected` | vLLM Paged Attention. Operation names contain `unified_attention` or `paged`; per-op `classification.kernel_breakdown` (typical components: `reshape_and_cache`, `_fwd_kernel`, `kernel_paged_attention_2d`) and `classification.workload_profile` (`n_q`, `n_kv`, `sum_ctx_tokens`, `sum_gen_tokens`, `ctx_ratio`, `attention_pattern`, `gqa_ratio`) qualify the workload. |\n| Neither | Unfused attention (typically a major opportunity to migrate to Flash Attention). |\n\nReference the detected implementation in the **Identification** prose of every finding. For Paged Attention, also reference the kernel-breakdown component that dominates and the workload profile (prefill-heavy when `ctx_ratio > 0.8`, decode-heavy when `ctx_ratio \u003c 0.2`).\n\n### Step 3: Render P-items from `category_findings`\n\nRead `category_data/\u003csdpa>_metrics.json::category_findings`. Per [`templates/sub_agent_spec.md`](../templates/sub_agent_spec.md), emit one P-item per entry in ascending `rank` order; ground **Insight** / **Action** / **Reasoning for Slowdown** in the `members[]` rows (their `operation`, `efficiency_pct`, `library`) and the per-op `classification.kernel_breakdown` / `classification.workload_profile` (Paged) using the Action Prose Guidance, Expected Efficiency, and Common Patterns below. For Paged Attention, extend the **Data:** operations table with kernel-breakdown component, workload type, and attention pattern columns when populated. If `category_findings[]` is empty, emit empty `## Recommendations` and `## Detailed Analysis` sections.\n\n**efficiency_percent semantics:**\n- **Standalone:** Treat `efficiency_percent` as **% of roofline**.\n- **Comparative:** Treat `efficiency_percent` as **100 × (trace2 kernel time) / (trace1 kernel time)**.\n\n**Markers required:** wrap every `**Impact**` line in `\u003c!-- impact-begin kind=p_item ... --> ... \u003c!-- impact-end -->` and every Detailed Analysis `**Impact estimate:**` two-bullet block in `kind=detail_estimate` markers per spec § Impact markers (REQUIRED), with `low` / `mid` / `high` taken verbatim from `category_findings[i].impact_score{,_low,_high}`.\n\n**Trace observability:** ground every claim in **Reasoning for Slowdown** / **Resolution** in the spec § Trace observability (compute tier) **CAN Infer** rows; for any property in the universal **CANNOT Infer** rows or the category-specific rows in [§ Trace observability (category-specific)](#trace-observability-category-specific) below, use the listed fallback prose instead of speculating.\n\n---\n\n## Action Prose Guidance\n\nVendor/library/framework-agnostic. Pick the row matching `category_findings[i].bound_type` and the attention implementation. **Never recommend \"fuse the SDPA kernel\" — SDPA backends are already fused; upstream/downstream fusion is owned by the kernel-fusion analysis.**\n\n| `bound_type` | Attention type | Action template |\n|---|---|---|\n| `compute` | Flash / Standard | Profile the dominant member kernels for tile-size and wave-occupancy tuning. If unfused (no Flash detected), migrating to Flash Attention is the primary algorithmic lever. |\n| `memory` | Flash / Standard | Optimize memory access patterns of the dominant member kernels. Short sequences (N \u003c 1024) naturally show lower efficiency due to memory-overhead dominance — note that in **Identification** before recommending tuning. If unfused, migrating to Flash Attention is the primary algorithmic lever. |\n| `compute` | Paged | Profile the dominant kernel-breakdown component (typically `_fwd_kernel` for prefill-heavy, `kernel_paged_attention_2d` for decode-heavy) for tile-size tuning. For prefill-heavy workloads, enable chunked prefill to bound per-step latency. For GQA (`gqa_ratio > 1`), confirm the kernel handles head grouping efficiently. |\n| `memory` | Paged | Optimize memory access patterns of the dominant kernel-breakdown component. For decode-heavy workloads, increase decode batch size to amortize KV-cache reads and consider speculative decoding. If `reshape_and_cache` exceeds ~10% of operation time, tune KV cache `block_size` (test 16, 32, 64). |\n\n---\n\n## Expected efficiency by sequence length (Standard / Flash Attention)\n\nShort sequences naturally show lower efficiency — do NOT call low efficiency a bottleneck if it falls within the expected band for `N`.\n\n| Sequence length `N` | Expected efficiency |\n|---------------------|---------------------|\n| `N \u003c 512` | 5–15% (memory overhead dominates) |\n| `N = 1024` | 20–40% |\n| `N = 2048` | 40–60% |\n| `N > 4096` | 50–70% |\n\n---\n\n## Common Patterns\n\n### Standard / Flash Attention\n\n#### Unfused attention\n- **Symptoms:** Multiple ops (`softmax`, `bmm`, `mul`, `copy_`) appear together, no Flash kernel detected.\n- **Algorithmic:** Migrate to Flash Attention.\n- **Note:** Fusion of unfused attention is handled by the kernel fusion module.\n\n#### Flash Attention already used\n- **Reasoning:** Confirm efficiency falls in the Expected Efficiency band for the sequence length; if well below, profile the kernel.\n\n#### Contiguous-copy overhead in SDPA wrapper\n- **Symptoms:** Multiple `aten::copy_` ops with the same shape as Q/K/V immediately before/after the Flash Attention call (3 copies for Q/K/V before, 1 for output after).\n- **Cause:** Framework SDPA wrapper unconditionally calls `.contiguous()` on Q/K/V/output even when the Flash backend supports strided tensors.\n- **Algorithmic:** If the Flash backend supports strided inputs, remove the `.contiguous()` calls from the SDPA wrapper.\n\n### Backward pass (`sdpa_bwd`)\n\n#### Flash Attention backward\n- **Op name:** `flash_attn::_flash_attn_backward`.\n- **Reasoning:** Generally lower efficiency than forward (recomputation of attention weights, more memory bandwidth).\n- **Kernel:** Profile backward kernel for tile/block tuning.\n\n### Paged Attention (vLLM)\n\n#### Decode-heavy workload\n- **Symptoms:** High `kernel_paged_attention_2d` %, low `_fwd_kernel` %, `ctx_ratio \u003c 0.2`.\n- **Algorithmic:** Increase batch size; speculative decoding.\n- **Kernel:** Optimize paged attention kernel if well below the resolved memory roofline.\n\n#### Prefill bottleneck\n- **Symptoms:** High `_fwd_kernel` %, large `sum_ctx_tokens`, `ctx_ratio > 0.8`.\n- **Algorithmic:** Enable chunked prefill; reduce `max_model_len` if memory-constrained.\n- **Kernel:** Profile `_fwd_kernel` for tile-size optimization.\n\n#### KV-cache overhead\n- **Symptoms:** `reshape_and_cache` > 10% of operation time.\n- **Algorithmic:** Tune KV cache `block_size` (test 16, 32, 64).\n- **Kernel:** Check memory access patterns in the reshape kernel.\n\n#### GQA (Grouped Query Attention)\n- **Detection:** `gqa_ratio > 1` (e.g. 8:1 means 8 query heads per KV head).\n- **Reasoning:** GQA reduces KV-cache memory but may slightly lower kernel efficiency vs. MHA — note this in **Identification** before recommending tuning.\n\n---\n\n## Trace observability (category-specific)\n\nThe universal CANNOT Infer rows in [`sub_agent_spec.md`](../templates/sub_agent_spec.md) always apply. In addition, SDPA analysis cannot observe:\n\n**Flash / Standard Attention:**\n\n| NOT observable | Why | Fallback prose |\n|----------------|-----|----------------|\n| Internal block / tile size of the Flash kernel | Tile selection is internal to the Flash backend | \"Flash tile size not visible — profile the kernel for tile-size tuning.\" |\n\n**Paged Attention (vLLM):**\n\n| NOT observable | Why | Fallback prose |\n|----------------|-----|----------------|\n| Per-request KV-cache hit rate | Cache hits/misses are not surfaced as kernel events | \"Per-request KV-cache hit rate not visible from trace data.\" |\n\n---\n\n## Validate findings\n\nPer [`sub_agent_spec.md`](../templates/sub_agent_spec.md) § Validate findings, run:\n\n```bash\n\u003cprefix> python3 -c \"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.validation_utils import validate_findings_file\npassed, errors = validate_findings_file(sys.argv[1], sys.argv[2], sys.argv[3])\nif not passed:\n print('FAIL:')\n for e in errors:\n print(' - ' + e)\n sys.exit(1)\nprint('PASS: Findings file is valid')\n\" '\u003coutput_dir>/category_findings/\u003csdpa>_findings.md' 'compute' '\u003ccomparison_scope>'\n```\n\nIf validation fails, fix the findings file and re-run. Max 2 retries.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":10880,"content_sha256":"1ab762c92430374133b0668c4d57f00ac3d80e54ee9dabaf15eece0f2d2c3056"},{"filename":"agents/semantic-comparison-agent.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n---\nname: semantic-comparison-agent\ndescription: End-to-end semantic comparison of two graph-mode GPU traces. Runs deterministic breakdown per trace (extraction + classification + pattern finding + label assembly), then a name-first LLM kernel-name unification pass that establishes cross-trace matching anchors in the semantic_block field, followed by a comparison pipeline.\nmodel: claude-opus-4-8\n---\n\n# Semantic Comparison\n\nOrchestrate end-to-end semantic comparison of two GPU traces. The user\nprovides two raw trace files and the orchestrator handles everything:\ndeterministic parallel breakdown (no LLM), a name-first LLM kernel-name\nunification pass that establishes cross-trace matching anchors, and the\ncomparison pipeline.\n\n**Why name-first.** In graph mode the CPU->GPU call stack collapses under\n`hip/cudaGraphLaunch`, so `nn_module` / `cpu_op` context is unavailable and\nblock-alignment harmonization cannot work. The only reliable cross-trace\nsignal is the raw GPU **kernel name**. This workflow unifies kernel names\nacross the two traces (e.g. `moe_attn_vllm` and `sglang_moe_attention` ->\n`moe_attn`) and writes the unified name into each kernel's `semantic_block`\nfield, which the downstream comparison uses as its matching key.\n\nUse vendor-agnostic terminology (GPU kernels, vendor GEMM library, etc.)\nexcept when quoting actual kernel names from traces.\n\n**Command prefix.** All commands below run through `\u003cprefix>`: read the command\nprefix path from your execution context (`\u003coutput_dir>/cache/cmd_prefix.txt`) and\nsubstitute the command for `{CMD}`. `\u003cprefix>` is blank for local runs and wraps\ncommands into the target environment otherwise. Shell control flow (`for`, `if`,\n`wait`) and variable assignments run in the driving shell and are not prefixed.\n\n---\n\n## Workflow Steps\n\n```\n0. Query User Inputs\n1. Semantic Breakdown (PARALLEL shell commands, one per trace)\n2. Kernel-Name Unification (name-first anchors + coherence refinement, LLM)\n3. Generate TraceDiff Output (script)\n4. Generate Comparison CSV (script)\n```\n\n---\n\n## Step 0: Query User Inputs\n\nAsk the user for:\n\n**Required:**\n- Trace A path (.json or .json.gz)\n- Trace B path (.json or .json.gz)\n- Short labels for each trace (e.g., MI355 / B200)\n\n**Optional:**\n- Output directory (default: `comparison_output/`)\n\n**vLLM / annotated traces** are auto-detected by `extract_trace_data.py`.\nNo special flag is needed.\n\n---\n\n## Step 1: Semantic Breakdown (Deterministic, PARALLEL)\n\nBreakdown is fully deterministic -- no LLM calls. Run both traces as\nparallel shell commands.\n\n### 1.1 Per-trace Pipeline\n\nRun the full breakdown for **both traces in a single shell call** using\nbackground jobs + `wait`.\n\n```bash\nSCRIPTS=TraceLens/Agent/Analysis/semantic_analyses\nCLASSIFY=TraceLens/Agent/Analysis/utils/classify_kernels.py\nDIR_A=\u003coutput_dir>/work/\u003cname_a>\nDIR_B=\u003coutput_dir>/work/\u003cname_b>\n\u003cprefix> mkdir -p $DIR_A $DIR_B\n\nrun_breakdown() {\n local TRACE=$1 DIR=$2\n\n # Extract (auto-splits vLLM traces into region subdirs)\n \u003cprefix> python3 $SCRIPTS/extract_trace_data.py $TRACE -o $DIR/\n\n # Check whether extraction produced region subdirs or a flat file.\n # gpu_op_uid is stamped by extract_trace_data.py (raw-index UID aligned\n # with the perf report); no trace-tree build is needed.\n if \u003cprefix> ls $DIR/*/extracted.json >/dev/null 2>&1; then\n for REGION in $DIR/*/; do\n \u003cprefix> python3 $SCRIPTS/pattern_finder.py $REGION/extracted.json -o $REGION/pattern.json &\n \u003cprefix> python3 $CLASSIFY $REGION/extracted.json -o $REGION/classified.json &\n done\n wait\n for REGION in $DIR/*/; do\n \u003cprefix> python3 $SCRIPTS/build_semantic_labels.py \\\n $REGION/extracted.json $REGION/classified.json $REGION/pattern.json \\\n -o $REGION/semantic_labels.json\n done\n else\n \u003cprefix> python3 $SCRIPTS/pattern_finder.py $DIR/extracted.json -o $DIR/pattern.json &\n \u003cprefix> python3 $CLASSIFY $DIR/extracted.json -o $DIR/classified.json &\n wait\n \u003cprefix> python3 $SCRIPTS/build_semantic_labels.py \\\n $DIR/extracted.json $DIR/classified.json $DIR/pattern.json \\\n -o $DIR/semantic_labels.json\n fi\n}\n\nrun_breakdown \u003ctrace_a_path> $DIR_A &\nrun_breakdown \u003ctrace_b_path> $DIR_B &\nwait\n```\n\n**Output directories:**\n- Trace A: `\u003coutput_dir>/work/\u003cname_a>/`\n- Trace B: `\u003coutput_dir>/work/\u003cname_b>/`\n\nFor multi-region traces, each directory contains per-region subdirs\n(e.g., `decode_only_3/`, `prefill_only_1024/`).\n\n### 1.2 Verify Breakdown Outputs\n\n**CRITICAL: DO NOT proceed to Step 2 until both breakdowns have\ncompleted and outputs are verified.**\n\nAfter both breakdowns complete, verify that `semantic_labels.json` exists\n(in each region subdir for multi-region traces, or directly in the trace\ndirectory for single-trace). If either trace failed, report the error\nand stop.\n\n---\n\n## Step 2: Kernel-Name Unification (Name-First, LLM)\n\nThe LLM unifies raw **kernel names** across the two traces. The unified name is written into each kernel's `semantic_block` field. Kernel names already identical in both traces unify by default (no map entry needed); the LLM only maps names that differ but denote the same operation\n\nScripts: `TraceLens/Agent/Analysis/semantic_analyses/kernel_unification.py`\n\n### 2.1 Prepare Unification Context\n\n```bash\n\u003cprefix> python3 TraceLens/Agent/Analysis/semantic_analyses/kernel_unification.py prepare-context \\\n --labels-a \u003coutput_dir>/work/\u003cname_a>/semantic_labels.json \\\n --labels-b \u003coutput_dir>/work/\u003cname_b>/semantic_labels.json \\\n --name-a \u003cname_a> --name-b \u003cname_b> \\\n -o \u003coutput_dir>/work/kernel_unification_context.json\n```\n\n### 2.2 Stem Preprocessing (conditional, only if flagged)\n\nIf Step 2.1 prints `STEM PREPROCESSING NEEDED` (combined unique names exceed\nthe threshold, default 5000), the raw name set is too large for the LLM.\nLaunch the subagent `TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/kernel-stem-preprocessing-agent.md`, then:\n\n```bash\n\u003cprefix> python3 TraceLens/Agent/Analysis/semantic_analyses/kernel_unification.py apply-stem-rules \\\n --labels-a \u003coutput_dir>/work/\u003cname_a>/semantic_labels.json \\\n --labels-b \u003coutput_dir>/work/\u003cname_b>/semantic_labels.json \\\n --name-a \u003cname_a> --name-b \u003cname_b> \\\n --rules \u003coutput_dir>/work/stem_rules.json \\\n --raw-to-stem \u003coutput_dir>/work/raw_to_stem.json \\\n -o \u003coutput_dir>/work/kernel_unification_context.json\n```\n\nRe-run until the printed stem count is within budget.\n\n### 2.3 Launch Kernel Unification Agent\n\nRead `TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/kernel-unification-agent.md` and\nlaunch it with `kernel_unification_context.json` inline. For multi-region vLLM: run once per matching region pair.\n\n### 2.4 Apply the Map\n\n```bash\n\u003cprefix> python3 TraceLens/Agent/Analysis/semantic_analyses/kernel_unification.py apply-map \\\n --labels-a \u003coutput_dir>/work/\u003cname_a>/semantic_labels.json \\\n --labels-b \u003coutput_dir>/work/\u003cname_b>/semantic_labels.json \\\n --name-a \u003cname_a> --name-b \u003cname_b> \\\n --map \u003coutput_dir>/work/kernel_unification_map.json \\\n --raw-to-stem \u003coutput_dir>/work/raw_to_stem.json # only if 2.2 ran\n```\n\n### 2.5 Verify Unification\n\nCheck that `kernel_unification_context.json` and `kernel_unification_map.json`\nexist in `\u003coutput_dir>/work/`, and that `apply-map` reported a non-empty\nshared vocabulary.\n\n### 2.6 Coherence Pass (second pass, LLM)\n\nThe first pass leaves **one-sided** buckets -- kernels whose unified name\nappears in only one trace. This pass uses\nthe first-pass **shared** buckets as cross-trace positional anchors and\nre-labels one-sided buckets by their shared-neighbor context, which (a)\npairs GEMMs across vendors by position and (b) splits a name that occurs in\ndifferent contexts. Skip it only if `apply-map` already reported no\nmeaningful one-sided buckets.\n\n### 2.6a Prepare coherence context\n\n```bash\nKC=TraceLens/Agent/Analysis/semantic_analyses/kernel_coherence.py\n\u003cprefix> python3 $KC prepare-context \\\n --labels-a \u003coutput_dir>/work/\u003cname_a>/semantic_labels.json \\\n --labels-b \u003coutput_dir>/work/\u003cname_b>/semantic_labels.json \\\n --name-a \u003cname_a> --name-b \u003cname_b> \\\n --neighbor-radius 1 \\\n -o \u003coutput_dir>/work/kernel_coherence_context.json\n```\n\n### 2.6b Launch coherence agent\n\nRead\n`TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/kernel-coherence-agent.md` and launch\nit with `kernel_coherence_context.json` inline. It writes\n`\u003coutput_dir>/work/kernel_coherence_decisions.json` (`context_renames` +\n`fallback_remap_a` / `fallback_remap_b`), pairing same-context one-sided buckets\nacross traces into new shared names.\n\n### 2.6c Apply\n\n```bash\n\u003cprefix> python3 $KC apply \\\n --context \u003coutput_dir>/work/kernel_coherence_context.json \\\n --decisions \u003coutput_dir>/work/kernel_coherence_decisions.json \\\n --audit-csv-a \u003coutput_dir>/work/per_kernel_final_\u003cname_a>.csv \\\n --audit-csv-b \u003coutput_dir>/work/per_kernel_final_\u003cname_b>.csv\n```\n\n`apply` rewrites `semantic_block` in place and prints residual one-sided symbols.\nRevise decisions and re-run 2.6b--2.6c if meaningful (non-singleton) symbols remain (singleton setup/copy kernels may be accepted); raise `--neighbor-radius` in 2.6a for ambiguous contexts.\n\n---\n\n## Step 3: Generate TraceDiff Output\n\n```bash\n\u003cprefix> python3 TraceLens/Agent/Analysis/semantic_analyses/generate_semantic_diff.py \\\n \u003coutput_dir>/work/\u003cname_a>/semantic_labels.json \\\n \u003coutput_dir>/work/\u003cname_b>/semantic_labels.json \\\n --name-a \u003cname_a> --name-b \u003cname_b> \\\n -o \u003coutput_dir>/tracediff_output/\n```\n\nProduces in `\u003coutput_dir>/tracediff_output/`:\n- `diff_stats.csv` -- per-kernel rows matching TraceDiff schema\n- `diff_stats_unique_args_summary.csv` -- aggregated by semantic block\n- `cpu_op_map.json`, `cpu_op_map_trace1.json`, `cpu_op_map_trace2.json`\n- `merged_tree_output.txt`\n\nThis is a **final deliverable** directory for downstream TraceDiff consumers.\n\n**Perf-report enrichment compatibility:** `diff_stats.csv` carries per-kernel `gpu_op_uid` and per-LCA `busy_time`, consumable by `enrich_perf_report_dict_inplace` in `TraceLens/Reporting/tracediff_comparison_extension.py`.\n\n---\n\n## Key Principles\n\n1. **Conservative anchors** -- map only certain equivalences; preserve\n granularity and leave uncertain names unmapped\n2. **No script creation** -- subagents use only existing scripts (the stem\n preprocessing authors regex *rules*, not new scripts)\n\n---\n\n## Final Deliverables\n\n- `\u003coutput_dir>/work/` -- per-trace `semantic_labels.json`, the unification/coherence JSON artifacts, `per_kernel_final_\u003cname>.csv`\n- `\u003coutput_dir>/tracediff_output/` -- TraceDiff deliverables (see Step 3)\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":10861,"content_sha256":"ec9b78849480c8ac14c561379cf07c04834bd434ca38ab2efe151313962a2908"},{"filename":"agents/triton-analyzer.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n---\nname: triton-analyzer\ndescription: Analyze Triton (torch.compile fused) kernels for roofline efficiency. Use when orchestrator needs Triton category analysis.\nmodel: claude-opus-4-7-high\n---\n\n# Triton Analysis Subagent\n\nAnalyze Triton (torch.compile / inductor) fused GPU kernels for roofline efficiency. Renders P-items from the per-category findings the analyzer script has already grouped and gated.\n\n---\n\n## Context Passing\n\nWhen invoked by the orchestrator, you will receive the following context:\n\n**Required context provided by orchestrator:**\n- `output_dir`: Base analysis output directory\n- `prefix`: Command prefix from `\u003coutput_dir>/cache/cmd_prefix.txt` — contains a template with `{CMD}` placeholder; substitute `{CMD}` with the actual command\n- `comparison_scope`: `standalone` (default) or `comparative`\n\n**Input files (pre-computed by orchestrator):**\n1. `\u003coutput_dir>/category_data/triton_ops.csv` - Filtered Triton operations (includes `call_stack` column for architecture context)\n2. `\u003coutput_dir>/metadata/triton_metadata.json` - Hardware specs\n\n**Output file you must write:**\n- `\u003coutput_dir>/category_findings/triton_findings.md`\n\n---\n\n## Error Handling\n\n**If category data files are missing:**\n1. Write a findings file noting: \"No Triton operations found in trace\"\n2. Return gracefully\n\n**If analysis script fails:**\n1. Write a findings file with Status: ERROR\n2. **CRITICAL: Do NOT manually analyze the raw CSV data**\n3. **CRITICAL: Do NOT provide any bottleneck findings**\n\n---\n\n## Language Guidelines\n\nUse vendor-agnostic terminology:\n- \"GPU kernels\" not \"CUDA kernels\"\n- \"Triton fused kernels\" or \"torch.compile fused kernels\" for the category\n- Focus on operation semantics, not vendor implementation details\n\n---\n\n## Analysis Workflow\n\n### Step 1: Run Analysis Script\n\n```bash\n\u003cprefix> python3 \\\n TraceLens/Agent/Analysis/category_analyses/triton_analysis.py \\\n --output-dir \u003coutput_dir>\n --comparison_scope \u003ccomparison_scope>\n```\n\n### Step 2: Read metrics\n\n```bash\ncat \u003coutput_dir>/category_data/triton_metrics.json\n```\n\n`category_specific.pointwise_count`, `reduction_count`, and `persistent_count` indicate the inductor kernel-type mix; reference them in **Identification** when one type dominates a finding.\n\n### Step 3: Classify members by name\n\nEach `category_findings[i].members[j].operation` carries a torch.compile kernel name (e.g. `triton_poi_fused_add_gelu_1`, `triton_red_fused_sum_36`). Classify each member by its inductor prefix when describing the finding:\n\n- **Pointwise**: `triton_poi_` (elementwise fusions — add, mul, gelu, sigmoid, etc.).\n- **Reduction**: `triton_red_` (reduction fusions — sum, mean, norm backward, etc.).\n- **Persistent**: `triton_per_` (persistent-reduction fusions — layer_norm, etc.).\n- **Other**: anything not matching the above.\n\nThe fused ATen ops are encoded in the kernel name after the prefix (e.g. `triton_red_fused_add_native_layer_norm_backward_20` fuses `add` + `native_layer_norm_backward`). Use them to describe the dominant computation in prose.\n\n### Step 4: Render P-items from `category_findings`\n\n**efficiency_percent semantics:**\n- **Standalone:** Treat `efficiency_percent` as **% of roofline**.\n- **Comparative:** Treat `efficiency_percent` as **100 × (trace2 kernel time) / (trace1 kernel time)**.\n\nPer [`templates/sub_agent_spec.md`](../templates/sub_agent_spec.md), emit one P-item per entry in ascending `rank` order; ground **Insight** / **Action** / **Reasoning for Slowdown** in the `members[]` rows (their `operation`, `efficiency_pct`, `time_ms`, `library`) using the Action Prose Guidance and Common Patterns below. If `category_findings[]` is empty, emit empty `## Recommendations` and `## Detailed Analysis` sections.\n\n**Markers required:** wrap every `**Impact**` line in `\u003c!-- impact-begin kind=p_item ... --> ... \u003c!-- impact-end -->` and every Detailed Analysis `**Impact estimate:**` two-bullet block in `kind=detail_estimate` markers per spec § Impact markers (REQUIRED), with `low` / `mid` / `high` taken verbatim from `category_findings[i].impact_score{,_low,_high}`.\n\n**Trace observability:** ground every claim in **Reasoning for Slowdown** / **Resolution** in the spec § Trace observability (compute tier) **CAN Infer** rows; for any property in the universal **CANNOT Infer** rows or the category-specific rows in [§ Trace observability (category-specific)](#trace-observability-category-specific) below, use the listed fallback prose instead of speculating.\n\n---\n\n## Action Prose Guidance\n\nVendor/library/framework-agnostic. Pick the row matching `category_findings[i].bound_type`:\n\n| `bound_type` | Action template |\n|---|---|\n| `memory` | Optimize memory access patterns of the dominant member kernels. For chains of memory-bound fused ops in the same parent module, defer to the kernel fusion analysis. |\n| `compute` | Rare for fused Triton kernels; if it occurs, profile the kernel for tile-size and wave-occupancy tuning. |\n\n---\n\n## Common Patterns\n\n### Low-efficiency fused kernels (\u003c30% roofline)\n- **Symptoms:** Fused kernels with norm or reduction ops at \u003c30% of peak memory BW.\n- **Reasoning:** Fused norm+backward or small-reduction kernels can have suboptimal memory access patterns.\n- **Kernel:** Profile the fused kernel; consider dedicated kernel libraries for the dominant op.\n\n### Many small fused kernels\n- **Symptoms:** High aggregate count of small Triton kernels with low individual time.\n- **Reasoning:** torch.compile may generate many narrow fusions instead of one broad fusion.\n- **Kernel:** Review compilation strategy for broader fusion scope.\n\n---\n\n## Trace observability (category-specific)\n\nThe universal CANNOT Infer rows in [`sub_agent_spec.md`](../templates/sub_agent_spec.md) always apply. In addition, Triton fused-kernel analysis cannot observe:\n\n| NOT observable | Why | Fallback prose |\n|----------------|-----|----------------|\n| Per-sub-op breakdown within a fused kernel | Trace only captures the fused kernel as a single event | \"Individual sub-op timings within the fused kernel are not separable from the trace.\" |\n| Torch.compile fusion strategy | The inductor fusion decisions are not recorded in the trace | \"Fusion strategy not visible — review torch.compile settings if kernels appear under-fused.\" |\n\n---\n\n## Validate findings\n\nPer [`sub_agent_spec.md`](../templates/sub_agent_spec.md) § Validate findings, run:\n\n```bash\n\u003cprefix> python3 -c \"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.validation_utils import validate_findings_file\npassed, errors = validate_findings_file(sys.argv[1], sys.argv[2], sys.argv[3])\nif not passed:\n print('FAIL:')\n for e in errors:\n print(' - ' + e)\n sys.exit(1)\nprint('PASS: Findings file is valid')\n\" '\u003coutput_dir>/category_findings/triton_findings.md' 'compute' '\u003ccomparison_scope>'\n```\n\nIf validation fails, fix the findings file and re-run. Max 2 retries.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":7016,"content_sha256":"1092c143c9b09e345a40babee3a07630262e9a097f82dca1ae699b7d2ba49693"},{"filename":"evals/analysis_tests/combined_traces_comparative.csv","content":"id,sub_category,trace1_path,trace2_path,reference_dir,platform,platform2,capture_folder1,capture_folder2\ntest01_moe_fused_vs_unfused,moe,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test01_moe_fused_vs_unfused/trace1.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test01_moe_fused_vs_unfused/trace2.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test01_moe_fused_vs_unfused/analysis_output_ref,MI300X,H100,,\ntest02_attention_both_unfused,attention,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test02_attention_both_unfused/trace1.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test02_attention_both_unfused/trace2.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test02_attention_both_unfused/analysis_output_ref,MI300X,H100,,\ntest03_attention_fused_vs_unfused,attention,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test03_attention_fused_vs_unfused/trace1.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test03_attention_fused_vs_unfused/trace2.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test03_attention_fused_vs_unfused/analysis_output_ref,MI300X,H100,,\ntest04_gemm_mm_vs_linear,gemm,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test04_gemm_mm_vs_linear/trace1.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test04_gemm_mm_vs_linear/trace2.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test04_gemm_mm_vs_linear/analysis_output_ref,MI300X,H100,,\ntest05_gemm_identical_runtime_roofline,gemm,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test05_gemm_identical_runtime_roofline/trace1.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test05_gemm_identical_runtime_roofline/trace2.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test05_gemm_identical_runtime_roofline/analysis_output_ref,MI300X,H100,,\ntest06_conv_gap_exceeds_roofline,conv,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test06_conv_gap_exceeds_roofline/trace1.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test06_conv_gap_exceeds_roofline/trace2.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_comparative/test06_conv_gap_exceeds_roofline/analysis_output_ref,MI300X,H100,,\nbert_small,full_model,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_comparative/bert_small/gaunernst_bert-small-uncased__1016001_mi300.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_comparative/bert_small/gaunernst_bert-small-uncased__1016001_h100.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_comparative/bert_small/analysis_output_ref,MI300X,H100,,\nllama3_2,full_model,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_comparative/llama3_2/llama3_2_mi300.pt.trace_mi300.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_comparative/llama3_2/llama3_2_h100.pt.trace_h100.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_comparative/llama3_2/analysis_output_ref,MI300X,H100,,\ntimesformer,full_model,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_comparative/timesformer/facebook_timesformer-base-finetuned-k400__1016002_mi300.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_comparative/timesformer/facebook_timesformer-base-finetuned-k400__1016002_h100.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_comparative/timesformer/analysis_output_ref,MI300X,H100,,\ngptoss_graph_capture,full_model,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_comparative/gptoss_graph_capture/vllm_v21_mi300.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_comparative/gptoss_graph_capture/vllm_v19_mi300.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_comparative/gptoss_graph_capture/analysis_output_ref,MI300X,MI300X,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_comparative/gptoss_graph_capture/capture_folder_v21,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_comparative/gptoss_graph_capture/capture_folder_v19","content_type":"text/csv; charset=utf-8","language":"csv","size":5191,"content_sha256":"5174580a890f63aa283e4e2f91a09916967acfe6438c2b87c8d7b8b8752ea9cb"},{"filename":"evals/analysis_tests/combined_traces_standalone.csv","content":"id,sub_category,trace_path,reference_dir,platform\ngemm_01_compute_few_tiles,gemm,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/gemm_01_compute_few_tiles/gemm_01_compute_few_tiles.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/gemm_01_compute_few_tiles/analysis_output_ref,MI300X\ngemm_02_compute_non_pow2,gemm,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/gemm_02_compute_non_pow2/gemm_02_compute_non_pow2.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/gemm_02_compute_non_pow2/analysis_output_ref,MI300X\ngemm_03_compute_misaligned,gemm,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/gemm_03_compute_misaligned/gemm_03_compute_misaligned.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/gemm_03_compute_misaligned/analysis_output_ref,MI300X\ngemm_04_memory_tall_skinny,gemm,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/gemm_04_memory_tall_skinny/gemm_04_memory_tall_skinny.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/gemm_04_memory_tall_skinny/analysis_output_ref,MI300X\nsdpa_01_long_seq_low_eff,attention,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/sdpa_01_long_seq_low_eff/sdpa_01_long_seq_low_eff.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/sdpa_01_long_seq_low_eff/analysis_output_ref,MI300X\nsdpa_02_very_long_seq_head_waste,attention,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/sdpa_02_very_long_seq_head_waste/sdpa_02_very_long_seq_head_waste.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/sdpa_02_very_long_seq_head_waste/analysis_output_ref,MI300X\nsdpa_03_short_seq_overhead,attention,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/sdpa_03_short_seq_overhead/sdpa_03_short_seq_overhead.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/sdpa_03_short_seq_overhead/analysis_output_ref,MI300X\nsdpa_04_large_head_dim,attention,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/sdpa_04_large_head_dim/sdpa_04_large_head_dim.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/sdpa_04_large_head_dim/analysis_output_ref,MI300X\nelem_01_add_poor_bw,elementwise,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/elem_01_add_poor_bw/elem_01_add_poor_bw.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/elem_01_add_poor_bw/analysis_output_ref,MI300X\nconv_01_large_spatial_small_ch,conv,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/conv_01_large_spatial_small_ch/conv_01_large_spatial_small_ch.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/conv_01_large_spatial_small_ch/analysis_output_ref,MI300X\nconv_02_small_spatial_many_ch,conv,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/conv_02_small_spatial_many_ch/conv_02_small_spatial_many_ch.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/conv_02_small_spatial_many_ch/analysis_output_ref,MI300X\nconv_03_asymmetric_kernel,conv,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/conv_03_asymmetric_kernel/conv_03_asymmetric_kernel.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/conv_03_asymmetric_kernel/analysis_output_ref,MI300X\nconv_04_depthwise,conv,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/conv_04_depthwise/conv_04_depthwise.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/conv_04_depthwise/analysis_output_ref,MI300X\nconv_05_pointwise_1x1,conv,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/conv_05_pointwise_1x1/conv_05_pointwise_1x1.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/conv_05_pointwise_1x1/analysis_output_ref,MI300X\nconv_06_large_dilation,conv,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/conv_06_large_dilation/conv_06_large_dilation.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/conv_06_large_dilation/analysis_output_ref,MI300X\nconv_07_transposed,conv,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/conv_07_transposed/conv_07_transposed.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/conv_07_transposed/analysis_output_ref,MI300X\nmoe_01_many_experts_few_tokens,moe,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/moe_01_many_experts_few_tokens/moe_01_many_experts_few_tokens.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/moe_01_many_experts_few_tokens/analysis_output_ref,MI300X\nmoe_02_few_experts_many_tokens,moe,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/moe_02_few_experts_many_tokens/moe_02_few_experts_many_tokens.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/moe_02_few_experts_many_tokens/analysis_output_ref,MI300X\nmoe_03_top1_sparse,moe,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/moe_03_top1_sparse/moe_03_top1_sparse.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/moe_03_top1_sparse/analysis_output_ref,MI300X\nmoe_04_top2_dense,moe,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/moe_04_top2_dense/moe_04_top2_dense.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/moe_04_top2_dense/analysis_output_ref,MI300X\nmoe_05_large_capacity,moe,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/moe_05_large_capacity/moe_05_large_capacity.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/moe_05_large_capacity/analysis_output_ref,MI300X\nmoe_06_tiny_batch_many_experts,moe,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/moe_06_tiny_batch_many_experts/moe_06_tiny_batch_many_experts.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/moe_06_tiny_batch_many_experts/analysis_output_ref,MI300X\nqwen1.5_mi300,full_model,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/qwen1.5_mi300/Qwen_Qwen1.5-0.5B-Chat__1016005.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/qwen1.5_mi300/analysis_output_ref,MI300X\n16_owlv2_mi300,full_model,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/owlv2_mi300/google_owlv2-large-patch14-ensemble__1016001.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/owlv2_mi300/analysis_output_ref,MI300X\n01_act_checkpoint_mi300,full_model,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/act_checkpoint_mi300/rank0_iter16_18_act.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/act_checkpoint_mi300/analysis_output_ref,MI300X\n04_ddp_resnet_mi325,full_model,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/ddp_resnet_mi325/mi325_rank4_iter16_18.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/ddp_resnet_mi325/analysis_output_ref,MI325X\n06_llama3_2_mi300,full_model,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/llama3_2_mi300/llama3_2_mi300.pt.trace.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/llama3_2_mi300/analysis_output_ref,MI300X\n10_timesformer_mi300,full_model,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/timesformer_mi300/facebook_timesformer-base-finetuned-k400__1016002.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/timesformer_mi300/analysis_output_ref,MI300X\n12_nsfw_detection_mi300,full_model,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/nsfw_detection_mi300/Falconsai_nsfw_image_detection__1016002.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/nsfw_detection_mi300/analysis_output_ref,MI300X\n14_bert_small_mi300,full_model,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/bert_small_mi300/gaunernst_bert-small-uncased__1016001.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/bert_small_mi300/analysis_output_ref,MI300X\n21_llama_70b,full_model,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/llama_70b/rank0_trace_no_pyfn.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/llama_70b/analysis_output_ref,MI300X\ndlrm_05_mi300,full_model,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/dlrm_mi300/rank5.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/e2e_tests_standalone/dlrm_mi300/analysis_output_ref,MI300X\n","content_type":"text/csv; charset=utf-8","language":"csv","size":10541,"content_sha256":"1cd59fd5d63d6ef8f0bf7eb2d3423613831f63cca6210b763565832d04362168"},{"filename":"evals/EVAL_RUBRICS.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n# TraceLens Eval Rubrics\n\nSingle-source reference for every eval check, its pass/fail criteria, tolerances,\nroot-cause categories, and where the implementation lives.\n\n> **Canonical location:** `TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/EVAL_RUBRICS.md`\n> Update this file whenever an eval is added, removed, or its criteria change.\n\n---\n\n## Overview\n\n| Range | Type | Count | Engine |\n|-------|------|:-----:|--------|\n| workflow\\_eval\\_1–8 | Scripted (file/dir existence) | 8 | `workflow_scripted_evals.py` |\n| workflow\\_eval\\_9 | Scripted (per-header) | 7 sub-indices | `workflow_scripted_evals.py` |\n| workflow\\_eval\\_10 | Scripted (per-row + CSV cross-check) | 5 sub-indices | `workflow_scripted_evals.py` |\n| workflow\\_eval\\_11 | Scripted (per-P-item) | Dynamic | `workflow_scripted_evals.py` |\n| workflow\\_eval\\_12 | LLM (multi-dimensional scoring) | 1 | `skills/workflow-llm-eval/` |\n| workflow\\_eval\\_13 | Scripted (per-category) | Dynamic | `workflow_scripted_evals.py` |\n| quality\\_eval\\_1 | Scripted (CSV alignment) | 1 (standalone) / 2 (comparative) | `quality_scripted_evals.py` |\n| quality\\_eval\\_2 | LLM (multi-dimensional scoring) | 1 | `skills/quality-llm-eval/` |\n| quality\\_eval\\_3 | LLM (multi-dimensional scoring) | 1 | `skills/quality-llm-eval/` |\n| marker\\_eval\\_1 | Scripted (structural) | 1 | `workflow_scripted_evals.py` |\n| marker\\_eval\\_2 | Scripted (per-P-item) | Dynamic | `workflow_scripted_evals.py` |\n| marker\\_eval\\_3 | Scripted (per-P-item) | Dynamic | `workflow_scripted_evals.py` |\n\n**Total:** 15 scripted evals (expanding to ~30+ sub-indices) + 3 LLM evals = 18 logical evals.\n\n---\n\n## Pre-Check Gates\n\nBefore any eval runs, hard gates are checked in order. If any trips, **all**\nworkflow evals are set to FAIL with `root_cause=pipeline`.\n\n| Gate | Condition | Implemented in |\n|------|-----------|----------------|\n| 1 | `output_dir` does not exist | `workflow_scripted_evals.py` |\n| 2 | `analysis.md` missing or \u003c 100 bytes | `workflow_scripted_evals.py` |\n| 3 | `analysis.md` is garbled (> 50% non-ASCII) | `workflow_scripted_evals.py` |\n| 4 | `analysis.md` contains raw JSON instead of markdown | `workflow_scripted_evals.py` |\n| 5 | `output_dir` or `reference_dir` missing | `quality_scripted_evals.py` |\n| 6 | `perf_report_csvs/` (standalone) or `perf_report_trace1_csvs/` (comparative) missing in generated or reference | `quality_scripted_evals.py` |\n\n---\n\n## Workflow Evals 1–8: File & Directory Existence\n\nAll binary PASS/FAIL. Root cause is always `pipeline`.\n\n| Index | Summary | Pass Criteria | Key Details |\n|-------|---------|---------------|-------------|\n| `workflow_eval_1` | Directory structure created | `metadata/`, `category_data/`, `system_findings/`, `category_findings/` all exist | — |\n| `workflow_eval_2` | Metadata files exist on disk | Every `metadata_file` listed in `category_manifest.json` exists on disk | Requires manifest |\n| `workflow_eval_3` | Model info JSON exists and valid | `metadata/model_info.json` exists, is valid JSON, has keys `{model, architecture, scale, precision}`, none empty | — |\n| `workflow_eval_4` | Unified perf report exists | `perf_report_csvs/unified_perf_summary.csv` exists (standalone) or `perf_report_trace1_csvs/unified_perf_summary.csv` (comparative) | — |\n| `workflow_eval_5` | Tree data files exist on disk | Every `tree_data_file` listed in `category_manifest.json` exists on disk | Requires manifest |\n| `workflow_eval_6` | Categorical findings .md files exist | For every category in manifest (except `cpu_idle` when idle ≤ 15%), the corresponding `*_findings.md` exists in the correct tier directory | — |\n| `workflow_eval_7` | All findings correctly placed | No findings file is in the wrong tier directory (`system_findings/` vs `category_findings/`) | — |\n| `workflow_eval_8` | Plot generated on disk | `perf_improvement.png` exists **OR** no kernel tuning recommendations exist (legitimately skipped) | Checks `plot_data.json` and `*_metrics.json` for zero recommendations |\n\n---\n\n## Workflow Eval 9: Report Template Rendering\n\n**Type:** Scripted, per-header. **Root cause on fail:** `template`.\n\nChecks `analysis.md` for required `##` section headers and a metrics table.\n\n| Sub-index | Pass Criteria |\n|-----------|---------------|\n| `workflow_eval_9_executive_summary` | `## Executive Summary` header present (regex: `^## Executive Summary`) |\n| `workflow_eval_9_compute` | `## Compute Kernel Optimizations` header present |\n| `workflow_eval_9_system` | `## System-Level Optimizations` header present |\n| `workflow_eval_9_detailed` | `## Detailed Analysis` header present |\n| `workflow_eval_9_appendix` | `## Appendix` header present |\n| `workflow_eval_9_metrics_table` | Executive Summary section contains at least one markdown table row (`\\|.*\\|`) |\n| `workflow_eval_9_kernel_fusion` | `## Kernel Fusion Opportunities (Experimental)` header present |\n\n---\n\n## Workflow Eval 10: Executive Summary Metrics Table\n\n**Type:** Scripted, per-row + CSV cross-check. **Root cause on fail:** `template`.\n\nParses the metrics table in `## Executive Summary` and optionally cross-checks numeric values against `gpu_timeline.csv`.\n\n### Standalone (3-column table: Metric | Value)\n\nCross-checks against `perf_report_csvs/gpu_timeline.csv`.\n\n| Sub-index | Row Labels (synonyms) | CSV Cross-Check | Tolerance |\n|-----------|----------------------|-----------------|-----------|\n| `workflow_eval_10_total_time` | \"Total Compute Time\", \"Total Time\" | — | — |\n| `workflow_eval_10_compute_pct` | \"Computation\", \"Compute %\", \"Compute\" | `computation_time` percent | ±1.0% absolute |\n| `workflow_eval_10_idle_pct` | \"Idle Time\", \"Idle %\", \"Idle\" | `idle_time` percent | ±1.0% absolute |\n| `workflow_eval_10_comm_pct` | \"Exposed Communication\", \"Exposed Communication %\" | — | — |\n| `workflow_eval_10_bottleneck` | \"Top Bottleneck Category\", \"Top Bottleneck\" | — | — |\n\n### Comparative (4-column table: Metric | T1 | T2 | Difference)\n\nCross-checks T1 values against `perf_report_trace1_csvs/gpu_timeline.csv` and T2 values against `perf_report_trace2_csvs/gpu_timeline.csv`. Same 5 sub-indices as standalone apply, but fail if a T1 or T2 cell is empty for a row that has a CSV cross-check.\n\nAdditionally, for every numeric row: verifies `|reported_diff − (T1 − T2)| / max(|T1|, |T2|) ≤ 0.01`. Non-numeric rows (e.g., Top Bottleneck Category) are skipped.\n\n**Pass:** Row found in table. If CSV cross-check applies, report value within tolerance of CSV value.\n**Fail:** Row not found, or numeric value exceeds tolerance.\n\n---\n\n## Workflow Eval 11: Issue Template Rendering\n\n**Type:** Scripted, per-P-item. **Root cause on fail:** `template`.\n\nFinds every priority item (`### ...P{N}:` headers) and checks for required bold fields.\n\n| Section | Required Fields | Example Sub-index |\n|---------|----------------|-------------------|\n| `## Compute Kernel Optimizations` | `**Insight**` or `**Issue**`, `**Action**`, `**Impact**` | `workflow_eval_11_compute_P1` |\n| `## System-Level Optimizations` | `**Insight**` or `**Issue**`, `**Action**` (no Impact) | `workflow_eval_11_system_P1` |\n\n**Pass:** All required bold fields present in the P-item block.\n**Fail:** Any required field missing. Details list which fields are absent.\n\n**Note:** Either `**Insight**` or `**Issue**` is acceptable — both are valid as the first field.\n\n---\n\n## Workflow Eval 12: Hardware Reference in Appendix\n\n**Type:** LLM, multi-dimensional weighted scoring. **Root cause on fail:** `template`.\n\n**Implementation:** `TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/skills/workflow-llm-eval/SKILL.md` (see `reference.md` for rubric detail)\n\nChecks the `## Appendix` section for hardware reference values (only for trace1 if in comparative mode):\n- Platform name (e.g., MI300X)\n- Peak memory BW value (e.g., 5.3 TB/s)\n- At least one Peak MAF value (e.g., 708 TFLOPS)\n\n### Scoring Dimensions\n\n| Dimension | Weight | Scale |\n|-----------|--------|-------|\n| **correctness** | 50% | 10 = all values correct and plausible, 7 = present but imprecise, 0 = wrong/invented |\n| **completeness** | 50% | 10 = all 3 items present, 7 = 2 of 3, 4 = 1 of 3, 0 = none |\n\n### Pass/Fail\n\n```\noverall_score = correctness × 0.50 + completeness × 0.50\n```\n\n- **FAIL** if correctness = 0\n- **FAIL** if overall\\_score \u003c 7.0\n- **PASS** otherwise\n\n**Details format:** `correctness=N/10 completeness=N/10 overall=N.N | \u003cexplanation>`\n\n---\n\n## Workflow Eval 13: Model Identification in Report\n\n**Type:** Scripted, per-field. **Root cause on fail:** `template`.\n\nReads `metadata/model_info.json` and checks that each field value appears\n(case-insensitive substring match) in the `## Appendix` section of the report.\n\n| Sub-index | JSON Field | Pass Criteria |\n|-----------|-----------|---------------|\n| `workflow_eval_13_model` | `model` | Value appears in Appendix text |\n| `workflow_eval_13_architecture` | `architecture` | Value appears in Appendix text |\n| `workflow_eval_13_scale` | `scale` | Value appears in Appendix text |\n| `workflow_eval_13_precision` | `precision` | Value appears in Appendix text |\n\n**Special case:** If a field value is empty or `\"Cannot be inferred from trace\"`,\nthe check is **skipped (PASS)** with a note in details.\n\n---\n\n## Quality Eval 1: Perf Report CSV Alignment\n\n**Type:** Scripted. **Root cause on fail:** `data`.\n\n**Implementation:** `quality_scripted_evals.py`\n\nCompares CSVs against the reference directory.\n\n| Check | Criteria |\n|-------|----------|\n| File presence | Every reference CSV must exist in generated output |\n| Column presence | All non-optional reference columns must exist (optional prefixes: `Pct Roofline`, `Roofline Time`) |\n| Row count | Generated row count must match reference |\n| Numeric columns | Relative diff ≤ 1% **AND** absolute diff ≤ 0.05 (both must exceed to fail) |\n| String columns | Exact match after stripping numpy type wrappers (e.g., `np.int64(135)` → `135`) |\n\n**Pass:** All reference CSVs match within tolerances.\n**Fail:** Details list up to 5 mismatches.\n\n---\n\n## Quality Eval 2: Compute Issue Title Alignment\n\n**Type:** LLM, multi-dimensional weighted scoring. **Root cause on fail:** `data`.\n\n**Implementation:** `TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/skills/quality-llm-eval/SKILL.md` (see `reference.md` for rubric detail)\n\nCompares P-item titles in the generated report against the reference report.\nThis is a **semantic** comparison — not a string match.\n\n### Scoring Dimensions\n\n| Dimension | Weight | Scale |\n|-----------|--------|-------|\n| **correctness** | 40% | 10 = same bottleneck identified (e.g., both say \"GEMM low CU occupancy\"), 7 = same category and root cause but framed differently (e.g., ref says \"low CU occupancy due to small tiles\" vs generated says \"insufficient tile coverage on CUs\"), 4 = same category but different root cause (e.g., ref says \"occupancy\" vs generated says \"memory bandwidth\"), 0 = wrong category or fabricated bottleneck |\n| **completeness** | 30% | 10 = all reference P-items matched, 7 = one miss, 4 = several misses, 0 = most unmatched |\n| **precision** | 30% | 10 = same priority level, 7 = off by 1, 4 = off by 2+, 0 = completely different |\n\n### Pass/Fail\n\n```\noverall_score = correctness × 0.40 + completeness × 0.30 + precision × 0.30\n```\n\n- **FAIL** if correctness = 0\n- **FAIL** if overall\\_score \u003c 7.0\n- **PASS** otherwise\n\n**Details format:** `correctness=N/10 completeness=N/10 precision=N/10 overall=N.N | \u003cexplanation>`\n\n---\n\n## Quality Eval 3: Compute Issue Content Alignment\n\n**Type:** LLM, multi-dimensional weighted scoring. **Root cause on fail:** `data`.\n\n**Implementation:** `TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/skills/quality-llm-eval/SKILL.md` (see `reference.md` for rubric detail)\n\nFor each matched compute P-item pair (from eval 2), compares content values.\n**Only compares Compute Kernel P-items** — System-Level P-items are skipped entirely.\n\n### Value Tolerances\n\n| Value Type | Tolerance | Special Rules |\n|-----------|-----------|---------------|\n| Performance numbers (kernel time, efficiency %) | 2% relative | — |\n| Shapes (matrix dimensions, batch sizes) | Exact match | — |\n| Gap to roofline (efficiency %) | 2% relative | — |\n| Estimated savings (ms) | 2% relative | If savings \u003c 5 ms, accept ≤ 1 ms absolute diff |\n| Non-numeric gains (\"Not quantifiable\") | Semantic match | Mismatch only if one side numeric, other not |\n| Diff values — cross-trace deltas (comparative only) | 2% relative | e.g., \"−12.3 ms\" or \"2.1× slower\" |\n\n### Scoring Dimensions\n\n| Dimension | Weight | Scale |\n|-----------|--------|-------|\n| **correctness** | 40% | 10 = all values match, 7 = minor discrepancies, 0 = wrong values |\n| **completeness** | 30% | 10 = all fields present (time, efficiency, shapes, gains), 7 = one missing, 0 = most missing |\n| **precision** | 30% | See precision scale below |\n\n**Precision scale** (based on the [Value Tolerances](#value-tolerances) above):\n\n| Score | Meaning |\n|-------|---------|\n| 10 | All numeric values within 2% relative tolerance |\n| 7 | Most values within 2%; 1–2 values between 2–5% off |\n| 4 | Several values 2–5% off, or 1–2 values >5% off |\n| 0 | Most values >5% off, or systematically wrong |\n\n### Pass/Fail\n\n```\noverall_score = correctness × 0.40 + completeness × 0.30 + precision × 0.30\n```\n\n- **FAIL** if correctness = 0\n- **FAIL** if overall\\_score \u003c 7.0\n- **PASS** otherwise\n\n**Details format:** `correctness=N/10 completeness=N/10 precision=N/10 overall=N.N | \u003cexplanation>`\n\n---\n\n## Marker Identification Evals\n\n**Type:** Scripted (structural / deterministic). **Root cause on fail:** `template`.\n\n**Implementation:** `TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_utils/workflow_scripted_evals.py`\n\nValidates the structural correctness of impact markers (`\u003c!-- impact-begin ... -->` /\n`\u003c!-- impact-end -->`) in generated `analysis.md` files. These checks are purely\nregex-based and do not validate numeric correctness of marker attribute values.\n\n### marker\\_eval\\_1: Top Operations Markers\n\nChecks `analysis.md` for:\n\n| Check | Pass Criteria |\n|-------|---------------|\n| `kind=top_ops` wrapper | A `\u003c!-- impact-begin kind=top_ops -->` / `\u003c!-- impact-end -->` pair exists |\n| Inline row markers | Each table row within the wrapper has a `\u003c!-- top-ops-row low=... high=... -->` marker |\n| Row marker attributes | Each `top-ops-row` marker contains both `low` and `high` attributes |\n\n**Pass:** All structural checks satisfied.\n**Fail:** Any marker missing or malformed. Details list specific issues.\n\n### marker\\_eval\\_2: P-item Impact Markers\n\n**Sub-indices:** `marker_eval_2_P{N}` (one per compute P-item)\n\nFor each P-item (`### ...P{N}:`) under `## Compute Kernel Optimizations`:\n\n| Check | Pass Criteria |\n|-------|---------------|\n| `kind=p_item` marker present | A `\u003c!-- impact-begin kind=p_item ... -->` / `\u003c!-- impact-end -->` pair exists |\n| Required attributes | Marker contains `category`, `low`, `mid`, `high` attributes |\n| Pairing | Every `impact-begin` has a matching `impact-end` |\n\n**Pass:** All structural checks satisfied for the P-item.\n**Fail:** Marker missing, attributes missing, or unpaired. Details list specific issues.\n\n### marker\\_eval\\_3: Detail Estimate Markers\n\n**Sub-indices:** `marker_eval_3_P{N}` (one per compute P-item in Detailed Analysis) and `marker_eval_3_fusion_P{N}` (one per P-item under `### Kernel Fusion Insights`)\n\nFor each compute P-item section under `## Detailed Analysis`, and each fusion P-item under `### Kernel Fusion Insights`:\n\n| Check | Pass Criteria |\n|-------|---------------|\n| `kind=detail_estimate` marker or sentinel | Either a `\u003c!-- impact-begin kind=detail_estimate ... -->` / `\u003c!-- impact-end -->` pair exists, or the text \"not quantifiable from trace data\" is present |\n| Required attributes | If marker present, it contains `low`, `mid` and `high` attributes (compute and fusion) |\n| Pairing | Every `impact-begin` has a matching `impact-end` |\n\n**Pass:** Marker with correct attributes present, or not-quantifiable sentinel present.\n**Fail:** Neither marker nor sentinel found, or marker has missing attributes. Details list specific issues.\n\n---\n\n## Stability Classification\n\nAfter multiple repeated runs, the aggregation script (`aggregate_repeatability.py`)\nclassifies each (trace, eval\\_index) pair:\n\n| Classification | Criteria |\n|----------------|----------|\n| `STABLE_PASS` | 100% of runs passed |\n| `FLAKY_PASS` | > 50% passed but not all |\n| `FLAKY_FAIL` | > 0% passed but ≤ 50% |\n| `STABLE_FAIL` | 0% of runs passed |\n\n---\n\n## Root Cause Categories\n\nEvery FAIL row includes a `root_cause` field for triage:\n\n| Root Cause | Meaning | Typical Fix |\n|-----------|---------|-------------|\n| `pipeline` | Analysis pipeline didn't produce expected output | Re-run pipeline step or fix subagent |\n| `template` | Report formatting / structure issue | Fix report assembly logic |\n| `data` | Generated data doesn't match reference | Regenerate golden refs or fix data pipeline |\n\n---\n\n## CSV Output Schema\n\nAll eval results use the same 7-column schema:\n\n```\nindex,category,issue_summary,result,details,root_cause,recommended_fix\n```\n\n| Column | Description |\n|--------|-------------|\n| `index` | Eval identifier (e.g., `workflow_eval_9_compute`, `quality_eval_2`) |\n| `category` | `Workflow`, `Quality`, or `Marker Identification` |\n| `issue_summary` | Human-readable name of the check |\n| `result` | `PASS` or `FAIL` |\n| `details` | Failure specifics, scoring breakdown, or match confirmation |\n| `root_cause` | `pipeline`, `template`, `data`, or empty if PASS |\n| `recommended_fix` | Actionable fix suggestion, or empty if PASS |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":17847,"content_sha256":"4a59f845107074f04424685cdbd25d739aa542786e2869db5123a4cca3b2364a"},{"filename":"evals/eval_scripts/generate_ref.sh","content":"#!/usr/bin/env bash\nset -uo pipefail\n\n# Tags this process and all descendants with TRACELENS_EVAL_JOB=1.\nif [[ \"${TRACELENS_EVAL_JOB:-}\" != 1 ]]; then\n export TRACELENS_EVAL_JOB=1\n exec bash \"$0\" \"$@\"\nfi\n\n# ---------------------------------------------------------------------------\n# Usage: bash generate_ref.sh [standalone|comparative]\n#\n# COMPARISON_SCOPE can also be set via the COMPARISON_SCOPE environment variable.\n# Defaults to standalone.\n#\n# USE_PI=1 (or --pi on run_repeatability_parallel.sh) uses `pi` instead of the\n# Cursor `agent` CLI, matching the repeatability harness agent backend.\n#\n# NUM_REPEATS retries each failed reference generation (default: 3). When invoked\n# from run_repeatability_parallel.sh, NUM_REPEATS is passed through from the harness.\n# ---------------------------------------------------------------------------\nCOMPARISON_SCOPE=\"${1:-${COMPARISON_SCOPE:-standalone}}\"\n\nif [[ \"$COMPARISON_SCOPE\" != \"standalone\" && \"$COMPARISON_SCOPE\" != \"comparative\" ]]; then\n echo \"ERROR: Unknown comparison scope '$COMPARISON_SCOPE'. Use 'standalone' or 'comparative'.\" >&2\n exit 1\nfi\n\n# ---------------------------------------------------------------------------\n# Configuration (run from repo root on the node)\n# ---------------------------------------------------------------------------\nCONTAINER=\"${CONTAINER:-}\"\nMAX_PARALLEL=\"${MAX_PARALLEL:-5}\"\nSLEEP_BETWEEN=\"${SLEEP_BETWEEN:-30}\"\nLAUNCH_STAGGER=\"${LAUNCH_STAGGER:-8}\"\nTEST_IDS=\"${TEST_IDS:-}\"\nNUM_REPEATS=\"${NUM_REPEATS:-3}\"\nUSE_PI=\"${USE_PI:-false}\"\nAGENT_MODEL=\"${AGENT_MODEL:-claude-opus-4-8-thinking-medium}\"\nPI_VENV_PREFIX=\"use venv_tracelens for all commands and tool calls. \"\n\nREPO_ROOT=\"${REPO_ROOT:-$(pwd)}\"\nANALYSIS_DIR=\"$REPO_ROOT/TraceLens/Agent/Analysis\"\nEVALS_DIR=\"$REPO_ROOT/TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals\"\nTEST_TRACES_CSV=\"${TEST_TRACES_CSV:-$EVALS_DIR/analysis_tests/combined_traces_${COMPARISON_SCOPE}.csv}\"\nSTATUS_FILE=\"$(mktemp)\"\n\nif [[ -n \"$CONTAINER\" ]]; then\n DEXEC=(docker exec -w \"$REPO_ROOT\" \"$CONTAINER\")\n RUNTIME_LABEL=\"container $CONTAINER\"\n NODE_LABEL=\"node $(hostname)\"\nelse\n DEXEC=()\n RUNTIME_LABEL=\"host (no container)\"\n NODE_LABEL=\"local\"\nfi\n\nif [[ \"$USE_PI\" == true || \"$USE_PI\" == 1 || \"$USE_PI\" == \"1\" ]]; then\n USE_PI=true\n AGENT_BACKEND=\"pi\"\nelse\n USE_PI=false\n AGENT_BACKEND=\"cursor agent\"\nfi\n\n# ---------------------------------------------------------------------------\n# Auto-extract test archives if trace CSV references them\n# ---------------------------------------------------------------------------\nfor archive in \"$EVALS_DIR\"/analysis_tests/e2e_tests_${COMPARISON_SCOPE}.tar.gz \"$EVALS_DIR\"/analysis_tests/unit_tests_${COMPARISON_SCOPE}.tar.gz; do\n [ -f \"$archive\" ] || continue\n target_dir=\"${archive%.tar.gz}\"\n if [ ! -d \"$target_dir\" ]; then\n echo \"Extracting $(basename \"$archive\")...\"\n # Archives store paths as agent_evals/Analysis/analysis_tests/...\n tar -xzf \"$archive\" -C \"$EVALS_DIR\" --strip-components=2\n fi\ndone\n\n# ---------------------------------------------------------------------------\n# Helpers\n# ---------------------------------------------------------------------------\n\nts() { date \"+%H:%M:%S\"; }\n\nlog_status() {\n flock 1 echo \"$@\"\n}\n\nrepo_abs_path() {\n local p=\"$1\"\n if [[ \"$p\" = /* ]]; then\n echo \"$p\"\n else\n echo \"$REPO_ROOT/$p\"\n fi\n}\n\n# Run an LLM agent step. Optional second arg: non-empty enables a 1800s timeout (cursor only).\nrun_llm_agent() {\n local prompt=\"$1\"\n local with_timeout=\"${2:-}\"\n\n if [[ \"$USE_PI\" == true ]]; then\n pi --mode json \"${PI_VENV_PREFIX}${prompt}\"\n elif [[ -n \"$with_timeout\" ]]; then\n timeout 1800 agent --model \"$AGENT_MODEL\" --print --force --trust --output-format stream-json \\\n \"$prompt\"\n else\n agent --model \"$AGENT_MODEL\" --print --force --trust --output-format stream-json \\\n \"$prompt\"\n fi\n}\n\n# ---------------------------------------------------------------------------\n# Single job: generate one golden reference\n#\n# Args: id trace1_path trace2_path reference_dir platform\n# For standalone, trace2_path is empty string \"\".\n# ---------------------------------------------------------------------------\n\ngenerate_single_ref() {\n local id=\"$1\" trace1_path=\"$2\" trace2_path=\"$3\" reference_dir=\"$4\" platform=\"$5\" platform2=\"$6\" capture_folder1=\"${7:-}\" capture_folder2=\"${8:-}\"\n local tag=\"[$id]\"\n\n local REF_DIR=\"$REPO_ROOT/$reference_dir\"\n local CASE_DIR\n CASE_DIR=\"$(dirname \"$REF_DIR\")\"\n local OUTPUT_DIR=\"$CASE_DIR/analysis_output\"\n\n trace1_path=\"$(repo_abs_path \"$trace1_path\")\"\n if [[ -n \"$trace2_path\" ]]; then\n trace2_path=\"$(repo_abs_path \"$trace2_path\")\"\n fi\n if [[ -n \"$capture_folder1\" ]]; then\n capture_folder1=\"$(repo_abs_path \"$capture_folder1\")\"\n fi\n if [[ -n \"$capture_folder2\" ]]; then\n capture_folder2=\"$(repo_abs_path \"$capture_folder2\")\"\n fi\n\n # Verify trace file(s) exist\n if [ ! -f \"$trace1_path\" ]; then\n log_status \" $tag ERROR: Trace file not found: $trace1_path — skipping.\"\n flock \"$STATUS_FILE\" bash -c \"echo 'failed' >> '$STATUS_FILE'\"\n return 1\n fi\n if [[ \"$COMPARISON_SCOPE\" == \"comparative\" ]] && [ ! -f \"$trace2_path\" ]; then\n log_status \" $tag ERROR: Trace2 file not found: $trace2_path — skipping.\"\n flock \"$STATUS_FILE\" bash -c \"echo 'failed' >> '$STATUS_FILE'\"\n return 1\n fi\n\n local ref_attempt=0\n while (( ref_attempt \u003c NUM_REPEATS )); do\n ref_attempt=$((ref_attempt + 1))\n if (( ref_attempt > 1 )); then\n log_status \" $tag [$(ts)] Retry $ref_attempt/$NUM_REPEATS...\"\n \"${DEXEC[@]}\" rm -rf \"$OUTPUT_DIR\" 2>/dev/null || rm -rf \"$OUTPUT_DIR\"\n rm -f \"$CASE_DIR/analysis_stream.ndjson\"\n sleep \"$SLEEP_BETWEEN\"\n else\n log_status \" $tag [$(ts)] Generating golden reference...\"\n fi\n \"${DEXEC[@]}\" bash -c \"mkdir -p $OUTPUT_DIR && chmod -R 777 $OUTPUT_DIR\"\n\n # Run analysis with retry + backoff on transient agent errors\n local agent_success=false\n local agent_attempts=0\n while [ \"$agent_success\" = false ] && [ \"$agent_attempts\" -lt 3 ]; do\n agent_attempts=$((agent_attempts + 1))\n (\n cd \"$ANALYSIS_DIR\" || exit\n if [[ \"$COMPARISON_SCOPE\" == \"comparative\" ]]; then\n local capture_suffix=\"\"\n [[ -n \"$capture_folder1\" ]] && capture_suffix+=\" capture folder for trace1 $capture_folder1\"\n [[ -n \"$capture_folder2\" ]] && capture_suffix+=\" capture folder for trace2 $capture_folder2\"\n local analysis_mode=\"default\"\n [[ -n \"$capture_folder1\" || -n \"$capture_folder2\" ]] && analysis_mode=\"inference\"\n run_llm_agent \\\n \"Follow the analysis orchestrator installed with the TraceLens pip package (look under TraceLens/Agent/Analysis/skills/analysis-orchestrator/ in the package installation directory) and run the full agentic analysis workflow on $trace1_path and $trace2_path${capture_suffix} with platform $platform (trace1) and $platform2 (trace2), analysis mode $analysis_mode, $NODE_LABEL, $RUNTIME_LABEL, output to $OUTPUT_DIR\" \\\n 1\n else\n run_llm_agent \\\n \"Follow the analysis orchestrator installed with the TraceLens pip package (look under TraceLens/Agent/Analysis/skills/analysis-orchestrator/ in the package installation directory) and run the full agentic analysis workflow on $trace1_path with platform $platform, analysis mode default, $NODE_LABEL, $RUNTIME_LABEL, output to $OUTPUT_DIR\" \\\n 1\n fi\n ) \u003c /dev/null > \"$CASE_DIR/analysis_stream.ndjson\" 2>&1\n\n if [[ \"$USE_PI\" == true ]]; then\n if head -c 2048 \"$CASE_DIR/analysis_stream.ndjson\" | grep -qiE 'Error:.*unavailable|Service Unavailable'; then\n log_status \" $tag Agent attempt $agent_attempts/3 failed (agent unavailable). Backing off 30s...\"\n sleep 30\n else\n agent_success=true\n fi\n elif grep -qiE 'Error:.*unavailable|Service Unavailable|usage limit|out of usage|You'\\''ve reached your' \"$CASE_DIR/analysis_stream.ndjson\" 2>/dev/null; then\n log_status \" $tag Agent attempt $agent_attempts/3 failed (agent unavailable or usage limit). Backing off 30s...\"\n sleep 30\n else\n agent_success=true\n fi\n done\n\n if [ \"$agent_success\" = false ]; then\n log_status \" $tag Reference attempt $ref_attempt/$NUM_REPEATS failed (agent unavailable or usage limit).\"\n \"${DEXEC[@]}\" rm -rf \"$OUTPUT_DIR\" 2>/dev/null || rm -rf \"$OUTPUT_DIR\"\n rm -f \"$CASE_DIR/analysis_stream.ndjson\"\n continue\n fi\n\n if [ ! -f \"$OUTPUT_DIR/analysis.md\" ]; then\n log_status \" $tag Reference attempt $ref_attempt/$NUM_REPEATS failed (analysis.md not found).\"\n \"${DEXEC[@]}\" rm -rf \"$OUTPUT_DIR\" 2>/dev/null || rm -rf \"$OUTPUT_DIR\"\n rm -f \"$CASE_DIR/analysis_stream.ndjson\"\n continue\n fi\n\n # Copy output as reference (remove old ref first, then copy contents directly)\n rm -rf \"$REF_DIR\"\n cp -r \"$OUTPUT_DIR\" \"$REF_DIR\"\n\n # Remove unwanted files from reference dir (keep only analysis.md + perf_report_csvs/)\n rm -rf \"$REF_DIR/category_data\" \\\n \"$REF_DIR/category_findings\" \\\n \"$REF_DIR/system_findings\" \\\n \"$REF_DIR/metadata\" \\\n \"$REF_DIR/cache\" \\\n \"$REF_DIR/perf_improvement.png\" \\\n \"$REF_DIR/perf_improvement_base64.txt\" \\\n \"$REF_DIR/plot_data.json\" \\\n \"$REF_DIR/perf_report.xlsx\" \\\n \"$REF_DIR/priority_data.json\"\n\n # Remove intermediate analysis output (docker-owned files need container cleanup)\n \"${DEXEC[@]}\" rm -rf \"$OUTPUT_DIR\"\n rm -f \"$CASE_DIR/analysis_stream.ndjson\"\n\n log_status \" $tag [$(ts)] Reference saved to $reference_dir (cleaned)\"\n flock \"$STATUS_FILE\" bash -c \"echo 'generated' >> '$STATUS_FILE'\"\n return 0\n done\n\n log_status \" $tag FAILED after $NUM_REPEATS reference generation attempt(s).\"\n flock \"$STATUS_FILE\" bash -c \"echo 'failed' >> '$STATUS_FILE'\"\n return 1\n}\n\n# ---------------------------------------------------------------------------\n# FIFO semaphore for concurrency control\n# ---------------------------------------------------------------------------\n\nFIFO=\"$(mktemp -u)\"\ncleanup() {\n rm -f \"$FIFO\" \"$STATUS_FILE\"\n}\n\nsetup_semaphore() {\n mkfifo \"$FIFO\"\n exec 4\u003c>\"$FIFO\"\n for ((t = 0; t \u003c MAX_PARALLEL; t++)); do echo >&4; done\n trap cleanup EXIT\n}\n\n# ---------------------------------------------------------------------------\n# Main\n# ---------------------------------------------------------------------------\n\necho \"=========================================\"\necho \" Golden Reference Generation\"\necho \" Mode: $COMPARISON_SCOPE\"\necho \" Agent: $AGENT_BACKEND\"\necho \" Node: $NODE_LABEL\"\necho \" Runtime: $RUNTIME_LABEL\"\necho \" Max parallel: $MAX_PARALLEL\"\necho \" Retries: $NUM_REPEATS\"\necho \" CSV: $TEST_TRACES_CSV\"\nif [[ -n \"$TEST_IDS\" ]]; then\n echo \" Test filter: $TEST_IDS\"\nfi\necho \"=========================================\"\necho \"\"\n\n# Return 0 if $id should run given TEST_IDS (empty = all).\n# Supports exact match and underscore-delimited prefix (e.g. gemm_01 -> gemm_01_compute_few_tiles).\nshould_run_id() {\n local id=\"$1\"\n [[ -z \"$TEST_IDS\" ]] && return 0\n local token\n for token in $TEST_IDS; do\n if [[ \"$id\" == \"$token\" || \"$id\" == \"${token}_\"* ]]; then\n return 0\n fi\n done\n return 1\n}\n\nsetup_semaphore\n\nif [[ \"$COMPARISON_SCOPE\" == \"comparative\" ]]; then\n # comparative CSV: id,sub_category,trace1_path,trace2_path,reference_dir,platform,platform2,capture_folder1,capture_folder2\n while IFS=, read -r id sub_category trace1_path trace2_path reference_dir platform platform2 capture_folder1 capture_folder2 \u003c&3; do\n [[ -z \"$id\" ]] && continue\n should_run_id \"$id\" || continue\n\n read -u4 # acquire semaphore slot\n (\n generate_single_ref \"$id\" \"$trace1_path\" \"$trace2_path\" \"$reference_dir\" \"$platform\" \"$platform2\" \"${capture_folder1:-}\" \"${capture_folder2:-}\" || true\n sleep \"$SLEEP_BETWEEN\"\n echo >&4 # release semaphore slot\n ) &\n sleep \"$LAUNCH_STAGGER\" # stagger agent startup to avoid ~/.cursor/cli-config.json rename race\n done 3\u003c \u003c(tail -n +2 \"$TEST_TRACES_CSV\"; echo)\nelse\n # standalone CSV: id,sub_category,trace_path,reference_dir,platform\n while IFS=, read -r id sub_category trace_path reference_dir platform \u003c&3; do\n [[ -z \"$id\" ]] && continue\n should_run_id \"$id\" || continue\n\n read -u4 # acquire semaphore slot\n (\n generate_single_ref \"$id\" \"$trace_path\" \"\" \"$reference_dir\" \"$platform\" \"\" || true\n sleep \"$SLEEP_BETWEEN\"\n echo >&4 # release semaphore slot\n ) &\n sleep \"$LAUNCH_STAGGER\" # stagger agent startup to avoid ~/.cursor/cli-config.json rename race\n done 3\u003c \u003c(tail -n +2 \"$TEST_TRACES_CSV\"; echo)\nfi\n\nwait\n\n# Tally results from status file\ngenerated=\"$(grep -c '^generated

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

\"$STATUS_FILE\" 2>/dev/null || true)\"\nfailed=\"$(grep -c '^failed

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

\"$STATUS_FILE\" 2>/dev/null || true)\"\ngenerated=\"${generated:-0}\"\nfailed=\"${failed:-0}\"\ntotal=$(( generated + failed ))\n\nif [[ -n \"$TEST_IDS\" && \"$total\" -eq 0 ]]; then\n echo \"\"\n echo \"WARNING: TEST_IDS='$TEST_IDS' matched no trace ids in $TEST_TRACES_CSV.\" >&2\n echo \" Use exact ids or underscore-delimited prefixes (e.g. gemm_01 -> gemm_01_compute_few_tiles).\" >&2\nfi\n\necho \"\"\necho \"=========================================\"\necho \" Golden Reference Generation Complete\"\necho \" Total: $total | Generated: $generated | Failed: $failed\"\necho \"=========================================\"\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":14339,"content_sha256":"49928a54ceb0b9313672aeb283ca6e30f0d480c3288899a3a898134278351828"},{"filename":"evals/eval_scripts/kill_eval_jobs.sh","content":"#!/usr/bin/env bash\n###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\nset -uo pipefail\n\n# ---------------------------------------------------------------------------\n# Kill every process the eval pipeline may have spawned.\n# Covers jobs started by generate_ref.sh and run_repeatability_parallel.sh.\n#\n# Usage:\n# bash kill_eval_jobs.sh # SIGTERM, then SIGKILL, rescanning until clear\n# bash kill_eval_jobs.sh --list # show matching processes, kill nothing\n# bash kill_eval_jobs.sh -9 # SIGKILL from the first pass (no grace period)\n# ---------------------------------------------------------------------------\n\nusage() {\n cat \u003c\u003c'EOF'\nUsage: bash kill_eval_jobs.sh [--list|-9|-h]\n\n (no args) Terminate matching jobs: SIGTERM, then SIGKILL, rescanning until\n no marked processes remain (bounded number of passes).\n --list List matching processes without killing anything.\n -9 SIGKILL from the first pass (skip the graceful SIGTERM).\n -h,--help Show this help.\n\nKills processes spawned by generate_ref.sh and run_repeatability_parallel.sh:\nthe scripts themselves, their subshells, agent/pi analysis + eval calls (and the\nnode/python workers underneath), and the scripted eval + merge python. Scoped to\nthe current user; unrelated agent/claude/python sessions are not touched.\nEOF\n}\n\nMODE=\"kill\" # kill | list | force\ncase \"${1:-}\" in\n \"\" ) MODE=\"kill\" ;;\n --list ) MODE=\"list\" ;;\n -9 ) MODE=\"force\" ;;\n -h|--help ) usage; exit 0 ;;\n *) echo \"ERROR: Unknown argument '$1'.\" >&2; usage >&2; exit 1 ;;\nesac\n\nMAX_PASSES=6\n\n# Print the PIDs of this user's processes carrying the TRACELENS_EVAL_JOB=1\n# marker, one per line, unique + numerically sorted. environ entries are\n# NUL-separated, so grep -z matches across them. Guard on ownership (-O): only\n# this user's /proc entries are readable, but be explicit.\nscan_marked() {\n local envf pid\n for envf in /proc/[0-9]*/environ; do\n pid=\"${envf#/proc/}\"; pid=\"${pid%/environ}\"\n [[ \"$pid\" =~ ^[0-9]+$ ]] || continue\n [[ -O \"$envf\" ]] || continue\n if grep -qzs 'TRACELENS_EVAL_JOB=1' \"$envf\" 2>/dev/null; then\n printf '%s\\n' \"$pid\"\n fi\n done | sort -un\n}\n\n# --- list mode: show matches and exit, killing nothing ----------------------\nif [[ \"$MODE\" == \"list\" ]]; then\n mapfile -t UNIQ \u003c \u003c(scan_marked)\n if [[ ${#UNIQ[@]} -eq 0 ]]; then\n echo \"No eval jobs found.\"\n exit 0\n fi\n echo \"Matching eval jobs (${#UNIQ[@]}):\"\n for pid in \"${UNIQ[@]}\"; do\n printf ' %s %s\\n' \"$pid\" \"$(ps -o args= -p \"$pid\" 2>/dev/null | cut -c1-100)\"\n done\n exit 0\nfi\n\n# --- kill mode: scan + signal, rescanning until clear or MAX_PASSES ----------\nkilled_any=false\nfor ((pass = 1; pass \u003c= MAX_PASSES; pass++)); do\n mapfile -t UNIQ \u003c \u003c(scan_marked)\n if [[ ${#UNIQ[@]} -eq 0 ]]; then\n if [[ \"$killed_any\" == true ]]; then\n echo \"All eval jobs cleared.\"\n else\n echo \"No eval jobs found.\"\n fi\n exit 0\n fi\n\n # SIGTERM on the first pass (unless -9) to allow graceful shutdown; SIGKILL\n # on every pass thereafter (and always in force mode).\n if [[ \"$MODE\" == \"force\" || \"$pass\" -gt 1 ]]; then\n echo \"Pass $pass: SIGKILL ${#UNIQ[@]} job(s): ${UNIQ[*]}\"\n kill -9 \"${UNIQ[@]}\" 2>/dev/null\n else\n echo \"Pass $pass: SIGTERM ${#UNIQ[@]} job(s): ${UNIQ[*]}\"\n kill \"${UNIQ[@]}\" 2>/dev/null\n fi\n killed_any=true\n sleep 2\ndone\n\n# Final check after the last pass.\nmapfile -t UNIQ \u003c \u003c(scan_marked)\nif [[ ${#UNIQ[@]} -gt 0 ]]; then\n echo \"WARNING: ${#UNIQ[@]} job(s) still alive after $MAX_PASSES passes: ${UNIQ[*]}\" >&2\n echo \" (possibly stuck in uninterruptible I/O -- inspect with: ps -o pid,stat,args -p ${UNIQ[*]// /,})\" >&2\n exit 1\nfi\necho \"All eval jobs cleared.\"\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":4127,"content_sha256":"917707fe6789c0bab104315fbb2e63d1aa86295404f8067c9b1d1149f2b1ca3d"},{"filename":"evals/eval_scripts/pack_test_archives.sh","content":"#!/usr/bin/env bash\n###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\nset -euo pipefail\n\n# ---------------------------------------------------------------------------\n# Pack analysis_tests/{unit,e2e}_tests_{standalone,comparative} into tar.gz\n# archives that generate_ref.sh and run_repeatability_parallel.sh extract with:\n# tar xzf \u003carchive> -C \"$EVALS_DIR\" --strip-components=2\n#\n# Usage (from repo root):\n# bash TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_scripts/pack_test_archives.sh\n# bash TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_scripts/pack_test_archives.sh standalone\n# bash TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_scripts/pack_test_archives.sh unit comparative\n# bash TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_scripts/pack_test_archives.sh unit_tests_standalone\n# ---------------------------------------------------------------------------\n\nusage() {\n cat \u003c\u003c'EOF'\nUsage: bash pack_test_archives.sh [suite|scope|archive_name ...]\n\nWith no arguments, packs all four archives:\n unit_tests_standalone.tar.gz\n unit_tests_comparative.tar.gz\n e2e_tests_standalone.tar.gz\n e2e_tests_comparative.tar.gz\n\nArguments can be mixed:\n standalone | comparative both unit and e2e for that scope\n unit | e2e both scopes for that suite\n unit_tests_standalone that archive only (same for the other three names)\n\nArchives are written next to the source directories under\nTraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/. Intermediate analysis_output/ trees and\nanalysis_stream.ndjson files are excluded; analysis_output_ref/ is kept.\nEOF\n}\n\nREPO_ROOT=\"${REPO_ROOT:-$(pwd)}\"\nEVALS_DIR=\"$REPO_ROOT/TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals\"\nTESTS_DIR=\"$EVALS_DIR/analysis_tests\"\n\nALL_ARCHIVES=(\n unit_tests_standalone\n unit_tests_comparative\n e2e_tests_standalone\n e2e_tests_comparative\n)\n\nif [[ ! -d \"$TESTS_DIR\" ]]; then\n echo \"ERROR: test directory not found: $TESTS_DIR\" >&2\n echo \"Run this script from the TraceLens repo root (or set REPO_ROOT).\" >&2\n exit 1\nfi\n\nresolve_names() {\n local arg suite scope name\n local -a selected=()\n\n if [[ $# -eq 0 ]]; then\n printf '%s\\n' \"${ALL_ARCHIVES[@]}\"\n return 0\n fi\n\n for arg in \"$@\"; do\n case \"$arg\" in\n -h|--help)\n usage\n exit 0\n ;;\n standalone|comparative)\n selected+=(\"unit_tests_${arg}\" \"e2e_tests_${arg}\")\n ;;\n unit|e2e)\n selected+=(\"${arg}_tests_standalone\" \"${arg}_tests_comparative\")\n ;;\n unit_tests_standalone|unit_tests_comparative|e2e_tests_standalone|e2e_tests_comparative)\n selected+=(\"$arg\")\n ;;\n *.tar.gz)\n name=\"${arg##*/}\"\n name=\"${name%.tar.gz}\"\n selected+=(\"$name\")\n ;;\n *)\n echo \"ERROR: Unknown argument '$arg'.\" >&2\n usage >&2\n exit 1\n ;;\n esac\n done\n\n # Deduplicate while preserving order\n local -A seen=()\n for name in \"${selected[@]}\"; do\n [[ -n \"${seen[$name]:-}\" ]] && continue\n seen[$name]=1\n printf '%s\\n' \"$name\"\n done\n}\n\npack_one() {\n local name=\"$1\"\n local src=\"$TESTS_DIR/$name\"\n local archive=\"$TESTS_DIR/${name}.tar.gz\"\n local rel=\"analysis_tests/$name\"\n\n if [[ ! -d \"$src\" ]]; then\n echo \"ERROR: source directory not found: $src\" >&2\n return 1\n fi\n\n echo \"Packing $rel -> $archive\"\n\n # Archive members keep the legacy agent_evals/Analysis/ prefix, which the\n # extractors strip and the skill's evals.json prompt refers to.\n tar -czf \"$archive\" \\\n --exclude='analysis_output' \\\n --exclude='analysis_stream.ndjson' \\\n --exclude='__pycache__' \\\n --transform='s,^,agent_evals/Analysis/,' \\\n -C \"$EVALS_DIR\" \\\n \"$rel\"\n\n ls -lh \"$archive\"\n}\n\nmapfile -t NAMES \u003c \u003c(resolve_names \"$@\")\n\nif [[ ${#NAMES[@]} -eq 0 ]]; then\n echo \"ERROR: no archives selected.\" >&2\n exit 1\nfi\n\necho \"Repo root: $REPO_ROOT\"\necho \"\"\n\nfailed=0\nfor name in \"${NAMES[@]}\"; do\n pack_one \"$name\" || failed=$((failed + 1))\ndone\n\necho \"\"\nif [[ \"$failed\" -gt 0 ]]; then\n echo \"Finished with $failed failure(s).\" >&2\n exit 1\nfi\necho \"Packed ${#NAMES[@]} archive(s).\"\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":4732,"content_sha256":"176bfd4fee136941a2cfa269a96e7919e95ff2f3ab75dfbfd62cd7f7cd3a32e7"},{"filename":"evals/eval_scripts/run_pi_analysis_in_docker.sh","content":"#!/usr/bin/env bash\n###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\nset -euo pipefail\n\n# ---------------------------------------------------------------------------\n# Start an inference-server Docker container, install pi + TraceLens inside it,\n# configure pi from /v1/models, and run Analysis evals with --pi.\n#\n# Usage:\n# bash run_pi_analysis_in_docker.sh \u003ctracelens_root> [both|standalone|comparative] [options] -- \u003cserver-cmd...>\n#\n# Arguments after -- are the full command used to start the inference server\n# (executable plus flags). They are passed to the shell unchanged.\n#\n# Examples:\n# bash run_pi_analysis_in_docker.sh /data/tracelens_local_testing/tracelens -- \\\n# \u003cserver-cmd> Qwen/Qwen3.6-35B-A3B --port 30000 --tensor-parallel-size 8\n#\n# TEST_IDS=\"gemm_01\" NUM_REPEATS=1 \\ # prefix ok: matches gemm_01_compute_few_tiles\n# bash run_pi_analysis_in_docker.sh /data/tracelens_local_testing/tracelens standalone -- \\\n# \u003cserver-cmd> MiniMaxAI/MiniMax-M3 --port 30000\n#\n# Options:\n# --docker-image IMAGE Inference server image\n# --container-name NAME Docker container name (default: tracelens_pi_evals)\n# --work-dir DIR Host directory mounted at /workspace (default: \u003cparent>)\n# --hf-cache-dir DIR Host HuggingFace cache mounted at /root/.cache/huggingface\n# --hf-cache-chown chown HF cache mount to caller UID/GID on exit (default: off)\n# --port PORT API port when not set in server args (default: 30000)\n# -h, --help Show help\n#\n# Environment:\n# DOCKER_IMAGE Same as --docker-image\n# CONTAINER_NAME Same as --container-name\n# WORK_DIR Host dir mounted at /workspace (default: parent of tracelens_root)\n# HF_CACHE_DIR Host HuggingFace cache dir (same as --hf-cache-dir)\n# HF_CACHE_CHOWN=1 chown HF cache mount to caller UID/GID on exit (default: off)\n# PORT API port (default: 30000; overridden by server --port)\n# READY_TIMEOUT Seconds to wait for http://localhost:\u003cport>/v1/models (default: 1800)\n# DOCKER_RUN_ARGS Extra whitespace-separated docker run arguments\n# PI_NPM_PACKAGE npm package for pi (default: @earendil-works/pi-coding-agent)\n# SKIP_EVAL=1 Set up server + pi + TraceLens only; skip eval harness\n# SKIP_SERVER=1 Skip inference server (npm/pi/TraceLens setup only)\n# TEST_IDS, NUM_REPEATS, … Harness env vars passed to run_repeatability_parallel.sh\n# ---------------------------------------------------------------------------\n\nDOCKER_IMAGE=\"${DOCKER_IMAGE:-vllm/vllm-openai-rocm:nightly}\"\nCONTAINER_NAME=\"${CONTAINER_NAME:-tracelens_pi_evals}\"\nHF_CACHE_DIR=\"${HF_CACHE_DIR:-}\"\nHF_CACHE_CHOWN=\"${HF_CACHE_CHOWN:-0}\"\nCONTAINER_HF_HOME=\"/root/.cache/huggingface\"\nPORT=\"${PORT:-30000}\"\nREADY_TIMEOUT=\"${READY_TIMEOUT:-1800}\"\nSKIP_EVAL=\"${SKIP_EVAL:-0}\"\nSKIP_SERVER=\"${SKIP_SERVER:-0}\"\n\nDEFAULT_DOCKER_RUN_ARGS=(\n --device /dev/dri\n --device /dev/kfd\n --device /dev/infiniband\n --network host\n --ipc host\n --group-add video\n --cap-add SYS_PTRACE\n --security-opt seccomp=unconfined\n --privileged\n --shm-size 128G\n)\n\nusage() {\n cat \u003c\u003cEOF\nUsage: bash run_pi_analysis_in_docker.sh \u003ctracelens_root> [both|standalone|comparative] [options] -- \u003cserver-cmd...>\n\n tracelens_root TraceLens repo on the host (correct branch already checked out)\n both|standalone|comparative Harness comparison scope (default: both)\n\nOptions:\n --docker-image IMAGE Inference server Docker image (default: $DOCKER_IMAGE)\n --container-name NAME Docker container name (default: tracelens_pi_evals)\n --work-dir DIR Host directory mounted at /workspace (default: \u003cparent>)\n --hf-cache-dir DIR Host HuggingFace cache mounted at /root/.cache/huggingface\n --hf-cache-chown chown HF cache mount to caller UID/GID on exit (default: off)\n --port PORT API port when not set in server args (default: 30000)\n --setup-only Install npm, pi, and TraceLens only; skip server and evals\n -h, --help Show this help\n\n Arguments after -- are the full inference-server command (executable plus flags).\n Not required with --setup-only (or SKIP_SERVER=1).\n\nEnvironment:\n PORT API port (default: 30000)\n HF_CACHE_DIR Host HuggingFace cache dir (same as --hf-cache-dir)\n HF_CACHE_CHOWN=1 chown HF cache mount to caller UID/GID on exit (default: off)\n READY_TIMEOUT Seconds to wait for /v1/models (default: 1800)\n DOCKER_RUN_ARGS Extra arguments appended to docker run\n PI_NPM_PACKAGE npm package for pi (default: @earendil-works/pi-coding-agent)\n SKIP_EVAL=1 Set up server + pi + TraceLens only; skip eval harness\n SKIP_SERVER=1 Skip inference server startup and models.json discovery\n\nExample:\n bash run_pi_analysis_in_docker.sh /data/tracelens_local_testing/tracelens -- \\\\\n \u003cserver-cmd> Qwen/Qwen3.6-35B-A3B --port 30000 --tensor-parallel-size 8\nEOF\n}\n\ndie() {\n echo \"ERROR: $*\" >&2\n exit 1\n}\n\nTRACELENS_ROOT=\"\"\nCOMPARISON_SCOPE=\"${COMPARISON_SCOPE:-both}\"\nSERVER_ARGS=()\n\nwhile [[ $# -gt 0 ]]; do\n case \"$1\" in\n -h|--help)\n usage\n exit 0\n ;;\n --docker-image)\n [[ $# -ge 2 ]] || die \"--docker-image requires a value\"\n DOCKER_IMAGE=\"$2\"\n shift 2\n ;;\n --container-name)\n [[ $# -ge 2 ]] || die \"--container-name requires a value\"\n CONTAINER_NAME=\"$2\"\n shift 2\n ;;\n --work-dir)\n [[ $# -ge 2 ]] || die \"--work-dir requires a value\"\n WORK_DIR=\"$2\"\n shift 2\n ;;\n --hf-cache-dir)\n [[ $# -ge 2 ]] || die \"--hf-cache-dir requires a value\"\n HF_CACHE_DIR=\"$2\"\n shift 2\n ;;\n --hf-cache-chown)\n HF_CACHE_CHOWN=1\n shift\n ;;\n --port)\n [[ $# -ge 2 ]] || die \"--port requires a value\"\n PORT=\"$2\"\n shift 2\n ;;\n --setup-only)\n SKIP_SERVER=1\n SKIP_EVAL=1\n shift\n ;;\n both|standalone|comparative)\n COMPARISON_SCOPE=\"$1\"\n shift\n ;;\n --)\n shift\n SERVER_ARGS=(\"$@\")\n break\n ;;\n -*)\n die \"Unknown option: $1\"\n ;;\n *)\n if [[ -z \"$TRACELENS_ROOT\" ]]; then\n TRACELENS_ROOT=\"$1\"\n shift\n else\n die \"Unexpected argument: $1 (use -- before server command arguments)\"\n fi\n ;;\n esac\ndone\n\nif [[ \"${SKIP_SERVER:-}\" == \"1\" || \"${SKIP_SERVER:-}\" == \"true\" ]]; then\n SKIP_SERVER=1\nelse\n SKIP_SERVER=0\nfi\n\nif [[ \"${HF_CACHE_CHOWN:-}\" == \"1\" || \"${HF_CACHE_CHOWN:-}\" == \"true\" ]]; then\n HF_CACHE_CHOWN=1\nelse\n HF_CACHE_CHOWN=0\nfi\n\nif [[ -z \"$TRACELENS_ROOT\" ]]; then\n usage >&2\n die \"tracelens_root is required\"\nfi\n\nif [[ \"$SKIP_SERVER\" != \"1\" && ${#SERVER_ARGS[@]} -eq 0 ]]; then\n die \"inference server command is required after -- (or use --setup-only)\"\nfi\n\nif [[ ! -d \"$TRACELENS_ROOT\" ]]; then\n die \"TraceLens root not found: $TRACELENS_ROOT\"\nfi\n\nif [[ \"$COMPARISON_SCOPE\" != \"both\" && \"$COMPARISON_SCOPE\" != \"standalone\" && \"$COMPARISON_SCOPE\" != \"comparative\" ]]; then\n die \"Unknown comparison scope '$COMPARISON_SCOPE'. Use 'both', 'standalone', or 'comparative'.\"\nfi\n\nTRACELENS_ROOT=\"$(cd \"$TRACELENS_ROOT\" && pwd)\"\nREPO_BASENAME=\"$(basename \"$TRACELENS_ROOT\")\"\nWORK_DIR=\"${WORK_DIR:-$(dirname \"$TRACELENS_ROOT\")}\"\nWORK_DIR=\"$(cd \"$WORK_DIR\" && pwd)\"\n\nif [[ \"$TRACELENS_ROOT\" != \"$WORK_DIR/\"* && \"$TRACELENS_ROOT\" != \"$WORK_DIR\" ]]; then\n die \"tracelens_root must be inside WORK_DIR ($WORK_DIR)\"\nfi\n\nif [[ -n \"$HF_CACHE_DIR\" ]]; then\n mkdir -p \"$HF_CACHE_DIR\"\n HF_CACHE_DIR=\"$(cd \"$HF_CACHE_DIR\" && pwd)\"\nfi\n\nCONTAINER_REPO=\"/workspace/$REPO_BASENAME\"\nHARNESS=\"$TRACELENS_ROOT/TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_scripts/run_repeatability_parallel.sh\"\nif [[ ! -f \"$HARNESS\" ]]; then\n die \"Eval harness not found: $HARNESS\"\nfi\n\nfor ((i = 0; i \u003c ${#SERVER_ARGS[@]}; i++)); do\n if [[ \"${SERVER_ARGS[i]}\" == --port && $((i + 1)) -lt ${#SERVER_ARGS[@]} ]]; then\n PORT=\"${SERVER_ARGS[i + 1]}\"\n elif [[ \"${SERVER_ARGS[i]}\" == --port=* ]]; then\n PORT=\"${SERVER_ARGS[i]#--port=}\"\n fi\ndone\n\nSERVER_CMD_QUOTED=\"\"\nif [[ ${#SERVER_ARGS[@]} -gt 0 ]]; then\n for arg in \"${SERVER_ARGS[@]}\"; do\n SERVER_CMD_QUOTED+=\"$(printf '%q ' \"$arg\")\"\n done\nfi\n\nmkdir -p \"$WORK_DIR/.pi/agent\" \"$WORK_DIR/venv_tracelens\"\n\nHOST_UID=\"$(id -u)\"\nHOST_GID=\"$(id -g)\"\n\nif [[ \"$SKIP_SERVER\" == \"1\" ]]; then\n DOCKER_ARGS=(--network host)\nelse\n DOCKER_ARGS=(\"${DEFAULT_DOCKER_RUN_ARGS[@]}\")\nfi\nif [[ -n \"${DOCKER_RUN_ARGS:-}\" ]]; then\n # shellcheck disable=SC2206\n EXTRA=($DOCKER_RUN_ARGS)\n DOCKER_ARGS+=(\"${EXTRA[@]}\")\nfi\n\ncleanup() {\n if docker ps --format '{{.Names}}' | grep -qx \"$CONTAINER_NAME\"; then\n echo \"Fixing ownership of mounted directories (${HOST_UID}:${HOST_GID})...\"\n docker exec \"$CONTAINER_NAME\" chown -R \"${HOST_UID}:${HOST_GID}\" /workspace >/dev/null 2>&1 || true\n if [[ -n \"$HF_CACHE_DIR\" && \"$HF_CACHE_CHOWN\" == \"1\" ]]; then\n docker exec \"$CONTAINER_NAME\" chown -R \"${HOST_UID}:${HOST_GID}\" \"$CONTAINER_HF_HOME\" >/dev/null 2>&1 || true\n fi\n fi\n if [[ -n \"${CONTAINER_NAME:-}\" ]]; then\n docker rm -f \"$CONTAINER_NAME\" >/dev/null 2>&1 || true\n fi\n}\ntrap cleanup EXIT\n\nif docker ps -a --format '{{.Names}}' | grep -qx \"$CONTAINER_NAME\"; then\n echo \"Removing existing container: $CONTAINER_NAME\"\n docker rm -f \"$CONTAINER_NAME\" >/dev/null\nfi\n\necho \"=========================================\"\necho \" pi Analysis Eval Launcher\"\necho \" TraceLens root: $TRACELENS_ROOT\"\necho \" Work dir: $WORK_DIR -> /workspace\"\necho \" Container repo: $CONTAINER_REPO\"\necho \" Docker image: $DOCKER_IMAGE\"\necho \" Container: $CONTAINER_NAME\"\necho \" Run as UID:GID: ${HOST_UID}:${HOST_GID}\"\necho \" API port: $PORT\"\nif [[ -n \"$HF_CACHE_DIR\" ]]; then\n echo \" HF cache: $HF_CACHE_DIR -> $CONTAINER_HF_HOME\"\n echo \" HF cache chown: $([[ \"$HF_CACHE_CHOWN\" == \"1\" ]] && echo yes || echo no)\"\nfi\nif [[ \"$SKIP_SERVER\" == \"1\" ]]; then\n echo \" Mode: setup-only (no inference server)\"\nelse\n echo \" Server command: ${SERVER_ARGS[*]}\"\nfi\necho \" Comparison: $COMPARISON_SCOPE\"\necho \"=========================================\"\necho \"\"\n\necho \"Starting Docker container...\"\nDOCKER_VOLUMES=(-v \"$WORK_DIR:/workspace:rw\")\nif [[ -n \"$HF_CACHE_DIR\" ]]; then\n DOCKER_VOLUMES+=(-v \"$HF_CACHE_DIR:$CONTAINER_HF_HOME:rw\")\nfi\n# shellcheck disable=SC2086\ndocker run -d --name \"$CONTAINER_NAME\" \\\n \"${DOCKER_ARGS[@]}\" \\\n \"${DOCKER_VOLUMES[@]}\" \\\n --entrypoint bash \\\n \"$DOCKER_IMAGE\" \\\n -c 'sleep infinity' >/dev/null\n\necho \"Container started. Running setup inside container...\"\n\nEXEC_ENV=(\n -e \"PORT=$PORT\"\n -e \"READY_TIMEOUT=$READY_TIMEOUT\"\n -e \"CONTAINER_REPO=$CONTAINER_REPO\"\n -e \"COMPARISON_SCOPE=$COMPARISON_SCOPE\"\n -e \"SKIP_EVAL=$SKIP_EVAL\"\n -e \"SKIP_SERVER=$SKIP_SERVER\"\n -e \"SERVER_CMD=$SERVER_CMD_QUOTED\"\n -e \"HOST_UID=$HOST_UID\"\n -e \"HOST_GID=$HOST_GID\"\n -e \"HF_CACHE_CHOWN=$HF_CACHE_CHOWN\"\n)\nif [[ -n \"$HF_CACHE_DIR\" ]]; then\n EXEC_ENV+=(\n -e \"HF_HOME=$CONTAINER_HF_HOME\"\n -e \"HUGGINGFACE_HUB_CACHE=$CONTAINER_HF_HOME/hub\"\n -e \"TRANSFORMERS_CACHE=$CONTAINER_HF_HOME/hub\"\n )\nfi\nfor var in TEST_IDS NUM_REPEATS MAX_PARALLEL SLEEP_BETWEEN TEST_TRACES_CSV RESULTS_ROOT REPORT_DIR SUITE_NAME SKIP_POST_PROCESSING PI_NPM_PACKAGE; do\n if [[ -n \"${!var:-}\" ]]; then\n EXEC_ENV+=(-e \"$var=${!var}\")\n fi\ndone\n\ndocker exec \"${EXEC_ENV[@]}\" -i \"$CONTAINER_NAME\" bash -s \u003c\u003c'INNER'\nset -euo pipefail\n\nPORT=\"${PORT:-30000}\"\nREADY_TIMEOUT=\"${READY_TIMEOUT:-1800}\"\nCONTAINER_REPO=\"${CONTAINER_REPO:?}\"\nCOMPARISON_SCOPE=\"${COMPARISON_SCOPE:-both}\"\nSKIP_EVAL=\"${SKIP_EVAL:-0}\"\nSKIP_SERVER=\"${SKIP_SERVER:-0}\"\nHF_CACHE_CHOWN=\"${HF_CACHE_CHOWN:-0}\"\nPI_AGENT_DIR=\"/workspace/.pi/agent\"\nVENV_DIR=\"/workspace/venv_tracelens\"\nSERVER_LOG=\"/workspace/inference_server.log\"\n\nif [[ -n \"${HF_HOME:-}\" ]]; then\n export HF_HOME\n export HUGGINGFACE_HUB_CACHE=\"${HUGGINGFACE_HUB_CACHE:-$HF_HOME/hub}\"\n export TRANSFORMERS_CACHE=\"${TRANSFORMERS_CACHE:-$HUGGINGFACE_HUB_CACHE}\"\n echo \"HF_HOME=$HF_HOME\"\nfi\n\nfix_mount_ownership() {\n if [[ -z \"${HOST_UID:-}\" || -z \"${HOST_GID:-}\" ]]; then\n return 0\n fi\n echo \"==> Fixing ownership of mounted directories (${HOST_UID}:${HOST_GID})...\"\n chown -R \"${HOST_UID}:${HOST_GID}\" /workspace || true\n if [[ \"$HF_CACHE_CHOWN\" == \"1\" && -n \"${HF_HOME:-}\" && -d \"$HF_HOME\" ]]; then\n chown -R \"${HOST_UID}:${HOST_GID}\" \"$HF_HOME\" || true\n fi\n}\n\non_inner_exit() {\n if [[ -n \"${SERVER_PID:-}\" ]] && kill -0 \"$SERVER_PID\" 2>/dev/null; then\n kill \"$SERVER_PID\" 2>/dev/null || true\n fi\n fix_mount_ownership\n}\ntrap on_inner_exit EXIT\n\ndie() {\n echo \"ERROR: $*\" >&2\n exit 1\n}\n\nif [[ \"$SKIP_SERVER\" == \"1\" ]]; then\n echo \"==> SKIP_SERVER=1 — skipping inference server startup\"\nelse\n echo \"==> Starting inference server: ${SERVER_CMD}\"\n echo \"==> Server output logged to $SERVER_LOG\"\n # shellcheck disable=SC2086\n eval \"${SERVER_CMD}\" >\"$SERVER_LOG\" 2>&1 &\n SERVER_PID=$!\n\n echo \"==> Waiting for inference server at http://localhost:${PORT}/v1/models (timeout ${READY_TIMEOUT}s)...\"\n models_json=\"\"\n deadline=$((SECONDS + READY_TIMEOUT))\n wait_started=$SECONDS\n last_progress=$SECONDS\n while (( SECONDS \u003c deadline )); do\n if models_json=\"$(python3 - \"$PORT\" \u003c\u003c'PY' 2>/dev/null\nimport sys\nimport urllib.request\n\nport = sys.argv[1]\nurl = f\"http://localhost:{port}/v1/models\"\nwith urllib.request.urlopen(url, timeout=5) as resp:\n if resp.status != 200:\n raise SystemExit(1)\n body = resp.read().decode()\n if not body.strip():\n raise SystemExit(1)\n print(body, end=\"\")\nPY\n)\"; then\n echo \"==> Inference server ready (after $((SECONDS - wait_started))s)\"\n break\n fi\n if ! kill -0 \"$SERVER_PID\" 2>/dev/null; then\n echo \"Inference server exited early. Last log lines:\" >&2\n tail -n 40 \"$SERVER_LOG\" >&2 || true\n die \"Inference server died before /v1/models became available\"\n fi\n if (( SECONDS - last_progress >= 60 )); then\n echo \"==> Still waiting for /v1/models ($((SECONDS - wait_started))s elapsed)...\"\n last_progress=$SECONDS\n fi\n sleep 5\n done\n\n if [[ -z \"$models_json\" ]]; then\n echo \"Inference server log tail:\" >&2\n tail -n 40 \"$SERVER_LOG\" >&2 || true\n die \"Timed out waiting for http://localhost:${PORT}/v1/models\"\n fi\n\n echo \"==> Discovering model id from /v1/models...\"\n mkdir -p \"$PI_AGENT_DIR\"\n python3 - \"$PI_AGENT_DIR\" \"$PORT\" \u003c\u003c'PY'\nimport json\nimport pathlib\nimport sys\nimport urllib.request\n\nagent_dir = pathlib.Path(sys.argv[1])\nport = sys.argv[2]\nurl = f\"http://localhost:{port}/v1/models\"\n\nwith urllib.request.urlopen(url, timeout=30) as resp:\n payload = json.load(resp)\n\nmodels = payload.get(\"data\") or []\nif not models:\n raise SystemExit(\"No models returned from /v1/models\")\n\nentries = []\nfor item in models:\n model_id = item.get(\"id\")\n if not model_id:\n continue\n context = (\n item.get(\"max_model_len\")\n or item.get(\"context_length\")\n or item.get(\"contextWindow\")\n or 1_000_000\n )\n entries.append({\"id\": model_id, \"contextWindow\": int(context)})\n\nif not entries:\n raise SystemExit(\"Could not parse model id from /v1/models response\")\n\nconfig = {\n \"providers\": {\n \"local\": {\n \"baseUrl\": f\"http://localhost:{port}/v1\",\n \"api\": \"openai-completions\",\n \"apiKey\": \"none\",\n \"models\": entries,\n }\n }\n}\n\nout = agent_dir / \"models.json\"\nout.write_text(json.dumps(config, indent=2) + \"\\n\", encoding=\"utf-8\")\nprint(f\"Wrote {out} with model id(s): {', '.join(m['id'] for m in entries)}\")\nPY\n\n export PI_CODING_AGENT_DIR=\"$PI_AGENT_DIR\"\n echo \"PI_CODING_AGENT_DIR=$PI_CODING_AGENT_DIR\"\nfi\n\nPI_NPM_PACKAGE=\"${PI_NPM_PACKAGE:-@earendil-works/pi-coding-agent}\"\nNODE_MIN_MAJOR=22\n\nnode_major_version() {\n if ! command -v node >/dev/null 2>&1; then\n echo 0\n return\n fi\n node -v | sed 's/^v//' | cut -d. -f1\n}\n\ninstall_nodejs_modern() {\n echo \"==> Installing Node.js ${NODE_MIN_MAJOR}+ and npm...\"\n if command -v apt-get >/dev/null 2>&1; then\n apt-get update -qq\n DEBIAN_FRONTEND=noninteractive apt-get install -y -qq ca-certificates curl gnupg\n curl -fsSL \"https://deb.nodesource.com/setup_${NODE_MIN_MAJOR}.x\" | bash -\n DEBIAN_FRONTEND=noninteractive apt-get install -y -qq nodejs\n elif command -v dnf >/dev/null 2>&1; then\n curl -fsSL \"https://rpm.nodesource.com/setup_${NODE_MIN_MAJOR}.x\" | bash -\n dnf install -y -q nodejs\n elif command -v yum >/dev/null 2>&1; then\n curl -fsSL \"https://rpm.nodesource.com/setup_${NODE_MIN_MAJOR}.x\" | bash -\n yum install -y -q nodejs\n elif command -v apk >/dev/null 2>&1; then\n apk add --no-cache nodejs npm\n else\n die \"npm not found and no supported package manager (apt, dnf, yum, apk)\"\n fi\n command -v npm >/dev/null 2>&1 || die \"npm installation failed\"\n major=\"$(node_major_version)\"\n if [[ \"$major\" -lt \"$NODE_MIN_MAJOR\" ]]; then\n die \"Node.js $(node -v) is too old; pi requires >= ${NODE_MIN_MAJOR}.x\"\n fi\n echo \"==> Node $(node -v), npm $(npm -v)\"\n}\n\ninstall_npm_if_needed() {\n major=\"$(node_major_version)\"\n if command -v npm >/dev/null 2>&1 && [[ \"$major\" -ge \"$NODE_MIN_MAJOR\" ]]; then\n echo \"==> Node $(node -v), npm $(npm -v)\"\n return 0\n fi\n install_nodejs_modern\n}\n\ninstall_pi_if_needed() {\n if command -v pi >/dev/null 2>&1; then\n return 0\n fi\n install_npm_if_needed\n echo \"==> Installing pi via npm ($PI_NPM_PACKAGE)...\"\n npm install -g --ignore-scripts \"$PI_NPM_PACKAGE\"\n NPM_PREFIX=\"$(npm prefix -g 2>/dev/null || npm config get prefix 2>/dev/null || echo /usr/local)\"\n export PATH=\"${NPM_PREFIX}/bin:${HOME}/.local/bin:${PATH}\"\n}\n\ninstall_pi_if_needed\ncommand -v pi >/dev/null 2>&1 || die \"pi not found on PATH after npm install\"\n\necho \"==> Verifying pi installation...\"\npi --version\n\nif [[ ! -f \"$VENV_DIR/bin/activate\" ]]; then\n echo \"==> Creating Python venv at $VENV_DIR...\"\n python3 -m venv \"$VENV_DIR\"\nfi\n\n# shellcheck source=/dev/null\nsource \"$VENV_DIR/bin/activate\"\n\necho \"==> Installing TraceLens editable from $CONTAINER_REPO...\"\npip install -q --upgrade pip\npip install -q -e \"$CONTAINER_REPO\"\n\nif [[ \"$SKIP_EVAL\" == \"1\" ]]; then\n echo \"SKIP_EVAL=1 — setup complete, skipping eval harness.\"\n exit 0\nfi\n\nHARNESS=\"$CONTAINER_REPO/TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_scripts/run_repeatability_parallel.sh\"\n[[ -f \"$HARNESS\" ]] || die \"Harness not found in container: $HARNESS\"\n\nexport REPO_ROOT=\"$CONTAINER_REPO\"\n\necho \"==> Running eval harness: $HARNESS --pi $COMPARISON_SCOPE\"\nbash \"$HARNESS\" --pi \"$COMPARISON_SCOPE\"\nINNER\n\necho \"\"\necho \"Eval run finished. Results are under:\"\necho \" $WORK_DIR/$REPO_BASENAME/TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/\"\necho \"\"\necho \"Inference server log inside container: /workspace/inference_server.log\"\necho \"pi config: $WORK_DIR/.pi/agent/models.json\"\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":20031,"content_sha256":"58b8b6c37c6b4c164930babb739e4e41ec0d4f7cf0d220253f2b089703a2ba90"},{"filename":"evals/eval_scripts/run_repeatability_parallel.sh","content":"#!/usr/bin/env bash\n###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\nset -uo pipefail\n\n# Tags this process and all descendants with TRACELENS_EVAL_JOB=1.\nif [[ \"${TRACELENS_EVAL_JOB:-}\" != 1 ]]; then\n export TRACELENS_EVAL_JOB=1\n exec bash \"$0\" \"$@\"\nfi\n\n# ---------------------------------------------------------------------------\n# Usage: bash run_repeatability_parallel.sh [both|standalone|comparative] [--pi]\n# or: bash run_repeatability_parallel.sh --pi [both|standalone|comparative]\n#\n# Default scope is 'both'. With 'both' (or no scope argument) the script runs\n# the full pipeline sequentially:\n# 1. Generate golden references for standalone (via generate_ref.sh)\n# 2. Generate golden references for comparative (via generate_ref.sh)\n# 3. Repeatability eval for standalone\n# 4. Repeatability eval for comparative\n# 5. A single combined pr_report.md + fix_ticket_report.md\n#\n# Passing 'standalone' or 'comparative' (or COMPARISON_SCOPE=standalone|comparative)\n# restricts the run to that one scope only (gen-ref + repeatability + report\n# over just that scope). Golden references are always regenerated from scratch\n# as local directories under TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/ before the\n# repeatability stage, so the repeatability evals compare against the freshly\n# generated references.\n#\n# --pi Use `pi` instead of the Cursor `agent` CLI. Also settable via USE_PI=1.\n# Applies to golden-reference generation (via generate_ref.sh), repeatability\n# evals, and post-processing.\n#\n# CONTAINER is optional. If set, python/setup commands run via\n# docker exec -w $REPO_ROOT $CONTAINER ... ; if unset, they run on the host.\n# ---------------------------------------------------------------------------\n\nusage() {\n cat \u003c\u003c'EOF'\nUsage: bash run_repeatability_parallel.sh [both|standalone|comparative] [--pi]\n\n both|standalone|comparative Comparison scope (default: both)\n --pi Use pi instead of the Cursor agent CLI (or USE_PI=1)\n\nWith 'both' (or no scope argument) the script runs, sequentially:\n 1. generate golden references (standalone) via generate_ref.sh\n 2. generate golden references (comparative) via generate_ref.sh\n 3. repeatability eval (standalone)\n 4. repeatability eval (comparative)\n 5. a single combined pr_report.md + fix_ticket_report.md\n\nTo kill a running pipeline (and any orphaned jobs it spawned), use:\n bash eval_scripts/kill_eval_jobs.sh\nEOF\n}\n\nUSE_PI=\"${USE_PI:-false}\"\n# Comparison scope: both (default), standalone, or comparative.\nSCOPE_FILTER=\"${COMPARISON_SCOPE:-both}\"\n\nwhile [[ $# -gt 0 ]]; do\n case \"$1\" in\n --pi)\n USE_PI=true\n shift\n ;;\n both|standalone|comparative)\n SCOPE_FILTER=\"$1\"\n shift\n ;;\n -h|--help)\n usage\n exit 0\n ;;\n *)\n echo \"ERROR: Unknown argument '$1'.\" >&2\n usage >&2\n exit 1\n ;;\n esac\ndone\n\nif [[ \"$USE_PI\" == true || \"$USE_PI\" == 1 || \"$USE_PI\" == \"1\" ]]; then\n USE_PI=true\nelse\n USE_PI=false\nfi\n\nif [[ \"$SCOPE_FILTER\" != \"both\" && \"$SCOPE_FILTER\" != \"standalone\" && \"$SCOPE_FILTER\" != \"comparative\" ]]; then\n echo \"ERROR: Unknown comparison scope '$SCOPE_FILTER'. Use 'both', 'standalone', or 'comparative'.\" >&2\n exit 1\nfi\n\nif [[ \"$SCOPE_FILTER\" == \"both\" ]]; then\n SCOPES=(standalone comparative)\nelse\n SCOPES=(\"$SCOPE_FILTER\")\nfi\n\n# ---------------------------------------------------------------------------\n# Configuration (scope-independent)\n# ---------------------------------------------------------------------------\nMAX_PARALLEL=\"${MAX_PARALLEL:-5}\"\nNUM_REPEATS=\"${NUM_REPEATS:-3}\"\nSLEEP_BETWEEN=\"${SLEEP_BETWEEN:-30}\"\nCONTAINER=\"${CONTAINER:-}\"\nTEST_IDS=\"${TEST_IDS:-}\"\nSUITE_NAME=\"${SUITE_NAME:-eval}\"\nSKIP_GENERATE_REF=\"${SKIP_GENERATE_REF:-}\"\nSKIP_POST_PROCESSING=\"${SKIP_POST_PROCESSING:-}\"\n\nAGENT_MODEL=\"${AGENT_MODEL:-claude-opus-4-8-thinking-medium}\"\nPI_VENV_PREFIX=\"use venv_tracelens for all commands and tool calls. \"\n\n# Paths (REPO_ROOT may differ from the shell cwd)\nREPO_ROOT=\"${REPO_ROOT:-$(pwd)}\"\nANALYSIS_DIR=\"$REPO_ROOT/TraceLens/Agent/Analysis\"\nEVALS_DIR=\"$REPO_ROOT/TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals\"\n\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nGENERATE_REF_SCRIPT=\"$SCRIPT_DIR/generate_ref.sh\"\n\n# Single combined report + intermediate combined results tree for post-processing.\nREPORT_DIR=\"${REPORT_DIR:-$EVALS_DIR/reports}\"\nCOMBINED_RESULTS_ROOT=\"${COMBINED_RESULTS_ROOT:-$EVALS_DIR/repeatability_results_combined}\"\n\n# Scope-specific globals, (re)set per scope by run_repeatability_for_scope().\nCOMPARISON_SCOPE=\"\"\nRESULTS_ROOT=\"\"\nTEST_TRACES_CSV=\"\"\n\nif [[ -n \"$CONTAINER\" ]]; then\n DEXEC=(docker exec -w \"$REPO_ROOT\" \"$CONTAINER\")\n RUNTIME_LABEL=\"container $CONTAINER\"\n NODE_LABEL=\"node $(hostname)\"\nelse\n DEXEC=()\n RUNTIME_LABEL=\"host (no container)\"\n NODE_LABEL=\"local\"\nfi\n\nif [[ \"$USE_PI\" == true ]]; then\n AGENT_BACKEND=\"pi\"\nelse\n AGENT_BACKEND=\"cursor agent\"\nfi\n\n# ---------------------------------------------------------------------------\n# Helpers\n# ---------------------------------------------------------------------------\n\nts() { date \"+%H:%M:%S\"; }\n\nlog_status() {\n flock 1 echo \"$@\"\n}\n\nrepo_abs_path() {\n local p=\"$1\"\n if [[ \"$p\" = /* ]]; then\n echo \"$p\"\n else\n echo \"$REPO_ROOT/$p\"\n fi\n}\n\n# Run an LLM agent step. Optional second arg: non-empty enables a 1800s timeout (cursor only).\nrun_llm_agent() {\n local prompt=\"$1\"\n local with_timeout=\"${2:-}\"\n\n if [[ \"$USE_PI\" == true ]]; then\n pi --mode json \"${PI_VENV_PREFIX}${prompt}\"\n elif [[ -n \"$with_timeout\" ]]; then\n timeout 1800 agent --model \"$AGENT_MODEL\" --print --force --trust --output-format stream-json \\\n \"$prompt\"\n else\n agent --model \"$AGENT_MODEL\" --print --force --trust --output-format stream-json \\\n \"$prompt\"\n fi\n}\n\nexpand_archive() {\n local name=\"$1\"\n local archive=\"$EVALS_DIR/analysis_tests/${name}.tar.gz\"\n local target=\"$EVALS_DIR/analysis_tests/$name\"\n if [[ -f \"$archive\" ]] && [[ ! -d \"$target\" ]]; then\n echo \"Expanding ${name}.tar.gz...\"\n # Archives store paths as agent_evals/Analysis/analysis_tests/...\n tar xzf \"$archive\" -C \"$EVALS_DIR\" --strip-components=2\n echo \"Done.\"\n fi\n}\n\n# Return 0 if $id should run given TEST_IDS (empty = all).\n# Supports exact match and underscore-delimited prefix (e.g. gemm_01 -> gemm_01_compute_few_tiles).\nshould_run_id() {\n local id=\"$1\"\n [[ -z \"$TEST_IDS\" ]] && return 0\n local token\n for token in $TEST_IDS; do\n if [[ \"$id\" == \"$token\" || \"$id\" == \"${token}_\"* ]]; then\n return 0\n fi\n done\n return 1\n}\n\nprint_scheduled_tests() {\n local -a scheduled_ids=()\n local id sub_category trace1_path trace2_path trace_path reference_dir platform platform2 capture_folder1 capture_folder2\n\n if [[ \"$COMPARISON_SCOPE\" == \"comparative\" ]]; then\n while IFS=, read -r id sub_category trace1_path trace2_path reference_dir platform platform2 capture_folder1 capture_folder2; do\n [[ -z \"$id\" ]] && continue\n should_run_id \"$id\" && scheduled_ids+=(\"$id\")\n done \u003c \u003c(tail -n +2 \"$TEST_TRACES_CSV\"; echo)\n else\n while IFS=, read -r id sub_category trace_path reference_dir platform; do\n [[ -z \"$id\" ]] && continue\n should_run_id \"$id\" && scheduled_ids+=(\"$id\")\n done \u003c \u003c(tail -n +2 \"$TEST_TRACES_CSV\"; echo)\n fi\n\n if [[ ${#scheduled_ids[@]} -eq 0 ]]; then\n return\n fi\n\n echo \"Tests to run (${#scheduled_ids[@]}):\"\n local scheduled_id\n for scheduled_id in \"${scheduled_ids[@]}\"; do\n echo \" - $scheduled_id ($NUM_REPEATS repeat(s))\"\n done\n echo \"\"\n}\n\n# ---------------------------------------------------------------------------\n# Single job: one (test_case, repeat) iteration\n#\n# Args: id repeat trace1_path trace2_path reference_dir platform platform2\n# capture_folder1 capture_folder2 (comparative mode; empty string if absent)\n# ---------------------------------------------------------------------------\n\nrun_single_job() {\n local id=\"$1\" repeat=\"$2\" trace1_path=\"$3\" trace2_path=\"$4\" reference_dir=\"$5\" platform=\"$6\" platform2=\"$7\"\n local capture_folder1=\"${8:-}\" capture_folder2=\"${9:-}\"\n local tag=\"[$id|run_$repeat]\"\n\n log_status \" $tag [$(ts)] Running\"\n\n trace1_path=\"$(repo_abs_path \"$trace1_path\")\"\n if [[ -n \"$trace2_path\" ]]; then\n trace2_path=\"$(repo_abs_path \"$trace2_path\")\"\n fi\n reference_dir=\"$(repo_abs_path \"$reference_dir\")\"\n if [[ -n \"$capture_folder1\" ]]; then\n capture_folder1=\"$(repo_abs_path \"$capture_folder1\")\"\n fi\n if [[ -n \"$capture_folder2\" ]]; then\n capture_folder2=\"$(repo_abs_path \"$capture_folder2\")\"\n fi\n\n local capture_suffix=\"\"\n local analysis_mode=\"default\"\n if [[ -n \"$capture_folder1\" ]]; then\n capture_suffix+=\" capture folder for trace1 $capture_folder1\"\n fi\n if [[ -n \"$capture_folder2\" ]]; then\n capture_suffix+=\" capture folder for trace2 $capture_folder2\"\n fi\n if [[ -n \"$capture_folder1\" || -n \"$capture_folder2\" ]]; then\n analysis_mode=\"inference\"\n fi\n\n local CASE_RESULTS=\"$RESULTS_ROOT/$id/run_${repeat}\"\n local OUTPUT_DIR=\"$CASE_RESULTS/analysis_output\"\n \"${DEXEC[@]}\" rm -rf \"$CASE_RESULTS\" 2>/dev/null || true\n rm -rf \"$CASE_RESULTS\" 2>/dev/null || true\n mkdir -p \"$OUTPUT_DIR\"\n \"${DEXEC[@]}\" bash -c \"mkdir -p $OUTPUT_DIR && chmod -R 777 $OUTPUT_DIR\"\n \"${DEXEC[@]}\" bash -c \"mkdir -p $CASE_RESULTS && chmod -R 777 $CASE_RESULTS\"\n\n # -- Phase 1: Agent analysis with retry + backoff -----------------------\n log_status \" $tag [$(ts)] Phase 1: analysis starting\"\n\n local agent_success=false\n local agent_attempts=0\n while [ \"$agent_success\" = false ] && [ \"$agent_attempts\" -lt 3 ]; do\n agent_attempts=$((agent_attempts + 1))\n (\n cd \"$ANALYSIS_DIR\" || exit\n if [[ \"$COMPARISON_SCOPE\" == \"comparative\" ]]; then\n run_llm_agent \\\n \"Follow the analysis orchestrator installed with the TraceLens pip package (look under TraceLens/Agent/Analysis/skills/analysis-orchestrator/ in the package installation directory) and run the full agentic analysis workflow on $trace1_path and $trace2_path${capture_suffix} with platform $platform (trace1) and $platform2 (trace2), analysis mode $analysis_mode, $NODE_LABEL, $RUNTIME_LABEL, output to $OUTPUT_DIR\" \\\n 1\n else\n run_llm_agent \\\n \"Follow the analysis orchestrator installed with the TraceLens pip package (look under TraceLens/Agent/Analysis/skills/analysis-orchestrator/ in the package installation directory) and run the full agentic analysis workflow on $trace1_path with platform $platform, $NODE_LABEL, $RUNTIME_LABEL, output to $OUTPUT_DIR\" \\\n 1\n fi\n ) \u003c /dev/null > \"$CASE_RESULTS/analysis_stream.ndjson\" 2>&1\n\n if head -c 2048 \"$CASE_RESULTS/analysis_stream.ndjson\" | grep -qiE 'Error:.*unavailable|Service Unavailable'; then\n log_status \" $tag Attempt $agent_attempts/3 failed (agent unavailable). Backing off 30s...\"\n sleep 30\n else\n agent_success=true\n fi\n done\n\n if [ \"$agent_success\" = false ]; then\n log_status \" $tag FAILED after 3 attempts (agent unavailable). Skipping evals.\"\n return 1\n fi\n\n log_status \" $tag [$(ts)] Phase 1 complete.\"\n sleep \"$SLEEP_BETWEEN\"\n\n # -- Phase 2: 4 parallel evals ------------------------------------------\n log_status \" $tag [$(ts)] Phase 2: evals starting\"\n local eval_pids=()\n\n \"${DEXEC[@]}\" python3 \"$EVALS_DIR/eval_utils/workflow_scripted_evals.py\" \\\n --output-dir \"$OUTPUT_DIR\" \\\n --results \"$CASE_RESULTS/workflow_scripted_results.csv\" \\\n --comparison-scope \"$COMPARISON_SCOPE\" \\\n > \"$CASE_RESULTS/workflow_scripted_eval.log\" 2>&1 &\n eval_pids+=($!)\n\n (\n cd \"$EVALS_DIR\" || exit\n run_llm_agent \\\n \"Run workflow LLM eval skill on $OUTPUT_DIR for test case $id mode=$COMPARISON_SCOPE. Write results to $CASE_RESULTS/workflow_llm_results.csv\"\n ) \u003c /dev/null > \"$CASE_RESULTS/workflow_llm_eval.ndjson\" 2>&1 &\n eval_pids+=($!)\n\n \"${DEXEC[@]}\" python3 \"$EVALS_DIR/eval_utils/quality_scripted_evals.py\" \\\n --output-dir \"$OUTPUT_DIR\" --reference-dir \"$reference_dir\" \\\n --results \"$CASE_RESULTS/quality_scripted_results.csv\" \\\n --comparison-scope \"$COMPARISON_SCOPE\" \\\n > \"$CASE_RESULTS/quality_scripted_eval.log\" 2>&1 &\n eval_pids+=($!)\n\n (\n cd \"$EVALS_DIR\" || exit\n run_llm_agent \\\n \"Run quality LLM eval skill on $OUTPUT_DIR with reference $reference_dir for test case $id mode=$COMPARISON_SCOPE. Write results to $CASE_RESULTS/quality_llm_results.csv\"\n ) \u003c /dev/null > \"$CASE_RESULTS/quality_llm_eval.ndjson\" 2>&1 &\n eval_pids+=($!)\n\n for pid in \"${eval_pids[@]}\"; do\n wait \"$pid\" 2>/dev/null || true\n done\n\n log_status \" $tag [$(ts)] Phase 2 complete.\"\n\n # -- Merge results -------------------------------------------------------\n \"${DEXEC[@]}\" python3 \"$EVALS_DIR/eval_utils/merge_results.py\" \\\n --results-dir \"$CASE_RESULTS\" \\\n --output \"$CASE_RESULTS/eval_summary.csv\" || true\n log_status \" $tag Summary -> $CASE_RESULTS/eval_summary.csv\"\n log_status \" $tag [$(ts)] Finished\"\n}\n\n# ---------------------------------------------------------------------------\n# FIFO semaphore for concurrency control\n# ---------------------------------------------------------------------------\n\nFIFO=\"\"\ncleanup() {\n [[ -n \"$FIFO\" ]] && rm -f \"$FIFO\"\n}\n\nsetup_semaphore() {\n FIFO=\"$RESULTS_ROOT/.job_fifo\"\n rm -f \"$FIFO\"\n mkfifo \"$FIFO\"\n exec 4\u003c>\"$FIFO\"\n for ((t = 0; t \u003c MAX_PARALLEL; t++)); do echo >&4; done\n trap cleanup EXIT\n}\n\n_spawn_jobs() {\n local id=\"$1\" trace1_path=\"$2\" trace2_path=\"$3\" reference_dir=\"$4\" platform=\"$5\" platform2=\"$6\"\n local capture_folder1=\"${7:-}\" capture_folder2=\"${8:-}\"\n\n should_run_id \"$id\" || return\n JOBS_SPAWNED=$((JOBS_SPAWNED + 1))\n\n for ((i = 0; i \u003c NUM_REPEATS; i++)); do\n read -r -u4 # acquire semaphore slot\n (\n run_single_job \"$id\" \"$i\" \"$trace1_path\" \"$trace2_path\" \"$reference_dir\" \"$platform\" \"$platform2\" \"$capture_folder1\" \"$capture_folder2\" || true\n echo >&4 # release semaphore slot\n sleep 2 # stagger agent startup to avoid ~/.cursor/cli-config.json rename race\n ) &\n sleep 2 # stagger agent startup to avoid ~/.cursor/cli-config.json rename race\n done\n}\n\n# ---------------------------------------------------------------------------\n# Stage 1: golden reference generation for one scope (from scratch)\n#\n# Delegates to generate_ref.sh so the golden references are produced with the\n# exact same flow, written as local analysis_output_ref/ directories under\n# TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/. generate_ref.sh already removes each\n# case's existing reference before regenerating it.\n# ---------------------------------------------------------------------------\n\ngenerate_refs_for_scope() {\n local scope=\"$1\"\n\n echo \"=========================================\"\n echo \" Stage 1: Golden Reference Generation\"\n echo \" Scope: $scope\"\n echo \" Agent: $AGENT_BACKEND\"\n echo \" Node: $NODE_LABEL\"\n echo \" Runtime: $RUNTIME_LABEL\"\n echo \"=========================================\"\n echo \"\"\n\n if [[ ! -f \"$GENERATE_REF_SCRIPT\" ]]; then\n echo \"ERROR: generate_ref.sh not found at $GENERATE_REF_SCRIPT\" >&2\n return 1\n fi\n\n REPO_ROOT=\"$REPO_ROOT\" \\\n CONTAINER=\"$CONTAINER\" \\\n MAX_PARALLEL=\"$MAX_PARALLEL\" \\\n SLEEP_BETWEEN=\"$SLEEP_BETWEEN\" \\\n TEST_IDS=\"$TEST_IDS\" \\\n NUM_REPEATS=\"$NUM_REPEATS\" \\\n USE_PI=\"$USE_PI\" \\\n AGENT_MODEL=\"$AGENT_MODEL\" \\\n bash \"$GENERATE_REF_SCRIPT\" \"$scope\"\n}\n\n# ---------------------------------------------------------------------------\n# Stage 2: repeatability evals for one scope\n# ---------------------------------------------------------------------------\n\nrun_repeatability_for_scope() {\n COMPARISON_SCOPE=\"$1\"\n RESULTS_ROOT=\"$EVALS_DIR/repeatability_results_${COMPARISON_SCOPE}\"\n TEST_TRACES_CSV=\"$EVALS_DIR/analysis_tests/combined_traces_${COMPARISON_SCOPE}.csv\"\n\n if [[ ! -f \"$TEST_TRACES_CSV\" ]]; then\n echo \"ERROR: trace CSV not found: $TEST_TRACES_CSV\" >&2\n return 1\n fi\n\n mkdir -p \"$RESULTS_ROOT\"\n \"${DEXEC[@]}\" bash -c \"mkdir -p $RESULTS_ROOT && chmod -R 777 $RESULTS_ROOT\"\n\n if [[ \"$COMPARISON_SCOPE\" == \"comparative\" ]]; then\n expand_archive unit_tests_comparative\n expand_archive e2e_tests_comparative\n else\n expand_archive unit_tests_standalone\n expand_archive e2e_tests_standalone\n fi\n\n echo \"=========================================\"\n echo \" Stage 2: Analysis Repeatability Test\"\n echo \" Mode: $COMPARISON_SCOPE\"\n echo \" Agent: $AGENT_BACKEND\"\n echo \" Node: $NODE_LABEL\"\n echo \" Runtime: $RUNTIME_LABEL\"\n echo \" Repeats: $NUM_REPEATS\"\n echo \" Max parallel: $MAX_PARALLEL\"\n echo \" CSV: $TEST_TRACES_CSV\"\n if [[ -n \"$TEST_IDS\" ]]; then\n echo \" Test filter: $TEST_IDS\"\n fi\n echo \"=========================================\"\n echo \"\"\n\n print_scheduled_tests\n\n setup_semaphore\n\n JOBS_SPAWNED=0\n\n if [[ \"$COMPARISON_SCOPE\" == \"comparative\" ]]; then\n # comparative CSV: id,sub_category,trace1_path,trace2_path,reference_dir,platform,platform2,capture_folder1,capture_folder2\n while IFS=, read -r id sub_category trace1_path trace2_path reference_dir platform platform2 capture_folder1 capture_folder2 \u003c&3; do\n [[ -z \"$id\" ]] && continue\n _spawn_jobs \"$id\" \"$trace1_path\" \"$trace2_path\" \"$reference_dir\" \"$platform\" \"$platform2\" \"${capture_folder1:-}\" \"${capture_folder2:-}\"\n done 3\u003c \u003c(tail -n +2 \"$TEST_TRACES_CSV\"; echo)\n else\n # standalone CSV: id,sub_category,trace_path,reference_dir,platform\n while IFS=, read -r id sub_category trace_path reference_dir platform \u003c&3; do\n [[ -z \"$id\" ]] && continue\n _spawn_jobs \"$id\" \"$trace_path\" \"\" \"$reference_dir\" \"$platform\" \"\"\n done 3\u003c \u003c(tail -n +2 \"$TEST_TRACES_CSV\"; echo)\n fi\n\n wait\n rm -f \"$FIFO\"\n\n if [[ -n \"$TEST_IDS\" && \"$JOBS_SPAWNED\" -eq 0 ]]; then\n echo \"\"\n echo \"WARNING: TEST_IDS='$TEST_IDS' matched no trace ids in $TEST_TRACES_CSV.\" >&2\n echo \" Use exact ids or underscore-delimited prefixes (e.g. gemm_01 -> gemm_01_compute_few_tiles).\" >&2\n fi\n\n echo \"\"\n echo \" Repeatability ($COMPARISON_SCOPE) finished. Results in: $RESULTS_ROOT\"\n echo \"\"\n}\n\n# ---------------------------------------------------------------------------\n# Build a single unified trace CSV covering every scope that ran, using the\n# standalone 5-column schema (id,sub_category,trace_path,reference_dir,platform)\n# so the post-processing skill can consume both scopes in one pass. Comparative\n# rows are down-projected: trace1_path -> trace_path, platform (trace1) kept.\n# ---------------------------------------------------------------------------\n\nbuild_combined_csv() {\n local out=\"$1\"\n local scope csv id sub t1 t2 ref plat plat2 cf1 cf2\n\n echo \"id,sub_category,trace_path,reference_dir,platform\" > \"$out\"\n\n for scope in \"${SCOPES[@]}\"; do\n csv=\"$EVALS_DIR/analysis_tests/combined_traces_${scope}.csv\"\n [[ -f \"$csv\" ]] || continue\n if [[ \"$scope\" == \"comparative\" ]]; then\n while IFS=, read -r id sub t1 t2 ref plat plat2 cf1 cf2; do\n [[ -z \"$id\" ]] && continue\n echo \"$id,$sub,$t1,$ref,$plat\" >> \"$out\"\n done \u003c \u003c(tail -n +2 \"$csv\"; echo)\n else\n while IFS= read -r line; do\n [[ -z \"$line\" ]] && continue\n echo \"$line\" >> \"$out\"\n done \u003c \u003c(tail -n +2 \"$csv\"; echo)\n fi\n done\n}\n\n# ---------------------------------------------------------------------------\n# Stage 3: single combined post-processing across all scopes that ran.\n# Merges the per-scope results trees (via symlinks) and CSVs into one, then\n# invokes the post-processing skill once to emit a single pr_report.md and a\n# single fix_ticket_report.md in $REPORT_DIR.\n# ---------------------------------------------------------------------------\n\nrun_post_processing() {\n if [[ \"$SKIP_POST_PROCESSING\" == \"1\" ]]; then\n echo \"\"\n echo \" Post-processing skipped -- SKIP_POST_PROCESSING=1.\"\n echo \" To run later, first rebuild the combined results tree + CSV, then:\"\n if [[ \"$USE_PI\" == true ]]; then\n echo \" pi --mode json '${PI_VENV_PREFIX}Run eval post processing on results_root=$COMBINED_RESULTS_ROOT suite=$SUITE_NAME test_traces_csv=$REPORT_DIR/combined_traces.csv report_dir=$REPORT_DIR container=${CONTAINER:-} $NODE_LABEL $RUNTIME_LABEL'\"\n else\n echo \" agent 'Run eval post processing on results_root=$COMBINED_RESULTS_ROOT suite=$SUITE_NAME test_traces_csv=$REPORT_DIR/combined_traces.csv report_dir=$REPORT_DIR container=${CONTAINER:-} $NODE_LABEL $RUNTIME_LABEL'\"\n fi\n return 0\n fi\n\n mkdir -p \"$REPORT_DIR\"\n\n # Merge per-scope results trees into one combined tree via symlinks so a\n # single aggregate/report pass sees every trace. Trace ids are unique\n # across scopes, so there are no name collisions.\n rm -rf \"$COMBINED_RESULTS_ROOT\"\n mkdir -p \"$COMBINED_RESULTS_ROOT\"\n\n local scope rr child name src\n for scope in \"${SCOPES[@]}\"; do\n rr=\"$EVALS_DIR/repeatability_results_${scope}\"\n [[ -d \"$rr\" ]] || continue\n for child in \"$rr\"/*/; do\n [[ -d \"$child\" ]] || continue\n src=\"${child%/}\"\n name=\"$(basename \"$src\")\"\n ln -sfn \"$src\" \"$COMBINED_RESULTS_ROOT/$name\"\n done\n done\n\n local combined_csv=\"$REPORT_DIR/combined_traces.csv\"\n build_combined_csv \"$combined_csv\"\n\n echo \"\"\n echo \"=========================================\"\n echo \" Stage 3: Combined eval post-processing\"\n echo \" Scopes: ${SCOPES[*]}\"\n echo \" Results tree: $COMBINED_RESULTS_ROOT\"\n echo \" CSV: $combined_csv\"\n echo \" Report dir: $REPORT_DIR\"\n echo \"=========================================\"\n\n (\n cd \"$EVALS_DIR\" || exit\n run_llm_agent \\\n \"Run eval post processing on results_root=$COMBINED_RESULTS_ROOT suite=$SUITE_NAME test_traces_csv=$combined_csv report_dir=$REPORT_DIR container=${CONTAINER:-} $NODE_LABEL $RUNTIME_LABEL\"\n ) \u003c /dev/null > \"$REPORT_DIR/post_processing.ndjson\" 2>&1\n\n echo \" Post-processing complete.\"\n echo \" PR report: $REPORT_DIR/pr_report.md\"\n echo \" Fix-ticket report: $REPORT_DIR/fix_ticket_report.md\"\n}\n\n# ---------------------------------------------------------------------------\n# Main\n# ---------------------------------------------------------------------------\n\necho \"=========================================\"\necho \" Analysis Eval Pipeline\"\necho \" Scopes: ${SCOPES[*]}\"\necho \" Agent: $AGENT_BACKEND\"\necho \" Node: $NODE_LABEL\"\necho \" Runtime: $RUNTIME_LABEL\"\necho \" Repeats: $NUM_REPEATS\"\necho \" Max parallel: $MAX_PARALLEL\"\nif [[ -n \"$TEST_IDS\" ]]; then\n echo \" Test filter: $TEST_IDS\"\nfi\nif [[ \"$SKIP_GENERATE_REF\" == \"1\" ]]; then\n echo \" Skip reference generation: TRUE\"\nfi\necho \" Report dir: $REPORT_DIR\"\necho \"=========================================\"\necho \"\"\n\n# Stage 1: regenerate golden references from scratch for each scope.\nif [[ \"$SKIP_GENERATE_REF\" == \"1\" ]]; then\n echo \" Stage 1 skipped -- SKIP_GENERATE_REF=1.\"\n echo \"\"\nelse\n for scope in \"${SCOPES[@]}\"; do\n generate_refs_for_scope \"$scope\"\n done\nfi\n\n# Stage 2: run repeatability evals for each scope against the fresh references.\nfor scope in \"${SCOPES[@]}\"; do\n run_repeatability_for_scope \"$scope\"\ndone\n\n# Stage 3: one combined post-processing pass -> single reports.\nrun_post_processing\n\necho \"\"\necho \"=========================================\"\necho \" Pipeline finished.\"\necho \" Reports in: $REPORT_DIR\"\necho \"=========================================\"\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":24915,"content_sha256":"025077e3012383dd49d411d13fa0de3c8431f8964ae7b55b6716a07b29a94927"},{"filename":"evals/eval_utils/aggregate_repeatability.py","content":"#!/usr/bin/env python3\n###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\n\"\"\"Aggregate repeatability test results from eval_summary.csv and analysis_stream.ndjson.\"\"\"\n\nimport csv\nimport json\nimport os\nimport re\nfrom collections import defaultdict\n\nfrom TraceLens.PerfModel.utils import optional_int\n\nRESULTS_ROOT = os.environ.get(\n \"RESULTS_ROOT\",\n os.path.join(\n os.path.dirname(os.path.dirname(os.path.abspath(__file__))),\n \"repeatability_results\",\n ),\n)\nOUTPUT_DIR = os.environ.get(\n \"OUTPUT_DIR\",\n os.path.join(os.path.dirname(os.path.abspath(__file__)), \"output\"),\n)\n\n\ndef find_runs(results_root):\n runs = []\n for trace_id in sorted(os.listdir(results_root)):\n trace_dir = os.path.join(results_root, trace_id)\n if not os.path.isdir(trace_dir):\n continue\n for entry in sorted(os.listdir(trace_dir)):\n m = re.match(r\"run_(\\d+)$\", entry)\n if m:\n runs.append((trace_id, int(m.group(1)), os.path.join(trace_dir, entry)))\n return runs\n\n\ndef aggregate_eval_summaries(runs):\n rows = []\n for trace_id, run_num, run_dir in runs:\n csv_path = os.path.join(run_dir, \"eval_summary.csv\")\n if not os.path.isfile(csv_path):\n rows.append(\n {\n \"trace_id\": trace_id,\n \"run_id\": run_num,\n \"eval_index\": \"MISSING\",\n \"eval_category\": \"\",\n \"issue_summary\": \"eval_summary.csv not found\",\n \"result\": \"MISSING\",\n \"details\": \"\",\n }\n )\n continue\n with open(csv_path, newline=\"\") as f:\n for row in csv.DictReader(f):\n rows.append(\n {\n \"trace_id\": trace_id,\n \"run_id\": run_num,\n \"eval_index\": row.get(\"index\", \"\"),\n \"eval_category\": row.get(\"category\", \"\"),\n \"issue_summary\": row.get(\"issue_summary\", \"\"),\n \"result\": row.get(\"result\", \"\"),\n \"details\": row.get(\"details\", \"\"),\n \"root_cause\": row.get(\"root_cause\", \"\"),\n \"recommended_fix\": row.get(\"recommended_fix\", \"\"),\n }\n )\n return rows\n\n\ndef _classify_stability(pass_count: int, total: int) -> str:\n \"\"\"Classify eval stability across repeated runs.\n\n Returns one of:\n STABLE_PASS - all runs passed\n FLAKY_PASS - >50% passed but not all\n FLAKY_FAIL - >0% passed but \u003c=50%\n STABLE_FAIL - all runs failed\n \"\"\"\n if total == 0:\n return \"N/A\"\n rate = pass_count / total\n if rate == 1.0:\n return \"STABLE_PASS\"\n elif rate > 0.5:\n return \"FLAKY_PASS\"\n elif rate > 0.0:\n return \"FLAKY_FAIL\"\n else:\n return \"STABLE_FAIL\"\n\n\ndef build_pass_rate_summary(rows):\n counts = defaultdict(lambda: {\"pass\": 0, \"fail\": 0, \"total\": 0})\n trace_totals = defaultdict(lambda: {\"pass\": 0, \"fail\": 0, \"total\": 0})\n for r in rows:\n if r[\"result\"] == \"MISSING\":\n continue\n key = (r[\"trace_id\"], r[\"eval_index\"])\n counts[key][\"total\"] += 1\n trace_totals[r[\"trace_id\"]][\"total\"] += 1\n if r[\"result\"] == \"PASS\":\n counts[key][\"pass\"] += 1\n trace_totals[r[\"trace_id\"]][\"pass\"] += 1\n else:\n counts[key][\"fail\"] += 1\n trace_totals[r[\"trace_id\"]][\"fail\"] += 1\n all_evals = sorted({k[1] for k in counts})\n all_traces = sorted({k[0] for k in counts})\n summary_rows = []\n for trace_id in all_traces:\n row = {\"trace_id\": trace_id}\n for ev in all_evals:\n c = counts.get((trace_id, ev))\n row[ev] = (\n \"{}/{}\".format(c[\"pass\"], c[\"total\"]) if c and c[\"total\"] > 0 else \"N/A\"\n )\n tt = trace_totals[trace_id]\n row[\"overall_pass_rate\"] = (\n \"{}/{} ({:.0f}%)\".format(\n tt[\"pass\"], tt[\"total\"], 100 * tt[\"pass\"] / tt[\"total\"]\n )\n if tt[\"total\"] > 0\n else \"N/A\"\n )\n summary_rows.append(row)\n return summary_rows, [\"trace_id\"] + all_evals + [\"overall_pass_rate\"]\n\n\ndef build_stability_summary(rows):\n \"\"\"Build a per-(trace, eval) stability classification table.\"\"\"\n counts = defaultdict(lambda: {\"pass\": 0, \"total\": 0})\n for r in rows:\n if r[\"result\"] == \"MISSING\":\n continue\n key = (r[\"trace_id\"], r[\"eval_index\"])\n counts[key][\"total\"] += 1\n if r[\"result\"] == \"PASS\":\n counts[key][\"pass\"] += 1\n\n all_evals = sorted({k[1] for k in counts})\n all_traces = sorted({k[0] for k in counts})\n stability_rows = []\n class_totals = defaultdict(int)\n for trace_id in all_traces:\n row = {\"trace_id\": trace_id}\n for ev in all_evals:\n c = counts.get((trace_id, ev))\n if c and c[\"total\"] > 0:\n label = _classify_stability(c[\"pass\"], c[\"total\"])\n row[ev] = label\n class_totals[label] += 1\n else:\n row[ev] = \"N/A\"\n stability_rows.append(row)\n return stability_rows, [\"trace_id\"] + all_evals, dict(class_totals)\n\n\ndef _extract_tool_stdout(result):\n \"\"\"Pull stdout text from pi-style tool_execution_end result payloads.\"\"\"\n if not result:\n return \"\"\n parts = []\n for item in result.get(\"content\") or []:\n if isinstance(item, dict) and item.get(\"type\") == \"text\":\n parts.append(item.get(\"text\", \"\"))\n return \"\".join(parts)\n\n\ndef _accumulate_usage(usage, totals):\n \"\"\"Sum token usage fields across pi turn records.\"\"\"\n if not usage:\n return\n totals[\"input\"] += (\n usage.get(\"inputTokens\") or usage.get(\"input_tokens\") or usage.get(\"input\") or 0\n )\n totals[\"output\"] += (\n usage.get(\"outputTokens\")\n or usage.get(\"output_tokens\")\n or usage.get(\"output\")\n or 0\n )\n totals[\"cache_read\"] += (\n usage.get(\"cacheReadTokens\")\n or usage.get(\"cached_input_tokens\")\n or usage.get(\"cache_read_input_tokens\")\n or usage.get(\"cacheRead\")\n or usage.get(\"cache_read\")\n or 0\n )\n\n\ndef _detect_steps_from_shell(cmd, stdout, steps):\n if \"generate_perf_report\" in cmd:\n steps.add(\"step1_perf_report\")\n if \"orchestrator_prepare\" in cmd:\n steps.add(\"step2_5_prepare\")\n if \"category_manifest\" in stdout or \"category_manifest\" in cmd:\n steps.add(\"step2_5_prepare\")\n if \"_findings.md\" in cmd or \"_findings.md\" in stdout:\n steps.add(\"step7_subagent_findings\")\n if \"multi_kernel_findings\" in stdout or \"cpu_idle\" in cmd:\n steps.add(\"step6_system_analysis\")\n if \"analysis.md\" in cmd:\n steps.add(\"step11_report\")\n\n\ndef parse_ndjson_stream(ndjson_path):\n diag = {\n \"outcome\": \"unknown\",\n \"duration_ms\": \"\",\n \"input_tokens\": \"\",\n \"output_tokens\": \"\",\n \"cache_read_tokens\": \"\",\n \"turns\": 0,\n \"tool_calls\": 0,\n \"report_written\": False,\n \"report_headers\": \"\",\n \"last_step_reached\": \"none\",\n }\n if not os.path.isfile(ndjson_path):\n diag[\"outcome\"] = \"missing_file\"\n return diag\n if os.path.getsize(ndjson_path) \u003c 100:\n with open(ndjson_path) as f:\n raw = f.read().strip()\n if \"unavailable\" in raw.lower() or \"service unavailable\" in raw.lower():\n diag[\"outcome\"] = \"agent_cli_unavailable\"\n elif \"error\" in raw.lower():\n diag[\"outcome\"] = \"agent_cli_error\"\n else:\n diag[\"outcome\"] = \"empty_or_minimal\"\n return diag\n\n turn_ids, tool_call_count, result_record, steps_reached, report_content = (\n set(),\n 0,\n None,\n set(),\n None,\n )\n agent_end_record = None\n pi_turn_count = 0\n pi_tool_calls = 0\n first_ts = None\n last_ts = None\n token_totals = {\"input\": 0, \"output\": 0, \"cache_read\": 0}\n\n with open(ndjson_path) as f:\n for line in f:\n line = line.strip()\n if not line:\n continue\n try:\n rec = json.loads(line)\n except json.JSONDecodeError:\n if (\n \"unavailable\" in line.lower()\n or \"service unavailable\" in line.lower()\n ):\n diag[\"outcome\"] = \"agent_cli_unavailable\"\n return diag\n continue\n rec_type = rec.get(\"type\", \"\")\n if rec_type == \"result\":\n result_record = rec\n elif rec_type == \"agent_end\":\n agent_end_record = rec\n elif rec_type == \"turn_end\":\n pi_turn_count += 1\n msg = rec.get(\"message\") or {}\n ts = msg.get(\"timestamp\")\n if ts is not None:\n if first_ts is None:\n first_ts = ts\n last_ts = ts\n _accumulate_usage(msg.get(\"usage\") or {}, token_totals)\n elif rec_type == \"tool_execution_start\":\n pi_tool_calls += 1\n cmd = (rec.get(\"args\") or {}).get(\"command\", \"\")\n if cmd:\n _detect_steps_from_shell(cmd, \"\", steps_reached)\n c = _detect_report_write_from_command(\n {\"shellToolCall\": {\"args\": {\"command\": cmd}}}\n )\n if c is not None:\n report_content = c\n elif rec_type == \"tool_execution_end\":\n args = rec.get(\"args\") or {}\n cmd = args.get(\"command\", \"\")\n stdout = _extract_tool_stdout(rec.get(\"result\"))\n if cmd or stdout:\n _detect_steps_from_shell(cmd, stdout, steps_reached)\n tc = {\n \"shellToolCall\": {\n \"args\": {\"command\": cmd},\n \"result\": {\"success\": {\"stdout\": stdout}},\n }\n }\n c = _detect_report_write(tc)\n if c is not None:\n report_content = c\n elif rec_type == \"tool_call\":\n mcid = rec.get(\"model_call_id\", \"\")\n if mcid:\n parts = mcid.rsplit(\"-\", 2)\n if len(parts) >= 3:\n turn_id = optional_int(parts[-2])\n if turn_id is not None:\n turn_ids.add(turn_id)\n if rec.get(\"subtype\") == \"started\":\n tool_call_count += 1\n tc = rec.get(\"tool_call\", {})\n if rec.get(\"subtype\") == \"completed\":\n _detect_steps(tc, steps_reached)\n c = _detect_report_write(tc)\n if c is not None:\n report_content = c\n elif rec.get(\"subtype\") == \"started\":\n c = _detect_report_write_from_command(tc)\n if c is not None:\n report_content = c\n elif rec_type == \"assistant\":\n mcid = rec.get(\"model_call_id\", \"\")\n if mcid:\n parts = mcid.rsplit(\"-\", 2)\n if len(parts) >= 3:\n turn_id = optional_int(parts[-2])\n if turn_id is not None:\n turn_ids.add(turn_id)\n\n diag[\"turns\"] = pi_turn_count if pi_turn_count else len(turn_ids)\n diag[\"tool_calls\"] = pi_tool_calls if pi_tool_calls else tool_call_count\n\n completion = result_record or agent_end_record\n if completion:\n diag[\"outcome\"] = \"error\" if completion.get(\"is_error\") else \"success\"\n duration_ms = completion.get(\"duration_ms\")\n if duration_ms:\n diag[\"duration_ms\"] = duration_ms\n usage = completion.get(\"usage\") or {}\n if usage:\n diag[\"input_tokens\"] = (\n usage.get(\"inputTokens\")\n or usage.get(\"input_tokens\")\n or usage.get(\"input\")\n or \"\"\n )\n diag[\"output_tokens\"] = (\n usage.get(\"outputTokens\")\n or usage.get(\"output_tokens\")\n or usage.get(\"output\")\n or \"\"\n )\n diag[\"cache_read_tokens\"] = (\n usage.get(\"cacheReadTokens\")\n or usage.get(\"cacheReadInputTokens\")\n or usage.get(\"cached_input_tokens\")\n or usage.get(\"cache_read_input_tokens\")\n or usage.get(\"cacheRead\")\n or usage.get(\"cache_read\")\n or \"\"\n )\n elif diag[\"outcome\"] == \"unknown\":\n diag[\"outcome\"] = \"no_result_record\"\n\n if not diag[\"duration_ms\"] and first_ts is not None and last_ts is not None:\n diag[\"duration_ms\"] = last_ts - first_ts\n\n if not diag[\"input_tokens\"] and token_totals[\"input\"]:\n diag[\"input_tokens\"] = token_totals[\"input\"]\n if not diag[\"output_tokens\"] and token_totals[\"output\"]:\n diag[\"output_tokens\"] = token_totals[\"output\"]\n if not diag[\"cache_read_tokens\"] and token_totals[\"cache_read\"]:\n diag[\"cache_read_tokens\"] = token_totals[\"cache_read\"]\n\n if report_content is not None:\n diag[\"report_written\"] = True\n headers = re.findall(r\"^##\\s+(.+)$\", report_content, re.MULTILINE)\n diag[\"report_headers\"] = \" | \".join(headers) if headers else \"(no headers)\"\n diag[\"last_step_reached\"] = _compute_last_step(steps_reached)\n return diag\n\n\ndef _detect_steps(tc, steps):\n shell = tc.get(\"shellToolCall\", {})\n cmd = shell.get(\"args\", {}).get(\"command\", \"\")\n res = shell.get(\"result\", {}).get(\"success\", {})\n stdout = res.get(\"stdout\", \"\")\n _detect_steps_from_shell(cmd, stdout, steps)\n rp = tc.get(\"readToolCall\", {}).get(\"args\", {}).get(\"path\", \"\")\n if \"_metrics.json\" in rp:\n steps.add(\"step7_subagent_findings\")\n if \"category_manifest\" in rp:\n steps.add(\"step2_5_prepare\")\n\n\ndef _detect_report_write(tc):\n \"\"\"Detect report write from completed tool_call (check stdout and interleavedOutput).\"\"\"\n shell = tc.get(\"shellToolCall\", {})\n cmd = shell.get(\"args\", {}).get(\"command\", \"\")\n res = shell.get(\"result\", {}).get(\"success\", {})\n if \"analysis.md\" in cmd:\n for key in (\"stdout\", \"interleavedOutput\"):\n content = res.get(key, \"\")\n if content and len(content) > 100:\n return content\n wc = tc.get(\"writeToolCall\", {})\n if wc and \"analysis.md\" in wc.get(\"args\", {}).get(\"path\", \"\"):\n content = wc.get(\"args\", {}).get(\"content\", \"\")\n if content:\n return content\n return None\n\n\ndef _detect_report_write_from_command(tc):\n \"\"\"Extract report content from heredoc in the command string (started tool_call).\"\"\"\n shell = tc.get(\"shellToolCall\", {})\n cmd = shell.get(\"args\", {}).get(\"command\", \"\")\n if \"analysis.md\" not in cmd:\n return None\n if \"tee \" not in cmd and \"cat >\" not in cmd:\n return None\n for marker in (\"REPORT_EOF\", \"EOF\", \"HEREDOC\", \"MD_EOF\", \"ANALYSIS_EOF\"):\n pattern = r\"\u003c\u003c\\s*'?\" + re.escape(marker) + r\"'?\\n(.*?)\\n\" + re.escape(marker)\n m = re.search(pattern, cmd, re.DOTALL)\n if m:\n return m.group(1)\n eof_match = re.search(r\"\u003c\u003c\\s*'?(\\w+)'?\\n(.*)\", cmd, re.DOTALL)\n if eof_match:\n delimiter = eof_match.group(1)\n rest = eof_match.group(2)\n end_pos = rest.rfind(\"\\n\" + delimiter)\n if end_pos > 0:\n return rest[:end_pos]\n return None\n\n\ndef _compute_last_step(steps):\n for key, label in [\n (\"step11_report\", \"Step 11: Report\"),\n (\"step7_subagent_findings\", \"Step 7: Subagent findings\"),\n (\"step6_system_analysis\", \"Step 6: System analysis\"),\n (\"step2_5_prepare\", \"Steps 2-5: Prepare\"),\n (\"step1_perf_report\", \"Step 1: Perf report\"),\n ]:\n if key in steps:\n return label\n return \"none\"\n\n\ndef build_run_level_summary(rows):\n \"\"\"Build per-(trace, run) failure counts and flag catastrophic runs.\n\n A run is catastrophic when >50% of its checks fail, which typically\n indicates the agent crashed or produced no analysis.md rather than\n individual eval issues.\n \"\"\"\n counts = defaultdict(lambda: {\"pass\": 0, \"fail\": 0, \"total\": 0})\n for r in rows:\n if r[\"result\"] in (\"MISSING\", \"\"):\n continue\n key = (r[\"trace_id\"], r[\"run_id\"])\n counts[key][\"total\"] += 1\n if r[\"result\"] == \"PASS\":\n counts[key][\"pass\"] += 1\n else:\n counts[key][\"fail\"] += 1\n\n run_rows = []\n for (trace_id, run_id), c in sorted(counts.items()):\n is_catastrophic = c[\"fail\"] > c[\"total\"] * 0.5 if c[\"total\"] > 0 else False\n run_rows.append(\n {\n \"trace_id\": trace_id,\n \"run_id\": run_id,\n \"pass\": c[\"pass\"],\n \"fail\": c[\"fail\"],\n \"total\": c[\"total\"],\n \"is_catastrophic\": is_catastrophic,\n }\n )\n return run_rows\n\n\ndef build_failure_nature_summary(rows, run_level_rows):\n \"\"\"Classify total failures into stable vs flaky vs catastrophic.\n\n - catastrophic_pipeline: failures from runs where >50% of checks failed\n - stable: failures for evals that fail in every run for a given trace\n - flaky: everything else\n \"\"\"\n catastrophic_runs = {\n (r[\"trace_id\"], r[\"run_id\"]) for r in run_level_rows if r[\"is_catastrophic\"]\n }\n\n per_trace_runs = defaultdict(set)\n per_eval_fail_runs = defaultdict(set)\n for r in rows:\n if r[\"result\"] in (\"MISSING\", \"\"):\n continue\n per_trace_runs[r[\"trace_id\"]].add(r[\"run_id\"])\n if r[\"result\"] == \"FAIL\":\n per_eval_fail_runs[(r[\"trace_id\"], r[\"issue_summary\"])].add(r[\"run_id\"])\n\n catastrophic_count = 0\n stable_count = 0\n flaky_count = 0\n\n for r in rows:\n if r[\"result\"] != \"FAIL\":\n continue\n key = (r[\"trace_id\"], r[\"run_id\"])\n if key in catastrophic_runs:\n catastrophic_count += 1\n elif (\n per_eval_fail_runs[(r[\"trace_id\"], r[\"issue_summary\"])]\n == per_trace_runs[r[\"trace_id\"]]\n ):\n stable_count += 1\n else:\n flaky_count += 1\n\n return {\n \"catastrophic_pipeline\": catastrophic_count,\n \"stable\": stable_count,\n \"flaky\": flaky_count,\n \"total\": catastrophic_count + stable_count + flaky_count,\n }\n\n\nRUN_LEVEL_COLS = [\"trace_id\", \"run_id\", \"pass\", \"fail\", \"total\", \"is_catastrophic\"]\n\n\ndef aggregate_stream_diagnostics(runs):\n rows = []\n for trace_id, run_num, run_dir in runs:\n diag = parse_ndjson_stream(os.path.join(run_dir, \"analysis_stream.ndjson\"))\n diag[\"trace_id\"] = trace_id\n diag[\"run_id\"] = run_num\n rows.append(diag)\n return rows\n\n\nSTREAM_COLS = [\n \"trace_id\",\n \"run_id\",\n \"outcome\",\n \"duration_ms\",\n \"input_tokens\",\n \"output_tokens\",\n \"cache_read_tokens\",\n \"turns\",\n \"tool_calls\",\n \"report_written\",\n \"report_headers\",\n \"last_step_reached\",\n]\nEVAL_COLS = [\n \"trace_id\",\n \"run_id\",\n \"eval_index\",\n \"eval_category\",\n \"issue_summary\",\n \"result\",\n \"details\",\n \"root_cause\",\n \"recommended_fix\",\n]\n\n\ndef write_csv(path, rows, fieldnames):\n with open(path, \"w\", newline=\"\") as f:\n w = csv.DictWriter(f, fieldnames=fieldnames, extrasaction=\"ignore\")\n w.writeheader()\n w.writerows(rows)\n print(\" Written: {} ({} rows)\".format(path, len(rows)))\n\n\ndef main():\n os.makedirs(OUTPUT_DIR, exist_ok=True)\n print(\"Scanning: {}\".format(RESULTS_ROOT))\n runs = find_runs(RESULTS_ROOT)\n print(\"Found {} run directories\\n\".format(len(runs)))\n\n print(\"Part A: Aggregating eval_summary.csv files...\")\n eval_rows = aggregate_eval_summaries(runs)\n write_csv(os.path.join(OUTPUT_DIR, \"aggregated_results.csv\"), eval_rows, EVAL_COLS)\n summary_rows, summary_cols = build_pass_rate_summary(eval_rows)\n write_csv(\n os.path.join(OUTPUT_DIR, \"pass_rate_summary.csv\"), summary_rows, summary_cols\n )\n\n print(\"\\nPart A2: Building run-level summary...\")\n run_level_rows = build_run_level_summary(eval_rows)\n write_csv(\n os.path.join(OUTPUT_DIR, \"run_level_summary.csv\"),\n run_level_rows,\n RUN_LEVEL_COLS,\n )\n catastrophic = [r for r in run_level_rows if r[\"is_catastrophic\"]]\n if catastrophic:\n print(\n \" Catastrophic runs (>50% fail): {}\".format(\n \", \".join(\n \"{}/run_{}\".format(r[\"trace_id\"], r[\"run_id\"]) for r in catastrophic\n )\n )\n )\n\n print(\"\\nPart A3: Building failure nature summary...\")\n failure_nature = build_failure_nature_summary(eval_rows, run_level_rows)\n write_csv(\n os.path.join(OUTPUT_DIR, \"failure_nature_summary.csv\"),\n [failure_nature],\n [\"catastrophic_pipeline\", \"stable\", \"flaky\", \"total\"],\n )\n print(\" Failure nature: {}\".format(failure_nature))\n\n print(\"\\nPart A4: Building stability classification...\")\n stability_rows, stability_cols, class_totals = build_stability_summary(eval_rows)\n write_csv(\n os.path.join(OUTPUT_DIR, \"stability_summary.csv\"),\n stability_rows,\n stability_cols,\n )\n print(\" Stability breakdown: {}\".format(class_totals))\n\n print(\"\\nPart B: Parsing analysis_stream.ndjson files...\")\n stream_rows = aggregate_stream_diagnostics(runs)\n write_csv(\n os.path.join(OUTPUT_DIR, \"stream_diagnostics.csv\"), stream_rows, STREAM_COLS\n )\n\n print(\"\\n=== Summary ===\")\n total_evals = sum(1 for r in eval_rows if r[\"result\"] not in (\"MISSING\", \"\"))\n passed = sum(1 for r in eval_rows if r[\"result\"] == \"PASS\")\n failed = sum(1 for r in eval_rows if r[\"result\"] == \"FAIL\")\n missing = sum(1 for r in eval_rows if r[\"result\"] == \"MISSING\")\n print(\n \"Eval results: {} PASS, {} FAIL, {} MISSING (total rows: {})\".format(\n passed, failed, missing, len(eval_rows)\n )\n )\n if total_evals > 0:\n print(\"Overall pass rate: {:.1f}%\".format(100 * passed / total_evals))\n\n if class_totals:\n stable_pass = class_totals.get(\"STABLE_PASS\", 0)\n flaky_pass = class_totals.get(\"FLAKY_PASS\", 0)\n flaky_fail = class_totals.get(\"FLAKY_FAIL\", 0)\n stable_fail = class_totals.get(\"STABLE_FAIL\", 0)\n ct = stable_pass + flaky_pass + flaky_fail + stable_fail\n print(\n \"Stability: {} STABLE_PASS, {} FLAKY_PASS, {} FLAKY_FAIL, {} STABLE_FAIL \"\n \"(of {} trace-eval pairs)\".format(\n stable_pass, flaky_pass, flaky_fail, stable_fail, ct\n )\n )\n\n if failure_nature[\"total\"] > 0:\n fn = failure_nature\n print(\n \"Failure nature: {} catastrophic ({:.0f}%), {} stable ({:.0f}%), \"\n \"{} flaky ({:.0f}%) of {} total failures\".format(\n fn[\"catastrophic_pipeline\"],\n 100 * fn[\"catastrophic_pipeline\"] / fn[\"total\"],\n fn[\"stable\"],\n 100 * fn[\"stable\"] / fn[\"total\"],\n fn[\"flaky\"],\n 100 * fn[\"flaky\"] / fn[\"total\"],\n fn[\"total\"],\n )\n )\n if catastrophic:\n print(\n \"Catastrophic runs: {} (agent produced no analysis.md, \"\n \"causing all evals to fail)\".format(len(catastrophic))\n )\n\n outcomes = defaultdict(int)\n for r in stream_rows:\n outcomes[r[\"outcome\"]] += 1\n print(\"\\nStream outcomes: {}\".format(dict(outcomes)))\n\n report_written = sum(1 for r in stream_rows if r[\"report_written\"])\n print(\n \"Runs with report written (detected in stream): {}/{}\".format(\n report_written, len(stream_rows)\n )\n )\n\n step_counts = defaultdict(int)\n for r in stream_rows:\n step_counts[r[\"last_step_reached\"]] += 1\n print(\"Last step reached: {}\".format(dict(step_counts)))\n\n print(\"\\nAll outputs in: {}\".format(OUTPUT_DIR))\n\n\nif __name__ == \"__main__\":\n main()\n","content_type":"text/x-python; charset=utf-8","language":"python","size":24657,"content_sha256":"e290d529587c3f21ece57a77763a08f2db9535976fc8d30b50247499c22b0549"},{"filename":"evals/eval_utils/compare_lca_partitions.py","content":"#!/usr/bin/env python3\n###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\n\"\"\"Compare LCA (lowest-common-ancestor) partitions between the \"with capture\"\n(gold standard) and \"no capture\" comparative analyses of the same traces, and\nscore the no-capture partition against semantics-free baselines.\n\nBoth analyses emit a ``diff_stats.csv`` in which every row is a single GPU kernel\n(identified by ``source`` = trace1/trace2 and ``gpu_op_uid``) tagged with a\n``lowest_common_ancestor_id`` (LCA). LCA ids are opaque cluster labels: their\nnumeric value is only meaningful within a single analysis, so we never compare\nids across analyses directly. Instead we measure how well the two LCA\n*partitions* of the shared kernel set agree, using bidirectional cluster purity.\n\nForward purity (gold -> no-capture):\n For every gold LCA group, take the majority no-capture LCA among its kernels\n and count how many kernels in the group carry that majority label. Sum across\n groups and divide by the number of matched kernels.\n\nReverse purity (no-capture -> gold): the same metric with the roles swapped.\n\nReporting both catches degenerate collapses: if the no-capture analysis dumped\nevery kernel into one LCA, forward purity stays high (each gold group trivially\nagrees with the single label) but reverse purity collapses.\n\nStrict consistency\n------------------\nAn all-or-nothing variant of purity: a kernel is *consistent* iff its ENTIRE\ngroup maps to a single group on the other side (no partial credit for the\nmajority). Strict forward = fraction of kernels whose whole gold group lands in\none no-capture LCA; strict reverse swaps the roles. Always \u003c= the matching\npurity, with equality only when every multi-kernel group is perfectly pure.\n\nRandom baselines\n----------------\nTo judge whether the real no-capture purity reflects genuine semantic signal or\nis merely an artifact of the bucket-size distribution and the metric's structure,\nwe compare against semantics-free baselines. All are evaluated on the SAME\nmatched kernels and PRESERVE the no-capture bucket sizes as observed on the\nmatched set -- only the kernel->bucket assignment changes.\n\nBaseline 1 (random): randomly shuffle which matched kernels fall into each\n bucket, preserving bucket sizes. Reported as mean +/- std over many seeds.\nBaseline 2 (sequential blocks, key string order): order buckets from most to\n least popular, then walk the matched kernels ordered by the lexicographic key\n string (\"source:uid\"), assigning the first k1 to bucket 1 (largest), the next\n k2 to bucket 2, and so on.\nBaseline 3 (sequential blocks, integer uid order): identical block assignment,\n but kernels are ordered by (source, integer gpu_op_uid).\n\nUsage:\n compare_lca_partitions.py \u003cwith_capture_diff_stats.csv> \u003cno_capture_diff_stats.csv>\n\"\"\"\n\nimport argparse\nfrom pathlib import Path\n\nimport numpy as np\nimport pandas as pd\n\nKEY_COLS = [\"source\", \"gpu_op_uid\"]\nLCA_COL = \"lowest_common_ancestor_id\"\nLCA_NAME = \"lowest_common_ancestor_name\"\n\n# Number of random-baseline seeds (Baseline 1). Kept modest by default; the\n# baseline loop is O(trials * kernels), so for large kernel sets it is capped to\n# REDUCED_TRIALS (see the guard in main()).\nDEFAULT_TRIALS = 200\nLARGE_KERNEL_THRESHOLD = 5000\nREDUCED_TRIALS = 2\nSEED = 0\n\n\ndef load(path: Path) -> pd.DataFrame:\n \"\"\"Load a diff_stats.csv, validate identity columns, and add a unique key.\"\"\"\n df = pd.read_csv(path)\n missing = [c for c in KEY_COLS + [LCA_COL, LCA_NAME, \"name\"] if c not in df.columns]\n if missing:\n raise ValueError(f\"{path} is missing required columns: {missing}\")\n df = df.copy()\n df[\"key\"] = df[\"source\"].astype(str) + \":\" + df[\"gpu_op_uid\"].astype(str)\n if df[\"key\"].duplicated().any():\n n = int(df[\"key\"].duplicated().sum())\n raise ValueError(\n f\"{path}: {n} duplicate (source, gpu_op_uid) keys; not a unique kernel id\"\n )\n return df\n\n\ndef purity(df: pd.DataFrame, group_col: str, label_col: str):\n \"\"\"Sum over each group of the majority-label count within that group.\n\n Returns (matched_count, total, fraction, per_group_records).\"\"\"\n total = len(df)\n matched = 0\n records = []\n for gid, grp in df.groupby(group_col, sort=True):\n counts = grp[label_col].value_counts() # sorted desc by count\n majority_label = counts.index[0]\n majority_count = int(counts.iloc[0])\n matched += majority_count\n records.append(\n {\n \"group_id\": gid,\n \"group_size\": len(grp),\n \"majority_label\": majority_label,\n \"majority_count\": majority_count,\n \"group_purity\": majority_count / len(grp),\n }\n )\n frac = matched / total if total else float(\"nan\")\n return matched, total, frac, records\n\n\ndef purity_frac(labels_a: np.ndarray, labels_b: np.ndarray) -> float:\n \"\"\"Fraction of items whose A-group carries the A-group's majority B-label.\n\n Sum over groups of A of the modal B-count, divided by the total item count.\n \"\"\"\n df = pd.DataFrame({\"a\": labels_a, \"b\": labels_b})\n matched = 0\n for _, grp in df.groupby(\"a\", sort=False):\n matched += int(grp[\"b\"].value_counts().iloc[0])\n return matched / len(df) if len(df) else float(\"nan\")\n\n\ndef both_purities(gold: np.ndarray, nc: np.ndarray):\n \"\"\"Return (forward, reverse).\n\n forward = gold group -> majority nc label\n reverse = nc group -> majority gold label\n \"\"\"\n return purity_frac(gold, nc), purity_frac(nc, gold)\n\n\ndef strict_consistency_frac(\n group_labels: np.ndarray, other_labels: np.ndarray\n) -> float:\n \"\"\"Fraction of items whose entire group is homogeneous in ``other_labels``.\n\n Group items by ``group_labels``; an item is *consistent* iff every item that\n shares its group carries the same ``other_labels`` value (so the whole group\n maps to a single other-bin). Report the fraction of consistent items.\n\n This is an all-or-nothing variant of ``purity_frac``: a group contributes its\n full size when perfectly pure, else 0 (no partial credit for the majority).\n Hence ``strict_consistency_frac \u003c= purity_frac`` always, with equality only\n when every multi-item group is perfectly pure.\n \"\"\"\n df = pd.DataFrame({\"g\": group_labels, \"o\": other_labels})\n consistent = 0\n for _, grp in df.groupby(\"g\", sort=False):\n if grp[\"o\"].nunique() == 1:\n consistent += len(grp)\n return consistent / len(df)\n\n\ndef both_strict(gold: np.ndarray, nc: np.ndarray):\n \"\"\"Return (strict_forward, strict_reverse).\n\n strict_forward = fraction of items whose gold group is fully pure in nc\n strict_reverse = fraction of items whose nc group is fully pure in gold\n \"\"\"\n return strict_consistency_frac(gold, nc), strict_consistency_frac(nc, gold)\n\n\ndef agreement_report(gold: pd.DataFrame, ncap: pd.DataFrame) -> None:\n \"\"\"Print the LCA-partition agreement (forward/reverse purity) section.\"\"\"\n print(\"=\" * 78)\n print(\"LCA assignment agreement: with-capture (GOLD) vs no-capture\")\n print(\"=\" * 78)\n print(\n f\"with-capture kernels : {len(gold)} | distinct gold LCAs: {gold[LCA_COL].nunique()}\"\n )\n print(\n f\"no-capture kernels : {len(ncap)} | distinct nc LCAs: {ncap[LCA_COL].nunique()}\"\n )\n\n g = gold[[\"key\", \"name\", LCA_COL]].rename(\n columns={LCA_COL: \"lca_gold\", \"name\": \"name_gold\"}\n )\n n = ncap[[\"key\", \"name\", LCA_COL]].rename(\n columns={LCA_COL: \"lca_nc\", \"name\": \"name_nc\"}\n )\n merged = g.merge(n, on=\"key\")\n name_agree = (\n (merged[\"name_gold\"] == merged[\"name_nc\"]).mean()\n if len(merged)\n else float(\"nan\")\n )\n\n gold_only = len(gold) - len(merged)\n nc_only = len(ncap) - len(merged)\n print(\"-\" * 78)\n print(f\"matched kernels (in both) : {len(merged)}\")\n print(f\"gold-only (unmatched) : {gold_only}\")\n print(f\"no-capture-only (unmatched) : {nc_only}\")\n print(\n f\"kernel-name agreement on matched keys: {name_agree:.4f} (sanity: should be 1.0)\"\n )\n\n df = merged[[\"key\", \"lca_gold\", \"lca_nc\"]]\n print(\"-\" * 78)\n print(f\"matched-set distinct gold LCAs: {df['lca_gold'].nunique()}\")\n print(f\"matched-set distinct nc LCAs: {df['lca_nc'].nunique()}\")\n\n # Forward purity: gold groups -> majority no-capture LCA.\n f_matched, f_total, f_frac, _ = purity(df, \"lca_gold\", \"lca_nc\")\n # Reverse purity: no-capture groups -> majority gold LCA.\n r_matched, r_total, r_frac, r_recs = purity(df, \"lca_nc\", \"lca_gold\")\n\n # Strict consistency (all-or-nothing per group; no partial credit).\n gold_lbl = df[\"lca_gold\"].to_numpy()\n nc_lbl = df[\"lca_nc\"].to_numpy()\n sf = strict_consistency_frac(gold_lbl, nc_lbl) # gold groups fully pure in nc\n sr = strict_consistency_frac(nc_lbl, gold_lbl) # nc groups fully pure in gold\n n_tot = len(df)\n\n print(\"=\" * 78)\n print(\"RESULTS\")\n print(\"=\" * 78)\n print(\n f\"Forward purity (gold group -> majority no-capture LCA): \"\n f\"{f_matched}/{f_total} = {f_frac:.4f}\"\n )\n print(\n f\"Reverse purity (no-capture group -> majority gold LCA): \"\n f\"{r_matched}/{r_total} = {r_frac:.4f}\"\n )\n print(\n f\"Strict forward consistency (whole gold group -> one nc LCA): \"\n f\"{int(round(sf * n_tot))}/{n_tot} = {sf:.4f}\"\n )\n print(\n f\"Strict reverse consistency (whole nc group -> one gold LCA): \"\n f\"{int(round(sr * n_tot))}/{n_tot} = {sr:.4f}\"\n )\n print(\"=\" * 78)\n\n # Extra context: the largest no-capture groups (where collapse would show up).\n top_nc = pd.DataFrame(r_recs).sort_values(\"group_size\", ascending=False).head(5)\n print(\"Largest no-capture LCA groups (matched set):\")\n print(top_nc.to_string(index=False))\n\n\ndef baseline_report(merged: pd.DataFrame, title: str, n_trials: int) -> None:\n \"\"\"Compute real + baseline purities for a given matched-kernel frame.\n\n ``merged`` must have columns: lca_gold, lca_nc, nc_name, source, gpu_op_uid,\n key. Bucket sizes are recomputed from ``merged`` so the \"total matched\n preserved\" property holds for whatever subset is passed in.\n \"\"\"\n merged = merged.sort_values(\"key\").reset_index(drop=True)\n n = len(merged)\n\n gold = merged[\"lca_gold\"].to_numpy()\n nc_real = merged[\"lca_nc\"].to_numpy()\n\n # No-capture bucket sizes on this (sub)set (sum == n).\n sizes = merged[\"lca_nc\"].value_counts() # sorted desc by count\n bucket_ids = sizes.index.to_numpy()\n bucket_sizes = sizes.to_numpy()\n names = merged.drop_duplicates(\"lca_nc\").set_index(\"lca_nc\")[\"nc_name\"].to_dict()\n assert bucket_sizes.sum() == n, (bucket_sizes.sum(), n)\n\n # Label pool: bucket id repeated by its size (length == n).\n pool = np.repeat(bucket_ids, bucket_sizes)\n assert len(pool) == n\n\n # ---- Real no-capture ----\n real_fwd, real_rev = both_purities(gold, nc_real)\n real_sfwd, real_srev = both_strict(gold, nc_real)\n\n # ---- Baseline 1: random shuffle of the pool over matched kernels ----\n rng = np.random.default_rng(SEED)\n fwds = np.empty(n_trials)\n revs = np.empty(n_trials)\n sfwds = np.empty(n_trials)\n srevs = np.empty(n_trials)\n for t in range(n_trials):\n shuffled = pool.copy()\n rng.shuffle(shuffled)\n fwds[t], revs[t] = both_purities(gold, shuffled)\n sfwds[t], srevs[t] = both_strict(gold, shuffled)\n b1_fwd_m, b1_fwd_s = fwds.mean(), fwds.std()\n b1_rev_m, b1_rev_s = revs.mean(), revs.std()\n b1_sfwd_m, b1_sfwd_s = sfwds.mean(), sfwds.std()\n b1_srev_m, b1_srev_s = srevs.mean(), srevs.std()\n\n # ---- Baseline 2: sequential blocks, buckets ordered most->least popular ----\n # Kernels ordered by key (lexicographic string: source then string uid).\n seq = pool.copy()\n b2_fwd, b2_rev = both_purities(gold, seq)\n b2_sfwd, b2_srev = both_strict(gold, seq)\n\n # ---- Baseline 3: sequential blocks, kernels ordered by INT gpu_op_uid ----\n order3 = np.lexsort((merged[\"gpu_op_uid\"].to_numpy(), merged[\"source\"].to_numpy()))\n gold3 = gold[order3]\n b3_fwd, b3_rev = both_purities(gold3, pool)\n b3_sfwd, b3_srev = both_strict(gold3, pool)\n\n # ---------------- report ----------------\n print(\"=\" * 74)\n print(f\"{title}\")\n print(\"Matched kernels:\", n, \"| no-capture buckets:\", len(bucket_ids))\n print(\"Distinct gold LCAs:\", merged[\"lca_gold\"].nunique())\n print(\"=\" * 74)\n print(\"No-capture bucket sizes (preserved by all baselines):\")\n for bid, sz in zip(bucket_ids, bucket_sizes):\n print(f\" LCA {bid:>3} {names.get(bid, '?'):\u003c18} {sz:>4}\")\n print(\"-\" * 74)\n hdr = f\"{'partition':\u003c26}{'forward (gold->nc)':>24}{'reverse (nc->gold)':>24}\"\n print(hdr)\n print(\"-\" * 74)\n print(f\"{'Real no-capture':\u003c26}{real_fwd:>24.4f}{real_rev:>24.4f}\")\n print(\n f\"{'Baseline 1 (random)':\u003c26}\"\n f\"{f'{b1_fwd_m:.4f} +/- {b1_fwd_s:.4f}':>24}\"\n f\"{f'{b1_rev_m:.4f} +/- {b1_rev_s:.4f}':>24}\"\n )\n print(f\"{'Baseline 2 (seq, key str)':\u003c26}{b2_fwd:>24.4f}{b2_rev:>24.4f}\")\n print(f\"{'Baseline 3 (seq, int uid)':\u003c26}{b3_fwd:>24.4f}{b3_rev:>24.4f}\")\n print(\"-\" * 74)\n print(\"Strict consistency (fraction of items in a FULLY pure group):\")\n sh = f\"{'partition':\u003c26}{'strict fwd (gold pure)':>24}{'strict rev (nc pure)':>24}\"\n print(sh)\n print(\"-\" * 74)\n print(f\"{'Real no-capture':\u003c26}{real_sfwd:>24.4f}{real_srev:>24.4f}\")\n print(\n f\"{'Baseline 1 (random)':\u003c26}\"\n f\"{f'{b1_sfwd_m:.4f} +/- {b1_sfwd_s:.4f}':>24}\"\n f\"{f'{b1_srev_m:.4f} +/- {b1_srev_s:.4f}':>24}\"\n )\n print(f\"{'Baseline 2 (seq, key str)':\u003c26}{b2_sfwd:>24.4f}{b2_srev:>24.4f}\")\n print(f\"{'Baseline 3 (seq, int uid)':\u003c26}{b3_sfwd:>24.4f}{b3_srev:>24.4f}\")\n print(\"=\" * 74)\n\n\ndef main() -> None:\n ap = argparse.ArgumentParser(\n description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter\n )\n ap.add_argument(\n \"with_capture\", type=Path, help=\"path to with-capture (GOLD) diff_stats.csv\"\n )\n ap.add_argument(\"no_capture\", type=Path, help=\"path to no-capture diff_stats.csv\")\n ap.add_argument(\n \"--trials\",\n type=int,\n default=DEFAULT_TRIALS,\n help=f\"random-baseline seeds (default: {DEFAULT_TRIALS}; auto-capped to \"\n f\"{REDUCED_TRIALS} when matched kernels > {LARGE_KERNEL_THRESHOLD})\",\n )\n args = ap.parse_args()\n\n gold = load(args.with_capture)\n ncap = load(args.no_capture)\n\n # Guard: with no shared identity keys the matched set is empty and every\n # downstream report divides by zero / sorts an empty frame. Fail cleanly.\n if not (set(gold[\"key\"]) & set(ncap[\"key\"])):\n print(\n f\"No matched keys between the two files ({len(gold)} gold rows, \"\n f\"{len(ncap)} no-capture rows, 0 shared keys). Nothing to compare.\"\n )\n return\n\n # ---- Part 1: LCA-partition agreement ----\n agreement_report(gold, ncap)\n print()\n\n # Matched set for the baselines: identity present in both. Deterministic by key.\n merged = (\n gold[[\"key\", LCA_COL]]\n .rename(columns={LCA_COL: \"lca_gold\"})\n .merge(\n ncap[[\"key\", \"source\", \"gpu_op_uid\", LCA_COL, LCA_NAME]].rename(\n columns={LCA_COL: \"lca_nc\", LCA_NAME: \"nc_name\"}\n ),\n on=\"key\",\n )\n )\n\n # Guard: the random baseline is O(trials * kernels); cap trials on large sets.\n n_trials = args.trials\n if len(merged) > LARGE_KERNEL_THRESHOLD:\n n_trials = REDUCED_TRIALS\n print(\n f\"[guard] matched kernels = {len(merged)} > {LARGE_KERNEL_THRESHOLD}; \"\n f\"reducing random-baseline trials {args.trials} -> {n_trials}\"\n )\n print()\n\n # ---- Part 2: random / sequential baselines ----\n baseline_report(merged, \"ALL MATCHED KERNELS\", n_trials)\n\n # Exclude singleton gold groups: gold LCAs with exactly one matched kernel.\n gsz = merged.groupby(\"lca_gold\")[\"key\"].transform(\"size\")\n merged_ns = merged[gsz > 1].copy()\n n_dropped = len(merged) - len(merged_ns)\n print()\n baseline_report(\n merged_ns,\n f\"EXCLUDING SINGLETON GOLD GROUPS (dropped {n_dropped} kernels)\",\n n_trials,\n )\n print(f\"(Baseline 1 over {n_trials} random seeds; Baselines 2 & 3 deterministic.)\")\n\n\nif __name__ == \"__main__\":\n main()\n","content_type":"text/x-python; charset=utf-8","language":"python","size":16624,"content_sha256":"fdae0d19985f7f94e5b13dc767147d2353a1738ae08568e2f6e1f3bb72b2c728"},{"filename":"evals/eval_utils/manual_eval_checklist.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n# Manual Evaluation Checklist\n\nChecks **not** covered by `validate_report()`.\nRun these against every generated `analysis.md` after the programmatic validator passes.\n\n## Link / Anchor Integrity\n\n- [ ] Every P-item card contains a link to its Detailed Analysis anchor (e.g., `[Detailed Analysis: Compute kernel insights > P1](#detailed-analysis-compute-p1)`)\n- [ ] Every Detailed Analysis P-block has a matching HTML anchor (`\u003ca id=\"detailed-analysis-compute-pN\">` or `\u003ca id=\"detailed-analysis-system-pN\">`)\n\n## Label Formatting\n\n- [ ] Each of the five required labels (`**Identification:**`, `**Data:**`, `**Reasoning for Slowdown:**`, `**Resolution:**`, `**Impact estimate:**`) starts on its own line with a newline between consecutive labels\n\n## Card–Detailed Analysis Consistency\n\n- [ ] Every operation mentioned in the card **Insight** appears in the Detailed Analysis **Data** table\n- [ ] Efficiency figures cited in the card match the Detailed Analysis **Data** table values (allowing for rounding)\n- [ ] Time values and percentages in the card match the Detailed Analysis **Data** table (allowing for rounding)\n- [ ] Card **Impact** range is consistent with Detailed Analysis **Impact estimate** bullets\n- [ ] No numbers or claims appear in the card that are absent from the Detailed Analysis block, and vice versa\n\n## Resolution Quality\n\n- [ ] No tautological roofline restatements beyond what the regex validator catches (e.g., novel phrasings of \"getting closer to roofline = faster\")\n- [ ] Resolution explains the **mechanism** by which the optimization raises utilization when inferable from the trace\n- [ ] If the mechanism is not inferable, Resolution states only the action without a roofline truism\n\n## Field Name Placement\n\n- [ ] JSON keys, dotted field paths, internal CSV column names, and internal variable names (e.g. `matrix_bf16`, `peak_mem_bw`) appear **only** in the **Identification** `(source: ...)` parenthetical\n- [ ] **Data**, **Reasoning for Slowdown**, **Resolution**, and **Impact estimate** use plain-language display headers and narrative only\n\n## Metadata Integrity\n\n- [ ] `impact_estimates` in each `metadata/*.json` is an **array** (not a bare object) with exactly one entry per `reasoning-candidate` block, ordered by ascending `rank`. Each entry sums only the operations that its candidate covers — not the entire category\n- [ ] Reasoning candidate blocks (`\u003c!-- reasoning-candidate tier=... rank=... -->`) are present in the corresponding findings files under `category_findings/` or `system_findings/`\n\n## Sentence Quality\n\n- [ ] No run-on sentences used to artificially compress content into the template's suggested sentence count (e.g., cramming 3–4 ideas into 2 sentences with semicolons or em-dashes)\n- [ ] Sentence counts in the template are treated as **guidance for thoroughness**, not hard limits — if the content naturally requires a few more sentences, that is preferred over run-on compression\n\n## Cross-Section Consistency\n\n- [ ] No claim or calculation in the Detailed Analysis section contradicts the corresponding Optimization Insight card\n- [ ] Top Operations table totals and percentages are consistent with per-P-item breakdowns\n- [ ] System-Level Optimizations section content is consistent with `### System-level insights` under Detailed Analysis\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3436,"content_sha256":"4caddfad1cb49330a6802b21f952a22194ad5a9fc9fba5df2b292b3e9d1adc8b"},{"filename":"evals/eval_utils/merge_results.py","content":"###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\n\"\"\"Merge per-test-case eval CSVs into a single summary.\"\"\"\n\nimport argparse\nimport glob\nimport os\n\nimport pandas as pd\n\n\ndef main():\n parser = argparse.ArgumentParser(description=\"Merge eval results\")\n parser.add_argument(\"--results-dir\", required=True)\n parser.add_argument(\"--output\", default=None)\n args = parser.parse_args()\n\n expected_csvs = [\n \"workflow_scripted_results.csv\",\n \"workflow_llm_results.csv\",\n \"quality_scripted_results.csv\",\n \"quality_llm_results.csv\",\n ]\n expected_cols = [\n \"index\",\n \"category\",\n \"issue_summary\",\n \"result\",\n \"details\",\n \"root_cause\",\n \"recommended_fix\",\n ]\n out = args.output or os.path.join(args.results_dir, \"eval_summary.csv\")\n\n frames = []\n missing_csvs = []\n csv_pattern = os.path.join(args.results_dir, \"*_results.csv\")\n found_files = {os.path.basename(p) for p in glob.glob(csv_pattern)}\n\n for csv_path in sorted(glob.glob(csv_pattern)):\n try:\n df = pd.read_csv(csv_path, on_bad_lines=\"warn\")\n except Exception:\n try:\n df = pd.read_csv(\n csv_path,\n sep=None,\n engine=\"python\",\n on_bad_lines=\"warn\",\n )\n except Exception as exc:\n print(f\"Warning: could not parse {csv_path}: {exc}\")\n continue\n frames.append(df)\n\n for expected in expected_csvs:\n if expected not in found_files:\n missing_csvs.append(expected)\n\n if not frames:\n merged = pd.DataFrame(\n [\n {\n \"index\": \"merge_error\",\n \"category\": \"Merge\",\n \"issue_summary\": \"No eval results produced\",\n \"result\": \"FAIL\",\n \"details\": f\"Missing: {', '.join(missing_csvs) if missing_csvs else 'all result CSVs'}\",\n \"root_cause\": \"pipeline\",\n \"recommended_fix\": \"Re-run eval pipeline to produce result CSVs\",\n }\n ]\n )\n merged.to_csv(out, index=False)\n print(f\"No result CSVs found. Missing: {', '.join(missing_csvs)}\")\n print(f\"Partial summary written to {out}\")\n return\n\n merged = pd.concat(frames, ignore_index=True)\n merged = merged[[c for c in expected_cols if c in merged.columns]]\n merged.to_csv(out, index=False)\n\n total = len(merged)\n passed = (merged[\"result\"] == \"PASS\").sum()\n if missing_csvs:\n print(f\"Warning: missing eval CSVs: {', '.join(missing_csvs)}\")\n print(f\"Summary: {passed}/{total} PASS | Written to {out}\")\n\n\nif __name__ == \"__main__\":\n main()\n","content_type":"text/x-python; charset=utf-8","language":"python","size":3023,"content_sha256":"cffbe981a673a15363d52e7600966d55e9fcca5cc5e47d30678bb4ac6d22e2d9"},{"filename":"evals/eval_utils/quality_scripted_evals.py","content":"###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\n\"\"\"Pythonic quality evals.\n\nCompares generated perf report CSVs against reference CSVs\nwith numeric tolerance.\n\"\"\"\n\nimport argparse\nimport csv\nimport os\nimport re\nimport sys\n\nimport pandas as pd\n\nCSV_COLUMNS = [\n \"index\",\n \"category\",\n \"issue_summary\",\n \"result\",\n \"details\",\n \"root_cause\",\n \"recommended_fix\",\n]\nNUMERIC_TOLERANCE = 0.01\nABS_TOLERANCE = 0.05\nOPTIONAL_COLUMN_PREFIXES = (\"Pct Roofline\", \"Roofline Time\")\nOPTIONAL_COLUMNS = {\"num_kernels\"}\n\n_NUMPY_TYPE_RE = re.compile(r\"np\\.\\w+\$([^)]+)\$\")\n\n\ndef _normalize_numpy_reprs(s: str) -> str:\n \"\"\"Strip numpy type wrappers so np.int64(135) becomes 135, etc.\"\"\"\n return _NUMPY_TYPE_RE.sub(r\"\\1\", s)\n\n\ndef _check_csv_alignment_dirs(gen_dir: str, ref_dir: str) -> tuple[str, str]:\n \"\"\"Compare all CSVs in gen_dir against ref_dir. Returns (result, details).\"\"\"\n if not os.path.isdir(ref_dir):\n return \"FAIL\", f\"Reference CSV directory not found: {ref_dir}\"\n if not os.path.isdir(gen_dir):\n return \"FAIL\", f\"Generated CSV directory not found: {gen_dir}\"\n\n ref_files = {f for f in os.listdir(ref_dir) if f.endswith(\".csv\")}\n if not ref_files:\n return \"FAIL\", \"No reference CSVs found\"\n\n mismatches = []\n for fname in sorted(ref_files):\n gen_path = os.path.join(gen_dir, fname)\n if not os.path.isfile(gen_path):\n mismatches.append(f\"{fname}: missing\")\n continue\n\n try:\n ref_df = pd.read_csv(os.path.join(ref_dir, fname), on_bad_lines=\"warn\")\n gen_df = pd.read_csv(gen_path, on_bad_lines=\"warn\")\n except Exception as exc:\n mismatches.append(f\"{fname}: parse error: {exc}\")\n continue\n\n missing_cols = set(ref_df.columns) - set(gen_df.columns)\n missing_cols = {\n c\n for c in missing_cols\n if not any(c.startswith(p) for p in OPTIONAL_COLUMN_PREFIXES)\n and c not in OPTIONAL_COLUMNS\n }\n if missing_cols:\n mismatches.append(f\"{fname}: missing required columns: {missing_cols}\")\n continue\n compare_cols = [c for c in ref_df.columns if c in gen_df.columns]\n gen_df = gen_df[compare_cols]\n if len(ref_df) != len(gen_df):\n mismatches.append(f\"{fname}: row count {len(gen_df)} vs ref {len(ref_df)}\")\n continue\n\n for col in compare_cols:\n if pd.api.types.is_bool_dtype(ref_df[col]) or pd.api.types.is_bool_dtype(\n gen_df[col]\n ):\n if not ref_df[col].equals(gen_df[col]):\n n_diff = (ref_df[col] != gen_df[col]).sum()\n mismatches.append(f\"{fname}:{col} {n_diff} bool value(s) differ\")\n elif pd.api.types.is_numeric_dtype(ref_df[col]):\n if not ref_df[col].equals(gen_df[col]):\n diff = (ref_df[col] - gen_df[col]).abs()\n denom = ref_df[col].abs().replace(0, 1)\n rel_diff = (diff / denom).max()\n abs_diff = diff.max()\n if rel_diff > NUMERIC_TOLERANCE and abs_diff > ABS_TOLERANCE:\n mismatches.append(\n f\"{fname}:{col} max relative diff {rel_diff:.4f}\"\n )\n else:\n ref_norm = (\n ref_df[col].fillna(\"\").astype(str).map(_normalize_numpy_reprs)\n )\n gen_norm = (\n gen_df[col].fillna(\"\").astype(str).map(_normalize_numpy_reprs)\n )\n mask = ref_norm != gen_norm\n if mask.any():\n rows = list(mask[mask].index[:3])\n mismatches.append(f\"{fname}:{col} differs at rows {rows}\")\n\n if mismatches:\n return \"FAIL\", \"; \".join(mismatches[:5])\n return \"PASS\", \"\"\n\n\ndef _pre_check_gates(\n output_dir: str, reference_dir: str, comparison_scope: str\n) -> str | None:\n \"\"\"Return a failure reason if a hard pre-check gate trips, else None.\"\"\"\n if not os.path.isdir(output_dir):\n return \"generated output directory does not exist\"\n if not os.path.isdir(reference_dir):\n return \"reference directory does not exist\"\n\n if comparison_scope == \"comparative\":\n csv_dir = \"perf_report_trace1_csvs\"\n else:\n csv_dir = \"perf_report_csvs\"\n\n if not os.path.isdir(os.path.join(output_dir, csv_dir)):\n return f\"generated {csv_dir}/ directory not found\"\n if not os.path.isdir(os.path.join(reference_dir, csv_dir)):\n return f\"reference {csv_dir}/ directory not found\"\n\n if comparison_scope == \"comparative\":\n if not os.path.isdir(os.path.join(output_dir, \"perf_report_trace2_csvs\")):\n return \"generated perf_report_trace2_csvs/ directory not found\"\n if not os.path.isdir(os.path.join(reference_dir, \"perf_report_trace2_csvs\")):\n return \"reference perf_report_trace2_csvs/ directory not found\"\n\n return None\n\n\ndef run(\n output_dir: str,\n reference_dir: str,\n results_path: str,\n comparison_scope: str = \"standalone\",\n) -> list[dict]:\n rows = []\n\n gate_fail = _pre_check_gates(output_dir, reference_dir, comparison_scope)\n if gate_fail is not None:\n index = (\n \"quality_eval_1\"\n if comparison_scope == \"standalone\"\n else \"quality_eval_1_trace1\"\n )\n rows.append(\n {\n \"index\": index,\n \"category\": \"Quality\",\n \"issue_summary\": \"TraceLens Perf report CSVs alignment\",\n \"result\": \"FAIL\",\n \"details\": f\"Pre-check gate: {gate_fail}\",\n \"root_cause\": \"pipeline\",\n \"recommended_fix\": \"Fix pre-check gate failure before running evals\",\n }\n )\n with open(results_path, \"w\", newline=\"\") as f:\n writer = csv.DictWriter(f, fieldnames=CSV_COLUMNS)\n writer.writeheader()\n writer.writerows(rows)\n return rows\n\n if comparison_scope == \"standalone\":\n gen_dir = os.path.join(output_dir, \"perf_report_csvs\")\n ref_dir = os.path.join(reference_dir, \"perf_report_csvs\")\n result, details = _check_csv_alignment_dirs(gen_dir, ref_dir)\n rows.append(\n {\n \"index\": \"quality_eval_1\",\n \"category\": \"Quality\",\n \"issue_summary\": \"TraceLens Perf report CSVs alignment\",\n \"result\": result,\n \"details\": details,\n \"root_cause\": \"data\" if result == \"FAIL\" else \"\",\n \"recommended_fix\": (\n \"Regenerate golden refs with current TraceLens version\"\n if result == \"FAIL\"\n else \"\"\n ),\n }\n )\n else:\n # Comparative: check trace1 and trace2 CSVs separately\n for trace_num in (1, 2):\n gen_dir = os.path.join(output_dir, f\"perf_report_trace{trace_num}_csvs\")\n ref_dir = os.path.join(reference_dir, f\"perf_report_trace{trace_num}_csvs\")\n result, details = _check_csv_alignment_dirs(gen_dir, ref_dir)\n rows.append(\n {\n \"index\": f\"quality_eval_1_trace{trace_num}\",\n \"category\": \"Quality\",\n \"issue_summary\": f\"TraceLens Perf report CSVs alignment (trace{trace_num})\",\n \"result\": result,\n \"details\": details,\n \"root_cause\": \"data\" if result == \"FAIL\" else \"\",\n \"recommended_fix\": (\n \"Regenerate golden refs with current TraceLens version\"\n if result == \"FAIL\"\n else \"\"\n ),\n }\n )\n\n with open(results_path, \"w\", newline=\"\") as f:\n writer = csv.DictWriter(f, fieldnames=CSV_COLUMNS)\n writer.writeheader()\n writer.writerows(rows)\n return rows\n\n\ndef main():\n parser = argparse.ArgumentParser(description=\"Pythonic quality evals\")\n parser.add_argument(\"--output-dir\", required=True)\n parser.add_argument(\"--reference-dir\", required=True)\n parser.add_argument(\"--results\", required=True)\n parser.add_argument(\n \"--comparison-scope\",\n choices=[\"standalone\", \"comparative\"],\n default=\"standalone\",\n help=\"Analysis mode (default: standalone)\",\n )\n args = parser.parse_args()\n\n rows = run(args.output_dir, args.reference_dir, args.results, args.comparison_scope)\n passed = sum(1 for r in rows if r[\"result\"] == \"PASS\")\n sys.exit(0 if passed == len(rows) else 1)\n\n\nif __name__ == \"__main__\":\n main()\n","content_type":"text/x-python; charset=utf-8","language":"python","size":8988,"content_sha256":"ea2f00117b36dfabe5f54277a2b71a5fd43e60c6f1a874ff8a8e7c87dec1c8fa"},{"filename":"evals/eval_utils/report_section_rules.yaml","content":"###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\nbase_section_rules:\n - section: \"Kernel Fusion\"\n eval_index_regex: \"^workflow_eval_(13|16)$\"\n - section: \"Kernel Fusion\"\n issue_summary_regex: \"(?i)kernel\\\\s*fusion|fusion\\\\s+metrics|fusion\\\\s+insights\"\n - section: \"Reasoning\"\n eval_index_regex: \"^quality_eval_(2|3)$\"\n - section: \"Reasoning\"\n issue_summary_regex: \"(?i)content\\\\s+alignment|title\\\\s+alignment|reasoning\"\n\nstandalone_section_rules:\n - section: \"Executive Summary\"\n eval_index_regex: \"^workflow_eval_10$\"\n - section: \"Executive Summary\"\n issue_summary_regex: \"(?i)executive\\\\s+summary\"\n - section: \"Compute Kernel Optimizations\"\n eval_index_regex: \"^quality_eval_(2|3)$\"\n - section: \"Compute Kernel Optimizations\"\n eval_index_regex: \"^workflow_eval_11$\"\n - section: \"Compute Kernel Optimizations\"\n issue_summary_regex: \"(?i)compute\\\\s+issue|compute\\\\s+kernel|issue\\\\s+template\"\n - section: \"Kernel Fusion Opportunities\"\n eval_index_regex: \"^workflow_eval_(13|16)$\"\n - section: \"Kernel Fusion Opportunities\"\n issue_summary_regex: \"(?i)kernel\\\\s*fusion|fusion\\\\s+metrics|fusion\\\\s+insights\"\n - section: \"System-Level Optimizations\"\n eval_index_regex: \"^workflow_eval_6$\"\n - section: \"System-Level Optimizations\"\n issue_summary_regex: \"(?i)cpu_idle|system[-\\\\s]level|categorical\\\\s+findings\"\n - section: \"Detailed Analysis\"\n eval_index_regex: \"^workflow_eval_(9|12)$\"\n - section: \"Detailed Analysis\"\n issue_summary_regex: \"(?i)impact_estimates|impact\\\\s+estimate\"\n - section: \"Detailed Analysis\"\n issue_summary_regex: \"(?i)detailed\\\\s+analysis|five-label|anchors?\"\n - section: \"Appendix\"\n eval_index_regex: \"^workflow_eval_13$\"\n - section: \"Appendix\"\n issue_summary_regex: \"(?i)hardware\\\\s+reference|model\\\\s+identification|model\\\\s+info\"\n\nfailure_mode_rules:\n - match_regex: \"(?i)not\\\\s+quantifiable|impact\\\\s+estimate\"\n likely_cause: \"Impact estimate data exists but is not consistently rendered into final cards.\"\n suggested_fix: \"Verify the sub-agent emitted impact markers with correct low/high values from category_findings[].\"\n - match_regex: \"(?i)efficiency|alignment\"\n likely_cause: \"Generated reasoning drifts from metrics JSON or reference wording.\"\n suggested_fix: \"Add a final consistency pass that checks efficiency, shapes, and gains against metrics artifacts.\"\n - match_regex: \"(?i)kernel\\\\s*fusion\"\n likely_cause: \"Fusion findings/metrics are not always propagated to report tables.\"\n suggested_fix: \"Render Kernel Fusion Insights from deterministic metrics outputs and validate required tables before publish.\"\n - match_regex: \"(?i)model\\\\s+info|model\\\\s+identification|appendix\"\n likely_cause: \"Metadata fallback is missing or late in the pipeline.\"\n suggested_fix: \"Always write a fallback model metadata skeleton during prepare and enforce appendix fill checks.\"\n - match_regex: \"(?i)csv\\\\s+alignment|SDPA_fwd|SDPA_bwd|unified_perf_summary\"\n likely_cause: \"Trace-dependent CSV presence and schema differ from fixed expectations.\"\n suggested_fix: \"Make CSV checks trace-aware and guarantee stable summary column emission.\"\n - match_regex: \"(?i)cpu_idle|findings\\\\.md|subagent\"\n likely_cause: \"Parallel subagent execution is occasionally flaky under load.\"\n suggested_fix: \"Add retry-on-missing-findings and/or prioritize system subagents before compute fan-out.\"\n\ndefaults:\n base_section: \"Others\"\n standalone_section: \"Detailed Analysis\"\n likely_cause: \"Pipeline or LLM variability caused this failure mode.\"\n suggested_fix: \"Add deterministic post-generation validation and targeted retries for missing artifacts.\"\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":3890,"content_sha256":"762e4871f915039db287187baddf86a439b13d3a8265176cc59575a7724b6274"},{"filename":"evals/eval_utils/workflow_scripted_evals.py","content":"###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\n\"\"\"Pythonic workflow evals.\n\nChecks directory structure, file existence, and output completeness\nagainst the category_manifest.json contract. Also includes deterministic\nmarker identification checks (top_ops, p_item, detail_estimate).\n\nSupports both standalone and comparative analysis modes via --comparison-scope flag.\n\"\"\"\n\nimport argparse\nimport csv\nimport json\nimport os\nimport re\nimport sys\n\nCSV_COLUMNS = [\n \"index\",\n \"category\",\n \"issue_summary\",\n \"result\",\n \"details\",\n \"root_cause\",\n \"recommended_fix\",\n]\n_REQUIRED_MODEL_KEYS = {\"model\", \"architecture\", \"scale\", \"precision\"}\n\n_MIN_REPORT_BYTES = 100\n_GARBLED_THRESHOLD = 0.50\n\n# Marker validation regexes — identical to those in MarkerValidator\n# (TraceLens/Agent/Analysis/utils/validation_utils.py). Defined here directly\n# because validation_utils.py has relative imports that prevent standalone loading.\n_MV_BEGIN_RE = re.compile(r\"\u003c!--\\s*impact-begin\\s+([^>]*?)-->\", re.DOTALL)\n_MV_END_RE = re.compile(r\"\u003c!--\\s*impact-end\\s*-->\")\n_MV_KIND_ATTR_RE = re.compile(r\"\\bkind=(\\w+)\\b\")\n_MV_ATTR_RE = re.compile(r\"\\b(\\w+)=([^\\s]+)\")\n_MV_REPORT_BEGIN_RE = re.compile(r\"\u003c!--\\s*report-begin\\s+([^>]*?)-->\", re.DOTALL)\n_TOP_OPS_ROW_RE = re.compile(r\"\u003c!--\\s*top-ops-row\\s+([^>]*?)-->\")\n_DETAIL_P_HEADER_RE = re.compile(r\"^####\\s+.+P(\\d+):\", re.MULTILINE)\n_DETAIL_P_HEADER_FALLBACK_RE = re.compile(r\"^(?:####|###)\\s+.+P(\\d+):\", re.MULTILINE)\n_REASONING_CANDIDATE_RE = re.compile(\n r\"\u003c!--\\s*reasoning-candidate\\s+tier=(\\w+)\\s+rank=(\\d+)\\s*-->\"\n)\n_NOT_QUANTIFIABLE_RE = re.compile(\n r\"not\\s+quantifiable\\s+from\\s+trace\\s+data\", re.IGNORECASE\n)\n\n\ndef _pre_check_gates(output_dir: str) -> str | None:\n \"\"\"Return a failure reason if a hard pre-check gate trips, else None.\n\n Gates (applied in order, first failure wins):\n 1. output_dir does not exist\n 2. analysis.md missing or too small (\u003c 100 bytes)\n 3. analysis.md is garbled (> 50% non-ASCII or looks like JSON)\n \"\"\"\n if not os.path.isdir(output_dir):\n return \"output directory does not exist\"\n\n report = os.path.join(output_dir, \"analysis.md\")\n if not os.path.isfile(report):\n return \"analysis.md not found\"\n size = os.path.getsize(report)\n if size \u003c _MIN_REPORT_BYTES:\n return f\"analysis.md too small ({size} bytes)\"\n\n with open(report, encoding=\"utf-8\", errors=\"replace\") as f:\n content = f.read()\n if not content.strip():\n return \"analysis.md is empty\"\n\n non_ascii = sum(1 for c in content if ord(c) > 127)\n if len(content) > 0 and non_ascii / len(content) > _GARBLED_THRESHOLD:\n return (\n f\"analysis.md appears garbled ({non_ascii}/{len(content)} non-ASCII chars)\"\n )\n\n stripped = content.strip()\n if stripped.startswith(\"{\") and stripped.endswith(\"}\"):\n return \"analysis.md contains raw JSON instead of markdown\"\n\n return None\n\n\ndef _load_manifest(output_dir: str) -> dict | None:\n path = os.path.join(output_dir, \"category_data\", \"category_manifest.json\")\n if not os.path.isfile(path):\n return None\n with open(path) as f:\n return json.load(f)\n\n\ndef _rebase_path(manifest_path: str, output_dir: str) -> str:\n \"\"\"Rebase an absolute manifest path to the actual output_dir.\n\n The manifest stores absolute paths using the original output_dir.\n When evaluating against a different directory (e.g. analysis_output_ref),\n we extract the relative suffix and resolve it against the actual dir.\n \"\"\"\n for marker in (\n \"category_data/\",\n \"metadata/\",\n \"system_findings/\",\n \"category_findings/\",\n ):\n idx = manifest_path.find(marker)\n if idx != -1:\n return os.path.join(output_dir, manifest_path[idx:])\n return manifest_path\n\n\ndef _check_directories(output_dir: str) -> tuple[str, str]:\n required = [\"metadata\", \"category_data\", \"system_findings\", \"category_findings\"]\n missing = [d for d in required if not os.path.isdir(os.path.join(output_dir, d))]\n if missing:\n return \"FAIL\", f\"Missing directories: {', '.join(missing)}\"\n return \"PASS\", \"\"\n\n\ndef _check_metadata_files(output_dir: str) -> tuple[str, str]:\n manifest = _load_manifest(output_dir)\n if manifest is None:\n return \"FAIL\", \"category_manifest.json not found\"\n missing = []\n for cat in manifest.get(\"categories\", []):\n mf = cat.get(\"metadata_file\", \"\")\n if mf and not os.path.isfile(_rebase_path(mf, output_dir)):\n missing.append(os.path.basename(mf))\n if missing:\n return \"FAIL\", f\"Missing metadata files: {', '.join(missing)}\"\n return \"PASS\", \"\"\n\n\ndef _check_model_info(output_dir: str) -> tuple[str, str]:\n path = os.path.join(output_dir, \"metadata\", \"model_info.json\")\n if not os.path.isfile(path):\n return \"FAIL\", \"metadata/model_info.json not found\"\n try:\n with open(path) as f:\n data = json.load(f)\n except json.JSONDecodeError as e:\n return \"FAIL\", f\"Invalid JSON: {e}\"\n missing = _REQUIRED_MODEL_KEYS - data.keys()\n if missing:\n return \"FAIL\", f\"Missing keys: {', '.join(missing)}\"\n empty = [k for k in _REQUIRED_MODEL_KEYS if not str(data[k]).strip()]\n if empty:\n return \"FAIL\", f\"Empty values: {', '.join(empty)}\"\n return \"PASS\", \"\"\n\n\ndef _check_unified_perf_report(\n output_dir: str, comparison_scope: str = \"standalone\"\n) -> tuple[str, str]:\n if comparison_scope == \"comparative\":\n path = os.path.join(\n output_dir, \"perf_report_trace1_csvs\", \"unified_perf_summary.csv\"\n )\n label = \"perf_report_trace1_csvs/unified_perf_summary.csv\"\n else:\n path = os.path.join(output_dir, \"perf_report_csvs\", \"unified_perf_summary.csv\")\n label = \"perf_report_csvs/unified_perf_summary.csv\"\n if not os.path.isfile(path):\n return \"FAIL\", f\"{label} not found\"\n return \"PASS\", \"\"\n\n\ndef _check_tree_data_files(output_dir: str) -> tuple[str, str]:\n manifest = _load_manifest(output_dir)\n if manifest is None:\n return \"FAIL\", \"category_manifest.json not found\"\n missing = []\n for cat in manifest.get(\"categories\", []):\n tf = cat.get(\"tree_data_file\")\n if tf and not os.path.isfile(_rebase_path(tf, output_dir)):\n missing.append(os.path.basename(tf))\n if missing:\n return \"FAIL\", f\"Missing tree data files: {', '.join(missing)}\"\n return \"PASS\", \"\"\n\n\ndef _check_findings_exist(output_dir: str) -> tuple[str, str]:\n manifest = _load_manifest(output_dir)\n if manifest is None:\n return \"FAIL\", \"category_manifest.json not found\"\n gpu_util = manifest.get(\"gpu_utilization\", {})\n idle_pct = gpu_util.get(\"idle_time_percent\", 0)\n missing = []\n for cat in manifest.get(\"categories\", []):\n name = cat[\"name\"]\n if name == \"cpu_idle\" and idle_pct \u003c= 15:\n continue\n tier = cat.get(\"tier\", \"compute_kernel\")\n subdir = \"system_findings\" if tier == \"system\" else \"category_findings\"\n path = os.path.join(output_dir, subdir, f\"{name}_findings.md\")\n if not os.path.isfile(path):\n missing.append(f\"{subdir}/{name}_findings.md\")\n if missing:\n return \"FAIL\", f\"Parallel subagent failed to invoke for: {', '.join(missing)}\"\n return \"PASS\", \"\"\n\n\ndef _check_findings_placement(output_dir: str) -> tuple[str, str]:\n manifest = _load_manifest(output_dir)\n if manifest is None:\n return \"FAIL\", \"category_manifest.json not found\"\n gpu_util = manifest.get(\"gpu_utilization\", {})\n idle_pct = gpu_util.get(\"idle_time_percent\", 0)\n misplaced = []\n for cat in manifest.get(\"categories\", []):\n name = cat[\"name\"]\n if name == \"cpu_idle\" and idle_pct \u003c= 15:\n continue\n tier = cat.get(\"tier\", \"compute_kernel\")\n correct_dir = \"system_findings\" if tier == \"system\" else \"category_findings\"\n wrong_dir = \"category_findings\" if tier == \"system\" else \"system_findings\"\n correct = os.path.join(output_dir, correct_dir, f\"{name}_findings.md\")\n wrong = os.path.join(output_dir, wrong_dir, f\"{name}_findings.md\")\n if not os.path.isfile(correct) and os.path.isfile(wrong):\n misplaced.append(f\"{name} in {wrong_dir}/ instead of {correct_dir}/\")\n if misplaced:\n return \"FAIL\", f\"Misplaced findings: {'; '.join(misplaced)}\"\n return \"PASS\", \"\"\n\n\ndef _check_plot(output_dir: str) -> tuple[str, str]:\n plot_path = os.path.join(output_dir, \"perf_improvement.png\")\n plot_data_path = os.path.join(output_dir, \"priority_data.json\")\n\n if os.path.isfile(plot_path):\n return \"PASS\", \"\"\n\n if os.path.isfile(plot_data_path):\n with open(plot_data_path) as f:\n plot_data = json.load(f)\n recs = plot_data.get(\"recommendations\", [])\n if not recs:\n return \"PASS\", \"No kernel tuning recommendations — plot correctly skipped\"\n\n cat_data_dir = os.path.join(output_dir, \"category_data\")\n if os.path.isdir(cat_data_dir):\n total_estimates = 0\n for f in os.listdir(cat_data_dir):\n if f.endswith(\"_metrics.json\"):\n with open(os.path.join(cat_data_dir, f)) as fh:\n metrics = json.load(fh)\n total_estimates += len(metrics.get(\"impact_estimates\", []))\n if total_estimates == 0:\n return (\n \"PASS\",\n \"No kernel tuning recommendations in any metrics — plot correctly skipped\",\n )\n\n return \"FAIL\", \"perf_improvement.png not found\"\n\n\n# Each entry: (summary, func, root_cause, recommended_fix, mode_aware)\n# mode_aware=True → called as func(output_dir, comparison_scope)\n# mode_aware=False → called as func(output_dir)\nEVAL_REGISTRY = [\n (\n \"Directory structure created\",\n _check_directories,\n \"pipeline\",\n \"Re-run analysis pipeline from Step 1\",\n False,\n ),\n (\n \"Metadata files exist on disk\",\n _check_metadata_files,\n \"pipeline\",\n \"Re-run orchestrator_prepare.py to regenerate metadata\",\n False,\n ),\n (\n \"Model info JSON exists and valid\",\n _check_model_info,\n \"pipeline\",\n \"Check model-identification subagent; add fallback skeleton in prepare step\",\n False,\n ),\n (\n \"Unified perf. report exists\",\n _check_unified_perf_report,\n \"pipeline\",\n \"Re-run TraceLens_generate_perf_report_from_config (Step 1)\",\n True,\n ),\n (\n \"Tree data files exist on disk\",\n _check_tree_data_files,\n \"pipeline\",\n \"Re-run orchestrator_prepare.py to regenerate tree data\",\n False,\n ),\n (\n \"Categorical findings .md files exist\",\n _check_findings_exist,\n \"pipeline\",\n \"Retry failed subagent(s) for missing findings files\",\n False,\n ),\n (\n \"All findings exist\",\n _check_findings_placement,\n \"pipeline\",\n \"Fix tier assignment in orchestrator_prepare.py\",\n False,\n ),\n (\n \"Plot generated on disk\",\n _check_plot,\n \"pipeline\",\n \"Check priority_data.json; add fallback empty plot_data.json\",\n False,\n ),\n]\n\n\n# ---------------------------------------------------------------------------\n# Helpers for per-item deterministic checks (evals 9-14, excl. 12 = LLM)\n# ---------------------------------------------------------------------------\n\n\ndef _make_row(index, summary, result, details, root_cause, fix, category=\"Workflow\"):\n return {\n \"index\": index,\n \"category\": category,\n \"issue_summary\": summary,\n \"result\": result,\n \"details\": details,\n \"root_cause\": root_cause,\n \"recommended_fix\": fix,\n }\n\n\ndef _read_report(output_dir):\n path = os.path.join(output_dir, \"analysis.md\")\n if not os.path.isfile(path):\n return None\n with open(path, encoding=\"utf-8\", errors=\"replace\") as f:\n return f.read()\n\n\ndef _extract_section(content, header):\n \"\"\"Extract text between a ## header and the next ## header or end of file.\"\"\"\n escaped = re.escape(header)\n m = re.search(\n rf\"^{escaped}[^\\n]*\\n(.*?)(?=^## |\\Z)\",\n content,\n re.MULTILINE | re.DOTALL,\n )\n return m.group(1).strip() if m else None\n\n\ndef _find_table_row(section_text, label_synonyms):\n \"\"\"Find a markdown table row matching any synonym. Returns value cell or None.\"\"\"\n for line in section_text.split(\"\\n\"):\n stripped = line.strip()\n if not stripped.startswith(\"|\"):\n continue\n cells = [c.strip() for c in stripped.strip(\"|\").split(\"|\")]\n if len(cells) \u003c 2:\n continue\n if all(set(c) \u003c= {\"-\", \":\", \" \"} for c in cells):\n continue\n label = cells[0]\n for synonym in label_synonyms:\n if synonym.lower() in label.lower():\n return cells[1] if len(cells) > 1 else \"\"\n return None\n\n\ndef _find_table_row_all_cells(section_text, label_synonyms):\n \"\"\"Find a markdown table row matching any synonym. Returns all cells or None.\"\"\"\n for line in section_text.split(\"\\n\"):\n stripped = line.strip()\n if not stripped.startswith(\"|\"):\n continue\n cells = [c.strip() for c in stripped.strip(\"|\").split(\"|\")]\n if len(cells) \u003c 2:\n continue\n if all(set(c) \u003c= {\"-\", \":\", \" \"} for c in cells):\n continue\n label = cells[0]\n for synonym in label_synonyms:\n if synonym.lower() in label.lower():\n return cells # [label, val1, val2, ...]\n return None\n\n\ndef _extract_percent(value_str):\n \"\"\"Extract a percentage value from a string like '82.2%'.\"\"\"\n m = re.search(r\"([\\d.]+)\\s*%\", value_str)\n return float(m.group(1)) if m else None\n\n\ndef _extract_numeric(value_str):\n \"\"\"Extract first numeric value from a string (handles %, ms, ±, sign prefix).\"\"\"\n # Strip commas used as thousands separators\n s = value_str.replace(\",\", \"\")\n # Match optional leading sign then digits\n m = re.search(r\"([-+]?\\s*[\\d.]+)\", s)\n if m:\n return float(m.group(1).replace(\" \", \"\"))\n return None\n\n\ndef _load_gpu_timeline(output_dir, comparison_scope=\"standalone\", trace_num=1):\n \"\"\"Load gpu_timeline.csv as {type_name: percent}.\"\"\"\n if comparison_scope == \"standalone\":\n path = os.path.join(output_dir, \"perf_report_csvs\", \"gpu_timeline.csv\")\n else:\n path = os.path.join(\n output_dir, f\"perf_report_trace{trace_num}_csvs\", \"gpu_timeline.csv\"\n )\n if not os.path.isfile(path):\n return None\n data = {}\n with open(path) as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n data[row[\"type\"]] = float(row[\"percent\"])\n except (KeyError, ValueError):\n continue\n return data\n\n\ndef _extract_p_items(section_text):\n \"\"\"Extract P-items from a section. Returns [(p_number, text_block), ...].\"\"\"\n p_pattern = re.compile(r\"^### .+P(\\d+):\", re.MULTILINE)\n matches = list(p_pattern.finditer(section_text))\n items = []\n for i, m in enumerate(matches):\n pnum = int(m.group(1))\n start = m.start()\n end = matches[i + 1].start() if i + 1 \u003c len(matches) else len(section_text)\n items.append((pnum, section_text[start:end]))\n return items\n\n\ndef _check_p_item_fields(p_text, is_compute):\n \"\"\"Check P-item for required bold fields. Returns list of missing field names.\"\"\"\n missing = []\n if not re.search(r\"\\*\\*Insight\\*\\*\", p_text) and not re.search(\n r\"\\*\\*Issue\\*\\*\", p_text\n ):\n missing.append(\"**Insight** or **Issue**\")\n if not re.search(r\"\\*\\*Action\\*\\*\", p_text):\n missing.append(\"**Action**\")\n if is_compute and not re.search(r\"\\*\\*Impact\\*\\*\", p_text):\n missing.append(\"**Impact**\")\n return missing\n\n\n# ---------------------------------------------------------------------------\n# Per-item check functions (evals 9, 10, 11, 13, 14)\n# ---------------------------------------------------------------------------\n\n_REPORT_HEADERS = {\n \"executive_summary\": \"Executive Summary\",\n \"compute\": \"Compute Kernel Optimizations\",\n \"kernel_fusion\": \"Kernel Fusion Opportunities (Experimental)\",\n \"system\": \"System-Level Optimizations\",\n \"detailed\": \"Detailed Analysis\",\n \"appendix\": \"Appendix\",\n}\n\n\ndef _check_report_template(output_dir, comparison_scope=\"standalone\"):\n \"\"\"Eval 9 — per-header check for required section headers + table presence.\"\"\"\n content = _read_report(output_dir)\n rows = []\n for key, header_text in _REPORT_HEADERS.items():\n found = bool(re.search(rf\"^## {re.escape(header_text)}\", content, re.MULTILINE))\n rows.append(\n _make_row(\n f\"workflow_eval_9_{key}\",\n f\"Report header: {header_text}\",\n \"PASS\" if found else \"FAIL\",\n \"\" if found else f\"Missing '## {header_text}' header\",\n \"template\" if not found else \"\",\n \"Add missing section header to report\" if not found else \"\",\n )\n )\n\n exec_section = _extract_section(content, \"## Executive Summary\")\n has_table = bool(re.search(r\"\\|.*\\|\", exec_section)) if exec_section else False\n rows.append(\n _make_row(\n \"workflow_eval_9_metrics_table\",\n \"Metrics table in Executive Summary\",\n \"PASS\" if has_table else \"FAIL\",\n \"\" if has_table else \"No markdown table in Executive Summary\",\n \"template\" if not has_table else \"\",\n \"Add metrics table to Executive Summary\" if not has_table else \"\",\n )\n )\n return rows\n\n\n_EXEC_SUMMARY_CHECKS_STANDALONE = [\n (\"total_time\", [\"Total Compute Time\", \"Total Time\"], None, None),\n (\"compute_pct\", [\"Computation\", \"Compute %\", \"Compute\"], \"computation_time\", 1.0),\n (\"idle_pct\", [\"Idle Time\", \"Idle %\", \"Idle\"], \"idle_time\", 1.0),\n (\"comm_pct\", [\"Exposed Communication\", \"Exposed Communication %\"], None, None),\n (\"bottleneck\", [\"Top Bottleneck Category\", \"Top Bottleneck\"], None, None),\n]\n\n# comparative: (key, labels, csv_type_t1, tol_t1, csv_type_t2, tol_t2)\n# None csv_type means no CSV cross-check for that trace\n_EXEC_SUMMARY_CHECKS_COMPARATIVE = [\n (\"total_time\", [\"Total Compute Time\", \"Total Time\"], None, None, None, None),\n (\n \"compute_pct\",\n [\"Computation\", \"Compute %\", \"Compute\"],\n \"computation_time\",\n 1.0,\n \"computation_time\",\n 1.0,\n ),\n (\"idle_pct\", [\"Idle Time\", \"Idle %\", \"Idle\"], \"idle_time\", 1.0, \"idle_time\", 1.0),\n (\n \"comm_pct\",\n [\"Exposed Communication\", \"Exposed Communication %\"],\n None,\n None,\n None,\n None,\n ),\n (\n \"bottleneck\",\n [\"Top Bottleneck Category\", \"Top Bottleneck\"],\n None,\n None,\n None,\n None,\n ),\n]\n\n\ndef _check_exec_summary(output_dir, comparison_scope=\"standalone\"):\n \"\"\"Eval 10 — per-row check for exec summary metrics + CSV cross-validation.\"\"\"\n content = _read_report(output_dir)\n exec_section = _extract_section(content, \"## Executive Summary\")\n if not exec_section:\n return [\n _make_row(\n \"workflow_eval_10\",\n \"Executive Summary has metrics table\",\n \"FAIL\",\n \"Executive Summary section not found\",\n \"template\",\n \"Add Executive Summary section\",\n )\n ]\n\n if comparison_scope == \"standalone\":\n return _check_exec_summary_standalone(output_dir, exec_section)\n return _check_exec_summary_comparative(output_dir, exec_section)\n\n\ndef _check_exec_summary_standalone(output_dir, exec_section):\n csv_data = _load_gpu_timeline(output_dir, comparison_scope=\"standalone\")\n rows = []\n for key, labels, csv_type, tolerance in _EXEC_SUMMARY_CHECKS_STANDALONE:\n value_str = _find_table_row(exec_section, labels)\n if value_str is None:\n rows.append(\n _make_row(\n f\"workflow_eval_10_{key}\",\n f\"Exec Summary row: {labels[0]}\",\n \"FAIL\",\n f\"Row not found (looked for: {', '.join(labels)})\",\n \"template\",\n \"Add missing row to Executive Summary table\",\n )\n )\n continue\n\n result, detail = \"PASS\", \"\"\n if csv_type and csv_data and tolerance:\n csv_val = csv_data.get(csv_type)\n if csv_val is not None:\n report_val = _extract_percent(value_str)\n if report_val is not None:\n diff = abs(report_val - csv_val)\n if diff > tolerance:\n result = \"FAIL\"\n detail = (\n f\"Value mismatch: report={report_val:.1f}%, \"\n f\"csv={csv_val:.2f}%, diff={diff:.2f}% \"\n f\"(tolerance={tolerance}%)\"\n )\n else:\n detail = (\n f\"Matches CSV (report={report_val:.1f}%, \"\n f\"csv={csv_val:.2f}%)\"\n )\n\n rows.append(\n _make_row(\n f\"workflow_eval_10_{key}\",\n f\"Exec Summary row: {labels[0]}\",\n result,\n detail,\n \"template\" if result == \"FAIL\" else \"\",\n \"Fix value in Executive Summary table\" if result == \"FAIL\" else \"\",\n )\n )\n return rows\n\n\ndef _check_exec_summary_comparative(output_dir, exec_section):\n \"\"\"Comparative exec summary: 4-column table (Metric | T1 | T2 | Difference).\"\"\"\n csv_t1 = _load_gpu_timeline(output_dir, comparison_scope=\"comparative\", trace_num=1)\n csv_t2 = _load_gpu_timeline(output_dir, comparison_scope=\"comparative\", trace_num=2)\n rows = []\n\n for (\n key,\n labels,\n csv_type_t1,\n tol_t1,\n csv_type_t2,\n tol_t2,\n ) in _EXEC_SUMMARY_CHECKS_COMPARATIVE:\n cells = _find_table_row_all_cells(exec_section, labels)\n if cells is None:\n rows.append(\n _make_row(\n f\"workflow_eval_10_{key}\",\n f\"Exec Summary row: {labels[0]}\",\n \"FAIL\",\n f\"Row not found (looked for: {', '.join(labels)})\",\n \"template\",\n \"Add missing row to Executive Summary table\",\n )\n )\n continue\n\n # Expect [label, t1_val, t2_val, diff_val]\n t1_str = cells[1] if len(cells) > 1 else \"\"\n t2_str = cells[2] if len(cells) > 2 else \"\"\n\n result, detail_parts = \"PASS\", []\n\n if csv_type_t1 and not t1_str:\n result = \"FAIL\"\n detail_parts.append(\"T1 cell missing or empty\")\n if csv_type_t2 and not t2_str:\n result = \"FAIL\"\n detail_parts.append(\"T2 cell missing or empty\")\n\n # Cross-check T1\n if csv_type_t1 and csv_t1 and tol_t1:\n csv_val = csv_t1.get(csv_type_t1)\n if csv_val is not None:\n report_val = _extract_percent(t1_str)\n if report_val is not None:\n diff = abs(report_val - csv_val)\n if diff > tol_t1:\n result = \"FAIL\"\n detail_parts.append(\n f\"T1 mismatch: report={report_val:.1f}%, csv={csv_val:.2f}%, diff={diff:.2f}%\"\n )\n else:\n detail_parts.append(\n f\"T1 OK ({report_val:.1f}% vs csv {csv_val:.2f}%)\"\n )\n\n # Cross-check T2\n if csv_type_t2 and csv_t2 and tol_t2:\n csv_val = csv_t2.get(csv_type_t2)\n if csv_val is not None:\n report_val = _extract_percent(t2_str)\n if report_val is not None:\n diff = abs(report_val - csv_val)\n if diff > tol_t2:\n result = \"FAIL\"\n detail_parts.append(\n f\"T2 mismatch: report={report_val:.1f}%, csv={csv_val:.2f}%, diff={diff:.2f}%\"\n )\n else:\n detail_parts.append(\n f\"T2 OK ({report_val:.1f}% vs csv {csv_val:.2f}%)\"\n )\n\n # Check Difference column exists\n if len(cells) \u003c 4:\n result = \"FAIL\"\n detail_parts.append(\"Difference column missing (expected 4-column table)\")\n else:\n # Arithmetic check: |Difference| ≈ |T1 - T2| within 1%\n diff_str = cells[3]\n t1_val = _extract_numeric(t1_str)\n t2_val = _extract_numeric(t2_str)\n diff_val = _extract_numeric(diff_str)\n if t1_val is not None and t2_val is not None and diff_val is not None:\n expected_magnitude = abs(t1_val - t2_val)\n denom = max(abs(t1_val), abs(t2_val), 1.0)\n rel_err = abs(abs(diff_val) - expected_magnitude) / denom\n if rel_err > 0.01:\n result = \"FAIL\"\n detail_parts.append(\n f\"Difference arithmetic error: reported={diff_str}, \"\n f\"expected |diff|≈{expected_magnitude:.2f}, rel_err={rel_err:.3f}\"\n )\n\n rows.append(\n _make_row(\n f\"workflow_eval_10_{key}\",\n f\"Exec Summary row: {labels[0]}\",\n result,\n \"; \".join(detail_parts),\n \"template\" if result == \"FAIL\" else \"\",\n \"Fix value in Executive Summary table\" if result == \"FAIL\" else \"\",\n )\n )\n return rows\n\n\ndef _check_issue_template(output_dir, comparison_scope=\"standalone\"):\n \"\"\"Eval 11 — per-P-item check for correct bold fields in each priority item.\"\"\"\n content = _read_report(output_dir)\n compute_section = _extract_section(content, \"## Compute Kernel Optimizations\")\n system_section = _extract_section(content, \"## System-Level Optimizations\")\n\n rows = []\n if compute_section:\n for pnum, p_text in _extract_p_items(compute_section):\n missing = _check_p_item_fields(p_text, is_compute=True)\n result = \"PASS\" if not missing else \"FAIL\"\n rows.append(\n _make_row(\n f\"workflow_eval_11_compute_P{pnum}\",\n f\"Compute P{pnum} template\",\n result,\n f\"Missing: {', '.join(missing)}\" if missing else \"\",\n \"template\" if result == \"FAIL\" else \"\",\n \"Add missing fields to P-item\" if result == \"FAIL\" else \"\",\n )\n )\n\n if system_section:\n for pnum, p_text in _extract_p_items(system_section):\n missing = _check_p_item_fields(p_text, is_compute=False)\n result = \"PASS\" if not missing else \"FAIL\"\n rows.append(\n _make_row(\n f\"workflow_eval_11_system_P{pnum}\",\n f\"System P{pnum} template\",\n result,\n f\"Missing: {', '.join(missing)}\" if missing else \"\",\n \"template\" if result == \"FAIL\" else \"\",\n \"Add missing fields to P-item\" if result == \"FAIL\" else \"\",\n )\n )\n\n if not rows:\n all_sections = (compute_section or \"\") + (system_section or \"\")\n intentionally_empty = (\n \"No compute kernel optimization opportunities identified\" in all_sections\n or \"No system-level bottlenecks detected\" in all_sections\n )\n rows.append(\n _make_row(\n \"workflow_eval_11\",\n \"Issue Template rendering\",\n \"PASS\" if intentionally_empty else \"FAIL\",\n \"\" if intentionally_empty else \"No P-items found in report\",\n \"\" if intentionally_empty else \"template\",\n \"\" if intentionally_empty else \"Ensure report contains priority items\",\n )\n )\n return rows\n\n\n_MODEL_INFO_FIELDS = [\"model\", \"architecture\", \"scale\", \"precision\"]\n\n\ndef _check_model_id(output_dir, comparison_scope=None):\n \"\"\"Eval 13 — per-field check for model_info.json values in Appendix.\"\"\"\n model_info_path = os.path.join(output_dir, \"metadata\", \"model_info.json\")\n report_path = os.path.join(output_dir, \"analysis.md\")\n\n if not os.path.isfile(model_info_path):\n return [\n _make_row(\n \"workflow_eval_13\",\n \"Model identification in report\",\n \"FAIL\",\n \"metadata/model_info.json not found\",\n \"pipeline\",\n \"Run model identification subagent\",\n )\n ]\n if not os.path.isfile(report_path):\n return [\n _make_row(\n \"workflow_eval_13\",\n \"Model identification in report\",\n \"FAIL\",\n \"analysis.md not found\",\n \"pipeline\",\n \"Re-run report generation\",\n )\n ]\n\n try:\n with open(model_info_path) as f:\n model_info = json.load(f)\n except (json.JSONDecodeError, OSError):\n return [\n _make_row(\n \"workflow_eval_13\",\n \"Model identification in report\",\n \"FAIL\",\n \"Invalid model_info.json\",\n \"pipeline\",\n \"Fix model identification subagent\",\n )\n ]\n\n with open(report_path, encoding=\"utf-8\", errors=\"replace\") as f:\n content = f.read()\n\n appendix = _extract_section(content, \"## Appendix\")\n if not appendix:\n return [\n _make_row(\n \"workflow_eval_13\",\n \"Model identification in report\",\n \"FAIL\",\n \"Appendix section not found in report\",\n \"template\",\n \"Add Appendix section to report\",\n )\n ]\n\n appendix_lower = appendix.lower()\n rows = []\n for field in _MODEL_INFO_FIELDS:\n value = str(model_info.get(field, \"\")).strip()\n if not value or value.lower() == \"cannot be inferred from trace\":\n rows.append(\n _make_row(\n f\"workflow_eval_13_{field}\",\n f\"Model ID: {field}\",\n \"PASS\",\n f\"Field not determined ('{value}') — skipped\",\n \"\",\n \"\",\n )\n )\n continue\n\n found = value.lower() in appendix_lower\n rows.append(\n _make_row(\n f\"workflow_eval_13_{field}\",\n f\"Model ID: {field}\",\n \"PASS\" if found else \"FAIL\",\n \"\" if found else f\"'{value}' not found in Appendix\",\n \"template\" if not found else \"\",\n \"Add model info field to Appendix\" if not found else \"\",\n )\n )\n return rows\n\n\n# ---------------------------------------------------------------------------\n# Marker Identification checks (deterministic, structural)\n# ---------------------------------------------------------------------------\n\n\ndef _marker_attrs_from_inner(inner):\n \"\"\"Parse key=value pairs from a marker's inner content.\"\"\"\n return dict(_MV_ATTR_RE.findall(inner))\n\n\ndef _count_table_data_rows(text):\n \"\"\"Count markdown table data rows in text\"\"\"\n n_rows = sum(\n 1\n for ln in text.splitlines()\n if ln.lstrip().startswith(\"|\") and not re.match(r\"^\\s*\\|[\\s\\-:|]+\\|\\s*$\", ln)\n )\n return max(0, n_rows - 1)\n\n\ndef _make_marker_row(index, summary, result, details, root_cause=\"\", fix=\"\"):\n return _make_row(\n index,\n summary,\n result,\n details,\n root_cause,\n fix,\n category=\"Marker Identification\",\n )\n\n\ndef _marker_not_found(index, summary):\n \"\"\"Single FAIL row for a marker eval whose report file is missing.\"\"\"\n return [\n _make_marker_row(\n index,\n summary,\n \"FAIL\",\n \"analysis.md not found\",\n \"pipeline\",\n \"Re-run report generation\",\n )\n ]\n\n\ndef _check_marker_top_ops(output_dir, comparison_scope=None):\n \"\"\"Marker eval 1: top_ops wrapper + inline top-ops-row markers.\"\"\"\n content = _read_report(output_dir)\n if content is None:\n return _marker_not_found(\n \"marker_eval_1\", \"Top Operations markers (kind=top_ops)\"\n )\n\n errors = []\n top_ops_begin = None\n for m in _MV_BEGIN_RE.finditer(content):\n inner = m.group(1)\n km = _MV_KIND_ATTR_RE.search(inner)\n if km and km.group(1) == \"top_ops\":\n top_ops_begin = m\n break\n\n if top_ops_begin is None:\n errors.append(\"No \u003c!-- impact-begin kind=top_ops --> marker found\")\n else:\n end_after = _MV_END_RE.search(content, top_ops_begin.end())\n if end_after is None:\n errors.append(\"kind=top_ops marker has no matching \u003c!-- impact-end -->\")\n else:\n block = content[top_ops_begin.end() : end_after.start()]\n row_markers = list(_TOP_OPS_ROW_RE.finditer(block))\n n_data_rows = _count_table_data_rows(block)\n if not row_markers:\n if n_data_rows > 0:\n errors.append(\n \"No \u003c!-- top-ops-row ... --> markers found inside \"\n \"top_ops block\"\n )\n else:\n for i, rm in enumerate(row_markers, start=1):\n inner = rm.group(1)\n attrs = _marker_attrs_from_inner(inner)\n missing_attrs = []\n if \"low\" not in attrs:\n missing_attrs.append(\"low\")\n if \"high\" not in attrs:\n missing_attrs.append(\"high\")\n if missing_attrs:\n errors.append(\n f\"top-ops-row #{i} missing attributes: \"\n f\"{', '.join(missing_attrs)}\"\n )\n\n result = \"PASS\" if not errors else \"FAIL\"\n details = \"; \".join(errors) if errors else \"\"\n return [\n _make_marker_row(\n \"marker_eval_1\",\n \"Top Operations markers (kind=top_ops)\",\n result,\n details,\n \"template\" if result == \"FAIL\" else \"\",\n \"Add or fix top_ops markers in analysis.md\" if result == \"FAIL\" else \"\",\n )\n ]\n\n\ndef _check_marker_p_items(output_dir, comparison_scope=None):\n \"\"\"Marker eval 2: kind=p_item markers for each compute P-item.\"\"\"\n content = _read_report(output_dir)\n if content is None:\n return _marker_not_found(\"marker_eval_2\", \"P-item markers (kind=p_item)\")\n\n compute_section = _extract_section(content, \"## Compute Kernel Optimizations\")\n if compute_section is None:\n return [\n _make_marker_row(\n \"marker_eval_2\",\n \"P-item markers (kind=p_item)\",\n \"FAIL\",\n \"Compute Kernel Optimizations section not found\",\n \"template\",\n \"Ensure report contains Compute Kernel Optimizations section\",\n )\n ]\n\n p_items = _extract_p_items(compute_section)\n if not p_items:\n intentionally_empty = (\n \"No compute kernel optimization opportunities identified\" in compute_section\n )\n return [\n _make_marker_row(\n \"marker_eval_2\",\n \"P-item markers (kind=p_item)\",\n \"PASS\" if intentionally_empty else \"FAIL\",\n (\n \"\"\n if intentionally_empty\n else \"No P-items found in Compute Kernel Optimizations\"\n ),\n \"\" if intentionally_empty else \"template\",\n \"\" if intentionally_empty else \"Ensure report contains P-items\",\n )\n ]\n\n rows = []\n for pnum, p_text in p_items:\n errors = []\n p_item_markers = []\n for m in _MV_BEGIN_RE.finditer(p_text):\n inner = m.group(1)\n km = _MV_KIND_ATTR_RE.search(inner)\n if km and km.group(1) == \"p_item\":\n p_item_markers.append(inner)\n\n if not p_item_markers:\n errors.append(\"No \u003c!-- impact-begin kind=p_item ... --> marker found\")\n else:\n for inner in p_item_markers:\n attrs = _marker_attrs_from_inner(inner)\n required = (\"category\", \"low\", \"mid\", \"high\")\n missing = [a for a in required if a not in attrs]\n if missing:\n errors.append(\n f\"kind=p_item marker missing attributes: \"\n f\"{', '.join(missing)}\"\n )\n\n end_count = len(_MV_END_RE.findall(p_text))\n begin_count = len(_MV_BEGIN_RE.findall(p_text))\n if begin_count > end_count:\n errors.append(\"Unpaired impact-begin (missing impact-end)\")\n\n result = \"PASS\" if not errors else \"FAIL\"\n details = \"; \".join(errors) if errors else \"\"\n rows.append(\n _make_marker_row(\n f\"marker_eval_2_P{pnum}\",\n f\"Compute P{pnum} impact marker (kind=p_item)\",\n result,\n details,\n \"template\" if result == \"FAIL\" else \"\",\n f\"Fix kind=p_item marker for P{pnum}\" if result == \"FAIL\" else \"\",\n )\n )\n return rows\n\n\ndef _p_blocks(tier, subsection, matches):\n \"\"\"(tier, P number, block text) for each P-item header in a Detailed Analysis subsection.\"\"\"\n for i, match in enumerate(matches):\n end = matches[i + 1].start() if i + 1 \u003c len(matches) else len(subsection)\n yield tier, int(match.group(1)), subsection[match.start() : end]\n\n\ndef _check_marker_detail_estimates(output_dir, comparison_scope=None):\n \"\"\"Marker eval 3: kind=detail_estimate markers in Detailed Analysis.\"\"\"\n content = _read_report(output_dir)\n if content is None:\n return _marker_not_found(\n \"marker_eval_3\", \"Detail estimate markers (kind=detail_estimate)\"\n )\n\n detailed_section = _extract_section(content, \"## Detailed Analysis\")\n if detailed_section is None:\n return [\n _make_marker_row(\n \"marker_eval_3\",\n \"Detail estimate markers (kind=detail_estimate)\",\n \"FAIL\",\n \"Detailed Analysis section not found\",\n \"template\",\n \"Ensure report contains Detailed Analysis section\",\n )\n ]\n\n m = re.search(\n r\"^### Compute Kernel Insights[^\\n]*\\n(.*?)(?=^### (?![^\\n]*P\\d)|^## |\\Z)\",\n detailed_section,\n re.MULTILINE | re.DOTALL,\n )\n compute_subsection = m.group(0) if m else detailed_section\n\n matches = list(_DETAIL_P_HEADER_RE.finditer(compute_subsection))\n if not matches:\n matches = list(_DETAIL_P_HEADER_FALLBACK_RE.finditer(compute_subsection))\n\n blocks = list(_p_blocks(\"compute\", compute_subsection, matches))\n fusion_subsection = _extract_detailed_analysis_subsection(\n content, \"### Kernel Fusion Insights\"\n )\n if fusion_subsection:\n fusion_matches = list(_DETAIL_P_HEADER_RE.finditer(fusion_subsection))\n blocks += _p_blocks(\"fusion\", fusion_subsection, fusion_matches)\n\n if not blocks:\n intentionally_empty = (\n \"No compute kernel optimization opportunities identified\"\n in compute_subsection\n )\n return [\n _make_marker_row(\n \"marker_eval_3\",\n \"Detail estimate markers (kind=detail_estimate)\",\n \"PASS\" if intentionally_empty else \"FAIL\",\n (\n \"\"\n if intentionally_empty\n else \"No P-item headers found in Detailed Analysis\"\n ),\n \"\" if intentionally_empty else \"template\",\n (\n \"\"\n if intentionally_empty\n else \"Ensure Detailed Analysis contains P-item sections\"\n ),\n )\n ]\n\n rows = []\n for tier, pnum, block in blocks:\n pname = f\"P{pnum}\" if tier == \"compute\" else f\"{tier} P{pnum}\"\n\n errors = []\n detail_markers = []\n for bm in _MV_BEGIN_RE.finditer(block):\n inner = bm.group(1)\n km = _MV_KIND_ATTR_RE.search(inner)\n if km and km.group(1) == \"detail_estimate\":\n detail_markers.append(inner)\n\n has_not_quantifiable = bool(_NOT_QUANTIFIABLE_RE.search(block))\n\n if not detail_markers and not has_not_quantifiable:\n errors.append(\n \"No \u003c!-- impact-begin kind=detail_estimate ... --> marker \"\n \"and no 'not quantifiable from trace data' sentinel found\"\n )\n elif detail_markers:\n for inner in detail_markers:\n attrs = _marker_attrs_from_inner(inner)\n missing = [a for a in (\"low\", \"mid\", \"high\") if a not in attrs]\n if missing:\n errors.append(\n f\"kind=detail_estimate marker missing attributes: \"\n f\"{', '.join(missing)}\"\n )\n\n total_begin = len(_MV_BEGIN_RE.findall(block))\n end_count = len(_MV_END_RE.findall(block))\n if total_begin > end_count:\n errors.append(\"Unpaired impact-begin (missing impact-end)\")\n\n result = \"PASS\" if not errors else \"FAIL\"\n details = \"; \".join(errors) if errors else \"\"\n rows.append(\n _make_marker_row(\n f\"marker_eval_3_{pname.replace(' ', '_')}\",\n f\"Detailed Analysis {pname} estimate marker (kind=detail_estimate)\",\n result,\n details,\n \"template\" if result == \"FAIL\" else \"\",\n (\n f\"Fix kind=detail_estimate marker for {pname} in Detailed Analysis\"\n if result == \"FAIL\"\n else \"\"\n ),\n )\n )\n return rows\n\n\ndef _extract_detailed_analysis_subsection(content, subsection_header):\n \"\"\"Extract a ### subsection from ## Detailed Analysis.\"\"\"\n da_match = re.search(\n r\"^## Detailed Analysis[^\\n]*\\n(.*?)(?=^## |\\Z)\",\n content,\n re.MULTILINE | re.DOTALL,\n )\n if not da_match:\n return None\n da_text = da_match.group(0)\n sub_start = da_text.find(subsection_header)\n if sub_start \u003c 0:\n return None\n sub_text = da_text[sub_start + len(subsection_header) :]\n next_h3 = re.search(r\"^### \", sub_text, re.MULTILINE)\n next_h2 = re.search(r\"^## \", sub_text, re.MULTILINE)\n ends = [pos.start() for pos in [next_h3, next_h2] if pos is not None]\n end = min(ends) if ends else len(sub_text)\n return sub_text[:end]\n\n\ndef _check_marker_reasoning_candidates(output_dir, comparison_scope=None):\n \"\"\"Marker eval 4: reasoning-candidate markers match P-item headings per tier.\"\"\"\n content = _read_report(output_dir)\n if content is None:\n return _marker_not_found(\"marker_eval_4\", \"Reasoning-candidate markers\")\n\n rows = []\n checks = [\n (\"compute\", \"### Compute Kernel Insights\"),\n (\"fusion\", \"### Kernel Fusion Insights\"),\n (\"system\", \"### System-Level Insights\"),\n ]\n for tier, header in checks:\n subsection = _extract_detailed_analysis_subsection(content, header)\n if subsection is None:\n continue\n n_headings = len(_DETAIL_P_HEADER_RE.findall(subsection))\n n_markers = sum(\n 1\n for m in _REASONING_CANDIDATE_RE.finditer(subsection)\n if m.group(1).lower() == tier\n )\n if n_headings == 0:\n continue\n result = \"PASS\" if n_markers == n_headings else \"FAIL\"\n detail = (\n \"\"\n if result == \"PASS\"\n else (\n f\"{n_headings} #### P\u003cN>: headings but \"\n f\"{n_markers} reasoning-candidate tier={tier} markers\"\n )\n )\n rows.append(\n _make_marker_row(\n f\"marker_eval_4_{tier}\",\n f\"reasoning-candidate markers ({tier})\",\n result,\n detail,\n \"template\" if result == \"FAIL\" else \"\",\n (\n f\"Add \u003c!-- reasoning-candidate tier={tier} rank=N --> before each \"\n f\"#### P\u003cN>: heading in {header}\"\n if result == \"FAIL\"\n else \"\"\n ),\n )\n )\n\n if not rows:\n rows.append(\n _make_marker_row(\n \"marker_eval_4\",\n \"Reasoning-candidate markers\",\n \"PASS\",\n \"No Detailed Analysis subsections with P-items found\",\n \"\",\n \"\",\n )\n )\n return rows\n\n\ndef _count_csv_impacts(inner):\n \"\"\"Count impacts= CSV items, tolerant of whitespace around commas.\"\"\"\n m = re.search(r\"impacts=(\\S.*?)(?:\\s+\\w+=|\\s*-->|\\s*$)\", inner)\n raw = m.group(1).strip() if m else \"\"\n return len([v for v in raw.split(\",\") if v.strip()]) if raw else 0\n\n\ndef _count_data_table_rows(block):\n \"\"\"Count body rows of the first **Data:** markdown table in a block.\"\"\"\n lines = block.splitlines()\n data_idx = next(\n (i for i, ln in enumerate(lines) if ln.strip() == \"**Data:**\"), None\n )\n if data_idx is None:\n return 0\n header_idx = next(\n (\n i\n for i in range(data_idx + 1, len(lines))\n if lines[i].lstrip().startswith(\"|\") and lines[i].rstrip().endswith(\"|\")\n ),\n None,\n )\n if header_idx is None or header_idx + 1 >= len(lines):\n return 0\n n_rows = 0\n for row in lines[header_idx + 2 :]:\n if not row.strip().startswith(\"|\"):\n break\n n_rows += 1\n return n_rows\n\n\ndef _check_marker_op_rows(output_dir, comparison_scope=None):\n \"\"\"Marker eval 5: kind=op_row per-row impact markers per compute candidate.\"\"\"\n content = _read_report(output_dir)\n if content is None:\n return _marker_not_found(\n \"marker_eval_5\", \"Per-row impact markers (kind=op_row)\"\n )\n\n candidates = list(_REASONING_CANDIDATE_RE.finditer(content))\n rows = []\n for i, cm in enumerate(candidates):\n if cm.group(1).lower() != \"compute\":\n continue\n start = cm.end()\n end = candidates[i + 1].start() if i + 1 \u003c len(candidates) else len(content)\n block = content[start:end]\n\n op_row_markers = [\n m.group(1)\n for m in _MV_BEGIN_RE.finditer(block)\n if (km := _MV_KIND_ATTR_RE.search(m.group(1))) and km.group(1) == \"op_row\"\n ]\n if not op_row_markers:\n continue\n\n rank = cm.group(2)\n errors = []\n n_rows = _count_data_table_rows(block)\n for inner in op_row_markers:\n attrs = _marker_attrs_from_inner(inner)\n missing = [a for a in (\"rank\", \"impacts\") if a not in attrs]\n if missing:\n errors.append(\n f\"kind=op_row marker missing attributes: {', '.join(missing)}\"\n )\n continue\n n_csv = _count_csv_impacts(inner)\n if n_csv != n_rows:\n errors.append(\n f\"kind=op_row has {n_csv} impacts but Data table has {n_rows} rows\"\n )\n\n if len(_MV_BEGIN_RE.findall(block)) > len(_MV_END_RE.findall(block)):\n errors.append(\"Unpaired impact-begin (missing impact-end)\")\n\n result = \"PASS\" if not errors else \"FAIL\"\n rows.append(\n _make_marker_row(\n f\"marker_eval_5_P{rank}\",\n f\"Compute P{rank} per-row impact marker (kind=op_row)\",\n result,\n \"; \".join(errors) if errors else \"\",\n \"template\" if result == \"FAIL\" else \"\",\n f\"Fix kind=op_row marker for P{rank}\" if result == \"FAIL\" else \"\",\n )\n )\n\n if not rows:\n rows.append(\n _make_marker_row(\n \"marker_eval_5\",\n \"Per-row impact markers (kind=op_row)\",\n \"PASS\",\n \"No compute candidates with op_row markers found\",\n \"\",\n \"\",\n )\n )\n return rows\n\n\ndef _check_marker_report_mode(output_dir, comparison_scope=None):\n \"\"\"Marker eval 6: exactly one top-of-report kind=report_mode marker.\n\n Gates the LLM path so agentic/comparative reports carry the report_mode\n marker parallel to the op_row eval. Expected mode = comparative when\n comparison_scope is comparative, else agentic.\n \"\"\"\n content = _read_report(output_dir)\n if content is None:\n return _marker_not_found(\n \"marker_eval_6\", \"Report mode marker (kind=report_mode)\"\n )\n\n modes = [\n _marker_attrs_from_inner(inner).get(\"mode\")\n for inner in _MV_REPORT_BEGIN_RE.findall(content)\n if (km := _MV_KIND_ATTR_RE.search(inner)) and km.group(1) == \"report_mode\"\n ]\n expected = \"comparative\" if comparison_scope == \"comparative\" else \"agentic\"\n errors = []\n if len(modes) != 1:\n errors.append(\n f\"expected exactly one kind=report_mode marker, found {len(modes)}\"\n )\n elif modes[0] != expected:\n errors.append(f\"report_mode mode={modes[0]!r}, expected {expected!r}\")\n\n result = \"PASS\" if not errors else \"FAIL\"\n return [\n _make_marker_row(\n \"marker_eval_6\",\n \"Report mode marker (kind=report_mode)\",\n result,\n \"; \".join(errors) if errors else \"\",\n \"template\" if result == \"FAIL\" else \"\",\n \"Fix top-of-report kind=report_mode marker\" if result == \"FAIL\" else \"\",\n )\n ]\n\n\n_MULTI_EVAL_CHECKS = [\n _check_report_template,\n _check_exec_summary,\n _check_issue_template,\n _check_model_id,\n _check_marker_top_ops,\n _check_marker_p_items,\n _check_marker_detail_estimates,\n _check_marker_reasoning_candidates,\n _check_marker_op_rows,\n _check_marker_report_mode,\n]\n\n_GATE_FAIL_NEW_EVALS = [\n (\"workflow_eval_9\", \"Report Template Rendering\"),\n (\"workflow_eval_10\", \"Executive Summary has metrics table\"),\n (\"workflow_eval_11\", \"Issue Template rendering\"),\n (\"workflow_eval_13\", \"Model identification in report\"),\n (\"marker_eval_1\", \"Top Operations markers (kind=top_ops)\"),\n (\"marker_eval_2\", \"P-item markers (kind=p_item)\"),\n (\"marker_eval_3\", \"Detail estimate markers (kind=detail_estimate)\"),\n (\"marker_eval_4\", \"Reasoning-candidate markers\"),\n (\"marker_eval_5\", \"Per-row impact markers (kind=op_row)\"),\n (\"marker_eval_6\", \"Report mode marker (kind=report_mode)\"),\n]\n\n\ndef run(\n output_dir: str, results_path: str, comparison_scope: str = \"standalone\"\n) -> list[dict]:\n rows = []\n\n gate_fail = _pre_check_gates(output_dir)\n if gate_fail is not None:\n for i, (summary, _func, rc, fix, _mode_aware) in enumerate(\n EVAL_REGISTRY, start=1\n ):\n rows.append(\n {\n \"index\": f\"workflow_eval_{i}\",\n \"category\": \"Workflow\",\n \"issue_summary\": summary,\n \"result\": \"FAIL\",\n \"details\": f\"Pre-check gate: {gate_fail}\",\n \"root_cause\": \"pipeline\",\n \"recommended_fix\": \"Fix pre-check gate failure before running evals\",\n }\n )\n for idx, summary in _GATE_FAIL_NEW_EVALS:\n rows.append(\n {\n \"index\": idx,\n \"category\": \"Workflow\",\n \"issue_summary\": summary,\n \"result\": \"FAIL\",\n \"details\": f\"Pre-check gate: {gate_fail}\",\n \"root_cause\": \"pipeline\",\n \"recommended_fix\": \"Fix pre-check gate failure before running evals\",\n }\n )\n with open(results_path, \"w\", newline=\"\") as f:\n writer = csv.DictWriter(f, fieldnames=CSV_COLUMNS)\n writer.writeheader()\n writer.writerows(rows)\n return rows\n\n for i, (summary, func, rc, fix, mode_aware) in enumerate(EVAL_REGISTRY, start=1):\n result, details = (\n func(output_dir, comparison_scope) if mode_aware else func(output_dir)\n )\n rows.append(\n {\n \"index\": f\"workflow_eval_{i}\",\n \"category\": \"Workflow\",\n \"issue_summary\": summary,\n \"result\": result,\n \"details\": details,\n \"root_cause\": rc if result == \"FAIL\" else \"\",\n \"recommended_fix\": fix if result == \"FAIL\" else \"\",\n }\n )\n\n for check_fn in _MULTI_EVAL_CHECKS:\n rows.extend(check_fn(output_dir, comparison_scope))\n\n with open(results_path, \"w\", newline=\"\") as f:\n writer = csv.DictWriter(f, fieldnames=CSV_COLUMNS)\n writer.writeheader()\n writer.writerows(rows)\n return rows\n\n\ndef main():\n parser = argparse.ArgumentParser(description=\"Pythonic workflow evals\")\n parser.add_argument(\"--output-dir\", required=True)\n parser.add_argument(\"--results\", required=True)\n parser.add_argument(\n \"--comparison-scope\",\n choices=[\"standalone\", \"comparative\"],\n default=\"standalone\",\n help=\"Analysis comparison scope (default: standalone)\",\n )\n args = parser.parse_args()\n\n rows = run(args.output_dir, args.results, args.comparison_scope)\n passed = sum(1 for r in rows if r[\"result\"] == \"PASS\")\n sys.exit(0 if passed == len(rows) else 1)\n\n\nif __name__ == \"__main__\":\n main()\n","content_type":"text/x-python; charset=utf-8","language":"python","size":54748,"content_sha256":"837b9378e9e6bec52226769a03001d6a3b1e7e0277765a271214941bb17192a5"},{"filename":"evals/evals.json","content":"{\n \"evaluations\": [\n {\n \"id\": \"gemm-01-repeatability\",\n \"skill_should_trigger\": true,\n \"prompt\": \"Run the full standalone TraceLens analysis workflow on locally with the default analysis mode, and write everything to analysis_output/ in the current directory. The trace ships inside https://github.com/AMD-AGI/TraceLens/raw/main/TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone.tar.gz. Unpack that and analyze agent_evals/Analysis/analysis_tests/unit_tests_standalone/gemm_01_compute_few_tiles/gemm_01_compute_few_tiles.json.\",\n \"files_exist\": [\n \"analysis_output/analysis.md\",\n \"analysis_output/perf_report.xlsx\",\n \"analysis_output/category_data/category_manifest.json\",\n \"analysis_output/priority_data.json\",\n \"analysis_output/metadata/model_info.json\",\n \"analysis_output/perf_improvement.png\"\n ],\n \"expected_behavior\": [\n \"Filled the Executive Summary of analysis.md with measured numbers, leaving no template placeholders such as 'X ms', 'Y%' or '\u003cBrief Title>' behind\"\n ]\n },\n {\n \"id\": \"analyze-torch-trace\",\n \"skill_should_trigger\": true,\n \"prompt\": \"My training step is way slower than I expected. I ran it under the PyTorch profiler and the output is at https://github.com/AMD-AGI/TraceLens/raw/main/tests/traces/mi300/resnet_act_checkpoint.json.gz. Work out where the time is actually going and write it up so I know what to fix first.\"\n },\n {\n \"id\": \"compare-two-traces\",\n \"skill_should_trigger\": true,\n \"prompt\": \"I profiled the same model on two different machines and one is clearly slower, but I can't tell why. The captures are at https://github.com/AMD-AGI/TraceLens/raw/main/tests/traces/mi300/facebook_timesformer-base-finetuned-k400__1016002.json.gz and https://github.com/AMD-AGI/TraceLens/raw/main/tests/traces/h100/facebook_timesformer-base-finetuned-k400__1016002.json.gz. Tell me which GPU kernels are worse on the slow one, worst first, and write it up.\"\n },\n {\n \"id\": \"agentic-analysis-workflow\",\n \"skill_should_trigger\": true,\n \"prompt\": \"Run the agentic analysis workflow on the trace at https://github.com/AMD-AGI/TraceLens/raw/main/tests/traces/mi300/gaunernst_bert-small-uncased__1016001.json.gz and produce analysis.md.\"\n },\n {\n \"id\": \"kernel-perf-report\",\n \"skill_should_trigger\": true,\n \"prompt\": \"I captured a GPU kernel trace from a distributed training run: https://github.com/AMD-AGI/TraceLens/raw/main/tests/traces/mi300/llama_70b_fsdp/rank0_trace_no_pyfn.json.gz. I need a prioritized report of the worst performance offenders that I can show stakeholders.\"\n },\n {\n \"id\": \"node-event-loop\",\n \"skill_should_trigger\": false,\n \"prompt\": \"My Node.js server shows high event loop latency under load. Help me profile it and find the blocking call.\"\n },\n {\n \"id\": \"cprofile-python-script\",\n \"skill_should_trigger\": false,\n \"prompt\": \"Profile my Python script with cProfile and tell me which function is eating the most time.\"\n }\n ]\n}\n","content_type":"application/json; charset=utf-8","language":"json","size":3133,"content_sha256":"2e4823156d3ac31b0ef8b02060af6d63e63f1e96cb864bd2fee295b6254bbf80"},{"filename":"evals/machine.yml","content":"###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\nos: [Linux]\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":285,"content_sha256":"84c1fba8ef08613c98f8bfa6c4ce182ef5bb6019c0a21ab8272648043e1c4fa6"},{"filename":"evals/partial_tests/eval_utils/semantic_partition_scripted_evals.py","content":"###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\n\"\"\"Scripted eval: LCA-partition purity vs. a with-capture (gold) reference.\n\nSelf-gating: only test cases whose reference_dir contains a\n``semantic_purity_gold_diff_stats.csv`` (which ships pre-baked in each\npartial-test fixture; regenerated only rarely by\n``generate_gold_ref.sh``) are scored. Any other test case\nproduces zero rows here, so this eval is inert unless a gold reference is\npresent.\n\nFor gated test cases, this reuses compare_lca_partitions.py's forward/reverse\npurity and strict forward/reverse consistency metrics, computed between the\ngold partition and the candidate's semantic-bucketing output\n(``tracediff_output/diff_stats.csv``, produced by the semantic-comparison\nworkflow through its \"Generate TraceDiff Output\" step), and records them.\n\nIMPORTANT: this per-run result is informational only. Because the semantic\nmethod has real run-to-run variance, a single\nrun's metrics are not a reliable regression signal by themselves -- the\nactual pass/fail decision is made by semantic_purity_aggregate.py, which\naverages strict_forward across NUM_REPEATS runs and compares against a\nfloor derived from currently-observed performance. See that script and the\npartial_tests/README.md \"Semantic-purity quality gate\" section.\n\nConsequently the \"result\" written here is PASS whenever the metrics were\ncomputed at all; it only turns FAIL for genuine pipeline errors (candidate\noutput missing, no matched keys).\n\"\"\"\n\nimport argparse\nimport csv\nimport os\nimport sys\n\n# compare_lca_partitions lives two dirs up in eval_utils/ (not a package).\n_HERE = os.path.dirname(os.path.abspath(__file__))\n_ANALYSIS_EVAL_UTILS = os.path.abspath(os.path.join(_HERE, \"..\", \"..\", \"eval_utils\"))\nsys.path.insert(0, _ANALYSIS_EVAL_UTILS)\nfrom compare_lca_partitions import (\n LCA_COL,\n both_purities,\n both_strict,\n load,\n) # noqa: E402\n\nCSV_COLUMNS = [\n \"index\",\n \"category\",\n \"issue_summary\",\n \"result\",\n \"details\",\n \"root_cause\",\n \"recommended_fix\",\n]\n\nGOLD_FILENAME = \"semantic_purity_gold_diff_stats.csv\"\nCANDIDATE_RELPATH = os.path.join(\"tracediff_output\", \"diff_stats.csv\")\n\n\ndef _write(results_path: str, rows: list[dict]) -> None:\n with open(results_path, \"w\", newline=\"\") as f:\n writer = csv.DictWriter(f, fieldnames=CSV_COLUMNS)\n writer.writeheader()\n writer.writerows(rows)\n\n\ndef run(output_dir: str, reference_dir: str, results_path: str) -> list[dict]:\n gold_path = os.path.join(reference_dir, GOLD_FILENAME)\n if not os.path.isfile(gold_path):\n # No gold reference here; self-gate to a no-op.\n _write(results_path, [])\n return []\n\n candidate_path = os.path.join(output_dir, CANDIDATE_RELPATH)\n if not os.path.isfile(candidate_path):\n rows = [\n {\n \"index\": \"semantic_purity_1\",\n \"category\": \"Quality\",\n \"issue_summary\": \"LCA-partition purity vs with-capture gold\",\n \"result\": \"FAIL\",\n \"details\": f\"Candidate semantic diff-stats not found: {candidate_path}\",\n \"root_cause\": \"pipeline\",\n \"recommended_fix\": \"Semantic workflow did not produce tracediff_output/diff_stats.csv; check the run log\",\n }\n ]\n _write(results_path, rows)\n return rows\n\n gold = load(gold_path)\n cand = load(candidate_path)\n\n merged = (\n gold[[\"key\", LCA_COL]]\n .rename(columns={LCA_COL: \"lca_gold\"})\n .merge(\n cand[[\"key\", LCA_COL]].rename(columns={LCA_COL: \"lca_cand\"}),\n on=\"key\",\n )\n )\n\n if merged.empty:\n rows = [\n {\n \"index\": \"semantic_purity_1\",\n \"category\": \"Quality\",\n \"issue_summary\": \"LCA-partition purity vs with-capture gold\",\n \"result\": \"FAIL\",\n \"details\": \"No matched (source, gpu_op_uid) keys between gold and candidate\",\n \"root_cause\": \"data\",\n \"recommended_fix\": \"Check that candidate and gold were generated from the same DECODE trace pair\",\n }\n ]\n _write(results_path, rows)\n return rows\n\n gold_lbl = merged[\"lca_gold\"].to_numpy()\n cand_lbl = merged[\"lca_cand\"].to_numpy()\n fwd, rev = both_purities(gold_lbl, cand_lbl)\n sfwd, srev = both_strict(gold_lbl, cand_lbl)\n\n # Informational only -- see module docstring. The regression gate is\n # applied post-hoc, across repeats, by semantic_purity_aggregate.py.\n details = (\n f\"matched={len(merged)} forward_purity={fwd:.4f} reverse_purity={rev:.4f} \"\n f\"strict_forward={sfwd:.4f} strict_reverse={srev:.4f} \"\n f\"shared_buckets={merged['lca_cand'].nunique()} \"\n f\"(informational only -- see semantic_purity_aggregate.py for the actual gate)\"\n )\n rows = [\n {\n \"index\": \"semantic_purity_1\",\n \"category\": \"Quality\",\n \"issue_summary\": \"LCA-partition purity vs with-capture gold\",\n \"result\": \"PASS\",\n \"details\": details,\n \"root_cause\": \"\",\n \"recommended_fix\": \"\",\n }\n ]\n _write(results_path, rows)\n return rows\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"LCA-partition purity vs with-capture gold\"\n )\n parser.add_argument(\"--output-dir\", required=True)\n parser.add_argument(\"--reference-dir\", required=True)\n parser.add_argument(\"--results\", required=True)\n args = parser.parse_args()\n\n rows = run(args.output_dir, args.reference_dir, args.results)\n if not rows:\n sys.exit(0)\n passed = sum(1 for r in rows if r[\"result\"] == \"PASS\")\n sys.exit(0 if passed == len(rows) else 1)\n\n\nif __name__ == \"__main__\":\n main()\n","content_type":"text/x-python; charset=utf-8","language":"python","size":6004,"content_sha256":"6f6a36d422de84ceafaa386f8a3ebea20238a18fea75448e00e7ba38bd6cf302"},{"filename":"evals/partial_tests/eval_utils/semantic_purity_aggregate.py","content":"###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\n\"\"\"Aggregate the semantic-purity regression gate across repeated runs.\n\nsemantic_partition_scripted_evals.py records per-run LCA-partition purity\nmetrics but does not gate on them (see that module's docstring) -- the\nsemantic-bucketing method has real run-to-run variance, so a single run is not a reliable regression signal.\n\nThis script is the actual gate. For each semantic-purity test case it:\n 1. Scans results_root/\u003ctest_id>/run_*/semantic_purity_results.csv,\n parsing out each run's strict_forward value (the \"strict forward\n consistency\" metric from compare_lca_partitions.py: the fraction of\n gold groups that map, in their entirety, to a single candidate LCA).\n strict_forward is used as the sole gating metric -- it is more\n discriminating than forward_purity because it does not give partial\n credit to a majority-only match.\n 2. Averages strict_forward across however many runs were found.\n 3. Compares that average against MIN_STRICT_FORWARD_AVG[test_id] -- a\n floor set BELOW the worst single independently-observed run (with\n margin), so that even a quick NUM_REPEATS=1 check does not\n false-alarm on ordinary variance, while still sitting far above the\n random-shuffle baseline (Baseline 1 in compare_lca_partitions.py,\n ~0.04-0.06 strict_forward for both models). The intent is regression\n detection (\"don't get materially worse than today\"), not an absolute\n quality bar.\n\nWrites one verdict row (7-column eval schema) per test id to\nresults_root/\u003ctest_id>/semantic_purity_aggregate_verdict.csv, and prints a\nsummary table.\n\nUsage: python3 semantic_purity_aggregate.py --results-root \u003cRESULTS_ROOT>\n\"\"\"\n\nimport argparse\nimport csv\nimport glob\nimport os\nimport re\nimport statistics\nimport sys\n\nCSV_COLUMNS = [\n \"index\",\n \"category\",\n \"issue_summary\",\n \"result\",\n \"details\",\n \"root_cause\",\n \"recommended_fix\",\n]\n\n# Floors are set below the worst single strict_forward run observed during\n# decode-only method validation (with margin), NOT at the mean -- so a\n# one-off NUM_REPEATS=1 quick check cannot false-alarm on ordinary\n# variance. Averaging across repeats leaves even more headroom.\n#\n# Observed decode-only strict_forward (compare_lca_partitions.py vs the\n# pre-baked gold):\n# deepseek_r1: 0.7280, 0.9780, 0.9748, 0.9748, 0.9780 (worst 0.7280)\n# qwen3_30b_a3b: 0.3710, 0.5671, 0.4677, 0.7304, 0.5691 (worst 0.3710)\n# Both floors sit far above the random-shuffle baseline (~0.04-0.06). If the\n# method is intentionally improved, raise these; do not lower them without a\n# documented reason.\nMIN_STRICT_FORWARD_AVG = {\n \"semantic_purity_deepseek_r1\": 0.60,\n \"semantic_purity_qwen3_30b_a3b\": 0.30,\n}\n\n# Degenerate-collapse guard. strict_forward saturates to ~1.0 when the candidate\n# partition collapses toward a single LCA bucket (every gold group is trivially\n# homogeneous inside one bucket), so the floor above cannot catch total collapse.\n# Distinct candidate LCA buckets on the matched set of observed healthy runs\n# ranged 26-32 for both models; total collapse is 1. A floor of 10 sits ~2.6x\n# below the worst healthy run and ~10x above collapse, so ordinary bucketing\n# variance never false-alarms while any real collapse fails hard.\nMIN_SHARED_BUCKETS = {\n \"semantic_purity_deepseek_r1\": 10,\n \"semantic_purity_qwen3_30b_a3b\": 10,\n}\n\nDETAILS_RE = re.compile(r\"strict_forward=([0-9.]+)\")\nFORWARD_RE = re.compile(r\"forward_purity=([0-9.]+)\")\nBUCKETS_RE = re.compile(r\"shared_buckets=([0-9]+)\")\n\n\ndef find_run_csvs(results_root: str, test_id: str) -> list[str]:\n pattern = os.path.join(\n results_root, test_id, \"run_*\", \"semantic_purity_results.csv\"\n )\n return sorted(glob.glob(pattern))\n\n\ndef parse_run(csv_path: str):\n with open(csv_path, newline=\"\") as f:\n rows = list(csv.DictReader(f))\n if not rows:\n return None # self-gated no-op for this test id (shouldn't happen here)\n row = rows[0]\n details = row.get(\"details\", \"\")\n m_sfwd = DETAILS_RE.search(details)\n m_fwd = FORWARD_RE.search(details)\n m_buckets = BUCKETS_RE.search(details)\n if not m_sfwd:\n return None # pipeline failure row, no metrics to parse\n return {\n \"strict_forward\": float(m_sfwd.group(1)),\n \"forward_purity\": float(m_fwd.group(1)) if m_fwd else None,\n \"shared_buckets\": int(m_buckets.group(1)) if m_buckets else None,\n \"source\": csv_path,\n }\n\n\ndef aggregate_one(results_root: str, test_id: str) -> dict:\n run_csvs = find_run_csvs(results_root, test_id)\n parsed = [p for p in (parse_run(c) for c in run_csvs) if p is not None]\n\n if not parsed:\n return {\n \"index\": \"semantic_purity_aggregate\",\n \"category\": \"Quality\",\n \"issue_summary\": f\"Semantic-purity regression gate ({test_id})\",\n \"result\": \"FAIL\",\n \"details\": f\"No usable runs found under {results_root}/{test_id}/run_*/semantic_purity_results.csv\",\n \"root_cause\": \"pipeline\",\n \"recommended_fix\": \"Ensure the run(s) completed for this test id before aggregating\",\n }\n\n values = [p[\"strict_forward\"] for p in parsed]\n avg = statistics.mean(values)\n floor = MIN_STRICT_FORWARD_AVG.get(test_id)\n if floor is None:\n return {\n \"index\": \"semantic_purity_aggregate\",\n \"category\": \"Quality\",\n \"issue_summary\": f\"Semantic-purity regression gate ({test_id})\",\n \"result\": \"FAIL\",\n \"details\": f\"No floor configured for test id {test_id}; add one to MIN_STRICT_FORWARD_AVG\",\n \"root_cause\": \"config\",\n \"recommended_fix\": \"Add this test id to MIN_STRICT_FORWARD_AVG in semantic_purity_aggregate.py\",\n }\n\n passed = avg >= floor\n\n # Collapse guard: fail hard if the distinct candidate LCA count on the\n # matched set drops below the floor, which would saturate strict_forward.\n bucket_floor = MIN_SHARED_BUCKETS.get(test_id)\n bucket_vals = [\n p[\"shared_buckets\"] for p in parsed if p.get(\"shared_buckets\") is not None\n ]\n min_buckets = min(bucket_vals) if bucket_vals else None\n collapsed = (\n bucket_floor is not None\n and min_buckets is not None\n and min_buckets \u003c bucket_floor\n )\n passed = passed and not collapsed\n\n n_note = (\n \"\"\n if len(parsed) > 1\n else \" (single run; a multi-run average is a stronger estimate)\"\n )\n bucket_note = (\n f\" min_shared_buckets={min_buckets} bucket_floor={bucket_floor}\"\n if bucket_floor is not None\n else \"\"\n )\n details = (\n f\"n_runs={len(parsed)} strict_forward_values={[round(v, 4) for v in values]} \"\n f\"avg_strict_forward={avg:.4f} floor={floor}{bucket_note}{n_note}\"\n )\n if collapsed:\n details += (\n f\" -- FAIL: candidate partition collapsed toward a single LCA bucket \"\n f\"(min {min_buckets} \u003c {bucket_floor}); strict_forward is unreliable\"\n )\n\n if passed:\n recommended_fix = \"\"\n elif collapsed:\n recommended_fix = (\n \"Candidate LCA partition collapsed (near-single-bucket); strict_forward \"\n \"is saturated and meaningless. Investigate the clustering/coherence step.\"\n )\n else:\n recommended_fix = (\n \"Semantic-bucketing quality regressed below the observed-performance \"\n \"floor; investigate the clustering change\"\n )\n\n return {\n \"index\": \"semantic_purity_aggregate\",\n \"category\": \"Quality\",\n \"issue_summary\": f\"Semantic-purity regression gate ({test_id})\",\n \"result\": \"PASS\" if passed else \"FAIL\",\n \"details\": details,\n \"root_cause\": \"\" if passed else \"quality\",\n \"recommended_fix\": recommended_fix,\n }\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"Aggregate semantic-purity gate across repeats\"\n )\n parser.add_argument(\"--results-root\", required=True)\n parser.add_argument(\n \"--test-ids\",\n default=\",\".join(MIN_STRICT_FORWARD_AVG.keys()),\n help=\"Comma-separated test ids to aggregate (default: all known semantic-purity test ids)\",\n )\n args = parser.parse_args()\n\n test_ids = [t.strip() for t in args.test_ids.split(\",\") if t.strip()]\n overall_pass = True\n for test_id in test_ids:\n case_dir = os.path.join(args.results_root, test_id)\n if not os.path.isdir(case_dir):\n print(f\"[{test_id}] skipped (no results directory found)\")\n continue\n result = aggregate_one(args.results_root, test_id)\n out_path = os.path.join(case_dir, \"semantic_purity_aggregate_verdict.csv\")\n with open(out_path, \"w\", newline=\"\") as f:\n writer = csv.DictWriter(f, fieldnames=CSV_COLUMNS)\n writer.writeheader()\n writer.writerow(result)\n print(f\"[{test_id}] {result['result']}: {result['details']}\")\n print(f\" -> {out_path}\")\n if result[\"result\"] != \"PASS\":\n overall_pass = False\n\n sys.exit(0 if overall_pass else 1)\n\n\nif __name__ == \"__main__\":\n main()\n","content_type":"text/x-python; charset=utf-8","language":"python","size":9397,"content_sha256":"94b694f4fb99b4bad05e7817e4ac5beb597336d5e28978e0aa796e47828db5f2"},{"filename":"evals/partial_tests/generate_gold_ref.sh","content":"#!/usr/bin/env bash\n###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\nset -uo pipefail\n\n# ---------------------------------------------------------------------------\n# OPTIONAL, RARELY RUN. The gold reference ships pre-baked inside each\n# partial-test fixture tarball -- normal test runs never call this script.\n#\n# Run it only to regenerate the gold reference (e.g. after an intentional,\n# reviewed change to the with-capture TraceDiff path). It reproduces gold the\n# same way it was originally produced: the with-capture perf-report / TraceDiff\n# path on the single-execution DECODE trace pair (batch-16 decode), which is\n# what makes gold's gpu_op_uid range line up with the no-capture candidate.\n#\n# It sources the DECODE traces AND their capture_traces folders from the\n# original full-fidelity trace location (NOT from the slimmed fixture, which\n# has no capture data), writes the refreshed CSV into each fixture's\n# analysis_output_ref/, and rebuilds the fixture tarball so gold stays shipped.\n#\n# Usage: bash generate_gold_ref.sh [test_id ...] (default: all)\n# ---------------------------------------------------------------------------\n\nREPO_ROOT=\"${REPO_ROOT:-$(pwd)}\"\nPARTIAL_DIR=\"$REPO_ROOT/TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/partial_tests\"\nGOLD_FILENAME=\"semantic_purity_gold_diff_stats.csv\"\nARCH_JSON=\"${ARCH_JSON:-$REPO_ROOT/TraceLens/Agent/Analysis/utils/arch/MI300X.json}\"\nPERF_CLI=\"${PERF_CLI:-TraceLens_generate_perf_report_pytorch_inference}\"\n\n# Original full-capture source locations (repo-relative). Override via env if\n# the traces move. Format: \"\u003cdecode_a> \u003ccap_a> \u003cdecode_b> \u003ccap_b>\".\nSEM_ROOT=\"tests/traces/semantic\"\ndeclare -A SOURCES=(\n [semantic_purity_deepseek_r1]=\"\\\n$SEM_ROOT/deepseek_R1/Deepseek-R1-Distill-LLama-8B/MI300/torch_trace/1782841132.2905657-TP-0-DECODE.trace.json.gz \\\n$SEM_ROOT/deepseek_R1/Deepseek-R1-Distill-LLama-8B/MI300/torch_trace/capture_traces \\\n$SEM_ROOT/deepseek_R1/Deepseek-R1-Distill-LLama-8B/B300/torch_trace/1782843274.7730265-TP-0-DECODE.trace.json.gz \\\n$SEM_ROOT/deepseek_R1/Deepseek-R1-Distill-LLama-8B/B300/torch_trace/capture_traces\"\n [semantic_purity_qwen3_30b_a3b]=\"\\\n$SEM_ROOT/qwen3_30b_a3b/Qwen3-30B-A3B/MI300/torch_trace/1782859605.7198358-TP-0-DECODE.trace.json.gz \\\n$SEM_ROOT/qwen3_30b_a3b/Qwen3-30B-A3B/MI300/torch_trace/capture_traces \\\n$SEM_ROOT/qwen3_30b_a3b/Qwen3-30B-A3B/B300/torch_trace/1782799835.6885648-TP-0-DECODE.trace.json.gz \\\n$SEM_ROOT/qwen3_30b_a3b/Qwen3-30B-A3B/B300/torch_trace/capture_traces\"\n)\n\ngenerate_one() {\n local id=\"$1\"\n local spec=\"${SOURCES[$id]:-}\"\n if [[ -z \"$spec\" ]]; then\n echo \"ERROR: unknown test id '$id' (no source mapping)\" >&2\n return 1\n fi\n # shellcheck disable=SC2086\n set -- $spec\n local decode_a=\"$REPO_ROOT/$1\" cap_a=\"$REPO_ROOT/$2\" decode_b=\"$REPO_ROOT/$3\" cap_b=\"$REPO_ROOT/$4\"\n\n for p in \"$decode_a\" \"$cap_a\" \"$decode_b\" \"$cap_b\" \"$ARCH_JSON\"; do\n [[ -e \"$p\" ]] || { echo \"ERROR [$id]: missing source $p\" >&2; return 1; }\n done\n\n local scratch=\"$PARTIAL_DIR/_gold_scratch_$id\"\n local out_csvs=\"$scratch/perf_report_trace1_csvs\"\n rm -rf \"$scratch\"; mkdir -p \"$scratch\"\n\n echo \"[$id] running with-capture TraceDiff on the DECODE pair...\"\n # trace1 (MI300) with capture + comparison against trace2 (B300) with its\n # capture triggers TraceDiff internally and writes diff_stats.csv into\n # perf_report_trace1_csvs/. Only platform1's arch JSON is needed.\n \"$PERF_CLI\" \\\n --profile_json_path \"$decode_a\" --capture_folder \"$cap_a\" \\\n --gpu_arch_json_path \"$ARCH_JSON\" \\\n --group_by_parent_module --enable_pseudo_ops --group_by_num_kernels --include_call_stack \\\n --comparison_json_path \"$decode_b\" --comparison_capture_folder \"$cap_b\" \\\n --output_xlsx_path \"$scratch/perf_report_trace1.xlsx\" \\\n --output_csvs_dir \"$out_csvs\" \\\n || { echo \"ERROR [$id]: perf-report CLI failed\" >&2; return 1; }\n\n local gold_src=\"$out_csvs/diff_stats.csv\"\n [[ -f \"$gold_src\" ]] || { echo \"ERROR [$id]: $gold_src not produced\" >&2; return 1; }\n\n local ref_dir=\"$PARTIAL_DIR/$id/analysis_output_ref\"\n mkdir -p \"$ref_dir\"\n cp \"$gold_src\" \"$ref_dir/$GOLD_FILENAME\"\n echo \"[$id] gold -> $ref_dir/$GOLD_FILENAME ($(wc -l \u003c \"$ref_dir/$GOLD_FILENAME\") lines)\"\n\n # Re-pack the fixture tarball so the refreshed gold ships with it.\n tar czf \"$PARTIAL_DIR/fixtures/${id}.tar.gz\" -C \"$PARTIAL_DIR\" --transform='s,^,agent_evals/Analysis/partial_tests/,' \"$id\"\n echo \"[$id] fixture repacked -> fixtures/${id}.tar.gz\"\n rm -rf \"$scratch\"\n}\n\nids=(\"$@\")\nif [[ ${#ids[@]} -eq 0 ]]; then\n ids=(\"${!SOURCES[@]}\")\nfi\n\nrc=0\nfor id in \"${ids[@]}\"; do\n generate_one \"$id\" || rc=1\ndone\nexit \"$rc\"\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":4977,"content_sha256":"8202e90478f97edac804974d00d3f11f05d8a4c16be811acaae420b5cc271cf7"},{"filename":"evals/partial_tests/partial_test_cases.csv","content":"id,workflow,trace_a,trace_b,reference_dir,platform_a,platform_b\nsemantic_purity_deepseek_r1,semantic_comparison,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/partial_tests/semantic_purity_deepseek_r1/MI300/1782841132.2905657-TP-0-DECODE.trace.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/partial_tests/semantic_purity_deepseek_r1/B300/1782843274.7730265-TP-0-DECODE.trace.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/partial_tests/semantic_purity_deepseek_r1/analysis_output_ref,MI300,B300\nsemantic_purity_qwen3_30b_a3b,semantic_comparison,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/partial_tests/semantic_purity_qwen3_30b_a3b/MI300/1782859605.7198358-TP-0-DECODE.trace.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/partial_tests/semantic_purity_qwen3_30b_a3b/B300/1782799835.6885648-TP-0-DECODE.trace.json.gz,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/partial_tests/semantic_purity_qwen3_30b_a3b/analysis_output_ref,MI300,B300\n","content_type":"text/csv; charset=utf-8","language":"csv","size":1044,"content_sha256":"15e3865508af1f9758289d7b05924218dd8db9729f32dc862915243f231aac24"},{"filename":"evals/partial_tests/README.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n# Partial-workflow tests\n\nThis folder holds tests that run only **part** of an analysis workflow — just\nfar enough to produce an artifact that a scripted eval scores against a\npre-baked reference. This is deliberately lighter than\n`../eval_scripts/run_repeatability_parallel.sh`, which runs the full 11-step\nanalysis orchestrator per case.\n\nThe first (and currently only) partial-workflow test is **`semantic_purity`**.\nThe folder is named generically so other partial-workflow tests can be added\nbeside it (give them a new value in the `workflow` column of\n`partial_test_cases.csv` and a matching branch in `run_partial_tests.sh`).\n\n## Layout\n\n```\npartial_tests/\n├── run_partial_tests.sh # the runner\n├── generate_gold_ref.sh # OPTIONAL, rare: regenerate gold\n├── partial_test_cases.csv # manifest (id,workflow,trace_a,trace_b,reference_dir,platform_a,platform_b)\n├── eval_utils/\n│ ├── semantic_partition_scripted_evals.py # per-run purity metrics (informational)\n│ └── semantic_purity_aggregate.py # the regression gate\n├── fixtures/*.tar.gz # DECODE trace pair + pre-baked gold (small)\n└── \u003ctest_id>/ # expanded fixture (MI300/, B300/, analysis_output_ref/)\n```\n\n`compare_lca_partitions.py` — the shared purity/consistency math — lives in\n`../eval_utils/` (it is also imported by `tests/test_compare_lca_partitions.py`).\n\n## What `semantic_purity` does\n\nFor each model (DeepSeek-R1, Qwen3-30B-A3B) the fixture ships:\n- the **DECODE** trace for MI300 and B300 (single batch-16 decode execution — the\n only trace kind these tests use), and\n- a pre-baked **gold** `analysis_output_ref/semantic_purity_gold_diff_stats.csv`.\n\nThe runner invokes the **semantic-comparison workflow** (the no-capture,\nname-first + coherence bucketing method) on the DECODE pair, running only\nthrough its \"Generate TraceDiff Output\" step — equivalent to the main\norchestrator through Step 1 pre-processing. That produces\n`analysis_output/tracediff_output/diff_stats.csv`, whose LCA partition is then\ncompared to gold.\n\nGold itself is the *with-capture* TraceDiff partition on the same DECODE pair.\nBecause both sides derive from the same single-execution DECODE trace, their\n`gpu_op_uid` numbering lines up, so the `(source, gpu_op_uid)` join used by\n`compare_lca_partitions.py` is meaningful.\n\n## Running\n\n```bash\n# from the repo root\nbash TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/partial_tests/run_partial_tests.sh\n\n# one model, single run (quick check)\nTEST_IDS=semantic_purity_deepseek_r1 NUM_REPEATS=1 \\\n bash TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/partial_tests/run_partial_tests.sh\n\n# variance study\nNUM_REPEATS=5 bash TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/partial_tests/run_partial_tests.sh\n```\n\nEnv knobs: `NUM_REPEATS` (default 1), `TEST_IDS` (space-separated whitelist),\n`MAX_PARALLEL` (default 3), `CONTAINER` (docker container to exec python in),\n`AGENT_MODEL`. Results land under `partial_tests/results/\u003cid>/run_*/`.\n\n## Semantic-purity quality gate\n\nThe semantic bucketing method uses an LLM for its unification/coherence steps,\nso it has **real run-to-run variance**. Therefore:\n\n- `semantic_partition_scripted_evals.py` writes each run's metrics\n (`forward_purity`, `strict_forward`, etc.) but is **informational only** — its\n per-run `result` is PASS unless the pipeline itself failed (missing candidate,\n no matched keys).\n- `semantic_purity_aggregate.py` is the **actual gate**. It averages\n `strict_forward` across the runs found and compares to a per-model floor.\n The floors — and the observed-run data they're derived from — live in that\n script's `MIN_STRICT_FORWARD_AVG` and its module docstring, which are the\n single source of truth (not duplicated here). The floors are set below the\n worst single observed run with margin, so a quick `NUM_REPEATS=1` check does\n not false-alarm, while staying well above the random-shuffle baseline. Raise\n them if the method is intentionally improved; do not lower them without a\n documented reason.\n\n## Regenerating gold (optional, rare — maintainer only)\n\nGold ships **pre-baked** inside the fixture tarballs, so normal test runs never\nneed this. Regenerate only after an intentional, reviewed change to the\nwith-capture TraceDiff path.\n\n**Precondition — not runnable from a clean clone.** Regeneration needs the\noriginal **full-capture** traces (the DECODE traces *and* their\n`capture_traces/` folders). Those are large and are **not committed to the\nrepo** — only the slimmed DECODE-only traces ship in the fixtures. You must\nalready have the full-capture traces available locally (on the machine where\nthey were captured). The script's source paths default to\n`tests/traces/semantic/...` and are set in the `SEM_ROOT` / `SOURCES` entries at\nthe top of the script; edit them to point at wherever you keep the traces. The\nscript fails with a clear \"missing source\" error if they aren't present.\n\n```bash\nbash TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/partial_tests/generate_gold_ref.sh\n```\n\nIt runs the with-capture perf-report / TraceDiff path on the DECODE pair, writes\nthe refreshed CSV into each `analysis_output_ref/`, and repacks the fixture\ntarball.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":5465,"content_sha256":"071688904f23d8f40cb5e0b8444b86624feda6b7b1d7c5534203eefb67902423"},{"filename":"evals/partial_tests/run_partial_tests.sh","content":"#!/usr/bin/env bash\n###############################################################################\n# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n#\n# See LICENSE for license information.\n###############################################################################\n\nset -uo pipefail\n\n# ---------------------------------------------------------------------------\n# Partial-workflow test runner.\n#\n# Unlike run_repeatability_parallel.sh (which runs the FULL 11-step analysis\n# orchestrator per case), this harness runs only PART of a workflow -- just\n# far enough to produce the artifact a scripted eval scores against a\n# pre-baked reference. The first such test is semantic_purity, which runs the\n# semantic-comparison workflow through its \"Generate TraceDiff Output\" step\n# (== main orchestrator through Step 1 pre-processing) to produce\n# \u003cOUTPUT_DIR>/tracediff_output/diff_stats.csv\n# and scores its LCA-partition purity against the committed gold.\n#\n# New partial-workflow tests can be added by giving them a distinct value in\n# the `workflow` column of partial_test_cases.csv and a matching branch in\n# run_single_job below.\n#\n# Usage: bash run_partial_tests.sh\n# NUM_REPEATS repeats per case (default 1; raise for a variance study)\n# TEST_IDS space-separated id whitelist (default: all)\n# MAX_PARALLEL concurrent jobs (default 3)\n# CONTAINER optional docker container to exec python in\n# ---------------------------------------------------------------------------\n\nMAX_PARALLEL=\"${MAX_PARALLEL:-3}\"\nNUM_REPEATS=\"${NUM_REPEATS:-1}\"\nTEST_IDS=\"${TEST_IDS:-}\"\nCONTAINER=\"${CONTAINER:-}\"\nAGENT_MODEL=\"${AGENT_MODEL:-claude-opus-4-8-thinking-medium}\"\n\nREPO_ROOT=\"${REPO_ROOT:-$(pwd)}\"\nANALYSIS_DIR=\"$REPO_ROOT/TraceLens/Agent/Analysis\"\nEVALS_DIR=\"$REPO_ROOT/TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals\"\nPARTIAL_DIR=\"$EVALS_DIR/partial_tests\"\nTEST_CASES_CSV=\"${TEST_CASES_CSV:-$PARTIAL_DIR/partial_test_cases.csv}\"\nRESULTS_ROOT=\"${RESULTS_ROOT:-$PARTIAL_DIR/results}\"\n\nif [[ -n \"$CONTAINER\" ]]; then\n DEXEC=(docker exec -w \"$REPO_ROOT\" \"$CONTAINER\")\nelse\n DEXEC=()\nfi\n\nts() { date \"+%H:%M:%S\"; }\nlog_status() { flock 1 echo \"$@\"; }\n\n# Expand a fixture tarball (members carry a legacy agent_evals/ prefix, stripped on extract).\nexpand_fixture() {\n local id=\"$1\"\n local archive=\"$PARTIAL_DIR/fixtures/${id}.tar.gz\"\n local target=\"$PARTIAL_DIR/$id\"\n if [[ -f \"$archive\" ]] && [[ ! -d \"$target\" ]]; then\n echo \"Expanding fixtures/${id}.tar.gz...\"\n tar xzf \"$archive\" -C \"$PARTIAL_DIR\" --strip-components=3\n fi\n}\n\n# ---------------------------------------------------------------------------\n# One (test_case, repeat) iteration.\n# Args: id workflow repeat trace_a trace_b reference_dir platform_a platform_b\n# ---------------------------------------------------------------------------\nrun_single_job() {\n local id=\"$1\" workflow=\"$2\" repeat=\"$3\" trace_a=\"$4\" trace_b=\"$5\" reference_dir=\"$6\" platform_a=\"$7\" platform_b=\"$8\"\n local tag=\"[$id|run_$repeat]\"\n\n local CASE_RESULTS=\"$RESULTS_ROOT/$id/run_${repeat}\"\n local OUTPUT_DIR=\"$CASE_RESULTS/analysis_output\"\n rm -rf \"$CASE_RESULTS\" 2>/dev/null || true\n mkdir -p \"$OUTPUT_DIR\"\n\n local abs_a=\"$REPO_ROOT/$trace_a\"\n local abs_b=\"$REPO_ROOT/$trace_b\"\n local abs_ref=\"$REPO_ROOT/$reference_dir\"\n\n log_status \" $tag [$(ts)] workflow=$workflow starting\"\n\n case \"$workflow\" in\n semantic_comparison)\n local agent_success=false\n local attempts=0\n while [ \"$agent_success\" = false ] && [ \"$attempts\" -lt 3 ]; do\n attempts=$((attempts + 1))\n (\n cd \"$ANALYSIS_DIR\" || exit\n timeout 1800 agent --model \"$AGENT_MODEL\" --print --force --trust --output-format stream-json \\\n \"Follow the semantic-comparison-agent workflow (TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/semantic-comparison-agent.md). Run it on trace A=$abs_a (platform $platform_a) and trace B=$abs_b (platform $platform_b), output to $OUTPUT_DIR. Run only through Step 3 'Generate TraceDiff Output' so that $OUTPUT_DIR/tracediff_output/diff_stats.csv is produced; do NOT run the full analysis orchestrator or any downstream report/category steps.\"\n ) \u003c /dev/null > \"$CASE_RESULTS/semantic_stream.ndjson\" 2>&1\n\n if head -c 2048 \"$CASE_RESULTS/semantic_stream.ndjson\" | grep -qiE 'Error:.*unavailable|Service Unavailable'; then\n log_status \" $tag Attempt $attempts/3 failed (agent unavailable). Backing off 30s...\"\n sleep 30\n else\n agent_success=true\n fi\n done\n if [ \"$agent_success\" = false ]; then\n log_status \" $tag FAILED after 3 attempts (agent unavailable). Skipping eval.\"\n return 1\n fi\n\n \"${DEXEC[@]}\" python3 \"$PARTIAL_DIR/eval_utils/semantic_partition_scripted_evals.py\" \\\n --output-dir \"$OUTPUT_DIR\" --reference-dir \"$abs_ref\" \\\n --results \"$CASE_RESULTS/semantic_purity_results.csv\" \\\n > \"$CASE_RESULTS/semantic_purity_eval.log\" 2>&1 || true\n ;;\n *)\n log_status \" $tag ERROR: unknown workflow '$workflow'\"\n return 1\n ;;\n esac\n\n log_status \" $tag [$(ts)] done -> $CASE_RESULTS/semantic_purity_results.csv\"\n}\n\n# ---------------------------------------------------------------------------\n# FIFO semaphore for concurrency control\n# ---------------------------------------------------------------------------\nFIFO=\"$RESULTS_ROOT/.job_fifo\"\ncleanup() { rm -f \"$FIFO\"; }\nsetup_semaphore() {\n rm -f \"$FIFO\"; mkfifo \"$FIFO\"; exec 4\u003c>\"$FIFO\"\n for ((t = 0; t \u003c MAX_PARALLEL; t++)); do echo >&4; done\n trap cleanup EXIT\n}\n\n# ---------------------------------------------------------------------------\n# Main\n# ---------------------------------------------------------------------------\nmkdir -p \"$RESULTS_ROOT\"\n\n# Expand every fixture referenced by the manifest.\nwhile IFS=, read -r id workflow trace_a trace_b reference_dir platform_a platform_b \u003c&3; do\n [[ -z \"$id\" ]] && continue\n expand_fixture \"$id\"\ndone 3\u003c \u003c(tail -n +2 \"$TEST_CASES_CSV\"; echo)\n\necho \"=========================================\"\necho \" Partial-workflow tests\"\necho \" Repeats: $NUM_REPEATS\"\necho \" Max parallel: $MAX_PARALLEL\"\necho \" CSV: $TEST_CASES_CSV\"\nif [[ -n \"$TEST_IDS\" ]]; then echo \" Test filter: $TEST_IDS\"; fi\necho \"=========================================\"\necho \"\"\n\n_spawn_jobs() {\n local id=\"$1\" workflow=\"$2\" trace_a=\"$3\" trace_b=\"$4\" reference_dir=\"$5\" platform_a=\"$6\" platform_b=\"$7\"\n if [[ -n \"$TEST_IDS\" ]]; then\n case \" $TEST_IDS \" in\n *\" $id \"*) ;;\n *) return ;;\n esac\n fi\n for ((i = 0; i \u003c NUM_REPEATS; i++)); do\n read -r -u4\n (\n run_single_job \"$id\" \"$workflow\" \"$i\" \"$trace_a\" \"$trace_b\" \"$reference_dir\" \"$platform_a\" \"$platform_b\" || true\n echo >&4\n sleep 2 # stagger agent startup to avoid ~/.cursor/cli-config.json rename race\n ) &\n sleep 2\n done\n}\n\nsetup_semaphore\n\n# manifest: id,workflow,trace_a,trace_b,reference_dir,platform_a,platform_b\nwhile IFS=, read -r id workflow trace_a trace_b reference_dir platform_a platform_b \u003c&3; do\n [[ -z \"$id\" ]] && continue\n _spawn_jobs \"$id\" \"$workflow\" \"$trace_a\" \"$trace_b\" \"$reference_dir\" \"$platform_a\" \"$platform_b\"\ndone 3\u003c \u003c(tail -n +2 \"$TEST_CASES_CSV\"; echo)\n\nwait\n\necho \"\"\necho \"=========================================\"\necho \" Runs finished. Applying regression gate...\"\necho \"=========================================\"\n\"${DEXEC[@]}\" python3 \"$PARTIAL_DIR/eval_utils/semantic_purity_aggregate.py\" \\\n --results-root \"$RESULTS_ROOT\" || true\n\necho \"\"\necho \" Results in: $RESULTS_ROOT\"\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":7962,"content_sha256":"2f25a3575f6bd1c816589ceacc23ac9936e37d750cb0ae8f72daf09481d58fd5"},{"filename":"evals/README.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n# Analysis Agent Evals\n\nEvaluation harness for the TraceLens Analysis Agent pipeline. Validates both **workflow correctness** (directory structure, file existence, report formatting) and **output quality** (comparison against reference reports).\n\nThe framework uses a hybrid approach:\n- **Deterministic scripted evals** (Python) for structural and factual checks — pre-check gates, per-item sub-scoring, root-cause metadata\n- **LLM-judged evals** with multi-dimensional weighted scoring for semantic checks that require reasoning\n- **Stability classification** (STABLE_PASS / FLAKY_PASS / FLAKY_FAIL / STABLE_FAIL) across repeated runs to separate deterministic bugs from LLM variability\n\nAll eval results use a 7-column CSV schema: `index, category, issue_summary, result, details, root_cause, recommended_fix`. See [EVAL_RUBRICS.md](EVAL_RUBRICS.md) for the full rubric reference.\n\nLLM and post-processing **agent skills** ship under `TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/skills/\u003cskill-name>/` (`SKILL.md` + `reference.md`). Cursor’s default skill discovery uses `.cursor/skills/` at the workspace root — symlink or copy those folders if you need automatic attachment.\n\n## Prerequisites / Setup\n\n### 1. Clone TraceLens\n\n```bash\ngit clone https://github.com/AMD-AGI/TraceLens.git\ncd TraceLens\n```\n\n### 2. Install TraceLens inside your container\n\nSSH into your node and exec into the container:\n\n```bash\nssh \u003cnode>\ndocker exec -it \u003ccontainer> bash\n```\n\nInstall TraceLens:\n\n```bash\ncd /path/to/TraceLens\npip install -e .\n```\n\n### 3. Install Cursor CLI\n\nThe eval scripts invoke the Cursor agent via the `agent` CLI command. All scripts are designed to run **directly on the node** (not the head node). The `agent` CLI must be installed on the node.\n\n```bash\ncurl https://cursor.com/install -fsS | bash\n```\nVerify with `agent --version`\n\n### 4. Set Model to Claude Opus 4.8 (Thinking, Medium)\n\nThe `agent` invocations in this repo use model `claude-opus-4-8-thinking-medium`. The eval shell scripts under `eval_scripts/` pass `--model claude-opus-4-8-thinking-medium` explicitly so they work regardless of your CLI default, but for ad-hoc `agent` runs it is convenient to set the same default in `~/.cursor/cli-config.json` (or via the Cursor IDE model picker).\n\nThe eval scripts pass `--model claude-opus-4-8-thinking-medium` to the top-level orchestrator agent. The 13 sub-agents still declare `claude-opus-4-7-high` in their own front matter under ``TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/``; update those separately if you want the sub-agents on the same model.\n\n## Running Scripts\n\nAll scripts run **on the node** from the repo root. They use `docker exec` to run Python scripts inside the container, while `agent` commands run directly on the node. The `CONTAINER` environment variable is required for all scripts.\n\n### Repeatability Study\n\nDispatches all `(test_case, repeat)` jobs concurrently with a configurable concurrency limit. After all jobs finish, the harness automatically invokes a Cursor agent to aggregate results and generate reports (see [Post-Processing Skill](#post-processing-skill)). `NUM_REPEATS=3` is the recommended setting (good stability signal at lower cost than the default 5); for a single-pass eval, set `NUM_REPEATS=1`.\n\n```bash\nCONTAINER=my_container bash TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_scripts/run_repeatability_parallel.sh\n```\n\nEnvironment variables:\n\n| Variable | Default | Description |\n|---|---|---|\n| `CONTAINER` | (required) | Docker container name |\n| `MAX_PARALLEL` | 5 | Max concurrent jobs |\n| `NUM_REPEATS` | 5 | Repeats per test case |\n| `SLEEP_BETWEEN` | 30 | Seconds between Phase 1 and Phase 2 |\n| `TEST_TRACES_CSV` | `TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/combined_traces_${COMPARISON_SCOPE}.csv` | Path to the trace CSV to use |\n| `RESULTS_ROOT` | `TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/repeatability_results` | Where per-run results are written |\n| `REPORT_DIR` | `\u003cRESULTS_ROOT>/../reports` | Where reports and reproducers are written |\n| `SUITE_NAME` | `eval` | Suite label used in reports (e.g. `unit`, `e2e`) |\n| `TEST_IDS` | (empty = all) | Space-separated trace IDs to run (filter) |\n| `SKIP_GENERATE_REF` | (empty) | Set to `1` to skip golden reference generation |\n| `SKIP_POST_PROCESSING` | (empty) | Set to `1` to skip report generation after the eval loop |\n\nExample (subset of traces, 3 repeats, 5 parallel):\n\n```bash\nCONTAINER=my_container \\\nTEST_IDS=\"qwen1.5_mi300 06_llama3_2_mi300\" \\\nNUM_REPEATS=3 MAX_PARALLEL=5 \\\n bash TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_scripts/run_repeatability_parallel.sh\n```\n\nUse a `screen` session to prevent disconnects during long runs. Each job cleans its output directory before starting to prevent stale results.\n\nTo run evals without generating reports (e.g. to re-process later):\n\n```bash\nSKIP_POST_PROCESSING=1 CONTAINER=my_container \\\n bash TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_scripts/run_repeatability_parallel.sh\n```\n\n### Generate Golden References\n\nGenerates `analysis_output_ref/` for test cases listed in `analysis_tests/combined_traces_standalone.csv` / `combined_traces_comparative.csv`. Runs analysis, copies the output as the reference, and strips intermediate files (keeps only `analysis.md` and `perf_report_csvs/`). Runs in parallel with `MAX_PARALLEL`. Skips test cases with missing trace files.\n\n```bash\nCONTAINER=my_container bash TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_scripts/generate_ref.sh\n```\n\nOr with more parallelism:\n\n```bash\nMAX_PARALLEL=5 CONTAINER=my_container bash TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_scripts/generate_ref.sh\n```\n\n### Stopping Eval Jobs\n\nKills every process `generate_ref.sh` or `run_repeatability_parallel.sh` may have spawned. Processes are identified by the `TRACELENS_EVAL_JOB=1` environment variable marker.\n\n```bash\nbash TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_scripts/kill_eval_jobs.sh\n```\n\n| Invocation | Behavior |\n|---|---|\n| (no args) | SIGTERM, then SIGKILL if processes are still alive |\n| `--list` | List eval processes without killing anything |\n| `-9` | SIGKILL from the first pass |\n| `-h` / `--help` | Show usage |\n\nExit code is 0 if all jobs are cleared (or none were found), 1 if any remain after the maximum passes (e.g., stuck in uninterruptible I/O).\n\n### Individual Manual Runs\n\nYou can run each stage independently using the `agent` CLI. Examples:\n\n**Analysis:**\n\n```bash\ncd TraceLens/Agent/Analysis\nagent --model claude-opus-4-8-thinking-medium --print --force --trust \\\n \"Follow the analysis orchestrator installed with the TraceLens pip package (look under raceLens/Agent/Analysis/skills/analysis-orchestrator/ in the package installation directory) and run the full agentic analysis workflow on \u003ctrace_path> with platform \u003cplatform>, analysis mode default, node \u003cnode>, container \u003ccontainer>, output to \u003coutput_dir>\"\n```\n\n**Workflow Eval:**\n\n```bash\ncd TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals\nagent --model claude-opus-4-8-thinking-medium --print --force --trust \\\n \"Run the workflow eval skill on \u003coutput_dir> for test case \u003cid>. Write results to \u003cresults_path>\"\n```\n\n**Quality Eval:**\n\n```bash\ncd TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals\nagent --model claude-opus-4-8-thinking-medium --print --force --trust \\\n \"Run the quality eval skill on \u003coutput_dir> with reference \u003creference_dir> for test case \u003cid>. Write results to \u003cresults_path>\"\n```\n\n## Post-Processing Skill\n\nAfter the repeatability harness finishes, a Cursor agent is automatically invoked to aggregate results and generate reports. The skill lives under `TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/skills/eval-post-processing/` (`SKILL.md` + `reference.md`).\n\nThe skill performs four steps:\n\n1. **Aggregate** -- Runs `aggregate_repeatability.py` to merge all per-run `eval_summary.csv` files into `aggregated_results.csv`, `pass_rate_summary.csv`, `stability_summary.csv`, and `stream_diagnostics.csv`. The stability summary classifies each (trace, eval) pair as `STABLE_PASS`, `FLAKY_PASS`, `FLAKY_FAIL`, or `STABLE_FAIL` based on pass rates across repeated runs.\n2. **Classify** -- Reads the aggregate CSVs and `report_section_rules.yaml` to classify failures into report sections and failure modes. Uses `root_cause` and `recommended_fix` columns from eval results.\n3. **Write reports** -- Generates `pr_report.md` (concise summary for a PR comment) and `fix_ticket_report.md` (detailed failure modes with root causes, suggested fixes, and reproducer commands).\n4. **Build reproducer packages** -- Creates a self-contained folder per failure issue with sanitized stream JSONs, eval summaries, and a README. Each folder is compressed into a `.tar.gz` that can be assigned to a developer for debugging with Cursor.\n\n### Re-running post-processing on existing results\n\nYou can re-run the post-processing skill on any previous results without re-running the eval loop:\n\n```bash\ncd TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals\nagent --model claude-opus-4-8-thinking-medium --print --force --trust \\\n \"Run eval post processing on results_root=\u003cresults_root> suite=\u003csuite> test_traces_csv=\u003ccsv_path> report_dir=\u003creport_dir> container=\u003ccontainer>\"\n```\n\nExample:\n\n```bash\ncd TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals\nagent --model claude-opus-4-8-thinking-medium --print --force --trust \\\n \"Run eval post processing on results_root=TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_reports/my_run/results/repeatability_results suite=eval test_traces_csv=TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/combined_traces_standalone.csv report_dir=TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/eval_reports/my_run/reports container=modular_evals\"\n```\n\n### Error handling\n\n- Each step logs its exit code. Failures are non-fatal — the pipeline continues and reports all failures in the summary.\n- Root-owned files from Docker are cleaned using `docker exec rm` before host `rm`.\n- The underlying scripts have built-in retry with backoff (3 attempts, 30s) for agent unavailability.\n\n## Adding New Test Cases\n\n### 1. Create the test case directory\n\nEach test case lives under `TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/\u003cunit_tests_standalone|unit_tests_comparative|e2e_tests_standalone|e2e_tests_comparative>/\u003ctest_id>/` and must contain:\n\n- The profiling trace JSON file.\n- `analysis_output_ref/` -- a reference analysis output to compare against (used by quality evals). Should include `analysis.md` and `perf_report_csvs/`. Generate this with `generate_ref.sh`.\n\n### 2. Add a row to `analysis_tests/combined_traces_standalone.csv` (or `combined_traces_comparative.csv`)\n\nThe CSV has the following columns:\n\n| Column | Description |\n|---|---|\n| `id` | Unique test case ID (e.g. `gemm_01_compute_few_tiles`) |\n| `sub_category` | Category label (e.g. `gemm`) |\n| `trace_path` | Relative path to the trace JSON |\n| `reference_dir` | Relative path to the reference output (`analysis_output_ref`) |\n| `platform` | Hardware platform (e.g. `MI300X`) |\n\nExample row:\n\n```\ngemm_01_compute_few_tiles,gemm,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/gemm_01_compute_few_tiles/gemm_01_compute_few_tiles.json,TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/analysis_tests/unit_tests_standalone/gemm_01_compute_few_tiles/analysis_output_ref,MI300X\n```\n\n## Eval Pipeline Summary\n\n### Shell Scripts\n\nFor each test case in the traces CSV, the scripts run two phases:\n\n1. **Phase 1 -- Analysis:** Invokes the Analysis agent on the trace file. Output is written to `analysis_output/`. Agent output is logged as stream JSON (`.ndjson`). The orchestrator includes targeted subagent retry (1 retry per failed subagent in Steps 6, 7, and 9) to reduce flaky pipeline failures.\n2. **Phase 2 -- Workflow + Quality Evals (4 parallel tasks):** Launches four tasks concurrently:\n - Scripted workflow evals — 13 evals with per-item sub-scoring (via `docker exec`)\n - LLM workflow eval — 1 eval with multi-dimensional weighted scoring (via `agent`)\n - Scripted quality evals — 1 eval (via `docker exec`)\n - LLM quality evals — 2 evals with multi-dimensional weighted scoring (via `agent`)\n3. **Merge Results:** Runs `eval_utils/merge_results.py` to combine per-eval CSVs into a single `eval_summary.csv`.\n\n### Eval Skills\n\nThree Cursor agent skills under `TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/skills/` define the eval and pipeline logic:\n\n**eval-post-processing** -- Aggregates repeatability results, classifies failures using `report_section_rules.yaml`, and generates PR + fix-ticket reports with reproducer packages. Invoked automatically by `run_repeatability_parallel.sh` or manually on existing results.\n\nThe remaining two skills define the per-case eval logic:\n\n**workflow-eval** -- 14 evals (13 scripted + 1 LLM):\n\n| Eval | Type | What it checks |\n|------|------|---------------|\n| 1–8 | Scripted | Directory structure, metadata files, model info, perf report, tree data, findings existence, plot |\n| 9 | Scripted | Report template rendering — all required `##` headers present. Per-header sub-indices (`_executive_summary`, `_compute`, `_system`, `_detailed`, `_appendix`, `_metrics_table`) |\n| 10 | Scripted | Executive Summary metrics — values match `gpu_timeline.csv` within 1%. Per-metric sub-indices (`_compute_pct`, `_idle_pct`, `_comm_pct`, `_total_time`, `_bottleneck`) |\n| 11 | Scripted | Issue template rendering — each P-item has Insight/Action/Impact fields. Per-P-item sub-indices (`_compute_P1`, `_system_P2`, etc.) |\n| 12 | LLM | Hardware Reference in Appendix — platform, memory BW, MAF values. Multi-dimensional weighted scoring: correctness (50%) + completeness (50%), pass threshold ≥ 7.0 |\n| 13 | Scripted | Model identification in Appendix — all 4 `model_info.json` fields present. Per-field sub-indices (`_model`, `_architecture`, `_scale`, `_precision`) |\n\nScripted evals (1–11, 13–14) run via `eval_utils/workflow_scripted_evals.py`. LLM eval (12) runs via `skills/workflow-llm-eval/` (`SKILL.md` + `reference.md`).\n\nAll scripted evals include **pre-check gates** that immediately FAIL all evals with a clear message if the output directory is missing, `analysis.md` is absent/garbled, or other fundamental prerequisites are unmet.\n\n**quality-eval** -- 3 evals (1 scripted + 2 LLM):\n\n| Eval | Type | What it checks |\n|------|------|---------------|\n| 1 | Scripted | CSV value alignment between generated and reference `perf_report_csvs/`. Run via `eval_utils/quality_scripted_evals.py` |\n| 2 | LLM | Compute Issue Title Alignment — semantic comparison of P-item titles against reference. Multi-dimensional scoring: correctness (40%) + completeness (30%) + precision (30%), pass threshold ≥ 7.0 |\n| 3 | LLM | Compute Issue Content Alignment — performance numbers, shapes, efficiency, gains against reference. Same 3-dimension scoring as eval 2 |\n\nLLM evals (2–3) run via `skills/quality-llm-eval/` (`SKILL.md` + `reference.md`). System-level P-items are skipped (no Impact field to compare).\n\n## Results\n\nResults are written to `TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/repeatability_results/\u003cid>/run_\u003cn>/` and include:\n\n- `analysis_stream.ndjson` -- stream JSON output from the analysis agent.\n- `workflow_scripted_eval.log` / `workflow_scripted_results.csv` -- scripted workflow eval output.\n- `workflow_llm_eval.ndjson` / `workflow_llm_results.csv` -- LLM workflow eval output.\n- `quality_scripted_eval.log` / `quality_scripted_results.csv` -- scripted quality eval output.\n- `quality_llm_eval.ndjson` / `quality_llm_results.csv` -- LLM quality eval output.\n- `eval_summary.csv` -- merged results for the test case.\n\n### CSV Schema\n\nAll eval result CSVs use the same 7-column schema:\n\n```\nindex,category,issue_summary,result,details,root_cause,recommended_fix\n```\n\n| Column | Description |\n|--------|-------------|\n| `index` | Eval identifier (e.g., `workflow_eval_9_compute`, `quality_eval_2`) |\n| `category` | `Workflow` or `Quality` |\n| `issue_summary` | Human-readable name of the check |\n| `result` | `PASS` or `FAIL` |\n| `details` | Failure specifics, scoring breakdown, or match confirmation |\n| `root_cause` | `pipeline`, `template`, or `data` (empty if PASS) |\n| `recommended_fix` | Actionable fix suggestion (empty if PASS) |\n\n### Aggregated Results and Reports\n\nAfter a repeatability run, the post-processing skill generates reports in `\u003cREPORT_DIR>/`:\n\n- `aggregates/aggregated_results.csv` -- full eval results for every (trace, run) pair\n- `aggregates/pass_rate_summary.csv` -- per-trace, per-eval pass rates across all runs\n- `aggregates/stability_summary.csv` -- per-(trace, eval) stability classification (see below)\n- `aggregates/stream_diagnostics.csv` -- per-run metadata (outcome, token usage, last step reached, whether the report was written)\n- `pr_report.md` -- concise summary for posting as a PR comment\n- `fix_ticket_report.md` -- detailed failure analysis with root causes, suggested fixes, and reproducer commands\n- `reproducers/` -- per-issue reproducer packages (`.tar.gz` files with sanitized stream JSONs for debugging)\n\n### Stability Classification\n\nThe `stability_summary.csv` classifies each (trace, eval) pair across repeated runs:\n\n| Classification | Meaning | Action |\n|---|---|---|\n| `STABLE_PASS` | Passed 100% of runs | No action needed |\n| `FLAKY_PASS` | Passed >50% but not all | Investigate LLM variability or edge cases |\n| `FLAKY_FAIL` | Passed >0% but ≤50% | Likely a significant issue with intermittent masking |\n| `STABLE_FAIL` | Failed 100% of runs | Deterministic bug — fix the pipeline code |\n\n**What to look for:**\n\n- **Overall pass rate** should not regress compared to the baseline on `main`.\n- **STABLE_FAIL entries** in `stability_summary.csv` — these are deterministic bugs that need pipeline code fixes. Check the `root_cause` column to understand whether it's a `pipeline`, `template`, or `data` issue.\n- **FLAKY_PASS / FLAKY_FAIL entries** — indicate LLM scoring variability or edge-case inconsistency. Check if the LLM eval scores hover near the 7.0 threshold (visible in the `details` column scoring breakdown).\n- **Pre-check gate failures** — if many evals show `\"Pre-check gate: ...\"` in the details column, the analysis pipeline itself failed to produce output. Focus on fixing the pipeline before investigating individual eval failures.\n- **Stream diagnostics regressions** -- runs where the report was not written or the last step reached regressed (e.g., stuck at Step 7 instead of reaching Step 11) signal a reliability problem.\n- **Reproducer packages** -- assign the `.tar.gz` for a specific issue to a developer. They can extract it, load the stream JSON into Cursor, and debug the root cause.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":19259,"content_sha256":"86aafa4acbf14c2d9c49b94240f18f283116e356a7737ed7140c387be44ad0c6"},{"filename":"reference.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n# Analysis orchestrator — reference\n\nThis document is the detailed specification for the TraceLens **analysis-orchestrator** skill ([SKILL.md](SKILL.md)). Read it when executing the workflow: step-by-step user prompts, CLI commands, subagent contracts, validation, and report assembly.\n\n## Workflow overview\n\nThe orchestrator runs a staged pipeline (Steps 0–12): collect inputs and environment prefix, generate perf reports, prepare category data via `orchestrator_prepare.py`, run system-level and compute-kernel subagents in parallel, aggregate and validate findings, identify the model, render plots when no extension is present, and write `analysis.md` via remote `tee` heredocs. Only Steps 7, 8, and 10 delegate to Task subagents; all other steps run in the main agent.\n\n---\n\n## Language Guidelines\n\nUse vendor-agnostic terminology throughout such as GPU kernels, collective communication, vendor GEMM library, DNN primitives, GPU graph, etc. Focus on operation semantics, not vendor implementation details\n\n**Exception:** When quoting kernel names from traces, it's acceptable to include the actual name for identification.\n \n---\n\n## Workflow Steps\n\n```\n0. Query User Inputs (interactive answers or a run YAML, Environment Setup) → cache/run_config.yaml\n1. Generate Performance Report (one wrapper call on cache/run_config.yaml; semantic pre-processing first)\n2. Trace-Quality Gate (deterministic fallback branch for graph-under-recorded traces)\n3-6. Prepare Category Data (GPU Util, Top Ops, Tree Data, Multi-Kernel Data, Category Filtering)\n7. System-Level Analysis (PARALLEL, CPU/Idle + Kernel Fusion + Multi-Kernel) → system_findings/\n8. Invoke Compute Kernel Subagents (PARALLEL, read category_findings[] from _metrics.json) → category_findings/\n 8.5. Aggregate per-category category_findings[] → priority_data.json::findings[] (globally sorted)\n9. Validate Subagent Outputs (system_findings/ + category_findings/)\n10. Prepare Report Data (load_findings) + Model Identification (subagent) → metadata/model_info.json\n11. Render performance PNG IF agent_extension.py is absent\n12. Generate Final Report (composable System + Compute sections), validate it,\n optionally invoke agent_extension.py (when present), then embed the PNG into the report.\n```\n\n**Subagent usage:** Only invoke Task subagents in steps that explicitly say \"subagent\" (Step 1 pre-processing semantic diff, Steps 7, 8, 10). All other steps (including Step 8.5) must be performed directly by the orchestrator using the command prefix.\n\n---\n\n## Step 0: Query User Inputs\n\nThe prompt selects the input mode:\n\n- **YAML mode**: the prompt says \"from config `\u003cconfig_path>`\", optionally with a scope (`standalone`, `comparative`, `semantic` or `multi_trace`). Resolve `\u003cconfig_path>` to an absolute path. Read `output_dir` from the file and ask only the environment questions (item 5), unless the prompt answers them. Skip the other items.\n- **Interactive mode**: the prompt names no config.\n\n**In Interactive mode, when this skill is invoked, immediately ask the user for:**\n\n### Required Information:\n\n1. **Scope** → `\u003cscope>`\n\n | `\u003cscope>` | `\u003ccomparison_scope>` |\n |---|---|\n | `standalone` | standalone |\n | `multi_trace` | standalone |\n | `comparative` | comparative |\n | `semantic` | comparative |\n\n - Set from the user’s intent before asking for paths:\n - **`comparative`** if the user asks for a comparative analysis, to compare two traces, or supplies two trace paths.\n - **`multi_trace`** (multi-trace standalone) if the user supplies several named jobs of one workload on one platform (for example prefill and decode captures).\n - **`standalone`** otherwise, including a request for standalone analysis or a single trace.\n\n2. **Trace File Path(s)** (PyTorch trace, `.json` or `.json.gz`)\n - **`standalone`:** **Trace File Path** → `\u003ctrace_path>`\n - Ask: \"Please provide the full path to your trace file\"\n - **`comparative`:** ask for both:\n - **Primary trace (trace1)** → `\u003ctrace_path>`\n - **Comparison trace (trace2)** → `\u003ctrace2_path>`\n - Ask: \"Please provide the full path to your primary trace file and your comparison trace file\"\n - **`multi_trace`:** ask for each job (2 or more) a name, a phase and **Trace File Path**; jobs that share a phase are combined\n - Ask: \"Please provide a name, a phase and the full path to the trace file for each job, starting with the job that kernel fusion and system analysis should focus on\"\n\n3. **Platform** → `\u003cplatform>`\n **`standalone`, `multi_trace`**: Ask: \"Which platform are you analyzing?\" (`multi_trace`: one platform for all jobs)\n **`comparative`**: Ask: \"Which platform is baseline trace (trace1)?\"\n - Options:\n 1. **MI300X**\n 2. **MI325X**\n 3. **MI350X**\n 4. **MI355X**\n 5. **MI455X**\n **`comparative`:** Ask: \"Which platform is target trace (trace2)?\" Assign `\u003cplatform2>` (`\u003cplatform2>` does not need to be one of the platform options)\n\n4. **Analysis Mode** → `\u003canalysis_mode>`\n - If the user's prompt explicitly specifies an analysis mode or mentions inference/vLLM/SGLang/ATOM, use that. Otherwise, default to `default` without asking.\n - Options:\n 1. **Default (training and non-vLLM/SGLang/ATOM eager inference)** (`\u003canalysis_mode>` = `default`)\n 2. **Inference analysis (vLLM/SGLang/ATOM)** (`\u003canalysis_mode>` = `inference`)\n - If **Inference (vLLM/SGLang/ATOM)** is selected, ask **Execution Mode** → `\u003cinference_exec_mode>`:\n 1. **Eager mode** (`\u003cinference_exec_mode>` = `eager`) — only the trace file is needed\n 2. **Graph replay + capture** (`\u003cinference_exec_mode>` = `graph_capture`) — also requires a capture folder path\n \n Ask for the **Capture Folder Path(s)**:\n - `standalone`: one folder → `\u003ccapture_folder_path_1>`. Ask: \"Please provide the full path to the graph capture traces folder\"\n - `comparative`: one folder per trace → `\u003ccapture_folder_path_1>` (primary/trace1) and `\u003ccapture_folder_path_2>` (comparison/trace2). Ask: \"Please provide the graph capture traces folder for the primary trace and for the comparison trace.\"\n - `multi_trace`: one folder per job, optional. Ask: \"Please provide the graph capture traces folder for each job that has one.\"\n\n 3. **Graph replay only** (`\u003cinference_exec_mode>` = `graph_replay_only`) — graph replay with no capture folder available:\n - `standalone`, `multi_trace`: continue without a capture folder; the Step 2 gate routes to the deterministic fallback.\n - `comparative`: use scope `semantic`.\n\n5. **Environment Setup**\n - Ask: \"Are you running locally or on a cluster?\"\n - If **local**: No further environment questions,prefix is blank (commands run directly).\n - If **cluster**:\n - Ask \"Which node should we use?\" → `\u003cnode>`\n - Ask \"Are you working in a containerized environment (e.g. Docker)?\" → if yes, ask for container name → `\u003ccontainer>`\n - Ask \"Are you using a virtual environment?\" → if yes, ask for venv path → `\u003cvenv_path>`\n\n6. **Output Directory** (Optional)\n - Ask: \"Where should we save analysis results? (Press Enter for default: \u003ctrace_directory>/analysis_output)\"\n - Default: Same directory as trace file, in `analysis_output/` subdirectory\n\n7. **Extension File** (Optional) → `\u003cextension_file>`\n - Ask: \"Do you have a TraceLens extension file to apply? Press Enter to skip.\"\n - If provided, resolve to an absolute path and assign to `\u003cextension_file>`.\n - If skipped, set `\u003cextension_file>` to empty (no `extension_file` is set in the config).\n\n### Build and Cache Command Prefix\n\nAfter collecting inputs, build a command template. Create the directory with `mkdir -p \u003coutput_dir>/cache`.\n\nThe template uses `{CMD}` as a placeholder for the actual command.\n\n**Local:** When `\u003cvenv_path>` is set, use `source \u003cvenv_path>/bin/activate && cd \u003ctracelens_dir> && {CMD}`. When `\u003cvenv_path>` is empty, use `cd \u003ctracelens_dir> && {CMD}` if `\u003ctracelens_dir>` is known; otherwise run `{CMD}` directly.\n\n**Cluster:** Before building the prefix, locate the TraceLens project root on the remote environment.\n\nRun the following command (adjust for container if applicable):\n\n```bash\n# Without container:\nssh \u003cnode> \"find / -maxdepth 5 -type d -name 'TraceLens' 2>/dev/null | head -5\"\n\n# With container:\nssh \u003cnode> \"docker exec \u003ccontainer> bash -c 'find / -maxdepth 5 -type d -name TraceLens 2>/dev/null | head -5'\"\n```\n\nPick the result containing `Agent/` and strip the trailing `/TraceLens` to get `\u003ctracelens_dir>`.\n\nBuild the cluster prefix using this lookup:\n\n| Container | Venv | Template |\n|-----------|------|----------|\n| No | No | `ssh \u003cnode> \"cd \u003ctracelens_dir> && {CMD}\"` |\n| Yes | No | `ssh \u003cnode> \"docker exec \u003ccontainer> bash -c 'cd \u003ctracelens_dir> && {CMD}'\"` |\n| No | Yes | `ssh \u003cnode> \"bash -c 'source \u003cvenv_path>/bin/activate && cd \u003ctracelens_dir> && {CMD}'\"` |\n\nWrite the resolved template to `\u003coutput_dir>/cache/cmd_prefix.txt`. Then validate that TraceLens is importable in the target environment through the prefix:\n\n```bash\n\u003cprefix> python3 -c \"import TraceLens; print('PREFIX_OK')\"\n```\n\nIf this fails, the cause is either a bad prefix or a missing package:\n- Verify `\u003ctracelens_dir>` is the parent of TraceLens (not the repo root itself) and that the container/venv is accessible; if wrong, rebuild prefix and retry once. If it still fails, inform the user with `[DIAG:pipeline:PREFIX_FAIL]`.\n- If the environment is reachable but TraceLens is not installed, ask the user where to install it and if they require a new venv \u003cvenv_path>.\n ```bash\n # Install into the container / bare metal directly (no prefix rebuild needed):\n \u003cprefix> pip install git+https://github.com/AMD-AGI/TraceLens\n\n # Create a new Python venv (if requested)\n \u003cprefix> python3 -m venv \u003cvenv_path>\n\n # Install into the python venv:\n \u003cprefix> \u003cvenv_path>/bin/pip install git+https://github.com/AMD-AGI/TraceLens\n ```\nRebuild the prefix and re-validate after installing.\n\nDo NOT proceed to Step 1 until validation passes.\n\n### Command Execution Pattern\n\n**Before executing any command**, read `\u003coutput_dir>/cache/cmd_prefix.txt`. It contains a template with a `{CMD}` placeholder. Substitute `{CMD}` with the actual command. All commands below use `\u003cprefix>` to represent this resolved template.\n\n### Resolve the Run Config\n\nRun one `config_utils` command. It completes the config, writes `\u003coutput_dir>/cache/run_config.yaml` (the input to Step 1), and prints the change lines, then the variables for Steps 1-12 as `key=value` lines.\n\n**YAML mode:**\n\n```bash\n\u003cprefix> python3 -m TraceLens.Agent.Analysis.utils.config_utils check \\\n --config \u003cconfig_path> \\\n [--scope \u003cscope>]\n```\n\n**Interactive mode:**\n\n```bash\n\u003cprefix> python3 -m TraceLens.Agent.Analysis.utils.config_utils build \\\n --scope \u003cscope> \\\n --analysis_mode \u003canalysis_mode> \\\n --output_dir \u003coutput_dir> \\\n --job trace_path=\u003ctrace_path>,platform=\u003cplatform>[,capture_folder=\u003ccapture_folder_path_1>] \\\n [--job trace_path=\u003ctrace2_path>,platform=\u003cplatform2>[,capture_folder=\u003ccapture_folder_path_2>]] \\\n [--set extension_file=\u003cextension_file>]\n```\n\nPass the second `--job` for `comparative` and `semantic`, a `capture_folder` only for graph replay + capture, and `--set extension_file` only when an extension file was provided. For `multi_trace`, pass one `--job name=\u003cname>,phase=\u003cphase>,...` per job (2 or more, same platform), the focus job first; the variables then describe the focus job and include `focus_job`.\n\nIf it exits non-zero, the errors are on stderr. In YAML mode, show them and stop. In Interactive mode, re-ask the field an error names once and rebuild.\n\nOtherwise, show the change lines to the user and assign each `key=value` line to the variable `\u003ckey>`. Assign the variables once, from this run. Never run `check` on `\u003coutput_dir>/cache/run_config.yaml`: it is the wrapper's input only, and it no longer holds the original `platform2`.\n\n---\n\n## Step 1: Generate Performance Report\n\n### Pre-processing\n\nOnly when `\u003cscope>` = semantic; otherwise skip to Generate Report.\n\n1. **Semantic diff (subagent)** — launch a Task subagent that reads and follows the FULL instructions in `TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/semantic-comparison-agent.md`. Prompt context:\n\n```\nRead and follow the FULL instructions in:\n TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/semantic-comparison-agent.md\n\n**Execution Context:**\n- Trace A (primary/trace1): \u003ctrace_path> (platform \u003cplatform>)\n- Trace B (comparison/trace2): \u003ctrace2_path> (platform \u003cplatform2>)\n- Labels: name-a trace1, name-b trace2\n- Output directory: \u003coutput_dir>/semantic/\n- Command prefix: read \u003coutput_dir>/cache/cmd_prefix.txt — substitute {CMD}\n\nRun the full semantic comparison through \"Generate TraceDiff Output\" so that\n\u003coutput_dir>/semantic/tracediff_output/diff_stats.csv is produced. Return \"DONE\".\n```\n\n Verify `\u003coutput_dir>/semantic/tracediff_output/diff_stats.csv` exists before continuing. If it is missing, retry the subagent once; if it still fails, stop and report.\n\n2. **Record the diff stats**: point trace1's job at that file:\n\n ```bash\n \u003cprefix> python3 -m TraceLens.Agent.Analysis.utils.config_utils set-diff-stats \\\n --config \u003coutput_dir>/cache/run_config.yaml \\\n --path \u003coutput_dir>/semantic/tracediff_output/diff_stats.csv\n ```\n\n It rewrites `\u003coutput_dir>/cache/run_config.yaml` and prints one `added` line; show it. If it fails, stop and report.\n\n### Generate Report\n\nOne wrapper call runs every report the config describes, in order:\n\n```bash\n\u003cprefix> TraceLens_generate_perf_report_from_config --config \u003coutput_dir>/cache/run_config.yaml\n```\n\nThis call can run for several minutes, longer for comparative graph-capture runs. Run it with the longest command timeout your runner allows.\n\nIf it exits non-zero or times out, show the tail of its output, emit `[DIAG:pipeline:STEP1_FAIL]`, and stop.\n\n---\n\n## Step 2: Trace-Quality Gate\n\n\n`\u003cunified_perf_csv>` is under `\u003coutput_dir>/perf_report_csvs/` (`standalone`), `\u003coutput_dir>/perf_report_aggregated_csvs/` (`multi_trace`), or `\u003coutput_dir>/perf_report_trace1_csvs/` (`comparative`, primary trace), as `unified_perf_summary.csv`.\n\n```bash\n\u003cprefix> python3 -c \\\"\nfrom TraceLens.Agent.Analysis.utils import check_graph_replay_coverage\ncoverage = check_graph_replay_coverage(\u003cunified_perf_csv>)\nstatus = 'GRAPH_UNDER_RECORDED' if coverage.graph_under_recorded else 'OK'\nprint(f'STATUS={status}')\nprint(f'GRAPH_REPLAY_FRACTION={coverage.graph_replay_fraction}')\n\" \n```\n\u003cgraph_replay_fraction> refers to the returned GRAPH_REPLAY_FRACTION value.\n\n**`STATUS=OK`:** Proceed\n**`STATUS=GRAPH_UNDER_RECORDED`:**\nIf \u003cscope> = semantic, skip the rest of this step and proceed with Steps 3-6.\n\nOtherwise,\n\n1. Emit `[DIAG:trace_quality:GRAPH_UNDER_RECORDED]`: Deterministic fallback report\n2. Run the deterministic fallback writer on `\u003cunified_perf_csv>`:\n ```bash\n \u003cprefix> python3 -m TraceLens.Agent.Analysis.utils.deterministic_fallback \\\n --unified-perf-csv '\u003cunified_perf_csv>' \\\n --output-dir '\u003coutput_dir>' \\ \n --graph-replay-fraction \u003cgraph_replay_fraction>\n ```\n3. **Skip Steps 3-12.** The writer already produced `\u003coutput_dir>/analysis.md`\n\n---\n\n## Steps 3-6: Prepare Category Data\n\nExecute the TraceLens Agentic Mode orchestrator preparation script:\n\n```bash\n\u003cprefix> python3 \\\n TraceLens/Agent/Analysis/utils/orchestrator_prepare.py \\\n --trace-path \u003ctrace_path> \\\n --platform \u003cplatform> \\\n --output-dir \u003coutput_dir> \\\n --comparison-scope \u003ccomparison_scope> \\\n [--platform2 \u003cplatform2>] \\\n [--capture-folder \u003ccapture_folder>]\n```\n\nPass `--platform2` only for comparative and `--capture-folder` only if `config_utils` printed one. Use your longest command timeout.\n\nThis script performs:\n- **Step 3:** Assess GPU utilization (computation, idle, communication times)\n- **Step 4:** Identify top 10 operations by GPU time\n- **Step 5:** Pre-compute tree data for bottleneck operations (load trace ONCE)\n- **Step 5.5:** Pre-compute multi-kernel issue data (memcpy by direction, NCCL events, overlap metrics)\n- **Step 6:** Filter and export category-specific data\n\n**Outputs:**\n- `category_data/\u003ccategory>_ops.csv` - Filtered operations per category\n- `metadata/\u003ccategory>_metadata.json` - Platform specs, GPU utilization, config\n- `category_data/multi_kernel_data.json` - Memcpy/NCCL/overlap pre-computed data\n- `category_data/category_manifest.json` - Workflow metadata with categories (includes `tier` field: `system` or `compute_kernel`)\n- `system_findings/` - Directory for system-level analysis outputs\n- `category_findings/` - Directory for compute kernel analysis outputs\n\n---\n\n## Step 7: System-Level Analysis (PARALLEL)\n\nSystem-level analysis examines issues that affect the GPU pipeline as a whole -- idle time, memory transfer patterns, and communication/compute overlap. These are **not** about individual kernel efficiency.\n\n**Output directory:** `system_findings/`\n\n### 7.1 Read Manifest and Identify System-Level Subagents\n\n```bash\n\u003cprefix> python3 -c \\\"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.report_utils import load_manifest_categories\nload_manifest_categories(sys.argv[1])\n\\\" '\u003coutput_dir>'\"\n```\n\nThis prints `system_categories` and `compute_categories` lists. Use `system_categories` for Step 7 and `compute_categories` for Step 8.\n\n### 7.2 Launch System-Level Subagents in PARALLEL\n\nLaunch system-level sub-agents simultaneously using the Task tool. Do NOT wait between invocations.\n\n**System-Level Agent File Map:**\n\n**Base path:** `TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/`\n\n| Category | Agent file |\n|----------|-----------|\n| `cpu_idle` | `cpu-idle-analyzer.md` |\n| `multi_kernel` | `multi-kernel-analyzer.md` |\n| `kernel_fusion` | `kernel-fusion-analyzer.md` |\n\n**Invocation conditions:**\n- **CPU/Idle**: Read `category_data/category_manifest.json` and check `gpu_utilization.idle_time_percent`. On multi-trace standalone runs system analysis is scoped to the focus job, so apply this and the Multi-Kernel condition to `gpu_utilization.jobs[\u003cfocus_job>]` (what Step 7.1 prints). Only invoke the subagent if `idle_time_percent > 15`.\n- **Multi-Kernel**: `multi_kernel` category exists in manifest OR `gpu_util['exposed_comm_time_percent'] > 0` OR `gpu_util['exposed_memcpy_time_percent'] > 0`\n- **Kernel Fusion**: `kernel_fusion` category exists in manifest\n\n**Task prompt structure for each system-level subagent:**\n\nThe subagent reads its own agent file — the orchestrator does NOT read or paste agent file contents.\n\n```\nRead and follow the FULL instructions in:\n TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/\u003cagent-file>.md\n\n**Execution Context:**\n- Comparison scope: `\u003ccomparison_scope>`\n- Output directory: \u003coutput_dir>\n- Command prefix: read `\u003coutput_dir>/cache/cmd_prefix.txt` — contains a template\n with `{CMD}` placeholder; substitute `{CMD}` with the actual command\n- Input files: \u003clist from agent file's \"Input files\" section>\n- Output file: \u003cfrom agent file's \"Output file\" section>\n\nExecute every step in the agent file. Return \"DONE\" when complete.\n```\n\n**CRITICAL:** The orchestrator does NOT read agent files or run analysis scripts. Each sub-agent is responsible for:\n1. Reading its own agent `.md` file\n2. Running its Python script using the command prefix\n3. Reading the metrics JSON output\n4. Identifying issues and generating findings\n\n### 7.3 Wait for System-Level Subagents to Complete\n\nThe three subagents must complete before proceeding to Step 7.4.\nEach writes findings to `system_findings/\u003cname>_findings.md`.\n\n### 7.4 Verify System Outputs and Retry Failures (up to 1 retry per subagent)\n\nAfter all system-level subagents complete:\n\n1. For each expected system category from the manifest, check:\n - Does `system_findings/\u003ccategory>_findings.md` exist?\n - If it exists, does it contain \"Status: ERROR\"?\n2. Collect a list of **failed** categories (missing file OR Status: ERROR).\n3. **Retry each failed category exactly once** by re-launching its subagent with the same prompt from Step 7.2. Wait for all retries to complete before proceeding.\n4. After retries, re-check outputs. Any category that still fails is excluded from aggregation.\n5. **CRITICAL: Do NOT attempt manual analysis of failed system checks — only automated subagent retry is allowed.**\n\n---\n\n## Step 8: Invoke Compute Kernel Subagents (PARALLEL)\n\nCompute kernel analysis examines individual operation category efficiency.\n\n**Output directory:** `category_findings/`\n\n### 8.1 Read Manifest and Identify Compute Kernel Categories\n\nUse `compute_categories` from the `load_manifest_categories()` call in Step 7.1.\n\n### 8.2 Launch Compute Kernel Subagents in PARALLEL\n\nFor each entry in `compute_categories` (loaded in Step 7.1), resolve `{agent_file}` as `{entry.skill}.md` and launch a subagent with agent file `TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/{agent_file}`. Fall back to `generic-op-analyzer.md` if the file is absent.\n\nLaunch all subagents simultaneously in a single parallel batch.\n\n---\n\n#### Shared Compute Kernel Preamble\n\nInclude this block in every compute kernel subagent prompt:\n\n\u003cShared Compute Kernel Preamble>:\n```\ncomparison_scope: {comparison_scope}\n\n**CRITICAL - READ FIRST:**\n- Use GPU kernel time (not CPU duration) for all bottleneck analysis\n- `efficiency_percent` semantics differ by mode:\n - **Standalone:** % of roofline. Flag > 100% as \"[ANOMALY] - verify measurement\".\n - **Comparative:** `100 × (Trace 2 kernel time) / (Trace 1 kernel time)`.\n - **\u003c 100%** → Trace 1 is slower than Trace 2. **This is an optimization opportunity — flag it.**\n - **> 100%** → Trace 2 is slower than Trace 1. **NOT an anomaly; no Trace-1 optimization needed.**\n\n**CRITICAL CONSTRAINTS:**\n1. **Standalone:** Any efficiency > 100% → `[ANOMALY] - verify measurement`. **Comparative:** efficiency > 100% means Trace 2 is slower — NOT an anomaly; efficiency \u003c 100% means Trace 1 is slower — flag as optimization opportunity.\n2. Status must be SUCCESS or ERROR; times in ms; efficiencies as percentages\n3. Operations with `fusion_flagged: true` in the metrics JSON are already covered by\n a high-confidence kernel fusion candidate — do NOT flag them as bottlenecks or write\n kernel_tuning recommendations. The analysis scripts already exclude them from `impact_estimates`.\n\n**Execution Context:**\n- Output directory: \u003coutput_dir>\n- Command prefix: read `\u003coutput_dir>/cache/cmd_prefix.txt` — contains a template\n with `{CMD}` placeholder; substitute `{CMD}` with the actual command\n```\n\n---\n\n#### Compute Kernel Subagent Prompt\n\nFor each category, launch a Task (subagent_type: generalPurpose):\n\n```\nYou are analyzing {category} operations for a PyTorch trace on {platform}.\n\n\u003cShared Compute Kernel Preamble>\n\nRead and follow the FULL instructions in:\n TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/{agent_file}\n\n- Category: {category}\n- Input files: category_data/{category}_ops.csv, metadata/{category}_metadata.json,\n category_data/{category}_metrics.json (P-items come from `category_findings[]`; `operations[i].module_chain` provides model layer context)\n- Output file: category_findings/{category}_findings.md\n\nExecute every step in the agent file. Return \"DONE\" when complete.\n```\n\n### 8.3 Wait for All Compute Kernel Subagents to Complete\n\nAll subagents must complete before proceeding to Step 8.4.\nEach subagent writes its findings to `category_findings/\u003ccategory>_findings.md`.\n\n### 8.4 Verify Outputs and Retry Failures (up to 1 retry per subagent)\n\nAfter all compute kernel subagents complete:\n\n1. For each category in the manifest with `tier: compute_kernel`, check:\n - Does `category_findings/\u003ccategory>_findings.md` exist?\n - If it exists, does it contain \"Status: ERROR\"?\n2. Collect a list of **failed** categories (missing file OR Status: ERROR).\n3. **Retry each failed category exactly once** by re-launching its subagent with the same prompt structure from Step 8.2. Launch all retries in parallel and wait for completion.\n4. After retries, re-check outputs. Any category that still fails is excluded from aggregation and recommendations.\n5. **CRITICAL: Do NOT attempt to manually analyze failed categories — only automated subagent retry is allowed.**\n\n### 8.5 Aggregate findings → priority_data.json\n\nAfter all compute sub-agent `_metrics.json` files exist (each carrying its own `category_findings[]`), concatenate them into a globally-sorted `priority_data.json::findings[]` for the report template.\n\n```bash\n\u003cprefix> python3 -c \\\"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.report_utils import generate_priority_data\ngenerate_priority_data(sys.argv[1])\n\\\" '\u003coutput_dir>'\n```\n\n---\n\n## Step 9: Validate Subagent Outputs\n\nBefore aggregating results, validate outputs from **both** tiers (system_findings/ and category_findings/).\n\n```bash\n\u003cprefix> python3 -c \\\"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.validation_utils import validate_subagent_outputs\nvalidate_subagent_outputs(sys.argv[1])\n\\\" '\u003coutput_dir>'\"\n```\n\nThis runs four checks:\n1. **Time Sanity** -- category GPU kernel time sum vs computation time (WARN if >15% discrepancy)\n2. **Efficiency Anomalies** -- findings with efficiency >100% (measurement issues) when `\u003ccomparison_scope>` = `standalone`\n3. **Coverage** -- all expected system and compute findings present\n4. **Priority Consistency** -- `priority_data.json` invariants: `findings[]` sorted desc by `impact_score`, contiguous `global_rank` / `priorities[].rank`, and per-category `priorities[].impact_score` ≈ `sum(findings[].impact_score)`\n\n---\n\n## Step 10: Prepare Report Data + Model Identification\n\n```bash\n\u003cprefix> python3 -c \\\"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.report_utils import load_findings\nload_findings(sys.argv[1])\n\\\" '\u003coutput_dir>'\"\n```\n\n### 10.1 Model Identification (Subagent, retry once on failure)\n\nLaunch a Task subagent (generalPurpose) that reads and follows `TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/model-identification-agent.md` with context: \u003coutput_dir>. Wait for completion.\n\n**On failure (subagent error, timeout, or `model_info.json` not written):**\n1. **Retry exactly once** by re-launching the same subagent with the same prompt.\n2. If the retry also fails, write fallback `metadata/model_info.json` with all four fields set to `\"Cannot be inferred from trace\"`.\n\nAssign \u003cModel> to model value in `\u003coutput_dir>/metadata/model_info.json` or \"Workload\" if model is \"Cannot be inferred from trace\".\n\n---\n\n## Step 11: Render Plot (conditional)\n\n**Important:** Plot data is sourced from `priority_data.json` (written in Step 8.5). This step only renders the PNG when `agent_extension.py` is absent.\nLook at the environment variable TL_EXTENSION to find python packages and directories to recursively search for `agent_extension.py`.\nIf this environment variable is not present or the it is not found look in TraceLens/Agent/Analysis/utils/.\nIf the file is present, **skip this step** — Step 12.2 will produce `perf_improvement.png` and Step 12.3 will embed it.\nUse \u003cagent_extension_file> to represent the location of this file.\n\n```bash\nEXT='\u003cagent_extension_file>'\nif [ ! -f \"$EXT\" ]; then\n \u003cprefix> python3 -c \\\"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.plot_utils import generate_perf_plot\ngenerate_perf_plot(sys.argv[1], sys.argv[2])\n\\\" '\u003coutput_dir>' '\u003cModel> on \u003cPlatform> — Performance Breakdown'\nfi\n```\n\nIf the plot fails (extension-absent branch), retry once. If still failing, proceed to Step 12 without the plot.\n\n---\n\n## Step 12: Generate Final Report (\u003coutput_dir>/analysis.md)\n\n**CRITICAL: Do NOT delegate Step 12 to a Task subagent.** The orchestrator must write the report directly.\n\n1. **Read** the report template: `TraceLens/Agent/Analysis/skills/analysis-orchestrator/templates/analysis_template.md`\n2. **Write the report in sections** to `\u003coutput_dir>/analysis.md` using **only** `\u003cprefix> tee` / `\u003cprefix> tee -a` with single-quoted heredoc delimiters (see write order below). You MUST NOT use the IDE Write tool, Edit tool, StrReplace tool, `cat >`, `echo >`, `>>` redirect, or any other write method for `analysis.md` unless tee fails.\n3. **Fill in** each section by substituting placeholders with actual data. Never retain template placeholders (`\u003cBrief Title>`, `X ms`, `Y%`, `\u003cplatform>`, `\u003cmodel>`) — every field must contain actual data.\n\n**Write order (one heredoc per step):**\n\n a. **Initialize** — truncate and write the title line, the `report_mode` + any `kind=warning` markers (see the template title block and the Warnings rule below), then `## Executive Summary` (metrics table, `{{PERF_PLOT}}` placeholder). Use `\u003cprefix> tee \u003coutput_dir>/analysis.md \u003c\u003c 'SECTION_EOF'` (truncating `tee`, not append) for this first write only.\n - Data sources: `category_data/category_manifest.json` (`gpu_utilization` keys), `priority_data.json` (top bottleneck).\n - `multi_trace`: Total Time is the combined time; add one sentence giving each phase's time, summed over its jobs in `gpu_utilization.jobs`.\n\n b. **Compute Kernel Optimizations** — append `## Compute Kernel Optimizations` with `### Top Operations` table and P-item cards. Use `\u003cprefix> tee -a \u003coutput_dir>/analysis.md \u003c\u003c 'SECTION_EOF'`.\n - Data sources: `priority_data.json` — P1 = `findings[0]`, P2 = `findings[1]`, ... ; each card joins its sub-agent's Detailed Analysis block by `(findings[i].category, findings[i].category_rank)`. The Top Operations table materializes `priorities[]` verbatim (one row per entry, array order, no re-sorting).\n - `category_findings/*.md` — for each findings file, copy its `## Recommendations` P-items into the report card slots. **Copy table cells verbatim** from the source `category_findings/\u003ccat>_findings.md`.\n - Heuristic findings (`findings[i].estimate_method == \"heuristic\"`) carry a numeric estimated impact and sort by `impact_score` like any other compute finding — render them (do NOT skip them) per `sub_agent_spec.md § Heuristic findings`.\n\n c. **Kernel Fusion** — append `## Kernel Fusion Opportunities (Experimental)`. Use `\u003cprefix> tee -a \u003coutput_dir>/analysis.md \u003c\u003c 'SECTION_EOF'`.\n - Data source: `system_findings/kernel_fusion_findings.md`.\n - `multi_trace`: add the template's scope line.\n\n d. **System-Level** — append `## System-Level Optimizations`. Use `\u003cprefix> tee -a \u003coutput_dir>/analysis.md \u003c\u003c 'SECTION_EOF'`.\n - Data sources: remaining `system_findings/*.md` (cpu_idle, multi_kernel).\n - `multi_trace`: add the template's scope line.\n\n e. **Detailed Analysis** — append `## Detailed Analysis` with `### Compute Kernel Insights`, `### Kernel Fusion Insights`, `### System-Level Insights` subsections. Use `\u003cprefix> tee -a \u003coutput_dir>/analysis.md \u003c\u003c 'SECTION_EOF'`.\n - Data sources: copy the `## Detailed Analysis` blocks verbatim from each `*_findings.md` file. Follow the template for formatting.\n - `category_data/*_metrics.json` (per-op tables, impact estimates).\n\n f. **Appendix** — append `## Appendix` with `### Model Architecture` and `### Hardware Reference`. Use `\u003cprefix> tee -a \u003coutput_dir>/analysis.md \u003c\u003c 'SECTION_EOF'`.\n - `metadata/model_info.json` — substitute `\u003cmodel>`, `\u003carchitecture>`, `\u003cscale>`, `\u003cprecision>` with the four field values.\n - Platform arch file — read `platform` from `category_manifest.json`, then read `TraceLens/Agent/Analysis/utils/arch/\u003cplatform>.json`. For `### Hardware Reference`: substitute `\u003cplatform>`, Peak Memory BW = `mem_bw_gbps / 1000` TB/s, Peak MAF (BF16) = `max_achievable_tflops.matrix_bf16` TFLOPS, Peak MAF (FP8) = `max_achievable_tflops.matrix_fp8` TFLOPS if present.\n\n**Failure exclusion:** Skip any category listed in `load_findings()` output as `failed_system` or `failed_compute`; emit one top `kind=warning` marker per failure and per `high_variance` operation (CoV > 1.0 in `*_metrics.json`) instead. No mid-document `## Warnings` section — validation rejects it.\n\nThe report at `\u003coutput_dir>/analysis.md` must use these exact `##` headers — do NOT rename them:\n1. `## Executive Summary`\n2. `## Compute Kernel Optimizations`\n3. `## Kernel Fusion Opportunities (Experimental)`\n4. `## System-Level Optimizations`\n5. `## Detailed Analysis`\n6. `## Appendix`\n\n### 12.1 Validate Report Structure (Retry up to 2x)\n\nAfter writing `analysis.md`, validate that the report contains all required `##` section headers. If validation fails, modify the report with the missing sections.\n\n**Validation procedure:**\n\n```bash\n\u003cprefix> python3 -c \\\"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.validation_utils import validate_report\npassed, missing = validate_report(sys.argv[1], comparison_scope=sys.argv[2])\nif not passed:\n print('FAIL:')\n for m in missing:\n print(' - ' + m)\n sys.exit(1)\nprint('PASS: All required sections present')\n\\\" '\u003coutput_dir>' '\u003ccomparison_scope>'\n```\n\n**If validation fails (exit code 1):**\n\n1. Read the FAIL output to identify the issue. Fix in-place, do NOT rewrite the report from scratch. Edit sections in place and not regenerate the entire output.\na. Check if the report contains similar but incorrectly named headers and rename them to match the exact required names. \nb. If sections are entirely absent, add them with the correct `##` headers, keeping existing content.\nc. For \"Missing metrics row\" errors: add the row to the Executive Summary table using values from `category_data/category_manifest.json` (`gpu_utilization` keys) and `priority_data.json` (top bottleneck).\nd. For placeholder values (`X ms`, `Y%`, `Z%`, `W%`) in the Executive Summary metrics table: replace each with the actual value from `category_manifest.json` -> `gpu_utilization`.\ne. For unfilled `\u003cBrief Title>` / `\u003cLibrary>` / `\u003cplatform>` placeholders: substitute the real title/backend/platform from the corresponding findings file or `metadata/*_metadata.json`.\nf. For Args cell mismatches: copy the matching `operations[].args` value verbatim (preserving `\u003cbr>`) from the corresponding `category_data/\u003ccat>_metrics.json` and string-replace the bad cell.\ng. For marker errors: restore or add the missing/broken marker in place — never delete a card or block to silence an error. Source numeric values from `priority_data.json` (P-items) or `\u003ccat>_metrics.json::impact_estimates[]` (detail estimates); use `null` or the sentinel `not quantifiable from trace data` for non-quantifiable items.\nh. For priority-consistency errors (R1 P-item count mismatch, R2 P-item category-order mismatch, R3 marker numeric mismatch, R4 Top Ops row-count mismatch, R6 phase mismatch): re-render the affected card(s) by re-reading `priority_data.json::findings[N-1]` for `category`, `phase`, `low` (impact_score_low), `mid` (impact_score), `high` (impact_score_high), and `priorities[]` for the Top Operations table rows (one row per entry, in array order).\n2. Run validation again.\n3. Maximum 2 retry attempts. If still failing after retry, proceed with a warning.\n\n---\n\n### 12.2 Optional extension (auto-detected)\n\nIf `\u003cagent_extension_file>` exists, run it as shown below. Its behavior is documented in the extension itself; the orchestrator does not need to inspect or reason about it.\n\nIf the file is absent, skip this step silently. The analysis is complete; the simple plot from Step 11 stays in place.\n\n```bash\nEXT='\u003cagent_extension_file>'\nif [ -f \"$EXT\" ]; then\n \u003cprefix> python3 \"$EXT\" --output-dir '\u003coutput_dir>' --title '\u003cModel> on \u003cPlatform> — Kernel Tuning Potential' --comparison-scope \u003ccomparison_scope>\nfi\n```\n\nThis step is a hook for an optional extension; if `agent_extension.py` is not present, skip it.\n\n**Do NOT re-run `validate_report` after this step.**\n\n---\n\n### 12.3 Embed Performance Improvement Plot\n\nThe PNG (`perf_improvement.png`) is already on disk from either Step 11 or Step 12.2 (whichever ran). This step only embeds its base64 sidecar into the report at the `{{PERF_PLOT}}` placeholder.\n\n```bash\n\u003cprefix> python3 -c \\\"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.plot_utils import embed_plot_in_report\nembed_plot_in_report(sys.argv[1])\n\\\" '\u003coutput_dir>'\n```\n\nIf the plot is skipped, the `{{PERF_PLOT}}` placeholder is removed so the report remains clean.","content_type":"text/markdown; charset=utf-8","language":"markdown","size":36459,"content_sha256":"2c879a09380c675b739207351c841936d0844128b50c7d1588ac08889bb56e30"},{"filename":"skill-card.md","content":"# Skill Card\n\n## Description\n\nOrchestrates modular PyTorch profiler trace analysis with TraceLens: generates perf reports, prepares category data, runs system-level and compute-kernel subagents in parallel, validates outputs, and writes a prioritized stakeholder report (analysis.md).\n\n## Owner\n\nAMD-AGI (federated from [AMD-AGI/TraceLens](https://github.com/AMD-AGI/TraceLens))\n\n## License\n\nMIT\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":396,"content_sha256":"4fffef8a6e61d49990b3183f541cd43060153a0446ac9bdbb876a8234aece7f2"},{"filename":"templates/analysis_template.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n\u003c!--\n=== FORMATTING RULES (for the agent filling in this template) ===\n\n=== MODE SELECTION ===\nThis template supports two modes determined by `comparison_scope`:\n - **standalone**: Single-trace roofline analysis (default). Use sections marked STANDALONE.\n - **comparative**: Two-trace analysis (Trace 1 = primary, Trace 2 = target). Use sections marked COMPARATIVE.\nWhen filling in this template, select the block matching the active `comparison_scope` for each\nsection that has STANDALONE / COMPARATIVE variants. Delete the unused variant.\n\n=== COMPARATIVE TERMINOLOGY ===\n - **Trace 1** = trace (primary). **Trace 2** = trace (target/comparison).\n - Impact semantics: standalone uses roofline gap; comparative uses\n trace 2 kernel time as the optimization target (gap = trace1 time − trace2 time).\n - Comparative speed semantics: express as \"X% faster\" or \"X% slower\" relative to Trace 1. If t2 \u003c t1 → \"X% faster\"; if t2 > t1 → \"X% slower\".\n Standalone efficiency semantics: % of roofline (unchanged).\n\n=== GENERAL RULES ===\n1. Report markers: emit the `report_mode` marker + one `kind=warning` marker per subagent failure and per `high_variance: true` operation at the top\n2. Executive Summary: Max ~20 lines.\n3. Performance plot: The {{PERF_PLOT}} placeholder is replaced by Step 12.3 with a base64-embedded\n PNG data URI (![Performance Breakdown](data:image/png;base64,...)) of a single horizontal stacked\n bar showing the run's compute-time breakdown by kernel category. The plot is purely descriptive\n (no error bars, no throughput cone, no savings estimates). If the plot was not generated\n (Step 11 / Step 12.2 failed), the placeholder is removed.\n4. Compute Kernel Optimizations: One P-item per entry in `priority_data.json::findings[]`,\n numbered P1, P2, ... in `findings[]` order (already globally sorted by `impact_score`). The\n P-item Impact line uses the canonical mid `impact_score` value; low/high values appear only\n in Detailed Analysis. Cards join the corresponding sub-agent's Detailed Analysis block by\n `(findings[i].category, findings[i].category_rank)`.\n5. System-Level Optimizations: If all system-level analyses report no actionable issues\n (NONE/N/A severity), use a single \"✅ No system-level bottlenecks detected\" summary instead of\n P1/P2/P3 recommendations. Only generate numbered priorities when at least one actionable issue\n exists (number sequentially from P1, including CPU/Idle first if invoked).\n6. Each section is independently composable -- can be shared standalone.\n7. All three tiers (Compute, Kernel Fusion, System) use separate sequential P1/P2/P3 numbering (no gaps).\n8. Priority icons are assigned by PRIORITY NUMBER, not severity, in every section (Compute Kernel, Kernel Fusion, System-Level): 🔴 P1 → 🟡 P2 → 🟢 P3 → 🟢 P4 ... (System-Level only when actionable issues exist)\n9. Field labels — each section uses EXACTLY these labels:\n\n OPTIMIZATION CARDS (§Compute Kernel Optimizations, §Kernel Fusion, §System-Level):\n - Compute Kernel P-items: **Insight** / **Action** / **Impact**\n - Kernel Fusion P-items: **Insight** / **Action** / **Impact** / **Confidence**\n - System-Level P-items: **Insight** / **Action** / **Impact**\n\n DETAILED ANALYSIS (§Detailed Analysis only):\n - Compute / System blocks: **Identification:** / **Data:** / **Reasoning for Slowdown:** / **Resolution:** / **Impact estimate:**\n - Kernel Fusion blocks: **Identification:** / **Data:** / **Resolution:** / **Impact estimate:**\n\n10. Detailed Analysis: three subsections (`### Compute Kernel Insights`, `### Kernel Fusion Insights`, `### System-Level Insights`) with `#### 🔴/🟡/🟢 Pn: \u003cBrief Title>` blocks matching card titles and order.\n11. Model and appendix: Use `model_info[\"model\"]` from `metadata/model_info.json` for the\n report title (fall back to \"Workload\" if \"Cannot be inferred from trace\"). Fill Appendix\n **Model Architecture** with the raw `model`, `architecture`, `scale`, `precision` values.\n12. Library parenthetical: Compute Kernel card titles and Detailed Analysis headings must include\n the library name(s) in parentheses when present in the sub-agent findings. Omit when no\n library is identified. System-Level and Kernel Fusion titles do NOT include a library\n parenthetical.\n13. Multi-trace standalone (`multi_trace` scope): use the STANDALONE\n blocks. Impact scores are a percent of the combined time of all jobs. Compute P-items carry their\n phase. Fusion and System sections cover the focus job only and open with a scope line.\n-->\n\n\u003c!-- === STANDALONE title === -->\n# \u003cModel> - \u003cPlatform> Standalone Analysis\n\u003c!-- report-begin kind=report_mode mode=agentic -->\n\u003c!-- report-end -->\n\n\u003c!-- === COMPARATIVE title === -->\n# \u003cModel> - Comparative Analysis: \u003cPlatform1> vs \u003cPlatform2>\n\u003c!-- report-begin kind=report_mode mode=comparative -->\n\u003c!-- report-end -->\n\n\u003c!-- Warning markers (one per flagged item). Omit entirely if nothing flagged. -->\n\n\u003c!-- report-begin kind=warning -->\n> **⚠ Analysis incomplete.** Excluded from recommendations (script failures):\n> - \u003cname> (\u003ctier>) — \u003cbrief error>\n\u003c!-- report-end -->\n\n\u003c!-- report-begin kind=warning -->\n> **⚠ Data Quality.** Unreliable kernel-time (CoV > 1.0, extreme variance across instances):\n> - \u003cname> (\u003ccategory>) — CoV X.X\n\u003c!-- report-end -->\n\n## Executive Summary\n\n\u003c!-- === STANDALONE Executive Summary === -->\n[1 paragraph overview + key metrics table. multi_trace: Total Time is the combined time; add one sentence giving each phase's time, summed over its jobs in `gpu_utilization.jobs`.]\n\n\u003c!-- MANDATORY: This table must contain exactly these 5 rows:\n Total Time | Compute % | Idle % | Exposed Communication % | Top Bottleneck Category\n Top Bottleneck Category V% = gpu_kernel_time_ms of top category / (gpu_utilization.total_time_ms * computation_time_percent / 100) -->\n| Metric | Value |\n|--------|-------|\n| Total Time | X ms |\n| Compute % | Y% |\n| Idle % | Z% |\n| Exposed Communication % | W% |\n| Top Bottleneck Category | Category (V%) |\n\n\u003c!-- === COMPARATIVE Executive Summary === -->\n[1 paragraph comparative overview: summarize which trace is faster overall, by how much, and the dominant gap categories]\n\n\u003c!-- Top Bottleneck Category X% = top category's gpu_kernel_time_ms / (manifest.gpu_utilization.total_time_ms * manifest.gpu_utilization.computation_time_percent / 100)\n Top Bottleneck Category Y% = top category's gpu_kernel_time_ms / (manifest.trace2_gpu_utilization.total_time_ms * manifest.trace2_gpu_utilization.computation_time_percent / 100)\n Difference = Trace 2 value − Trace 1 value -->\n| Metric | Trace 1 - (\u003cPlatform1>) | Trace 2 - (\u003cPlatform2>) | Difference |\n|--------|----------------------------|-------------------------------|------------|\n| Total Time | X ms | Y ms | +/-Z ms (+/-W%) |\n| Compute % | X% | Y% | +/-Z% |\n| Idle % | X% | Y% | +/-Z% |\n| Exposed Communication % | X% | Y% | +/-Z% |\n| Top Bottleneck Category | Category (X%) | Category (Y%) | — |\n\n{{PERF_PLOT}}\n\n## Compute Kernel Optimizations\n\nFindings from per-category kernel analysis (GEMM, SDPA, elementwise, etc.).\nSummaries of recommendations from Step 8 sub-agents, focused on individual kernel efficiency.\n\n### Top Operations\n\nOne row per entry in `priority_data.json::priorities[]`, in array order (no manifest-sort, no extra rows). For row N (= `priorities[N-1]`): `Rank`/`Category` = `rank`/`display_name`; `Time (ms)` = matching `manifest.categories[].gpu_kernel_time_ms` (verbatim); `Ops` = matching `manifest.categories[].ops_count`; `% of Compute Time` = `Time (ms) / (gpu_utilization.total_time_ms * computation_time_percent / 100)`; trailer `low`/`high` = `priorities[N-1].impact_score_low`/`impact_score_high` (use `null` for `source: \"manifest_fallback\"`). Wrap the whole block (header + separator + rows) in the `kind=top_ops` marker.\n\n\u003c!-- === STANDALONE Top Operations === -->\n\u003c!-- === MULTI_TRACE Top Operations: add this line above the standalone table, outside the marker. Phase names are the distinct `phase` values in gpu_utilization.jobs === -->\n_Times and percentages are summed across all phases (\u003cphase_1> + \u003cphase_2>). See each P-item's Phase field for where the time is spent._\n\n\u003c!-- impact-begin kind=top_ops -->\n| Rank | Category | Time (ms) | % of Compute Time | Ops |\n|------|----------|-----------|-------------------|-----|\n| 1 | ... | ... | ... | ... | \u003c!-- top-ops-row low=\u003cimpact_score_low> high=\u003cimpact_score_high> -->\n\u003c!-- impact-end -->\n\n\u003c!-- === COMPARATIVE Top Operations === -->\n`Trace 2 Time (ms)` = matching `manifest.trace2_ops_summary_by_category[][\"total_direct_kernel_time_ms\"]` where `\"op category\"` matches the row Category **case-insensitively**; use — if no match.\n`Difference (ms)` = Trace 2 Time − Trace 1 Time.\n\u003c!-- impact-begin kind=top_ops -->\n| Rank | Category | Trace 1 Time (ms) | Trace 2 Time (ms) | % of Compute Time | Ops | Difference (ms) |\n|------|----------|-------------------|-------------------|-------------------|-----|-----------------|\n| 1 | ... | ... | ... | ... | ... | +/-X.X or — | \u003c!-- top-ops-row low=\u003cimpact_score_low> high=\u003cimpact_score_high> -->\n\u003c!-- impact-end -->\n\n\u003c!-- === NO ACTIONABLE FINDINGS (all quantified compute categories have empty category_findings[] in *_metrics.json) === -->\n\u003c!-- Use when priority_data / per-category metrics show no compute P-items to render (category_findings[] empty everywhere that applies). -->\n✅ No compute kernel optimization opportunities identified. All categories are within target performance bounds.\n\n\u003c!-- === ACTIONABLE FINDINGS (at least one compute category has P-items) === -->\n\u003c!-- Icon mapping by PRIORITY NUMBER (not severity): P1=🔴, P2=🟡, P3+=🟢 -->\n\u003c!-- One card per entry in priority_data.findings[] in array order. Title uses the entry's category and library; Action text is category-appropriate. Do NOT recommend \"fuse the SDPA kernel\" (already fused — defer upstream/downstream fusion to Kernel Fusion section). -->\n\u003c!-- Skip categories that have empty category_findings[] in category_data/\u003ccat>_metrics.json (no P-items for that category). -->\n\u003c!-- The **Phase** line and `phase=` appear only when the finding has a phase. -->\n\u003c!-- Heuristic findings (priority_data.findings[i].estimate_method == \"heuristic\") carry a numeric estimated impact and sort by impact_score like any other compute finding; render per sub_agent_spec.md § Heuristic findings. -->\n\n### 🔴 P1: \u003cBrief Title> (\u003cLibrary>)\n\n**Phase**: \u003cpriority_data.findings[0].phase>\n\n**Insight**: [1 sentence - what's wrong]\n\n**Action**: [1-2 sentences - category-appropriate: GEMM fusion/tile/library; SDPA tile/backend; elementwise fusion; etc.]\n\n\u003c!-- impact-begin kind=p_item category=\u003cpriority_data.findings[0].category> low=\u003cpriority_data.findings[0].impact_score_low> mid=\u003cpriority_data.findings[0].impact_score> high=\u003cpriority_data.findings[0].impact_score_high> phase=\u003cpriority_data.findings[0].phase> -->\n**Impact**: [impact_score: X.X, OR \"Not quantifiable from trace data\"]\n\u003c!-- impact-end -->\n\n→ *See [Detailed Analysis: Compute kernel insights > P1](#detailed-analysis-compute-p1) for details*\n\n---\n\n### 🟡 P2: \u003cBrief Title> (\u003cLibrary>)\n\n**Insight**: [1 sentence]\n\n**Action**: [1-2 sentences]\n\n\u003c!-- impact-begin kind=p_item category=\u003cpriority_data.findings[1].category> low=\u003cpriority_data.findings[1].impact_score_low> mid=\u003cpriority_data.findings[1].impact_score> high=\u003cpriority_data.findings[1].impact_score_high> phase=\u003cpriority_data.findings[1].phase> -->\n**Impact**: [impact_score: X.X, OR \"Not quantifiable from trace data\"]\n\u003c!-- impact-end -->\n\n→ *See [Detailed Analysis: Compute kernel insights > P2](#detailed-analysis-compute-p2) for details*\n\n---\n\n### 🟢 P3: \u003cBrief Title> (\u003cLibrary>)\n\n**Insight**: [1 sentence]\n\n**Action**: [1-2 sentences]\n\n\u003c!-- impact-begin kind=p_item category=\u003cpriority_data.findings[2].category> low=\u003cpriority_data.findings[2].impact_score_low> mid=\u003cpriority_data.findings[2].impact_score> high=\u003cpriority_data.findings[2].impact_score_high> phase=\u003cpriority_data.findings[2].phase> -->\n**Impact**: [impact_score: X.X, OR \"Not quantifiable from trace data\"]\n\u003c!-- impact-end -->\n\n→ *See [Detailed Analysis: Compute kernel insights > P3](#detailed-analysis-compute-p3) for details*\n\n\u003c!-- All additional P-items (P4, P5, ...) follow the same pattern, sourcing markers from priority_data.findings[N-1]. Detailed Analysis links: → *See [Detailed Analysis: Compute kernel insights > PN](#detailed-analysis-compute-pN) for details* -->\n\n---\n\n## Kernel Fusion Opportunities (Experimental)\n\u003c!-- === STANDALONE Kernel Fusion === -->\n> **Note:** Kernel fusion analysis is experimental. impact_score projections estimate the recoverable fraction of E2E with \u003ckernel_fusion_metrics.overlap_efficiency as a percent> memory/compute pipeline overlap. Kernels without perf models use their measured trace time as-is. Candidates are not reported unless at least 50% of their kernels have perf models (75% for candidates of 5 or more kernels). Actual recoverable time depends on implementation feasibility and interaction effects.\n\u003c!-- === MULTI_TRACE scope line (after the note; the same line goes under the System-Level note) === -->\nScope: \u003cfocus_job> only (trace `\u003cfocus trace>`, capture `\u003cmanifest.capture_folder or raw>`)\n\u003c!-- === COMPARATIVE Kernel Fusion === -->\n> **Note:** Kernel fusion analysis is experimental.\n\n\u003c!-- Populate from system_findings/kernel_fusion_findings.md if kernel_fusion category exists in manifest. -->\n\u003c!-- Each finding uses Insight / Action / Impact / Confidence format, with Impact from kernel_fusion_metrics.json. -->\n\u003c!-- P1/P2/P3+ ordered by impact_score, highest first. -->\n\u003c!-- Icon mapping by PRIORITY NUMBER (not severity): P1=🔴, P2=🟡, P3+=🟢 -->\n\u003c!-- If no findings or kernel_fusion category not in manifest, replace the cards below with: \"No kernel fusion opportunities detected.\" -->\n\n### 🔴 P1: \u003cCandidate Name>\n\n**Insight**: [1 sentence - what fusion pattern was detected]\n\n**Action**: [1-2 sentences - which kernels to fuse and how]\n\n\u003c!-- impact-begin kind=p_item low=\u003ckernel_fusion_metrics.impact_estimates[0].impact_score_low> mid=\u003ckernel_fusion_metrics.impact_estimates[0].impact_score> high=\u003ckernel_fusion_metrics.impact_estimates[0].impact_score_high> -->\n**Impact**: [impact_score: X.X (perf-model coverage Y/Z kernels)]\n\u003c!-- impact-end -->\n\n**Confidence**: [high / medium / low - fusion pattern quality]\n\n→ *See [Detailed Analysis: Kernel fusion insights > P1](#detailed-analysis-fusion-P1) for details*\n\n---\n\n### 🟡 P2: \u003cCandidate Name>\n\n**Insight**: [1 sentence]\n\n**Action**: [1-2 sentences]\n\n\u003c!-- impact-begin kind=p_item low=\u003ckernel_fusion_metrics.impact_estimates[1].impact_score_low> mid=\u003ckernel_fusion_metrics.impact_estimates[1].impact_score> high=\u003ckernel_fusion_metrics.impact_estimates[1].impact_score_high> -->\n**Impact**: [impact_score: X.X (perf-model coverage Y/Z kernels)]\n\u003c!-- impact-end -->\n\n**Confidence**: [high / medium / low]\n\n→ *See [Detailed Analysis: Kernel fusion insights > P2](#detailed-analysis-fusion-P2) for details*\n\n---\n\n### 🟢 P3: \u003cCandidate Name>\n\n**Insight**: [1 sentence]\n\n**Action**: [1-2 sentences]\n\n\u003c!-- impact-begin kind=p_item low=\u003ckernel_fusion_metrics.impact_estimates[2].impact_score_low> mid=\u003ckernel_fusion_metrics.impact_estimates[2].impact_score> high=\u003ckernel_fusion_metrics.impact_estimates[2].impact_score_high> -->\n**Impact**: [impact_score: X.X (perf-model coverage Y/Z kernels)]\n\u003c!-- impact-end -->\n\n**Confidence**: [high / medium / low]\n\n→ *See [Detailed Analysis: Kernel fusion insights > P3](#detailed-analysis-fusion-P3) for details*\n\n\u003c!-- All additional fusion P-items (P4, P5, ...) follow the same pattern with Detailed Analysis links: → *See [Detailed Analysis: Kernel fusion insights > PN](#detailed-analysis-fusion-PN) for details* -->\n\n---\n\n## System-Level Optimizations\n\n> **Note:** System-level analysis is exploratory. The patterns and recommendations below are under active development and may be refined as system-level analysis matures.\n\n\u003c!-- === MULTI_TRACE: add the scope line from Kernel Fusion here === -->\n\u003c!-- === COMPARATIVE system-level note === -->\n\u003c!-- In comparative mode, add this note immediately after the blockquote above: -->\n\u003c!-- > **Comparative Note:** System-level analysis is performed on the primary trace (Trace 1) only. Cross-trace system-level comparison is not yet supported. -->\n\nFindings from system-level analysis (GPU utilization, memory transfer patterns,\ncommunication/compute overlap). These affect the GPU pipeline as a whole.\n\n\u003c!-- CONDITIONAL: If NO actionable system-level issues found (idle \u003c= 15% and all multi-kernel assessments flagged: false), use Template A. -->\n\u003c!-- Otherwise, number priorities sequentially: CPU/Idle first (if idle > 15%), then multi-kernel issues by severity. -->\n\u003c!-- Icon mapping by PRIORITY NUMBER (not severity): P1=🔴, P2=🟡, P3+=🟢 -->\n\u003c!-- Title format: Descriptive name only. -->\n\u003c!-- System-level recommendations always include **Impact**: \"Not quantifiable from trace data\" with null markers. -->\n\u003c!-- De-dup rule: If CPU/Idle and Multi-Kernel propose the same mechanism/action, keep one merged system card with combined evidence (do not render two near-duplicate cards). -->\n\n\u003c!-- === TEMPLATE A: No actionable system-level issues === -->\n\u003c!-- Use this when idle \u003c= 15% and all multi-kernel assessments have flagged: false -->\n\n✅ No system-level bottlenecks detected. GPU activity breakdown shows X% computation, with negligible memcpy and communication overhead.\n\n\u003c!-- === TEMPLATE B: Actionable issues found === -->\n\u003c!-- Use this when idle > 15% or at least one multi-kernel assessment has flagged: true -->\n\n### 🔴 P1: \u003cCPU/Idle Title OR Multi-Kernel Issue Title>\n\n**Insight**: [1-2 sentences - what's wrong]\n\n**Action**: [1-2 sentences - what to do]\n\n\u003c!-- impact-begin kind=p_item low=null mid=null high=null -->\n**Impact**: Not quantifiable from trace data\n\u003c!-- impact-end -->\n\n→ *See [Detailed Analysis: System-level insights > P1](#detailed-analysis-system-p1) for details*\n\n---\n\n### 🟡 P2: \u003cMulti-Kernel Issue Title>\n\n**Insight**: [1 sentence - what's wrong]\n\n**Action**: [1-2 sentences - what to do]\n\n\u003c!-- impact-begin kind=p_item low=null mid=null high=null -->\n**Impact**: Not quantifiable from trace data\n\u003c!-- impact-end -->\n\n→ *See [Detailed Analysis: System-level insights > P2](#detailed-analysis-system-p2) for details*\n\n---\n\n### 🟢 P3: \u003cNext Multi-Kernel Issue>\n\n**Insight**: [1 sentence]\n\n**Action**: [1-2 sentences]\n\n\u003c!-- impact-begin kind=p_item low=null mid=null high=null -->\n**Impact**: Not quantifiable from trace data\n\u003c!-- impact-end -->\n\n→ *See [Detailed Analysis: System-level insights > P3](#detailed-analysis-system-p3) for details*\n\n\u003c!-- All additional system P-items follow the same pattern with Detailed Analysis links -->\n\n---\n\n## Detailed Analysis\n\n\u003c!-- Paste reasoning blocks from sub-agent findings, augment headings with P-numbers, icons, and HTML anchors. Everything else should be copied verbatim-->\n\u003c!-- Detailed Analysis labels per rule 9 — do not use these labels in optimization cards above -->\n\u003c!-- Impact estimate bullets are rendered by each sub-agent from metadata/*.json → impact_estimates (same source as card Impact). -->\n\u003c!-- MARKER CONTRACT: Every #### P\u003cN>: heading in Detailed Analysis MUST be\n preceded by \u003c!-- reasoning-candidate tier=\u003cTIER> rank=\u003cR> --> where TIER = compute | fusion | system (matching the ### subsection), R= 1, 2, 3, … incrementing per tier (rank=1 for first item, rank=2 for second, etc.). -->\n\n### Compute Kernel Insights\n\n\u003c!-- One #### 🔴/🟡/🟢 Pn: \u003ctitle> block per entry in priority_data.findings[], in array order. -->\n\u003c!-- Source the body block from the sub-agent's findings.md by joining on (findings[i].category, findings[i].category_rank): the sub-agent emits its P-items ordered by intra-category rank, so its rank-N block becomes this report's PN where N matches the position in priority_data.findings[]. -->\n\u003c!-- Each block has an HTML anchor: \u003ca id=\"detailed-analysis-compute-pN\">\u003c/a> -->\n\u003c!-- If the Compute Kernel Optimizations section rendered the no-opportunities summary, mirror it here: \"✅ No compute kernel optimization opportunities identified. All categories are within target performance bounds.\" -->\n\n\u003c!-- === STANDALONE Compute Kernel Data table === Use this schema for standalone mode ONLY. Use these 10 exact columns (must match sub_agent_spec.md § Operations Table Schema) -->\n\n\u003ca id=\"detailed-analysis-compute-p1\">\u003c/a>\n\u003c!-- reasoning-candidate tier=compute rank=1 -->\n#### 🔴 P1: \u003cBrief Title> (\u003cLibrary>)\n**Phase**: \u003cpriority_data.findings[0].phase>\n**Identification:**\n**Data:**\n\n| Operation | Args | Kernel Path | Kernel Name | Time (ms) | %E2E | Count | FLOPS/Byte | Efficiency | Bound |\n|-----------|------|-------------|-------------|-----------|------|-------|------------|------------|-------|\n| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |\n\n**Reasoning for Slowdown:**\n**Resolution:**\n**Impact estimate:**\n\n\u003c!-- === COMPARATIVE Compute Kernel Data table === Use this schema for comparative mode ONLY. Use these 10 exact columns (must match sub_agent_spec.md § Operations Table Schema) -->\n\u003c!-- Trace 1 ms = operations[i].time_ms. Trace 2 ms = operations[i].t2_time_ms.\n Count T1/T2 = operations[i].count / operations[i].count_trace2 when present.\n Difference (ms) = operations[i].difference_ms (negative ⇒ Trace 1 slower), or —. -->\n\n\u003ca id=\"detailed-analysis-compute-p1\">\u003c/a>\n\u003c!-- reasoning-candidate tier=compute rank=1 -->\n#### 🔴 P1: \u003cBrief Title>\n**Identification:** [1-2 sentences - How this opportunity was surfaced relative to the target trace. Must end with (source: \u003cartifact> → \u003ckeys>).]\n**Data:** [1 sentence summary of table]\n\n| Operation | Args (T1) | Kernel Path | Kernel Name | Trace 1 Time (ms) | Trace 2 Time (ms) | Count (T1/T2) | Difference (ms) | FLOPS/Byte (T1) | Bound (T1) |\n|-----------|-----------|-------------|-------------|-------------------|-------------------|---------------|-----------------|-----------------|------------|\n| ... | ... | ... | ... | ... | ... | .../... | ... | ... | ... |\n\n**Reasoning for Slowdown:** [2-3 sentences - Why Trace 1 is slower than Trace 2 for these operations as the traces show. No micro-architecture speculation.]\n**Resolution:** [1-2 sentences - Why the suggested optimization helps close the gap — not merely restating what to do.]\n**Impact estimate:** [Rendered from metadata → impact_estimates]\n\n\u003ca id=\"detailed-analysis-compute-p2\">\u003c/a>\n\u003c!-- reasoning-candidate tier=compute rank=2 -->\n#### 🟡 P2: \u003cBrief Title>\n**Identification:**\n**Data:**\n**Reasoning for Slowdown:**\n**Resolution:**\n**Impact estimate:**\n\n\u003ca id=\"detailed-analysis-compute-p3\">\u003c/a>\n\u003c!-- reasoning-candidate tier=compute rank=3 -->\n#### 🟢 P3: \u003cBrief Title>\n**Identification:**\n**Data:**\n**Reasoning for Slowdown:**\n**Resolution:**\n**Impact estimate:**\n\n### Kernel Fusion Insights\n\u003c!-- === STANDALONE Kernel Fusion === -->\n> **Note:** Kernel fusion analysis is experimental. impact_score projections estimate the recoverable fraction of E2E with \u003ckernel_fusion_metrics.overlap_efficiency as a percent> memory/compute pipeline overlap. Kernels without perf models use their measured trace time as-is. Actual recoverable time depends on implementation feasibility and interaction effects.\n\u003c!-- === COMPARATIVE Kernel Fusion === -->\n> **Note:** Kernel fusion analysis is experimental.\n\n\u003c!-- Paste reasoning blocks from kernel_fusion_findings.md, ordered by impact_score, highest first (matching card order). -->\n\u003c!-- Comparative: replace the single Data table with a **Trace 1** (E2E \u003cms> ms) table and a **Trace 2** (E2E \u003cms> ms) table. -->\n\u003c!-- If kernel_fusion category is not in the manifest or findings are empty, show \"No fusion impact estimates available.\" -->\n\n\u003ca id=\"detailed-analysis-fusion-P1\">\u003c/a>\n\u003c!-- reasoning-candidate tier=fusion rank=1 -->\n#### 🔴 P1: \u003cCandidate Name>\n\n**Identification:**\n\n**Data:**\n\n| Operation | Args | Kernel Path | Kernel Name | Time (ms) | %E2E | Count | FLOPS/Byte | Efficiency | Bound |\n|---|---|---|---|---|---|---|---|---|---|\n| \u003cop> | \u003cargs> | \u003ckernel path> | \u003ckernel name (truncated)> | X.XXX | X.XX | N | X.XX | X.XX% of Y TFLOPS/TB/s | compute/memory-bound |\n\n**Resolution:**\n\n**Impact estimate:**\n\n\u003ca id=\"detailed-analysis-fusion-P2\">\u003c/a>\n\u003c!-- reasoning-candidate tier=fusion rank=2 -->\n#### 🟡 P2: \u003cCandidate Name>\n**Identification:**\n**Data:**\n**Resolution:**\n**Impact estimate:**\n\n\u003ca id=\"detailed-analysis-fusion-P3\">\u003c/a>\n\u003c!-- reasoning-candidate tier=fusion rank=3 -->\n#### 🟢 P3: \u003cCandidate Name>\n**Identification:**\n**Data:**\n**Resolution:**\n**Impact estimate:**\n\n\u003c!-- Repeat for P4 onward with anchor detailed-analysis-fusion-PN. -->\n\n### System-Level Insights\n\n\u003c!-- One #### 🔴/🟡/🟢 Pn: \u003ctitle> block per promoted system P-item, in priority order. -->\n\u003c!-- Each block has an HTML anchor: \u003ca id=\"detailed-analysis-system-pN\">\u003c/a> -->\n\u003c!-- System-level detailed analysis uses the same format for both standalone and comparative modes.\n In comparative mode, system-level analysis covers Trace 1 () only. -->\n\u003c!-- If the System-Level Optimizations section rendered the no-bottlenecks summary, mirror it here: \"✅ No system-level bottlenecks detected.\" -->\n\n\u003ca id=\"detailed-analysis-system-p1\">\u003c/a>\n\u003c!-- reasoning-candidate tier=system rank=1 -->\n#### 🔴 P1: \u003cBrief Title>\n**Identification:**\n**Data:**\n**Reasoning for Slowdown:**\n**Resolution:**\n**Impact estimate:**\n\n\u003ca id=\"detailed-analysis-system-p2\">\u003c/a>\n\u003c!-- reasoning-candidate tier=system rank=2 -->\n#### 🟡 P2: \u003cBrief Title>\n**Identification:**\n**Data:**\n**Reasoning for Slowdown:**\n**Resolution:**\n**Impact estimate:**\n\n\u003ca id=\"detailed-analysis-system-p3\">\u003c/a>\n\u003c!-- reasoning-candidate tier=system rank=3 -->\n#### 🟢 P3: \u003cBrief Title>\n**Identification:**\n**Data:**\n**Reasoning for Slowdown:**\n**Resolution:**\n**Impact estimate:**\n\n---\n\n## Appendix\n\n### Model Architecture\n- **Model**: \u003cmodel>\n- **Architecture**: \u003carchitecture>\n- **Scale**: \u003cscale>\n- **Precision**: \u003cprecision>\n\n### Hardware Reference\n- **Platform**: \u003cplatform>\n- **Peak Memory BW**: X TB/s\n- **Peak MAF (BF16)**: Y TFLOPS\n- **Peak MAF (FP8)**: Z TFLOPS (if supported)\n- **Peak MAF (FP4)**: W TFLOPS (if supported)\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":26630,"content_sha256":"dd3d4402c670c4f6211e23adff94d9b8ce22c27af7e18a182b3721e8b035dd1e"},{"filename":"templates/sub_agent_spec.md","content":"\u003c!--\nCopyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.\n\nSee LICENSE for license information.\n-->\n\n# Sub-Agent Findings Specification\n\nCanonical reference for the output that sub-agents write into their findings\nfiles. The orchestrator extracts these sections when composing the final\n`analysis.md` report.\n\n> **Usage:** Link here from every `*-analyzer.md` instead of duplicating the\n> schema. Replace `\u003ccategory>` with the actual category name.\n> This spec receives `comparison_scope`: `standalone` (default) or `comparative`.\n---\n\n## Orchestrator-consumed sections\n\nEvery findings file must end with these two sections, in this order:\n\n1. `## Recommendations`\n2. `## Detailed Analysis`\n\nApplies to both tiers (compute → `category_findings/`, system → `system_findings/`). Agents may include any other sections (Overview, Operations Breakdown, Key Bottlenecks, …) before them — those are agent-internal and not parsed by the orchestrator.\n\n---\n\n## No actionable findings\n\n**Compute tier:** There is no actionable bottleneck when the analyzer left\n`category_data/\u003ccategory>_metrics.json::category_findings` as an **empty array**\n`[]`. In that case emit **empty** `## Recommendations` and **empty**\n`## Detailed Analysis` exactly as in § Empty category_findings.\n\n**System tier:** Follow the structured output your analyzer JSON supports.\n\n---\n\n## Recommendations\n\nEach P-item maps 1:1 to a `## Detailed Analysis` reasoning candidate at the same rank.\n\n```markdown\n### P1: \u003cBrief Title> (\u003cLibrary>) \u003c!-- (\u003cLibrary>) only on compute tier -->\n**Phase**: \u003cphase> \u003c!-- only when the finding has a phase -->\n**Insight**: [1 sentence — what's wrong]\n**Action**: [1-2 sentences — what to do]\n\u003c!-- impact-begin kind=p_item low=\u003cimpact_score_low> mid=\u003cimpact_score> high=\u003cimpact_score_high> phase=\u003cphase> -->\n**Impact**: [impact_score: X.X, OR \"Not quantifiable from trace data\"]\n\u003c!-- impact-end -->\n```\n\n- **Compute tier**: include all three fields. Pull `**Impact**` from `category_data/\u003ccategory>_metrics.json::category_findings[i]`, ordered by `rank` (one card per entry).\n- **System tier**: omit the `(\u003cLibrary>)` title suffix. Always emit `**Impact**: Not quantifiable from trace data` wrapped in `kind=p_item` markers with `low=null mid=null high=null`.\n- **Field labels are exact** — `**Insight**`, `**Action**`, `**Impact**`.\n- **`(\u003cLibrary>)` suffix**: the single `category_findings[i].library` for this card (one library per finding by construction). Omit the parenthetical when the value is `Unknown`.\n- **Phase**: only when `category_findings[i].phase` is present (multi-trace standalone runs), emit the `**Phase**` line on the card and its Detailed Analysis block and the marker's `phase=`; omit all three otherwise. Do not repeat the phase in the title, insight text, or the Operation column.\n- **Marker required** — see § Impact markers (REQUIRED). The `low`/`mid`/`high` attributes carry the raw `impact_score_low/impact_score/impact_score_high` values from `category_findings[i]`. For non-quantifiable cards (system tier) use `low=null mid=null high=null`.\n\n---\n\n## Detailed Analysis block schema\n\nEach candidate block lives inside a `## Detailed Analysis` section. It starts\nwith an HTML comment and an `####` heading:\n\n```markdown\n## Detailed Analysis\n\n\u003c!-- reasoning-candidate tier=\u003ccompute|system> rank=\u003cN> -->\n#### \u003cinsight_title>\n**Phase**: \u003cphase> \u003c!-- only when the finding has a phase -->\n\n**Identification:** …\n\n**Data:** …\n\n**Reasoning for Slowdown:** …\n\n**Resolution:** …\n\n**Impact estimate:** …\n```\n\n### HTML comment fields\n\n| Field | Values | Meaning |\n|-------|--------|---------|\n| `tier` | `compute` \\| `system` | Must match the findings directory (`category_findings/` → compute, `system_findings/` → system). |\n| `rank` | Integer ≥ 1 | Compute tier: `category_findings[i].rank`. System tier: agent-local priority within this file (1 = highest). |\n\n### Required labels\n\nThe five labels below must appear **in this order**, each on its own line with a\nblank line between them. The validator checks for these as substring matches.\nFusion tier (`tier=fusion`): the same labels except **Reasoning for Slowdown:**, which fusion blocks omit.\n\n| Label | Purpose |\n|-------|---------|\n| `**Identification:**` | Why these operations were flagged. Body text must be plain language — JSON keys, dotted paths, and internal variable names belong **only** in the closing `(source: \\`artifact\\` → \\`keys\\`)` parenthetical (artifact + keys backticked, e.g. `(source: \\`\u003ccat>_metrics.json\\` → \\`operations[].efficiency.efficiency_percent\\` \u003c 70)`). When any flagged op has a non-null `library` (e.g. `Tensile`, `CK`, `AITER`, `Triton`, `rocBLAS`), state the backend in prose and include `operations[].library` in the `(source:)` parenthetical. When `operations[i].module_chain` is non-empty, name the model layer the ops belong to. When `operations[i].call_chain` is present, use it for deeper context. |\n| `**Data:**` | **Compute** (`tier=compute`): exactly one trace-grounded kernel breakdown table (see § Operations Table Schema). **All columns in the schema are mandatory — never drop a column.** Use `—` for any individual cell whose value is missing or null. **System** (`tier=system`): **must not** include kernel breakdown tables. include metric table (see § Metric Table Schema). |\n| `**Reasoning for Slowdown:**` | Why the workload is slow *as the trace shows*: **Standalone:** low % of roofline, low arithmetic intensity, unfused patterns, etc. **Comparative:** how Trace 1 is slower than Trace 2 for these operations — express speed differences as \"X% faster\" or \"X% slower\", plus absolute time gaps. Never use raw efficiency ratios or `efficiency_percent` values in prose. **Forbidden:** micro-architecture speculation (bank conflicts, L1 miss rates, etc.). |\n| `**Resolution:**` | **Why** the suggested optimization helps — not merely restating *what* to do. Must align with the P-item **Action** on the card. **Forbidden tautologies:** Do not restate the roofline definition (e.g. \"raising bandwidth toward the roofline reduces kernel time\"). Instead, explain the **mechanism** (e.g. \"fusion eliminates the intermediate write-back, cutting bytes moved per invocation in half\"). If the mechanism is not inferable from the trace, state only the action. |\n| `**Impact estimate:**` | Compute tier: rendered from `category_findings[i]` (matched by `rank`), two-bullet low/high `impact_score` format (see § Impact estimate rendering). System tier: `Impact estimate is not quantifiable from trace data.` |\n\n### Sentence quality\n\n- Each sentence should convey **one main idea**. Do not chain independent\n observations with em-dashes, semicolons, or \"while\" bridges. Avoid run-on\n sentences.\n\n### Trace observability (compute tier)\n\nThis is the single source of truth for what compute-tier sub-agents can and\ncannot infer from a kernel-level PyTorch trace. Ground every claim in\n**Reasoning for Slowdown** / **Resolution** in a **CAN Infer** row; for any\nproperty in the **CANNOT Infer** rows, use the listed fallback prose instead\nof speculating.\n\n#### CAN Infer (universal — all compute categories)\n\n| Observable | Source |\n|------------|--------|\n| Kernel names | `trunc_kernel_details` column |\n| Kernel durations | Trace events |\n| Achieved TFLOPS/s or TB/s | Calculated from duration + FLOPs/bytes |\n| Efficiency % vs roofline | Achieved / resolved peak (MAF or memory BW) |\n| Invocation counts | Number of trace events per signature |\n| Library / backend | `library` column / kernel-name heuristics |\n| Bound type | `efficiency.bound_type` (compute / memory) |\n| Input shape dimensions | `Input Dims` column (semantics per category — e.g. M/N/K for GEMM, B/H/S/D for SDPA, expert/token counts for MoE, NCHW for convolution) |\n\n#### CANNOT Infer (universal — all compute categories)\n\nThese require hardware counters or profiler tools, not a trace.\n\n| NOT Observable | Why | Fallback prose |\n|----------------|-----|----------------|\n| Bank conflicts | Requires hardware counters | \"Low efficiency — profile with hardware counters to diagnose.\" |\n| Cache hit rates | Requires hardware counters | \"Large working set may exceed cache.\" |\n| Wave / SM occupancy | Requires hardware counters | \"Kernel running slower than expected — profile occupancy with hardware counters.\" |\n| Shared-memory / LDS usage | Requires hardware counters | \"Shared-memory usage not visible — profile with hardware counters.\" |\n| Intra-warp shuffle efficiency | Requires hardware counters | \"Warp-shuffle efficiency not visible — profile with hardware counters.\" |\n| Root causes generally | Traces show WHAT, not WHY | \"Bottleneck identified — generate reproducer for kernel team.\" |\n\n#### CANNOT Infer (category-specific)\n\nEach analyzer owns its own category-specific blind spots under a\n`## Trace observability (category-specific)` section in its `*-analyzer.md` file.\nThe universal rows above always apply on top of those.\n\n---\n\n## Operations Table Schema (compute tier)\n\nStandard column schema for operations breakdown tables and the `**Data:**` table\ninside `## Detailed Analysis` blocks.\n\n### Standalone (`comparison_scope` = `standalone`)\n\n```markdown\n| Operation | Args | Kernel Path | Kernel Name | Time (ms) | %E2E | Count |FLOPS/Byte| Efficiency | Bound |\n|-----------|--------|-----------------------------------------|-------------|-----------|------|-------|----------|------------|-------|\n```\n\n**All ten columns above are mandatory.** Never drop a column because some or all of its values are missing — render `—` in any cell whose value is null/absent and keep the column. The header row of every `**Data:**` table must contain exactly these ten column names in this order. (Agents may append extra columns at the end when needed, e.g. `Sub-Category` in the generic-op analyzer, but must not remove or reorder the ten standard columns.)\n\n**Per-row impact marker (REQUIRED).** Immediately after the last row of each compute-tier `**Data:**` table, emit one paired marker carrying the per-row impact scores as a CSV, in table order:\n\n```markdown\n\u003c!-- impact-begin kind=op_row rank=\u003cN> impacts=\u003cv0>,\u003cv1>,... -->\n\u003c!-- impact-end -->\n```\n\n`\u003cN>` is this candidate's `rank`; `impacts` is one value per data row in table order (see § Impact markers, `op_row` row, for the attr contract).\n\n**Column mappings** (source: `metrics['operations']`):\n- **Operation**: `operations[i].name`. Bare op name only — shape/dtype go in Args. Allowed suffix: `(decode)`/`(prefill)` to disambiguate the same op at multiple shapes.\n- **Args**: `operations[i].args`. Pre-rendered shape/dtype string, already joined with `\u003cbr>` — paste verbatim, do not reformat or re-join. `—` when absent.\n- **Kernel Path**: `operations[i].launcher_path`. Relative Python path that launched the kernel (e.g. `sglang/srt/layers/quantization/fp8_utils.py(549): aiter_w8a8_block_fp8_linear`). **Copy the value exactly as-is — do NOT truncate, shorten, or extract just the function name.** `—` when absent.\n- **Kernel Name**: `operations[i].kernel_name_trunc`. Truncated GPU kernel name(s) launched by this operation. For multi-kernel ops, formatted as `Kernel 1: \u003cname>\u003cbr>Kernel 2: \u003cname>`. **Copy the value exactly as-is.** `—` when absent. (The full untruncated name is available in `operations[i].kernel_name` if needed for identification.)\n- **Time (ms)**: `operations[i].time_ms` — kernel time in milliseconds.\n- **%E2E**: `operations[i].percent_of_total` — kernel time as % of E2E GPU time. `—` when null. (`percent_of_category` is still in the JSON for screening thresholds but no longer rendered.)\n- **Count**: `operations[i].count` — total invocations, not unique signatures. `—` when absent.\n- **FLOPS/Byte**: `operations[i].efficiency.flops_per_byte` — note the nested path under `efficiency`, NOT a top-level field. `—` when null.\n- **Efficiency**: `operations[i].efficiency.efficiency_percent`, formatted by `bound_type`:\n - `compute-bound`: `X.XX% of Y TFLOPS` (Y = `resolved_peak_maf`)\n - `memory-bound`: `X.XX% of Y TB/s` (Y = `resolved_peak_mem_bw`)\n- **Bound**: `operations[i].efficiency.bound_type` + `-bound` suffix (e.g., `memory-bound`). Must reflect compute/memory bound type — never use `classification.gemm_type` or similar.\n\n### Comparative (`comparison_scope` = `comparative`)\n\n```markdown\n| Operation | Args (T1) | Kernel Path | Kernel Name | Trace 1 Time (ms) | Trace 2 Time (ms) | Count (T1/T2) | Difference (ms) | FLOPS/Byte (T1) | Bound (T1) |\n|-----------|-----------|-------------|-------------|-------------------|-------------------|---------------|-----------------|-----------------|------------|\n```\n\n**Column mappings** (all sourced from `metrics['operations']`; do **not** re-join the CSV):\n- **Operation**: `operations[i].name`. Bare op name only.\n- **Args (T1)**: `operations[i].args`. Pre-rendered shape/dtype string, already joined with `\u003cbr>` — paste verbatim. `—` when absent.\n- **Kernel Path**: `operations[i].launcher_path`. Relative Python path that launched the kernel. **Copy the value exactly as-is — do NOT truncate, shorten, or extract just the function name.** `—` when absent.\n- **Kernel Name**: `operations[i].kernel_name_trunc`. Truncated GPU kernel name(s). For multi-kernel ops, formatted as `Kernel 1: \u003cname>\u003cbr>Kernel 2: \u003cname>`. **Copy the value exactly as-is.** `—` when absent.\n- **Trace 1 Time (ms)**: `operations[i].time_ms`\n- **Trace 2 Time (ms)**: `operations[i].t2_time_ms`. `—` when absent.\n- **Count (T1/T2)**: T1 = `operations[i].count`; T2 = `operations[i].count_trace2`. Format `T1 / T2` (use `—` for missing T2).\n- **Difference (ms)**: `operations[i].difference_ms`. `—` when absent.\n- **FLOPS/Byte (T1)**: `operations[i].efficiency.flops_per_byte`\n- **Bound (T1)**: `operations[i].efficiency.bound_type` with a `-bound` suffix\n\nAgents may add extra columns when needed (e.g. `Sub-Category` in the generic-op analyzer).\n\n---\n\n## Metric Table Schema (system tier)\n\nStandard schema for the `**Data:**` table inside system-tier `## Detailed Analysis` blocks. In comparative mode, report Trace 1 metrics only — do not add Trace 2 columns or comparisons.\n\n```markdown\n| Metric | Value | Flagged |\n|--------|-------|---------|\n```\n\n**Column rules:**\n- **Metric**: Copy metric label directly from earlier findings sections — do not rename or reformat.\n- **Value**: `X.X ms` or `X.X%` or `X.X ms (X.X%)`\n- **Flagged**: `true` when the metric's threshold is exceeded (issue present); `false` otherwise.\n\n---\n\n## Peak Reference (compute tier)\n\nWhen citing peak performance for a bottleneck, select the correct peak based on\n`operations[i].efficiency.bound_type`:\n- **compute-bound**: Use `operations[i].efficiency.resolved_peak_maf` (TFLOPS).\n Report achieved TFLOPS/s vs peak TFLOPS.\n- **memory-bound**: Use `operations[i].efficiency.resolved_peak_mem_bw` (TB/s).\n Report achieved TB/s vs peak TB/s.\n\nDo not look up peaks independently from the metadata dict.\n\n---\n\n## Impact estimate rendering\n\nCompute-tier sub-agents READ their P-items from `category_data/\u003ccategory>_metrics.json::category_findings[]`, one card per entry ordered by `rank`. The analyzer script has already grouped per-op estimates by `(bound_type, library, eff_bucket)` in standalone mode (or `(bound_type, library)` in comparative mode), summed impact, and dropped sub-threshold groups; the sub-agent renders one card per surviving entry.\n\nThe set of P-items is decided by `category_findings[]` alone — `MIN_PITEM_IMPACT_SCORE` already gated upstream. **Per-category efficiency tables, expected-band thresholds, and Common Patterns in analyzer files are interpretation context for the prose** (cite in **Reasoning for Slowdown** when a member matches the band/symptom); they MUST NOT be used to add or drop P-items.\n\n### Reading category_findings[]\n\n| Field | Use |\n|-------|-----|\n| `rank` | Card order within your category (1 = highest impact). Also the `rank=` value in `\u003c!-- reasoning-candidate -->`. |\n| `bound_type` | `compute` \\| `memory`. Selects the matching Action Prose Guidance row. |\n| `library` | One per finding. Drives the `(\u003cLibrary>)` title suffix. |\n| `phase` | Multi-trace standalone runs only. |\n| `eff_bucket` | Roofline-efficiency band: `\"0-30\"`, `\"30-60\"`, `\"60-100\"`, or `\"unknown\"` (standalone); `\"all\"` (comparative). Members within a finding share the same band. |\n| `impact_score` / `_low` / `_high` | Group-summed % of E2E. Render verbatim into `kind=p_item` and `kind=detail_estimate` markers. |\n| `estimate_method` | `\"quantified\"` (impact from a perf model — standalone roofline gap or comparative t2/t1 ratio) or `\"heuristic\"` (op has no perf model; impact estimated from E2E share — see below). |\n| `percent_of_total` | Heuristic findings only: the op's combined E2E GPU-time share (summed across shapes). Drives the warning line in § Heuristic findings. Absent on quantified findings. |\n| `member_count`, `members[]` | Underlying per-op estimate rows (operation, time_ms, efficiency_pct, `type`, …) — rows of the `**Data:**` table. `members[].type == \"unmodeled_significant\"` marks a heuristic finding; `\"kernel_tuning\"` is a quantified (perf-modeled) finding. |\n\n### Empty category_findings\n\nIf `category_findings[]` is empty, emit `## Recommendations` with no P-items\nand `## Detailed Analysis` with no candidates. Do not manufacture sub-threshold\ncards to fill the section — that is the honest \"no actionable issues\" answer.\n\n### Heuristic findings (`estimate_method == \"heuristic\"`)\n\nAn op with no perf model: `impact_score` / `_low` / `_high` are estimated from a\nrecoverable fraction of its E2E share (`percent_of_total`, summed across shapes)\nand ranked alongside quantified findings.\n\nRender like a normal P-item card (numeric `low`/`mid`/`high` on `kind=p_item`,\nnormal `kind=detail_estimate` bullets) with two additions:\n- Immediately **after** the `kind=p_item` impact marker block (i.e. *outside* the `impact-begin`/`impact-end` markers), add a warning line, substituting the finding's `percent_of_total`:\n\n```markdown\n> **Warning — estimated:** No performance model for this op; impact is derived from its E2E GPU-time share (\u003cpercent_of_total>%), not a gap projection.\n```\n- In the `## Detailed Analysis` `**Data:**` row, render `—` for `Efficiency` and `Bound`.\n\n### Rendering in `## Detailed Analysis` (compute tier)\n\nTwo bullets — low and high. Wrap in `kind=detail_estimate` markers (see\n§ Impact markers).\n\n```markdown\n\u003c!-- impact-begin kind=detail_estimate low=\u003cimpact_score_low> mid=\u003cimpact_score> high=\u003cimpact_score_high> -->\n- Low end impact_score: \u003cimpact_score_low>\n- High end impact_score: \u003cimpact_score_high>\n\u003c!-- impact-end -->\n```\n\n## Impact markers (REQUIRED)\n\nEvery block whose contents depend on `impact_score*` values must be wrapped in\na paired HTML-comment marker. The markers carry the underlying numeric data as\nkey=value attributes so that optional downstream tooling can re-process the\nblock deterministically without re-parsing prose.\n\n### Marker shape\n\n```\n\u003c!-- impact-begin kind=KIND attr1=value1 attr2=value2 -->\n...rendered markdown content for this block...\n\u003c!-- impact-end -->\n```\n\nThe block between them is exactly the `impact_score`-based markdown you would otherwise emit.\n\n### `kind` values you must emit\n\n| `kind` | Where | Required attributes | Optional attributes |\n|--------|-------|--------------------|---------------------|\n| `p_item` | Around every P-item `**Impact**` line in `## Recommendations`. | `low`, `mid`, `high` (all three; use `null` only for system-tier non-quantifiable); `phase` when the finding has one. | `category` is reserved for the orchestrator template; sub-agents do **not** emit it. |\n| `detail_estimate` | Around the two-bullet `Low end ... / High end ...` block under `**Impact estimate:**` in each `## Detailed Analysis` candidate. Skip only for system-tier non-quantifiable estimates. | `low`, `mid`, `high` (impact_score values, % of E2E). | none |\n| `op_row` | After the last row of each compute-tier `**Data:**` table (see § Operations Table Schema → Per-row impact marker). | `rank`, `impacts` (CSV of per-row `impact_score`, one per data row in table order; `—` for null). | none |\n\n### Value-source rule\n\nThe numbers in marker attributes (`low`, `mid`, `high`) **must** be transcribed\nverbatim from `category_data/\u003ccategory>_metrics.json::category_findings[i]`\n(`impact_score_low`, `impact_score`, `impact_score_high` respectively), matched\nby `rank`. Do not re-derive, round, or scale them. Do not pull them from any\nother source.\n\n---\n\n## Validate findings (required before returning)\n\nAfter writing the findings file and impact estimates, run the programmatic\nvalidator. This replaces the previous manual self-check. The validator also\nenforces marker structure (pairing, `kind=`, per-kind required attrs,\nmandatory `kind=p_item` for category/system findings unless exempt) per\n§ Impact markers above.\n\n```bash\n\u003cprefix> python3 -c \"\nimport sys\nfrom TraceLens.Agent.Analysis.utils.validation_utils import validate_findings_file\npassed, errors = validate_findings_file(sys.argv[1], sys.argv[2], sys.argv[3])\nif not passed:\n print('FAIL:')\n for e in errors:\n print(' - ' + e)\n sys.exit(1)\nprint('PASS: Findings file is valid')\n\" '\u003coutput_dir>/\u003csubdir>/\u003ccategory>_findings.md' '\u003ctier>' '\u003ccomparison_scope>'\n```\n\nWhere `\u003ctier>` is `compute` or `system`, `\u003csubdir>` is `category_findings`\nor `system_findings` respectively, and `\u003ccomparison_scope>` is `standalone` or\n`comparative`.\n\n**If validation fails (exit code 1):**\n\n1. Read the FAIL output — error messages are self-explanatory and include the fix hint.\n2. Fix the findings file accordingly. Edit sections in place and not regenerate the entire output.\n3. When fixing table cells (Args, Kernel Path, or any multi-row issue), **re-emit the entire table in one edit** — do NOT patch rows individually. Batch together edits together.\n4. Re-run validation. Maximum 2 retry attempts; if still failing, return with a warning.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":22192,"content_sha256":"672f1a24f527361499b3498f00458c9fe3e6b8007f85141ab4d35d55b79dd78a"}],"content_json":{"type":"doc","content":[{"type":"paragraph","content":[{"text":"\u003c!-- Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.","type":"text"}]},{"type":"paragraph","content":[{"text":"See LICENSE for license information. -->","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Analysis orchestrator","type":"text"}]},{"type":"paragraph","content":[{"text":"Coordinate ","type":"text"},{"text":"system-level","type":"text","marks":[{"type":"strong"}]},{"text":" analysis (CPU/idle, kernel fusion, multi-kernel / comm / memcpy) and ","type":"text"},{"text":"compute-kernel","type":"text","marks":[{"type":"strong"}]},{"text":" analysis (GEMM, SDPA, elementwise, etc.): one trace load, shared prep, parallel subagents, then aggregation into ","type":"text"},{"text":"analysis.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Full procedure","type":"text"}]},{"type":"paragraph","content":[{"text":"Follow ","type":"text"},{"text":"reference.md","type":"text","marks":[{"type":"link","attrs":{"href":"reference.md","title":null}},{"type":"strong"}]},{"text":" for every step (user prompts, ","type":"text"},{"text":"\u003cprefix>","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"{CMD}","type":"text","marks":[{"type":"code_inline"}]},{"text":" usage, CLI commands, subagent launch text, validation, report ","type":"text"},{"text":"tee","type":"text","marks":[{"type":"code_inline"}]},{"text":" order, and plot embedding).","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Workflow index","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"0. Query User Inputs (interactive answers or a run YAML, Environment Setup) → cache/run_config.yaml\n1. Generate Performance Report (one wrapper call on cache/run_config.yaml; semantic pre-processing first)\n2. Trace-Quality Gate → Deterministic fallback analysis\n3-6. Prepare Category Data (GPU Util, Top Ops, Tree Data, Multi-Kernel Data, Category Filtering)\n7. System-Level Analysis (PARALLEL) → system_findings/\n8. Compute Kernel Subagents (PARALLEL) → category_findings/\n 8.5. Aggregate → priority_data.json::findings[]\n9. Validate Subagent Outputs\n10. load_findings + Model Identification (subagent) → metadata/model_info.json\n11. Render performance PNG if agent_extension.py is absent\n12. Generate analysis.md (orchestrator writes via \u003cprefix> tee), optional extension, embed PNG","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Rules","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Subagents:","type":"text","marks":[{"type":"strong"}]},{"text":" Use your runner's subagent primitive (the Task tool in Claude Code or Cursor, ","type":"text"},{"text":"spawn_agent","type":"text","marks":[{"type":"code_inline"}]},{"text":" under Codex) ","type":"text"},{"text":"only","type":"text","marks":[{"type":"strong"}]},{"text":" where reference.md says “subagent” (Steps ","type":"text"},{"text":"7","type":"text","marks":[{"type":"strong"}]},{"text":", ","type":"text"},{"text":"8","type":"text","marks":[{"type":"strong"}]},{"text":", ","type":"text"},{"text":"10","type":"text","marks":[{"type":"strong"}]},{"text":"). The orchestrator runs everything else, including Step 8.5, using the command prefix from ","type":"text"},{"text":"\u003coutput_dir>/cache/cmd_prefix.txt","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"{CMD}","type":"text","marks":[{"type":"code_inline"}]},{"text":" substitution).","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Language:","type":"text","marks":[{"type":"strong"}]},{"text":" Prefer vendor-agnostic terms (GPU kernels, collective communication, vendor GEMM library, DNN primitives, GPU graph). When quoting trace data, real kernel names are fine.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Subagent prompts:","type":"text","marks":[{"type":"strong"}]},{"text":" Point each subagent at the checked-in agent file under ","type":"text"},{"text":"TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/\u003cname>.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" (see reference.md for exact paths and prompt shells).","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Primary outputs","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Deliverable:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"\u003coutput_dir>/analysis.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Internals:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"system_findings/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"category_findings/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"category_data/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"metadata/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"perf_report*.xlsx","type":"text","marks":[{"type":"code_inline"}]},{"text":", CSV folders — see package README for layout.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Agent layout","type":"text"}]},{"type":"paragraph","content":[{"text":"Project subagents ship with this skill: ","type":"text"},{"text":"TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/*.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},"metadata":{"date":"2026-11-02","name":"tracelens-analysis-orchestrator","author":"@skillopedia","source":{"stars":473,"repo_name":"skills","origin_url":"https://github.com/amd/skills/blob/HEAD/skills/tracelens-analysis-orchestrator/SKILL.md","repo_owner":"amd","body_sha256":"aeb351c5ccc44f044c1d22411c0532d4c59f434916975fb6fd32a99ef5c0dd2b","cluster_key":"63dbe7feeac70cdae11c12fbf05dcb6ef8083f8d22ad003305aa17e1665a3dcd","clean_bundle":{"format":"clean-skill-bundle-v1","source":"amd/skills/skills/tracelens-analysis-orchestrator/SKILL.md","text_bytes":497308,"source_commit":"c67366c8d1ab85bbe5f5e9000f40617a1bfb9edb","skills_sh_rank":13011,"attachment_count":45,"text_attachments":45,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"skills/tracelens-analysis-orchestrator/SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"v1","category":"ai-agent-development","category_label":"AI"},"exact_dupes_collapsed_into_this":0},"license":"MIT","version":"v1","category":"ai-agent-development","import_tag":"skills-sh-top20k-2026-11","description":"Orchestrates modular PyTorch profiler trace analysis with TraceLens: generates perf reports, prepares category data, runs system-level and compute-kernel subagents in parallel, validates outputs, and writes a prioritized stakeholder report (analysis.md). Use when the user asks to follow the analysis orchestrator, run the agentic analysis workflow, run it from config \u003crun.yaml>, analyze a trace, compare two traces, or mentions standalone or comparative TraceLens analysis."}},"renderedAt":1793755483436}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.