perf-tuning — Skillopedia

Performance Tuning When the user complains about slowness ("Hermes is slow", "work is lagging", "builds are crawling", "everything feels sluggish"), do not guess. Diagnose, then act. Mental model The most common failure mode is assuming "slow" means "CPU-bound". On these 32-thread Xeon boxes, CPU is rarely the bottleneck. The usual suspects in order: 1. Filesystem driver (NTFS/ntfs-3g via FUSE on is single-threaded and serializes IO across all 32 cores) 2. CPU frequency governor ( default keeps cores at 1.6 GHz) 3. Background indexers ( , , ) 4. Memory pressure / swap (rare on 31 GiB box, but…

\\t' scripts/benchmarks/results.tsv | tail -10\n```\n**Reference numbers on ace-linux-1 (governor=performance, NTFS-3G workspace):**\n- `cpu_single` ~500 ms — anything >800 ms means CPU contention\n- `cpu_multi` ~1400 ms — should be close to `cpu_single` on a dedicated box; >3× means IO/scheduling contention\n- `git_status` was 155,707 ms with NTFS-3G — anything >2 s is a filesystem-driver smell\n- `git_log` ~50 ms — should always be fast (cached)\n\nIf `git_status` is in the seconds, suspect IO contention. If in the minutes, suspect filesystem driver.\n\n## Fixes (apply via the runbook)\n\nThe runbook at `docs/runbooks/perf-tuning.md` is the canonical procedure. This skill's job is to **reach the right runbook section quickly**:\n\n| Symptom | Runbook section |\n|---------|-----------------|\n| `git status` >5 s on workspace-hub | \"Migrate /mnt/local-analysis from NTFS\" + interim ntfs3 mitigation |\n| Idle scaling MHz \u003c80% | \"CPU frequency governor → performance (persistent)\" |\n| Cross-socket memory thrashing on heavy job | \"NUMA-pin heavy batch jobs\" |\n| Compute-bound solver (FEA/CFD) under-performing | \"Disable Hyper-Threading\" + benchmark |\n| `tracker-miner-fs-3` >20% sustained | \"Turn off tracker3 for paths you don't search\" |\n\n## Anti-patterns to avoid\n\n- **Don't reach for BIOS toggles before running the benchmark.** HT-off blindly can hurt the overnight 5-terminal batch pattern. Numbers first.\n- **Don't recommend `tracker reset --hard`** without checking `tracker3 status` first — on ace-linux-1 it's idle and indexes only `$HOME` (not workspace-hub). Resetting it would just re-index later.\n- **Don't remount the live workspace filesystem** (`umount /mnt/local-analysis`) while the user has a shell or editor in it. Plan filesystem changes for a maintenance window.\n- **Don't skip the \"filesystem of the working directory\" check.** This is the diagnostic most likely to surface a 100× win and the one most often missed.\n\n## Persisting findings\n\nAfter diagnosis, append a row to `scripts/benchmarks/results.tsv` with a meaningful label so the next session can compare. Memorable labels: `pre-{change}`, `post-{change}`, `diag-{date}`.\n\nIf you discover a new bottleneck class not in the runbook, propose an addition rather than fixing in-place. The runbook is the durable artifact; this skill is the routing layer.\n---","attachment_filenames":[],"attachments":[],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"Performance Tuning","type":"text"}]},{"type":"paragraph","content":[{"text":"When the user complains about slowness (\"Hermes is slow\", \"work is lagging\", \"builds are crawling\", \"everything feels sluggish\"), do not guess. Diagnose, then act.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Mental model","type":"text"}]},{"type":"paragraph","content":[{"text":"The most common failure mode is assuming \"slow\" means \"CPU-bound\". On these 32-thread Xeon boxes, CPU is rarely the bottleneck. The usual suspects in order:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Filesystem driver","type":"text","marks":[{"type":"strong"}]},{"text":" (NTFS/ntfs-3g via FUSE on ","type":"text"},{"text":"/mnt/local-analysis","type":"text","marks":[{"type":"code_inline"}]},{"text":" is single-threaded and serializes IO across all 32 cores)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"CPU frequency governor","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"powersave","type":"text","marks":[{"type":"code_inline"}]},{"text":" default keeps cores at ~1.6 GHz)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Background indexers","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"tracker-miner-fs-3","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"baloo","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"mlocate.updatedb","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Memory pressure / swap","type":"text","marks":[{"type":"strong"}]},{"text":" (rare on 31 GiB box, but check)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Network mounts","type":"text","marks":[{"type":"strong"}]},{"text":" (sshfs, NFS — slow when remote host is unreachable)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Genuinely CPU-bound","type":"text","marks":[{"type":"strong"}]},{"text":" (only after eliminating the above)","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Diagnostic order (5 minutes total)","type":"text"}]},{"type":"paragraph","content":[{"text":"Run these in order. Stop as soon as you find a clear answer.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"1. System pulse","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"uptime # load avg vs nproc — if load \u003c\u003c threads, CPU is NOT the bottleneck\nfree -h # check available memory and swap usage\nps -eo pid,user,pcpu,pmem,etime,comm --sort=-pcpu | head -10","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"2. Filesystem of the working directory (the most-missed diagnostic)","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"mount | grep \"$(git rev-parse --show-toplevel | cut -d/ -f1-3)\"","type":"text"}]},{"type":"paragraph","content":[{"text":"If the line contains ","type":"text"},{"text":"ntfs","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"ntfs-3g","type":"text","marks":[{"type":"code_inline"}]},{"text":", or ","type":"text"},{"text":"fuseblk","type":"text","marks":[{"type":"code_inline"}]},{"text":" → ","type":"text"},{"text":"this is the bottleneck","type":"text","marks":[{"type":"strong"}]},{"text":". No further CPU diagnosis needed; route to runbook section \"Migrate /mnt/local-analysis from NTFS\".","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"3. Governor state","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor\nlscpu | grep \"scaling MHz\"","type":"text"}]},{"type":"paragraph","content":[{"text":"If governor is not ","type":"text"},{"text":"performance","type":"text","marks":[{"type":"code_inline"}]},{"text":" or scaling is \u003c80%, install the cpupower service from the runbook.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"4. Background indexer noise","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"ps -eo pcpu,comm --sort=-pcpu | grep -E \"tracker|baloo|updatedb|spotlight\" | head","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"5. Run the benchmark for hard numbers","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"scripts/benchmarks/perf-bench.sh diag-$(date +%s)\ncolumn -t -s

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

\\t' scripts/benchmarks/results.tsv | tail -10","type":"text"}]},{"type":"paragraph","content":[{"text":"Reference numbers on ace-linux-1 (governor=performance, NTFS-3G workspace):","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"cpu_single","type":"text","marks":[{"type":"code_inline"}]},{"text":" ~500 ms — anything >800 ms means CPU contention","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"cpu_multi","type":"text","marks":[{"type":"code_inline"}]},{"text":" ~1400 ms — should be close to ","type":"text"},{"text":"cpu_single","type":"text","marks":[{"type":"code_inline"}]},{"text":" on a dedicated box; >3× means IO/scheduling contention","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"git_status","type":"text","marks":[{"type":"code_inline"}]},{"text":" was 155,707 ms with NTFS-3G — anything >2 s is a filesystem-driver smell","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"git_log","type":"text","marks":[{"type":"code_inline"}]},{"text":" ~50 ms — should always be fast (cached)","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"If ","type":"text"},{"text":"git_status","type":"text","marks":[{"type":"code_inline"}]},{"text":" is in the seconds, suspect IO contention. If in the minutes, suspect filesystem driver.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Fixes (apply via the runbook)","type":"text"}]},{"type":"paragraph","content":[{"text":"The runbook at ","type":"text"},{"text":"docs/runbooks/perf-tuning.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" is the canonical procedure. This skill's job is to ","type":"text"},{"text":"reach the right runbook section quickly","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Symptom","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Runbook section","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"git status","type":"text","marks":[{"type":"code_inline"}]},{"text":" >5 s on workspace-hub","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"\"Migrate /mnt/local-analysis from NTFS\" + interim ntfs3 mitigation","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Idle scaling MHz \u003c80%","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"\"CPU frequency governor → performance (persistent)\"","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Cross-socket memory thrashing on heavy job","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"\"NUMA-pin heavy batch jobs\"","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Compute-bound solver (FEA/CFD) under-performing","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"\"Disable Hyper-Threading\" + benchmark","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"tracker-miner-fs-3","type":"text","marks":[{"type":"code_inline"}]},{"text":" >20% sustained","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"\"Turn off tracker3 for paths you don't search\"","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Anti-patterns to avoid","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Don't reach for BIOS toggles before running the benchmark.","type":"text","marks":[{"type":"strong"}]},{"text":" HT-off blindly can hurt the overnight 5-terminal batch pattern. Numbers first.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Don't recommend ","type":"text","marks":[{"type":"strong"}]},{"text":"tracker reset --hard","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" without checking ","type":"text"},{"text":"tracker3 status","type":"text","marks":[{"type":"code_inline"}]},{"text":" first — on ace-linux-1 it's idle and indexes only ","type":"text"},{"text":"$HOME","type":"text","marks":[{"type":"code_inline"}]},{"text":" (not workspace-hub). Resetting it would just re-index later.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Don't remount the live workspace filesystem","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"umount /mnt/local-analysis","type":"text","marks":[{"type":"code_inline"}]},{"text":") while the user has a shell or editor in it. Plan filesystem changes for a maintenance window.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Don't skip the \"filesystem of the working directory\" check.","type":"text","marks":[{"type":"strong"}]},{"text":" This is the diagnostic most likely to surface a 100× win and the one most often missed.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Persisting findings","type":"text"}]},{"type":"paragraph","content":[{"text":"After diagnosis, append a row to ","type":"text"},{"text":"scripts/benchmarks/results.tsv","type":"text","marks":[{"type":"code_inline"}]},{"text":" with a meaningful label so the next session can compare. Memorable labels: ","type":"text"},{"text":"pre-{change}","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"post-{change}","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"diag-{date}","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"If you discover a new bottleneck class not in the runbook, propose an addition rather than fixing in-place. The runbook is the durable artifact; this skill is the routing layer.","type":"text"}]}]},"metadata":{"date":"2026-06-05","name":"perf-tuning","author":"@skillopedia","source":{"stars":10,"repo_name":"workspace-hub","origin_url":"https://github.com/vamseeachanta/workspace-hub/blob/HEAD/.agents/skills/operations/devops/perf-tuning/SKILL.md","repo_owner":"vamseeachanta","body_sha256":"9f4a45cb7b46362aaba1c3dc42d107db7ebbbff700c03a1dbcdd101000011fb1","cluster_key":"1954b82b3c306e8101db650f624e1f6deb7c5c70219c915ae2171f4e8844b626","clean_bundle":{"format":"clean-skill-bundle-v1","source":"vamseeachanta/workspace-hub/.agents/skills/operations/devops/perf-tuning/SKILL.md","bundle_sha256":"ad5fb608a6e4a5bfe3f8da6e39400634f1081a8403a36658936757d3dc98f058","attachment_count":0,"text_attachments":0,"binary_attachments":0},"cluster_size":2,"skill_md_path":".agents/skills/operations/devops/perf-tuning/SKILL.md","import_metadata":{"date":"2026-06-05","author":"@skillopedia","version":"v1","category":"web-development","category_label":"Web"},"exact_dupes_collapsed_into_this":1},"version":"v1","category":"web-development","import_tag":"clean-skills-v1","description":"Diagnose local machine slowness and apply CPU/IO/governor tuning. Use when work feels slow on ace-linux-1 / ace-linux-2 — Hermes lagging, git operations taking minutes, builds dragging, Codex feeling sluggish. Runs perf-bench.sh, identifies the dominant bottleneck (CPU governor, filesystem driver, NUMA, memory pressure, I/O contention), and applies fixes from the perf-tuning runbook."}},"renderedAt":1782980736087}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.