TAO AutoML Standalone install? If this session was not initialized by the TAO skill bank plugin, run first (host preflight, credentials, cross-skill discovery). Run automated hyperparameter optimization for a TAO model by combining: 1. The selected model skill under . 2. The selected platform skill under . 3. , which generates recommendations, launches selected action jobs, extracts metrics, and feeds results back to the optimizer. Do not launch until model metadata, platform preflight, data visibility, credentials, image choice, and compute shape are all proven. Execution Runtime — Hard Gate…

\"$RD/progress.log\" 2>/dev/null | tail -1 | awk '{print $1}')\n case \"$LAST\" in\n \"\") CARD=00-preflight.md ;;\n preflight) CARD=10-baseline-eval.md ;;\n baseline_eval) CARD=20-launch-recs.md ;;\n recs_launched|runner_finished) CARD=30-interpret.md ;;\n done) CARD=30-interpret.md ;;\n *) CARD=30-interpret.md ;;\n esac\n\n CMDS=\"\"; [ -f \"$RD/commands.log\" ] && CMDS=$(tail -20 \"$RD/commands.log\" 2>/dev/null | head -c 2500)\n\n PROMPT=\"Execute exactly ONE stage card of a TAO AutoML workflow, then end your turn.\nConstants (all exported as environment variables in your bash shell — use them verbatim):\nWS=$WS RD=$RD SB=$SB VENV=$VENV TRAIN_IMG=$TRAIN_IMG\nRules: follow the card exactly; one step = one command, run it verbatim; never retype or\nexpand commands; variables you set inside one bash call do not survive to the next call;\nprogress marks are appended ONLY by the exact card commands; after any detached launch or\nwhen the card says so, print the exact STAGE_DONE token and STOP — no extra commands.\nYou have a hard budget of $PI_KIT_TURN_BUDGET tool calls this session.\n===== CARD =====\n$(cat \"$CARDS/$CARD\")\n===== RECENT COMMANDS (reuse, don't re-derive) =====\n${CMDS:-\u003cnone>}\"\n\n echo \"[a-driver] round $round -> $CARD (last=$LAST) $(date)\" >> \"$LOG\"\n before=$( [ -f \"$RD/progress.log\" ] && wc -l \u003c \"$RD/progress.log\" || echo 0 )\n touch \"$RUN_HOME/.round_marker\"\n timeout 2400 pi \"${PI_FLAGS[@]}\" --model \"$MODEL\" \"$PROMPT\" >> \"$LOG\" 2>&1\n after=$( [ -f \"$RD/progress.log\" ] && wc -l \u003c \"$RD/progress.log\" || echo 0 )\n # Progress = new log mark OR active work OR any new JOB OUTPUT under RD\n # (fast jobs can start AND finish inside one session window). Exclude the\n # recorder's commands.log: it is harness telemetry, appended even on stuck\n # rounds, and counting it would defeat the 5-round no-progress abort.\n NEWF=$(find \"$RD\" -newer \"$RUN_HOME/.round_marker\" -type f ! -name 'commands.log' 2>/dev/null | head -1)\n # Grace before counting no-progress: a detached launch can take a few seconds\n # to appear in docker ps after the session exits.\n if [ \"$after\" -eq \"$before\" ] && ! working && [ -z \"$NEWF\" ]; then\n sleep 20\n NEWF=$(find \"$RD\" -newer \"$RUN_HOME/.round_marker\" -type f ! -name 'commands.log' 2>/dev/null | head -1)\n fi\n if [ \"$after\" -eq \"$before\" ] && ! working && [ -z \"$NEWF\" ]; then\n noop=$((noop+1)); echo \"[a-driver] no progress ($noop)\" >> \"$LOG\"\n [ $noop -ge 5 ] && { echo \"[a-driver] ABORT: 5 no-progress rounds\" >> \"$LOG\"; exit 1; }\n else noop=0; fi\ndone\nfinish_if_terminal\necho \"[a-driver] 40-round cap $(date)\" >> \"$LOG\"\nexit 1\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":8182,"content_sha256":"302d36eef8d61d76aefde224439e1b58b3a421b3f8380e9ba3cae677910b6fa7"},{"filename":"cards/README.md","content":"# AutoML card pack\n\nCompiled stage cards for running `tao-run-automl` (AutoMLRunner wheel,\nbayesian search) via the token-efficient execution kit\n(`skills/core/tao-token-efficient-execution`) — fresh headless sessions, one\ncard per stage, progress marks in `$RD/progress.log`.\n\n**Measured vs running the skill in one long conversation: 5.8M → 0.5M billed\ntokens (-91%), peak context 198k → 42k.** Validated end-to-end on a ~35B-class\nexecution model with zero guard fires (VCN classify, 4 bayesian recs); the\nraw-skill arm on the same model failed to complete honestly.\n\n## Stage flow\n\n```\n00-preflight → 10-baseline-eval → 20-launch-recs (runner detached) → 30-interpret\n```\n\n- `10-baseline-eval` scores the untrained init via a zero-LR trick and gates\n on a PID file (never on pgrep of command text).\n- `20-launch-recs` starts the AutoMLRunner detached with `session_id` pinned\n so interrupted runs resume the same search instead of restarting it.\n- `30-interpret` harvests per-rec metrics through the wheel's own result keys\n (`rec_id`, `metric`, `metric_value`), writes `$RD/results.md`, marks DONE.\n\n`driver.sh` owns run-dir selection (`automl2_*` under `$WS/results`), routes\non the last ok mark, and halts when the latest mark is a FAIL (no auto-retry).\n\n## Run it\n\n```bash\n# prerequisites: kit install done, WS + VENV set in ~/.tao-kit/kit.env, API key exported.\n# Standalone (no plugin): export TAO_SKILL_BANK_PATH=/path/to/tao-skill-bank first.\nnohup bash \"$TAO_SKILL_BANK_PATH/skills/applications/tao-run-automl/cards/driver.sh\" > /dev/null &\ntail -f ~/.tao-kit/automl/driver.log\n```\n\nConfig (env or `~/.tao-kit/kit.env`): `WS`, `VENV` required; `MODEL`,\n`TRAIN_IMG`, `SB` (skill bank the cards read references from, defaults to\nthis bank), `PI_KIT_TURN_BUDGET` optional.\n\n## Version pin\n\n- Authored against `tao-run-automl` with the 7.0.1 AutoMLRunner wheel\n (DockerSDK path: explicit `mounts` required via platform kwargs), VCN\n classify on an AOI workspace, Pi harness.\n- The default `TRAIN_IMG` is resolved from `images.tao_toolkit.pyt` in\n `$SB/versions.yaml`, using the same bank that supplies recommendation templates.\n An explicit override is preserved, but must match that bank's template schema;\n the old 6.26.3 image rejects the current `train.checkpointer` field.\n Baseline eval still copies `$WS/specs/baseline_spec.yaml`; validate that\n workspace spec against the selected image during preflight.\n- Historical measurements used older images/wheels. Resolving the current\n image fixes the known default mismatch; it is not an end-to-end validation\n claim for the current image and AutoMLRunner combination.\n- The dataset/model specifics (VCN classify spec fields, AOI CSV layout) are\n baked into the cards — that is what makes them cheap to execute. For a\n different network or dataset, re-author the pack (kit skill → authoring\n prompt) rather than hand-editing these cards.\n- Relaunching after a completed run starts a fresh run (the driver refreshes\n its launch marker on DONE). Never delete\n `~/.tao-kit/automl/.launch_marker` while a run is active.\n\n## Baseline honesty note\n\nThe pack's \"baseline\" is the untrained initialization scored with the\nzero-LR trick — a floor for \"did the search improve anything\", not a trained\nreference. Improvements over it are search-quality signals, not deploy gates.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3365,"content_sha256":"0a19d50c44e0fb741bdccd981208683b49593b4b95baeb6e68ecb71c50c904de"},{"filename":"config/skillspector-baseline.yaml","content":"version: 2\n# Reviewed false positives — NVIDIA/nvskills-ci#67. No content removed.\nrules:\n - id: \"AE*\"\n path: \"*\"\n reason: \"Reviewed FP: SkillSpector 2.11.0 (SkillEvaluator 1.5.4+) introduced hard runtime budgets (MAX_WORKFLOW_SECONDS=60) that cannot be raised externally; large skills are only partially inspected and the partial read is reported as evasion. Upstream: nvcarps-ci!252. Artifacts are present, small and plain-text; nothing is hidden from analysis.\"\n - id: \"skillspector_execution_failure\"\n path: \"*\"\n reason: \"Reviewed FP: scan aborts via the same 2.11.0 runtime budget and on reference_resolution of slash-containing prose and template paths (PASS/NO_PASS, schemas/\u003caction>.schema.json, nvcr.io image URIs, frontmatter version strings). Not a property of the skill content. Upstream: nvcarps-ci!252.\"\n - id: \"PE3\"\n path: \"eval.config\"\n reason: \"Reviewed FP (#67): documented infrastructure path reference (e.g. ~/.docker/config.json, ~/.ssh keys) appearing in documentation or example config; the skill performs no credential access.\"\n - id: \"TM1\"\n path: \"eval.config\"\n reason: \"Reviewed FP (#67): scoped maintenance command operating on the skill's own workspace or eval fixture; not tool misuse.\"\n - id: \"TM1\"\n path: \"references/best_rec_adapter.py\"\n reason: \"Reviewed FP (#67): scoped maintenance command operating on the skill's own workspace or eval fixture; not tool misuse.\"\n - id: \"SQP-2\"\n path: \"eval.config\"\n reason: \"Reviewed FP (#67): eval-fixture cleanup scoped to the eval's own $WORKSPACE_DIR and its containers; not an unscoped destructive command.\"\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":1627,"content_sha256":"ee42ca5446b1c03bd72a3dfc422a34ba1ec5ba6389fba2addcdcf5a3e13aed7f"},{"filename":"evals/evals.json","content":"[\n {\n \"id\": \"tao-run-automl-basic\",\n \"question\": \"A user request: \\\"Run AutoML / hyperparameter optimization (HPO) for NVIDIA TAO networks using AutoMLRunner.\\\" Identify which TAO skill applies and, reading only that skill's documentation, outline the steps it prescribes. Do NOT run any commands, scripts, web searches, or other tools \\u2014 describe the plan only.\",\n \"expected_skill\": \"tao-run-automl\",\n \"expected_script\": null,\n \"ground_truth\": \"Identify tao-run-automl as the applicable skill and summarize its documented workflow from SKILL.md without executing anything.\",\n \"expected_behavior\": [\n \"Identifies tao-run-automl as the relevant skill\",\n \"Outlines the documented workflow steps from SKILL.md\",\n \"Does not run commands, scripts, or web searches\"\n ]\n },\n {\n \"id\": \"tao-run-automl-container-default\",\n \"question\": \"Plan TAO AutoML for a model whose model skill resolves image model-train:latest. The user chose local Docker but did not mention venv execution. Explain where recommendation training runs and what a host AutoML Python environment is for. Do not execute anything.\",\n \"expected_skill\": \"tao-run-automl\",\n \"expected_script\": null,\n \"ground_truth\": \"All child model actions run in model-train:latest. A host Python environment may run AutoML/SDK control-plane code only and does not become the training runtime. Venv model-action execution requires an explicit user request.\",\n \"expected_behavior\": [\n \"Uses the model skill's resolved container image for recommendation and evaluation jobs\",\n \"Treats a host AutoML or SDK venv as control-plane-only\",\n \"Does not infer venv training from local Docker or an installed Python stack\"\n ]\n }\n]\n","content_type":"application/json; charset=utf-8","language":"json","size":1741,"content_sha256":"b5de9cf1b262b6f9fccbcb804c5ea37b4c6479727321a9aea5318d08872ab033"},{"filename":"references/automl-advanced-monitoring.md","content":"# TAO AutoML Advanced Monitoring\n\nLLM/agentic options, optional hooks, progress monitoring, resume, result interpretation, pitfalls, and status queries.\n\nLoad this file only when the compact `SKILL.md` points here for the current task. If this reference conflicts with `SKILL.md`, `skill_info.yaml`, schemas, or platform/model skills, the compact/current source wins.\n\n## Contents\n\n- Natural Language Configuration\n- config = {\n- \"automl_algorithm\": \"bayesian\",\n- \"automl_hyperparameters\": [\"\u003cparam_from_model_schema>\", ...],\n- \"algorithm_specific_params\": {\"automl_max_recommendations\": 15},\n- \"metric\": \"\u003cmetric_from_model_skill_or_user_request>\",\n- \"reasoning\": \"...\"\n- }\n- LLM Analyzer (works with ANY algorithm)\n- After every 5 completed experiments, call:\n- analysis = {\n- \"patterns\": [\"...\"],\n- \"convergence_assessment\": \"improving\",\n- \"recommendations\": [\"...\"],\n- \"suggested_ranges\": {\"\u003cparam_name>\": {\"min\": ..., \"max\": ...}},\n- }\n- Autoresearch Agent Components\n- Research Programs\n- Validate before running\n- Advanced hooks (opt-in)\n- `metric_extractor(logs: str, metric_name: str) → float | None`\n- `eval_fn(rec, train_job_id: str) → float | None`\n- Step 4: Monitor Progress\n- Resume after interruption\n- Step 5: Interpret Results\n- How to report to the user\n- If all recs failed\n- Model-Specific Notes\n- Common Pitfalls\n- Querying Experiment Status\n- Progress summary\n- Best config\n- Per-rec details\n- In-flight jobs\n\n## LLM/Agentic Features Deep Dive\n\n### Natural Language Configuration\n\nDon't know which algorithm or parameters to use? The `NLConfigGenerator` translates plain English into a valid AutoML configuration:\n\n```python\nfrom tao_automl.brain.nl_config import NLConfigGenerator\n\ngenerator = NLConfigGenerator() # uses NVIDIA NIM by default\nconfig = generator.generate_config(\n user_prompt=user_goal,\n network=network_arch,\n available_parameters=param_records, # from generate_hyperparams_to_search()\n hardware_info=hardware_info,\n)\n# config = {\n# \"automl_algorithm\": \"bayesian\",\n# \"automl_hyperparameters\": [\"\u003cparam_from_model_schema>\", ...],\n# \"algorithm_specific_params\": {\"automl_max_recommendations\": 15},\n# \"metric\": \"\u003cmetric_from_model_skill_or_user_request>\",\n# \"reasoning\": \"...\"\n# }\n```\n\n### LLM Analyzer (works with ANY algorithm)\n\nThe `LLMAnalyzer` can be used alongside any classical algorithm to provide periodic analysis of experiment results:\n\n```python\nfrom tao_automl.brain.llm_analyzer import LLMAnalyzer\n\nanalyzer = LLMAnalyzer(analysis_interval=5, narrow_ranges=True)\n\n# After every 5 completed experiments, call:\nanalysis = analyzer.analyze(\n experiments=experiment_history,\n parameters=param_records,\n network=network_arch,\n metric_name=metric,\n metric_direction=direction,\n best_metric=best_metric,\n)\n# analysis = {\n# \"patterns\": [\"...\"],\n# \"convergence_assessment\": \"improving\",\n# \"recommendations\": [\"...\"],\n# \"suggested_ranges\": {\"\u003cparam_name>\": {\"min\": ..., \"max\": ...}},\n# }\n```\n\nWhen `narrow_ranges=True`, the analyzer suggests tighter search bounds based on observed patterns. These can be applied to dynamically focus the search.\n\n### Autoresearch Agent Components\n\nThe `autoresearch` algorithm integrates five AutoML-Agent concepts:\n\n| Component | What it does | When it runs |\n|---|---|---|\n| **KnowledgeRetriever** (RAP) | Retrieves built-in tuning knowledge for the requested network and optionally web-searched papers/benchmarks | Once at initialization |\n| **SpecPrescreener** | LLM predicts which of N candidate configs are worth running, WITHOUT training. Saves GPU budget by filtering unlikely-to-improve configs. | Before each trial — proposes 3 candidates, pre-screens to pick the best 1 |\n| **MultiStageVerifier** | Pre-launch: validates proposed changes won't crash/OOM. Post-result: checks metrics are plausible (not NaN, not anomalous). | Before launch + after result |\n| **ExperimentTracker** | Tracks full history with keep/discard decisions and reasoning | After each result |\n| **LLMAnalyzer** | Periodic pattern detection, convergence assessment, and optional range narrowing | Every N completed experiments |\n\n### Research Programs\n\nFor complex multi-phase optimization, define a research program:\n\n```python\nfrom tao_automl.brain.research_program import ResearchProgram, ResearchPhase\n\nprogram = ResearchProgram(\n objective=objective,\n network=network_arch,\n phases=[\n ResearchPhase(\n name=\"Phase 1\",\n algorithm=\"bayesian\",\n parameters=[\"\u003cparam_from_model_schema>\", \"...\"],\n trials=8,\n ),\n ResearchPhase(\n name=\"Phase 2\",\n algorithm=\"asha\",\n parameters=[\"\u003canother_param_from_model_schema>\", \"...\"],\n trials=15,\n carry_forward=\"best\", # best values carry into this phase\n ),\n ],\n)\n\n# Validate before running\nissues = program.validate(\n available_parameters=available_parameters,\n available_algorithms=[\"bayesian\", \"asha\"],\n)\n```\n\n---\n\n## Advanced hooks (opt-in)\n\nBoth hooks are optional. If neither is provided, the runner uses its built-in log regex extractor.\n\n### `metric_extractor(logs: str, metric_name: str) → float | None`\n\nCalled on every poll of the training container's logs. Return the most recent/final metric value seen, or `None` if the metric isn't yet present.\n\nUse it when:\n- Your container emits the metric in a non-standard log format the built-in regex misses.\n- You want to parse values from log lines instead of using the generic patterns.\n- Your metric needs derivation from multiple log fields.\n\n```python\nimport re\n\ndef extract_custom_metric(logs: str, metric_name: str):\n m = re.search(rf\"{re.escape(metric_name)}:\\s*([0-9.]+)\", logs)\n return float(m.group(1)) if m else None\n\nrunner.run(..., metric_extractor=extract_custom_metric)\n```\n\nExceptions raised inside the extractor are caught and logged; the runner continues polling.\n\n### `eval_fn(rec, train_job_id: str) → float | None`\n\nCalled once after a rec's training job reaches a terminal state, before the result is reported to the brain. Whatever it returns **overrides** any value captured by `metric_extractor` and becomes what the brain optimizes on.\n\nUse it when:\n- The real task metric lives outside the training logs.\n- You want a true-test-metric sweep without building surrounding plumbing yourself.\n- Per-rec cost is acceptable relative to `metric_extractor`.\n\n```python\ndef eval_on_held_out(rec, train_job_id):\n # Implement the model-specific evaluation flow documented in the model skill.\n metric_value = run_model_specific_eval(rec, train_job_id)\n return metric_value\n\nrunner.run(\n ...,\n automl_settings={\n \"metric\": task_metric,\n \"direction\": direction,\n \"session_id\": session_id,\n ...,\n },\n eval_fn=eval_on_held_out,\n)\n```\n\nExceptions from `eval_fn` are caught and logged — the runner falls back to the log-extracted metric for that rec.\n\n---\n\n## Step 4: Monitor Progress\n\n`runner.run()` blocks until all recommendations complete. Use callbacks for\nlive launch/result events from the process that owns the runner:\n\n```python\ndef on_rec(rec):\n print(f\"Rec {rec.id}: trying {rec.specs}\")\n\ndef on_result(rec, metric, status):\n print(f\"Rec {rec.id}: {status}, metric={metric}\")\n\nresult = runner.run(..., on_recommendation=on_rec, on_result=on_result)\n```\n\nFor status questions from a separate shell, agent turn, or detached monitor,\ndefault to the structured AutoML state instead of reading launcher logs:\n\n```python\nfrom tao_automl import query_status\n\nstatus = query_status(\"\u003cfull_workspace_path>/run_\u003ctimestamp>\")\n```\n\nUse `status[\"progress\"]`, `status[\"best\"]`, `status[\"recommendations\"]`, and\n`status[\"active_jobs\"]` as the source of truth for recommendation ids, specs,\nmetrics, success/failure/pending counts, and active TAO job ids. The state comes\nfrom `\u003cworkspace>/.automl/controller/`, `\u003cworkspace>/.automl/best_rec/`,\n`\u003cworkspace>/.automl/brain/`, and `\u003cworkspace>/active_jobs.json`.\n\nDo not comb through launcher logs for normal AutoML status. Logs are fallback\ndebug material only: use them when the user explicitly asks for logs, when\n`query_status()` reports no usable state, when diagnosing a failed child job, or\nwhen checking metric-extractor / LLM-client warnings that are not represented in\nthe structured state. If live backend queue state is needed, join the active TAO\njob ids from `query_status()[\"active_jobs\"]` with the selected SDK's job status\nAPI or the platform scheduler (`squeue` for SLURM); do not derive recommendation\nresults from logs.\n\nEach rec takes 10–90 minutes depending on model size, dataset, epochs, and checkpoint save cost. Don't assume failure during long uploads.\n\n### Checkpoint retention\n\n`automl_delete_intermediate_ckpt` defaults to `True`. Sequential searches\ndelete terminal failed and non-best trial artifacts as soon as they are no\nlonger needed. Multi-fidelity searches retain parent/rung checkpoints while\nthey are eligible for promotion or resume and prune them only after that\ndependency ends. Hybrid searches conservatively retain successful trial\nartifacts when full-fidelity provenance is ambiguous. For cleanup-supported\nsequential or provenance-verified scalar runs, completion retains the winning\ntraining artifacts and any separate final-evaluation result rather than one\nfull checkpoint set per recommendation. Multi-objective runs retain every\nnon-dominated Pareto-front checkpoint because there is no single winner across\nall objectives; conservative Hybrid runs can retain multiple successful\ntrials. A run\nstopped by a recommendation budget is resumable and therefore does not receive\ncompleted-run pruning.\n\nAutomatic cleanup covers SDK-routed job results: S3 prefixes, local absolute\nDocker `/results` binds, Lustre job directories, and VirtualEnv job results.\nFor pruned Docker and Kubernetes trials it also verifies and removes the exact\nterminal container or UID-bound Job/pods before reporting cleanup complete, so\nstopped writable layers and pod-local storage do not accumulate.\nKeep checkpoints under the SDK-routed job results directory. Explicit output\npaths outside it are not owned by the cleanup policy. Before the first trial,\ncleanup-aware SDKs validate that the selected output route and identity can be\nreclaimed. With retention enabled, writable named Docker volumes, remote binds,\nroot-output opt-outs, and unbound S3 identities are rejected rather than\nlaunched and silently accumulated. They may be used only with retention\nexplicitly disabled and an externally owned cleanup lifecycle.\nRaw S3 secrets are never written to the durable job store. After a process or\nhost restart, resume with the same explicit platform `env_vars`; retention\npreflight rebinds them to the recorded non-secret principal and endpoint before\nold trial cleanup is retried.\n\nSet the option to `False` only for an intentional all-trials debugging run and\ninclude the resulting storage cost and external cleanup owner in the launch\nreview.\n\n### Resume after interruption\n\nOn handled `SIGINT` or `SIGTERM`, the runner requests cancellation of active\nchild jobs and waits for the platform to confirm that their writers are\nterminal before deleting anything. If termination cannot be confirmed within\nthe bounded wait, the active-job record and artifacts remain for a later\nresume. An uncatchable process termination or host loss can also leave an\nin-flight backend job; recover that run with `resume=True` and the **full\nsuffixed path** (including the `run_\u003ctimestamp>` directory):\n\nResolve the controller identity before constructing the resumed runner:\n\n```bash\nSESSION_ID=$(python \"$TAO_SKILL_BANK_PATH/skills/applications/tao-run-automl/scripts/resolve_automl_session.py\" \\\n resolve --workspace ./my_experiment/run_20260423_183015)\n```\n\nThe command fails closed if no controller state exists. It also refuses to\nchoose arbitrarily when a workspace contains multiple controller files. For a\nworkspace contaminated by an older failed resume, inspect the controller\nstates and rerun with `--session-id \u003cintended-id>`; never select the first file\nreturned by the filesystem.\n\n```python\nresult = runner.run(\n ...,\n workspace_path=\"./my_experiment/run_20260423_183015\", # full suffixed path\n automl_settings={**automl_settings, \"session_id\": \"\u003cresolved-session-id>\"},\n resume=True,\n)\n```\n\nWhen `resume=True`, the runner does NOT append a new timestamp suffix — it reuses the path as-is.\n\nFor the original run, create the identity once after launch approval and before\ncalling the runner:\n\n```bash\nSESSION_ID=$(python \"$TAO_SKILL_BANK_PATH/skills/applications/tao-run-automl/scripts/resolve_automl_session.py\" new)\n```\n\nPlace that literal value in the sealed `automl_settings`. The resulting\n`.automl/controller/\u003csession_id>.json` is the durable binding used by later\nresume invocations.\n\nBefore either fresh or resumed `runner.run(...)`, call the bundled\n`validate_session_settings(automl_settings, resume=..., workspace=...)` gate\nshown in `SKILL.md`. Missing identities fail before the wheel can apply its\nrandom default; resumed identities must resolve to an existing controller.\n\nThis is a skill-boundary safeguard. The currently packaged\n`nvidia-tao-automl` wheel still permits `resume=True` without a matching\n`session_id` and may silently create a fresh controller for callers outside\nthis skill. NVBug 6662913 tracks that upstream fail-loud behavior; until the\nwheel changes, non-skill callers must apply the same explicit identity gate.\n\nBehaviour on resume:\n1. **Brain state** is reloaded from `\u003cworkspace>/.automl/*` for the explicit\n `session_id` — all completed rec results are already registered. Treat\n `Loaded controller state: 0 recommendations` as an error when the selected\n controller file contains completed recommendations; cancel before a new\n recommendation is submitted.\n2. **Any in-flight jobs** recorded in `\u003cworkspace>/active_jobs.json` (persisted after each submission) are polled to terminal, their recovered backend job IDs are restored on the recommendation, their metrics extracted, and reported to the brain — *before* the main propose-new-rec loop starts. No duplicate submissions or orphaned trial artifacts.\n3. After recovery, the loop continues normally until `automl.is_complete()`.\n\n---\n\n## Step 5: Interpret Results\n\nThe result is a plain dict:\n\n```python\n{\n \"best\": {\n \"rec_id\": 4,\n \"specs\": {\"\u003cparam_name>\": \"\u003cvalue>\", \"...\": \"...\"},\n \"metric_value\": 0.7077,\n },\n \"progress\": {\n \"completed\": 8, \"total\": 8,\n \"best_metric\": 0.7077, \"best_rec_id\": 4,\n \"algorithm\": \"bayesian\",\n },\n \"history\": [\n {\"rec_id\": 0, \"metric\": 0.6308, \"status\": \"success\"},\n {\"rec_id\": 1, \"metric\": 0.7077, \"status\": \"success\"},\n ...\n ],\n}\n```\n\nMetric values in `best` and `history` are always in the original scale the user provided — direction inversion (if any) is undone before the dict is returned.\n\nFor multi-fidelity algorithms (`hyperband`, `bohb`, `asha`, `dehb`, `hyperband_es`, and `pbt`), `best` is selected from the largest observed training budget once those candidates exist. Report any lower-budget metric that beats the selected final-budget metric as promotion context rather than treating it as the downstream checkpoint.\n\n### How to report to the user\n\n1. **Best config** — show the winning hyperparameters and metric value.\n2. **Comparison table** — rank recs by metric and highlight the best. For multi-fidelity algorithms, rank the largest-budget candidates first and include lower-budget winners as context when they differ from the selected final checkpoint.\n3. **Insights** — call out what the optimizer learned from the requested parameters and metric.\n4. **WandB link** — if tracking was enabled, provide the dashboard URL.\n5. **Next steps** — suggest:\n - More recs (re-run with `resume=True` + higher `automl_max_recommendations`).\n - Train longer with the best config using `sdk.create_job(specs=result[\"best\"][\"specs\"])`.\n - Run a downstream evaluation on the best checkpoint.\n - Run the model skill's recommended export/deploy workflow for the best model.\n\n### If all recs failed\n\nCheck common issues:\n- **Dataset path wrong** — verify the URI points to the layout required by the model skill.\n- **Metric never appears** — verify the model skill's required metric-related overrides and custom extractor are present.\n- **Checkpoint or eval artifact missing** — verify the model skill's checkpoint/export/eval requirements.\n- **Model or data download timeout** — inspect backend logs and model-skill error patterns.\n- **OOM** — reduce the model-specific batch, resolution, sequence length, or memory-heavy knobs recommended by the model skill.\n- **Cached data corruption** — inspect the model skill's dataset/cache error patterns and clear only the affected cache path if documented.\n- **LLM endpoint unreachable** (llm/hybrid/autoresearch only) — the brain falls back to random sampling. Check `AUTOML_LLM_ENDPOINT` and `AUTOML_LLM_API_KEY`. Verify with: `curl -s $AUTOML_LLM_ENDPOINT/models -H \"Authorization: Bearer $AUTOML_LLM_API_KEY\"`.\n\nIf the runner reports no new recommendations and there are no pending/running\nchild jobs, treat the AutoML run as exhausted instead of continuing to poll\nforever. Inspect the failed child job logs, fix the model skill/config/setup\nissue, then relaunch from a fresh runner or resume only after the failed cause\nis corrected.\n\n---\n\n## Model-Specific Notes\n\nModel-specific notes do not belong in this AutoML skill. For every requested model, resolve the packaged `model_skill`, read `\u003cbank-root>/skills/models/\u003cmodel_skill>/SKILL.md`, and use its **Training Requirements**, **Per-Action Dataset Requirements**, **Typical Spec Overrides**, **AutoML / HPO Notes**, and **Error Patterns** sections as the source of truth.\n\n---\n\n## Common Pitfalls\n\n1. **`skill_dir` not passed (or wrong path).** `AutoMLRunner(skill_dir=...)` requires an absolute path to a model directory inside the skill bank. The runner raises `FileNotFoundError: skill_info.yaml not found at \u003cskill_dir>/references/skill_info.yaml` if the path is wrong. Use the same bank root the agent loaded this SKILL.md from; combine with `skills/models/\u003cmodel_skill>/` after resolving aliases.\n2. **Wrong LLM endpoint (404).** Use `https://inference-api.nvidia.com` for NVIDIA inference API testing and pass it explicitly in `automl_settings`. The LLM brain falls back to random sampling on LLM failure, so check logs for \"LLM call failed\" before trusting the run as LLM-guided.\n3. **Model-specific training failures (data format, missing datasets, invalid params).** Each network has unique training requirements. ALWAYS read `\u003cbank-root>/skills/models/\u003cmodel_skill>/SKILL.md` — the \"Training Requirements\" and \"Error Patterns\" sections document model-specific failure modes that apply to AutoML recs too.\n4. **Workspace path collisions.** Running the same script twice overwrites the previous experiment. Always include a timestamp: `workspace_path=f\"./automl_workspace/{TIMESTAMP}\"` where `TIMESTAMP = datetime.now().strftime(\"%Y%m%d_%H%M%S\")`.\n5. **Using a weak proxy metric.** The brain can optimize a metric that does not reflect real task quality. Use the metric recommended by the model skill or provide `eval_fn`.\n6. **Implicit direction trap.** If the metric name does not imply the desired direction, set `direction` explicitly.\n7. **Spec-override typos.** `save_freq_in_epochs` (plural) used to silently do nothing; now raises `ValueError` with suggestion. If you see that error, it's the fix working.\n8. **Orchestrator dies mid-sweep.** Resolve the existing `session_id`, then relaunch with the same full `workspace_path`, that session id in `automl_settings`, and `resume=True`. In-flight jobs are recovered from `active_jobs.json`. Missing or ambiguous controller state is a blocker, not permission to start a new search.\n9. **Rec never reports a metric.** Check the model skill's metric-emission requirements and custom extractor guidance.\n10. **Parallel Bayesian arms.** Bayesian is inherently sequential. If you want parallelism, use `asha`. If you use multiple `AutoMLRunner` instances, give each its own `\u003cSDK>(state_file=...)` (e.g., `BrevSDK(state_file=...)`, `KubernetesSDK(state_file=...)`) to avoid SQLite write races on the SDK's job store.\n11. **LLM brain returning random configs.** If every LLM recommendation looks random, the LLM endpoint is probably failing silently. Check the logs for \"LLM call failed\" warnings. Verify your API key and endpoint are correct. Common cause: using the wrong endpoint URL (see pitfall #2).\n12. **`openai` package not installed.** The `llm`, `hybrid`, and `autoresearch` algorithms require the `openai` Python package. Install with `pip install openai` or reinstall tao-run-automl with the `llm` extra by resolving the platform wheel key from `versions.yaml` and appending `,llm` to the extra.\n13. **WandB not logging.** Ensure `wandb_config={\"enabled\": True}` is passed and either `api_key` is in the config or `WANDB_API_KEY` is set in the environment. Check logs for \"WandB initialized\" confirmation.\n14. **`No default specs found` for an action.** The skill bank model directory is missing `references/spec_template_\u003caction>.yaml`, or the packaged AutoML support check is missing `schemas/\u003caction>.schema.json`. Generate both during skill-bank maintenance and ship them with the plugin; do not expect `~/tao-core` to exist on the runtime machine.\n15. **`conda run` buffers output.** When running AutoML via `conda run -n tao_sdk python script.py`, all output is buffered until completion. Use `PYTHONUNBUFFERED=1 ~/miniconda3/envs/tao_sdk/bin/python script.py` for real-time output.\n\n---\n\n## Querying Experiment Status\n\nUse `query_status()` to check experiment progress from a separate process. This\nis the default status path for AutoML; do not parse launcher logs unless the user\nasks for log output or you are debugging missing/failed structured state.\n\n```python\nfrom tao_automl import query_status\n\nstatus = query_status(\"./my_experiment\")\n\n# Progress summary\np = status[\"progress\"]\nprint(f\"{p['completed']}/{p['total']} recs done, \"\n f\"{p['succeeded']} succeeded, {p['failed']} failed\")\n\n# Best config\nif status[\"best\"]:\n print(f\"Best: rec {status['best']['rec_id']}, \"\n f\"metric={status['best']['metric_value']}, \"\n f\"specs={status['best']['specs']}\")\n\n# Per-rec details\nfor rec in status[\"recommendations\"]:\n print(f\" Rec {rec['rec_id']}: {rec['status']} \"\n f\"metric={rec['metric_value']} specs={rec['specs']}\")\n\n# In-flight jobs\nfor job in status[\"active_jobs\"]:\n print(f\" Active: rec {job['rec_id']} job {job['job_id']}\")\n```\n\nThe function reads from the persisted state store (`\u003cworkspace>/.automl/`) and `active_jobs.json`. It is safe to call while the runner is active — no locking conflicts.\n\nThe `AutoML` class also exposes `get_status()` for in-process queries:\n\n```python\nautoml = AutoML(workspace=..., ...)\nstatus = automl.get_status()\n```\n\n---\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":23172,"content_sha256":"652858f4e456391bd4f469f5dfd2c2dda2ac4cbd260d9ed1001543127c294e83"},{"filename":"references/automl-common-pitfalls.md","content":"# AutoML common pitfalls\n\n- Do not expect `~/tao-core` at runtime. Schemas and templates must be packaged\n inside the model skill.\n- Do not infer dataset URIs from previous runs.\n- Do not precompute SDK-managed output paths; non-URI output values are routed\n by the SDK.\n- For SLURM, stage large datasets on Lustre rather than burning GPU allocation\n time on large S3 downloads.\n- For gated HuggingFace models, verify `HF_TOKEN` is set without reading it.\n- If all recommendations fail, stop and summarize the shared root cause instead\n of launching more trials.\n- Do not disable `automl_delete_intermediate_ckpt` by default. Keeping every\n trial can consume one full distributed checkpoint set per recommendation.\n- Do not bypass a retention-preflight failure by disabling cleanup unless the\n user has explicitly accepted external ownership and manual lifecycle\n management for every trial artifact.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":908,"content_sha256":"77ab6fb2a91869d888d8b31a44ac482be2042c8bbdfe1e64bb2cec33a4fd1c85"},{"filename":"references/automl-compression-literature.md","content":"# AutoML For Distill, Prune, And Quantize\n\nLiterature-backed policy for non-train TAO AutoML actions. Reviewed 2026-06-02.\n\n## Conclusion\n\nThe existing TAO AutoML algorithms are sufficient for first-class support across\n`distill`, `prune`, and `quantize` when the action exposes a valid schema and a\nmetric can be measured. Bayesian/BFBO search fits small single-shot compression\nspaces; Hyperband, ASHA, BOHB, DEHB, and PBT fit only when the action has a real\nfidelity axis such as epochs, checkpoints, pruning retrain epochs, calibration\nbudget, or population generations. LLM, hybrid, and autoresearch can plan search\nspaces, but must still delegate to schema-valid parameters and real metrics.\n\nNew algorithms are not a blocker for broad non-train action support. For best\naccuracy/latency/size tradeoffs, the most valuable future additions are\ncompression-aware candidate priors and multi-objective/hardware-aware search,\nnot a replacement for the existing HPO stack.\n\n## Action Policy\n\n`distill` is train-like when it optimizes over epochs and writes checkpoints.\nSearch temperature, hard/soft loss weights, teacher checkpoint choice, learning\nrate, augmentation, and schedule parameters with Bayesian/BFBO for small budgets\nor budgeted algorithms when an epoch/rung axis exists.\n\n`prune` is usually a single-shot or short retrain action. Use Bayesian/BFBO for\npruning threshold/ratio/granularity and retrain knobs. Use Hyperband/ASHA/BOHB/\nDEHB only when the action schema includes a meaningful partial-fidelity budget,\nfor example retrain epochs or progressive sparsity stages. Prefer downstream\n`eval_fn` when the prune job itself does not emit the task metric.\n\n`quantize` is often a calibration/search action with a discrete or mixed\nprecision space. Use Bayesian/BFBO for calibration size, batch count, precision\nmode, per-layer/group choices exposed by the schema, and QAT knobs. Use\nbudgeted algorithms only when lower-fidelity calibration/eval is representative\nenough to rank candidates. Use `eval_fn` for post-quantize evaluate/inference\nmetrics and latency measurements.\n\n## Evidence\n\n- Knowledge distillation is a direct compression method: Hinton, Vinyals, and\n Dean show that ensemble knowledge can be compressed into a smaller deployable\n model and improve a production acoustic model. This supports treating distill\n as a training-like action whose loss, teacher, and schedule knobs are valid\n AutoML parameters. Source: https://arxiv.org/abs/1503.02531\n- Hyperband frames HPO as adaptive resource allocation over configurations and\n reports large speedups over Bayesian baselines when a valid resource such as\n iterations, data, or features can be allocated. This supports budgeted\n algorithms only for non-train actions with a meaningful fidelity/resource\n field. Source: https://www.jmlr.org/papers/v18/16-558.html\n- ASHA is designed for large-scale parallel HPO with aggressive early stopping,\n so it is appropriate for action jobs that produce comparable partial metrics\n and are cheap enough to stop early. Source: https://arxiv.org/abs/1810.05934\n- BOHB combines Bayesian model guidance with Hyperband budgets and reports\n strong anytime and final performance across varied deep learning problems.\n This supports BOHB/DEHB for action spaces that have both search parameters\n and a valid budget axis. Source: https://proceedings.mlr.press/v80/falkner18a.html\n- AMC uses reinforcement learning for model compression policy search and shows\n improved compression quality and mobile speedups. This is evidence for future\n compression-aware candidate generators or priors when TAO needs more than\n generic HPO over exposed prune knobs. Source:\n https://openaccess.thecvf.com/content_ECCV_2018/html/Yihui_He_AMC_Automated_Model_ECCV_2018_paper.html\n- HAQ uses hardware feedback in the quantization loop and shows that optimal\n mixed-precision policies differ by architecture, hardware, and resource\n constraint. This is evidence for future hardware-aware or multi-objective\n AutoML when optimizing quantized accuracy, latency, energy, and size together.\n Source:\n https://openaccess.thecvf.com/content_CVPR_2019/html/Wang_HAQ_Hardware-Aware_Automated_Quantization_With_Mixed_Precision_CVPR_2019_paper.html\n- HAWQ uses Hessian sensitivity to choose mixed precision and fine-tuning order,\n showing that sensitivity-aware priors can improve quantization search quality.\n This argues for adding Hessian/sensitivity metadata as candidate priors rather\n than requiring a new optimizer before supporting quantize actions. Source:\n https://arxiv.org/abs/1905.03696\n- Network Slimming demonstrates channel sparsity and pruning can reduce model\n size and compute while preserving comparable accuracy, supporting schema-level\n search over sparsity/pruning controls plus downstream task evaluation. Source:\n https://arxiv.org/abs/1708.06519\n\n## Recommended Roadmap\n\n1. Ship broad non-train support with existing algorithms, gated by\n `schemas/\u003caction>.schema.json`, `spec_template_\u003caction>.yaml`, and metric\n extraction/evaluation hooks.\n2. Add optional multi-objective result handling for accuracy, latency, size,\n memory, and energy so compression workflows can report Pareto candidates\n instead of one scalar unless the user chooses a scalar objective.\n3. Add compression-aware priors: pruning sensitivity, layer saliency, Hessian or\n activation sensitivity, and hardware measurements. Feed them into existing\n Bayesian/BFBO/LLM/hybrid candidate generation.\n4. Add hardware-aware policy search only after the runner can collect reliable\n latency/energy/size signals per platform. RL/evolutionary compression search\n is useful, but only with that measurement loop in place.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":5733,"content_sha256":"e37ba414beced3c34c5135a157d2073f5b2c4e89b1ef65dd0421ff286eb867c1"},{"filename":"references/automl-examples.md","content":"# TAO AutoML Example Conversations\n\nExample user requests and the correct high-level agent response shape.\n\nLoad this file only when the compact `SKILL.md` points here for the current task. If this reference conflicts with `SKILL.md`, `skill_info.yaml`, schemas, or platform/model skills, the compact/current source wins.\n\n## Contents\n\n- User: \"Optimize `\u003cnetwork_arch>` training on my dataset\"\n- User: \"I want the real task metric, not the default proxy\"\n- User: \"Use the LLM to figure out the best hyperparameters\"\n- User: \"Run a fully autonomous search — let the AI figure everything out\"\n- User: \"Resume the experiment from yesterday\"\n- User: \"Switch to ASHA with 30 trials and track in WandB\"\n- User: \"I don't know what parameters to tune for `\u003cnetwork_arch>`\"\n\n## Example Conversations\n\n### User: \"Optimize `\u003cnetwork_arch>` training on my dataset\"\n\n```\nAgent: I'll read the model skill for `\u003cnetwork_arch>` first, then generate a fresh timestamped runner using its required datasets, spec overrides, metric, and recommended search space. I need:\n1. The dataset URI or URIs required by that model skill\n2. How many trials? (default: 10)\n\nUser: s3://bucket/data/train, 10 recs\n\nAgent: Launch review before I spend GPU budget:\n- model skill, network_arch, platform, image, GPU/node shape, workspace, dataset paths/counts, metric/direction, algorithm, budget, search bounds, estimated runtime, and automatic baseline eval job id/result path/metric\n- exact generated recommendation configs for the initial launch batch, created in a review-only step before any recommendation job is submitted\n\nUser: confirmed\n\n[executes runner.run(...)]\n```\n\n### User: \"I want the real task metric, not the default proxy\"\n\n```\nAgent: For a real task metric, I'll use the eval_fn hook described by the model skill. This adds per-rec cost, so I’ll adjust the budget if needed.\n[shows launch review with eval_fn cost and metric/direction]\nUser: confirmed\n[executes runner.run(metric=task_metric, direction=direction, eval_fn=model_specific_eval, ...)]\n```\n\n### User: \"Use the LLM to figure out the best hyperparameters\"\n\n```\nAgent: I'll use the LLM algorithm — it reasons about your network architecture and learns from each experiment.\nI need three things for the LLM brain:\n1. LLM endpoint URL (default: https://inference-api.nvidia.com)\n2. LLM model name (default: gcp/google/gemini-3.1-pro-preview)\n3. API key available in `AUTOML_LLM_API_KEY` or `NVIDIA_API_KEY`\n\nUser: endpoint \"https://inference-api.nvidia.com\", model \"gcp/google/gemini-3.1-pro-preview\"\n\nAgent: I verified the LLM endpoint/key env var is available, the selected train image is confirmed, and platform/data preflight passed. Launch review: 10 LLM-guided recs, metric/direction, search bounds, workspace, GPU/node shape, and estimated runtime.\nUser: confirmed\n[executes runner.run(automl_settings={\n \"algorithm\": \"llm\",\n \"llm_endpoint\": \"https://inference-api.nvidia.com\",\n \"llm_model\": \"gcp/google/gemini-3.1-pro-preview\",\n \"llm_api_key\": os.environ[\"AUTOML_LLM_API_KEY\"],\n \"session_id\": session_id,\n ...\n})]\n```\n\n### User: \"Run a fully autonomous search — let the AI figure everything out\"\n\n```\nAgent: I'll use the autoresearch algorithm. It will:\n1. Retrieve tuning knowledge for your network\n2. Propose modifications, pre-screen them, verify before launch\n3. Make keep/discard decisions with reasoning after each experiment\n4. Automatically stop when it detects diminishing returns\n\nI'll cap it at 30 experiments. I need:\n- Dataset URI\n- LLM endpoint (default: https://inference-api.nvidia.com)\n- LLM model (default: gcp/google/gemini-3.1-pro-preview)\n- LLM API key in `AUTOML_LLM_API_KEY` or `NVIDIA_API_KEY`\n\nUser: dataset s3://bucket/data, endpoint https://inference-api.nvidia.com, model gcp/google/gemini-3.1-pro-preview\n\n[shows launch review with 30-experiment cap, LLM config source, data/image/platform checks, exact generated configs for the initial launch batch, and estimated runtime]\nUser: confirmed\n\n[executes runner.run(automl_settings={\n \"algorithm\": \"autoresearch\", \"automl_max_experiments\": 30,\n \"llm_endpoint\": \"https://inference-api.nvidia.com\",\n \"llm_model\": \"gcp/google/gemini-3.1-pro-preview\",\n \"llm_api_key\": os.environ[\"AUTOML_LLM_API_KEY\"],\n \"session_id\": session_id,\n ...\n})]\n```\n\n### User: \"Resume the experiment from yesterday\"\n\n```\nAgent: Resuming. The runner will poll any jobs that were in flight when we stopped, report their results, then show a resume review before submitting any new recommendation jobs.\n[shows resume review with completed/failed/in-flight recs, remaining budget, metric state, image/platform/data checks, and ETA]\nUser: confirmed\n[executes runner.run(\n ...,\n workspace_path=\"./my_experiment/run_\u003ctimestamp>\",\n automl_settings={**automl_settings, \"session_id\": resolved_session_id},\n resume=True,\n)]\n```\n\n### User: \"Switch to ASHA with 30 trials and track in WandB\"\n\n```\nAgent: Switching to ASHA — parallel + early-stopping. I’ll first check the model skill for any warning that short trials are dominated by startup, validation, or checkpoint cost, then generate a fresh runner with WandB tracking.\n[shows launch review with ASHA rungs, max concurrent jobs, WandB project, search bounds, and estimated runtime]\nUser: confirmed\n[executes runner.run(\n automl_settings={\n \"algorithm\": \"asha\",\n \"automl_max_epochs\": 9,\n \"session_id\": session_id,\n ...,\n },\n wandb_config={\"enabled\": True, \"project\": \"tao-hpo\"},\n)]\n```\n\n### User: \"I don't know what parameters to tune for `\u003cnetwork_arch>`\"\n\n```\nAgent: I can generate a configuration from your goal description. What are you trying to achieve?\n\nUser: Maximize my task metric on a small custom dataset with 1 GPU.\n\nAgent: Based on your constraints, I recommend:\n- Algorithm: bayesian (sample-efficient for small budgets)\n- Parameters: from the model skill's AutoML / HPO Notes and the generated schema\n- Budget: 12 recs\n- Ranges: from the model skill and user constraints\n[uses NLConfigGenerator, shows launch review with generated config and estimated runtime]\nUser: confirmed\n[executes runner.run with the generated config]\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":6183,"content_sha256":"0e20ab33ab272285fb32eb7f13f91d2965e76b6e4e300d116ec55a4f32cccb45"},{"filename":"references/automl-intent-algorithms.md","content":"# TAO AutoML Intent And Algorithms\n\nUser intent parsing, train-vs-AutoML policy, algorithm selection, and the algorithm decision tree.\n\nLoad this file only when the compact `SKILL.md` points here for the current task. If this reference conflicts with `SKILL.md`, `skill_info.yaml`, schemas, or platform/model skills, the compact/current source wins.\n\n## Contents\n\n- network_arch is NOT a runner.run() arg anymore; resolve the user's model\n- request to a packaged skill_dir, then read network_arch from that skill's\n- metadata.\n- Step 2: Select Algorithm\n- Classical Algorithms\n- LLM/Agentic Algorithms (NEW)\n- Quick Reference: Algorithm Decision Tree\n\n## Step 1: Parse User Intent\n\nDefault to a quick-start run unless the user explicitly asks to customize AutoML or agrees to a customization offer. Do not present algorithm, budget, or search-space choices as required inputs for a normal \"run AutoML\" request.\n\nAny workflow/application that reaches an AutoML-capable model action must\nconsult the selected model's `automl_enabled` metadata. If it is `true`, use\nthis AutoML workflow for train-like optimization unless the run/workflow setting\nhas `automl_policy: off` or the user explicitly asks for a plain single action\nrun. Treat `automl_policy: on` as the default enabled state for train and\ndistill. For single-shot compression actions such as prune and quantize, use\nAutoML when the user asks to optimize/tune the action or when the workflow\nprofile explicitly sets `automl_policy: on`.\n\nExtract these fields for a default run:\n\n| Field | Required | Example | How to get it |\n|---|---|---|---|\n| `requested_model` | Yes | `\"cosmos-rl\"` or `\"tao-finetune-cosmos-reason\"` | User states the model, model family, or network alias. |\n| `model_skill` | Yes | `\"tao-finetune-cosmos-reason\"` | Resolve `requested_model` to a packaged directory under `skills/models/` by using model metadata. Do not assume `network_arch` is the directory name. |\n| `network_arch` | Yes | `\"cosmos-rl\"` | Read from `\u003cskill_dir>/references/skill_info.yaml` after resolving `model_skill`. |\n| `action` | Yes | `\"train\"`, `\"distill\"`, `\"prune\"`, `\"quantize\"` | Default to `train` only for train-stage requests. For compression requests, use the named action. |\n| `platform` | Yes | `\"brev\"`, `\"slurm\"`, `\"local-docker\"`, `\"kubernetes\"` | After the user confirms they want AutoML, discover the execution platforms from the installed platform skills (tao-run-on-docker / -slurm / -kubernetes / -brev, plus any external one); on a runtime that surfaces only the core router skills, read `skills/platform/tao-run-on-*/SKILL.md` frontmatter. Ask them to choose from those platforms. |\n| action dataset/checkpoint inputs | Yes | `\"s3://bucket/data/subset\"`, `\"/lustre/fsw/tao_datasets/\u003cmodel>/train\"`, `prune.checkpoint=/...`, or `quantize.model_path=/...` | User provides a root URI/path, exact spec-key paths, parent job IDs, or the model skill declares a default profile for this exact network/action/use case. |\n| `eval_dataset_uri` or direct eval spec paths | Model-dependent | `\"s3://bucket/data/eval\"`, `\"/lustre/fsw/tao_datasets/\u003cmodel>/eval\"`, or `custom.val_dataset.media_path=/...` | Ask only if the model skill's Per-Action Dataset Requirements require an eval/validation source and no default profile supplies it. |\n| `image` | Yes | `\"nvcr.io/...\"` | Resolve the default with `scripts/resolve_tao_image.py --model \u003crequested_model> --action \u003caction>`, show it to the user, and require confirmation or `image=\u003coverride>` before creating the AutoML runner. |\n| `metric` | No | `\"\u003cmetric_name>\"` | Use the model skill recommendation or ask if unclear. Do not choose model-specific metrics from this AutoML skill. |\n| `direction` | No | `\"minimize\"` or `\"maximize\"` | **Only needed if your metric name doesn't contain `\"loss\"` AND you want to minimize, or contains `\"loss\"` AND you want to maximize.** Otherwise the implicit \"contains 'loss' → minimize, else maximize\" rule applies. |\n| `skill_dir` | Yes | `\"\u003cbank-root>/skills/models/tao-train-dino\"` | Absolute path to the resolved model skill directory in the skill bank. Passed explicitly to `AutoMLRunner(skill_dir=...)` — no env-var fallback. |\n| `long_running_enabled` | Yes | `true` | Ask during launch intake. If enabled, keep the agent attached and emit status until completion. Default: enabled. |\n| `status_interval_minutes` | Yes | `5` | Ask during launch intake. Default: 5 minutes. |\n| required credentials | Platform/model-dependent | `SLURM_USER`, `SLURM_HOSTNAME`, `SSH_KEY_PATH` or `SSH_AUTH_SOCK`, `HF_TOKEN` | First read the chosen platform skill's `## Credentials` section and `references/skill_info.yaml` (required_credentials / credential_groups), satisfy required credential groups, then add selected-model credentials. Do not ask for unrelated platform credentials. |\n| compute shape | Model-dependent | `num_gpus=4`, `num_nodes=1` | Ask only for model-required hardware fields that are not provided by the platform/default profile. |\n| `llm_endpoint` / `base_url` | **Yes** (for `llm`/`hybrid`/`autoresearch`) | `\"https://inference-api.nvidia.com\"` | Resolve from user input or `AUTOML_LLM_ENDPOINT`; pass explicitly in `automl_settings`. |\n| `llm_model` / `model` | **Yes** (for `llm`/`hybrid`/`autoresearch`) | `\"gcp/google/gemini-3.1-pro-preview\"` | Resolve from user input or `AUTOML_LLM_MODEL`; pass explicitly in `automl_settings`. |\n| `llm_api_key` / `api_key` | **Yes** (for `llm`/`hybrid`/`autoresearch`) | `\"nvapi-...\"` or `\"sk-...\"` | Resolve from `AUTOML_LLM_API_KEY` or `NVIDIA_API_KEY`. Do not print or log the value; if unavailable, stop for credential setup instead of falling back silently. |\n\nUse these quick-start AutoML defaults without asking:\n\n| Field | Default |\n|---|---|\n| `algorithm` | `bayesian`, unless the user/model default profile explicitly selects another algorithm |\n| `automl_max_recommendations` | model/workflow default if declared, otherwise `10` |\n| `automl_hyperparameters` | `None` so AutoML uses dataclass-schema params with `automl_enabled=true` |\n| `custom_param_ranges` | `None` so ranges/options/defaults come from the generated dataclass schema |\n| `long_running_enabled` | `true` |\n| `status_interval_minutes` | `5` |\n\nIf any required field is missing, ask the user. Do NOT guess dataset paths, skill bank paths, credentials, or hardware that the model skill marks as required.\n\nWhen asking for missing AutoML launch inputs, use a first-time-user friendly\nprompt. Do not say only \"train dataset root\" / \"eval dataset root\", and do not\nsay \"attached monitoring every 5 minutes\" without explaining it. Include:\n\n- platform choices;\n- root-mode dataset examples for the selected platform;\n- direct spec-parameter mode as an equal option;\n- model-required spec keys from the model skill's Per-Action Dataset\n Requirements table;\n- resolved train container image and the option to override it with\n `image=\u003coverride>`;\n- monitoring meaning and cadence choices.\n\nBefore generating an AutoML script, verify platform access and dataset\nvisibility using the shared launch preflight. For SLURM, that means\npasswordless SSH to at least one login host and remote `test -e` checks for\neach required annotation/media path. If preflight fails, stop with remediation\nsteps instead of creating a runner that will immediately fail.\n\nIf the selected model skill's Per-Action Dataset Requirements or Typical Spec\nOverrides show train/evaluate/inference inputs that come from a prior\n`dataset_convert` action, run that conversion before calling\n`AutoMLRunner.run`. Use the `dataset_convert` action's own `container_image`\nwhen it overrides the model default, persist the conversion output under the\ncurrent run's results root, verify every required converted artifact named by\nthe model skill exists, and pass those current-run converted paths in\n`spec_overrides`. Do not reuse stale conversion paths from another AutoML\nalgorithm/run folder.\n\nAlso verify container image confirmation using the shared launch preflight.\nAutoML launches real action jobs for each recommendation, so the confirmed\naction image must be passed into `AutoMLRunner.run(..., image=chosen_image, ...)`\nor into the SDK adapter's `create_job(..., image=chosen_image, ...)`. Do not\nrely on an implicit default after the user has chosen a platform and dataset.\n\nAlso run any model-specific annotation content checks documented by the model\nskill. Missing required annotation fields are a preflight failure, not an\nAutoML recommendation failure.\n\nBefore submitting any recommendation job, generate the initial recommendation\nbatch in a review-only step and show the exact configs to the user together\nwith metric, direction, expected runtime, and effective-batch checks. Do not\nreplace concrete configs with only search bounds.\n\nAfter all platform, image, credential, data, and model preflight checks pass,\nsubmit the model's evaluate action once against the selected validation/eval\ndata before submitting AutoML recommendations. This is the automatic baseline\neval job for the run. Show its job id, result path, and metric value in the\nlaunch review, then ask for confirmation before spending training budget on\nrecommendations. If the eval job cannot run because the model has no packaged\nevaluate action, validation data is missing, or the eval job fails, stop and\nsurface the blocker. Proceed without this baseline only when the user\nexplicitly accepts a proxy-metric run with no impact baseline.\n\nIf the workflow starts from scratch and has no compatible checkpoint to\nevaluate, run a minimal default-configuration training job outside the AutoML\nrecommendation budget, resolve its exact epoch/step checkpoint, and evaluate\nthat checkpoint for the baseline. Then use the same evaluator task metric via\n`eval_fn` for every recommendation and `final_eval_fn` for the winner. An empty\ncheckpoint value or container directory is not a valid baseline model.\n\n**Customization gate:** After the required quick-start fields are resolved, you may briefly offer customization. If the user declines or does not ask for it, proceed with the defaults above. If the user chooses customization, then present the additional options below.\n\nCustomization-only fields:\n\n| Field | Example | Notes |\n|---|---|---|\n| `algorithm` | `bayesian`, `asha`, `hyperband`, `bohb`, `llm`, `hybrid`, `autoresearch` | Present the algorithm guide only in customization mode or when the user names an algorithm. |\n| `max_recommendations` | `5`, `10`, `20` | Explain that each recommendation is a real training job. |\n| `long_running_enabled` | `false` | Only use false when the user explicitly does not want the agent to keep monitoring. |\n| `status_interval_minutes` | `5`, `10`, `15` | Already asked during launch intake; customize only if the user wants a different cadence. |\n| `automl_hyperparameters` | `[\"train.optm_lr\", \"train.epoch\"]` | List choices from the generated schema JSON, not from hand-written guesses. |\n| `custom_param_ranges` | `{\"train.optm_lr\": {\"valid_min\": 1e-6, \"valid_max\": 1e-4}}` | Validate against schema type/range/options before using. |\n| `llm_endpoint`, `llm_model`, `llm_api_key` | `https://inference-api.nvidia.com`, `gcp/google/gemini-3.1-pro-preview`, `nvapi-...` | Required only when the selected algorithm is `llm`, `hybrid`, or `autoresearch`. Resolve credentials from env/secret files; do not echo keys. |\n\n**MANDATORY: Read the generated dataclass schema before configuring AutoML.**\n\nFor the selected model/action, read:\n\n- `${TAO_SKILL_BANK_PATH:-~/tao-skill-bank}/skills/models/\u003cmodel_skill>/schemas/\u003caction>.schema.json`\n- `${TAO_SKILL_BANK_PATH:-~/tao-skill-bank}/skills/models/\u003cmodel_skill>/schemas/manifest.json`\n\nAutoML is enabled by the model skill, but it can run only when\n`schemas/\u003caction>.schema.json` is packaged with the plugin and valid for the\nselected model/action. Do not fall back to hand-written model\nnotes, old runner scripts, or a local `~/tao-core` checkout for AutoML\nparameter metadata. If the action schema is missing, stop and report that AutoML\nis enabled for that model but not runnable for that action until the schema is\ngenerated and shipped in the skill bank.\n\nUse the schema JSON as the source of truth for `automl_default_parameters`,\n`automl_disabled_parameters`, per-parameter defaults, ranges, enums,\n`option_weights`, `math_cond`, `depends_on`, `parent_param`, and `popular`.\n\nWhen `automl_hyperparameters=None`, the runner automatically discovers all\nparams marked `automl_enabled=True` in the network's generated schema. Each\nnetwork has its own set; never hardcode them in this workflow skill.\n\nQuick-start runner shape:\n\n```python\n# network_arch is NOT a runner.run() arg anymore. It is read from the model\n# metadata at skill_dir, which was passed to AutoMLRunner(...) at construction.\nresult = runner.run(\n train_dataset_uri=TRAIN_DATASET_URI,\n automl_settings={\n \"algorithm\": \"bayesian\",\n \"metric\": metric,\n \"automl_max_recommendations\": 10,\n \"session_id\": session_id,\n },\n automl_hyperparameters=None, # use schema params marked automl_enabled=true\n custom_param_ranges=None, # use schema ranges/options/defaults\n spec_overrides={...}, # from model skill + dataset requirements\n workspace_path=f\"./automl/{TIMESTAMP}\",\n)\n```\n\nCustomization runner additions:\n\n```python\nresult = runner.run(\n ...,\n automl_hyperparameters=selected_param_names,\n custom_param_ranges={\n \"\u003cparam_name>\": {\"valid_min\": min_value, \"valid_max\": max_value},\n \"\u003ccategorical_param>\": {\n \"valid_options\": [\"option_a\", \"option_b\"],\n \"option_weights\": [0.7, 0.3],\n },\n },\n)\n```\n\n**MANDATORY LLM configuration for LLM-based algorithms (`llm`, `hybrid`, `autoresearch`):**\n\nWhen the user requests or customizes into an LLM-powered algorithm, resolve ALL THREE of the following before generating the script. Do not ask for these on default `bayesian` quick-start runs.\n\n1. **`llm_endpoint`** — user input -> `AUTOML_LLM_ENDPOINT` -> `https://inference-api.nvidia.com`\n2. **`llm_model`** — user input -> `AUTOML_LLM_MODEL` -> `gcp/google/gemini-3.1-pro-preview`\n3. **`llm_api_key`** — `AUTOML_LLM_API_KEY` -> `NVIDIA_API_KEY` -> declared local secret file when allowed. Do not print the value.\n\nIf the runner does not receive valid LLM settings, the LLM brain may silently fall back to random sampling — wasting GPU budget on random configs instead of intelligent ones. There is no error message; the only clue is \"LLM call failed... Falling back to random\" in the logs.\n\n**MANDATORY: Read the model skill before generating the script.**\n\nAutoML runs the selected TAO action. Before generating any AutoML script, read `\u003cbank-root>/skills/models/\u003cmodel_skill>/SKILL.md` (where `\u003cbank-root>` is wherever the agent loaded this SKILL.md from). The model skill contains all model-specific knowledge:\n\n- **Training Requirements** — dataset type, formats, monitoring metric, required dataset URIs to prompt for, required user prompts (data format, num_classes, etc.), and mandatory `spec_overrides`. Prompt the user for every required field. Apply mandatory spec_overrides exactly.\n- **Per-Action Dataset Requirements** — table mapping each action to its spec keys, data source, expected files, and whether the field is a list. Use this table to construct the correct data source `spec_overrides` for the requested action. If the model's Typical Spec Overrides mark data sources as \"mandatory\", construct them from this table and the user's dataset URIs.\n- **Typical Spec Overrides** — per-action override suggestions (train, evaluate, export, inference, etc.) extracted from SDK notebooks. Use these as the starting point for `spec_overrides` and suggest them to the user. When overrides are marked \"mandatory data sources\", they MUST be included — the runner cannot auto-resolve them. Merge with any other mandatory overrides from Training Requirements.\n- **AutoML / HPO Notes** — metric, direction, model-specific constraints, and any guidance that narrows or overrides the generated schema. Hyperparameter names/ranges/defaults come first from `schemas/\u003caction>.schema.json`.\n- **Error Patterns** — common training failure modes that apply to AutoML recs too.\n\nDo NOT hardcode model-specific knowledge in the AutoML script without reading the model skill first. Each network has different requirements.\n\n**MANDATORY: No model-specific constants in this AutoML skill.**\n\nThe AutoML skill must not define model-specific hyperparameter names, ranges, defaults, metric names, dataset layouts, archive names, class-count rules, spec override keys, container images, checkpoint quirks, or custom metric regexes. Hyperparameter metadata belongs in `\u003cbank-root>/skills/models/\u003cmodel_skill>/schemas/\u003caction>.schema.json`; model-specific runtime guidance belongs in the model skill's **Training Requirements**, **Typical Spec Overrides**, **AutoML / HPO Notes**, and **Error Patterns** sections. This skill may describe how to read and apply those sources, but not the concrete per-model values.\n\n**MANDATORY: Timestamped workspace folders.**\n\nALWAYS generate `workspace_path` with a timestamp suffix. Running the same script twice without a timestamp overwrites the previous experiment. Pattern:\n\n```python\nfrom datetime import datetime\nTIMESTAMP = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\nworkspace_path = f\"./experiment_name/{TIMESTAMP}\"\n```\n\nDo NOT use a flat path like `workspace_path=\"./my_experiment\"`. The user should never have to manually delete old workspace folders.\n\n**MANDATORY: Fresh runner per new AutoML request, after preflight passes.**\n\nEvery new user request to run AutoML MUST create a new runner script and launch a new AutoML job, even if an older runner script for the same network/algorithm already exists. This freshness rule starts only after platform and dataset preflight passes. Existing runner files and logs may be read only as references for dataset URIs, credentials patterns, and proven fixes; do not reuse them as the execution target for a new request.\n\nUse a unique timestamp in the new runner filename, log filename, PID filename, SDK `state_file`, and `workspace_path`. Derive path components from the requested `network_arch` and `algorithm`; do not hardcode any model or algorithm name unless it is the actual requested value.\n\n```python\nimport re\n\ndef slug(value):\n return re.sub(r\"[^A-Za-z0-9_.-]+\", \"_\", str(value)).strip(\"_\").lower()\n\nTIMESTAMP = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\nRUN_NAME = f\"{slug(network_arch)}_{slug(algorithm)}\"\nrunner_path = f\"automl_runs/run_{RUN_NAME}_{TIMESTAMP}.py\"\nlog_path = f\"automl_runs/{RUN_NAME}_{TIMESTAMP}.log\"\npid_path = f\"automl_runs/{RUN_NAME}_{TIMESTAMP}.pid\"\nstate_file = f\"tao_session_state_{RUN_NAME}_{TIMESTAMP}.json\"\nworkspace_path = f\"./automl_runs/{RUN_NAME}/{TIMESTAMP}\"\n```\n\nOnly resume an existing runner/workspace when the user explicitly asks to resume, continue, recover, or inspect an existing experiment. If the user says \"run automl\" or asks for a new AutoML run, treat it as a fresh job.\n\n**Best-practice on metric choice**:\n\n- Training loss is cheap, but can overfit on small fine-tuning datasets. Prefer the model skill's recommended validation or task metric when available.\n- If the model skill recommends a validation proxy, also apply the model skill's required validation-related `spec_overrides` so the metric is actually emitted.\n- A real task metric via `eval_fn` is often the most honest but adds per-rec cost. Use it when the model skill says log-based metrics are insufficient or the user explicitly wants downstream evaluation.\n- For AutoML runs with a runnable evaluate action and validation/eval data, run\n the automatic baseline eval job after preflight and before recommendations,\n then include that baseline metric in the final comparison.\n\n**Checkpoint / resume behavior**:\n\n- Resume-based algorithms (`hyperband`, `asha`, `bohb`, `dehb`, `pbt`, `hyperband_es`) must resume from the checkpoint for the stopped rung/generation epoch or step, not a generic `*_latest.*` file.\n- The runner records the intended `resume_from_epoch` / `resume_from_step` on promoted recommendations and resolves the model-specific checkpoint path through the SDK checkpoint resolver. Do not patch runner scripts to guess names like `model_latest.pth`.\n- If the intended checkpoint is epoch 1, prefer artifacts such as `epoch_1`, `epoch_001`, `model_epoch_001.pth`, `model_epoch_000_step_*.pth` when the trainer writes zero-indexed epoch files, or a checkpoint directory like `checkpoints/epoch_1`.\n- Use a `latest` checkpoint only when the requested action explicitly has no epoch/step target. If no epoch/step-specific resume artifact exists, report the model as blocked and fix the model/skill checkpoint metadata rather than silently resuming from latest.\n\n---\n\n## Step 2: Select Algorithm\n\n### Classical Algorithms\n\nThese require no external services — they use statistical/mathematical methods to pick hyperparameters.\n\n| Algorithm | Use when | Typical budget | How it works |\n|---|---|---|---|\n| `bayesian` | **Default choice.** Small budgets, few parameters. | 5–20 recs | Builds a Gaussian Process model of metric vs. hyperparameters. Sequential — waits for each result before proposing the next, so it learns fast but can't parallelize. |\n| `bfbo` | Alternative to bayesian with different acquisition function. | 5–20 recs | UCB-based Bayesian optimization with local penalization. Good when bayesian gets stuck. |\n| `hyperband` | Large search spaces, many parameters. | 20–50+ recs | Trains many configs cheaply for a few epochs, keeps the best, trains longer. Requires `automl_max_epochs` and `automl_reduction_factor`. |\n| `hyperband_es` | Hyperband + early stopping. | 20–50+ recs | Like hyperband but adds early-stop thresholds to halt clearly bad runs sooner. |\n| `asha` | Async variant of hyperband, supports parallel execution. | 10–30 recs | Same successive-halving idea as hyperband, but trials run concurrently. Best when you have many GPUs. Uses `automl_max_concurrent`. |\n| `bohb` | Best of both — Bayesian intelligence + Hyperband efficiency. | 15–40 recs | Combines KDE-based model (like Bayesian) with Hyperband's multi-fidelity scheduling. Good all-rounder for medium budgets. |\n| `dehb` | Evolutionary + multi-fidelity. | 15–40 recs | Differential evolution mutations + hyperband scheduling. Good for complex search spaces with many interacting parameters. |\n| `pbt` | Dynamic schedules — mutates hyperparameters during training. | population_size × generations | Population-Based Training. Starts N configs in parallel, periodically copies weights from winners and perturbs their hyperparameters. Best for long runs where hyperparameters should change over time (e.g. learning rate schedules). Final handoff selects the best member at the largest observed training budget; earlier lower-budget metrics are promotion evidence, not the final checkpoint choice. |\n\n### LLM/Agentic Algorithms (NEW)\n\nThese use a large language model to reason about hyperparameter choices. They require an LLM endpoint (NVIDIA NIM, OpenAI, vLLM, Ollama, etc.) and the `openai` Python package.\n\n| Algorithm | Use when | Typical budget | How it works |\n|---|---|---|---|\n| `llm` | Domain knowledge matters more than statistical rigor. | 5–20 recs | An LLM proposes hyperparameter configs based on the search space schema, experiment history, and its training knowledge. Falls back to random sampling on LLM failure. Sequential like bayesian. |\n| `hybrid` | You want the LLM to orchestrate multi-phase optimization. | 10–50 recs | An LLM strategist plans optimization phases over model-skill parameters. Each phase uses a classical sub-algorithm. Stops when the strategist detects diminishing returns. |\n| `autoresearch` | Fully autonomous agent loop. | 10–50 recs | The most powerful mode. Combines: (1) RAP knowledge retrieval about the network, (2) LLM-proposed spec modifications, (3) training-free pre-screening of candidates, (4) multi-stage verification (pre-launch + post-result), (5) keep/discard reasoning. Automatically stops on budget exhaustion or consecutive failures. |\n\n**Default to `bayesian` unless** the user specifically asks for something else, has a large GPU budget, or needs early-stopping on cheap intermediate metrics (ASHA / hyperband).\n\n**Use `llm` / `hybrid` / `autoresearch` when** the user wants LLM-guided search, has an API key for NVIDIA NIM or OpenAI, and wants richer reasoning about why certain hyperparameters are chosen.\n\n**Compression action policy (`distill`, `prune`, `quantize`):**\n\n- `distill` is train-like when it runs for epochs and emits checkpoints. Use\n `bayesian` by default; use `hyperband`, `asha`, `bohb`, `dehb`,\n `hyperband_es`, or `pbt` only when the action schema contains epoch/checkpoint\n budget fields and the model skill says short-rung metrics are meaningful.\n- `prune` is often a single-shot compression transform followed by optional\n retrain/evaluate. Default to `bayesian`, `bfbo`, `llm`, `hybrid`, or\n `autoresearch` over prune ratios, thresholds, skip/layer choices, and retrain\n knobs exposed by the action schema. Use `eval_fn` to run evaluate/retrain when\n the prune action itself does not emit the final task metric.\n- `quantize` is usually a single-shot calibration/export transform. Default to\n `bayesian`, `bfbo`, `llm`, `hybrid`, or `autoresearch` over calibration batch\n count, backend/layer/skip choices, precision policies, and calibration data\n knobs exposed by the schema. Treat latency/size/accuracy as a multi-objective\n metric policy in `eval_fn` until a dedicated Pareto optimizer is added.\n\n**Caveat on ASHA with expensive checkpoints:** ASHA's whole point is running many configs cheaply for early rungs, then promoting survivors. If the model skill warns that checkpoints, validation, or startup cost dominate short trials, prefer the model skill's recommended algorithm instead of assuming ASHA will be cheaper.\n\n---\n\n## Quick Reference: Algorithm Decision Tree\n\n```\nIs your budget tiny (≤10 recs)?\n YES → bayesian\n NO ↓\n\nDo you have an LLM API key and want AI-guided search?\n YES → Do you want full autonomy? → autoresearch\n Just LLM proposals? → llm\n LLM orchestrating phases? → hybrid\n NO ↓\n\nDo you need parallel execution?\n YES → asha (or bohb for smarter sampling)\n NO ↓\n\nIs your search space large (10+ parameters)?\n YES → hyperband or dehb\n NO ↓\n\nDo hyperparameters need to change during training (schedules)?\n YES → pbt\n NO → bayesian (safe default)\n```\n\n---\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":26694,"content_sha256":"4ad343828a0b6e433ea0ae57682b6ff22cd96de7daf9f5397a20c4f3e64bda2f"},{"filename":"references/automl-preflight-concepts.md","content":"# TAO AutoML Preflight And Concepts\n\nPreflight, platform prerequisites, support checks, and core AutoML concepts from the pre-refactor guide.\n\nLoad this file only when the compact `SKILL.md` points here for the current task. If this reference conflicts with `SKILL.md`, `skill_info.yaml`, schemas, or platform/model skills, the compact/current source wins.\n\n## Contents\n\n- Preflight\n- Prerequisites\n- Verify LLM features (optional)\n- Verify WandB (optional)\n- Concepts: What is TAO AutoML?\n- Quick Support Queries\n\n# TAO AutoML Skill\n\nThis is a skill-bank **workflow** skill at `skills/applications/tao-run-automl/`. The agent\ndiscovers it by reading this file directly (or via the `tao-skills` plugin).\n\nRun automated hyperparameter optimization (HPO) for any TAO network/action. The agent uses `AutoMLRunner` — a single interface that manages the full loop: generate hyperparameter recommendations, launch action jobs, extract metrics, and feed results back to the optimizer.\n\nThe runner is **platform-agnostic** — it takes any object implementing the standard SDK shape (`create_job`, `get_job_status`, `get_job_logs`, `get_failure_analysis`) and calls those methods. Pick whichever SDK matches where you want jobs to run; the runner doesn't care:\n\n| SDK | Best for AutoML |\n|---|---|\n| `BrevSDK` | Cost-tuned sweeps on Brev instances (single-instance per rec, multi-GPU OK). Multi-credential / multi-workspace accounts must pass `cloud_cred_id=` and `workspace_group_id=` to `create_job` — see `skills/platform/tao-run-on-brev/SKILL.md`. |\n| `SlurmSDK` | Large sweeps on shared HPC clusters with queue/quota |\n| `KubernetesSDK` | Sweeps on EKS / GKE / AKS / on-prem clusters with the NVIDIA GPU Operator |\n| `DockerSDK` | Local debugging or single-host sweeps with a few recs |\n| `VirtualEnvSDK` | Containerless model skills (`container_image: null`, `execution.type: python_script`) that run trials as local Python processes. Takes constructor args, not credentials: `VirtualEnvSDK(venv_path=..., work_dir=...)` — see `automl-runner-configuration.md` § \"VirtualEnvSDK (containerless venv runs)\". |\n\nMulti-node per rec works on SLURM and K8s (each rec is an N-node distributed training job). Brev and local Docker are single-host per rec — multi-GPU within one host still works (`gpu_count > 1`), but you can't parallelize one rec across multiple hosts.\n\n## Preflight\n\nThis skill needs `nvidia-tao-automl`, which pulls `nvidia-tao-sdk` as a\ntransitive dependency. Both packages are pinned in `versions.yaml` under\n`wheels:`. Resolve the selected platform extra with\n`scripts/resolve_versions_key.py`:\n\n```bash\npython -c \"import tao_automl\" 2>/dev/null || {\n SB=\"${TAO_SKILL_BANK_PATH:-~/tao-skill-bank}\"\n echo \"MISSING: nvidia-tao-automl not installed. Pick the platform extra you need:\"\n echo \" pip install \\\"$($SB/scripts/resolve_versions_key.py wheels.tao_automl_slurm)\\\" # on-prem SLURM cluster\"\n echo \" pip install \\\"$($SB/scripts/resolve_versions_key.py wheels.tao_automl_kubernetes)\\\" # K8s (EKS / GKE / on-prem)\"\n echo \" pip install \\\"$($SB/scripts/resolve_versions_key.py wheels.tao_automl_docker)\\\" # local Docker daemon\"\n echo \" pip install \\\"$($SB/scripts/resolve_versions_key.py wheels.tao_automl_brev)\\\" # Brev GPU instances\"\n echo \" pip install \\\"$($SB/scripts/resolve_versions_key.py wheels.tao_automl_all)\\\" # all platforms\"\n echo \" (append ,llm or ,wandb to the extra only when needed)\"\n exit 1\n}\n```\n\n(For local development against a checkout: `pip install -e '~/tao-run-automl[brev]'` from the cloned repo.)\n\nIf missing, the agent prompts the user to authorize the install via Bash, then re-runs the preflight before continuing.\n\n## Prerequisites\n\nBefore running AutoML:\n\n1. **Shared launch preflight**: Run the `tao-launch-workflow` intake pattern first. AutoML must not create runner files, workspaces, state files, logs, compatibility shims, or install dependencies until the selected platform's credentials, access check, dataset visibility, model credentials, container image confirmation, and compute shape are satisfied. This prevents wasting the AutoML budget on fake recommendation failures caused by SSH, storage, image, or credential setup.\n2. **SDK credentials**: env vars read from the session environment (export them in your shell before launching, or source a user-approved env file with `set -a; source /path/to/.env; set +a`). Required env vars depend on which SDK you choose — see each platform's SKILL.md (`skills/platform/tao-run-on-brev`, `skills/platform/tao-run-on-slurm`, `skills/platform/tao-run-on-kubernetes`, `skills/platform/tao-run-on-local-docker`). Before asking for credentials, read the chosen platform skill's `## Credentials` section and `references/skill_info.yaml` (required_credentials / credential_groups). Ask only for the credentials listed there. For example, SLURM needs SLURM credentials and not Brev or S3 credentials; Kubernetes and local Docker do not need SLURM or Brev credentials. Ask S3 credentials only when the selected platform and dataset/result URIs use `s3://`. For container pulls: `NGC_KEY`. The agent never reads values — only checks presence with `[ -n \"$VAR_NAME\" ]`. Construct the credential-reading SDKs with no arguments — e.g., `BrevSDK()`, `SlurmSDK()`, `KubernetesSDK()`, or `DockerSDK()`. `VirtualEnvSDK` is the exception: it reads no credentials but requires `venv_path=` and must be given an explicit `work_dir=`.\n\n **Verify the SDK actually constructs before generating runner files.** A successful `import tao_automl` does not prove the selected AutoML platform extra is installed. Instantiate the SDK class for the chosen platform, for example:\n\n ```bash\n python -c \"from tao_sdk.platforms.docker import DockerSDK; DockerSDK(); print('OK')\"\n ```\n\n Substitute the matching import for `brev`, `slurm`, or `kubernetes`. Catch a missing platform extra during preflight, before spending time on an image pull or recommendation. Local Docker launches with the host UID:GID when a writable results bind is present, so any model dataloader that writes through a spec path must point that path at a writable mount.\n\n **Baseline eval when training from scratch.** The required baseline is scoped to runs that have a starting checkpoint. With no pretrained, parent, or resume checkpoint, there is no starting model to evaluate, so the baseline is inapplicable rather than failed:\n\n - Record the baseline as unavailable with the reason and surface it in the launch review.\n - Proceed to recommendations; only the impact-versus-starting-point delta is unavailable.\n - Supply `baseline_fn` or `automl_settings[\"baseline_metric\"]` only when a checkpoint exists. `final_eval_fn` still applies; a from-scratch smoke test may explicitly use `reuse_best_metric_for_final_evaluation=True`.\n\n3. **Dataset**: Training data accessible from the compute backend. URI format depends on the SDK's platform:\n - Brev / cloud: `s3://bucket/path` (S3-compatible; do not generate `aws://...`)\n - Slurm / internal shared storage: an absolute shared filesystem path visible to the Slurm job, e.g. `/lustre/fsw/tao_datasets/\u003cmodel>/train` and `/lustre/fsw/tao_datasets/\u003cmodel>/eval`\n - Azure: `azure://container/path`\n - Local / Docker: local filesystem path\n Accept either dataset roots or exact spec-key paths. For exact spec paths,\n preserve user-supplied keys such as\n `custom.train_dataset.annotation_path=/lustre/.../annotations.json` and\n `custom.train_dataset.media_path=/lustre/.../videos.tar.gz`; do not force\n both files to share one parent directory.\n3. **Skill bank available**: the runner takes an explicit `skill_dir` — the **absolute path to a model directory** inside the skill bank, e.g. `\u003cbank-root>/skills/models/tao-train-dino`. No global env var; pass per run. The agent already knows the bank root (it loaded this SKILL.md from there) — use that same root. Resolve user model aliases to a packaged skill directory before constructing this path; do not assume `network_arch` equals the directory name. Common locations:\n - cloned standalone: `~/tao-skill-bank/` (or wherever the user cloned).\n - Installed skill-bank cache: `\u003cagent-cache>/tao-skill-bank/\u003cversion>/`.\n - Codex plugin: `~/.codex/plugins/cache/\u003cmarketplace>/tao-skill-bank/\u003cversion>/`.\n - submodule inside a cloned SDK: `\u003csdk>/tao-skill-bank/`.\n ```python\n from pathlib import Path\n SKILL_BANK = Path(\"\u003cbank-root>\") # substitute the actual path\n skill_dir = SKILL_BANK / \"skills\" / \"models\" / model_skill\n ```\n The bank structure is:\n ```\n tao-skill-bank/\n └── skills/\n ├── applications/ # workflow configs (this skill)\n ├── models/ # per-network skill packages\n │ ├── \u003cmodel_skill>/\n │ │ ├── SKILL.md\n │ │ ├── schemas/\n │ │ │ └── \u003caction>.schema.json # REQUIRED AutoML gate\n │ │ └── references/\n │ │ ├── skill_info.yaml # actions, data_sources, container image\n │ │ └── spec_template_train.yaml # default training spec (recommended)\n │ └── ...\n ├── data/\n └── platform/\n ```\n **CRITICAL**: AutoML requires a packaged generated dataclass schema at `\u003cbank-root>/skills/models/\u003cmodel_skill>/schemas/\u003caction>.schema.json`. The schema must exist and parse as JSON — it's the AutoML support gate because it defines `automl_enabled` parameters, defaults, ranges, options, weights, and popular metadata. Schemas are generated during skill-bank maintenance and shipped with the plugin; the runtime must not expect `~/tao-core` to exist. If the packaged action schema is missing, do not run AutoML for that model/action.\n\n `references/spec_template_\u003caction>.yaml` is required for **non-TAO-Core models** (cosmos-rl, clip, etc.) — without it the runner has no defaults and the trial spec will be missing keys. For **TAO Core / Hydra-based models** (DINO, BEVFusion, etc.) the template is optional; Hydra fills container-side defaults at runtime.\n4. **`nvidia-tao-automl` installed** with the platform extra you want. Resolve\n the pinned install command from `versions.yaml`:\n ```bash\n SB=\"${TAO_SKILL_BANK_PATH:-~/tao-skill-bank}\"\n pip install \"$($SB/scripts/resolve_versions_key.py wheels.tao_automl_brev)\" # or _slurm, _kubernetes, _docker, _all\n # With LLM/agentic algorithms, append ,llm to the resolved extra:\n pip install \"$($SB/scripts/resolve_versions_key.py wheels.tao_automl_brev | sed 's/]/,llm]/')\"\n ```\n For local development against a checkout: `pip install -e '~/tao-run-automl[brev]'`.\n\nVerify setup:\n```bash\npython3 -c \"from tao_automl.runner import AutoMLRunner; print('OK')\"\n\n# Verify LLM features (optional)\npython3 -c \"from tao_automl.brain.llm_brain import LLMBrain; print('LLM OK')\"\n\n# Verify WandB (optional)\npython3 -c \"import wandb; print('WandB OK')\"\n```\n\n---\n\n## Concepts: What is TAO AutoML?\n\nTAO AutoML automates the \"try different hyperparameter values → run action → compare results → repeat\" cycle. Instead of manually tweaking action settings, you tell AutoML:\n\n- **What network/action** to optimize (`network_arch`, `action`)\n- **Which hyperparameters** to search over (from the model skill and schema)\n- **What metric** to optimize (from the model skill or user request)\n- **How many trials** (budget)\n\nAutoML then:\n1. Picks hyperparameter values using a search algorithm (Bayesian, Hyperband, LLM, etc.)\n2. Launches a real action job on whichever backend the SDK targets (Brev, SLURM, Kubernetes, or local Docker)\n3. Reads the result metric from logs or an optional follow-up evaluator\n4. Feeds the result back to the algorithm so it learns what works\n5. Repeats until budget is exhausted\n6. Returns the best configuration found\n\nEach \"trial\" is called a **recommendation** (rec). One rec = one full action run with a specific set of hyperparameters.\n\n---\n\n## Quick Support Queries\n\nWhen the user asks what models/networks are supported for AutoML, run the\npackaged model-list helper in AutoML mode. AutoML enablement is **model-level**\nmetadata (`skills/models/\u003cnetwork>/references/skill_info.yaml` has\n`automl_enabled: true`), not workflow-level metadata. The helper reads that\nmodel metadata, then validates whether the model also has packaged,\nparseable selected-action dataclass schemas:\n\n```bash\n${TAO_SKILL_BANK_PATH:-~/tao-skill-bank}/scripts/list_tao_models.py \\\n --skill-bank ${TAO_SKILL_BANK_PATH:-~/tao-skill-bank} --scope automl --format text\n```\n\nThe compatibility wrapper below is also valid and delegates to the same logic:\n\n```bash\n${TAO_SKILL_BANK_PATH:-~/tao-skill-bank}/scripts/list_automl_support.py \\\n --skill-bank ${TAO_SKILL_BANK_PATH:-~/tao-skill-bank} --format text\n```\n\nReturn both sections from that output: runnable AutoML model/actions and\nAutoML-enabled model/actions still blocked on schema packaging. The support\nrule is: AutoML is enabled at model level; runnable AutoML for an action also\nrequires `skills/models/\u003cmodel_skill>/schemas/\u003caction>.schema.json` to be packaged and\nvalid. Use `--action evaluate`, `--action inference`, `--action distill`,\n`--action prune`, or `--action quantize` for a focused non-train action query.\n\n---\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":13377,"content_sha256":"aab8772c1fc676858023748ac3db1af068aae22cae9a20e1550ec4720793ff19"},{"filename":"references/automl-runner-configuration.md","content":"# TAO AutoML Runner Configuration\n\nRunner construction, SDK handoff examples, settings keys, metrics, custom parameter ranges, spec overrides, and WandB setup.\n\nLoad this file only when the compact `SKILL.md` points here for the current task. If this reference conflicts with `SKILL.md`, `skill_info.yaml`, schemas, or platform/model skills, the compact/current source wins.\n\n## Contents\n\n- Minimal Example\n- Pick whichever SDK matches where you want trials to run. AutoMLRunner is\n- platform-agnostic — none of the SDKs is a default; the user picks.\n- from tao_sdk.platforms.slurm import SlurmSDK # SLURM cluster\n- from tao_sdk.platforms.kubernetes import KubernetesSDK # K8s (EKS / GKE / on-prem)\n- from tao_sdk.platforms.docker import DockerSDK # local Docker daemon\n- VirtualEnvSDK (containerless venv runs)\n- Full Example (all options)\n- LLM-Powered Algorithm Example\n- Programmatic API (without runner)\n- `automl_settings` keys\n- `kpi` metric resolution\n- `custom_param_ranges` format\n- Model-specific search-space rules\n- LLM Analyzer (server-side range narrowing)\n- `spec_overrides`\n- WandB Experiment Tracking\n- Setup\n- or (when reinstalling tao-run-automl with the wandb extra):\n- `pip install \"$(${TAO_SKILL_BANK_PATH:?}/scripts/resolve_versions_key.py wheels.tao_automl_brev | sed 's/]/,wandb]/')\"`\n- How it works\n- Minimal WandB setup\n- Option 1: via config dict\n- Option 2: environment variable (simpler)\n- export WANDB_API_KEY=your-key\n- Dashboard features\n\n## Step 3: Configure and Run\n\n### Minimal Example\n\n```python\nfrom datetime import datetime\nfrom pathlib import Path\n\n# Pick whichever SDK matches where you want trials to run. AutoMLRunner is\n# platform-agnostic — none of the SDKs is a default; the user picks.\nfrom tao_sdk.platforms.brev import BrevSDK # Brev GPU instances\n# from tao_sdk.platforms.slurm import SlurmSDK # SLURM cluster\n# from tao_sdk.platforms.kubernetes import KubernetesSDK # K8s (EKS / GKE / on-prem)\n# from tao_sdk.platforms.docker import DockerSDK # local Docker daemon\nfrom tao_automl.runner import AutoMLRunner\n\nTIMESTAMP = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n\nsdk = BrevSDK() # reads platform credentials from env\nrunner = AutoMLRunner(\n sdk=sdk,\n skill_dir=SKILL_BANK / \"skills\" / \"models\" / model_skill, # resolved skill dir\n action=action, # train, evaluate, inference, distill, prune, quantize, ...\n)\nresult = runner.run(\n train_dataset_uri=train_dataset_uri,\n automl_settings={\n \"algorithm\": algorithm,\n \"metric\": metric,\n \"automl_max_recommendations\": max_recommendations,\n \"session_id\": session_id, # explicit, generated once\n },\n workspace_path=f\"./automl_workspace/{TIMESTAMP}\", # timestamped to avoid collisions\n # Platform-specific create_job kwargs go here as **platform_kwargs.\n # See each platform's SKILL.md for the kwargs each accepts.\n gpu_count=8,\n num_nodes=1,\n gpu_type=\"H100\", # Brev-specific\n)\n```\n\n### VirtualEnvSDK (containerless venv runs)\n\nModel skills with `container_image: null` and `execution.type: python_script`\n(the Kumo-Forecast and Kumo-Anomaly skills) run trials as local\nPython processes instead of containers. Their SDK is `VirtualEnvSDK`, and\nunlike the credential-reading platform SDKs it takes constructor arguments:\n\n```python\nfrom tao_sdk.platforms.virtualenv import VirtualEnvSDK\nfrom tao_automl.runner import AutoMLRunner\n\nsdk = VirtualEnvSDK(\n venv_path=\"/abs/path/to/model-venv\", # must contain pyvenv.cfg + bin/python\n work_dir=\"/scratch/automl-jobs\", # ALWAYS set this - see caution below\n)\nrunner = AutoMLRunner(sdk=sdk, skill_dir=str(skill_dir), action=\"train\")\n```\n\nFull signature: `VirtualEnvSDK(venv_path, work_dir=None, state_file=None)`.\n\n> **Caution - `work_dir` defaults into your home directory.** When `work_dir`\n> is omitted the SDK resolves it to `$TAO_SDK_STATE_DIR/virtualenv`, or\n> `~/.tao_sdk/virtualenv` when that env var is unset, and creates\n> `\u003cwork_dir>/jobs/` on construction. Every trial's checkpoints then land under\n> `$HOME` - roughly 1.4 GB per Kumo-Forecast trial. Always pass an\n> explicit `work_dir` on a filesystem with room for\n> `automl_max_recommendations` checkpoints. Setting `TAO_SDK_STATE_DIR`\n> relocates the default, but an explicit `work_dir` is the reviewable form and\n> is what the launch review must show.\n\nJob layout (this is where trial artifacts actually land):\n\n```text\n\u003cwork_dir>/jobs/\u003cjob-id>/spec/spec.\u003cyaml|json|toml>\n\u003cwork_dir>/jobs/\u003cjob-id>/results/\u003cdeclared_output_key_with_dots_as_underscores>/\n\u003cwork_dir>/jobs/\u003cjob-id>/logs/job.log\n```\n\nDeclared outputs from `references/skill_info.yaml` are rewritten into that\nper-job results directory, so `train.output_dir` becomes\n`\u003cwork_dir>/jobs/\u003cjob-id>/results/train_output_dir`. Resolve it at runtime with\n`sdk.get_job_results_dir(job_id)`; do not precompute it.\n\nDo not pass `image=` for `python_script` execution - the venv is the runtime.\nVerify construction during preflight. Importing the class is not enough: the\nconstructor is what validates `venv_path`, so an import-only check reports\nsuccess against a venv that cannot run a single trial.\n\n```bash\nVENV_PATH=/abs/path/to/model-venv \\\nWORK_DIR=/scratch/automl-jobs \\\npython - \u003c\u003c'PY'\nimport os\nfrom tao_sdk.platforms.virtualenv import VirtualEnvSDK\n\nVirtualEnvSDK(\n venv_path=os.environ[\"VENV_PATH\"],\n work_dir=os.environ[\"WORK_DIR\"],\n)\nprint(\"OK\")\nPY\n```\n\nUse the same `venv_path` and `work_dir` the runner will use. A missing or\nmalformed venv exits non-zero with\n`ValueError: Virtual environment does not exist: \u003cvenv_path>`, so the check\nfails before any trial launches. Construction also creates `\u003cwork_dir>/jobs/`,\nwhich is the second reason to pass `work_dir` explicitly here.\n\nThe `virtualenv` extra is not exposed as its own `versions.yaml` wheel key;\n`wheels.tao_automl_all` is currently the only key that includes it.\n\n### Full Example (all options)\n\n```python\ndef my_eval(rec, train_job_id):\n \"\"\"Optional post-training evaluator. Return a float (the real metric)\n or None to fall back to the log-based extractor.\"\"\"\n # e.g. read a results file uploaded by the container and compute the requested metric\n ...\n return 0.71\n\nresult = runner.run(\n # --- Required ---\n train_dataset_uri=train_dataset_uri,\n\n # --- Dataset + resources ---\n eval_dataset_uri=eval_dataset_uri,\n base_checkpoint=\"\",\n image=image, # only set to override skill_info's container_image\n\n # --- AutoML config ---\n automl_settings={\n \"algorithm\": algorithm,\n \"metric\": metric,\n \"direction\": direction, # explicit when needed\n \"automl_max_recommendations\": max_recommendations,\n \"session_id\": session_id, # explicit, generated/resolved once\n },\n automl_hyperparameters=automl_hyperparameters, # from model skill / schema\n custom_param_ranges=custom_param_ranges, # from model skill / user constraints\n\n # --- Per-rec spec overrides ---\n spec_overrides=spec_overrides, # mandatory model-specific overrides from model skill\n\n # --- State + durability ---\n workspace_path=f\"./my_experiment/{TIMESTAMP}\", # ALWAYS timestamp to avoid collisions\n resume=False, # True → recovers in-flight jobs\n\n # --- Hooks (all optional, opt-in) ---\n metric_extractor=None, # custom log→metric parser\n eval_fn=my_eval, # post-training real-metric eval\n on_recommendation=lambda r: print(f\"launching rec {r.id}: {r.specs}\"),\n on_result=lambda r, metric, status: print(f\"rec {r.id} {status} → {metric}\"),\n\n # --- Platform create_job kwargs (forwarded as **platform_kwargs) ---\n # SLURM: partition, account, num_nodes, gpu_count\n # Kubernetes: namespace, node_selector, tolerations, num_nodes, gpu_count\n # Docker: mounts, gpu_count\n # Brev: instance_id, gpu_type, gpu_count\n gpu_count=8,\n num_nodes=1,\n gpu_type=\"H100\",\n)\n```\n\n### LLM-Powered Algorithm Example\n\nFor `llm`, `hybrid`, or `autoresearch`, use the same generic runner shape as above, plus the required LLM endpoint, model, and key in `automl_settings`. All model-specific hyperparameters, metric extractors, and `spec_overrides` must still come from the model skill.\n\n**LLM endpoint configuration** (in order of precedence):\n1. `automl_settings` keys: `llm_endpoint`, `llm_model`, `llm_api_key`; aliases `base_url`, `model`, and `api_key` are also accepted.\n2. Environment variables: `AUTOML_LLM_ENDPOINT`, `AUTOML_LLM_MODEL`, `AUTOML_LLM_API_KEY`\n3. Fallback env var for API key: `NVIDIA_API_KEY`\n4. Defaults: NVIDIA inference endpoint (`https://inference-api.nvidia.com`) with `gcp/google/gemini-3.1-pro-preview`. Always pass the endpoint, model, and key explicitly for reproducible LLM-based testing.\n\n### Programmatic API (without runner)\n\nFor tighter control, use the `AutoML` class directly:\n\n```python\nfrom tao_automl import AutoML\n\nautoml = AutoML(\n workspace=\"/tmp/my_experiment\",\n network=network_arch,\n train_specs=my_train_spec_dict,\n action=action,\n settings={\n \"algorithm\": \"bayesian\",\n \"metric\": \"loss\",\n \"automl_max_recommendations\": 10,\n },\n wandb_config={\"enabled\": True, \"project\": \"my-project\"},\n)\n\nwhile not automl.is_complete():\n recs = automl.next_recommendation()\n for rec in recs:\n metric_value = train_model(rec.specs) # your training function\n automl.report_result(rec.id, metric_value)\n\nautoml.finish() # close WandB run\nprint(\"Best:\", automl.get_best().specs)\n```\n\n### `automl_settings` keys\n\n| Key | Type | Default | Description |\n|---|---|---|---|\n| `algorithm` | str | **required** | `bayesian`, `hyperband`, `bohb`, `asha`, `bfbo`, `dehb`, `pbt`, `hyperband_es`, `llm`, `hybrid`, `autoresearch` |\n| `session_id` | str | **required by this skill** | Stable controller identity. Generate it once for a fresh run with `scripts/resolve_automl_session.py new`; on resume recover it from the full run workspace. Never rely on the wheel's random default. |\n| `metric` | str | `\"loss\"` | Metric name. The implicit rule for direction is \"contains `'loss'` → minimize, else maximize\". Override with `direction`. |\n| `direction` | `\"minimize\"` \\| `\"maximize\"` | inferred | Explicit direction. Required only when it disagrees with the implicit rule. The runner transparently inverts reported values so callers always see their metric in its original scale. |\n| `automl_max_recommendations` | int | 20 | Max trials (bayesian, bfbo, llm) |\n| `automl_max_epochs` | int | 27 | Epoch budget (hyperband, bohb, asha, dehb) |\n| `automl_reduction_factor` | int | 3 | Halving factor (hyperband variants) |\n| `automl_max_concurrent` | int | 4 | Max parallel configs (asha only) |\n| `automl_population_size` | int | 10 | Population size (pbt only) |\n| `automl_max_experiments` | int | 50 | Max experiments (autoresearch only) |\n| `llm_endpoint` | str | `https://inference-api.nvidia.com` | OpenAI-compatible API endpoint (llm, hybrid, autoresearch) |\n| `llm_model` | str | `gcp/google/gemini-3.1-pro-preview` | LLM model name (llm, hybrid, autoresearch) |\n| `llm_api_key` | str | from env | API key for the LLM endpoint |\n| `research_program` | str | None | Free-text research directives for the autoresearch agent |\n| `evolvable_text_parameters` | str \\| list[str] | `[]` | Searchable text parameters that `autoresearch` may rewrite beyond schema seed enums. |\n| `automl_delete_intermediate_ckpt` | bool | True | Automatically delete cleanup-supported terminal artifacts once they are safe to prune. The active trial/current best, every non-dominated Pareto-front member in a multi-objective search, and Hyperband-family promotion/resume inputs are protected; Hybrid conservatively retains successful trials when full-fidelity provenance is ambiguous. Cleanup-aware SDKs reject remote Docker binds, named volumes, unsafe output ownership, and unbound S3 identities before the first trial because deletion cannot be verified. Set False only when all trial artifacts and their external cleanup lifecycle are intentionally owned. |\n| `override_automl_disabled_params` | bool | False | Include params whose schema `automl_enabled` is False. For advanced users who want to search over params the network author didn't flag for AutoML. |\n\n`AutoMLRunner.run(..., feedback_fn=...)` accepts a callback with signature\n`(recommendation, job_id) -> JSON-safe value`. For reflective searches, return\ncompact training-split records containing the input/query, generated output,\nand a failure-mode comment that does not reveal the correct answer. Do not return\ngold/expected labels, item or media identifiers, or paths. The runner removes\ncommon ground-truth and identifier fields, persists the sanitized feedback, and\nsupplies it to the next `autoresearch` proposal; it does not replace the numeric\nvalidation selection metric.\n\n### `kpi` metric resolution\n\nWhen `metric=\"kpi\"`, the controller resolves the actual metric key from the network config's `metrics.monitoring_metric` field. Whether `kpi` is appropriate, and whether a custom `metric_extractor` is needed, is model-specific. Follow the model skill's **AutoML / HPO Notes**.\n\n### `custom_param_ranges` format\n\nEach entry can include:\n\n| Field | Type | Description |\n|---|---|---|\n| `valid_min` | float/int/list | Min value. For list-valued parameters, pass the list shape required by the schema. |\n| `valid_max` | float/int/list | Max value. Same list rules as min. |\n| `valid_options` | list[str] | For categorical/ordered params: restrict to these values |\n| `option_weights` | list[float] | Sampling weights for `valid_options`. Must match length. Higher weight = more likely to be sampled. |\n| `disable_list` | bool | For params that can be float OR list: `True` keeps it as a single float for optimization, bypassing network list helpers. Use only when supported by the schema/model skill. |\n\nExample with all features:\n\n```python\ncustom_param_ranges={\n \"\u003cfloat_param>\": {\"valid_min\": min_value, \"valid_max\": max_value, \"disable_list\": True},\n \"\u003ccategorical_param>\": {\n \"valid_options\": [\"option_a\", \"option_b\"],\n \"option_weights\": [0.7, 0.3],\n },\n \"\u003clist_param>\": {\"valid_min\": [min_a, min_b], \"valid_max\": [max_a, max_b]},\n}\n```\n\n### Model-specific search-space rules\n\nSome networks have built-in search-space exclusions or algorithm restrictions. Do not document them here; read the model skill's **AutoML / HPO Notes** and let schema validation report unsupported combinations.\n\n### LLM Analyzer (server-side range narrowing)\n\nThe controller supports automatic range narrowing via the LLM analyzer. Enable via environment variables before launching:\n\n```python\nos.environ[\"AUTOML_LLM_ANALYZER_ENABLED\"] = \"true\"\nos.environ[\"AUTOML_LLM_ANALYZER_INTERVAL\"] = \"5\" # analyze every 5 completed recs\nos.environ[\"AUTOML_LLM_ANALYZER_NARROW_RANGES\"] = \"true\" # auto-tighten custom_param_ranges\n```\n\nWhen enabled, after every N completed experiments the analyzer reviews patterns, assesses convergence, and optionally narrows search ranges to focus on promising regions. This happens server-side and persists the narrowed ranges.\n\n### `spec_overrides`\n\n`spec_overrides` keys are model-specific. Read the model skill's **Training Requirements**, **Per-Action Dataset Requirements**, and **Typical Spec Overrides** sections, then pass only the keys required or recommended there. Do not infer override keys from examples in this AutoML skill.\n\nEvery key you pass is validated against the skill's spec schema. Typos that look like existing keys raise `ValueError` with a suggestion; genuinely-new keys are accepted with a warning.\n\n---\n\n## WandB Experiment Tracking\n\nAutoML optionally integrates with [Weights & Biases](https://wandb.ai) to track all experiments in a single dashboard.\n\n### Setup\n\n```bash\npip install wandb\n# or (when reinstalling tao-run-automl with the wandb extra):\n# pip install \"$(\"${TAO_SKILL_BANK_PATH:?}/scripts/resolve_versions_key.py\" wheels.tao_automl_brev | sed 's/]/,wandb]/')\"\n```\n\n### How it works\n\nWhen `wandb_config={\"enabled\": True}` is passed:\n\n1. The controller creates a WandB **run** named `automl_brain` in the specified project.\n2. All recommendations are grouped under a WandB **group** (e.g. `automl_abc123`) so parent + child training runs appear together in the dashboard.\n3. After every result, a **WandB table** (`automl_experiments`) is logged containing:\n - `experiment_id`, `job_id`, `status`, metric value, `best_epoch_number`\n - All varying hyperparameter values\n4. Call `automl.finish()` (or let `runner.run()` complete) to finalize the WandB run.\n\n### Minimal WandB setup\n\n```python\n# Option 1: via config dict\nresult = runner.run(\n ...,\n wandb_config={\n \"enabled\": True,\n \"project\": \"tao-hpo\",\n \"api_key\": \"your-key\", # or set WANDB_API_KEY env var\n },\n)\n\n# Option 2: environment variable (simpler)\n# export WANDB_API_KEY=your-key\nresult = runner.run(\n ...,\n wandb_config={\"enabled\": True, \"project\": \"tao-hpo\"},\n)\n```\n\n### Dashboard features\n\nOnce tracking is active, you can:\n- **Compare all trials** side-by-side in the WandB table view\n- **Sort by metric** to find the best config instantly\n- **Group by hyperparameter** to see which values correlate with good results\n- **Link to child training runs** if the compute backend also logs to WandB (group name is available via `automl.wandb_group`)\n\n---\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":17763,"content_sha256":"10bf45667e2dfa0057e56d1f18253ac700eb7d598e6d30318836097495151b20"},{"filename":"references/best_rec_adapter.py","content":"#!/usr/bin/env python3\n# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n# SPDX-License-Identifier: Apache-2.0\n\n\"\"\"Emit best_rec.json from an AutoMLRunner result — the read side of the\nAutoML -> DEFT warm-start seam.\n\nAutoMLRunner (the separate nvidia-tao-automl wheel) is unchanged; this adapter\nserializes its winning recommendation into the best_rec.json contract\n(skills/core/tao-artifacts/references/best_rec.schema.json), which the DEFT\npipeline reads to warm-start.\n\nThe load-bearing transform is BUDGET STRIPPING: ASHA/hyperband mutate the fidelity\nkey (num_epochs) per rung, so the winning spec carries a *rung* budget. Deep-merging\nthat into DEFT's hand-authored baseline spec would silently overwrite the real\nepoch count — so budget keys are moved OUT of `specs` into a separate\n`observed_budget`. And `metric_name` + `direction` are required: a bare score is\nmeaningless to `_pick_best` downstream. Output is validated against the schema so a\nmalformed hand-off fails loudly here, not after Phase-2 GPU-hours.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport argparse\nimport copy\nimport json\nimport sys\nfrom pathlib import Path\n\nSCHEMA_PATH = (Path(__file__).resolve().parents[3]\n / \"core/tao-artifacts/references/best_rec.schema.json\")\nDEFAULT_BUDGET_KEYS = (\"num_epochs\", \"train.num_epochs\", \"train.max_epochs\", \"epochs\")\n\n\ndef _pop_nested(spec: dict, dotted: str):\n parts = dotted.split(\".\")\n d = spec\n for p in parts[:-1]:\n if not isinstance(d, dict) or p not in d:\n return None\n d = d[p]\n if isinstance(d, dict) and parts[-1] in d:\n return d.pop(parts[-1])\n return None\n\n\ndef build_best_rec(*, experiment_id: str, metric_name: str, direction: str,\n best_rec_id, best_score, best_specs: dict, checkpoint_uri: str,\n all_recs=None, budget_keys=DEFAULT_BUDGET_KEYS,\n checkpoint_epoch=None, checkpoint_step=None) -> dict:\n if not metric_name:\n raise ValueError(\"metric_name is required (a bare score is meaningless downstream)\")\n if direction not in (\"maximize\", \"minimize\"):\n raise ValueError(f\"direction must be 'maximize' or 'minimize', got {direction!r}\")\n\n specs = copy.deepcopy(best_specs)\n observed_budget: dict = {}\n for key in budget_keys:\n val = _pop_nested(specs, key)\n if val is not None:\n observed_budget[key.split(\".\")[-1]] = val\n\n best = {\n \"rec_id\": str(best_rec_id),\n \"score\": float(best_score),\n \"specs\": specs,\n \"observed_budget\": observed_budget,\n \"checkpoint_uri\": checkpoint_uri,\n }\n if checkpoint_epoch is not None:\n best[\"checkpoint_epoch\"] = int(checkpoint_epoch)\n if checkpoint_step is not None:\n best[\"checkpoint_step\"] = int(checkpoint_step)\n\n return {\n \"schema_version\": 1,\n \"experiment_id\": experiment_id,\n \"metric_name\": metric_name,\n \"direction\": direction,\n \"best\": best,\n \"all_recs\": list(all_recs or []),\n }\n\n\ndef validate(rec: dict) -> None:\n \"\"\"Raise jsonschema.ValidationError if rec violates best_rec.schema.json.\"\"\"\n import jsonschema\n jsonschema.validate(rec, json.loads(SCHEMA_PATH.read_text()))\n\n\ndef main(argv: list[str] | None = None) -> int:\n p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)\n p.add_argument(\"input\", nargs=\"?\", help=\"JSON of the extracted runner fields; stdin if omitted\")\n p.add_argument(\"-o\", \"--out\", help=\"write best_rec.json here (else stdout)\")\n p.add_argument(\"--no-validate\", action=\"store_true\")\n args = p.parse_args(argv)\n\n raw = json.loads(Path(args.input).read_text() if args.input and args.input != \"-\" else sys.stdin.read())\n rec = build_best_rec(**raw)\n if not args.no_validate:\n validate(rec)\n out = json.dumps(rec, indent=2)\n if args.out:\n Path(args.out).write_text(out + \"\\n\")\n print(f\"wrote {args.out}\", file=sys.stderr)\n else:\n print(out)\n return 0\n\n\nif __name__ == \"__main__\":\n raise SystemExit(main())\n","content_type":"text/x-python; charset=utf-8","language":"python","size":4160,"content_sha256":"86fcd5600a14a7a8e85f543ed1f95fa65e177c4beed63440d8307349b4c5baf0"},{"filename":"references/detailed-guide.md","content":"# TAO AutoML Detailed Reference Map\n\nThis is a navigation file for split detailed references. Do not load every linked file by default; pick the smallest reference that matches the current task.\n\nIf a split reference conflicts with `SKILL.md`, `skill_info.yaml`, schemas, or platform/model skills, the compact/current source wins.\n\n## Reference Map\n\n- `automl-preflight-concepts.md` — preflight, platform prerequisites, support checks, or conceptual grounding\n- `automl-intent-algorithms.md` — intent parsing, algorithm choice, or search-policy decisions\n- `automl-compression-literature.md` — distill/prune/quantize literature review, algorithm sufficiency, and compression-search roadmap\n- `automl-runner-configuration.md` — runner configuration, SDK handoff, metrics, ranges, spec overrides, or WandB\n- `automl-advanced-monitoring.md` — LLM/agentic features, hooks, monitoring, resume, result interpretation, or pitfalls\n- `automl-examples.md` — conversation-level examples\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":989,"content_sha256":"20f5d465eb4ad66423836f54c524e450580d1abee4b39c92f1db09eefbc5808f"},{"filename":"references/skill_info.yaml","content":"name: tao-run-automl\ntype: workflow\nprerequisites:\n required:\n - name: model\n description: A TAO network skill with selected-action defaults and AutoML/HPO notes.\n - name: action_inputs\n description: Dataset, checkpoint, teacher, calibration, or artifact inputs required by the selected action.\n - name: platform\n description: Compute backend available through the TAO SDK, usually slurm, kubernetes, docker, or brev.\n - name: skill_bank_path\n description: Path to this skill bank so AutoMLRunner can load model skills and selected-action specs.\n optional:\n - name: eval_dataset_uri\n description: Validation or evaluation dataset URI when required by the selected model skill.\n - name: algorithm\n description: AutoML search algorithm such as bayesian, hyperband, asha, bohb, llm, hybrid, or autoresearch.\n - name: llm_endpoint\n description: Required for LLM-powered AutoML algorithms; alias base_url is accepted.\n - name: llm_model\n description: Required for LLM-powered AutoML algorithms; alias model is accepted.\n - name: llm_api_key\n description: Required for LLM-powered AutoML algorithms if not already available in the environment; alias api_key is accepted and must never be printed.\nactions:\n run:\n description: Create a fresh timestamped AutoML runner script and launch TAO HPO through AutoMLRunner.\n status:\n description: Inspect an existing AutoML workflow folder, runner log, PID, or SDK state for progress.\n resume:\n description: Resume or recover an explicitly requested existing AutoML experiment.\ndepends_on:\n models:\n - \u003cnetwork>\n platform:\n - tao-sdk\n - brev\ndescription: TAO AutoML/HPO workflow for launching model/action-agnostic hyperparameter optimization through AutoMLRunner while grounding\n model- and action-specific details in the selected model skill.\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":1836,"content_sha256":"0f047c1a60b55150ea09bfdd76aa253d8f951e34e3d54f520991290d817204df"},{"filename":"references/tests/test_best_rec_adapter.py","content":"# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n# SPDX-License-Identifier: Apache-2.0\n\n\"\"\"Tests for the AutoML -> best_rec.json adapter.\n\nThe two load-bearing behaviors (both red-team must-fixes on the contract): budget\nkeys are moved OUT of `specs` into `observed_budget` (so a rung's num_epochs can't\noverwrite DEFT's baseline), and metric_name+direction are mandatory. Output must\nvalidate against the M0 best_rec schema.\n\"\"\"\n\nimport copy\nimport json\nimport sys\nfrom pathlib import Path\n\nimport jsonschema\nimport pytest\n\nsys.path.insert(0, str(Path(__file__).resolve().parents[1]))\nimport best_rec_adapter as bra # noqa: E402\n\nSCHEMA = json.loads(bra.SCHEMA_PATH.read_text())\n\nWINNING_SPECS = {\n \"train\": {\"num_epochs\": 10, \"optim\": {\"lr\": 0.00013}}, # num_epochs = a rung budget\n \"dataset\": {\"batch_size\": 4},\n}\n\n\ndef build(**over):\n kw = dict(experiment_id=\"automl-exp-7\", metric_name=\"far_at_100_recall\", direction=\"minimize\",\n best_rec_id=\"rec_003\", best_score=0.0012, best_specs=WINNING_SPECS,\n checkpoint_uri=\"/lustre/results/exp7/rec_003/model_epoch_009_step_01200.pth\")\n kw.update(over)\n return bra.build_best_rec(**kw)\n\n\ndef test_output_validates_against_schema():\n jsonschema.validate(build(), SCHEMA)\n\n\ndef test_budget_stripped_from_specs_into_observed_budget():\n rec = build()\n assert \"num_epochs\" not in rec[\"best\"][\"specs\"][\"train\"] # moved out of specs\n assert rec[\"best\"][\"observed_budget\"][\"num_epochs\"] == 10 # preserved separately\n assert rec[\"best\"][\"specs\"][\"train\"][\"optim\"][\"lr\"] == 0.00013 # non-budget HPs kept\n\n\ndef test_original_specs_not_mutated():\n original = copy.deepcopy(WINNING_SPECS)\n build()\n assert WINNING_SPECS == original # deepcopy — caller's dict intact\n\n\ndef test_metric_identity_required():\n with pytest.raises(ValueError, match=\"metric_name\"):\n build(metric_name=\"\")\n with pytest.raises(ValueError, match=\"direction\"):\n build(direction=\"down\")\n\n\ndef test_metric_name_and_direction_carried():\n rec = build()\n assert rec[\"metric_name\"] == \"far_at_100_recall\" and rec[\"direction\"] == \"minimize\"\n\n\ndef test_checkpoint_epoch_step_optional_and_valid():\n rec = build(checkpoint_epoch=9, checkpoint_step=1200)\n jsonschema.validate(rec, SCHEMA)\n assert rec[\"best\"][\"checkpoint_epoch\"] == 9 and rec[\"best\"][\"checkpoint_step\"] == 1200\n\n\ndef test_top_level_num_epochs_also_stripped():\n rec = bra.build_best_rec(experiment_id=\"e\", metric_name=\"m\", direction=\"maximize\",\n best_rec_id=\"r1\", best_score=1.0, checkpoint_uri=\"/x\",\n best_specs={\"num_epochs\": 20, \"train\": {\"lr\": 0.1}})\n assert \"num_epochs\" not in rec[\"best\"][\"specs\"]\n assert rec[\"best\"][\"observed_budget\"][\"num_epochs\"] == 20\n jsonschema.validate(rec, SCHEMA)\n\n\ndef test_all_recs_passthrough():\n rec = build(all_recs=[{\"rec_id\": \"rec_001\", \"score\": 0.004, \"job_id\": \"dino-train-x1\"},\n {\"rec_id\": \"rec_002\", \"score\": None}])\n jsonschema.validate(rec, SCHEMA)\n assert len(rec[\"all_recs\"]) == 2\n\n\ndef test_cli_stdin_to_stdout(capsys, monkeypatch):\n payload = {\"experiment_id\": \"e\", \"metric_name\": \"acc\", \"direction\": \"maximize\",\n \"best_rec_id\": \"r1\", \"best_score\": 0.9, \"checkpoint_uri\": \"/ckpt.pth\",\n \"best_specs\": {\"train\": {\"num_epochs\": 12, \"lr\": 0.1}}}\n monkeypatch.setattr(\"sys.stdin\", __import__(\"io\").StringIO(json.dumps(payload)))\n assert bra.main([]) == 0\n out = json.loads(capsys.readouterr().out)\n assert out[\"best\"][\"observed_budget\"][\"num_epochs\"] == 12\n assert \"num_epochs\" not in out[\"best\"][\"specs\"][\"train\"]\n","content_type":"text/x-python; charset=utf-8","language":"python","size":3768,"content_sha256":"8f6dbfc26aebccdf9cce69583d232370f47c9236a9ba1f94793a0117cf91dd43"},{"filename":"scripts/resolve_automl_session.py","content":"#!/usr/bin/env python3\n# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n# SPDX-License-Identifier: Apache-2.0\n\n\"\"\"Create or recover the explicit session identity used by TAO AutoML.\n\nThe AutoML wheel otherwise generates a new random identity on each invocation,\nwhich makes ``resume=True`` silently start a second controller in the same\nworkspace. This helper is deliberately read-only: the runner persists the\nidentity in ``.automl/controller/\u003csession_id>.json``.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport argparse\nimport json\nimport re\nimport secrets\nimport sys\nfrom collections.abc import Mapping\nfrom pathlib import Path\nfrom typing import Any\n\n\nSESSION_ID_RE = re.compile(r\"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$\")\n\n\nclass SessionResolutionError(ValueError):\n \"\"\"Raised when an AutoML workspace cannot be resumed unambiguously.\"\"\"\n\n\ndef new_session_id() -> str:\n \"\"\"Return a wheel-compatible 12-character session identity.\"\"\"\n\n return secrets.token_hex(6)\n\n\ndef _controller_files(workspace: Path) -> list[Path]:\n controller_dir = workspace / \".automl\" / \"controller\"\n if not controller_dir.is_dir():\n raise SessionResolutionError(\n f\"resume state is missing: {controller_dir} does not exist; \"\n \"refuse to start a fresh search with resume=True\"\n )\n return sorted(controller_dir.glob(\"*.json\"))\n\n\ndef _validate_controller(path: Path) -> None:\n try:\n state = json.loads(path.read_text(encoding=\"utf-8\"))\n except (OSError, json.JSONDecodeError) as exc:\n raise SessionResolutionError(\n f\"controller state is unreadable: {path}: {exc}\"\n ) from exc\n if not isinstance(state, list):\n raise SessionResolutionError(\n f\"controller state has unexpected shape: {path} must contain a JSON list\"\n )\n\n\ndef resolve_session_id(workspace: Path, requested: str | None = None) -> str:\n \"\"\"Resolve exactly one existing controller identity for a resumed run.\"\"\"\n\n workspace = workspace.expanduser().resolve()\n files = _controller_files(workspace)\n\n if requested is not None:\n if not SESSION_ID_RE.fullmatch(requested):\n raise SessionResolutionError(f\"invalid session id: {requested!r}\")\n selected = workspace / \".automl\" / \"controller\" / f\"{requested}.json\"\n if selected not in files:\n available = \", \".join(path.stem for path in files) or \"none\"\n raise SessionResolutionError(\n f\"session {requested!r} has no controller state in {workspace}; \"\n f\"available sessions: {available}\"\n )\n else:\n if not files:\n raise SessionResolutionError(\n f\"resume state is missing: no controller JSON exists in \"\n f\"{workspace / '.automl' / 'controller'}; refuse to start a fresh \"\n \"search with resume=True\"\n )\n if len(files) != 1:\n available = \", \".join(path.stem for path in files)\n raise SessionResolutionError(\n f\"resume state is ambiguous: found {len(files)} controller sessions \"\n f\"in {workspace}: {available}; select the intended session explicitly \"\n \"with --session-id after inspecting its state\"\n )\n selected = files[0]\n\n if not SESSION_ID_RE.fullmatch(selected.stem):\n raise SessionResolutionError(\n f\"controller filename is not a valid session id: {selected.name}\"\n )\n _validate_controller(selected)\n return selected.stem\n\n\ndef validate_session_settings(\n automl_settings: Mapping[str, Any],\n *,\n resume: bool,\n workspace: Path | None = None,\n) -> str:\n \"\"\"Fail before runner construction when session identity is not durable.\"\"\"\n if not isinstance(automl_settings, Mapping):\n raise SessionResolutionError(\"automl_settings must be a mapping\")\n session_id = automl_settings.get(\"session_id\")\n if not isinstance(session_id, str) or not SESSION_ID_RE.fullmatch(session_id):\n raise SessionResolutionError(\n \"automl_settings.session_id must be an explicit valid session id\"\n )\n if resume:\n if workspace is None:\n raise SessionResolutionError(\n \"resume=True requires the full existing workspace path\"\n )\n resolved = resolve_session_id(workspace, session_id)\n if resolved != session_id: # defensive: requested resolution must be exact\n raise SessionResolutionError(\n f\"resolved session {resolved!r} does not match settings {session_id!r}\"\n )\n return session_id\n\n\ndef build_parser() -> argparse.ArgumentParser:\n parser = argparse.ArgumentParser(description=__doc__)\n subparsers = parser.add_subparsers(dest=\"command\", required=True)\n subparsers.add_parser(\"new\", help=\"emit a new explicit session id\")\n resume = subparsers.add_parser(\n \"resolve\", help=\"recover the session id from an existing run workspace\"\n )\n resume.add_argument(\"--workspace\", required=True, type=Path)\n resume.add_argument(\n \"--session-id\",\n help=\"select an existing session when a previously broken workspace is ambiguous\",\n )\n return parser\n\n\ndef main(argv: list[str] | None = None) -> int:\n args = build_parser().parse_args(argv)\n try:\n if args.command == \"new\":\n session_id = new_session_id()\n else:\n session_id = resolve_session_id(args.workspace, args.session_id)\n except SessionResolutionError as exc:\n print(f\"ERROR: {exc}\", file=sys.stderr)\n return 2\n print(session_id)\n return 0\n\n\nif __name__ == \"__main__\":\n raise SystemExit(main())\n","content_type":"text/x-python; charset=utf-8","language":"python","size":5743,"content_sha256":"89a4a9335e52fa2b1b92d9cc8b9c1cc9f80b89c4fdbe88d41b985d6f1ed52ffc"},{"filename":"skill-card.md","content":"## Description: \u003cbr>\nRun container-backed AutoML / hyperparameter optimization (HPO) for NVIDIA TAO networks using AutoMLRunner, handling algorithm selection (bayesian, hyperband, asha, bohb, llm, hybrid, autoresearch), WandB experiment tracking, job execution on Brev, SLURM, Kubernetes, or Docker, result interpretation, and per-recommendation custom evaluation hooks. \u003cbr>\n\nThis skill is ready for commercial/non-commercial use. \u003cbr>\n\n## Owner\nNVIDIA \u003cbr>\n\n### License/Terms of Use: \u003cbr>\nApache 2.0 \u003cbr>\n## Use Case: \u003cbr>\nDevelopers and engineers use this skill to run automated hyperparameter optimization for NVIDIA TAO models, covering baseline evaluation, algorithm and search-space selection, recommendation job execution across platforms, monitoring, and best-checkpoint handoff for train, evaluate, inference, distill, prune, or quantize actions. \u003cbr>\n\n### Deployment Geography for Use: \u003cbr>\nGlobal \u003cbr>\n\n## Requirements / Dependencies: \u003cbr>\n**Requires API Key or External Credential:** [Yes] \u003cbr>\n**Credential Type(s):** [API key, Cloud Credentials] \u003cbr>\n\nDo not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. \u003cbr>\n\n## Known Risks and Mitigations: \u003cbr>\nRisk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. \u003cbr>\nMitigation: Review and scan skill before deployment. \u003cbr>\n\n## Reference(s): \u003cbr>\n- [skill_info.yaml](references/skill_info.yaml) \u003cbr>\n- [Detailed Guide](references/detailed-guide.md) \u003cbr>\n- [Preflight Concepts](references/automl-preflight-concepts.md) \u003cbr>\n- [Intent Algorithms](references/automl-intent-algorithms.md) \u003cbr>\n- [Compression Literature](references/automl-compression-literature.md) \u003cbr>\n- [Runner Configuration](references/automl-runner-configuration.md) \u003cbr>\n- [Advanced Monitoring](references/automl-advanced-monitoring.md) \u003cbr>\n- [Examples](references/automl-examples.md) \u003cbr>\n- [Common Pitfalls](references/automl-common-pitfalls.md) \u003cbr>\n\n\n## Skill Output: \u003cbr>\n**Output Type(s):** [Shell commands, Code, Configuration instructions, Analysis] \u003cbr>\n**Output Format:** [Markdown with inline bash and Python code blocks] \u003cbr>\n**Output Parameters:** [1D] \u003cbr>\n**Other Properties Related to Output:** [None] \u003cbr>\n\n## Evaluation Agents Used: \u003cbr>\n- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`) \u003cbr>\n- Codex (`openai/openai/gpt-5.5`) \u003cbr>\n\n\n\n## Evaluation Tasks: \u003cbr>\n2 evaluation tasks (2 positive) against skill-evaluator-dataset-snapshot/1, one attempt per task in isolated sandbox pods. \u003cbr>\n\n## Evaluation Metrics Used: \u003cbr>\nReported benchmark dimensions: \u003cbr>\n- Security: Checks for unsafe operations, secret leakage, and unauthorized access. \u003cbr>\n- Correctness: Checks final-answer correctness against the reference answer. \u003cbr>\n- Discoverability: Checks whether the expected skill was selected, decoys were avoided, and the workflow executed. \u003cbr>\n- Effectiveness: Checks whether the user's goal was achieved and the expected workflow behavior was followed. \u003cbr>\n- Efficiency: Checks tool-call productivity and token efficiency. \u003cbr>\n\nUnderlying evaluation signals used in this run: \u003cbr>\n- `security`: Unsafe operations, secret leakage, and unauthorized access. \u003cbr>\n- `skill_execution`: Whether the expected skill was selected, decoys were avoided, and the workflow executed. \u003cbr>\n- `skill_efficiency`: Tool-call productivity. \u003cbr>\n- `accuracy`: Final-answer correctness against the reference answer. \u003cbr>\n- `goal_accuracy`: Whether the user's goal was achieved. \u003cbr>\n- `behavior_check`: Whether the expected workflow behavior was followed. \u003cbr>\n- `token_efficiency`: Actual uncached prompt plus completion usage. \u003cbr>\n\n\n\n## Evaluation Results: \u003cbr>\n| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |\n|---|---:|---:|\n| Overall | 99.2% — baseline ran, but no comparable score was available; uplift unavailable | 89.2% — baseline ran, but no comparable score was available; uplift unavailable |\n| Security | 100.0% → 100.0% (±0.0 points) | 100.0% → 100.0% (±0.0 points) |\n| Correctness | 50.0% → 100.0% (+50.0 points) | 50.0% → 100.0% (+50.0 points) |\n| Discoverability | 100.0% — baseline ran, but no comparable score was available; uplift unavailable | 47.5% — baseline ran, but no comparable score was available; uplift unavailable |\n| Effectiveness | 50.0% → 100.0% (+50.0 points) | 58.3% → 100.0% (+41.7 points) |\n| Efficiency | 95.8% — baseline ran, but no comparable score was available; uplift unavailable | 98.7% — baseline ran, but no comparable score was available; uplift unavailable |\n\n## Skill Version(s): \u003cbr>\n0.1.1 (source: frontmatter) \u003cbr>\n\n## Ethical Considerations: \u003cbr>\nNVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. \u003cbr>\n\n(For Release on NVIDIA Platforms Only) \u003cbr>\nPlease report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail). \u003cbr>\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":5308,"content_sha256":"9079071b66cc9a375c54c177efb5bffd96a9f17bdae73aed720b556129e1387d"},{"filename":"tests/test_resolve_automl_session.py","content":"# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n# SPDX-License-Identifier: Apache-2.0\n\nfrom __future__ import annotations\n\nimport importlib.util\nimport json\nimport subprocess\nimport sys\nfrom pathlib import Path\n\nimport pytest\n\n\nSCRIPT = Path(__file__).resolve().parents[1] / \"scripts\" / \"resolve_automl_session.py\"\nSPEC = importlib.util.spec_from_file_location(\"resolve_automl_session\", SCRIPT)\nassert SPEC and SPEC.loader\nMODULE = importlib.util.module_from_spec(SPEC)\nSPEC.loader.exec_module(MODULE)\n\n\ndef _write_controller(workspace: Path, session_id: str, state=None) -> Path:\n path = workspace / \".automl\" / \"controller\" / f\"{session_id}.json\"\n path.parent.mkdir(parents=True, exist_ok=True)\n path.write_text(json.dumps([] if state is None else state), encoding=\"utf-8\")\n return path\n\n\ndef test_new_session_id_matches_wheel_default_shape():\n first = MODULE.new_session_id()\n second = MODULE.new_session_id()\n assert first != second\n assert len(first) == 12\n assert MODULE.SESSION_ID_RE.fullmatch(first)\n\n\ndef test_resolve_unique_controller(tmp_path):\n _write_controller(tmp_path, \"abc123def456\", [{\"id\": 0, \"status\": \"success\"}])\n assert MODULE.resolve_session_id(tmp_path) == \"abc123def456\"\n\n\ndef test_missing_controller_state_fails_closed(tmp_path):\n with pytest.raises(MODULE.SessionResolutionError, match=\"resume state is missing\"):\n MODULE.resolve_session_id(tmp_path)\n\n\ndef test_ambiguous_controller_state_fails_closed(tmp_path):\n _write_controller(tmp_path, \"111111111111\")\n _write_controller(tmp_path, \"222222222222\")\n with pytest.raises(MODULE.SessionResolutionError, match=\"resume state is ambiguous\"):\n MODULE.resolve_session_id(tmp_path)\n\n\ndef test_explicit_session_resolves_legacy_ambiguous_workspace(tmp_path):\n _write_controller(tmp_path, \"111111111111\")\n _write_controller(tmp_path, \"222222222222\")\n assert MODULE.resolve_session_id(tmp_path, \"222222222222\") == \"222222222222\"\n\n\ndef test_malformed_controller_state_is_rejected(tmp_path):\n path = _write_controller(tmp_path, \"abc123def456\")\n path.write_text(\"{}\", encoding=\"utf-8\")\n with pytest.raises(MODULE.SessionResolutionError, match=\"unexpected shape\"):\n MODULE.resolve_session_id(tmp_path)\n\n\ndef test_runner_settings_without_explicit_session_fail_before_launch():\n with pytest.raises(\n MODULE.SessionResolutionError,\n match=\"automl_settings.session_id\",\n ):\n MODULE.validate_session_settings(\n {\"algorithm\": \"bayesian\"},\n resume=False,\n )\n\n\ndef test_resume_settings_must_bind_existing_controller(tmp_path):\n _write_controller(tmp_path, \"abc123def456\")\n\n with pytest.raises(MODULE.SessionResolutionError, match=\"has no controller state\"):\n MODULE.validate_session_settings(\n {\"algorithm\": \"bayesian\", \"session_id\": \"different123\"},\n resume=True,\n workspace=tmp_path,\n )\n\n assert (\n MODULE.validate_session_settings(\n {\"algorithm\": \"bayesian\", \"session_id\": \"abc123def456\"},\n resume=True,\n workspace=tmp_path,\n )\n == \"abc123def456\"\n )\n\n\ndef test_cli_returns_nonzero_instead_of_silently_starting_fresh(tmp_path):\n result = subprocess.run(\n [sys.executable, str(SCRIPT), \"resolve\", \"--workspace\", str(tmp_path)],\n text=True,\n capture_output=True,\n check=False,\n )\n assert result.returncode == 2\n assert \"refuse to start a fresh search with resume=True\" in result.stderr\n assert result.stdout == \"\"\n\n\ndef test_explicit_identity_resumes_real_automl_controller(tmp_path):\n tao_automl = pytest.importorskip(\"tao_automl\")\n schema_path = (\n Path(__file__).resolve().parents[3]\n / \"models/tao-finetune-kumo-forecast/schemas/train.schema.json\"\n )\n schema = json.loads(schema_path.read_text(encoding=\"utf-8\"))\n session_id = MODULE.new_session_id()\n workspace = tmp_path / \"run_resume\"\n settings = {\n \"algorithm\": \"bayesian\",\n \"metric\": \"val_mse\",\n \"direction\": \"minimize\",\n \"automl_max_recommendations\": 1,\n \"session_id\": session_id,\n }\n assert MODULE.validate_session_settings(settings, resume=False) == session_id\n fresh = tao_automl.AutoML(\n workspace=str(workspace),\n network=\"kumo_forecast\",\n train_specs=schema[\"default\"],\n settings=settings,\n action=\"train\",\n search_schema=schema,\n )\n rec = fresh.next_recommendation()[0]\n fresh.report_result(rec.id, 0.5)\n fresh.finish()\n\n assert MODULE.resolve_session_id(workspace) == session_id\n assert (\n MODULE.validate_session_settings(\n settings,\n resume=True,\n workspace=workspace,\n )\n == session_id\n )\n resumed = tao_automl.AutoML(\n workspace=str(workspace),\n network=\"kumo_forecast\",\n train_specs=schema[\"default\"],\n settings={**settings, \"automl_max_recommendations\": 2},\n action=\"train\",\n search_schema=schema,\n resume=True,\n )\n assert resumed.get_status()[\"progress\"][\"completed\"] == 1\n assert [item.id for item in resumed.next_recommendation()] == [1]\n assert [path.name for path in (workspace / \".automl/controller\").glob(\"*.json\")] == [\n f\"{session_id}.json\"\n ]\n","content_type":"text/x-python; charset=utf-8","language":"python","size":5401,"content_sha256":"34f852edc92c1be4e31d8abbb86690e023343aa08ebb152a957061a48e23ae33"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"TAO AutoML","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"Standalone install?","type":"text","marks":[{"type":"strong"}]},{"text":" If this session was not initialized by the TAO skill bank plugin, run ","type":"text"},{"text":"tao-setup","type":"text","marks":[{"type":"code_inline"}]},{"text":" first (host preflight, credentials, cross-skill discovery).","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Run automated hyperparameter optimization for a TAO model by combining:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"The selected model skill under ","type":"text"},{"text":"skills/models/\u003cmodel_skill>/","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"The selected platform skill under ","type":"text"},{"text":"skills/platform/\u003cplatform>/","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"AutoMLRunner","type":"text","marks":[{"type":"code_inline"}]},{"text":", which generates recommendations, launches selected action jobs, extracts metrics, and feeds results back to the optimizer.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Do not launch until model metadata, platform preflight, data visibility, credentials, image choice, and compute shape are all proven.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Execution Runtime — Hard Gate","type":"text"}]},{"type":"paragraph","content":[{"text":"Every recommendation, baseline evaluation, per-recommendation evaluation, and final evaluation runs in the selected model action's resolved ","type":"text"},{"text":"container_image","type":"text","marks":[{"type":"code_inline"}]},{"text":" by default. Resolve it from the model skill before any training-environment setup. A local checkpoint or Hugging Face model ID does not change this rule.","type":"text"}]},{"type":"paragraph","content":[{"text":"Use venv-based ","type":"text"},{"text":"model execution","type":"text","marks":[{"type":"strong"}]},{"text":" only when explicitly requested. Never infer venv mode from ","type":"text"},{"text":"local-docker","type":"text","marks":[{"type":"code_inline"}]},{"text":", local GPUs, Python, or ","type":"text"},{"text":"pyproject.toml","type":"text","marks":[{"type":"code_inline"}]},{"text":". If absent, execution is container-backed. A host/controller venv for ","type":"text"},{"text":"tao_automl","type":"text","marks":[{"type":"code_inline"}]},{"text":", TAO SDK, or a platform adapter is control-plane-only; keep child model actions in the resolved container image.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Reference Map","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/skill_info.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":": this workflow's structured metadata.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Split detailed references: ","type":"text"},{"text":"automl-preflight-concepts.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" for prerequisites and support checks; ","type":"text"},{"text":"automl-intent-algorithms.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" for search policy; ","type":"text"},{"text":"automl-compression-literature.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" for distill/prune/quantize algorithm sufficiency and future compression-search roadmap; ","type":"text"},{"text":"automl-runner-configuration.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" for runner/API/WandB details; ","type":"text"},{"text":"automl-advanced-monitoring.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" for hooks, resume, and pitfalls; and ","type":"text"},{"text":"automl-examples.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" for conversation examples; and ","type":"text"},{"text":"automl-common-pitfalls.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" for recurring safety checks. ","type":"text"},{"text":"detailed-guide.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" is only the map.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"skills/models/\u003cnetwork>/SKILL.md","type":"text","marks":[{"type":"code_inline"}]},{"text":": model-specific dataset requirements, metrics, HPO notes, checkpoint handoff, and known failures.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"skills/models/\u003cnetwork>/references/skill_info.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":": action contract, container image, inputs, outputs, upload exclusions, and ","type":"text"},{"text":"mode","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"skills/platform/\u003cplatform>/SKILL.md","type":"text","marks":[{"type":"code_inline"}]},{"text":": selected platform preflight, credentials, resource shape, monitoring, and cancellation.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"skills/core/tao-launch-workflow/SKILL.md","type":"text","marks":[{"type":"code_inline"}]},{"text":": shared intake pattern for platform, credentials, dataset visibility, image confirmation, and user confirmation.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Preflight","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Run the shared launch intake. If the user has not chosen a platform, ask; Brev, SLURM, Kubernetes, and Docker are equal peers.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Run the selected platform skill's preflight before generating runner files.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Verify ","type":"text"},{"text":"nvidia-tao-automl","type":"text","marks":[{"type":"code_inline"}]},{"text":" imports:","type":"text"}]}]}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python -c \"import tao_automl; from tao_automl.runner import AutoMLRunner; print('OK')\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Then verify the selected platform's SDK constructs — importing ","type":"text"},{"text":"tao_automl","type":"text","marks":[{"type":"code_inline"}]},{"text":" does not prove the platform backend is installed (e.g. ","type":"text"},{"text":"DockerSDK()","type":"text","marks":[{"type":"code_inline"}]},{"text":" raises ","type":"text"},{"text":"CredentialError","type":"text","marks":[{"type":"code_inline"}]},{"text":" without the ","type":"text"},{"text":"docker","type":"text","marks":[{"type":"code_inline"}]},{"text":" package). See ","type":"text"},{"text":"automl-preflight-concepts.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"If missing, show the exact install command from ","type":"text"},{"text":"versions.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ask before installing:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"SB=\"${TAO_SKILL_BANK_PATH:-~/tao-skill-bank}\"\npip install \"$($SB/scripts/resolve_versions_key.py wheels.tao_automl_\u003cplatform>)\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Valid platform wheel keys are ","type":"text"},{"text":"tao_automl_brev","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"tao_automl_slurm","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"tao_automl_kubernetes","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"tao_automl_docker","type":"text","marks":[{"type":"code_inline"}]},{"text":", and ","type":"text"},{"text":"tao_automl_all","type":"text","marks":[{"type":"code_inline"}]},{"text":". Use ","type":"text"},{"text":"all","type":"text","marks":[{"type":"code_inline"}]},{"text":" only for development machines that need every backend. Add ","type":"text"},{"text":",llm","type":"text","marks":[{"type":"code_inline"}]},{"text":" only when the user requests LLM-guided algorithms.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Model Support Gate","type":"text"}]},{"type":"paragraph","content":[{"text":"Before every run:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Read the model ","type":"text"},{"text":"SKILL.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"references/skill_info.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Confirm ","type":"text"},{"text":"automl_enabled: true","type":"text","marks":[{"type":"code_inline"}]},{"text":" for the model or that the model skill explicitly routes the selected action to AutoML.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Confirm ","type":"text"},{"text":"\u003cskill_dir>/schemas/\u003caction>.schema.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" exists and parses. This is the AutoML search-space gate.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For non-TAO-Core models such as Cosmos-RL and CLIP, also require ","type":"text"},{"text":"references/spec_template_\u003caction>.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":"; otherwise the runner has no complete action defaults.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"If any gate fails, do not improvise a search space. Report the missing package artifact.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Inputs","type":"text"}]},{"type":"paragraph","content":[{"text":"Collect these before runner construction:","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Input","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Requirement","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"model_skill","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Resolved model skill directory under ","type":"text"},{"text":"skills/models/","type":"text","marks":[{"type":"code_inline"}]},{"text":". Resolve user aliases such as ","type":"text"},{"text":"network_arch","type":"text","marks":[{"type":"code_inline"}]},{"text":" to the packaged skill directory first.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"network_arch","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Read from the resolved model skill metadata.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"action","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Action to optimize — ","type":"text"},{"text":"train","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"evaluate","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"inference","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"distill","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"prune","type":"text","marks":[{"type":"code_inline"}]},{"text":", or ","type":"text"},{"text":"quantize","type":"text","marks":[{"type":"code_inline"}]},{"text":" with a packaged schema/template.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"platform","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"One of the supported TAO platform skills.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"train_dataset","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"eval_dataset","type":"text","marks":[{"type":"code_inline"}]},{"text":" / action inputs","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use model-specific spec keys and layout. Non-train actions may also need parent/teacher checkpoints, calibration data, or pruned artifacts.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"results_root","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Local, Lustre, or S3 path appropriate for the platform.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"gpu_count","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"num_nodes","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Respect model and platform limits.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"container_image","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Resolve through model metadata and ","type":"text"},{"text":"versions.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":"; show it to the user.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"automl_algorithm","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Default ","type":"text"},{"text":"bayesian","type":"text","marks":[{"type":"code_inline"}]},{"text":" unless user asks for another algorithm or the model skill recommends one.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"metric","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"direction","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Prefer the model skill's validation/task metric.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"automl_budget","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Recommendation count, max epochs/rungs, concurrency, or population size as required by the algorithm.","type":"text"}]}]}]}]},{"type":"paragraph","content":[{"text":"Never ask for secret values. Verify required env vars with ","type":"text"},{"text":"[ -n \"$VAR_NAME\" ] && echo SET || echo UNSET","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Pre-Launch Review Gate","type":"text"}]},{"type":"paragraph","content":[{"text":"Before launching any recommendation jobs, show a concrete launch review and get user confirmation. This gate applies to every AutoML run for every AutoML-supported model/network; it is not Cosmos-specific and must not be scoped to a single model skill. This applies even when platform and image preflight already passed. The review must include:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"model/network, platform, image, GPU/node shape, and result/workspace root","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"dataset mode and concrete spec keys, including train/eval sample counts when they can be read cheaply","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"algorithm, budget, max concurrent jobs, metric, and direction","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"searchable parameters and ranges, including default values when the user did not provide an explicit search space","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"exact generated recommendation configs for the initial launch batch, produced in a review-only step before any recommendation job is submitted","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"estimated runtime per recommendation and total expected wall time, with the assumptions used","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"the automatic baseline eval job id, metric value, and result path from the post-preflight eval job, or an explicit blocker if the model has no runnable evaluate action or validation data","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"the post-AutoML final evaluation plan for the selected best checkpoint/model, including metric, dataset, and record path","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"If the estimate is longer than the user's stated limit or materially longer than a normal interactive run, ask whether to reduce recommendations, epochs, dataset size, validation frequency, or search space before launch. Do not hide multi-day estimates in logs.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Automatic Baseline Eval Job","type":"text"}]},{"type":"paragraph","content":[{"text":"After platform, image, credential, data, and model preflight pass, run the model's evaluate action once on the selected validation/eval data before submitting any AutoML recommendation jobs. This is required AutoML setup, not an optional \"pretrained eval\" question for the user. Use the same base model or checkpoint that the AutoML training run starts from, the model skill's evaluate spec/template, and the selected platform's normal job submission path. If the model skill recommends a smaller shape for evaluation than training, use that shape and call it out in the launch review.","type":"text"}]},{"type":"paragraph","content":[{"text":"Share the eval metric number in the launch review before asking for confirmation. If a starting checkpoint exists but the baseline cannot be produced — no packaged evaluate action, missing eval dataset, failed eval job — stop and report the blocker instead of silently falling back to a training-loss-only run.","type":"text"}]},{"type":"paragraph","content":[{"text":"For training from scratch, record the baseline as unavailable and proceed; do not evaluate an empty checkpoint.","type":"text"}]},{"type":"paragraph","content":[{"text":"The runner owns final evaluation. When eval is runnable, pass ","type":"text"},{"text":"final_eval_fn(best_rec, train_job_id)","type":"text","marks":[{"type":"code_inline"}]},{"text":" to ","type":"text"},{"text":"AutoMLRunner.run","type":"text","marks":[{"type":"code_inline"}]},{"text":"; the result then carries ","type":"text"},{"text":"result[\"final_evaluation\"]","type":"text","marks":[{"type":"code_inline"}]},{"text":". See ","type":"text"},{"text":"automl-preflight-concepts.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" for the callback, checkpoint, baseline, and from-scratch rules.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Dependency And Data Preflight","type":"text"}]},{"type":"paragraph","content":[{"text":"If the selected workflow needs object storage or a platform CLI and the tool is missing, report the missing dependency and offer the exact install command before continuing. After user approval, rerun ","type":"text"},{"text":"scripts/check_tao_launch_preflight.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" with ","type":"text"},{"text":"--install-missing-tools","type":"text","marks":[{"type":"code_inline"}]},{"text":" so it installs the smallest needed package and immediately retries path verification. For S3 paths, verify both credentials and path readability from the launch platform before creating runner artifacts.","type":"text"}]},{"type":"paragraph","content":[{"text":"For models that read large media archives or directories during every training trial, stage or extract the dataset once to storage visible from the execution platform, then point all recommendation specs at that staged path. Record the source URI, staged path, byte/file-count evidence when available, and timestamp in ","type":"text"},{"text":"\u003cworkspace>/evaluations/data_staging.json","type":"text","marks":[{"type":"code_inline"}]},{"text":". If staging is not possible, include the repeated S3 I/O risk in the pre-launch review and ask before spending a long AutoML budget on it.","type":"text"}]},{"type":"paragraph","content":[{"text":"When the model skill defines sample-count-sensitive constraints, enforce them before launch. Reject or cap every batch-size recommendation that would create zero training steps for the selected dataset and GPU shard count. Use ","type":"text"},{"text":"scripts/check_tao_launch_preflight.py --effective-batch-limit train_annotation=\u003cbatch_size>,\u003cshard_count>","type":"text","marks":[{"type":"code_inline"}]},{"text":" for each generated recommendation before submitting it. If a recommendation later fails because the data is too small for the effective batch size, classify it as an invalid configuration, replace or adjust it only when remaining budget exists, and report the correction in the final summary. When train sample count is known from an annotation file or cheap manifest read, pass it as ","type":"text"},{"text":"automl_settings[\"train_sample_count\"]","type":"text","marks":[{"type":"code_inline"}]},{"text":" to ","type":"text"},{"text":"AutoMLRunner.run","type":"text","marks":[{"type":"code_inline"}]},{"text":" so the runner can cap impossible recommendations before submitting a job and record the adjustment in ","type":"text"},{"text":"result[\"history\"][i][\"adjustments\"]","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Algorithm Policy","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Algorithm","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Good fit","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Required knobs","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"bayesian","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Default for small/medium budgets and few parameters.","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"num_recommendations","type":"text","marks":[{"type":"code_inline"}]},{"text":", metric, direction","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"hyperband","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"asha","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Many configs, cheap early rungs; ASHA is parallel-friendly.","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"max_epochs","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"reduction_factor","type":"text","marks":[{"type":"code_inline"}]},{"text":", optional ","type":"text"},{"text":"max_concurrent","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"bohb","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"dehb","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Mixed Bayesian/evolutionary search with multi-fidelity budgets.","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"same rung budget fields as Hyperband","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"pbt","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Long training where schedules should mutate during training.","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"population and generation budget","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"llm","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"hybrid","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"autoresearch","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"User explicitly wants LLM-guided search with a configured endpoint.","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"LLM endpoint config plus budget","type":"text"}]}]}]}]},{"type":"paragraph","content":[{"text":"For ","type":"text"},{"text":"evaluate","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"inference","type":"text","marks":[{"type":"code_inline"}]},{"text":", default to Bayesian/BFBO-style search over the selected action's prompt, decoding, preprocessing, or runtime config knobs. Use a task metric from the action outputs/logs and set ","type":"text"},{"text":"direction","type":"text","marks":[{"type":"code_inline"}]},{"text":" explicitly when the metric name is ambiguous. Do not use training-loss assumptions for actions that do not update weights.","type":"text"}]},{"type":"paragraph","content":[{"text":"For ","type":"text"},{"text":"distill","type":"text","marks":[{"type":"code_inline"}]},{"text":", use the same train-like policy when the distill action performs epoch-based optimization and writes checkpoints. For single-shot ","type":"text"},{"text":"prune","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"quantize","type":"text","marks":[{"type":"code_inline"}]},{"text":", default to ","type":"text"},{"text":"bayesian","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"bfbo","type":"text","marks":[{"type":"code_inline"}]},{"text":" unless the action schema/model skill declares an epoch-like or calibration-budget field that makes ","type":"text"},{"text":"hyperband","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"asha","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"bohb","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"dehb","type":"text","marks":[{"type":"code_inline"}]},{"text":" meaningful. Use ","type":"text"},{"text":"eval_fn","type":"text","marks":[{"type":"code_inline"}]},{"text":" when the selected metric must be computed by a follow-up evaluate/inference action after the compression action completes.","type":"text"}]},{"type":"paragraph","content":[{"text":"Prefer the model skill's recommendation over generic defaults. Avoid ASHA or Hyperband when the model skill says startup, validation, or checkpoint cost dominates short trials.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Spec And Search Space","type":"text"}]},{"type":"paragraph","content":[{"text":"Build specs as nested dictionaries. If a model skill lists paths in dotted notation for readability, walk the path and assign the nested leaf; do not store flat dotted strings as spec keys.","type":"text"}]},{"type":"paragraph","content":[{"text":"Use the packaged selected-action schema for:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"automl_default_parameters","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"automl_disabled_parameters","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"valid min/max ranges","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"enums, option weights, conditions, dependencies, and popular parameters","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"User-provided search spaces must stay inside schema constraints. For integer knobs with discrete choices, include the schema's required integer option shape instead of a loose list if the model skill calls that out.","type":"text"}]},{"type":"paragraph","content":[{"text":"Data source overrides are mandatory unless the model skill says the launcher can derive them. Preserve exact user-provided spec keys when the dataset uses direct annotation/media paths.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Metric Policy","type":"text"}]},{"type":"paragraph","content":[{"text":"Training loss is cheap but can be misleading. Prefer the model skill's task metric. Use one of these:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Log metric: ","type":"text"},{"text":"metric=\u003cname>","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"direction=maximize|minimize","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"metric_extractor(logs, metric_name)","type":"text","marks":[{"type":"code_inline"}]},{"text":": parse the model's logs when the default resolver is ambiguous.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"eval_fn(rec, train_job_id)","type":"text","marks":[{"type":"code_inline"}]},{"text":": run the model's evaluate action after each recommendation when the user wants a downstream task metric.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Do not map ","type":"text"},{"text":"kpi","type":"text","marks":[{"type":"code_inline"}]},{"text":" to a metric unless the model skill explicitly defines that mapping.","type":"text"}]},{"type":"paragraph","content":[{"text":"The final report must compare the baseline metric, each recommendation's metric, and the selected best metric so users can see the impact of tuning. For model skills that require an ","type":"text"},{"text":"eval_fn","type":"text","marks":[{"type":"code_inline"}]},{"text":" to compute the real task metric, use that evaluator instead of optimizing a convenient training loss unless the user explicitly accepts the proxy metric.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Runner Construction","type":"text"}]},{"type":"paragraph","content":[{"text":"Use the selected platform SDK only after its preflight passes. Construct SDKs without embedding credentials in code.","type":"text"}]},{"type":"paragraph","content":[{"text":"sdk","type":"text","marks":[{"type":"code_inline"}]},{"text":" is the platform SDK object; containerless venv models use ","type":"text"},{"text":"VirtualEnvSDK(venv_path=..., work_dir=...)","type":"text","marks":[{"type":"code_inline"}]},{"text":". Always pass ","type":"text"},{"text":"work_dir","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- the default ","type":"text"},{"text":"~/.tao_sdk/virtualenv","type":"text","marks":[{"type":"code_inline"}]},{"text":" fills the home directory with trial checkpoints. See ","type":"text"},{"text":"automl-runner-configuration.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"python"},"content":[{"text":"import sys\nfrom pathlib import Path\nfrom tao_automl.runner import AutoMLRunner\n\nskill_bank = Path(\"\u003cabsolute-tao-skill-bank>\")\nmodel_skill = \"\u003cresolved-model-skill-directory>\"\nskill_dir = skill_bank / \"skills\" / \"models\" / model_skill\nsys.path.insert(\n 0, str(skill_bank / \"skills/applications/tao-run-automl/scripts\")\n)\nfrom resolve_automl_session import validate_session_settings\n\nrunner = AutoMLRunner(\n sdk=sdk,\n skill_dir=str(skill_dir),\n action=action, # train, distill, prune, quantize, ...\n)\n\nworkspace_path = Path(\"\u003cautoml_workspace>\")\nresume = False\n# Mandatory fail-closed gate from this skill's bundled scripts directory.\nvalidate_session_settings(\n automl_settings,\n resume=resume,\n workspace=workspace_path if resume else None,\n)\nresult = runner.run(\n workspace_path=str(workspace_path), # timestamp it to avoid collisions\n automl_settings=automl_settings, # must contain an explicit session_id\n spec_overrides=spec_overrides,\n automl_hyperparameters=automl_hyperparameters,\n custom_param_ranges=custom_param_ranges,\n metric_extractor=metric_extractor, # optional\n eval_fn=eval_fn, # optional\n final_eval_fn=final_eval_fn, # optional but required when final eval is runnable\n resume=resume,\n)","type":"text"}]},{"type":"paragraph","content":[{"text":"Set ","type":"text"},{"text":"automl_settings[\"session_id\"]","type":"text","marks":[{"type":"code_inline"}]},{"text":" explicitly and call ","type":"text"},{"text":"validate_session_settings","type":"text","marks":[{"type":"code_inline"}]},{"text":" before every run. Generate a fresh ID once with ","type":"text"},{"text":"scripts/resolve_automl_session.py new","type":"text","marks":[{"type":"code_inline"}]},{"text":". Resume only when explicitly requested; resolve its controller with ","type":"text"},{"text":"scripts/resolve_automl_session.py resolve --workspace \u003cfull-run-path>","type":"text","marks":[{"type":"code_inline"}]},{"text":". Missing or ambiguous state is a blocker. See the resume section of ","type":"text"},{"text":"references/automl-advanced-monitoring.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" for the complete fresh/resume pattern.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Monitoring","type":"text"}]},{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"runner","type":"text","marks":[{"type":"code_inline"}]},{"text":" status output and the platform SDK's ","type":"text"},{"text":"get_job_status","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"get_job_logs","type":"text","marks":[{"type":"code_inline"}]},{"text":", and ","type":"text"},{"text":"get_failure_analysis","type":"text","marks":[{"type":"code_inline"}]},{"text":". For active jobs, report:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"recommendation id / trial id","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"platform job id","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"status","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"current metric","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"best metric so far","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"selected hyperparameters for the current/best recommendation","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"elapsed time and updated ETA when enough timing data exists","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"On failure, classify whether it is infrastructure, data visibility, image, credential, spec/schema, or model-code failure. Fix only the minimal cause and do not silently spend additional budget on repeated invalid recommendations. If a blocker is fixed during run setup, continue from the original task after showing the updated preflight/launch review instead of leaving the user to restate the request.","type":"text"}]},{"type":"paragraph","content":[{"text":"For LLM-based algorithms, inspect the brain logs before calling the run valid. Verify that LLM calls succeeded, proposals were generated, prior metrics were used to choose later parameter changes, and logs show keep/discard or equivalent algorithm decisions. If the brain falls back to random sampling, classify the LLM workflow as failed or blocked instead of treating it as a valid LLM-guided run.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Result Handoff","type":"text"}]},{"type":"paragraph","content":[{"text":"At completion:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Identify the best recommendation by the selected metric and direction.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Return the best child job id and its result path.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Resolve the model checkpoint or action artifact using the model skill's checkpoint/artifact metadata and SDK helpers; do not guess filenames such as ","type":"text"},{"text":"latest","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Report the exact search space, algorithm, budget, metric, and platform.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Report the automatic baseline eval job id/result path/metric, all recommendation metrics, final evaluation status/result path/metric, failed recommendations and root causes, elapsed time, and final runtime notes.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"If this feeds a workflow such as AutoML + DEFT, pass the winning spec overrides and checkpoint through the workflow's declared handoff fields.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"With the default retention policy, verify that cleanup-supported, safely prunable terminal trial artifacts were deleted and that the winning training artifacts remain. Report protected promotion/resume parents or conservative Hybrid results explicitly. A remote bind, named volume, or other output route the SDK cannot reclaim must fail retention preflight before the first trial rather than be silently retained.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Common Pitfalls","type":"text"}]},{"type":"paragraph","content":[{"text":"See ","type":"text"},{"text":"references/automl-common-pitfalls.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" before launching or recovering an AutoML run.","type":"text"}]}]},"metadata":{"date":"2026-11-02","name":"tao-run-automl","tags":["automl","hpo","workflow","training","optimization","llm"],"author":"@skillopedia","source":{"stars":3825,"repo_name":"skills","origin_url":"https://github.com/NVIDIA/skills/blob/HEAD/skills/tao-run-automl/SKILL.md","repo_owner":"NVIDIA","body_sha256":"0a2c969473b75e44a2bd1ae1fe3a91d335a6a2ed28dd76ce2905453e2c723aa6","cluster_key":"062015f5c7caefedb03e3516ef1fbeffe9b0d7df23e3ecd5a1dfb5f02409861b","clean_bundle":{"format":"clean-skill-bundle-v1","source":"NVIDIA/skills/skills/tao-run-automl/SKILL.md","text_bytes":159868,"source_commit":"25340619336e89bcc40365c28bfe9e790430468e","skills_sh_rank":9596,"attachment_count":23,"text_attachments":23,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"skills/tao-run-automl/SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"v1","category":"ai-agent-development","category_label":"AI"},"exact_dupes_collapsed_into_this":0},"license":"Apache-2.0","version":"v1","category":"ai-agent-development","metadata":{"author":"NVIDIA Corporation","version":"0.1.1"},"import_tag":"skills-sh-top20k-2026-11","description":"Run container-backed AutoML / hyperparameter optimization (HPO) for NVIDIA TAO networks using AutoMLRunner. Handles algorithm selection (bayesian, hyperband, asha, bohb, llm, hybrid, autoresearch), WandB experiment tracking, job execution on any TAO SDK platform, result interpretation, and per-rec custom evaluation hooks. Use when the user mentions TAO AutoML, hyperparameter optimization, HPO, automl, automl_settings, AutoMLRunner, tao_automl, bayesian search, hyperband, ASHA, LLM-guided search, autoresearch, or wants to tune train/evaluate/inference/distill/prune/quantize for a TAO network. Model actions use the resolved image; venv training requires an explicit request. Platform-agnostic — runs on any SDK (Brev, SLURM, Kubernetes, Docker). Do not use generic \"keep improving\" language alone to override a matching domain-specific DEFT workflow; attribute-labelled CLIP / SigLIP image-retrieval loops belong to tao-run-deft-pas unless HPO is explicit.","allowed-tools":"Read Bash Write","compatibility":"Requires docker + nvidia-container-toolkit. Workflows declare additional requirements."}},"renderedAt":1793671938601}

TAO AutoML Standalone install? If this session was not initialized by the TAO skill bank plugin, run first (host preflight, credentials, cross-skill discovery). Run automated hyperparameter optimization for a TAO model by combining: 1. The selected model skill under . 2. The selected platform skill under . 3. , which generates recommendations, launches selected action jobs, extracts metrics, and feeds results back to the optimizer. Do not launch until model metadata, platform preflight, data visibility, credentials, image choice, and compute shape are all proven. Execution Runtime — Hard Gate…