Kubernetes Standalone install? If this session was not initialized by the TAO skill bank plugin, run the skill first (host preflight, credentials, cross-skill discovery). Submits TAO container jobs as Kubernetes Jobs. Works on any cluster reachable via kubeconfig (EKS / GKE / AKS / on-prem) or in-cluster service account (when running inside a pod). Single-pod by default; opt into multi-node distributed training via (uses Indexed Job + headless Service, see Multi-node training below). Preflight Three checks: GPU host runtime ready, cluster reachable via , GPU Operator/device plugin present. Th…

| head -1)\nif [ -z \"$gpu\" ] || [ \"$gpu\" = \"0\" ]; then\n echo \"WARN: no nvidia.com/gpu allocatable on this cluster.\"\n echo \"Install the NVIDIA GPU Operator before submitting GPU jobs:\"\n echo \" https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html\"\nfi\n```\n\nThe GPU node runtime check is mandatory for self-managed nodes. For managed\nclusters where the client is not running on a GPU worker, verify the provider\nnode image or GPU Operator policy and set `TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1`\ninstead of running the installer on the client. The GPU-capacity warning here is\na soft check; the `submit` verb re-checks allocatable `nvidia.com/gpu` and\nhard-fails before applying the manifest (there is no gang scheduling, so a\ntoo-big Job would sit `Pending` forever).\n\n## Credentials & configuration\n\n- **Kubeconfig** (one of):\n - `~/.kube/config` — default discovery path\n - `$KUBECONFIG` — alternate path\n - In-cluster service account — used when running inside a pod (no kubeconfig needed)\n- **TAO_K8S_NAMESPACE** (optional): default namespace for Job submission. Defaults to `default`.\n- **TAO_K8S_CONTEXT** (optional): kubeconfig context name to switch clusters.\n- **NGC_KEY** (optional): for nvcr.io image pulls. If you've pre-created an image-pull secret in the target namespace, reference its name in the rendered manifest's `imagePullSecrets`.\n- **AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / S3_BUCKET_NAME / S3_ENDPOINT_URL** (optional): for S3 dataset I/O (storage tier C), injected into the pod via the per-job Secret (`envFrom.secretRef`), never inline. Legacy `ACCESS_KEY`/`SECRET_KEY` are mapped by `tao-data-io`.\n\nDo not ask for Brev or SLURM credentials for Kubernetes runs. Ask for\nS3 credentials only when the selected workflow uses `s3://` inputs or outputs,\nand ask for model-specific credentials such as `HF_TOKEN` only when the selected\nmodel requires them. Before launch, verify the selected namespace can create\nJobs, dataset/result paths are visible from the pod, and PVC/mounted filesystem\npaths are proven to be mounted into the job container; an agent-host local path\nis not sufficient proof.\n\n## Execution — the four verbs\n\n`tao-run-on-kubernetes` is a platform **consumer**: it runs a spec-bundle via\n`kubectl`, mutating only the job-record. No nvidia-tao-sdk, no `tao_sdk` import —\njobs are submitted with plain `kubectl apply`.\n`$BANK` = `${TAO_SKILL_BANK_PATH}`.\n\n### submit\n\n1. **GPU-capacity gate — hard-fail first** (no gang scheduling → a too-big Job\n sits `Pending` forever):\n ```bash\n ALLOC=$(kubectl get nodes -o jsonpath='{range .items[*]}{.status.allocatable.nvidia\\.com/gpu}{\"\\n\"}{end}' | awk '{s+=$1} END{print s+0}')\n [ \"${ALLOC:-0}\" -ge \"$NUM_GPUS\" ] || { echo \"insufficient GPUs: need $NUM_GPUS, allocatable $ALLOC\"; exit 1; }\n ```\n2. **Storage tier** (via `tao-data-io`): **A** = mount a bound PVC/NFS holding the\n data (author the mount paths, no fetch — the air-gap answer, and what the\n packaged template does); **C** = ephemeral: an initContainer fetches from S3\n into a shared `emptyDir` and a final step uploads results to S3 before TTL.\n\n **Tier C holds the GPU while it downloads.** A pod reserves `nvidia.com/gpu`\n for its whole lifetime, initContainers included, so a large tier-C fetch —\n or a first-time multi-GB image pull — is billed and reaper-eligible idle GPU\n time, exactly like pulling inside a SLURM allocation. Prefer tier A when the\n data is already on a PVC; choose tier C knowingly, for small inputs.\n\n A producer action request may declare several mounts, including duplicate-\n source aliases for logical and embedded absolute paths. Read\n `references/action-request.md` and use its staging map plus packaged\n renderer. For `mode=config`, materialize the producer's nested spec first,\n stage that exact generated file, and pass it back to the renderer so the\n pod receives a verified read-only config mount. The legacy single-root\n template cannot represent that contract.\n3. **Open the record — mints the id, binds `results_dir`, before launch:**\n ```bash\n JOB_ID=$(\"$BANK/scripts/tao_job_record.py\" open --platform kubernetes --image \"$IMAGE\" \\\n --network-arch \"$ARCH\" --action \"$ACTION\" --storage-tier \"$TIER\" --results-dir \"$RESULTS_DIR\")\n ```\n `results_dir` must be a **mounted (surviving) volume path or an S3 prefix** —\n `ttlSecondsAfterFinished` deletes the Job and its logs after it ends, so\n nothing is recoverable from the Job object later.\n4. **Render, gate, apply, and record RUNNING.** For a producer action request,\n follow `references/action-request.md`; it owns backend-name normalization,\n conditional Secret references, native argv rendering, server dry-run, and\n binding the applied object name to the job-record. For a simple one-root\n spec-bundle, render `templates/k8s/single-pod-job.yaml.tmpl`, run\n `redact_secrets.py lint` plus `kubectl apply --dry-run=server`, apply it, and\n mark the record with `backend-ref=\u003cnamespace>/\u003cactual-object-name>`.\n\nA submit that skipped the gate or the open has no id — so it cannot launch.\n\n### status\n\nKeep `K8S_JOB_NAME` from submit. On reattach, read the job-record's\n`backend_ref=\u003cnamespace>/\u003cname>` and recover both values from that field; do\nnot assume the Kubernetes name equals the record id.\n\n```bash\nkubectl get job \"$K8S_JOB_NAME\" -n \"$NAMESPACE\" \\\n -o jsonpath='{.status.conditions[0].type} {.status.active} {.status.succeeded} {.status.failed}'\n```\n\n| kubectl signal | vocab |\n|---|---|\n| no pods scheduled | `PENDING` (`kubectl get pods -n \"$NAMESPACE\" -l job-name=\"$K8S_JOB_NAME\"` → `ImagePullBackOff` / `Insufficient nvidia.com/gpu` in `message`) |\n| `active` ≥ 1 | `RUNNING` |\n| condition `Complete` | `COMPLETE` |\n| condition `Failed` | `ERROR` (classify from the pod's terminated reason — `OOMKilled` → `ERR_INFRA`) |\n| Job/pod not found | `UNKNOWN` (may be TTL-deleted — the job-record is the source of truth) |\n\n### logs\n\n```bash\nkubectl logs -n \"$NAMESPACE\" -l \"job-name=$K8S_JOB_NAME\" --tail \"${N:-200}\"\n```\n\n### cancel\n\n```bash\nkubectl delete job \"$K8S_JOB_NAME\" -n \"$NAMESPACE\" --cascade=foreground\nif [ -n \"${CRED_SECRET:-}\" ]; then\n kubectl delete secret \"$CRED_SECRET\" -n \"$NAMESPACE\" --ignore-not-found\nfi\n\"$BANK/scripts/tao_job_record.py\" mark \"$JOB_ID\" --state CANCELED --source agent\n```\n\n### Multi-node (nodes > 1)\n\nSame four verbs, plus:\n\n1. **Version gate:** require k8s ≥ 1.28 (`kubectl version -o json`) — the pod\n hostname `\u003cjob>-\u003cindex>` (PodIndexLabel) that `MASTER_ADDR=\u003cjob>-0.\u003csvc>`\n resolves to needs it; on older clusters rank-0 hangs at rendezvous.\n2. **Capacity gate ×nodes:** hard-fail unless allocatable GPUs ≥ `gpus_per_node ×\n nodes` (no gang scheduling → a partial start leaves rank-0 waiting forever).\n3. **Render `templates/k8s/indexed-job.yaml.tmpl`** — the headless Service +\n Indexed Job + rendezvous env (`WORLD_SIZE` = node count, `NODE_RANK` from\n `JOB_COMPLETION_INDEX`, `MASTER_ADDR=\u003cjob>-0.\u003csvc>`, `/dev/shm` 16Gi so NCCL\n doesn't silently hang). `kubectl apply -f` creates the Service and Job together;\n `cancel` deletes the Job (Foreground) and the Service.\n4. **NCCL probe first** (as SLURM) — a 2-node all-reduce with a timeout; on hang,\n set the cluster NCCL env and re-probe; cache per cluster.\n\n## Local cluster (development, CI, and evals)\n\nA throwaway minikube/kind cluster exercises admission, the four verbs,\njob-record wiring, and log plumbing without cluster quota — and is what an\nagent-driven eval should provision for itself. `kubectl` and `minikube` are\nsingle static binaries needing no root, so a non-root CI container can install\nthem itself.\n\nTwo prerequisites keep a rendered Job `Pending`, and the first masks the second:\nthe PVC the template mounts must exist (`persistentvolumeclaim \"\u003cname>\" not\nfound` fires before any GPU complaint), then a Job requesting `nvidia.com/gpu`\non a GPU-less cluster reports `Insufficient nvidia.com/gpu` and waits forever.\nRender `NUM_GPUS=0` for a lifecycle-only run and say GPU scheduling was not\nverified; on a Linux GPU host, `minikube start --driver=docker --gpus all`\npasses real GPUs through, so one GPU box suffices for a GPU-real smoke.\n\nInstall commands, driver choice, the container/host-networking caveat, and the\nfake-device-plugin middle option: `references/local-cluster.md`.\n\n## Container shell\n\nThe simple single-pod template invokes its command via `/bin/sh -c` (POSIX sh,\npresent in busybox/distroless as well as TAO images). For producer action\nrequests, an args-mode command and its arguments are native container argv; a\nproducer that needs a shell declares the shell and its script explicitly. A\nsimple config-mode command also becomes native argv after `{config_path}` is\nsubstituted. A producer-owned config command that is itself a multi-line shell\nscript is preserved verbatim under `/bin/sh -c`; the renderer never constructs\nshell text from config values.\n\n## GPU Operator dependency\n\nThe `submit` verb refuses to launch GPU jobs on a cluster with no `nvidia.com/gpu` allocatable. For self-managed clusters, first run the `tao-setup-nvidia-gpu-host` install action on every GPU worker node or bake the same package set into the node image:\n\n```bash\nbash skills/platform/tao-setup-nvidia-gpu-host/scripts/setup-nvidia-gpu-host.sh --backend kubernetes --install --yes\n```\n\nThen install the NVIDIA GPU Operator or device plugin:\n\n```bash\nhelm repo add nvidia https://helm.ngc.nvidia.com/nvidia\nhelm repo update\nhelm install --wait gpu-operator -n gpu-operator --create-namespace nvidia/gpu-operator\n```\n\nFull guide: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html\n\n## Multi-node training (distributed)\n\nSet `num_nodes > 1` (see the [Multi-node (nodes > 1)](#multi-node-nodes--1) verb\nsteps above) to run distributed training across N pods. Rendering\n`templates/k8s/indexed-job.yaml.tmpl` provisions:\n\n1. A **headless Service** named after the Job (selector: `job-name=\u003cjob-name>`, `clusterIP: None`, `publishNotReadyAddresses: true` so pods can rendezvous before they're all Ready).\n2. An **Indexed Job** with `parallelism = completions = num_nodes`, `completionMode: Indexed`. Each pod gets `JOB_COMPLETION_INDEX` injected by k8s automatically (= the node rank).\n3. A **command wrapper** that exports the rendezvous env vars before invoking the user command. Two naming conventions are exported simultaneously:\n\n | Env var | Value | Read by |\n |---|---|---|\n | `WORLD_SIZE` | `num_nodes` | TAO PyTorch container's `nvidia_tao_pytorch/core/entrypoint.py` (uses this to mean *node count*, even though PyTorch's own convention is *total processes*) |\n | `NUM_GPU_PER_NODE` | `gpu_count` | TAO PyTorch container's entrypoint |\n | `NNODES` | `num_nodes` | `torchrun` and PyTorch-standard rendezvous |\n | `NPROC_PER_NODE` | `gpu_count` | `torchrun` |\n | `NODE_RANK` | `$JOB_COMPLETION_INDEX` | both |\n | `MASTER_ADDR` | `\u003cjob-name>-0.\u003cjob-name>` (pod-0's DNS) | both |\n | `MASTER_PORT` | `29500` | both (TAO's default) |\n\n Both naming conventions are set so TAO entrypoints (`dino train`, etc.) and raw `torchrun` commands work without modification.\n\nFor a TAO entrypoint, the container reads `spec.train.num_nodes` and the wired\nenv vars — e.g. `dino train -e /tmp/spec.yaml` with `gpu_count=8`, `num_nodes=4`\n(4 × 8 = 32 GPUs total).\n\nFor raw `torchrun`-based commands (non-TAO containers), the wrapper invokes:\n\n```bash\ntorchrun --nnodes=$NNODES --nproc-per-node=$NPROC_PER_NODE --node-rank=$NODE_RANK \\\n --master-addr=$MASTER_ADDR --master-port=$MASTER_PORT train.py\n```\n\nThe capacity check sums across nodes: `gpu_count × num_nodes` ≤ cluster's allocatable `nvidia.com/gpu`.\n\n### Cluster requirements for multi-node\n\n- **k8s 1.28+** is required for stable pod hostnames in Indexed Jobs (the `PodIndexLabel` feature). On older clusters the `MASTER_ADDR=\u003cjob>-0.\u003csvc>` DNS lookup fails. Verify with `kubectl version`.\n- **Pod-to-pod networking** must be open on port 29500 (PyTorch default; configurable via `MASTER_PORT` env var). Most CNIs (Calico, Cilium, AWS VPC CNI) allow this by default; restrictive NetworkPolicies must be relaxed.\n- **NCCL** in the container talks GPU-to-GPU; if the cluster has multi-NIC nodes or RDMA, set `NCCL_SOCKET_IFNAME` / `NCCL_IB_HCA` in the container `env` of the rendered manifest.\n\n### Reference reading\n\n- Kubernetes Indexed Job: \u003chttps://kubernetes.io/docs/concepts/workloads/controllers/job/#completion-mode>\n- Indexed Job for batch ML: \u003chttps://kubernetes.io/blog/2022/06/01/indexed-jobs-mpi/>\n- PyTorch distributed (env-var rendezvous): \u003chttps://pytorch.org/docs/stable/elastic/run.html>\n- NCCL networking tuning (NCCL_SOCKET_IFNAME, NCCL_IB_HCA): \u003chttps://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html>\n\n### Kubernetes operator alternatives\n\nFor more sophisticated topologies (gang scheduling, PyTorch elastic / fault-tolerant training, MPI / Horovod, RDMA setup), reach for an operator instead of plain Indexed Job:\n\n- **MPI Operator** — \u003chttps://github.com/kubeflow/mpi-operator> — for MPI / Horovod workloads.\n- **Kubeflow Training Operator** (`PyTorchJob`, `TFJob`) — \u003chttps://www.kubeflow.org/docs/components/training/> — for elastic PyTorch training with built-in restart logic.\n- **Volcano** — \u003chttps://volcano.sh/> — gang scheduling, queues, fair-share. Useful in shared multi-tenant clusters.\n- **Kueue** — \u003chttps://kueue.sigs.k8s.io/> — quota / queue layer on top of any of the above.\n\nThis skill's Indexed Job path is intentionally simple and dependency-free; if you need elastic restart or gang scheduling, layer one of these on top and submit jobs through the operator's CRD instead.\n\n## Common error patterns\n\n**`No nvidia.com/gpu resources allocatable on the cluster`** — the GPU Operator (or NVIDIA Device Plugin) isn't installed. Install per the link above; verify with `kubectl get nodes -o jsonpath='{.items[*].status.allocatable}'`.\n\n**`ImagePullBackOff` / `ErrImagePull`** — the cluster can't pull the image. For nvcr.io: pre-create an image-pull secret in the namespace and reference it as the pod's `imagePullSecrets` in the rendered manifest:\nFeed the key over stdin — `--docker-password=$NGC_KEY` would put the secret in\nargv, where it is visible in the host's process table and shell history:\n```bash\nset -a; source /path/to/.env; set +a # omit if already exported\nkubectl create secret generic ngc-pull-secret -n tao-jobs \\\n --type=kubernetes.io/dockerconfigjson \\\n --from-file=.dockerconfigjson=/dev/stdin \u003c\u003cEOF\n{\"auths\": {\"nvcr.io\": {\"username\": \"\\$oauthtoken\", \"password\": \"${NGC_KEY}\"}}}\nEOF\n# Verify without reading the secret back:\nkubectl get secret ngc-pull-secret -n tao-jobs >/dev/null && echo SECRET_OK\n```\n\n**Pod stays `Pending` forever** — `kubectl describe pod -l job-name=$JOB_ID` shows the scheduling reason in the `Events`. Common causes: insufficient GPU capacity (`Insufficient nvidia.com/gpu`), no node matches the pod's `nodeSelector`, missing image-pull secret, or PVC mount failure.\n\n**`OOMKilled` (exit 137)** — container exceeded memory. Reduce batch size, lower max_length, or add a memory request/limit and target a larger node.\n\n**`CredentialError: Could not authenticate to a Kubernetes cluster`** — neither kubeconfig nor in-cluster auth worked. Run `kubectl get nodes` to verify your config, or set `$KUBECONFIG` to the right path.\n\n## What this skill does NOT support (yet)\n\n- **Elastic / fault-tolerant training.** Indexed Job has `backoff_limit=0` — failures fail the whole training run. For elastic restart (e.g., resume from checkpoint after a node death), use Kubeflow's `PyTorchJob` operator instead.\n- **Gang scheduling.** Indexed Job pods are scheduled independently — no all-or-nothing. Multi-node training will *partially* start if only some pods can be scheduled (rank-0 will hang waiting for peers). For all-or-nothing scheduling on shared clusters, use Volcano or Kueue.\n- **MPI / Horovod.** Use the MPI Operator. The Indexed Job path here is PyTorch-distributed-shaped (env-var rendezvous on `MASTER_ADDR:MASTER_PORT`).\n- **Auto-creating image-pull secrets from `$NGC_KEY`.** You pre-create the secret in the target namespace and pass the name. K8s namespace conventions vary widely, so we keep secret creation explicit.","attachment_filenames":["BENCHMARK.md","config.json","config/skillspector-baseline.yaml","evals/evals.json","references/action-request.md","references/local-cluster.md","references/skill_info.yaml","scripts/render_action_job.py","skill-card.md"],"attachments":[{"filename":"BENCHMARK.md","content":"# Skill Benchmark: tao-run-on-kubernetes\n\n> ✅ **Overall verdict: PASS — Recommended for publication**\n\n## Publication Recommendation\n\nRecommended for publication based on the completed evaluation evidence in this report.\n\n## Evaluation Metadata\n\n- Skill: `tao-run-on-kubernetes`\n- Evaluation date: 2026-09-21\n- Evaluator version: `1.5.6`\n- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`)\n- Tasks: 1 evaluation tasks (1 positive)\n- Dataset digest: `sha256:7ec1d30a7bdad9358b2758fb7f4f6bc53cd6b93ea77c8f66ccfae6b41327ba45` (skill-evaluator-dataset-snapshot/1)\n- Attempts per task: 3\n- Environment: `k8s-sandbox`\n- Tier 2 evidence: required for publication\n- Tier 3 evidence: required for publication\n\nEach task attempt ran in its own isolated sandbox pod.\n\n## What This Report Answers\n\nThe three-tier evaluation checks whether the skill:\n\n- is safe to use;\n- produces correct answers;\n- is discovered and activated when needed;\n- helps the agent complete the user's goal and expected workflow; and\n- avoids wasted skill and tool usage.\n\n## Results at a Glance\n\n| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |\n|---|---:|---:|\n| Overall | 94.2% — baseline ran, but no comparable score was available; uplift unavailable | 70.4% — baseline ran, but no comparable score was available; uplift unavailable |\n| Security | 100.0% → 100.0% (±0.0 points) | 100.0% → 100.0% (±0.0 points) |\n| Correctness | 0.0% → 100.0% (+100.0 points) | 13.3% → 100.0% (+86.7 points) |\n| Discoverability | 100.0% — baseline ran, but no comparable score was available; uplift unavailable | 0.0% — baseline ran, but no comparable score was available; uplift unavailable |\n| Effectiveness | 16.7% → 100.0% (+83.3 points) | 20.0% → 53.3% (+33.3 points) |\n| Efficiency | 71.1% — baseline ran, but no comparable score was available; uplift unavailable | 99.5% → 98.5% (-1.0 points) |\n\n**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.\n\nExample: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.\n\nA partial dimension was calculated from only the available configured signals; review the detailed report before relying on it.\n\n## Token Usage\n\nActual Tier 3 execution usage is reported for every observed agent/case pair and both conditions.\n\n| Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage |\n|---|---|---:|---:|---:|---:|---|\n| claude-code | All cases | 105,650 | 92,269 | N/A | N/A | skill 1/1; base 3/3 |\n| claude-code | tao-run-on-kubernetes-basic | 105,650 | 92,269 | N/A | N/A | skill 1/1; base 3/3 |\n| codex | All cases | 13,884 | 41,571 | N/A | N/A | skill 1/1; base 3/3 |\n| codex | tao-run-on-kubernetes-basic | 13,884 | 41,571 | N/A | N/A | skill 1/1; base 3/3 |\n| ALL AGENTS | Dataset aggregate | 119,534 | 133,840 | N/A | N/A | skill 2/2; base 6/6 |\n\nPrompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. N/A means the relevant trajectory counters were not available; coverage is never estimated.\n\n## Tier Status\n\n| Tier | Purpose | Status | Evidence |\n|---|---|---|---|\n| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 22 finding(s) |\n| Tier 2 | Semantic deduplication | **PASSED** | 2 validator(s); 0 finding(s) |\n| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 1 task(s) |\n\n## Findings and Observations\n\n\u003cdetails>\n\u003csummary>Show detailed findings and successful checks\u003c/summary>\n\n- **MEDIUM** QUALITY/quality_correctness: No documented scripts in table format (`skills/platform/tao-run-on-kubernetes/SKILL.md`)\n- **MEDIUM** QUALITY/quality_correctness: Instructions don't mention 'run_script' (`skills/platform/tao-run-on-kubernetes/SKILL.md`)\n- **MEDIUM** QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.tags' (`skills/platform/tao-run-on-kubernetes/SKILL.md`)\n- **MEDIUM** SCHEMA/frontmatter_field_placement: Root field 'tags' is ignored; use 'metadata.tags' (`skills/platform/tao-run-on-kubernetes/SKILL.md`)\n- **MEDIUM** SCHEMA/folder_hierarchy: Unexpected nesting depth for general skill (`skills/platform/tao-run-on-kubernetes`)\n- 17 additional finding(s) are available in the full evaluation artifacts.\n\n\u003c/details>\n\n## Scoring Methodology\n\n\u003cdetails>\n\u003csummary>Show dimension definitions, source signals, and thresholds\u003c/summary>\n\n| Dimension | Question | Scored signals |\n|---|---|---|\n| Security | Is it safe to use? | `security` (100%) |\n| Correctness | Is the answer correct? | `accuracy` (100%) |\n| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) |\n| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) |\n| Efficiency | Did it avoid wasted tool calls and token usage? | `skill_efficiency` (50%) + `token_efficiency` (50%) |\n\n- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.\n- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.\n- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.\n- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.\n- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).\n- Efficiency is 50% tool-call productivity (the backward-compatible `skill_efficiency` wire id) and 50% `token_efficiency`. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial.\n\nSignals present in this run:\n\n- `security` (Security): unsafe operations, secret leakage, and unauthorized access.\n- `skill_execution` (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed.\n- `skill_efficiency` (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability).\n- `accuracy` (Accuracy): final-answer correctness against the reference answer.\n- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved.\n- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed.\n- `token_efficiency` (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency).\n\n\u003c/details>\n\n## Freshness\n\nRegenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":7111,"content_sha256":"98ac7d224e5220e4102602fe8415b22ddf50445989c8f2a88b7d29b57c5649be"},{"filename":"config.json","content":"{\n \"name\": \"kubernetes\",\n \"type\": \"platform\",\n \"required_credentials\": [],\n \"optional_credentials\": [\n {\n \"name\": \"KUBECONFIG\",\n \"source\": \"env_var\"\n },\n {\n \"name\": \"TAO_K8S_NAMESPACE\",\n \"source\": \"env_var\"\n },\n {\n \"name\": \"TAO_K8S_CONTEXT\",\n \"source\": \"env_var\"\n },\n {\n \"name\": \"NGC_KEY\",\n \"source\": \"env_var\"\n },\n {\n \"name\": \"ACCESS_KEY\",\n \"source\": \"env_var\"\n },\n {\n \"name\": \"SECRET_KEY\",\n \"source\": \"env_var\"\n },\n {\n \"name\": \"S3_ENDPOINT_URL\",\n \"source\": \"env_var\"\n },\n {\n \"name\": \"S3_BUCKET_NAME\",\n \"source\": \"env_var\"\n },\n {\n \"name\": \"CLOUD_REGION\",\n \"source\": \"env_var\"\n },\n {\n \"name\": \"HF_TOKEN\",\n \"source\": \"env_var\"\n }\n ],\n \"resource_defaults\": {\n \"num_nodes\": 1,\n \"num_gpus\": 1,\n \"namespace\": \"default\"\n },\n \"cloud_storage\": {\n \"protocol\": \"aws\",\n \"uri_format\": \"s3://{bucket_name}/{path}\",\n \"metadata_key\": \"aws\"\n },\n \"tags\": [\n \"gpu\",\n \"compute\",\n \"kubernetes\",\n \"k8s\",\n \"remote\"\n ],\n \"description\": \"Kubernetes Job execution on a configured GPU cluster using kubeconfig or in-cluster service account authentication.\"\n}\n","content_type":"application/json; charset=utf-8","language":"json","size":1238,"content_sha256":"0a4633719d029de036465375ff7a2c03ee1200337542fafddb06845708aeccc1"},{"filename":"config/skillspector-baseline.yaml","content":"version: 2\n# Reviewed false positives — NVIDIA/nvskills-ci#67. No content removed.\nrules:\n - id: \"PE*\"\n path: \"SKILL.md\"\n reason: \"Reviewed FP (#67): standard infra path/flag refs in docs (SSH/kubeconfig/keyring/etc-passwd/docker), not privilege escalation.\"\n - id: \"PE*\"\n path: \"config.json\"\n reason: \"Reviewed FP (#67): standard infra path/flag refs in docs (SSH/kubeconfig/keyring/etc-passwd/docker), not privilege escalation.\"\n - id: \"PE*\"\n path: \"references/skill_info.yaml\"\n reason: \"Reviewed FP (#67): standard infra path/flag refs in docs (SSH/kubeconfig/keyring/etc-passwd/docker), not privilege escalation.\"\n - id: \"AE*\"\n path: \"*\"\n reason: \"Reviewed FP: SkillSpector 2.11.0 (SkillEvaluator 1.5.4+) introduced hard runtime budgets (MAX_WORKFLOW_SECONDS=60) that cannot be raised externally; large skills are only partially inspected and the partial read is reported as evasion. Upstream: nvcarps-ci!252. Artifacts are present, small and plain-text; nothing is hidden from analysis.\"\n - id: \"skillspector_execution_failure\"\n path: \"*\"\n reason: \"Reviewed FP: scan aborts via the same 2.11.0 runtime budget and on reference_resolution of slash-containing prose and template paths (PASS/NO_PASS, schemas/\u003caction>.schema.json, nvcr.io image URIs, frontmatter version strings). Not a property of the skill content. Upstream: nvcarps-ci!252.\"\n - id: \"LP1\"\n path: \"scripts/render_action_job.py\"\n reason: \"Reviewed FP (#67): MCP least-privilege rule applied to a non-MCP skill. The TAO skill schema declares tooling via allowed-tools and has no permissions field, so the capability cannot be declared.\"\n - id: \"PE5\"\n path: \"eval.config\"\n reason: \"Reviewed FP (#67): documented container runtime flag in a usage example; required for GPU/dataloader workloads and not a privilege escalation performed by the skill.\"\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":1866,"content_sha256":"56c204388212fcd536ec0e3447946b05d986b2a55549b90116b61f1552cf3b03"},{"filename":"evals/evals.json","content":"[\n {\n \"id\": \"tao-run-on-kubernetes-basic\",\n \"question\": \"A user request: \\\"Kubernetes execution platform \\u2014 submits TAO container jobs as single-pod k8s Jobs with NVIDIA GPU scheduling.\\\" Identify which TAO skill applies and, reading only that skill's documentation, outline the steps it prescribes. Do NOT run any commands, scripts, web searches, or other tools \\u2014 describe the plan only.\",\n \"expected_skill\": \"tao-run-on-kubernetes\",\n \"expected_script\": null,\n \"ground_truth\": \"Identify tao-run-on-kubernetes as the applicable skill and summarize its documented workflow from SKILL.md without executing anything.\",\n \"expected_behavior\": [\n \"Identifies tao-run-on-kubernetes as the relevant skill\",\n \"Outlines the documented workflow steps from SKILL.md\",\n \"Does not run commands, scripts, or web searches\"\n ]\n }\n]\n","content_type":"application/json; charset=utf-8","language":"json","size":862,"content_sha256":"6fff5924664d9778d65e67b273b57e6dc4236c7a80aa0a76498e100ae4d017df"},{"filename":"references/action-request.md","content":"# Producer Action Requests\n\nRead this reference before consuming a platform-neutral action request that\ncontains a `mounts` array. The simple one-root spec-bundle template cannot\npreserve duplicate-source aliases or per-target access modes.\n\nThe renderer accepts action-request schema version `\"1\"` only and fails closed\non a missing or unsupported `request.schema_version`; update the renderer and\nthis contract together before consuming a future version.\n\n## Materialize config-mode bundles, then stage every source\n\nFor `spec_bundle.mode=config`, the producer supplies a nested `spec` as\ncontent, not a launcher-local path. The Kubernetes consumer must first\nserialize that content and later replace the command's `{config_path}` with\nthe file's in-container path:\n\n```bash\nCONFIG_RENDER_ARGS=()\nif [ \"$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))[\"spec_bundle\"][\"mode\"])' \\\n \"$ACTION_REQUEST\")\" = \"config\" ]; then\n CONFIG_SOURCE=$(python3 \\\n \"$BANK/skills/platform/tao-run-on-kubernetes/scripts/render_action_job.py\" \\\n materialize-config --request \"$ACTION_REQUEST\" \\\n --output-dir \"$WORKSPACE/.tao/action-configs\")\n CONFIG_RENDER_ARGS+=(--config-source \"$CONFIG_SOURCE\")\nfi\n```\n\nThe materializer supports the contract's `yaml`, `json`, and `toml` formats.\nIt writes an atomic, content-addressed file and returns its absolute path.\nTreat that returned path as one additional consumer-derived staging source:\ncopy the exact file onto the PVC and add exactly one row for it to the staging\nreceipt. Do not rewrite the producer bundle as `mode=args`, hand-author a\nsecond config, or substitute a launcher path that the pod cannot see.\n\nUse `tao-data-io` when the producer source is not already visible from the\nKubernetes compute frame. Mirror each distinct `request.mounts[].source` into\na job-owned relative path on one bound PVC. If `CONFIG_SOURCE` is set, mirror\nthat exact file too. Use delete semantics when the producer requires remote\nfreshness. Record exactly one row per distinct source, including the generated\nconfig when present:\n\n```json\n{\n \"schema_version\": \"1\",\n \"sources\": [\n {\"source\": \"/launcher/run/results\", \"sub_path\": \"jobs/action-123/results\"},\n {\"source\": \"/launcher/run/config\", \"sub_path\": \"jobs/action-123/results/config\"},\n {\"source\": \"/launcher/cache\", \"sub_path\": \"jobs/action-123/cache\"},\n {\"source\": \"/launcher/.tao/action-configs/tao-action-config-\u003csha256>.yaml\", \"sub_path\": \"jobs/action-123/action-config.yaml\"}\n ]\n}\n```\n\nOmit the last row for `mode=args`. The staged subpaths must already exist.\nVerify read access and required write\naccess from a pod before opening the job-record. For an application freshness\ncontract, also verify every declared absent path and persist its staging\nreceipt before opening the record. Never add an undeclared source or use an\nincremental copy that can retain stale outputs.\n\n## Record, credentials, and backend name\n\nOpen the job-record only after the staging/freshness gate. Job-record ids allow\ncharacters and lengths that Kubernetes object names do not. Derive, never\nhand-edit, the collision-resistant backend name; the manifest retains the\noriginal record id in an annotation:\n\n```bash\nK8S_JOB_NAME=$(python3 \\\n \"$BANK/skills/platform/tao-run-on-kubernetes/scripts/render_action_job.py\" \\\n name --job-id \"$JOB_ID\")\n```\n\nWhen `request.forward_env` is non-empty, create `CRED_SECRET` after the record\nis open and feed exactly those approved variables through an env-file on\nstdin. For example, for a request that forwards these three names:\n\n```bash\nCRED_SECRET=\"${K8S_JOB_NAME}-creds\"\nset -a; source /path/to/.env; set +a # omit if already exported\nprintf 'AWS_ACCESS_KEY_ID=%s\\nAWS_SECRET_ACCESS_KEY=%s\\nHF_TOKEN=%s\\n' \\\n \"$AWS_ACCESS_KEY_ID\" \"$AWS_SECRET_ACCESS_KEY\" \"$HF_TOKEN\" \\\n | kubectl create secret generic \"$CRED_SECRET\" -n \"$NAMESPACE\" \\\n --from-env-file=/dev/stdin\n```\n\nThe manifest projects each approved `forward_env` name from the same-named key\nwith an individual `env.valueFrom.secretKeyRef`; unrelated keys in the Secret\nare not imported. Omit both the credential Secret and `--credential-secret`\nwhen `forward_env` is empty (the renderer rejects that mismatch). Likewise,\npass an image-pull Secret only when the namespace needs one; do not create a\ndependency on a placeholder Secret.\n\n## Render and gate\n\nThe renderer validates the immutable command/image/GPU shape, exact source\nmap, unique targets, safe relative PVC subpaths, writable coverage for fresh\noutputs, duplicate-source aliases, read-only flags, and conditional Secret\nreferences. For `mode=config`, it also verifies that `CONFIG_SOURCE` is the\ncanonical serialization of `spec_bundle.spec`, mounts that exact staged file\nread-only at a content-addressed path, and substitutes the path for every\n`{config_path}` occurrence. A stale, altered, unstaged, or symlinked config is\nrejected before a manifest is emitted.\n\nSimple commands are emitted as native `command`/`args` arrays. A producer-owned\nconfig command that is itself a multi-line shell script (for example, a\nCosmos-RL action that discovers its hook inside the image) is preserved\nverbatim under `/bin/sh -c`; values from the config are never interpolated as\nshell text:\n\n```bash\nRENDER_ARGS=()\nif [ -n \"${CRED_SECRET:-}\" ]; then\n RENDER_ARGS+=(--credential-secret \"$CRED_SECRET\")\nfi\nif [ -n \"${IMAGE_PULL_SECRET:-}\" ]; then\n RENDER_ARGS+=(--image-pull-secret \"$IMAGE_PULL_SECRET\")\nfi\n\npython3 \"$BANK/skills/platform/tao-run-on-kubernetes/scripts/render_action_job.py\" \\\n render --request \"$ACTION_REQUEST\" --staging-map \"$STAGING_MAP\" \\\n --job-id \"$JOB_ID\" --namespace \"$NAMESPACE\" --pvc-claim \"$PVC_CLAIM\" \\\n \"${CONFIG_RENDER_ARGS[@]}\" \"${RENDER_ARGS[@]}\" >\"$MANIFEST\"\n\n\"$BANK/scripts/redact_secrets.py\" lint \"$MANIFEST\"\nkubectl apply --dry-run=server -f \"$MANIFEST\"\n```\n\nThe renderer uses `templates/k8s/action-job.yaml.tmpl`. Do not use the legacy\n`single-pod-job.yaml.tmpl` for an action request with multiple mounts.\n\nApply and bind the real backend name to the record:\n\n```bash\nK8S_OBJECT=$(kubectl apply -f \"$MANIFEST\" -o name)\n[ \"$K8S_OBJECT\" = \"job.batch/$K8S_JOB_NAME\" ] || {\n echo \"unexpected Kubernetes object: $K8S_OBJECT\" >&2\n exit 1\n}\n\"$BANK/scripts/tao_job_record.py\" mark \"$JOB_ID\" --state RUNNING \\\n --backend-ref \"$NAMESPACE/$K8S_JOB_NAME\"\n```\n\nOn reattach, recover namespace and backend name from that `backend_ref`; never\nassume the Kubernetes object name equals the record id. Status, logs, and\ncancel use the backend name. Delete `CRED_SECRET` after terminal log/output\ncollection, when it was created.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":6613,"content_sha256":"9202c2b49b48c2522e2e523cf0ee05fc586d6783ebe90894b30b72f277c24cd3"},{"filename":"references/local-cluster.md","content":"# Local Kubernetes cluster (development, CI, and agent-driven evals)\n\nA throwaway local cluster exercises manifest admission, the four verbs,\njob-record wiring, and log plumbing without touching cluster quota. It is what\nan agent-driven eval should provision for itself.\n\n## Installing the tools\n\n`kubectl` and `minikube` are single static binaries, so neither needs root — a\npackage manager is not required, which matters in a CI or eval container that\nruns as a non-root user and has no `sudo`. Install to any directory on `$PATH`\nthat you can write (e.g. `~/.local/bin`):\n\n```bash\nmkdir -p \"$HOME/.local/bin\"; export PATH=\"$HOME/.local/bin:$PATH\"\n\n# kubectl\ncurl -fsSLo \"$HOME/.local/bin/kubectl\" \\\n \"https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl\"\nchmod +x \"$HOME/.local/bin/kubectl\"\n\n# minikube\ncurl -fsSLo \"$HOME/.local/bin/minikube\" \\\n https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64\nchmod +x \"$HOME/.local/bin/minikube\"\n```\n\nSwap `linux/amd64` for your platform (`darwin-arm64` on Apple Silicon).\n\n## Starting and tearing down\n\n```bash\n# Linux / Intel with a Docker daemon — the usual CI case.\nminikube start --driver=docker --cpus=2 --memory=3g\n\n# Apple Silicon, or any host where the Docker daemon is unavailable or\n# restricted: vfkit needs no daemon (brew install vfkit).\nminikube start --driver=vfkit --cpus=2 --memory=3g\n\nkubectl get nodes # STATUS Ready before submitting\n...\nminikube delete # the cluster is disposable — always tear it down\n```\n\n`kind` is an alternative (`kind create cluster` / `kind delete cluster`) but\nrequires a working Docker daemon; minikube's vfkit/qemu drivers do not.\n\n**Inside a container with the host Docker socket mounted** (the skill-eval\nsetup), `--driver=docker` makes minikube create its cluster as a *sibling*\ncontainer on the host. That works, and `kubectl` can reach the API server\n**only because the eval container uses host networking** — minikube binds the\nAPI server on the host, so a bridge-networked container would fail to connect.\nDo not switch that container to bridge networking.\n\n## GPUs on a local cluster\n\nA default local cluster advertises **no** `nvidia.com/gpu`, so a Job requesting\none waits forever. Three options, in increasing fidelity:\n\n1. **Lifecycle only** — render with `NUM_GPUS=0`. Validates admission, the four\n verbs, and logs. Cannot say anything about GPU scheduling; say so explicitly\n rather than implying GPU coverage.\n2. **Scheduling without hardware** — install a fake device plugin that\n advertises `nvidia.com/gpu` capacity. The Job then schedules and you exercise\n the resource key, limits, and node selection, though nothing CUDA runs.\n3. **Real GPUs** — on a Linux host with a GPU and the NVIDIA Container Toolkit,\n minikube passes them through:\n ```bash\n minikube start --driver=docker --container-runtime=docker --gpus all\n kubectl get nodes -o jsonpath='{.items[*].status.allocatable.nvidia\\.com/gpu}'\n ```\n This is why a single GPU box is enough for a GPU-real k8s smoke — no managed\n cluster required. Verify the allocatable count is non-zero before concluding\n the passthrough worked.\n\n## Two things that keep a rendered TAO Job `Pending`\n\nBoth are prerequisites rather than bugs, and both were reproduced against\nminikube. The order matters — the first masks the second:\n\n1. **The PVC must exist.** The templates mount `@@PVC_CLAIM@@`; without it the\n scheduler reports `persistentvolumeclaim \"\u003cname>\" not found`, and that fires\n *before* any GPU complaint. minikube's default StorageClass binds a plain\n claim immediately:\n ```bash\n kubectl create -f - \u003c\u003c'EOF'\n apiVersion: v1\n kind: PersistentVolumeClaim\n metadata: {name: edgeai-datasets}\n spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 1Gi}}}\n EOF\n ```\n2. **GPU capacity must be advertised.** With the PVC satisfied, a Job requesting\n `nvidia.com/gpu` on a GPU-less cluster reports\n `0/1 nodes are available: 1 Insufficient nvidia.com/gpu` and waits forever.\n\nSize any smoke test or eval against which of the three GPU options above is\nactually in play, and report that honestly — a GPU-less pass is not GPU\ncoverage.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":4270,"content_sha256":"19fefa61905635f2e8150d636e6ca65347600febf70a9728e8680b7755158060"},{"filename":"references/skill_info.yaml","content":"type: platform\nrequired_credentials: [] # Auth via kubeconfig (~/.kube/config) or in-cluster service account\noptional_credentials:\n- name: TAO_K8S_NAMESPACE\n source: env_var\n description: Default namespace for Job submission (defaults to 'default').\n- name: TAO_K8S_CONTEXT\n source: env_var\n description: kubeconfig context name to switch clusters.\n- name: KUBECONFIG\n source: env_var\n description: Alternate kubeconfig path (defaults to ~/.kube/config).\n- name: NGC_KEY\n source: env_var\n description: For nvcr.io image pulls (use with image_pull_secret).\n- name: ACCESS_KEY\n source: env_var\n- name: SECRET_KEY\n source: env_var\n- name: S3_ENDPOINT_URL\n source: env_var\n- name: S3_BUCKET_NAME\n source: env_var\n- name: CLOUD_REGION\n source: env_var\n- name: HF_TOKEN\n source: env_var\nresource_defaults:\n ttl_seconds_after_finished: 3600 # auto-clean Jobs 1h after terminal state\n backoff_limit: 0 # no auto-retry; surface failure to the user\n restart_policy: Never\n num_nodes: 1 # default only — multi-node runs via the Indexed Job template\ncloud_storage:\n protocol: aws\n uri_format: s3://{bucket_name}/{path}\n metadata_key: aws\nfeatures:\n- gpu-scheduling\n- failure-analysis\n- log-streaming\n- pod-diagnostics\n- multi-node\n- producer-action-mount-aliases\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":1278,"content_sha256":"05a2e11c1c3db4e739622a34a9fb5aa9bb1d8e3e375578573fb12b8f478c2937"},{"filename":"scripts/render_action_job.py","content":"#!/usr/bin/env python3\n# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n# SPDX-License-Identifier: Apache-2.0\n\n\"\"\"Render one producer action request as a single-pod Kubernetes Job.\n\nThe data mover first mirrors each distinct request ``mounts[].source`` into a\njob-owned directory on one PVC. Its receipt records the relative PVC path for\neach source. This renderer preserves every producer-declared target and access\nmode, including multiple target aliases for the same source, by mounting that\nPVC subPath once per target.\n\nCredentials are deliberately not accepted as values. Each ``forward_env``\nname is projected from the same key of an optional, pre-created Secret; keys\nthat the producer did not approve are never imported. Registry authentication\nis an independent optional image-pull Secret.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport argparse\nimport hashlib\nimport json\nimport math\nimport os\nimport pathlib\nimport posixpath\nimport re\nimport shlex\nimport stat\nimport sys\nimport tempfile\nfrom typing import Any\n\n\nREPO_ROOT = pathlib.Path(__file__).resolve().parents[4]\nDEFAULT_TEMPLATE = REPO_ROOT / \"templates/k8s/action-job.yaml.tmpl\"\nMARKER_RE = re.compile(r\"@@[A-Z][A-Z0-9_]*@@\")\nENV_NAME_RE = re.compile(r\"[A-Za-z_][A-Za-z0-9_]*\")\nDNS_LABEL_RE = re.compile(r\"[a-z0-9](?:[-a-z0-9]*[a-z0-9])?\")\nDNS_SUBDOMAIN_RE = re.compile(\n r\"[a-z0-9](?:[-a-z0-9.]*[a-z0-9])?\"\n)\nQUANTITY_RE = re.compile(r\"[1-9][0-9]*(?:Ki|Mi|Gi|Ti|Pi|Ei)\")\nMAX_JOB_NAME = 52\nREQUEST_SCHEMA_VERSION = \"1\"\nCONFIG_FORMATS = {\"json\": \"json\", \"toml\": \"toml\", \"yaml\": \"yaml\"}\nSHELL_META = (\"\\n\", \";\", \"&&\", \"||\", \"$(\", \"`\", \"|\", \">\", \"\u003c\")\n\n\nclass RenderError(ValueError):\n \"\"\"The action request or staging receipt cannot be rendered safely.\"\"\"\n\n\ndef _load_object(path: pathlib.Path, label: str) -> dict[str, Any]:\n try:\n payload = json.loads(path.read_text(encoding=\"utf-8\"))\n except (OSError, json.JSONDecodeError) as exc:\n raise RenderError(f\"cannot read {label} JSON {path}: {exc}\") from exc\n if not isinstance(payload, dict):\n raise RenderError(f\"{label} JSON root must be an object\")\n return payload\n\n\ndef _nonempty_string(value: Any, label: str) -> str:\n if not isinstance(value, str) or not value or \"\\x00\" in value:\n raise RenderError(f\"{label} must be a non-empty string without NUL\")\n return value\n\n\ndef _string_value(value: Any, label: str) -> str:\n if not isinstance(value, str) or \"\\x00\" in value:\n raise RenderError(f\"{label} must be a string without NUL\")\n return value\n\n\ndef _dns_name(value: str, label: str, *, label_only: bool = False) -> str:\n pattern = DNS_LABEL_RE if label_only else DNS_SUBDOMAIN_RE\n maximum = 63 if label_only else 253\n if len(value) > maximum or pattern.fullmatch(value) is None:\n kind = \"DNS label\" if label_only else \"DNS subdomain\"\n raise RenderError(f\"{label} must be a valid Kubernetes {kind}\")\n return value\n\n\ndef kubernetes_job_name(job_id: str) -> str:\n \"\"\"Map an immutable job-record id to a stable Kubernetes-safe name.\n\n Job-record components may contain underscores, uppercase letters, dots, or\n enough text to exceed Kubernetes' practical Job-name length. Preserve an\n already-safe short id; otherwise append a digest so normalization and\n truncation cannot merge two record ids into one backend object.\n \"\"\"\n original = _nonempty_string(job_id, \"job id\")\n if len(original) \u003c= MAX_JOB_NAME and DNS_LABEL_RE.fullmatch(original):\n return original\n slug = re.sub(r\"[^a-z0-9]+\", \"-\", original.lower()).strip(\"-\")\n if not slug:\n slug = \"tao-job\"\n digest = hashlib.sha256(original.encode(\"utf-8\")).hexdigest()[:10]\n prefix = slug[: MAX_JOB_NAME - len(digest) - 1].rstrip(\"-\") or \"tao-job\"\n return f\"{prefix}-{digest}\"\n\n\ndef _absolute_clean_path(value: Any, label: str) -> str:\n path = _nonempty_string(value, label)\n if (\n not path.startswith(\"/\")\n or path == \"/\"\n or \"\\\\\" in path\n or any(ord(ch) \u003c 32 for ch in path)\n ):\n raise RenderError(f\"{label} must be an absolute non-root POSIX path\")\n normalized = posixpath.normpath(path)\n if normalized != path or \"//\" in path:\n raise RenderError(f\"{label} must be normalized and contain no traversal\")\n return path\n\n\ndef _safe_sub_path(value: Any, label: str) -> str:\n path = _nonempty_string(value, label)\n if path.startswith(\"/\") or \"\\\\\" in path or any(ord(ch) \u003c 32 for ch in path):\n raise RenderError(f\"{label} must be a safe relative POSIX path\")\n normalized = posixpath.normpath(path)\n if normalized != path or path in {\".\", \"..\"} or path.startswith(\"../\"):\n raise RenderError(f\"{label} must be normalized and contain no traversal\")\n return path\n\n\ndef _toml_key(value: Any) -> str:\n key = _nonempty_string(value, \"TOML key\")\n if re.fullmatch(r\"[A-Za-z0-9_-]+\", key):\n return key\n return json.dumps(key)\n\n\ndef _toml_value(value: Any) -> str:\n if isinstance(value, bool):\n return \"true\" if value else \"false\"\n if isinstance(value, int):\n return str(value)\n if isinstance(value, float):\n if not math.isfinite(value):\n raise RenderError(\"TOML config values must be finite\")\n return repr(value)\n if isinstance(value, str):\n return json.dumps(value)\n if isinstance(value, list):\n return \"[\" + \", \".join(_toml_value(item) for item in value) + \"]\"\n if isinstance(value, dict):\n return \"{ \" + \", \".join(\n f\"{_toml_key(key)} = {_toml_value(item)}\"\n for key, item in sorted(value.items())\n ) + \" }\"\n raise RenderError(f\"no TOML representation for {type(value).__name__}\")\n\n\ndef _toml_document(spec: dict[str, Any], prefix: tuple[str, ...] = ()) -> str:\n scalars: list[str] = []\n tables: list[tuple[str, dict[str, Any]]] = []\n for raw_key in sorted(spec):\n key = _toml_key(raw_key)\n value = spec[raw_key]\n if isinstance(value, dict):\n tables.append((key, value))\n else:\n scalars.append(f\"{key} = {_toml_value(value)}\")\n sections: list[str] = []\n if scalars:\n sections.append(\"\\n\".join(scalars))\n for key, value in tables:\n name = \".\".join((*prefix, key))\n body = _toml_document(value, (*prefix, key)).rstrip(\"\\n\")\n sections.append(f\"[{name}]\" + (f\"\\n{body}\" if body else \"\"))\n return \"\\n\\n\".join(sections) + \"\\n\"\n\n\ndef _config_content(bundle: dict[str, Any]) -> tuple[str, str, str]:\n spec = bundle.get(\"spec\")\n if not isinstance(spec, dict):\n raise RenderError(\"spec_bundle.spec must be a nested object for mode=config\")\n raw_format = bundle.get(\"config_format\")\n if raw_format not in CONFIG_FORMATS:\n raise RenderError(\n \"spec_bundle.config_format must be one of json, toml, or yaml\"\n )\n extension = CONFIG_FORMATS[raw_format]\n try:\n if raw_format == \"toml\":\n content = _toml_document(spec)\n else:\n # JSON is valid YAML. Keeping both formats on the stdlib serializer\n # avoids making Kubernetes submission depend on a host YAML package.\n content = json.dumps(\n spec, allow_nan=False, indent=2, sort_keys=True\n ) + \"\\n\"\n except (TypeError, ValueError) as exc:\n raise RenderError(f\"spec_bundle.spec is not {raw_format}-serializable: {exc}\") from exc\n digest = hashlib.sha256(content.encode(\"utf-8\")).hexdigest()\n return content, extension, digest\n\n\ndef _read_regular_file(path: pathlib.Path, label: str) -> bytes:\n flags = os.O_RDONLY | getattr(os, \"O_NOFOLLOW\", 0)\n try:\n descriptor = os.open(path, flags)\n except OSError as exc:\n raise RenderError(f\"cannot open {label} {path}: {exc}\") from exc\n try:\n if not stat.S_ISREG(os.fstat(descriptor).st_mode):\n raise RenderError(f\"{label} must be a regular non-symlink file: {path}\")\n with os.fdopen(descriptor, \"rb\") as handle:\n descriptor = -1\n return handle.read()\n except OSError as exc:\n raise RenderError(f\"cannot read {label} {path}: {exc}\") from exc\n finally:\n if descriptor >= 0:\n os.close(descriptor)\n\n\ndef materialize_config(\n request: dict[str, Any], output_dir: pathlib.Path\n) -> pathlib.Path:\n \"\"\"Write one content-addressed config file for later PVC staging.\"\"\"\n if request.get(\"schema_version\") != REQUEST_SCHEMA_VERSION:\n raise RenderError(\n f\"request.schema_version must be {REQUEST_SCHEMA_VERSION!r}\"\n )\n if request.get(\"platform\") != \"kubernetes\":\n raise RenderError(\"request.platform must be 'kubernetes'\")\n bundle = request.get(\"spec_bundle\")\n if not isinstance(bundle, dict):\n raise RenderError(\"request.spec_bundle must be an object\")\n if bundle.get(\"mode\") != \"config\":\n raise RenderError(\"materialize-config requires spec_bundle.mode=config\")\n if \"args\" in bundle:\n raise RenderError(\"spec_bundle.mode=config must not carry args\")\n content, extension, digest = _config_content(bundle)\n\n raw_output_dir = output_dir.expanduser()\n if raw_output_dir.is_symlink():\n raise RenderError(\"config output directory must not be a symlink\")\n try:\n raw_output_dir.mkdir(parents=True, exist_ok=True)\n root = raw_output_dir.resolve(strict=True)\n except OSError as exc:\n raise RenderError(f\"cannot prepare config output directory: {exc}\") from exc\n if root == pathlib.Path(root.anchor) or not root.is_dir():\n raise RenderError(\"config output directory must be a non-root directory\")\n\n destination = root / f\"tao-action-config-{digest}.{extension}\"\n encoded = content.encode(\"utf-8\")\n if destination.exists() or destination.is_symlink():\n if _read_regular_file(destination, \"materialized config\") != encoded:\n raise RenderError(f\"materialized config content mismatch: {destination}\")\n return destination\n\n temporary: pathlib.Path | None = None\n try:\n with tempfile.NamedTemporaryFile(\n mode=\"wb\",\n prefix=destination.name + \".\",\n suffix=\".tmp\",\n dir=root,\n delete=False,\n ) as handle:\n temporary = pathlib.Path(handle.name)\n os.fchmod(handle.fileno(), 0o600)\n handle.write(encoded)\n handle.flush()\n os.fsync(handle.fileno())\n try:\n # link() publishes the complete file without ever replacing an\n # existing path. Concurrent materializers either create the same\n # content-addressed file or verify the winner below.\n os.link(temporary, destination)\n except FileExistsError:\n if _read_regular_file(destination, \"materialized config\") != encoded:\n raise RenderError(\n f\"materialized config content mismatch: {destination}\"\n )\n except OSError as exc:\n raise RenderError(f\"cannot materialize config in {root}: {exc}\") from exc\n finally:\n if temporary is not None:\n try:\n temporary.unlink()\n except FileNotFoundError:\n pass\n return destination\n\n\ndef _is_shell_script(command: str) -> bool:\n return any(marker in command for marker in SHELL_META)\n\n\ndef _command_bundle(\n request: dict[str, Any], config_path: str | None\n) -> tuple[list[str], list[str], str, int]:\n bundle = request.get(\"spec_bundle\")\n if not isinstance(bundle, dict):\n raise RenderError(\"request.spec_bundle must be an object\")\n mode = bundle.get(\"mode\")\n command_text = _nonempty_string(bundle.get(\"command\"), \"spec_bundle.command\")\n if mode == \"args\":\n if \"spec\" in bundle or \"config_format\" in bundle:\n raise RenderError(\n \"spec_bundle.mode=args must not carry spec or config_format\"\n )\n if config_path is not None:\n raise RenderError(\"--config-source must be omitted for spec_bundle.mode=args\")\n raw_args = bundle.get(\"args\")\n if not isinstance(raw_args, list):\n raise RenderError(\"spec_bundle.args must be an array\")\n args = [\n _string_value(value, f\"spec_bundle.args[{index}]\")\n for index, value in enumerate(raw_args)\n ]\n elif mode == \"config\":\n if \"args\" in bundle:\n raise RenderError(\"spec_bundle.mode=config must not carry args\")\n _config_content(bundle)\n if config_path is None:\n raise RenderError(\n \"spec_bundle.mode=config requires --config-source; run \"\n \"materialize-config, stage the returned file, then render\"\n )\n if \"{config_path}\" not in command_text:\n raise RenderError(\n \"spec_bundle.mode=config requires {config_path} in command\"\n )\n command_text = command_text.replace(\"{config_path}\", config_path)\n args = []\n else:\n raise RenderError(\"spec_bundle.mode must be 'args' or 'config'\")\n\n if mode == \"config\" and _is_shell_script(command_text):\n command = [\"/bin/sh\", \"-c\"]\n args = [command_text, \"tao-action\", *args]\n else:\n try:\n command = shlex.split(command_text)\n except ValueError as exc:\n raise RenderError(f\"spec_bundle.command cannot be parsed: {exc}\") from exc\n if not command:\n raise RenderError(\"spec_bundle.command must contain an executable\")\n image = _nonempty_string(bundle.get(\"image\"), \"spec_bundle.image\")\n workload_image = request.get(\"workload_image\")\n if workload_image is not None and workload_image != image:\n raise RenderError(\"request.workload_image must equal spec_bundle.image\")\n shape = bundle.get(\"compute_shape\")\n if not isinstance(shape, dict):\n raise RenderError(\"spec_bundle.compute_shape must be an object\")\n gpus = shape.get(\"gpus\")\n nodes = shape.get(\"nodes\")\n if not isinstance(gpus, int) or isinstance(gpus, bool) or gpus \u003c 0:\n raise RenderError(\"spec_bundle.compute_shape.gpus must be a non-negative integer\")\n if nodes != 1:\n raise RenderError(\"single-pod action rendering requires compute_shape.nodes=1\")\n return command, args, image, gpus\n\n\ndef _staged_sources(payload: dict[str, Any]) -> dict[str, str]:\n if payload.get(\"schema_version\") != \"1\":\n raise RenderError(\"staging map schema_version must be '1'\")\n rows = payload.get(\"sources\")\n if not isinstance(rows, list) or not rows:\n raise RenderError(\"staging map sources must be a non-empty array\")\n result: dict[str, str] = {}\n sub_paths: dict[str, str] = {}\n for index, row in enumerate(rows):\n if not isinstance(row, dict):\n raise RenderError(f\"staging map sources[{index}] must be an object\")\n source = _absolute_clean_path(row.get(\"source\"), f\"staging sources[{index}].source\")\n sub_path = _safe_sub_path(row.get(\"sub_path\"), f\"staging sources[{index}].sub_path\")\n if source in result:\n raise RenderError(f\"staging map repeats source: {source}\")\n if sub_path in sub_paths:\n raise RenderError(\n \"staging map assigns one PVC subPath to distinct sources: \"\n f\"{sub_paths[sub_path]} and {source}\"\n )\n result[source] = sub_path\n sub_paths[sub_path] = source\n return result\n\n\ndef _mounts(\n request: dict[str, Any],\n staged: dict[str, str],\n config_mount: tuple[str, str] | None = None,\n) -> tuple[list[dict[str, Any]], list[tuple[pathlib.PurePosixPath, bool]]]:\n rows = request.get(\"mounts\")\n if not isinstance(rows, list) or not rows:\n raise RenderError(\"request.mounts must be a non-empty array\")\n rendered: list[dict[str, Any]] = [\n {\"name\": \"dshm\", \"mountPath\": \"/dev/shm\", \"readOnly\": False} # nosec B108 - k8s emptyDir shm mount, not a temp file\n ]\n used_sources: set[str] = set()\n targets: set[str] = {\"/dev/shm\"} # nosec B108 - k8s emptyDir shm mount, not a temp file\n source_modes: list[tuple[pathlib.PurePosixPath, bool]] = []\n for index, row in enumerate(rows):\n if not isinstance(row, dict):\n raise RenderError(f\"request.mounts[{index}] must be an object\")\n source = _absolute_clean_path(row.get(\"source\"), f\"mounts[{index}].source\")\n target = _absolute_clean_path(row.get(\"target\"), f\"mounts[{index}].target\")\n read_only = row.get(\"read_only\")\n if not isinstance(read_only, bool):\n raise RenderError(f\"mounts[{index}].read_only must be boolean\")\n if source not in staged:\n raise RenderError(f\"mount source has no staged PVC subPath: {source}\")\n if target in targets:\n raise RenderError(f\"request repeats Kubernetes mount target: {target}\")\n targets.add(target)\n used_sources.add(source)\n source_modes.append((pathlib.PurePosixPath(source), read_only))\n rendered.append(\n {\n \"name\": \"workspace\",\n \"mountPath\": target,\n \"subPath\": staged[source],\n \"readOnly\": read_only,\n }\n )\n if config_mount is not None:\n source, target = config_mount\n if source in used_sources:\n raise RenderError(\n \"materialized config must have its own staged file source\"\n )\n if source not in staged:\n raise RenderError(\n f\"materialized config has no staged PVC subPath: {source}\"\n )\n if target in targets:\n raise RenderError(\n f\"materialized config repeats Kubernetes mount target: {target}\"\n )\n used_sources.add(source)\n targets.add(target)\n rendered.append(\n {\n \"name\": \"workspace\",\n \"mountPath\": target,\n \"subPath\": staged[source],\n \"readOnly\": True,\n }\n )\n extras = sorted(set(staged) - used_sources)\n if extras:\n raise RenderError(\"staging map contains undeclared mount sources: \" + \", \".join(extras))\n return rendered, source_modes\n\n\ndef _config_mount(\n request: dict[str, Any],\n staged: dict[str, str],\n config_source: pathlib.Path | None,\n) -> tuple[str | None, tuple[str, str] | None]:\n bundle = request.get(\"spec_bundle\")\n if not isinstance(bundle, dict):\n raise RenderError(\"request.spec_bundle must be an object\")\n mode = bundle.get(\"mode\")\n if mode != \"config\":\n if config_source is not None:\n raise RenderError(\"--config-source is valid only for spec_bundle.mode=config\")\n return None, None\n if config_source is None:\n return None, None\n\n lexical = config_source.expanduser()\n source = _absolute_clean_path(str(lexical), \"config source\")\n path = pathlib.Path(source)\n content, extension, digest = _config_content(bundle)\n if _read_regular_file(path, \"config source\") != content.encode(\"utf-8\"):\n raise RenderError(\n \"config source content does not match spec_bundle.spec; rerun \"\n \"materialize-config and restage the returned file\"\n )\n if source not in staged:\n raise RenderError(f\"materialized config has no staged PVC subPath: {source}\")\n target = f\"/tao-action-config/spec-{digest}.{extension}\"\n return target, (source, target)\n\n\ndef _require_writable_outputs(\n request: dict[str, Any], source_modes: list[tuple[pathlib.PurePosixPath, bool]]\n) -> None:\n outputs = request.get(\"fresh_outputs\", [])\n if not isinstance(outputs, list):\n raise RenderError(\"request.fresh_outputs must be an array\")\n for index, raw in enumerate(outputs):\n output = pathlib.PurePosixPath(\n _absolute_clean_path(raw, f\"fresh_outputs[{index}]\")\n )\n writable = False\n for source, read_only in source_modes:\n try:\n output.relative_to(source)\n except ValueError:\n continue\n if not read_only:\n writable = True\n break\n if not writable:\n raise RenderError(\n f\"fresh output is not covered by a writable declared mount: {output}\"\n )\n\n\ndef _environment(\n request: dict[str, Any], credential_secret: str | None\n) -> list[dict[str, Any]]:\n raw_environment = request.get(\"environment\", {})\n if not isinstance(raw_environment, dict):\n raise RenderError(\"request.environment must be an object\")\n environment: list[dict[str, Any]] = []\n for name in sorted(raw_environment):\n if ENV_NAME_RE.fullmatch(name) is None:\n raise RenderError(f\"invalid environment variable name: {name!r}\")\n value = _string_value(raw_environment[name], f\"environment.{name}\")\n environment.append({\"name\": name, \"value\": value})\n\n raw_forward = request.get(\"forward_env\", [])\n if not isinstance(raw_forward, list):\n raise RenderError(\"request.forward_env must be an array\")\n forward: list[str] = []\n for index, raw in enumerate(raw_forward):\n name = _nonempty_string(raw, f\"forward_env[{index}]\")\n if ENV_NAME_RE.fullmatch(name) is None:\n raise RenderError(f\"invalid forwarded environment variable name: {name!r}\")\n if name in forward:\n raise RenderError(f\"request.forward_env repeats {name}\")\n if name in raw_environment:\n raise RenderError(f\"credential {name} must not be present in request.environment\")\n forward.append(name)\n if forward and credential_secret is None:\n raise RenderError(\n \"request.forward_env is non-empty but no --credential-secret was supplied\"\n )\n if not forward and credential_secret is not None:\n raise RenderError(\n \"--credential-secret must be omitted when request.forward_env is empty\"\n )\n if credential_secret is not None:\n environment.extend(\n {\n \"name\": name,\n \"valueFrom\": {\n \"secretKeyRef\": {\"name\": credential_secret, \"key\": name}\n },\n }\n for name in forward\n )\n return environment\n\n\ndef render_action_job(\n request: dict[str, Any],\n staging_map: dict[str, Any],\n *,\n job_id: str,\n namespace: str,\n pvc_claim: str,\n credential_secret: str | None = None,\n image_pull_secret: str | None = None,\n config_source: pathlib.Path | None = None,\n ttl_seconds: int = 3600,\n shm_size: str = \"16Gi\",\n template_path: pathlib.Path = DEFAULT_TEMPLATE,\n) -> str:\n \"\"\"Validate inputs and return a complete Kubernetes Job manifest.\"\"\"\n if request.get(\"schema_version\") != REQUEST_SCHEMA_VERSION:\n raise RenderError(\n f\"request.schema_version must be {REQUEST_SCHEMA_VERSION!r}\"\n )\n if request.get(\"platform\") != \"kubernetes\":\n raise RenderError(\"request.platform must be 'kubernetes'\")\n job_id = _nonempty_string(job_id, \"job id\")\n job_name = kubernetes_job_name(job_id)\n namespace = _dns_name(namespace, \"namespace\", label_only=True)\n pvc_claim = _dns_name(pvc_claim, \"PVC claim\")\n if credential_secret is not None:\n credential_secret = _dns_name(credential_secret, \"credential secret\")\n if image_pull_secret is not None:\n image_pull_secret = _dns_name(image_pull_secret, \"image pull secret\")\n if (\n not isinstance(ttl_seconds, int)\n or isinstance(ttl_seconds, bool)\n or not 0 \u003c= ttl_seconds \u003c= 604800\n ):\n raise RenderError(\"ttl_seconds must be an integer from 0 through 604800\")\n if QUANTITY_RE.fullmatch(shm_size) is None:\n raise RenderError(\"shm_size must be a positive binary Kubernetes quantity such as 16Gi\")\n\n staged = _staged_sources(staging_map)\n config_path, config_mount = _config_mount(request, staged, config_source)\n command, args, image, gpus = _command_bundle(request, config_path)\n volume_mounts, source_modes = _mounts(request, staged, config_mount)\n _require_writable_outputs(request, source_modes)\n environment = _environment(request, credential_secret)\n image_pull_secrets = (\n [{\"name\": image_pull_secret}] if image_pull_secret is not None else []\n )\n\n values: dict[str, Any] = {\n \"JOB_NAME_JSON\": job_name,\n \"JOB_ID_JSON\": job_id,\n \"NAMESPACE_JSON\": namespace,\n \"TTL_SECONDS_JSON\": ttl_seconds,\n \"IMAGE_PULL_SECRETS_JSON\": image_pull_secrets,\n \"IMAGE_JSON\": image,\n \"COMMAND_JSON\": command,\n \"ARGS_JSON\": args,\n \"NUM_GPUS_JSON\": str(gpus),\n \"ENV_JSON\": environment,\n \"VOLUME_MOUNTS_JSON\": volume_mounts,\n \"SHM_SIZE_JSON\": shm_size,\n \"PVC_CLAIM_JSON\": pvc_claim,\n }\n try:\n rendered = template_path.read_text(encoding=\"utf-8\")\n except OSError as exc:\n raise RenderError(f\"cannot read Kubernetes action template: {exc}\") from exc\n for marker, value in values.items():\n rendered = rendered.replace(f\"@@{marker}@@\", json.dumps(value, separators=(\",\", \":\")))\n remaining = MARKER_RE.findall(rendered)\n if remaining:\n raise RenderError(f\"unresolved template markers: {sorted(set(remaining))}\")\n return rendered\n\n\ndef build_parser() -> argparse.ArgumentParser:\n parser = argparse.ArgumentParser(description=__doc__)\n commands = parser.add_subparsers(dest=\"operation\", required=True)\n name = commands.add_parser(\n \"name\", help=\"print the deterministic Kubernetes name for a job-record id\"\n )\n name.add_argument(\"--job-id\", required=True)\n\n materialize = commands.add_parser(\n \"materialize-config\",\n help=\"write a content-addressed mode=config file for PVC staging\",\n )\n materialize.add_argument(\"--request\", required=True, type=pathlib.Path)\n materialize.add_argument(\"--output-dir\", required=True, type=pathlib.Path)\n\n render = commands.add_parser(\"render\", help=\"render a complete Job manifest\")\n render.add_argument(\"--request\", required=True, type=pathlib.Path)\n render.add_argument(\"--staging-map\", required=True, type=pathlib.Path)\n render.add_argument(\"--job-id\", required=True)\n render.add_argument(\n \"--namespace\", default=os.environ.get(\"TAO_K8S_NAMESPACE\", \"default\")\n )\n render.add_argument(\"--pvc-claim\", required=True)\n render.add_argument(\"--credential-secret\")\n render.add_argument(\"--image-pull-secret\")\n render.add_argument(\"--config-source\", type=pathlib.Path)\n render.add_argument(\"--ttl-seconds\", type=int, default=3600)\n render.add_argument(\"--shm-size\", default=\"16Gi\")\n return parser\n\n\ndef main(argv: list[str] | None = None) -> int:\n args = build_parser().parse_args(argv)\n try:\n if args.operation == \"name\":\n print(kubernetes_job_name(args.job_id))\n return 0\n request = _load_object(args.request, \"action request\")\n if args.operation == \"materialize-config\":\n print(materialize_config(request, args.output_dir))\n return 0\n staging_map = _load_object(args.staging_map, \"staging map\")\n manifest = render_action_job(\n request,\n staging_map,\n job_id=args.job_id,\n namespace=args.namespace,\n pvc_claim=args.pvc_claim,\n credential_secret=args.credential_secret,\n image_pull_secret=args.image_pull_secret,\n config_source=args.config_source,\n ttl_seconds=args.ttl_seconds,\n shm_size=args.shm_size,\n )\n except RenderError as exc:\n print(f\"ERROR: {exc}\", file=sys.stderr)\n return 2\n sys.stdout.write(manifest)\n return 0\n\n\nif __name__ == \"__main__\":\n raise SystemExit(main())\n","content_type":"text/x-python; charset=utf-8","language":"python","size":27916,"content_sha256":"9f90bc888a779b1cdddd3ed1b3cd4d4f69093572ae4439baa92e5f5609718402"},{"filename":"skill-card.md","content":"## Description: \u003cbr>\nKubernetes execution platform that submits TAO container jobs as k8s Jobs with NVIDIA GPU scheduling, supporting single-pod and multi-node distributed training via Indexed Jobs. \u003cbr>\n\nThis skill is ready for commercial/non-commercial use. \u003cbr>\n\n## Owner\nNVIDIA \u003cbr>\n\n### License/Terms of Use: \u003cbr>\nApache-2.0 \u003cbr>\n## Use Case: \u003cbr>\nDevelopers and ML engineers who need to submit and manage NVIDIA TAO container-based training and inference jobs on Kubernetes clusters (EKS, GKE, AKS, on-prem) with NVIDIA GPU scheduling. \u003cbr>\n\n### Deployment Geography for Use: \u003cbr>\nGlobal \u003cbr>\n\n## Requirements / Dependencies: \u003cbr>\n**Requires API Key or External Credential:** [Yes] \u003cbr>\n**Credential Type(s):** [Cloud Credentials] \u003cbr>\n\nDo not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. \u003cbr>\n\n## Known Risks and Mitigations: \u003cbr>\nRisk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. \u003cbr>\nMitigation: Review and scan skill before deployment. \u003cbr>\n\n## Reference(s): \u003cbr>\n- [Action Request Reference](references/action-request.md) \u003cbr>\n- [Local Cluster Setup](references/local-cluster.md) \u003cbr>\n- [NVIDIA GPU Operator Getting Started](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html) \u003cbr>\n- [Kubernetes Indexed Job](https://kubernetes.io/docs/concepts/workloads/controllers/job/#completion-mode) \u003cbr>\n- [TAO Skill Bank Repository](https://github.com/NVIDIA-TAO/tao-skill-bank) \u003cbr>\n\n\n## Skill Output: \u003cbr>\n**Output Type(s):** [Shell commands, Configuration instructions] \u003cbr>\n**Output Format:** [Markdown with inline bash code blocks] \u003cbr>\n**Output Parameters:** [1D] \u003cbr>\n**Other Properties Related to Output:** [None] \u003cbr>\n\n## Evaluation Agents Used: \u003cbr>\n- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`) \u003cbr>\n- Codex (`openai/openai/gpt-5.5`) \u003cbr>\n\n\n\n## Evaluation Tasks: \u003cbr>\n1 evaluation task (1 positive), 3 attempts per task, in k8s-sandbox environment. Dataset digest: sha256:7ec1d30a7bdad9358b2758fb7f4f6bc53cd6b93ea77c8f66ccfae6b41327ba45. \u003cbr>\n\n## Evaluation Metrics Used: \u003cbr>\nReported benchmark dimensions: \u003cbr>\n- Security: Checks for unsafe operations, secret leakage, and unauthorized access. \u003cbr>\n- Correctness: Verifies final-answer correctness against the reference answer. \u003cbr>\n- Discoverability: Checks whether the expected skill was selected, decoys were avoided, and the workflow executed. \u003cbr>\n- Effectiveness: Measures goal completion (50%) and expected workflow behavior adherence (50%). \u003cbr>\n- Efficiency: Measures tool-call productivity (50%) and token efficiency (50%). \u003cbr>\n\nUnderlying evaluation signals used in this run: \u003cbr>\n- `security`: Checks for unsafe operations, secret leakage, and unauthorized access. \u003cbr>\n- `skill_execution`: Whether the expected skill was selected and the workflow executed. \u003cbr>\n- `accuracy`: Final-answer correctness against the reference answer. \u003cbr>\n- `goal_accuracy`: Whether the user's goal was achieved. \u003cbr>\n- `behavior_check`: Whether the expected workflow behavior was followed. \u003cbr>\n- `skill_efficiency`: Tool-call productivity; routing scored under Discoverability. \u003cbr>\n- `token_efficiency`: Actual uncached prompt plus completion token usage. \u003cbr>\n\n\n\n## Evaluation Results: \u003cbr>\n| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |\n|---|---:|---:|\n| Overall | 94.2% | 70.4% |\n| Security | 100.0% → 100.0% (±0.0 points) | 100.0% → 100.0% (±0.0 points) |\n| Correctness | 0.0% → 100.0% (+100.0 points) | 13.3% → 100.0% (+86.7 points) |\n| Discoverability | 100.0% | 0.0% |\n| Effectiveness | 16.7% → 100.0% (+83.3 points) | 20.0% → 53.3% (+33.3 points) |\n| Efficiency | 71.1% | 99.5% → 98.5% (-1.0 points) |\n\n## Skill Version(s): \u003cbr>\n0.1.0 (source: frontmatter) \u003cbr>\n\n## Ethical Considerations: \u003cbr>\nNVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. \u003cbr>\n\n(For Release on NVIDIA Platforms Only) \u003cbr>\nPlease report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail). \u003cbr>\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":4479,"content_sha256":"3fab7e2e0f38b2500f0115e5f50ad79513dac82950d9b22bace22a547bc39572"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"Kubernetes","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"Standalone install?","type":"text","marks":[{"type":"strong"}]},{"text":" If this session was not initialized by the TAO skill bank plugin, run the ","type":"text"},{"text":"tao-setup","type":"text","marks":[{"type":"code_inline"}]},{"text":" skill first (host preflight, credentials, cross-skill discovery).","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Submits TAO container jobs as Kubernetes Jobs. Works on any cluster reachable via kubeconfig (EKS / GKE / AKS / on-prem) or in-cluster service account (when running inside a pod).","type":"text"}]},{"type":"paragraph","content":[{"text":"Single-pod by default; opt into multi-node distributed training via ","type":"text"},{"text":"num_nodes > 1","type":"text","marks":[{"type":"code_inline"}]},{"text":" (uses Indexed Job + headless Service, see ","type":"text"},{"text":"Multi-node training","type":"text","marks":[{"type":"link","attrs":{"href":"#multi-node-training-distributed","title":null}}]},{"text":" below).","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Preflight","type":"text"}]},{"type":"paragraph","content":[{"text":"Three checks: GPU host runtime ready, cluster reachable via ","type":"text"},{"text":"kubectl","type":"text","marks":[{"type":"code_inline"}]},{"text":", GPU Operator/device plugin present.","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# 0. GPU node host runtime.\n# Run this on each self-managed GPU worker node or in the node image build.\n# Set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1 only when using managed GPU nodes whose\n# driver/toolkit lifecycle is owned by the cloud provider or GPU Operator policy.\nif [ \"${TAO_K8S_SKIP_NODE_RUNTIME_CHECK:-0}\" != \"1\" ]; then\n TAO_SKILL_BANK_ROOT=\"${TAO_SKILL_BANK_ROOT:-$PWD}\"\n SETUP_SCRIPT=\"${TAO_SKILL_BANK_ROOT}/skills/platform/tao-setup-nvidia-gpu-host/scripts/setup-nvidia-gpu-host.sh\"\n\n bash \"$SETUP_SCRIPT\" --backend kubernetes --check-only || {\n echo \"MISSING: TAO Kubernetes GPU node runtime is not ready.\"\n echo \"For self-managed GPU nodes, run after user approval:\"\n echo \" bash \\\"$SETUP_SCRIPT\\\" --backend kubernetes --install --yes\"\n echo \"For managed clusters, verify the node image/GPU Operator policy installs driver 580 and toolkit 1.19.0, then set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1.\"\n exit 1\n }\nfi\n\n# 1. Cluster reachable (kubeconfig OR in-cluster service account)\ncommand -v kubectl >/dev/null 2>&1 || {\n echo \"MISSING: kubectl not found on PATH. Install kubectl to submit Jobs.\"\n exit 1\n}\nkubectl cluster-info >/dev/null 2>&1 || {\n echo \"MISSING: no reachable cluster (kubeconfig at ~/.kube/config, \\$KUBECONFIG, or in-pod service account).\"\n echo \"Configure kubectl for your cluster, or set \\$KUBECONFIG:\"\n echo \" EKS: aws eks update-kubeconfig --name \u003ccluster> --region \u003cregion>\"\n echo \" GKE: gcloud container clusters get-credentials \u003ccluster> --region \u003cregion>\"\n echo \" AKS: az aks get-credentials --resource-group \u003crg> --name \u003ccluster>\"\n echo \" local: minikube start (see 'Local cluster' below)\"\n exit 1\n}\n\n# 2. NVIDIA GPU Operator present (soft check — warn, don't fail)\ngpu=$(kubectl get nodes -o jsonpath='{range .items[*]}{.status.allocatable.nvidia\\.com/gpu}{\"\\n\"}{end}' 2>/dev/null | grep -v '^

Kubernetes Standalone install? If this session was not initialized by the TAO skill bank plugin, run the skill first (host preflight, credentials, cross-skill discovery). Submits TAO container jobs as Kubernetes Jobs. Works on any cluster reachable via kubeconfig (EKS / GKE / AKS / on-prem) or in-cluster service account (when running inside a pod). Single-pod by default; opt into multi-node distributed training via (uses Indexed Job + headless Service, see Multi-node training below). Preflight Three checks: GPU host runtime ready, cluster reachable via , GPU Operator/device plugin present. Th…

| head -1)\nif [ -z \"$gpu\" ] || [ \"$gpu\" = \"0\" ]; then\n echo \"WARN: no nvidia.com/gpu allocatable on this cluster.\"\n echo \"Install the NVIDIA GPU Operator before submitting GPU jobs:\"\n echo \" https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html\"\nfi","type":"text"}]},{"type":"paragraph","content":[{"text":"The GPU node runtime check is mandatory for self-managed nodes. For managed clusters where the client is not running on a GPU worker, verify the provider node image or GPU Operator policy and set ","type":"text"},{"text":"TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1","type":"text","marks":[{"type":"code_inline"}]},{"text":" instead of running the installer on the client. The GPU-capacity warning here is a soft check; the ","type":"text"},{"text":"submit","type":"text","marks":[{"type":"code_inline"}]},{"text":" verb re-checks allocatable ","type":"text"},{"text":"nvidia.com/gpu","type":"text","marks":[{"type":"code_inline"}]},{"text":" and hard-fails before applying the manifest (there is no gang scheduling, so a too-big Job would sit ","type":"text"},{"text":"Pending","type":"text","marks":[{"type":"code_inline"}]},{"text":" forever).","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Credentials & configuration","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Kubeconfig","type":"text","marks":[{"type":"strong"}]},{"text":" (one of):","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"~/.kube/config","type":"text","marks":[{"type":"code_inline"}]},{"text":" — default discovery path","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"$KUBECONFIG","type":"text","marks":[{"type":"code_inline"}]},{"text":" — alternate path","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"In-cluster service account — used when running inside a pod (no kubeconfig needed)","type":"text"}]}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"TAO_K8S_NAMESPACE","type":"text","marks":[{"type":"strong"}]},{"text":" (optional): default namespace for Job submission. Defaults to ","type":"text"},{"text":"default","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"TAO_K8S_CONTEXT","type":"text","marks":[{"type":"strong"}]},{"text":" (optional): kubeconfig context name to switch clusters.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NGC_KEY","type":"text","marks":[{"type":"strong"}]},{"text":" (optional): for nvcr.io image pulls. If you've pre-created an image-pull secret in the target namespace, reference its name in the rendered manifest's ","type":"text"},{"text":"imagePullSecrets","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / S3_BUCKET_NAME / S3_ENDPOINT_URL","type":"text","marks":[{"type":"strong"}]},{"text":" (optional): for S3 dataset I/O (storage tier C), injected into the pod via the per-job Secret (","type":"text"},{"text":"envFrom.secretRef","type":"text","marks":[{"type":"code_inline"}]},{"text":"), never inline. Legacy ","type":"text"},{"text":"ACCESS_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"SECRET_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" are mapped by ","type":"text"},{"text":"tao-data-io","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Do not ask for Brev or SLURM credentials for Kubernetes runs. Ask for S3 credentials only when the selected workflow uses ","type":"text"},{"text":"s3://","type":"text","marks":[{"type":"code_inline"}]},{"text":" inputs or outputs, and ask for model-specific credentials such as ","type":"text"},{"text":"HF_TOKEN","type":"text","marks":[{"type":"code_inline"}]},{"text":" only when the selected model requires them. Before launch, verify the selected namespace can create Jobs, dataset/result paths are visible from the pod, and PVC/mounted filesystem paths are proven to be mounted into the job container; an agent-host local path is not sufficient proof.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Execution — the four verbs","type":"text"}]},{"type":"paragraph","content":[{"text":"tao-run-on-kubernetes","type":"text","marks":[{"type":"code_inline"}]},{"text":" is a platform ","type":"text"},{"text":"consumer","type":"text","marks":[{"type":"strong"}]},{"text":": it runs a spec-bundle via ","type":"text"},{"text":"kubectl","type":"text","marks":[{"type":"code_inline"}]},{"text":", mutating only the job-record. No nvidia-tao-sdk, no ","type":"text"},{"text":"tao_sdk","type":"text","marks":[{"type":"code_inline"}]},{"text":" import — jobs are submitted with plain ","type":"text"},{"text":"kubectl apply","type":"text","marks":[{"type":"code_inline"}]},{"text":". ","type":"text"},{"text":"$BANK","type":"text","marks":[{"type":"code_inline"}]},{"text":" = ","type":"text"},{"text":"${TAO_SKILL_BANK_PATH}","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"submit","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"GPU-capacity gate — hard-fail first","type":"text","marks":[{"type":"strong"}]},{"text":" (no gang scheduling → a too-big Job sits ","type":"text"},{"text":"Pending","type":"text","marks":[{"type":"code_inline"}]},{"text":" forever):","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"ALLOC=$(kubectl get nodes -o jsonpath='{range .items[*]}{.status.allocatable.nvidia\\.com/gpu}{\"\\n\"}{end}' | awk '{s+=$1} END{print s+0}')\n[ \"${ALLOC:-0}\" -ge \"$NUM_GPUS\" ] || { echo \"insufficient GPUs: need $NUM_GPUS, allocatable $ALLOC\"; exit 1; }","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Storage tier","type":"text","marks":[{"type":"strong"}]},{"text":" (via ","type":"text"},{"text":"tao-data-io","type":"text","marks":[{"type":"code_inline"}]},{"text":"): ","type":"text"},{"text":"A","type":"text","marks":[{"type":"strong"}]},{"text":" = mount a bound PVC/NFS holding the data (author the mount paths, no fetch — the air-gap answer, and what the packaged template does); ","type":"text"},{"text":"C","type":"text","marks":[{"type":"strong"}]},{"text":" = ephemeral: an initContainer fetches from S3 into a shared ","type":"text"},{"text":"emptyDir","type":"text","marks":[{"type":"code_inline"}]},{"text":" and a final step uploads results to S3 before TTL.","type":"text"}]},{"type":"paragraph","content":[{"text":"Tier C holds the GPU while it downloads.","type":"text","marks":[{"type":"strong"}]},{"text":" A pod reserves ","type":"text"},{"text":"nvidia.com/gpu","type":"text","marks":[{"type":"code_inline"}]},{"text":" for its whole lifetime, initContainers included, so a large tier-C fetch — or a first-time multi-GB image pull — is billed and reaper-eligible idle GPU time, exactly like pulling inside a SLURM allocation. Prefer tier A when the data is already on a PVC; choose tier C knowingly, for small inputs.","type":"text"}]},{"type":"paragraph","content":[{"text":"A producer action request may declare several mounts, including duplicate- source aliases for logical and embedded absolute paths. Read ","type":"text"},{"text":"references/action-request.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" and use its staging map plus packaged renderer. For ","type":"text"},{"text":"mode=config","type":"text","marks":[{"type":"code_inline"}]},{"text":", materialize the producer's nested spec first, stage that exact generated file, and pass it back to the renderer so the pod receives a verified read-only config mount. The legacy single-root template cannot represent that contract.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Open the record — mints the id, binds ","type":"text","marks":[{"type":"strong"}]},{"text":"results_dir","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":", before launch:","type":"text","marks":[{"type":"strong"}]}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"JOB_ID=$(\"$BANK/scripts/tao_job_record.py\" open --platform kubernetes --image \"$IMAGE\" \\\n --network-arch \"$ARCH\" --action \"$ACTION\" --storage-tier \"$TIER\" --results-dir \"$RESULTS_DIR\")","type":"text"}]},{"type":"paragraph","content":[{"text":"results_dir","type":"text","marks":[{"type":"code_inline"}]},{"text":" must be a ","type":"text"},{"text":"mounted (surviving) volume path or an S3 prefix","type":"text","marks":[{"type":"strong"}]},{"text":" — ","type":"text"},{"text":"ttlSecondsAfterFinished","type":"text","marks":[{"type":"code_inline"}]},{"text":" deletes the Job and its logs after it ends, so nothing is recoverable from the Job object later.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Render, gate, apply, and record RUNNING.","type":"text","marks":[{"type":"strong"}]},{"text":" For a producer action request, follow ","type":"text"},{"text":"references/action-request.md","type":"text","marks":[{"type":"code_inline"}]},{"text":"; it owns backend-name normalization, conditional Secret references, native argv rendering, server dry-run, and binding the applied object name to the job-record. For a simple one-root spec-bundle, render ","type":"text"},{"text":"templates/k8s/single-pod-job.yaml.tmpl","type":"text","marks":[{"type":"code_inline"}]},{"text":", run ","type":"text"},{"text":"redact_secrets.py lint","type":"text","marks":[{"type":"code_inline"}]},{"text":" plus ","type":"text"},{"text":"kubectl apply --dry-run=server","type":"text","marks":[{"type":"code_inline"}]},{"text":", apply it, and mark the record with ","type":"text"},{"text":"backend-ref=\u003cnamespace>/\u003cactual-object-name>","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"A submit that skipped the gate or the open has no id — so it cannot launch.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"status","type":"text"}]},{"type":"paragraph","content":[{"text":"Keep ","type":"text"},{"text":"K8S_JOB_NAME","type":"text","marks":[{"type":"code_inline"}]},{"text":" from submit. On reattach, read the job-record's ","type":"text"},{"text":"backend_ref=\u003cnamespace>/\u003cname>","type":"text","marks":[{"type":"code_inline"}]},{"text":" and recover both values from that field; do not assume the Kubernetes name equals the record id.","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"kubectl get job \"$K8S_JOB_NAME\" -n \"$NAMESPACE\" \\\n -o jsonpath='{.status.conditions[0].type} {.status.active} {.status.succeeded} {.status.failed}'","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"kubectl signal","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"vocab","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"no pods scheduled","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"PENDING","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"kubectl get pods -n \"$NAMESPACE\" -l job-name=\"$K8S_JOB_NAME\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" → ","type":"text"},{"text":"ImagePullBackOff","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"Insufficient nvidia.com/gpu","type":"text","marks":[{"type":"code_inline"}]},{"text":" in ","type":"text"},{"text":"message","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"active","type":"text","marks":[{"type":"code_inline"}]},{"text":" ≥ 1","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"RUNNING","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"condition ","type":"text"},{"text":"Complete","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"COMPLETE","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"condition ","type":"text"},{"text":"Failed","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"ERROR","type":"text","marks":[{"type":"code_inline"}]},{"text":" (classify from the pod's terminated reason — ","type":"text"},{"text":"OOMKilled","type":"text","marks":[{"type":"code_inline"}]},{"text":" → ","type":"text"},{"text":"ERR_INFRA","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Job/pod not found","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"UNKNOWN","type":"text","marks":[{"type":"code_inline"}]},{"text":" (may be TTL-deleted — the job-record is the source of truth)","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"logs","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"kubectl logs -n \"$NAMESPACE\" -l \"job-name=$K8S_JOB_NAME\" --tail \"${N:-200}\"","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"cancel","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"kubectl delete job \"$K8S_JOB_NAME\" -n \"$NAMESPACE\" --cascade=foreground\nif [ -n \"${CRED_SECRET:-}\" ]; then\n kubectl delete secret \"$CRED_SECRET\" -n \"$NAMESPACE\" --ignore-not-found\nfi\n\"$BANK/scripts/tao_job_record.py\" mark \"$JOB_ID\" --state CANCELED --source agent","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Multi-node (nodes > 1)","type":"text"}]},{"type":"paragraph","content":[{"text":"Same four verbs, plus:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Version gate:","type":"text","marks":[{"type":"strong"}]},{"text":" require k8s ≥ 1.28 (","type":"text"},{"text":"kubectl version -o json","type":"text","marks":[{"type":"code_inline"}]},{"text":") — the pod hostname ","type":"text"},{"text":"\u003cjob>-\u003cindex>","type":"text","marks":[{"type":"code_inline"}]},{"text":" (PodIndexLabel) that ","type":"text"},{"text":"MASTER_ADDR=\u003cjob>-0.\u003csvc>","type":"text","marks":[{"type":"code_inline"}]},{"text":" resolves to needs it; on older clusters rank-0 hangs at rendezvous.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Capacity gate ×nodes:","type":"text","marks":[{"type":"strong"}]},{"text":" hard-fail unless allocatable GPUs ≥ ","type":"text"},{"text":"gpus_per_node × nodes","type":"text","marks":[{"type":"code_inline"}]},{"text":" (no gang scheduling → a partial start leaves rank-0 waiting forever).","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Render ","type":"text","marks":[{"type":"strong"}]},{"text":"templates/k8s/indexed-job.yaml.tmpl","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" — the headless Service + Indexed Job + rendezvous env (","type":"text"},{"text":"WORLD_SIZE","type":"text","marks":[{"type":"code_inline"}]},{"text":" = node count, ","type":"text"},{"text":"NODE_RANK","type":"text","marks":[{"type":"code_inline"}]},{"text":" from ","type":"text"},{"text":"JOB_COMPLETION_INDEX","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"MASTER_ADDR=\u003cjob>-0.\u003csvc>","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"/dev/shm","type":"text","marks":[{"type":"code_inline"}]},{"text":" 16Gi so NCCL doesn't silently hang). ","type":"text"},{"text":"kubectl apply -f","type":"text","marks":[{"type":"code_inline"}]},{"text":" creates the Service and Job together; ","type":"text"},{"text":"cancel","type":"text","marks":[{"type":"code_inline"}]},{"text":" deletes the Job (Foreground) and the Service.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NCCL probe first","type":"text","marks":[{"type":"strong"}]},{"text":" (as SLURM) — a 2-node all-reduce with a timeout; on hang, set the cluster NCCL env and re-probe; cache per cluster.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Local cluster (development, CI, and evals)","type":"text"}]},{"type":"paragraph","content":[{"text":"A throwaway minikube/kind cluster exercises admission, the four verbs, job-record wiring, and log plumbing without cluster quota — and is what an agent-driven eval should provision for itself. ","type":"text"},{"text":"kubectl","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"minikube","type":"text","marks":[{"type":"code_inline"}]},{"text":" are single static binaries needing no root, so a non-root CI container can install them itself.","type":"text"}]},{"type":"paragraph","content":[{"text":"Two prerequisites keep a rendered Job ","type":"text"},{"text":"Pending","type":"text","marks":[{"type":"code_inline"}]},{"text":", and the first masks the second: the PVC the template mounts must exist (","type":"text"},{"text":"persistentvolumeclaim \"\u003cname>\" not found","type":"text","marks":[{"type":"code_inline"}]},{"text":" fires before any GPU complaint), then a Job requesting ","type":"text"},{"text":"nvidia.com/gpu","type":"text","marks":[{"type":"code_inline"}]},{"text":" on a GPU-less cluster reports ","type":"text"},{"text":"Insufficient nvidia.com/gpu","type":"text","marks":[{"type":"code_inline"}]},{"text":" and waits forever. Render ","type":"text"},{"text":"NUM_GPUS=0","type":"text","marks":[{"type":"code_inline"}]},{"text":" for a lifecycle-only run and say GPU scheduling was not verified; on a Linux GPU host, ","type":"text"},{"text":"minikube start --driver=docker --gpus all","type":"text","marks":[{"type":"code_inline"}]},{"text":" passes real GPUs through, so one GPU box suffices for a GPU-real smoke.","type":"text"}]},{"type":"paragraph","content":[{"text":"Install commands, driver choice, the container/host-networking caveat, and the fake-device-plugin middle option: ","type":"text"},{"text":"references/local-cluster.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Container shell","type":"text"}]},{"type":"paragraph","content":[{"text":"The simple single-pod template invokes its command via ","type":"text"},{"text":"/bin/sh -c","type":"text","marks":[{"type":"code_inline"}]},{"text":" (POSIX sh, present in busybox/distroless as well as TAO images). For producer action requests, an args-mode command and its arguments are native container argv; a producer that needs a shell declares the shell and its script explicitly. A simple config-mode command also becomes native argv after ","type":"text"},{"text":"{config_path}","type":"text","marks":[{"type":"code_inline"}]},{"text":" is substituted. A producer-owned config command that is itself a multi-line shell script is preserved verbatim under ","type":"text"},{"text":"/bin/sh -c","type":"text","marks":[{"type":"code_inline"}]},{"text":"; the renderer never constructs shell text from config values.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"GPU Operator dependency","type":"text"}]},{"type":"paragraph","content":[{"text":"The ","type":"text"},{"text":"submit","type":"text","marks":[{"type":"code_inline"}]},{"text":" verb refuses to launch GPU jobs on a cluster with no ","type":"text"},{"text":"nvidia.com/gpu","type":"text","marks":[{"type":"code_inline"}]},{"text":" allocatable. For self-managed clusters, first run the ","type":"text"},{"text":"tao-setup-nvidia-gpu-host","type":"text","marks":[{"type":"code_inline"}]},{"text":" install action on every GPU worker node or bake the same package set into the node image:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"bash skills/platform/tao-setup-nvidia-gpu-host/scripts/setup-nvidia-gpu-host.sh --backend kubernetes --install --yes","type":"text"}]},{"type":"paragraph","content":[{"text":"Then install the NVIDIA GPU Operator or device plugin:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"helm repo add nvidia https://helm.ngc.nvidia.com/nvidia\nhelm repo update\nhelm install --wait gpu-operator -n gpu-operator --create-namespace nvidia/gpu-operator","type":"text"}]},{"type":"paragraph","content":[{"text":"Full guide: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Multi-node training (distributed)","type":"text"}]},{"type":"paragraph","content":[{"text":"Set ","type":"text"},{"text":"num_nodes > 1","type":"text","marks":[{"type":"code_inline"}]},{"text":" (see the ","type":"text"},{"text":"Multi-node (nodes > 1)","type":"text","marks":[{"type":"link","attrs":{"href":"#multi-node-nodes--1","title":null}}]},{"text":" verb steps above) to run distributed training across N pods. Rendering ","type":"text"},{"text":"templates/k8s/indexed-job.yaml.tmpl","type":"text","marks":[{"type":"code_inline"}]},{"text":" provisions:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A ","type":"text"},{"text":"headless Service","type":"text","marks":[{"type":"strong"}]},{"text":" named after the Job (selector: ","type":"text"},{"text":"job-name=\u003cjob-name>","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"clusterIP: None","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"publishNotReadyAddresses: true","type":"text","marks":[{"type":"code_inline"}]},{"text":" so pods can rendezvous before they're all Ready).","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"An ","type":"text"},{"text":"Indexed Job","type":"text","marks":[{"type":"strong"}]},{"text":" with ","type":"text"},{"text":"parallelism = completions = num_nodes","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"completionMode: Indexed","type":"text","marks":[{"type":"code_inline"}]},{"text":". Each pod gets ","type":"text"},{"text":"JOB_COMPLETION_INDEX","type":"text","marks":[{"type":"code_inline"}]},{"text":" injected by k8s automatically (= the node rank).","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A ","type":"text"},{"text":"command wrapper","type":"text","marks":[{"type":"strong"}]},{"text":" that exports the rendezvous env vars before invoking the user command. Two naming conventions are exported simultaneously:","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Env var","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Value","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Read by","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"WORLD_SIZE","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"num_nodes","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"TAO PyTorch container's ","type":"text"},{"text":"nvidia_tao_pytorch/core/entrypoint.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" (uses this to mean ","type":"text"},{"text":"node count","type":"text","marks":[{"type":"em"}]},{"text":", even though PyTorch's own convention is ","type":"text"},{"text":"total processes","type":"text","marks":[{"type":"em"}]},{"text":")","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"NUM_GPU_PER_NODE","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"gpu_count","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"TAO PyTorch container's entrypoint","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"NNODES","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"num_nodes","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"torchrun","type":"text","marks":[{"type":"code_inline"}]},{"text":" and PyTorch-standard rendezvous","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"NPROC_PER_NODE","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"gpu_count","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"torchrun","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"NODE_RANK","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"$JOB_COMPLETION_INDEX","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"both","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"MASTER_ADDR","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"\u003cjob-name>-0.\u003cjob-name>","type":"text","marks":[{"type":"code_inline"}]},{"text":" (pod-0's DNS)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"both","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"MASTER_PORT","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"29500","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"both (TAO's default)","type":"text"}]}]}]}]},{"type":"paragraph","content":[{"text":"Both naming conventions are set so TAO entrypoints (","type":"text"},{"text":"dino train","type":"text","marks":[{"type":"code_inline"}]},{"text":", etc.) and raw ","type":"text"},{"text":"torchrun","type":"text","marks":[{"type":"code_inline"}]},{"text":" commands work without modification.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"For a TAO entrypoint, the container reads ","type":"text"},{"text":"spec.train.num_nodes","type":"text","marks":[{"type":"code_inline"}]},{"text":" and the wired env vars — e.g. ","type":"text"},{"text":"dino train -e /tmp/spec.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":" with ","type":"text"},{"text":"gpu_count=8","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"num_nodes=4","type":"text","marks":[{"type":"code_inline"}]},{"text":" (4 × 8 = 32 GPUs total).","type":"text"}]},{"type":"paragraph","content":[{"text":"For raw ","type":"text"},{"text":"torchrun","type":"text","marks":[{"type":"code_inline"}]},{"text":"-based commands (non-TAO containers), the wrapper invokes:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"torchrun --nnodes=$NNODES --nproc-per-node=$NPROC_PER_NODE --node-rank=$NODE_RANK \\\n --master-addr=$MASTER_ADDR --master-port=$MASTER_PORT train.py","type":"text"}]},{"type":"paragraph","content":[{"text":"The capacity check sums across nodes: ","type":"text"},{"text":"gpu_count × num_nodes","type":"text","marks":[{"type":"code_inline"}]},{"text":" ≤ cluster's allocatable ","type":"text"},{"text":"nvidia.com/gpu","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Cluster requirements for multi-node","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"k8s 1.28+","type":"text","marks":[{"type":"strong"}]},{"text":" is required for stable pod hostnames in Indexed Jobs (the ","type":"text"},{"text":"PodIndexLabel","type":"text","marks":[{"type":"code_inline"}]},{"text":" feature). On older clusters the ","type":"text"},{"text":"MASTER_ADDR=\u003cjob>-0.\u003csvc>","type":"text","marks":[{"type":"code_inline"}]},{"text":" DNS lookup fails. Verify with ","type":"text"},{"text":"kubectl version","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Pod-to-pod networking","type":"text","marks":[{"type":"strong"}]},{"text":" must be open on port 29500 (PyTorch default; configurable via ","type":"text"},{"text":"MASTER_PORT","type":"text","marks":[{"type":"code_inline"}]},{"text":" env var). Most CNIs (Calico, Cilium, AWS VPC CNI) allow this by default; restrictive NetworkPolicies must be relaxed.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NCCL","type":"text","marks":[{"type":"strong"}]},{"text":" in the container talks GPU-to-GPU; if the cluster has multi-NIC nodes or RDMA, set ","type":"text"},{"text":"NCCL_SOCKET_IFNAME","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"NCCL_IB_HCA","type":"text","marks":[{"type":"code_inline"}]},{"text":" in the container ","type":"text"},{"text":"env","type":"text","marks":[{"type":"code_inline"}]},{"text":" of the rendered manifest.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Reference reading","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Kubernetes Indexed Job: ","type":"text"},{"text":"https://kubernetes.io/docs/concepts/workloads/controllers/job/#completion-mode","type":"text","marks":[{"type":"link","attrs":{"href":"https://kubernetes.io/docs/concepts/workloads/controllers/job/#completion-mode","title":null}}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Indexed Job for batch ML: ","type":"text"},{"text":"https://kubernetes.io/blog/2022/06/01/indexed-jobs-mpi/","type":"text","marks":[{"type":"link","attrs":{"href":"https://kubernetes.io/blog/2022/06/01/indexed-jobs-mpi/","title":null}}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"PyTorch distributed (env-var rendezvous): ","type":"text"},{"text":"https://pytorch.org/docs/stable/elastic/run.html","type":"text","marks":[{"type":"link","attrs":{"href":"https://pytorch.org/docs/stable/elastic/run.html","title":null}}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NCCL networking tuning (NCCL_SOCKET_IFNAME, NCCL_IB_HCA): ","type":"text"},{"text":"https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html","title":null}}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Kubernetes operator alternatives","type":"text"}]},{"type":"paragraph","content":[{"text":"For more sophisticated topologies (gang scheduling, PyTorch elastic / fault-tolerant training, MPI / Horovod, RDMA setup), reach for an operator instead of plain Indexed Job:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"MPI Operator","type":"text","marks":[{"type":"strong"}]},{"text":" — ","type":"text"},{"text":"https://github.com/kubeflow/mpi-operator","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/kubeflow/mpi-operator","title":null}}]},{"text":" — for MPI / Horovod workloads.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Kubeflow Training Operator","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"PyTorchJob","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"TFJob","type":"text","marks":[{"type":"code_inline"}]},{"text":") — ","type":"text"},{"text":"https://www.kubeflow.org/docs/components/training/","type":"text","marks":[{"type":"link","attrs":{"href":"https://www.kubeflow.org/docs/components/training/","title":null}}]},{"text":" — for elastic PyTorch training with built-in restart logic.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Volcano","type":"text","marks":[{"type":"strong"}]},{"text":" — ","type":"text"},{"text":"https://volcano.sh/","type":"text","marks":[{"type":"link","attrs":{"href":"https://volcano.sh/","title":null}}]},{"text":" — gang scheduling, queues, fair-share. Useful in shared multi-tenant clusters.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Kueue","type":"text","marks":[{"type":"strong"}]},{"text":" — ","type":"text"},{"text":"https://kueue.sigs.k8s.io/","type":"text","marks":[{"type":"link","attrs":{"href":"https://kueue.sigs.k8s.io/","title":null}}]},{"text":" — quota / queue layer on top of any of the above.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"This skill's Indexed Job path is intentionally simple and dependency-free; if you need elastic restart or gang scheduling, layer one of these on top and submit jobs through the operator's CRD instead.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Common error patterns","type":"text"}]},{"type":"paragraph","content":[{"text":"No nvidia.com/gpu resources allocatable on the cluster","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" — the GPU Operator (or NVIDIA Device Plugin) isn't installed. Install per the link above; verify with ","type":"text"},{"text":"kubectl get nodes -o jsonpath='{.items[*].status.allocatable}'","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"ImagePullBackOff","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" / ","type":"text","marks":[{"type":"strong"}]},{"text":"ErrImagePull","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" — the cluster can't pull the image. For nvcr.io: pre-create an image-pull secret in the namespace and reference it as the pod's ","type":"text"},{"text":"imagePullSecrets","type":"text","marks":[{"type":"code_inline"}]},{"text":" in the rendered manifest: Feed the key over stdin — ","type":"text"},{"text":"--docker-password=$NGC_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" would put the secret in argv, where it is visible in the host's process table and shell history:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"set -a; source /path/to/.env; set +a # omit if already exported\nkubectl create secret generic ngc-pull-secret -n tao-jobs \\\n --type=kubernetes.io/dockerconfigjson \\\n --from-file=.dockerconfigjson=/dev/stdin \u003c\u003cEOF\n{\"auths\": {\"nvcr.io\": {\"username\": \"\\$oauthtoken\", \"password\": \"${NGC_KEY}\"}}}\nEOF\n# Verify without reading the secret back:\nkubectl get secret ngc-pull-secret -n tao-jobs >/dev/null && echo SECRET_OK","type":"text"}]},{"type":"paragraph","content":[{"text":"Pod stays ","type":"text","marks":[{"type":"strong"}]},{"text":"Pending","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" forever","type":"text","marks":[{"type":"strong"}]},{"text":" — ","type":"text"},{"text":"kubectl describe pod -l job-name=$JOB_ID","type":"text","marks":[{"type":"code_inline"}]},{"text":" shows the scheduling reason in the ","type":"text"},{"text":"Events","type":"text","marks":[{"type":"code_inline"}]},{"text":". Common causes: insufficient GPU capacity (","type":"text"},{"text":"Insufficient nvidia.com/gpu","type":"text","marks":[{"type":"code_inline"}]},{"text":"), no node matches the pod's ","type":"text"},{"text":"nodeSelector","type":"text","marks":[{"type":"code_inline"}]},{"text":", missing image-pull secret, or PVC mount failure.","type":"text"}]},{"type":"paragraph","content":[{"text":"OOMKilled","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" (exit 137)","type":"text","marks":[{"type":"strong"}]},{"text":" — container exceeded memory. Reduce batch size, lower max_length, or add a memory request/limit and target a larger node.","type":"text"}]},{"type":"paragraph","content":[{"text":"CredentialError: Could not authenticate to a Kubernetes cluster","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" — neither kubeconfig nor in-cluster auth worked. Run ","type":"text"},{"text":"kubectl get nodes","type":"text","marks":[{"type":"code_inline"}]},{"text":" to verify your config, or set ","type":"text"},{"text":"$KUBECONFIG","type":"text","marks":[{"type":"code_inline"}]},{"text":" to the right path.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"What this skill does NOT support (yet)","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Elastic / fault-tolerant training.","type":"text","marks":[{"type":"strong"}]},{"text":" Indexed Job has ","type":"text"},{"text":"backoff_limit=0","type":"text","marks":[{"type":"code_inline"}]},{"text":" — failures fail the whole training run. For elastic restart (e.g., resume from checkpoint after a node death), use Kubeflow's ","type":"text"},{"text":"PyTorchJob","type":"text","marks":[{"type":"code_inline"}]},{"text":" operator instead.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Gang scheduling.","type":"text","marks":[{"type":"strong"}]},{"text":" Indexed Job pods are scheduled independently — no all-or-nothing. Multi-node training will ","type":"text"},{"text":"partially","type":"text","marks":[{"type":"em"}]},{"text":" start if only some pods can be scheduled (rank-0 will hang waiting for peers). For all-or-nothing scheduling on shared clusters, use Volcano or Kueue.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"MPI / Horovod.","type":"text","marks":[{"type":"strong"}]},{"text":" Use the MPI Operator. The Indexed Job path here is PyTorch-distributed-shaped (env-var rendezvous on ","type":"text"},{"text":"MASTER_ADDR:MASTER_PORT","type":"text","marks":[{"type":"code_inline"}]},{"text":").","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Auto-creating image-pull secrets from ","type":"text","marks":[{"type":"strong"}]},{"text":"$NGC_KEY","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":".","type":"text","marks":[{"type":"strong"}]},{"text":" You pre-create the secret in the target namespace and pass the name. K8s namespace conventions vary widely, so we keep secret creation explicit.","type":"text"}]}]}]}]},"metadata":{"date":"2026-11-02","name":"tao-run-on-kubernetes","tags":["kubernetes","k8s","gpu","compute","container"],"author":"@skillopedia","source":{"stars":3825,"repo_name":"skills","origin_url":"https://github.com/NVIDIA/skills/blob/HEAD/skills/tao-run-on-kubernetes/SKILL.md","repo_owner":"NVIDIA","body_sha256":"5aea2bbd3d3eed55fbdf968c32e1a5463a44cad2ff20791ffdab00705ff37a46","cluster_key":"2713a19522558be7b92cc2f603c12d427095ab22c21b56fd0aeeb4c093cd3099","clean_bundle":{"format":"clean-skill-bundle-v1","source":"NVIDIA/skills/skills/tao-run-on-kubernetes/SKILL.md","text_bytes":55633,"source_commit":"25340619336e89bcc40365c28bfe9e790430468e","skills_sh_rank":9649,"attachment_count":9,"text_attachments":9,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"skills/tao-run-on-kubernetes/SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"v1","category":"devops-infrastructure","category_label":"DevOps"},"exact_dupes_collapsed_into_this":0},"license":"Apache-2.0","version":"v1","category":"devops-infrastructure","metadata":{"author":"NVIDIA Corporation","version":"0.1.0"},"import_tag":"skills-sh-top20k-2026-11","description":"Kubernetes execution platform — submits TAO container jobs as k8s Jobs with NVIDIA GPU scheduling; single-pod for one node, Indexed Jobs for multi-node distributed training. Use when running on EKS / GKE / AKS / on-prem clusters with the NVIDIA GPU Operator installed, or when integrating TAO into an existing k8s-native ML platform.","allowed-tools":"Read Bash","compatibility":"Requires GPU worker nodes with NVIDIA driver branch 580, CUDA Toolkit 13.0, and NVIDIA Container Toolkit 1.19.0; a `kubectl` client authenticated to the cluster; and the NVIDIA GPU Operator or device plugin. No nvidia-tao-sdk required — jobs are submitted with plain `kubectl`."}},"renderedAt":1793666686464}

Kubernetes Standalone install? If this session was not initialized by the TAO skill bank plugin, run the skill first (host preflight, credentials, cross-skill discovery). Submits TAO container jobs as Kubernetes Jobs. Works on any cluster reachable via kubeconfig (EKS / GKE / AKS / on-prem) or in-cluster service account (when running inside a pod). Single-pod by default; opt into multi-node distributed training via (uses Indexed Job + headless Service, see Multi-node training below). Preflight Three checks: GPU host runtime ready, cluster reachable via , GPU Operator/device plugin present. Th…