physical-ai-infrastructure-setup-and-resilient-scaling

Physical AI Infrastructure Setup And Resilient Scaling Canonical skill for the Physical AI infrastructure stack. Use it to compose cluster, inference, OSMO, and workload stages into a reproducible Physical AI SDG environment, then keep the environment observable and recoverable. Operating Rules - Read only the component references needed for the selected target. Do not load every component by default. - Keep the repo as the durable artifact. Fix checked-in config or scripts, then rerun. Do not recover a failed install with untracked one-off changes. - Run mutating cluster, OSMO, Helm, Terrafo…

] [--resume]\nosmo dataset delete \u003cname:tag> [--force]\nosmo dataset info \u003cname> [--all] [--count N] [--order asc|desc]\nosmo dataset list [--bucket \u003cb>] [--name \u003cn>] [--count N]\nosmo dataset inspect \u003cname:tag> [--format-type text|tree|json] [--regex ...] [--count N]\n```\n\n### Advanced\n\n```bash\nosmo dataset update \u003cname:tag> --add \u003clocal_path>:\u003cremote_path> [--remove 'regex']\nosmo dataset collect \u003ccollection_name> \u003cds1> \u003cds2:tag> ...\nosmo dataset rename \u003cold> \u003cnew> [--force]\nosmo dataset query \u003cquery.yaml> [--bucket \u003cb>]\nosmo dataset migrate \u003cname:tag> --target-bucket \u003cb>\nosmo dataset tag \u003cname:tag> --set \u003cnew_tag> # or --delete\nosmo dataset label \u003cname:tag> --set key=val # or --delete key\nosmo dataset metadata \u003cname:tag> --set key=val\nosmo dataset checksum \u003clocal_path> # local MD5 aggregate\nosmo dataset check \u003cname> [--access-type ...]\n```\n\nDataset names follow the pattern `[bucket/]name[:tag]`.\n\n---\n\n## `osmo data` (Direct Storage)\n\n```bash\nosmo data upload \u003cremote_uri> \u003clocal_path> [\u003cmore_paths>...] [--regex '...'] [-p N] [-T N]\nosmo data download \u003cremote_uri> \u003clocal_path> [--regex '...'] [--resume] [-p N] [-T N]\nosmo data list \u003cremote_uri> [--prefix \u003cp>] [--recursive] [--regex '...'] [--no-pager]\nosmo data delete \u003cremote_uri> [--regex '...']\nosmo data check \u003cremote_uri> [--access-type \u003ctype>] [--config-file \u003cpath>]\n```\n\n`-p` sets parallel processes, `-T` sets threads per process. Uses the multi-cloud\nstorage SDK (S3, Azure, GCS, Swift, TOS).\n\nFor workflow outputs, use `osmo data`, including local MicroK8s MinIO. Do not\nread `/var/snap/microk8s/common/default-storage/minio-operator-data*/` directly:\nMinIO chunks objects and encrypts them at rest, so the files are not usable\noutside MinIO. Use `--no-pager` whenever running non-interactively.\n\n```bash\n# Discover workflow folders\nosmo data list --no-pager s3://osmo-workflows\n\n# Inspect one workflow or output prefix\nosmo data list --no-pager s3://osmo-workflows/\u003cworkflow_id>/\nosmo data list --no-pager --recursive s3://osmo-workflows/\u003cworkflow_id>/\n\n# Download so the files are available locally\nosmo data download s3://osmo-workflows/\u003cworkflow_id>/ /tmp/\u003cworkflow_id>-data\n```\n\nLocal MinIO client access is also valid when it goes through MinIO. The local\n`mc` client may already have an `osmo` alias configured:\n\n```bash\nmc ls osmo/osmo-workflows/\nmc ls --recursive osmo/osmo-workflows/\u003cworkflow_id>/\nmc cp --recursive osmo/osmo-workflows/\u003cworkflow_id>/ /tmp/\u003cworkflow_id>-data/\n```\n\nDo not confuse this with direct disk access under `/var/snap/...`; that path is\nchunked/encrypted and not usable as workflow output files.\n\n---\n\n## `osmo app`\n\n```bash\nosmo app create \u003cname> --description \"desc\" [--file \u003cyaml>] # editor if no --file\nosmo app update \u003cname[:version]> [--file \u003cyaml>]\nosmo app info \u003cname[:version]> [--count N] [--order asc|desc]\nosmo app show \u003cname[:version]>\nosmo app spec \u003cname[:version]>\nosmo app list [--name \u003cn>] [--user \u003cu>] [--all-users] [--count N]\nosmo app delete \u003cname[:version]> [--all] [--force]\nosmo app rename \u003cold> \u003cnew> [--force]\nosmo app submit \u003cname[:version]> --pool \u003cpool> [--set key=val] [--dry-run] [--priority ...]\n```\n\n`app submit` delegates to workflow submission with app context.\n\n---\n\n## `osmo profile`\n\n```bash\nosmo profile list # show profile, pools, settings\nosmo profile set pool \u003cpool_name> # set default pool\nosmo profile set bucket \u003cbucket_name> # set default bucket\nosmo profile set notifications \u003cbool> # toggle notifications\n```\n\n---\n\n## `osmo user` (Admin)\n\n```bash\nosmo user list [--format-type json|text]\nosmo user create \u003cusername>\nosmo user update \u003cusername> [role flags]\nosmo user delete \u003cusername>\nosmo user get \u003cusername>\n```\n\n---\n\n## `osmo token` (Personal Access Tokens)\n\n```bash\nosmo token set \u003cname> [--expires-at \u003cdate>] [--description \"...\"] [--roles ...]\nosmo token delete \u003cname>\nosmo token list\nosmo token roles \u003cname>\n```\n\nAdmin variants accept `--user \u003cusername>` to manage other users' tokens.\n\n---\n\n## `osmo credential`\n\n```bash\nosmo credential set \u003cname> --type REGISTRY|DATA|GENERIC --payload key=value [k2=v2 ...]\nosmo credential list\nosmo credential delete \u003cname>\n```\n\n**The CLI takes `--payload key=value` pairs, NOT `--server/--username/--password`\nflags.** Those flags do not exist; using them will fail. See the full narrative\n(NGC key resolution, workflow wiring, common gotchas) in `SKILL.md` →\n**Set Up an Image Registry Credential**.\n\n### `--type REGISTRY`\n\nUsed for pulling private container images in OSMO workflows. Keys:\n\n| Key | Meaning |\n|-----|---------|\n| `registry` | Registry hostname (e.g. `nvcr.io`), no scheme, no path |\n| `username` | Registry username (for NGC: literal `$oauthtoken`) |\n| `auth` | **Raw** auth token (for NGC: the raw NGC API key). NOT the base64 `user:pass` string Docker writes to `~/.docker/config.json` |\n\n```bash\n# nvcr.io (NGC) — most common case\nosmo credential set nvcr --type REGISTRY \\\n --payload registry=nvcr.io username='$oauthtoken' auth=\"$NGC_API_KEY\"\n```\n\nReference it from a workflow task via the task-level `credentials:` map\n(key = this credential name). OSMO auto-wires a REGISTRY credential referenced\nthis way as an `imagePullSecret` on the task pod. Example:\n\n```yaml\ntasks:\n- name: train\n image: nvcr.io/nvidia/pytorch:24.01-py3\n credentials:\n nvcr: # same name used with `osmo credential set`\n NGC_CLI_API_KEY: auth # optional: also expose `auth` as an env var\n```\n\nName must match exactly — `nvcr_io` ≠ `nvcr-io`. Mismatches show up as\n`ImagePullBackOff` on the task pod. See `references/workflow-spec.md` for the\nfull `credentials:` schema.\n\n### `--type GENERIC`\n\nArbitrary key-value secrets surfaced into tasks (env vars or credential mount).\nCommonly used for HuggingFace tokens, API keys, etc. Keys are free-form — the\ntask spec decides how to consume them.\n\n```bash\nosmo credential set hf-token --type GENERIC --payload token=hf_YOUR_TOKEN\nosmo credential set ngc-api-key --type GENERIC --payload key=\"$NGC_API_KEY\"\n```\n\n### `--type DATA`\n\nObject-storage credentials for `osmo dataset` / `osmo data` and workflow\n`inputs:` / `outputs:` that point at S3/GCS/Azure/Swift/TOS URIs. Also updates\nthe local `config.yaml` so the storage SDK picks it up for subsequent\nclient-side calls.\n\n```bash\nosmo credential set my-s3 --type DATA --payload \\\n access_key=AKIA... secret_key=... endpoint=https://s3.amazonaws.com region=us-east-1\n```\n\nExact keys vary by provider; see `osmo credential set --help` and\n`references/workflow-spec.md` for the provider-specific fields. For non-AWS S3\n(MinIO, TOS, etc.), both `region` and `override_url`/`endpoint` are typically\nrequired.\n\n---\n\n## `osmo bucket`\n\n```bash\nosmo bucket list\n```\n\n---\n\n## `osmo task`\n\n```bash\nosmo task list [--workflow \u003cid>] [--status ...] [--pool ...] [--count N]\n```\n\n---\n\n## Configs (Admin)\n\nIn 6.3 ConfigMap mode (`services.configFile.enabled: true`), all configs live in the `osmo-service-configs` ConfigMap. The `osmo config` CLI subcommands no-op or 409 here.\n\n```bash\n# Read\nkubectl get cm osmo-service-configs -n osmo-minimal -o yaml\n\n# Apply a change (scripted, idempotent)\nkubectl patch cm osmo-service-configs -n osmo-minimal --type=merge -p ...\n```\n\nThe osmo-service container watches `/etc/osmo/configs/config.yaml` via inotify and reloads on change.\n\n\n---\n\n## `osmo standalone` (Local Docker Execution)\n\n```bash\nosmo standalone run -f \u003cworkflow.yaml> [flags]\n```\n\n| Flag | Description |\n|------|-------------|\n| `-f` / `--file` | Workflow YAML (required) |\n| `--work-dir` | Working directory for intermediate data |\n| `--keep` | Preserve containers after completion |\n| `--docker` | Docker binary path (default: `docker`) |\n| `--resume` | Resume from last successful step |\n| `--from-step \u003ctask>` | Resume from a specific task |\n| `--credential NAME=PATH` | Mount credential directory (repeatable) |\n| `--set key=value` | Template variable overrides |\n| `--set-string key=value` | String-only overrides |\n| `--shm-size \u003csize>` | Shared memory size (e.g. `16g`) |\n\nRuns tasks serially via `docker run`. Does NOT support `{{host:taskname}}` —\nuse `docker-compose` for inter-task networking.\n\n---\n\n## `osmo docker-compose` (Local Parallel Execution)\n\n```bash\nosmo docker-compose run -f \u003cworkflow.yaml> [flags]\n```\n\nSame flags as `standalone` except `--compose-cmd` replaces `--docker`:\n\n| Flag | Description |\n|------|-------------|\n| `--compose-cmd` | Compose command (default: `docker compose`) |\n\nSupports `{{host:taskname}}` via shared Docker networks. Tasks in the same\ngroup run in parallel. Groups execute in topological order (serial between groups,\nparallel within).\n\n---\n\n## Global Flags\n\nAll commands support:\n\n| Flag | Description |\n|------|-------------|\n| `--log-level` | Logging level (default: INFO) |\n\nMost list/query commands support:\n\n| Flag | Description |\n|------|-------------|\n| `--format-type` / `-t` | `json` or `text` (default: text) |\n\nUse `--format-type json` for machine-readable output (recommended for scripting\nand when parsing output programmatically).\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":14901,"content_sha256":"3c883dc1501dc01151644cfd91c9033a78c378738ff5f72376c6ea24590c8a34"},{"filename":"components/osmo-cli/references/workflow-patterns.md","content":"# OSMO Workflow Patterns Reference\n\nRead this file when generating a **multi-task, parallel, pipelined, or templated**\nworkflow. The basic single-task scaffold in SKILL.md is sufficient for simple jobs;\nthis reference covers everything beyond that.\n\n## Table of Contents\n\n- [Critical Rules](#critical-rules)\n- [Pattern 1: Independent Parallel Tasks](#pattern-1-independent-parallel-tasks)\n- [Pattern 2: Serial Tasks with Data Dependencies](#pattern-2-serial-tasks-with-data-dependencies)\n- [Pattern 3: Synchronized Groups (Parallel with Coordination)](#pattern-3-synchronized-groups-parallel-with-coordination)\n- [Pattern 4: Combination Pipelines (Serial Groups, Parallel Within)](#pattern-4-combination-pipelines-serial-groups-parallel-within)\n- [Pattern 5: Jinja Templates](#pattern-5-jinja-templates)\n\n---\n\n## Critical Rules\n\n- **`tasks:` and `groups:` are mutually exclusive** at the workflow level — never mix them\n- **Memory and storage must use binary units**: `Gi`, `Mi` — never `GB` or `MB`\n- **Every group must have exactly one `lead: true` task** — the group terminates when the lead exits, so make sure the lead outlives its non-lead siblings\n- **`{{input:N}}` is 0-indexed** and ordered by the `inputs:` list on that task\n\n---\n\n## Pattern 1: Independent Parallel Tasks\n\nTasks defined under `tasks:` with no `inputs:` between them run simultaneously. This is\nthe simplest form of parallelism — no coordination needed.\n\n```yaml\nworkflow:\n name: parallel-tasks\n tasks:\n - name: task-a\n image: alpine:3.18\n command: [\"echo\", \"Hello from task-a\"]\n\n - name: task-b\n image: alpine:3.18\n command: [\"echo\", \"Hello from task-b\"]\n\n - name: task-c\n image: alpine:3.18\n command: [\"echo\", \"Hello from task-c\"]\n```\n\nAll three tasks start at the same time. They cannot communicate with each other over\nthe network. Use this pattern for embarrassingly parallel workloads (batch processing,\nhyperparameter sweeps, independent eval runs).\n\n---\n\n## Pattern 2: Serial Tasks with Data Dependencies\n\nAdd an `inputs:` section to a task to declare that it depends on another task's output.\nOSMO automatically waits for the upstream task, then makes its output available at\n`{{input:N}}` (0-indexed, matching the order of the `inputs:` list).\n\n```yaml\nworkflow:\n name: serial-tasks\n tasks:\n\n - name: task1\n image: ubuntu:22.04\n command: [sh]\n args: [/tmp/run.sh]\n files:\n - contents: |\n echo \"Data from task 1\" > {{output}}/result.txt\n path: /tmp/run.sh\n\n - name: task2\n image: ubuntu:22.04\n command: [sh]\n args: [/tmp/run.sh]\n files:\n - contents: |\n # task1's output is at {{input:0}}\n cat {{input:0}}/result.txt\n echo \"Data from task 2\" > {{output}}/result.txt\n path: /tmp/run.sh\n inputs:\n - task: task1 # creates dependency AND data flow\n\n - name: task3\n image: ubuntu:22.04\n command: [sh]\n args: [/tmp/run.sh]\n files:\n - contents: |\n cat {{input:0}}/result.txt # from task1\n cat {{input:1}}/result.txt # from task2\n path: /tmp/run.sh\n inputs:\n - task: task1\n - task: task2\n```\n\nIf a task fails, all downstream dependents are automatically canceled.\n\n---\n\n## Pattern 3: Synchronized Groups (Parallel with Coordination)\n\nUse `groups:` when tasks need to **start together** and/or **communicate over the\nnetwork** (e.g. distributed training, client-server). All tasks in a group launch\nsimultaneously; the group is considered complete when the lead task exits.\n\n```yaml\nworkflow:\n name: grouped-workflow\n groups:\n - name: parallel-processing\n tasks:\n - name: processor-1\n lead: true # required — group ends when this task exits\n image: ubuntu:24.04\n command: [\"bash\", \"-c\"]\n args:\n - |\n echo \"Processor 1 running...\"\n sleep 30\n echo \"Processor 1 done\"\n\n - name: processor-2\n image: ubuntu:24.04\n command: [\"bash\", \"-c\"]\n args:\n - |\n echo \"Processor 2 running...\"\n sleep 10\n echo \"Processor 2 done\"\n```\n\n### Inter-task communication within a group\n\nTasks in the same group can reach each other using `{{host:task-name}}`, which resolves\nto the IP address of that task at runtime:\n\n```yaml\nworkflow:\n name: client-server\n groups:\n - name: parallel-processing\n tasks:\n - name: server\n lead: true\n image: alpine:3.18\n command: [\"sh\", \"-c\"]\n args:\n - |\n echo \"hello\" > /tmp/hello.txt\n nc -w 50 -l -p 24831 \u003c /tmp/hello.txt\n\n - name: client\n image: alpine:3.18\n command: [\"sh\", \"-c\"]\n args:\n - |\n nc -w 30 {{host:server}} 24831 > /tmp/received.txt\n echo \"Got: $(cat /tmp/received.txt)\"\n```\n\n`{{host:task-name}}` only works within the same group — tasks in different groups\ncannot use it.\n\n---\n\n## Pattern 4: Combination Pipelines (Serial Groups, Parallel Within)\n\nGroups can depend on each other through task-level `inputs:`. When any task in a group\ndeclares an input from a task in another group, **the entire downstream group waits for\nthe entire upstream group to complete**.\n\nThis gives you serial execution *between* groups and parallel execution *within* groups.\n\n```yaml\nworkflow:\n name: data-pipeline\n groups:\n # Group 1: runs first\n - name: prepare-data\n tasks:\n - name: generate-dataset\n lead: true\n image: ubuntu:24.04\n command: [\"bash\", \"-c\"]\n args:\n - |\n mkdir -p {{output}}/data\n echo \"sample_1,value_1\" >> {{output}}/data/dataset.csv\n echo \"sample_2,value_2\" >> {{output}}/data/dataset.csv\n\n - name: validate-data\n image: ubuntu:24.04\n command: [\"bash\", \"-c\"]\n args: [\"echo Validating...\"]\n\n # Group 2: waits for Group 1 via task inputs\n - name: train-models\n tasks:\n - name: train-model-a\n lead: true\n image: ubuntu:24.04\n command: [\"bash\", \"-c\"]\n args:\n - |\n cat {{input:0}}/data/dataset.csv\n echo \"Model A trained\"\n inputs:\n - task: generate-dataset # establishes group dependency\n\n - name: train-model-b\n image: ubuntu:24.04\n command: [\"bash\", \"-c\"]\n args:\n - |\n wc -l {{input:0}}/data/dataset.csv\n echo \"Model B trained\"\n inputs:\n - task: generate-dataset\n```\n\n**Execution flow:** `prepare-data` group completes → `train-models` group starts with\nboth `train-model-a` and `train-model-b` running in parallel.\n\n> **Lead task caution:** If the lead task finishes before non-lead tasks in its group,\n> the group terminates early. Ensure the lead task's duration covers its siblings, or\n> use a barrier script to synchronize completion.\n\n---\n\n## Pattern 5: Jinja Templates\n\nUse Jinja templates to make workflows configurable at submission time without editing\nthe YAML. Variables use `{{ }}` syntax; defaults live in a `default-values:` block at\nthe top level (outside `workflow:`).\n\n```yaml\nworkflow:\n name: \"{{workflow_name}}\"\n\n resources:\n training:\n cpu: 8\n memory: 32Gi\n gpu: {{gpu_count}}\n\n tasks:\n {% for i in range(num_tasks) %}\n - name: train-model-{{i}}\n image: {{training_image}}\n command: [\"python\", \"train.py\"]\n args:\n - \"--dataset={{dataset_name}}\"\n - \"--fold={{i}}\"\n resource: training\n outputs:\n - dataset:\n name: \"{{model_type}}_fold_{{i}}\"\n {% endfor %}\n\ndefault-values:\n workflow_name: ml-training\n dataset_name: imagenet\n model_type: resnet50\n num_tasks: 3\n gpu_count: 1\n training_image: nvcr.io/nvidia/pytorch:24.01-py3\n```\n\nSubmit with defaults or override at the command line:\n\n```bash\n# Use defaults\nosmo workflow submit template-workflow.yaml\n\n# Override specific values\nosmo workflow submit template-workflow.yaml \\\n --set model_type=efficientnet \\\n --set gpu_count=4 \\\n --set num_tasks=5\n```\n\n**Special tokens** (set automatically by OSMO, cannot be overridden with `--set`):\n\n| Token | Value |\n|---|---|\n| `{{output}}` | Path where this task should write output data |\n| `{{input:N}}` | Path to the Nth upstream task's output (0-indexed) |\n| `{{workflow_id}}` | Unique ID for this workflow run |\n| `{{host:task-name}}` | IP address of a task in the same group |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":8197,"content_sha256":"15d746e1fff6f06a2f9d7c93f3f269151dcfd16a3a7618a180975e08b7496c13"},{"filename":"components/osmo-cli/references/workflow-spec.md","content":"# OSMO Workflow Spec Reference\n\nComplete schema for OSMO workflow YAML files.\n\n## Table of Contents\n\n- [Top-Level Structure](#top-level-structure)\n- [Resources](#resources)\n- [Task Spec (`TaskSpec`)](#task-spec-taskspec)\n- [Inputs](#inputs)\n- [Outputs](#outputs)\n- [Groups](#groups)\n- [Special Tokens](#special-tokens)\n- [Jinja Templates](#jinja-templates)\n- [Cookbook Examples](#cookbook-examples)\n\n---\n\n## Top-Level Structure\n\n```yaml\nversion: 2 # optional; must be 2 if present (default)\n\nworkflow:\n name: \u003cstring> # workflow name\n pool: \u003cstring> # target pool (usually set via --pool flag instead)\n resources: ... # named resource profiles\n tasks: [...] # flat task list (mutually exclusive with groups)\n groups: [...] # grouped tasks (mutually exclusive with tasks)\n timeout:\n exec_timeout: \u003cduration> # max execution time\n queue_timeout: \u003cduration> # max queue wait time\n\ndefault-values: # Jinja template defaults (top-level, outside workflow:)\n var_name: value\n```\n\n**Rule:** Exactly one of `tasks:` or `groups:` must be present — never both.\n\n---\n\n## Resources\n\nNamed resource profiles referenced by tasks via the `resource:` field.\n\n```yaml\nresources:\n default: # every workflow has an implicit \"default\" profile\n cpu: 8\n gpu: 2\n memory: 32Gi # must use binary units: Gi, Mi\n storage: 100Gi # must use binary units: Gi, Mi\n platform: dgx-h100 # target hardware platform\n nodesExcluded: # exclude specific nodes\n - bad-node-01\n topology: # advanced placement constraints\n - key: \u003cstring>\n group: \u003cstring>\n requirementType: \u003cstring>\n\n gpu_heavy: # custom named profile\n cpu: 16\n gpu: 8\n memory: 128Gi\n storage: 200Gi\n```\n\nTasks use `resource: gpu_heavy` to select a profile. Default is `\"default\"`.\n\n---\n\n## Task Spec (`TaskSpec`)\n\nEach task defines a container to run.\n\n```yaml\ntasks:\n- name: \u003cstring> # unique task name\n image: \u003cstring> # container image\n command: [\u003cstring>, ...] # entrypoint (required, non-empty)\n args: [\u003cstring>, ...] # additional arguments\n resource: \u003cstring> # name of resource profile (default: \"default\")\n lead: \u003cbool> # required in multi-task groups (one per group)\n\n # Data I/O\n inputs: [...] # data inputs (see below)\n outputs: [...] # data outputs (see below)\n\n # Configuration\n environment: # environment variables\n KEY: \"value\"\n files: # inline files created in the container\n - path: /tmp/script.sh\n contents: |\n #!/bin/bash\n echo \"Hello\"\n - path: /tmp/data.bin\n contents: \u003cbase64_string>\n base64: true\n\n # Credentials\n credentials:\n my_credential: /mnt/creds # mount credential at path\n my_secret: # or map env vars to secret keys\n ENV_VAR: secret_key\n\n # Advanced\n privileged: \u003cbool>\n hostNetwork: \u003cbool>\n volumeMounts: [...] # host volume mounts\n downloadType: \u003cstring> # download behavior\n cacheSize: \u003cstring> # cache size\n backend: \u003cstring> # per-task backend override\n\n # Checkpointing\n checkpoint:\n - path: /tmp/checkpoints\n url: s3://bucket/checkpoints\n frequency: 60s\n regex: '.*\\.(pt|bin)

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

# optional filter\n\n # Error handling\n exitActions:\n COMPLETE: 0 # exit 0 = success\n RESCHEDULE: 1-255 # non-zero = retry\n FAIL: 137 # specific code = fail\n\n # Monitoring\n kpis:\n index: \u003cint>\n path: \u003cstring>\n```\n\n---\n\n## Inputs\n\nTasks can receive data from three sources:\n\n### Task-to-task (data dependency)\n\n```yaml\ninputs:\n- task: upstream_task_name\n regex: '.*\\.csv

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

# optional: filter files\n```\n\nCreates a DAG dependency. Upstream task must complete before this task starts.\nAccess via `{{input:N}}` (0-indexed by position in the inputs list) or\n`{{input:upstream_task_name}}`.\n\n### URL (cloud storage)\n\n```yaml\ninputs:\n- url: s3://bucket/data/\n regex: '.*\\.png

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

# optional filter\n```\n\nDownloads from S3, GCS, Azure, Swift, or TOS at task start.\n\n### Dataset (OSMO-managed)\n\n```yaml\ninputs:\n- dataset:\n name: my_dataset\n path: /custom/mount # optional\n regex: '.*' # optional\n```\n\n---\n\n## Outputs\n\n### Dataset output\n\n```yaml\noutputs:\n- dataset:\n name: my_output_dataset\n # optional: metadata, labels\n```\n\nThe task writes to `{{output}}`, which OSMO uploads as a dataset on completion.\nFetch it with `osmo dataset download \u003cname> \u003clocal-path>`.\n\n### URL output\n\n```yaml\noutputs:\n- url: s3://bucket/output/\n```\n\nFetch URL outputs with `osmo data list --no-pager \u003curl>` and\n`osmo data download \u003curl> \u003clocal-path>`.\n\n### Update existing dataset\n\n```yaml\noutputs:\n- update_dataset:\n name: existing_dataset\n```\n\n---\n\n## Groups\n\nUse groups when tasks need co-scheduling or network communication.\n\n```yaml\ngroups:\n- name: training_group\n barrier: true # default true; wait for all tasks in group\n ignoreNonleadStatus: true # default true; group status follows lead only\n tasks:\n - name: coordinator\n lead: true # exactly one lead per multi-task group\n image: my-image\n command: [\"python\", \"coord.py\"]\n - name: worker\n image: my-image\n command: [\"python\", \"worker.py\", \"--coord={{host:coordinator}}\"]\n```\n\n### Group rules\n\n- Exactly one task must have `lead: true` in multi-task groups.\n- The group terminates when the lead task exits.\n- `{{host:taskname}}` resolves to the DNS name of a task in the same group.\n- `{{host:taskname}}` does NOT work across groups.\n\n### Cross-group dependencies\n\nGroups depend on each other through task-level `inputs:`:\n\n```yaml\ngroups:\n- name: stage1\n tasks:\n - name: produce\n lead: true\n command: [\"bash\", \"-c\", \"echo data > {{output}}/out.txt\"]\n\n- name: stage2\n tasks:\n - name: consume\n lead: true\n command: [\"bash\", \"-c\", \"cat {{input:0}}/out.txt\"]\n inputs:\n - task: produce # stage2 waits for ALL of stage1 to complete\n```\n\n---\n\n## Special Tokens\n\nAutomatically set by OSMO — cannot be overridden with `--set`:\n\n| Token | Resolves To |\n|-------|-------------|\n| `{{output}}` | Output directory path for this task |\n| `{{input:N}}` | Nth input path (0-indexed by `inputs:` list order) |\n| `{{input:taskname}}` | Input path by upstream task name |\n| `{{host:taskname}}` | DNS name of a task in the same group |\n| `{{workflow_id}}` | Unique workflow run ID |\n\nThese tokens work in `command`, `args`, `environment` values, and `files` contents.\n\n---\n\n## Jinja Templates\n\nMake workflows configurable at submit time:\n\n```yaml\nworkflow:\n name: \"{{workflow_name}}\"\n resources:\n default:\n gpu: {{gpu_count}}\n tasks:\n - name: train\n image: \"{{image}}\"\n command: [\"python\", \"train.py\"]\n args: [\"--epochs={{epochs}}\"]\n\ndefault-values:\n workflow_name: my-training\n gpu_count: 1\n image: nvcr.io/nvidia/pytorch:24.01-py3\n epochs: 10\n```\n\n```bash\n# Submit with defaults\nosmo workflow submit template.yaml --pool my-pool\n\n# Override values\nosmo workflow submit template.yaml --pool my-pool --set gpu_count=4 epochs=50\n```\n\nJinja supports loops and conditionals:\n\n```yaml\ntasks:\n{% for i in range(num_workers) %}\n- name: worker-{{i}}\n image: my-image\n command: [\"python\", \"worker.py\", \"--id={{i}}\"]\n{% endfor %}\n```\n\n---\n\n## Cookbook Examples\n\nReal-world examples in the OSMO repo under `cookbook/`:\n\n| Example | Pattern |\n|---------|---------|\n| `tutorials/hello_world.yaml` | Minimal single task |\n| `tutorials/template_hello_world.yaml` | Jinja template with defaults |\n| `tutorials/serial_workflow.yaml` | Serial task chain |\n| `tutorials/parallel_tasks.yaml` | Independent parallel tasks |\n| `tutorials/group_tasks.yaml` | Synchronized group |\n| `tutorials/group_tasks_communication.yaml` | `{{host:...}}` inter-task networking |\n| `tutorials/combination_workflow_complex.yaml` | Multi-group pipeline with data flow |\n| `tutorials/data_download.yaml` | S3 URL input |\n| `tutorials/dataset_upload.yaml` | Dataset output with `{{output}}` |\n| `tutorials/resources_platforms.yaml` | Multiple resource profiles + platforms |\n| `tutorials/resources_basic.yaml` | Basic resource configuration |\n| `dnn_training/torchrun_multinode/train.yaml` | Multi-node distributed training |\n| `reinforcement_learning/single_gpu/train_policy.yaml` | RL training with GPU |\n| `integration_and_tools/jupyterlab/jupyter.yaml` | Interactive Jupyter session |\n| `integration_and_tools/vscode/vscode.yaml` | VS Code remote session |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":8698,"content_sha256":"f67065012b1b1d495385b9bcf6ecb2bb32f114b0a7149be4adb68b7647082ce9"},{"filename":"components/osmo-cli/scripts/preflight.sh","content":"#!/usr/bin/env bash\n# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n# SPDX-License-Identifier: Apache-2.0\n\nset -euo pipefail\n\nMIN_OSMO_CLI_VERSION=\"6.3.0\"\nPASS=true\nWARNINGS=0\n\nfail() {\n echo \"ERROR: $*\" >&2\n PASS=false\n}\n\nwarn() {\n echo \"WARNING: $*\" >&2\n WARNINGS=$((WARNINGS + 1))\n}\n\nok() {\n echo \"OK: $*\"\n}\n\nversion_ge() {\n local got=\"${1#v}\"\n local want=\"${2#v}\"\n got=\"${got%%[-+]*}\"\n want=\"${want%%[-+]*}\"\n awk -v got=\"${got}\" -v want=\"${want}\" '\n BEGIN {\n split(got, g, \".\")\n split(want, w, \".\")\n for (i = 1; i \u003c= 3; i++) {\n if ((g[i] + 0) > (w[i] + 0)) exit 0\n if ((g[i] + 0) \u003c (w[i] + 0)) exit 1\n }\n }\n '\n}\n\ncheck_min_version() {\n local name=\"$1\"\n local version=\"$2\"\n local min_version=\"$3\"\n if [[ -z \"${version}\" ]]; then\n fail \"could not determine ${name} version; need >= ${min_version}\"\n elif version_ge \"${version}\" \"${min_version}\"; then\n ok \"${name} ${version} >= ${min_version}\"\n else\n fail \"${name} ${version} \u003c ${min_version}\"\n fi\n}\n\nosmo_client_version() {\n local output=\"\"\n local major=\"\"\n local minor=\"\"\n local revision=\"\"\n if output=$(osmo version --format-type json 2>&1); then\n major=$(awk -F'\"' '/\"major\"/ { print $4; exit }' \u003c\u003c\u003c\"${output}\")\n minor=$(awk -F'\"' '/\"minor\"/ { print $4; exit }' \u003c\u003c\u003c\"${output}\")\n revision=$(awk -F'\"' '/\"revision\"/ { print $4; exit }' \u003c\u003c\u003c\"${output}\")\n if [[ -n \"${major}\" && -n \"${minor}\" && -n \"${revision}\" ]]; then\n printf \"%s.%s.%s\" \"${major}\" \"${minor}\" \"${revision}\"\n return 0\n fi\n else\n fail \"osmo version failed: ${output}\"\n return 1\n fi\n\n if output=$(osmo version 2>&1); then\n awk '\n match($0, /[0-9]+[.][0-9]+[.][0-9]+/) {\n print substr($0, RSTART, RLENGTH)\n exit\n }\n ' \u003c\u003c\u003c\"${output}\"\n else\n fail \"osmo version failed: ${output}\"\n return 1\n fi\n}\n\nfinish() {\n if [[ \"${PASS}\" != \"true\" ]]; then\n echo \"==> osmo-cli preflight failed\" >&2\n exit 1\n fi\n echo \"==> osmo-cli preflight passed (${WARNINGS} warning(s))\"\n}\n\necho \"==> osmo-cli preflight\"\nif command -v osmo >/dev/null 2>&1; then\n ok \"osmo found ($(command -v osmo))\"\nelse\n fail \"osmo not found in PATH\"\n finish\nfi\n\nosmo_version=\"\"\nif osmo_version=$(osmo_client_version); then\n :\nelse\n osmo_version=\"\"\nfi\ncheck_min_version \"osmo CLI\" \"${osmo_version}\" \"${MIN_OSMO_CLI_VERSION}\"\nfinish\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":2414,"content_sha256":"2025db9e0603f822203bd45fe337b90c4e6d7e3b89c4e97b00265c4017e21a3d"},{"filename":"components/osmo-cli/tests/orchestrator-runtime-failure.md","content":"# Test: Runtime Script Failure — Diagnosis and Auto-Recovery\n\n## Objective\n\nVerify that the phase-split orchestration pattern works end-to-end: the\nworkflow-expert agent handles setup/submission and failure diagnosis in its\nisolated context, while the **main conversation** monitors inline with live\nstatus updates visible to the user. The full cycle — submit, detect failure,\ndiagnose, fix, resubmit, complete — must run without additional user input.\n\n## Why This Test Matters\n\nOSMO validates workflow YAML structure and resource limits at submission time,\nbut cannot inspect the correctness of embedded shell scripts. A workflow can\npass all server-side validation yet fail seconds after launch due to a script\nbug. The phase-split pattern must handle this gracefully:\n\n1. **Workflow expert** submits the workflow and returns the workflow ID\n2. **Main agent** monitors inline and detects the FAILED status (user sees this)\n3. **Workflow expert** (resumed) pulls logs, diagnoses, fixes, and resubmits\n4. **Main agent** monitors the corrected workflow to completion (user sees this)\n\n## Bug Design\n\nThe workflow contains **two layered runtime bugs**, both invisible to OSMO\nserver-side validation:\n\n| # | Bug | Why it passes validation | Runtime symptom | Expected fix |\n| --- | --------------------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |\n| 1 | `jq` used but not installed in `ubuntu:22.04` | OSMO does not inspect script contents | `jq: command not found`, exit code 127 | Replace `jq` with pure bash, or prepend `apt-get update && apt-get install -y jq` |\n| 2 | `{{outputs}}` (plural) instead of `{{output}}` (singular) | OSMO does not validate template variables inside `files.contents` | `mkdir` creates a literal `{{outputs}}` directory; output dataset path is wrong | Change all `{{outputs}}` to `{{output}}` |\n\n**Bug ordering is deliberate.** Bug 1 triggers first because `set -e` aborts\non the `jq` failure before reaching the `{{outputs}}` lines. This tests\nwhether the workflow expert, when fixing Bug 1, also proactively reviews the rest\nof the script and catches Bug 2 in the same pass — or whether it requires a\nsecond failure-retry cycle.\n\n## Setup\n\nPlace the following workflow YAML at `workflow.yaml` in the working directory.\nThe comments in the YAML **do not** mention the bugs — they read as normal\ndeveloper comments so the agent gets no hints.\n\n```yaml\nworkflow:\n name: hello-world-test\n tasks:\n - name: hello\n image: ubuntu:22.04\n command: [\"bash\"]\n args: [\"/tmp/entry.sh\"]\n files:\n - contents: |\n #!/bin/bash\n set -e\n\n echo \"=========================================\"\n echo \" Hello World from OSMO!\"\n echo \"=========================================\"\n\n echo \"Hostname: $(hostname)\"\n echo \"Date: $(date)\"\n\n # Generate a structured JSON report\n echo \"Generating report...\"\n jq -n --arg host \"$(hostname)\" --arg ts \"$(date -u +%FT%TZ)\" \\\n '{message: \"Hello World\", host: $host, timestamp: $ts}' \\\n > /tmp/report.json\n\n # Write outputs to the dataset mount\n mkdir -p {{outputs}}\n cp /tmp/report.json {{outputs}}/report.json\n echo \"Hello World from OSMO!\" > {{outputs}}/hello.txt\n\n echo \"Done!\"\n path: /tmp/entry.sh\n outputs:\n - dataset:\n name: hello-world-test-output\n resources:\n default:\n cpu: 2\n gpu: 2\n memory: 8Gi\n storage: 10Gi\n```\n\n## Prompt\n\nGive the **main agent** (not the workflow expert directly) this prompt. Do not\nprovide any hints about the bugs. Pre-authorize submission to avoid the\nPhase 3 confirmation pause:\n\n> I have a hello world workflow ready in workflow.yaml. Submit it now to an\n> available GPU pool, monitor it, and report the results when it's done. If\n> it fails, diagnose from logs, fix the workflow, and resubmit automatically\n> without asking me. Run fully autonomously until completion or 3 retries.\n\n## Expected Behavior (Phase-Split Architecture)\n\nThe workflow expert handles setup/submit and failure diagnosis in its isolated\ncontext, while the main conversation monitors inline so the user sees live\nstatus updates.\n\n### Step 1: Main agent spawns workflow expert — Resource Check → Submit\n\n- The main agent reads the user prompt and matches it to the osmo skill's\n \"Orchestrate a Workflow End-to-End\" use case.\n- It spawns the `workflow-expert` agent with a prompt that asks it to\n **check resources and submit only** — NOT to monitor.\n- The workflow expert detects that `workflow.yaml` already exists and uses it.\n- It checks pool availability, picks a pool with free GPUs, and submits.\n- Submission **succeeds** — no YAML structure or resource validation errors.\n- The workflow expert returns: workflow ID, pool name, monitoring commands,\n and its agent ID for later resume.\n\n**What to verify:**\n- The prompt sent to the workflow expert does NOT include monitoring instructions\n (no \"poll status\", no \"monitor until complete\", no \"report progress\").\n- The workflow expert returns promptly after submission without polling.\n\n### Step 2: Main agent monitors inline — user sees live status updates\n\n- The main agent polls `osmo workflow query \u003cid> --format-type json` itself.\n- The user sees each status update in real time:\n - `Status: SCHEDULING (queued 5s)`\n - `Workflow transitioned: SCHEDULING → RUNNING`\n - `Status: RUNNING (task \"hello\" active)`\n- The workflow moves to RUNNING, then FAILED within seconds.\n- The main agent detects the FAILED status from its own polling.\n\n**What to verify:**\n- Status updates appear in the main conversation (not hidden in a subagent).\n- The main agent detects FAILED without the workflow expert's help.\n\n### Step 3: Main agent resumes workflow expert — Failure Diagnosis\n\n- The main agent resumes the workflow expert (using the `resume` parameter with\n the agent ID from Step 1) and passes the failure context.\n- The workflow expert fetches logs via `osmo workflow logs \u003cid> -n 10000`.\n- Logs contain: `jq: command not found` and a non-zero exit code.\n- **Root cause**: The workflow expert identifies `jq` is not in `ubuntu:22.04`.\n- **Fix**: Edits `workflow.yaml` to either:\n - (a) Replace the `jq` invocation with bash-native alternatives, OR\n - (b) Add `apt-get update && apt-get install -y jq` before the `jq` call\n- **Ideal**: While editing, the workflow expert also notices `{{outputs}}` should\n be `{{output}}` and fixes both bugs in one pass.\n- The workflow expert resubmits the corrected workflow and returns the new\n workflow ID.\n\n**What to verify:**\n- The workflow expert is resumed (not spawned fresh) — preserves prior context.\n- Root-cause explanation is clear and in plain language.\n- workflow.yaml is actually modified with the fix.\n\n### Step 4: Main agent resumes inline monitoring\n\n- The main agent monitors the new workflow inline (same as Step 2).\n- If the corrected workflow fails again (Bug 2 not caught in first pass):\n - The main agent resumes the workflow expert again for a second diagnosis.\n - The workflow expert catches `{{outputs}}` → `{{output}}`, fixes, resubmits.\n - The main agent monitors the third submission.\n- After the fully corrected workflow completes, proceed to Step 5.\n\n### Step 5: Main agent reports results\n\n- The main agent (not the workflow expert) reports to the user:\n - Workflow ID and COMPLETED status\n - OSMO Web link\n - What the workflow produced\n - Offers to download the `hello-world-test-output` dataset\n\n## Success Criteria\n\n| # | Criterion | Required | Notes |\n| --- | ---------------------------------------------------------- | -------- | ------------------------------------------------------- |\n| 1 | Workflow expert prompt does NOT include monitoring | Yes | Main agent follows SKILL.md orchestration pattern |\n| 2 | Submission succeeds on first attempt | Yes | No YAML/resource validation errors |\n| 3 | Live status updates visible to user during monitoring | Yes | Main agent polls and reports inline, not in subagent |\n| 4 | Main agent detects FAILED status from its own polling | Yes | Not detected by the workflow expert |\n| 5 | Workflow expert is resumed (not spawned fresh) for diagnosis | Yes | Uses `resume` parameter with agent ID |\n| 6 | Workflow expert fetches and reads logs | Yes | Must use `osmo workflow logs` |\n| 7 | Workflow expert identifies `jq: command not found` | Yes | Clear root-cause statement in plain language |\n| 8 | Workflow expert fixes Bug 1 in workflow.yaml | Yes | Either approach (a) or (b) |\n| 9 | Workflow expert resubmits without asking user | Yes | Mode 2 auto-retry |\n| 10 | Workflow expert fixes Bug 2 (`{{outputs}}` → `{{output}}`) | Yes | First or second pass |\n| 11 | Main agent reports final COMPLETED status | Yes | With workflow ID, web link, and output info |\n| 12 | Total retries ≤ 3 | Yes | No infinite loops |\n| 13 | No user input required after initial prompt | Yes | Core requirement |\n\n## Scoring\n\n- **A — Both bugs fixed in one pass + correct phase split**: The main agent\n monitors inline (user sees live updates), the workflow expert catches both bugs\n in a single diagnosis pass, and the second submission succeeds. Demonstrates\n both the phase-split pattern and thorough proactive script review.\n- **B — Correct phase split + bugs fixed across two retries**: The main agent\n monitors inline correctly. The workflow expert fixes Bug 1 first, then catches\n Bug 2 on the second failure. Acceptable — the retry loop works correctly.\n- **C — Bugs fixed but workflow expert monitors**: Both bugs are fixed and the\n workflow completes, but the main agent delegated monitoring to the workflow\n expert (user saw no live updates). The phase-split pattern failed.\n- **Fail**: Agent does not identify the root cause, asks the user what to do\n before exhausting retries, or loops more than 3 times without resolving.\n\n## Cleanup\n\nAfter the test, delete the test workflow and output dataset:\n```bash\necho \"y\" | osmo dataset delete hello-world-test-output\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":11285,"content_sha256":"465b7de14f885c9dca1f44955c7422be314c9b50727643bb489b3512ad2f6acd"},{"filename":"components/osmo-k8s/reference.md","content":"# Kubernetes OSMO\n\n## Prerequisites\n\n* MicroK8s v1.31+ with configured kubectl\n* helm 3.x\n* git (for shallow clone of https://github.com/nvidia/osmo)\n\n## Supporting files\n\n| Path | Use | When |\n|------|-----|------|\n| `scripts/preflight.sh` | Run first | Checks local MicroK8s/Kubernetes tools, Helm, Git, and repo `.env`; cluster state is verified during deploy. |\n\n# Deployment\n\n1. Run preflight\n\n```bash\nREPO=\"$(git rev-parse --show-toplevel)\"\n\"$REPO/skills/physical-ai-infrastructure-setup-and-resilient-scaling/components/osmo-k8s/scripts/preflight.sh\"\n```\n\n2. Clone https://github.com/nvidia/osmo - use `main` unless otherwise specified\n\n```bash\nOSMO_REF=\"${OSMO_REF:-main}\"\nOSMO_DIR=\"$HOME/.cache/physical-ai/osmo\"\nif [ -d \"$OSMO_DIR/.git\" ]; then\n git -C \"$OSMO_DIR\" fetch --depth 1 origin \"$OSMO_REF\"\n git -C \"$OSMO_DIR\" reset --hard FETCH_HEAD\nelse\n mkdir -p \"$(dirname \"$OSMO_DIR\")\"\n git clone --depth 1 --branch \"$OSMO_REF\" \\\n https://github.com/NVIDIA/OSMO.git \"$OSMO_DIR\"\nfi\n```\n\n3. Load environment secrets\n\n```bash\nREPO=\"$(git rev-parse --show-toplevel)\"\n[ -f \"$REPO/.env\" ] && { set -a; . \"$REPO/.env\"; set +a; }\n```\n\n4. Check for an existing OSMO install before deploying. Do not redeploy or\n upgrade a working install just because namespace `osmo` is empty; Physical AI\n infra uses the `osmo-minimal` namespace.\n\n```bash\nhelm status -n osmo-minimal osmo-minimal\nkubectl get pods -n osmo-minimal\nosmo workflow list --count 5\n```\n\nIf Helm status succeeds and the pods are healthy, reuse the existing install.\nOnly continue to deploy when OSMO is absent or the user explicitly approves a\nrepair/redeploy.\n\n5. Deploy OSMO in \"minimal\" configuration in MicroK8s mode with in-cluster PostgreSQL, Redis, MicroK8s MinIO add-on, ClusterIP gateway with port-forward watchdog\n\n```bash\n\"$OSMO_DIR/deployments/scripts/deploy-osmo-minimal.sh\" \\\n --provider microk8s \\\n --storage-backend minio \\\n --non-interactive\n```\n\nFor CPU-only instances, add `--no-gpu`.\n\n# Verify\n\nVerification is done as part of `deploy-osmo-minimal.sh`. If the script exits with exit code 0, the OSMO deployment is considered verified.\n\n# Re-run\n\nDo not re-run deployment scripts during demos without explicit user approval.\nUse the existing-install checks above first.\n\n# Cleanup\n\nThis intentionally only cleans up the OSMO install - the MicroK8s cluster\nremains up.\n\n```bash\npkill -f 'osmo-pf-watchdog:' || true\n$OSMO_DIR/deployments/scripts/deploy-osmo-minimal.sh \\\n --provider microk8s \\\n --destroy --non-interactive\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":2520,"content_sha256":"3fe8c37aa699caccbc1a674a8891f00073e2862db22e1b870f5c05170157cd88"},{"filename":"components/osmo-k8s/scripts/preflight.sh","content":"#!/usr/bin/env bash\n# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n# SPDX-License-Identifier: Apache-2.0\n\nset -euo pipefail\n\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nREPO_ROOT=\"$(cd \"${SCRIPT_DIR}/../../../../..\" && pwd)\"\nMIN_KUBECTL_VERSION=\"1.31.0\"\nMIN_HELM_VERSION=\"3.0.0\"\nMIN_GIT_VERSION=\"2.25.0\"\nMIN_MICROK8S_VERSION=\"1.31.0\"\nPASS=true\nWARNINGS=0\n\nfail() { echo \"ERROR: $*\" >&2; PASS=false; }\nwarn() { echo \"WARNING: $*\" >&2; WARNINGS=$((WARNINGS + 1)); }\nok() { echo \"OK: $*\"; }\n\nrequire_cmds() {\n local cmd\n for cmd in \"$@\"; do\n if command -v \"${cmd}\" >/dev/null 2>&1; then\n ok \"${cmd} found ($(command -v \"${cmd}\"))\"\n else\n fail \"${cmd} not found in PATH\"\n fi\n done\n}\n\nversion_ge() {\n local got=\"${1#v}\"\n local want=\"${2#v}\"\n got=\"${got%%[-+]*}\"\n want=\"${want%%[-+]*}\"\n awk -v got=\"${got}\" -v want=\"${want}\" '\n BEGIN {\n split(got, g, \".\")\n split(want, w, \".\")\n for (i = 1; i \u003c= 3; i++) {\n if ((g[i] + 0) > (w[i] + 0)) exit 0\n if ((g[i] + 0) \u003c (w[i] + 0)) exit 1\n }\n }\n '\n}\n\ncheck_min_version() {\n local name=\"$1\"\n local version=\"$2\"\n local min_version=\"$3\"\n if [[ -z \"${version}\" ]]; then\n fail \"could not determine ${name} version; need >= ${min_version}\"\n elif version_ge \"${version}\" \"${min_version}\"; then\n ok \"${name} ${version} >= ${min_version}\"\n else\n fail \"${name} ${version} \u003c ${min_version}\"\n fi\n}\n\nkubectl_semver() {\n local version=\"\"\n version=$(kubectl version --client=true --short 2>/dev/null | awk '/Client Version/ { sub(/^v/, \"\", $3); print $3; exit }' || printf \"\")\n [[ -n \"${version}\" ]] || version=$(kubectl version --client -o json 2>/dev/null | awk -F'\"' '/\"gitVersion\"/ { sub(/^v/, \"\", $4); print $4; exit }' || printf \"\")\n printf \"%s\" \"${version}\"\n}\n\nfinish() {\n if [[ \"${PASS}\" != \"true\" ]]; then\n echo \"==> osmo-k8s preflight failed\" >&2\n exit 1\n fi\n echo \"==> osmo-k8s preflight passed (${WARNINGS} warning(s))\"\n}\n\necho \"==> osmo-k8s preflight\"\nrequire_cmds kubectl helm git microk8s awk\nhelm_version=$(helm version --short 2>/dev/null | awk '{ sub(/^v/, \"\", $1); print $1; exit }' || printf \"\")\ncheck_min_version \"helm\" \"${helm_version}\" \"${MIN_HELM_VERSION}\"\ncheck_min_version \"kubectl\" \"$(kubectl_semver)\" \"${MIN_KUBECTL_VERSION}\"\ncheck_min_version \"git\" \"$(git --version 2>/dev/null | awk '{ print $3; exit }' || printf \"\")\" \"${MIN_GIT_VERSION}\"\ncheck_min_version \"microk8s\" \"$(microk8s version 2>/dev/null | awk '{ sub(/^v/, \"\", $2); print $2; exit }' || printf \"\")\" \"${MIN_MICROK8S_VERSION}\"\nif [[ -f \"${REPO_ROOT}/.env\" ]]; then\n set -a\n # shellcheck disable=SC1091\n source \"${REPO_ROOT}/.env\"\n set +a\n ok \"loaded ${REPO_ROOT}/.env\"\nelse\n warn \"${REPO_ROOT}/.env not found\"\nfi\n[[ -n \"${NGC_API_KEY:-}\" ]] && ok \"NGC_API_KEY set\" || fail \"NGC_API_KEY is unset\"\nfinish\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":2883,"content_sha256":"a171d06cc35e78f879e4572d76c23208d16299e3f0f40c3c37f2a308f3ce8ed0"},{"filename":"evals/evals.json","content":"{\n \"version\": \"1.0.0\",\n \"skill\": \"physical-ai-infrastructure-setup-and-resilient-scaling\",\n \"cases\": [\n {\n \"id\": \"infra-setup-azure-aks-positive\",\n \"question\": \"Set up resilient Physical AI infrastructure for a video data augmentation workflow on Azure AKS with OSMO and NIM Operator.\",\n \"expected_skill\": \"physical-ai-infrastructure-setup-and-resilient-scaling\",\n \"expected_script\": null,\n \"ground_truth\": \"The agent should select the infrastructure setup skill, classify the target as Azure AKS with OSMO and NIM Operator, run selected component preflights first, and stop at the first failed infrastructure gate.\",\n \"expected_behavior\": [\n \"Reads skills/physical-ai-infrastructure-setup-and-resilient-scaling/SKILL.md.\",\n \"Classifies the request as Azure AKS infrastructure setup for a Physical AI workload.\",\n \"Loads only the Azure access, Azure cluster, Azure OSMO, NIM Operator, and OSMO CLI component references needed for the target.\",\n \"Runs selected preflight scripts before provisioning or workflow submission.\",\n \"Reports Kubernetes, OSMO, inference, and workload readiness gates separately.\"\n ]\n },\n {\n \"id\": \"infra-setup-osmo-logs-negative\",\n \"question\": \"Summarize recent OSMO workflow logs for workflow abc123 and tell me which task failed.\",\n \"expected_skill\": null,\n \"expected_script\": null,\n \"ground_truth\": \"This is an OSMO workflow log analysis request, not an infrastructure setup, scaling, validation, or recovery request.\",\n \"expected_behavior\": [\n \"Does not select physical-ai-infrastructure-setup-and-resilient-scaling unless the prompt also asks to set up, scale, validate, or recover infrastructure.\",\n \"Does not run cluster provisioning, Terraform, Helm install, or NIM Operator setup.\",\n \"Routes to an OSMO log or workflow troubleshooting reference if available.\"\n ]\n },\n {\n \"id\": \"infra-setup-missing-azure-auth-blocked\",\n \"question\": \"Provision the Azure AKS stack for Physical AI, but no Azure login, subscription, or quota details are available.\",\n \"expected_skill\": \"physical-ai-infrastructure-setup-and-resilient-scaling\",\n \"expected_script\": null,\n \"ground_truth\": \"The agent should select the infrastructure setup skill, identify missing Azure prerequisites during preflight, and ask for the missing authorization or subscription details instead of attempting provisioning.\",\n \"expected_behavior\": [\n \"Reads skills/physical-ai-infrastructure-setup-and-resilient-scaling/SKILL.md.\",\n \"Loads the Azure access and Azure cluster component references before provisioning.\",\n \"Runs or plans Azure preflight checks before Terraform apply.\",\n \"Reports missing Azure authentication, subscription, or quota data as blockers.\",\n \"Does not claim an AKS cluster or OSMO deployment was created.\"\n ]\n }\n ]\n}\n","content_type":"application/json; charset=utf-8","language":"json","size":2949,"content_sha256":"517a6896ce659ac6fca325c8e6440de71e93fe6e0be72f6d630a04aac214cc30"},{"filename":"skill-card.md","content":"## Description: \u003cbr>\nUse when the user wants to set up, scale, validate, or harden NVIDIA physical AI infrastructure for synthetic data generation workflows across local MicroK8s or Azure AKS, including Kubernetes clusters, inference endpoint deployment, OSMO deployment, workload submission readiness, and infrastructure failure recovery. \u003cbr>\n\nThis skill is ready for commercial/non-commercial use. \u003cbr>\n\n## Owner\nNVIDIA \u003cbr>\n\n### License/Terms of Use: \u003cbr>\nApache-2.0 \u003cbr>\n## Use Case: \u003cbr>\nDevelopers and infrastructure engineers composing cluster, inference, OSMO, and workload stages into a reproducible Physical AI synthetic data generation environment on MicroK8s or Azure AKS, then keeping the environment observable and recoverable. \u003cbr>\n\n### Deployment Geography for Use: \u003cbr>\nGlobal \u003cbr>\n\n## Known Risks and Mitigations: \u003cbr>\nRisk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. \u003cbr>\nMitigation: Review and scan skill before deployment. \u003cbr>\n\n## Reference(s): \u003cbr>\n- [Cluster Azure Reference](components/cluster-azure/reference.md) \u003cbr>\n- [Cluster MicroK8s Reference](components/cluster-microk8s/reference.md) \u003cbr>\n- [NIM Operator Inference Reference](components/inference-nim-operator/reference.md) \u003cbr>\n- [NVCF Inference Reference](components/inference-nvcf/reference.md) \u003cbr>\n- [Azure AI Foundry Inference Reference](components/inference-azure/reference.md) \u003cbr>\n- [OSMO Azure Reference](components/osmo-azure/reference.md) \u003cbr>\n- [OSMO Kubernetes Reference](components/osmo-k8s/reference.md) \u003cbr>\n- [OSMO CLI Reference](components/osmo-cli/reference.md) \u003cbr>\n\n\n## Skill Output: \u003cbr>\n**Output Type(s):** [Shell commands, Configuration instructions, Analysis] \u003cbr>\n**Output Format:** [Markdown with inline bash code blocks] \u003cbr>\n**Output Parameters:** [1D] \u003cbr>\n**Other Properties Related to Output:** [None] \u003cbr>\n\n## Evaluation Tasks: \u003cbr>\nNVSkills-Eval 3-Tier Evaluation with external profile. Tier 1 static validation ran 9 checks (13 findings). Tier 2 deduplication ran 2 checks (16 findings). Tier 3 live agent evaluation was not available. \u003cbr>\n\n## Evaluation Metrics Used: \u003cbr>\nReported benchmark dimensions: \u003cbr>\n- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. \u003cbr>\n- Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. \u003cbr>\n- Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. \u003cbr>\n- Effectiveness: Checks whether the agent performs measurably better with the skill than without it. \u003cbr>\n- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. \u003cbr>\n\n\n\n## Skill Version(s): \u003cbr>\n1.0.0 (source: frontmatter) \u003cbr>\n\n## Ethical Considerations: \u003cbr>\nNVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. \u003cbr>\n\n(For Release on NVIDIA Platforms Only) \u003cbr>\nPlease report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail). \u003cbr>\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3453,"content_sha256":"4c55580838d6214e3a493dc0574bd935f25a21d58d91419efbb2396c81d14631"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"Physical AI Infrastructure Setup And Resilient Scaling","type":"text"}]},{"type":"paragraph","content":[{"text":"Canonical skill for the Physical AI infrastructure stack. Use it to compose cluster, inference, OSMO, and workload stages into a reproducible Physical AI SDG environment, then keep the environment observable and recoverable.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Operating Rules","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Read only the component references needed for the selected target. Do not load every component by default.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Keep the repo as the durable artifact. Fix checked-in config or scripts, then rerun. Do not recover a failed install with untracked one-off changes.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Run mutating cluster, OSMO, Helm, Terraform, or Azure operations through checked-in scripts when a script exists. Read-only diagnostics are allowed.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Stop at the first red gate. Fix the lowest owning layer in this order: config, script, then skill guidance.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Derive values from the environment when possible. Ask only for values that cannot be inferred, such as API keys, target choice, or quota tradeoffs.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Store secrets in ","type":"text"},{"text":"${REPO_ROOT}/.env","type":"text","marks":[{"type":"code_inline"}]},{"text":". Cluster-derived values such as storage, database, Redis, and endpoint names come from Terraform outputs or platform queries, not ","type":"text"},{"text":".env","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Preflight means no deployed state: no cluster API, Terraform outputs, Helm releases, OSMO pools, or workflow state. Those belong to deploy/verify gates.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Never print, echo, or paste raw keys into commands, YAML, logs, or transcripts. Prefer credential handles, Kubernetes ","type":"text"},{"text":"secretKeyRef","type":"text","marks":[{"type":"code_inline"}]},{"text":", and runtime-only secret injection. Scan raw transcript exports with ","type":"text"},{"text":"scripts/scan_transcript_secrets.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" before sharing.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Use absolute paths. Derive repo root with ","type":"text"},{"text":"git rev-parse --show-toplevel","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Component References","type":"text"}]},{"type":"paragraph","content":[{"text":"Each component lives inside this skill so the stack has one canonical trigger. Load the component reference only when the selected target needs that slice.","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Concern","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Load","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Assets","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Stage matrix and old driver notes","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/driver/reference.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"None","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"MicroK8s cluster","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/cluster-microk8s/reference.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/cluster-microk8s/scripts/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"components/cluster-microk8s/runtimeclass-nvidia-runc.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Azure AKS cluster","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/cluster-azure/reference.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/cluster-azure/scripts/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"components/cluster-azure/terraform/","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"NIM Operator inference","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/inference-nim-operator/reference.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/inference-nim-operator/scripts/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"components/inference-nim-operator/nims/","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"NVCF inference","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/inference-nvcf/reference.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/inference-nvcf/scripts/","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Azure AI Foundry inference","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/inference-azure/reference.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/inference-azure/scripts/","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"MicroK8s OSMO","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/osmo-k8s/reference.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/osmo-k8s/scripts/","type":"text","marks":[{"type":"code_inline"}]},{"text":", upstream OSMO deploy scripts","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Azure OSMO","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/osmo-azure/reference.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/osmo-azure/scripts/","type":"text","marks":[{"type":"code_inline"}]},{"text":", upstream OSMO deploy scripts plus Azure TF outputs","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Azure access setup","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/azure-access/reference.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"None","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"OSMO CLI and workflow operations","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/osmo-cli/reference.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/osmo-cli/scripts/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"components/osmo-cli/references/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"components/osmo-cli/agents/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"components/osmo-cli/tests/","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"OpenClaw Azure device login","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/openclaw-azure-login/reference.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"None","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"OSMO CLI Support Files","type":"text"}]},{"type":"paragraph","content":[{"text":"The OSMO CLI component has second-level support files because its command and workflow surface is large. Load these directly only for the stated case.","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"File","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Read when","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/osmo-cli/agents/workflow-expert.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Spawning a workflow-generation or workflow-failure subagent.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/osmo-cli/agents/logs-reader.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Spawning a log summarization subagent for OSMO workflow failures.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/osmo-cli/references/cli-commands.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Exact OSMO CLI flags, payloads, or command syntax are needed.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/osmo-cli/references/workflow-spec.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Workflow YAML schema, credentials, outputs, or provider fields are needed.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/osmo-cli/references/workflow-patterns.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Multi-task, data dependency, Jinja, serial, or parallel workflow design is needed.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/osmo-cli/references/advanced-patterns.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Checkpointing, retry/exit behavior, or node exclusion is needed.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"components/osmo-cli/tests/orchestrator-runtime-failure.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Validating or debugging the OSMO orchestration review pattern.","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Target Selection","type":"text"}]},{"type":"paragraph","content":[{"text":"Pick exactly one option per stage. Stage 2 follows stage 1.","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Kubernetes: ","type":"text"},{"text":"MicroK8s","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"Azure","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"OSMO: ","type":"text"},{"text":"MicroK8s OSMO","type":"text","marks":[{"type":"code_inline"}]},{"text":" when Kubernetes is MicroK8s, ","type":"text"},{"text":"Azure OSMO","type":"text","marks":[{"type":"code_inline"}]},{"text":" when Kubernetes is Azure","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Inference: ","type":"text"},{"text":"NIM Operator","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"NVCF","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"Azure AI Foundry","type":"text","marks":[{"type":"code_inline"}]},{"text":", or ","type":"text"},{"text":"None","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Workload: Video Data Augmentation, Defect Image Generation, NuRec Carline Adaptation, NRE, NCore, Asset Harvester, or custom workflow YAML","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Reject invalid combinations before provisioning:","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Cluster","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"NIM Operator","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"NVCF","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Azure AI Foundry","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"MicroK8s","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"yes","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"yes","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"no, Foundry requires Azure identities","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Azure","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"yes","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"yes","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"yes","type":"text"}]}]}]}]},{"type":"paragraph","content":[{"text":"For OpenClaw or any chat-only environment that cannot open a browser, read ","type":"text"},{"text":"components/openclaw-azure-login/reference.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" before Azure prerequisites. For any Azure target, read ","type":"text"},{"text":"components/azure-access/reference.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" before Azure component preflights.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Setup Flow","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Confirm target choices and workload compute requirements.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Load the selected component references.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Resolve prerequisites up front, including API keys, Azure access, caller CIDR, GPU quota, storage class, and OSMO login requirements.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Run ","type":"text"},{"text":"scripts/preflight.sh","type":"text","marks":[{"type":"code_inline"}]},{"text":" for every selected infrastructure component plus any OSMO CLI/workload preflight before provisioning; build the implementation plan from the results and stop on red preflight.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Deploy Kubernetes first. Nothing else starts until the cluster gate is green.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Deploy OSMO and inference after Kubernetes. These can proceed in parallel once the cluster exists, but workload submission waits for both selected gates.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Submit the workload only after OSMO, storage credentials, compute pool, and selected inference endpoints are verified. For VDA, this includes ","type":"text"},{"text":"preflight_credentials.sh","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"pre_submit_guard.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" with resolved ","type":"text"},{"text":"--set","type":"text","marks":[{"type":"code_inline"}]},{"text":" values, non-empty model-cache prefixes, and workflow-namespace endpoint smoke checks.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Monitor through completion. On failed workflow state, inspect events and logs from ","type":"text"},{"text":"components/osmo-cli/reference.md","type":"text","marks":[{"type":"code_inline"}]},{"text":"; do not resubmit blindly.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Inference Discovery","type":"text"}]},{"type":"paragraph","content":[{"text":"Avoid over-deploying expensive endpoints.","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Scan the chosen workflow spec and default values for endpoint references: ","type":"text"},{"text":"*.osmo-nims.svc.cluster.local","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"api.nvcf.nvidia.com/*","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"*.inference.ai.azure.com","type":"text","marks":[{"type":"code_inline"}]},{"text":", or ","type":"text"},{"text":"*.cognitiveservices.azure.com","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Map each reference to the selected backend:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NIM Operator: service name must match a directory under ","type":"text"},{"text":"components/inference-nim-operator/nims/","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NVCF: function URL or function ID must be supplied by the environment.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Azure AI Foundry: endpoint name must be deployed through ","type":"text"},{"text":"components/inference-azure/scripts/install.sh","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"If the workflow needs a capability the selected backend lacks, stop and report the mismatch. Do not silently substitute another model.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Verification Gates","type":"text"}]},{"type":"paragraph","content":[{"text":"Each stage has its own Verify section in the component reference. These gates are mandatory:","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Stage","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Gate","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Kubernetes","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Cluster API reachable, nodes Ready, GPU capacity advertised for GPU paths, and CPU+NVCF paths have ","type":"text"},{"text":"runtimeclass/nvidia","type":"text","marks":[{"type":"code_inline"}]},{"text":" mapped to ","type":"text"},{"text":"runc","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Inference","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Every endpoint referenced by the workload is reachable. NIM readiness uses ","type":"text"},{"text":"/v1/health/ready","type":"text","marks":[{"type":"code_inline"}]},{"text":"; NVCF and Foundry still need task-specific authenticated checks.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"OSMO","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"OSMO pods Ready, pool ONLINE, port-forward watchdogs alive, storage credentials configured, and verify-hello workflow COMPLETED.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Workload","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Selected workload pre-submit guards pass before submit. ","type":"text"},{"text":"osmo workflow query \u003cid>","type":"text","marks":[{"type":"code_inline"}]},{"text":" reports ","type":"text"},{"text":"COMPLETED","type":"text","marks":[{"type":"code_inline"}]},{"text":" and every task is green. Failed terminal states require events and logs before retry.","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Resilient Scaling","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Size the cluster from workload needs before provisioning. For Azure, check CPU and GPU quota for the selected VM families before ","type":"text"},{"text":"terraform apply","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For NIM Operator, deploy only the NIMServices referenced by the workload. Each service pins GPU and model-cache storage for the lifetime of the cluster.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Keep OSMO storage URL schemes aligned with the active backend. Local MicroK8s uses MinIO, Azure uses Blob-backed configuration.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Treat Pending, Unknown, ImagePullBackOff, unbound PVCs, or 0 Ready replicas as layer failures. Investigate scheduling, storage, image credentials, and adjacent platform state before retrying the same command.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For long deploys or workflow watches, provide heartbeat updates with current state, elapsed time, last useful observation, and next check.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Workload Routing","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Video Data Augmentation: use ","type":"text"},{"text":"skills/physical-ai-video-data-augmentation/SKILL.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Defect Image Generation: use ","type":"text"},{"text":"skills/physical-ai-defect-image-generation/SKILL.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NuRec carline adaptation: use ","type":"text"},{"text":"skills/carline-adaptation/SKILL.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NRE, NCore, and Asset Harvester live in the canonical NuRec catalog listed in ","type":"text"},{"text":"skills/INDEX.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Custom workload: submit the provided workflow YAML through OSMO after checking resource requests, image credentials, data credentials, and inference URLs.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Evaluation Prompts And Results","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Positive trigger: \"Set up resilient Physical AI infrastructure for VDA on Azure AKS with NIM Operator.\" Expected: use this skill.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Negative trigger: \"Summarize recent OSMO workflow logs for this workflow ID.\" Expected: do not use this infrastructure setup skill unless the request also involves setup, scaling, validation, or recovery of the infrastructure stack.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Latest static review: 2026-05-26, description keywords match the expected routes above.","type":"text"}]}]},"metadata":{"date":"2026-11-02","name":"physical-ai-infrastructure-setup-and-resilient-scaling","tags":["physical-ai","infrastructure","kubernetes","azure","microk8s","osmo","nim-operator","scaling"],"tools":["Read","Shell"],"author":"@skillopedia","source":{"stars":3825,"repo_name":"skills","origin_url":"https://github.com/NVIDIA/skills/blob/HEAD/skills/physical-ai-infrastructure-setup-and-resilient-scaling/SKILL.md","repo_owner":"NVIDIA","body_sha256":"ee96598544cb3e7399c0e12c245e94104aea847bcb7d647a5b25bcb2320727ba","cluster_key":"ab9b1dfc9e72e7e8e1de96420766b253e54e2891e16596fa131ecdeaa44dfded","clean_bundle":{"format":"clean-skill-bundle-v1","source":"NVIDIA/skills/skills/physical-ai-infrastructure-setup-and-resilient-scaling/SKILL.md","text_bytes":250144,"source_commit":"25340619336e89bcc40365c28bfe9e790430468e","skills_sh_rank":8365,"attachment_count":55,"text_attachments":55,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"skills/physical-ai-infrastructure-setup-and-resilient-scaling/SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"1.0.0","category":"devops-infrastructure","category_label":"DevOps"},"exact_dupes_collapsed_into_this":0},"license":"Apache-2.0","version":"1.0.0","category":"devops-infrastructure","metadata":{"tags":["physical-ai","infrastructure","kubernetes","azure","microk8s","osmo","nim-operator","scaling"],"author":"NVIDIA Physical AI","domain":"ai-ml","languages":["bash","hcl","yaml"]},"import_tag":"skills-sh-top20k-2026-11","description":"Use when the user wants to set up, scale, validate, or harden NVIDIA physical AI infrastructure for synthetic data generation workflows across local MicroK8s or Azure AKS, including Kubernetes clusters, inference endpoint deployment, OSMO deployment, workload submission readiness, and infrastructure failure recovery. Trigger keywords: physical ai infrastructure, resilient scaling, SDG infrastructure, microk8s, azure aks, NVCF deployment, NIM Operator, OSMO deploy, workflow scaling. Don't trigger for: OSMO log summarization or workload-only operations unless infrastructure setup, scaling, validation, or recovery is requested.","compatibility":"Requires the selected component prerequisites, usually kubectl plus either MicroK8s or Azure CLI/Terraform, and OSMO or inference credentials for the chosen target."}},"renderedAt":1793666782814}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.