tao-setup-nvidia-gpu-host

NVIDIA GPU Host Setup Standalone install? If this session was not initialized by the TAO skill bank plugin, run the skill first (host preflight, credentials, cross-skill discovery). Use this setup skill before TAO workflows run on the , , or backend. The TAO-wide default minimums are: - NVIDIA driver (open kernel module preferred) - CUDA Toolkit - NVIDIA Container Toolkit - Docker engine — only installed for / backends and only when Docker is missing. The package picked depends on the distro family ( on Debian-family by default, / from on RHEL-family, on SUSE-family). Pass to opt out. The che…

| head -n 1 || true)\"\n [[ -n \"$gpu\" && \"$gpu\" != \"0\" ]]\n}\n\nprint_status() {\n local detected_driver detected_cuda detected_toolkit\n detected_driver=\"$(driver_version || true)\"\n detected_cuda=\"$(cuda_version || true)\"\n detected_toolkit=\"$(container_toolkit_version || true)\"\n\n if driver_ok; then\n echo \"OK: NVIDIA driver ${detected_driver} >= ${MIN_DRIVER_VERSION}\"\n else\n echo \"MISSING: visible GPU with NVIDIA driver >= ${MIN_DRIVER_VERSION} (detected: ${detected_driver:-none})\"\n fi\n\n if cuda_ok; then\n echo \"OK: CUDA Toolkit ${detected_cuda} >= ${MIN_CUDA_VERSION}\"\n else\n echo \"MISSING: CUDA Toolkit >= ${MIN_CUDA_VERSION} (detected: ${detected_cuda:-none})\"\n fi\n\n if container_toolkit_ok; then\n echo \"OK: NVIDIA Container Toolkit ${detected_toolkit} >= ${MIN_CONTAINER_TOOLKIT_VERSION}\"\n else\n echo \"MISSING: NVIDIA Container Toolkit >= ${MIN_CONTAINER_TOOLKIT_VERSION} (detected: ${detected_toolkit:-none})\"\n fi\n\n if [[ \"$BACKEND\" == \"docker\" || \"$BACKEND\" == \"local-docker\" ]]; then\n if ! docker_installed_ok; then\n echo \"MISSING: Docker is not installed.\"\n case \"$PKG_FAMILY\" in\n debian)\n echo \" Rerun with --install (not --skip-docker-install) to install\"\n echo \" '${DOCKER_PACKAGE_DEBIAN}' via apt and finish the NVIDIA runtime wiring.\"\n ;;\n rhel)\n echo \" Rerun with --install (not --skip-docker-install) to install\"\n echo \" docker-ce / moby-engine via ${PKG_MANAGER} and finish the NVIDIA runtime wiring.\"\n ;;\n suse)\n echo \" Rerun with --install (not --skip-docker-install) to install\"\n echo \" 'docker' via zypper and finish the NVIDIA runtime wiring.\"\n ;;\n arch)\n echo \" Install Docker manually for Arch-family hosts:\"\n echo \" sudo pacman -S docker && sudo systemctl enable --now docker\"\n echo \" Then rerun this script to wire the NVIDIA Container Toolkit runtime.\"\n ;;\n *)\n echo \" Install Docker for your distribution (see\"\n echo \" https://docs.docker.com/engine/install/), then rerun this script.\"\n ;;\n esac\n elif docker_runtime_ok; then\n echo \"OK: Docker NVIDIA runtime configured\"\n else\n echo \"MISSING: Docker NVIDIA runtime not configured or Docker unreachable\"\n fi\n fi\n\n if [[ \"$BACKEND\" == \"kubernetes\" || \"$BACKEND\" == \"k8s\" ]]; then\n if kubernetes_gpu_ok; then\n echo \"OK: Kubernetes reports nvidia.com/gpu allocatable\"\n else\n local rc=$?\n if [[ \"$rc\" -eq 2 ]]; then\n echo \"WARN: kubectl not found; cannot check cluster GPU capacity\"\n else\n echo \"WARN: Kubernetes does not report nvidia.com/gpu allocatable\"\n fi\n fi\n fi\n\n if [[ -n \"$PKG_FAMILY\" && \"$PKG_FAMILY\" != \"unknown\" ]]; then\n echo \"INFO: detected ${DISTRO_PRETTY} (family=${PKG_FAMILY}, manager=${PKG_MANAGER:-n/a})\"\n elif [[ -n \"$DISTRO_PRETTY\" ]]; then\n echo \"INFO: detected ${DISTRO_PRETTY} (family=unknown — --install will print manual steps)\"\n fi\n}\n\nruntime_ok() {\n driver_ok && cuda_ok && container_toolkit_ok || return 1\n if [[ \"$BACKEND\" == \"docker\" || \"$BACKEND\" == \"local-docker\" ]]; then\n docker_runtime_ok\n return $?\n fi\n return 0\n}\n\nunsupported_install_family() {\n cat >&2 \u003c\u003cEOF\nERROR: --install does not yet automate this distribution.\n Detected: ${DISTRO_PRETTY:-unknown} (family=${PKG_FAMILY:-unknown})\n\nInstall these manually using your distribution's package manager, then rerun\nthis script with --check-only to verify:\n\n - NVIDIA driver >= ${MIN_DRIVER_VERSION} (open kernel module preferred)\n - CUDA Toolkit >= ${MIN_CUDA_VERSION} (NVIDIA package: ${CUDA_PACKAGE})\n - NVIDIA Container Toolkit >= ${MIN_CONTAINER_TOOLKIT_VERSION}\n - Docker engine (any flavor)\n\nNVIDIA documentation:\n - CUDA install guide (all distros):\n https://docs.nvidia.com/cuda/cuda-installation-guide-linux/\n - NVIDIA Container Toolkit install guide:\n https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html\n - Docker engine install guide (per distro):\n https://docs.docker.com/engine/install/\n\nThe selected model's runtime requirements take precedence over the TAO-wide\ndefaults. Rerun with the same --min-*-version flags after manual installation.\nEOF\n exit 1\n}\n\nconfirm_install() {\n if [[ \"$YES\" -eq 1 ]]; then\n return 0\n fi\n\n local driver_line cuda_line nct_line docker_line=\"\"\n case \"$PKG_FAMILY\" in\n debian)\n driver_line=\"${DRIVER_PACKAGE_DEBIAN} (must provide driver >= ${MIN_DRIVER_VERSION})\"\n cuda_line=\"${CUDA_PACKAGE}\"\n nct_line=\"current nvidia-container-toolkit packages (must be >= ${MIN_CONTAINER_TOOLKIT_VERSION})\"\n if [[ ( \"$BACKEND\" == \"docker\" || \"$BACKEND\" == \"local-docker\" ) \\\n && \"$INSTALL_DOCKER\" -eq 1 ]] && ! have docker; then\n docker_line=\"\n - ${DOCKER_PACKAGE_DEBIAN} (Docker engine, distribution apt repo)\"\n fi\n ;;\n rhel)\n driver_line=\"${DRIVER_PACKAGE_RHEL} + ${DRIVER_KMOD_RHEL} (must provide driver >= ${MIN_DRIVER_VERSION})\"\n cuda_line=\"${CUDA_PACKAGE}\"\n nct_line=\"current nvidia-container-toolkit packages (must be >= ${MIN_CONTAINER_TOOLKIT_VERSION})\"\n if [[ ( \"$BACKEND\" == \"docker\" || \"$BACKEND\" == \"local-docker\" ) \\\n && \"$INSTALL_DOCKER\" -eq 1 ]] && ! have docker; then\n case \"$DISTRO_ID\" in\n fedora) docker_line=\"\n - moby-engine + moby-cli (Fedora) — falls back to docker-ce from download.docker.com\" ;;\n *) docker_line=\"\n - docker-ce docker-ce-cli containerd.io (from download.docker.com)\" ;;\n esac\n fi\n ;;\n suse)\n driver_line=\"${DRIVER_PACKAGE_SUSE} (must provide driver >= ${MIN_DRIVER_VERSION})\"\n cuda_line=\"${CUDA_PACKAGE}\"\n nct_line=\"current nvidia-container-toolkit packages (must be >= ${MIN_CONTAINER_TOOLKIT_VERSION})\"\n if [[ ( \"$BACKEND\" == \"docker\" || \"$BACKEND\" == \"local-docker\" ) \\\n && \"$INSTALL_DOCKER\" -eq 1 ]] && ! have docker; then\n docker_line=\"\n - docker (zypper)\"\n fi\n ;;\n *)\n unsupported_install_family\n ;;\n esac\n\n cat \u003c\u003cEOF\nDetected: ${DISTRO_PRETTY:-unknown} — family=${PKG_FAMILY}, manager=${PKG_MANAGER}\n\nThis will install or repair:\n - ${driver_line}\n - ${cuda_line}\n - ${nct_line}${docker_line}\n\nIt will add NVIDIA's CUDA + Container Toolkit repositories if missing, and\nmay restart Docker. If your invoking user is not already in the 'docker'\ngroup, it will be added (log out / 'newgrp docker' for that to take effect).\nEOF\n read -r -p \"Continue? [y/N] \" answer\n case \"$answer\" in\n y|Y|yes|YES) ;;\n *) echo \"Aborted.\"; exit 1 ;;\n esac\n}\n\ninstall_prereqs() {\n case \"$PKG_FAMILY\" in\n debian)\n export DEBIAN_FRONTEND=noninteractive\n \"${SUDO[@]}\" apt-get update\n \"${SUDO[@]}\" apt-get install -y --no-install-recommends ca-certificates curl gnupg\n ;;\n rhel)\n \"${SUDO[@]}\" \"$PKG_MANAGER\" -y install ca-certificates curl\n # dnf-plugins-core is required for `dnf config-manager --add-repo` on\n # RHEL/Rocky/Alma; Fedora ships it by default. Silently best-effort.\n \"${SUDO[@]}\" \"$PKG_MANAGER\" -y install dnf-plugins-core >/dev/null 2>&1 \\\n || \"${SUDO[@]}\" \"$PKG_MANAGER\" -y install yum-utils >/dev/null 2>&1 \\\n || true\n ;;\n suse)\n \"${SUDO[@]}\" zypper --non-interactive refresh\n \"${SUDO[@]}\" zypper --non-interactive install ca-certificates curl gpg2\n ;;\n *)\n unsupported_install_family\n ;;\n esac\n}\n\ninstall_cuda_repo() {\n [[ -n \"$CUDA_REPO_DISTRO\" && -n \"$CUDA_REPO_ARCH\" ]] || {\n echo \"ERROR: cannot map ${DISTRO_PRETTY} to an NVIDIA CUDA repo path.\" >&2\n unsupported_install_family\n }\n\n case \"$PKG_FAMILY\" in\n debian)\n if dpkg-query -W cuda-keyring >/dev/null 2>&1; then\n return 0\n fi\n local deb\n deb=\"$(mktemp)\"\n curl -fsSL \\\n \"https://developer.download.nvidia.com/compute/cuda/repos/${CUDA_REPO_DISTRO}/${CUDA_REPO_ARCH}/cuda-keyring_1.1-1_all.deb\" \\\n --output \"$deb\"\n \"${SUDO[@]}\" dpkg -i \"$deb\"\n rm -f \"$deb\"\n ;;\n rhel)\n local repo_file=\"/etc/yum.repos.d/cuda-${CUDA_REPO_DISTRO}.repo\"\n if [[ -f \"$repo_file\" ]]; then\n return 0\n fi\n local repo_url=\"https://developer.download.nvidia.com/compute/cuda/repos/${CUDA_REPO_DISTRO}/${CUDA_REPO_ARCH}/cuda-${CUDA_REPO_DISTRO}.repo\"\n if \"${SUDO[@]}\" \"$PKG_MANAGER\" config-manager --add-repo \"$repo_url\" >/dev/null 2>&1; then\n :\n else\n # Fallback: drop the .repo file directly if config-manager is unavailable.\n \"${SUDO[@]}\" curl -fsSL \"$repo_url\" -o \"$repo_file\"\n fi\n \"${SUDO[@]}\" \"$PKG_MANAGER\" clean expire-cache >/dev/null 2>&1 || true\n ;;\n suse)\n local repo_url=\"https://developer.download.nvidia.com/compute/cuda/repos/${CUDA_REPO_DISTRO}/${CUDA_REPO_ARCH}/cuda-${CUDA_REPO_DISTRO}.repo\"\n if zypper lr --uri 2>/dev/null | grep -q \"$repo_url\"; then\n return 0\n fi\n \"${SUDO[@]}\" zypper --non-interactive addrepo --gpgcheck-strict \"$repo_url\"\n \"${SUDO[@]}\" zypper --non-interactive --gpg-auto-import-keys refresh\n ;;\n *)\n unsupported_install_family\n ;;\n esac\n}\n\ninstall_container_repo() {\n case \"$PKG_FAMILY\" in\n debian)\n local key_tmp keyring_tmp list_tmp\n key_tmp=\"$(mktemp)\"\n keyring_tmp=\"$(mktemp)\"\n list_tmp=\"$(mktemp)\"\n\n curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey --output \"$key_tmp\"\n gpg --dearmor --yes --output \"$keyring_tmp\" \"$key_tmp\"\n \"${SUDO[@]}\" install -m 0644 \"$keyring_tmp\" /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg\n\n curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list --output \"$list_tmp\"\n sed -i 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \"$list_tmp\"\n \"${SUDO[@]}\" install -m 0644 \"$list_tmp\" /etc/apt/sources.list.d/nvidia-container-toolkit.list\n\n rm -f \"$key_tmp\" \"$keyring_tmp\" \"$list_tmp\"\n ;;\n rhel)\n local repo_file=\"/etc/yum.repos.d/nvidia-container-toolkit.repo\"\n [[ -f \"$repo_file\" ]] && return 0\n \"${SUDO[@]}\" curl -fsSL https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo \\\n -o \"$repo_file\"\n ;;\n suse)\n local repo_url=\"https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo\"\n if zypper lr --uri 2>/dev/null | grep -q \"$repo_url\"; then\n return 0\n fi\n \"${SUDO[@]}\" zypper --non-interactive addrepo --gpgcheck-strict \"$repo_url\"\n \"${SUDO[@]}\" zypper --non-interactive --gpg-auto-import-keys refresh\n ;;\n *)\n unsupported_install_family\n ;;\n esac\n}\n\ninstall_runtime_packages() {\n case \"$PKG_FAMILY\" in\n debian)\n export DEBIAN_FRONTEND=noninteractive\n local kernel_headers\n kernel_headers=\"linux-headers-$(uname -r)\"\n \"${SUDO[@]}\" apt-get update\n \"${SUDO[@]}\" apt-get install -y --allow-downgrades \\\n \"$kernel_headers\" \\\n \"$DRIVER_PACKAGE_DEBIAN\" \\\n \"$CUDA_PACKAGE\" \\\n nvidia-container-toolkit \\\n nvidia-container-toolkit-base \\\n libnvidia-container-tools \\\n libnvidia-container1\n ;;\n rhel)\n # Kernel headers/devel package names match the running kernel.\n \"${SUDO[@]}\" \"$PKG_MANAGER\" -y install \\\n \"kernel-devel-$(uname -r)\" \\\n \"kernel-headers-$(uname -r)\" || true\n \"${SUDO[@]}\" \"$PKG_MANAGER\" -y install --allowerasing \\\n \"$DRIVER_PACKAGE_RHEL\" \\\n \"$DRIVER_KMOD_RHEL\" \\\n \"$CUDA_PACKAGE\" \\\n nvidia-container-toolkit \\\n nvidia-container-toolkit-base \\\n libnvidia-container-tools \\\n libnvidia-container1\n ;;\n suse)\n \"${SUDO[@]}\" zypper --non-interactive install --allow-downgrade \\\n \"kernel-default-devel\" \\\n \"$DRIVER_PACKAGE_SUSE\" \\\n \"$CUDA_PACKAGE\" \\\n nvidia-container-toolkit \\\n nvidia-container-toolkit-base \\\n libnvidia-container-tools \\\n libnvidia-container1\n ;;\n *)\n unsupported_install_family\n ;;\n esac\n}\n\ninstall_docker_package() {\n [[ \"$INSTALL_DOCKER\" -eq 1 ]] || return 0\n [[ \"$BACKEND\" == \"docker\" || \"$BACKEND\" == \"local-docker\" ]] || return 0\n have docker && return 0\n\n case \"$PKG_FAMILY\" in\n debian)\n echo \"Installing Docker package '${DOCKER_PACKAGE_DEBIAN}' (apt)...\"\n export DEBIAN_FRONTEND=noninteractive\n \"${SUDO[@]}\" apt-get install -y --no-install-recommends \"$DOCKER_PACKAGE_DEBIAN\"\n ;;\n rhel)\n if [[ \"$DISTRO_ID\" == \"fedora\" ]] \\\n && \"${SUDO[@]}\" \"$PKG_MANAGER\" -y install moby-engine moby-cli 2>/dev/null; then\n echo \"Installed Fedora's moby-engine + moby-cli.\"\n else\n echo \"Installing docker-ce from download.docker.com ...\"\n local docker_repo\n case \"$DISTRO_ID\" in\n fedora) docker_repo=\"https://download.docker.com/linux/fedora/docker-ce.repo\" ;;\n *) docker_repo=\"https://download.docker.com/linux/centos/docker-ce.repo\" ;;\n esac\n if ! \"${SUDO[@]}\" \"$PKG_MANAGER\" config-manager --add-repo \"$docker_repo\" >/dev/null 2>&1; then\n \"${SUDO[@]}\" curl -fsSL \"$docker_repo\" -o /etc/yum.repos.d/docker-ce.repo\n fi\n \"${SUDO[@]}\" \"$PKG_MANAGER\" -y install docker-ce docker-ce-cli containerd.io\n fi\n ;;\n suse)\n echo \"Installing Docker via zypper...\"\n \"${SUDO[@]}\" zypper --non-interactive install docker\n ;;\n *)\n echo \"WARN: --install cannot auto-install Docker on family '${PKG_FAMILY}'.\"\n echo \" Install Docker manually per https://docs.docker.com/engine/install/ ,\"\n echo \" then rerun this script to finish wiring the NVIDIA runtime.\"\n return 0\n ;;\n esac\n\n if have systemctl; then\n \"${SUDO[@]}\" systemctl enable --now docker || {\n echo \"WARN: could not enable/start docker via systemctl; start it manually.\"\n }\n fi\n}\n\nadd_invoker_to_docker_group() {\n [[ \"$ADD_USER_TO_DOCKER_GROUP\" -eq 1 ]] || return 0\n [[ \"$BACKEND\" == \"docker\" || \"$BACKEND\" == \"local-docker\" ]] || return 0\n have docker || return 0\n\n # Resolve the user that invoked us:\n # - via sudo: EUID=0, SUDO_USER=\u003coriginal> → use SUDO_USER\n # - as non-root: EUID!=0, SUDO_USER unset → use $USER\n # - as raw root: EUID=0, SUDO_USER unset → $USER=root → skip\n local target_user=\"${SUDO_USER:-$USER}\"\n [[ -n \"$target_user\" && \"$target_user\" != \"root\" ]] || return 0\n\n if ! getent group docker >/dev/null 2>&1; then\n \"${SUDO[@]}\" groupadd docker || return 0\n fi\n\n if id -nG \"$target_user\" 2>/dev/null | tr ' ' '\\n' | grep -qx docker; then\n return 0\n fi\n\n \"${SUDO[@]}\" usermod -aG docker \"$target_user\" || return 0\n echo \"NOTE: Added '${target_user}' to the 'docker' group. The new membership\"\n echo \" does NOT take effect in this shell. To use docker without sudo,\"\n echo \" log out and back in, or run 'newgrp docker' in each new shell.\"\n}\n\nconfigure_docker_runtime() {\n [[ \"$CONFIGURE_DOCKER\" -eq 1 ]] || return 0\n [[ \"$BACKEND\" == \"docker\" || \"$BACKEND\" == \"local-docker\" ]] || return 0\n\n if ! have docker; then\n if [[ \"$INSTALL_DOCKER\" -eq 0 ]]; then\n echo \"WARN: Docker is not installed and --skip-docker-install was passed;\"\n echo \" install Docker manually, then rerun this script to wire the\"\n echo \" NVIDIA Container Toolkit runtime into /etc/docker/daemon.json.\"\n else\n echo \"WARN: Docker is still not installed; skipping NVIDIA runtime configuration.\"\n fi\n return 0\n fi\n\n \"${SUDO[@]}\" nvidia-ctk runtime configure --runtime=docker\n if have systemctl; then\n \"${SUDO[@]}\" systemctl restart docker || {\n echo \"WARN: could not restart Docker; restart it manually before running GPU containers.\"\n }\n else\n echo \"WARN: systemctl not found; restart Docker manually before running GPU containers.\"\n fi\n}\n\ndetect_distro\n\nif [[ \"$INSTALL\" -eq 0 ]]; then\n print_status\n if runtime_ok; then\n exit 0\n fi\n echo\n case \"$PKG_FAMILY\" in\n debian|rhel|suse)\n echo \"Run with --install --yes after user approval to install or update the runtime.\"\n ;;\n *)\n echo \"Automatic install is not available for this distribution. See the\"\n echo \"MISSING messages above and the NVIDIA install guides at\"\n echo \" https://docs.nvidia.com/cuda/cuda-installation-guide-linux/\"\n echo \" https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html\"\n ;;\n esac\n exit 1\nfi\n\nif ! sudo_available; then\n echo \"MISSING: passwordless sudo/root is required for runtime installation.\" >&2\n exit 1\nfi\n\ncase \"$PKG_FAMILY\" in\n debian|rhel|suse) ;;\n *) unsupported_install_family ;;\nesac\n\nconfirm_install\ninstall_prereqs\ninstall_cuda_repo\ninstall_container_repo\ninstall_runtime_packages\ninstall_docker_package\nconfigure_docker_runtime\nadd_invoker_to_docker_group\n\nif have modprobe; then\n \"${SUDO[@]}\" modprobe nvidia || true\nfi\n\nprint_status\nruntime_ok\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":28007,"content_sha256":"b8fa35e8f00badd6126563665cbabc21df08dc7e95ef9cf9a9c75118bbc7b9e3"},{"filename":"skill-card.md","content":"## Description: \u003cbr>\nHost setup for TAO GPU backends that checks and, after user approval, installs minimum-compatible NVIDIA driver, CUDA Toolkit, and NVIDIA Container Toolkit versions for Docker/local-Docker and Kubernetes GPU worker hosts. \u003cbr>\n\nThis skill is ready for commercial/non-commercial use. \u003cbr>\n\n## Owner\nNVIDIA \u003cbr>\n\n### License/Terms of Use: \u003cbr>\nApache 2.0 \u003cbr>\n## Use Case: \u003cbr>\nDevelopers and engineers preparing NVIDIA GPU hosts for TAO training, evaluation, and inference workflows on Docker or Kubernetes backends. \u003cbr>\n\n### Deployment Geography for Use: \u003cbr>\nGlobal \u003cbr>\n\n## Requirements / Dependencies: \u003cbr>\n**Requires API Key or External Credential:** [Not Specified] \u003cbr>\n**Credential Type(s):** [None identified] \u003cbr>\n\nDo not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. \u003cbr>\n\n## Known Risks and Mitigations: \u003cbr>\nRisk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. \u003cbr>\nMitigation: Review and scan skill before deployment. \u003cbr>\n\n## Reference(s): \u003cbr>\n- [NVIDIA CUDA Installation Guide for Linux](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) \u003cbr>\n- [NVIDIA Container Toolkit Installation Guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) \u003cbr>\n- [Docker Engine Installation Guide](https://docs.docker.com/engine/install/) \u003cbr>\n- [Skill runtime profile schema](references/skill_info.yaml) \u003cbr>\n\n\n## Skill Output: \u003cbr>\n**Output Type(s):** [Shell commands, Configuration instructions] \u003cbr>\n**Output Format:** [Markdown with inline bash code blocks] \u003cbr>\n**Output Parameters:** [1D] \u003cbr>\n**Other Properties Related to Output:** [None] \u003cbr>\n\n## Evaluation Agents Used: \u003cbr>\n- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`) \u003cbr>\n- Codex (`openai/openai/gpt-5.5`) \u003cbr>\n\n\n\n## Evaluation Tasks: \u003cbr>\n1 evaluation task (1 positive), each attempt in an isolated sandbox pod. Dataset digest: sha256:906b47f67164bf4211efaf087e292e7014b9d8923002af0a07a00469b12ae6dc. \u003cbr>\n\n## Evaluation Metrics Used: \u003cbr>\nReported benchmark dimensions: \u003cbr>\n- Security: Whether the skill avoids unsafe operations, secret leakage, and unauthorized access. \u003cbr>\n- Correctness: Whether the final answer is correct against the reference answer. \u003cbr>\n- Discoverability: Whether the expected skill was selected and the workflow executed. \u003cbr>\n- Effectiveness: Whether the skill helped complete the user's goal and followed the expected workflow. \u003cbr>\n- Efficiency: Whether the skill avoided wasted tool calls and token usage. \u003cbr>\n\nUnderlying evaluation signals used in this run: \u003cbr>\n- `security`: Checks for unsafe operations, secret leakage, and unauthorized access. \u003cbr>\n- `skill_execution`: Whether the expected skill was selected, decoys were avoided, and the workflow executed. \u003cbr>\n- `accuracy`: Final-answer correctness against the reference answer. \u003cbr>\n- `goal_accuracy`: Whether the user's goal was achieved. \u003cbr>\n- `behavior_check`: Whether the expected workflow behavior was followed. \u003cbr>\n- `skill_efficiency`: Tool-call productivity; routing is scored under Discoverability. \u003cbr>\n- `token_efficiency`: Actual uncached prompt plus completion token usage. \u003cbr>\n\n\n\n## Evaluation Results: \u003cbr>\n| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |\n|---|---:|---:|\n| Overall | 99.5% | 95.3% |\n| Security | 100.0% → 100.0% (±0.0 points) | 100.0% → 100.0% (±0.0 points) |\n| Correctness | 6.7% → 100.0% (+93.3 points) | 20.0% → 100.0% (+80.0 points) |\n| Discoverability | 100.0% | 95.0% |\n| Effectiveness | 16.7% → 100.0% (+83.3 points) | 48.3% → 83.3% (+35.0 points) |\n| Efficiency | 97.5% | 98.1% |\n\n## Skill Version(s): \u003cbr>\n0.1.2 (source: frontmatter) \u003cbr>\n\n## Ethical Considerations: \u003cbr>\nNVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. \u003cbr>\n\n(For Release on NVIDIA Platforms Only) \u003cbr>\nPlease report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail). \u003cbr>\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":4420,"content_sha256":"064105f7ca15533258c0c32b41d441595f3fd58e390c30f988cd6952a5022095"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"NVIDIA GPU Host Setup","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"Standalone install?","type":"text","marks":[{"type":"strong"}]},{"text":" If this session was not initialized by the TAO skill bank plugin, run the ","type":"text"},{"text":"tao-setup","type":"text","marks":[{"type":"code_inline"}]},{"text":" skill first (host preflight, credentials, cross-skill discovery).","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Use this setup skill before TAO workflows run on the ","type":"text"},{"text":"docker","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"local-docker","type":"text","marks":[{"type":"code_inline"}]},{"text":", or ","type":"text"},{"text":"kubernetes","type":"text","marks":[{"type":"code_inline"}]},{"text":" backend. The TAO-wide default minimums are:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NVIDIA driver ","type":"text"},{"text":">=580","type":"text","marks":[{"type":"code_inline"}]},{"text":" (open kernel module preferred)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"CUDA Toolkit ","type":"text"},{"text":">=13.0","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NVIDIA Container Toolkit ","type":"text"},{"text":">=1.19.0","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Docker engine — only installed for ","type":"text"},{"text":"docker","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"local-docker","type":"text","marks":[{"type":"code_inline"}]},{"text":" backends and only when Docker is missing. The package picked depends on the distro family (","type":"text"},{"text":"docker.io","type":"text","marks":[{"type":"code_inline"}]},{"text":" on Debian-family by default, ","type":"text"},{"text":"moby-engine","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"docker-ce","type":"text","marks":[{"type":"code_inline"}]},{"text":" from ","type":"text"},{"text":"download.docker.com","type":"text","marks":[{"type":"code_inline"}]},{"text":" on RHEL-family, ","type":"text"},{"text":"docker","type":"text","marks":[{"type":"code_inline"}]},{"text":" on SUSE-family). Pass ","type":"text"},{"text":"--skip-docker-install","type":"text","marks":[{"type":"code_inline"}]},{"text":" to opt out.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"The check is safe and read-only by default — it works on any Linux distribution because it only probes ","type":"text"},{"text":"nvidia-smi","type":"text","marks":[{"type":"code_inline"}]},{"text":", the CUDA toolkit path, the installed container-toolkit package version (via ","type":"text"},{"text":"dpkg","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"rpm","type":"text","marks":[{"type":"code_inline"}]},{"text":"/the ","type":"text"},{"text":"nvidia-ctk","type":"text","marks":[{"type":"code_inline"}]},{"text":" binary version), and the Docker daemon's NVIDIA runtime.","type":"text"}]},{"type":"paragraph","content":[{"text":"Installation must be explicitly authorized by the user and rerun with ","type":"text"},{"text":"--install","type":"text","marks":[{"type":"code_inline"}]},{"text":". The install path is automated for these distro families:","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Family","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Tested distros","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Manager","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Notes","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"debian","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Ubuntu 22.04 / 24.04, Debian 12 (and derivatives Pop!_OS, Mint, Zorin, Raspbian, KDE Neon, etc. via ","type":"text"},{"text":"UBUNTU_CODENAME","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"VERSION_CODENAME","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"apt-get","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Adds NVIDIA ","type":"text"},{"text":"cuda-keyring","type":"text","marks":[{"type":"code_inline"}]},{"text":" + Container Toolkit ","type":"text"},{"text":".list","type":"text","marks":[{"type":"code_inline"}]},{"text":". Docker via ","type":"text"},{"text":"docker.io","type":"text","marks":[{"type":"code_inline"}]},{"text":" (override ","type":"text"},{"text":"$DOCKER_PACKAGE_DEBIAN","type":"text","marks":[{"type":"code_inline"}]},{"text":").","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"rhel","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Fedora 39+, RHEL / Rocky / AlmaLinux 9 and 10","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"dnf","type":"text","marks":[{"type":"code_inline"}]},{"text":" (or ","type":"text"},{"text":"yum","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Adds NVIDIA ","type":"text"},{"text":"cuda-\u003cdistro>.repo","type":"text","marks":[{"type":"code_inline"}]},{"text":" + Container Toolkit ","type":"text"},{"text":".repo","type":"text","marks":[{"type":"code_inline"}]},{"text":". Docker via Fedora ","type":"text"},{"text":"moby-engine","type":"text","marks":[{"type":"code_inline"}]},{"text":" when available, otherwise ","type":"text"},{"text":"docker-ce","type":"text","marks":[{"type":"code_inline"}]},{"text":" from ","type":"text"},{"text":"download.docker.com","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"suse","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"openSUSE Leap 15, SLES 15","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"zypper","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Adds the same NVIDIA ","type":"text"},{"text":".repo","type":"text","marks":[{"type":"code_inline"}]},{"text":" files. Docker via the distribution ","type":"text"},{"text":"docker","type":"text","marks":[{"type":"code_inline"}]},{"text":" package.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"other (Arch, Alpine, Gentoo, NixOS, FreeBSD, …)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"n/a","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"n/a","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"--install","type":"text","marks":[{"type":"code_inline"}]},{"text":" exits with a clear error listing the version targets and the NVIDIA install-guide URLs. Install manually, then rerun ","type":"text"},{"text":"--check-only","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Quick Start","type":"text"}]},{"type":"paragraph","content":[{"text":"From the skill bank root:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Check the local Docker backend host.\nbash skills/platform/tao-setup-nvidia-gpu-host/scripts/setup-nvidia-gpu-host.sh --backend docker --check-only\n\n# Install or repair after user approval.\nbash skills/platform/tao-setup-nvidia-gpu-host/scripts/setup-nvidia-gpu-host.sh --backend docker --install\n\n# Check a Kubernetes GPU worker host.\nbash skills/platform/tao-setup-nvidia-gpu-host/scripts/setup-nvidia-gpu-host.sh --backend kubernetes --check-only","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"⚠️ ","type":"text"},{"text":"Note — running non-interactively (agent/skill runs):","type":"text","marks":[{"type":"strong"}]},{"text":" a skill run has no terminal, so the installer's ","type":"text"},{"text":"Continue? [y/N]","type":"text","marks":[{"type":"code_inline"}]},{"text":" prompt cannot be answered. After running ","type":"text"},{"text":"--check-only","type":"text","marks":[{"type":"code_inline"}]},{"text":" to preview and getting the user's approval, append the assume-yes flag (","type":"text"},{"text":"--yes","type":"text","marks":[{"type":"code_inline"}]},{"text":") to the ","type":"text"},{"text":"--install","type":"text","marks":[{"type":"code_inline"}]},{"text":" command so it proceeds without a prompt — this auto-confirms installation of system packages (NVIDIA driver, CUDA Toolkit, NVIDIA Container Toolkit, and Docker for Docker backends) and modifies the host, so only do this on a host you control. A person running ","type":"text"},{"text":"--install","type":"text","marks":[{"type":"code_inline"}]},{"text":" directly at a terminal gets the prompt instead.","type":"text"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Workflow Contract","type":"text"}]},{"type":"paragraph","content":[{"text":"Docker and Kubernetes workflows must run the check before submitting GPU work:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"SB=\"${TAO_SKILL_BANK_PATH:-${TAO_SKILL_BANK_ROOT:-$PWD}}\"\nSETUP_SCRIPT=\"${SB}/skills/platform/tao-setup-nvidia-gpu-host/scripts/setup-nvidia-gpu-host.sh\"\n\nbash \"$SETUP_SCRIPT\" --backend docker --check-only || {\n echo \"MISSING: TAO GPU host runtime is not ready.\"\n echo \"After user approval, run (append --yes for non-interactive agent runs):\"\n echo \" bash \\\"$SETUP_SCRIPT\\\" --backend docker --install\"\n exit 1\n}","type":"text"}]},{"type":"paragraph","content":[{"text":"Never install silently. If the check fails, explain what is missing, ask the user to authorize the fix, then run the install command and rerun the check.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Model Runtime Override Contract","type":"text"}]},{"type":"paragraph","content":[{"text":"Platform defaults apply when a model has no override. A model that needs a different validated host stack declares it in ","type":"text"},{"text":"references/skill_info.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":":","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"yaml"},"content":[{"text":"runtime_requirements:\n gpu_host:\n min_driver_version: '\u003cversion>'\n min_cuda_version: '\u003cversion>'\n min_container_toolkit_version: '\u003cversion>'","type":"text"}]},{"type":"paragraph","content":[{"text":"Read those values before the final platform preflight and pass them to the matching ","type":"text"},{"text":"--min-*-version","type":"text","marks":[{"type":"code_inline"}]},{"text":" flags. Model minimums take precedence for that workflow only; do not rewrite the platform defaults or requirements for other models. Version checks use numeric lower bounds, so later compatible releases pass. Always retain the selected-image GPU smoke test because a version bound cannot prove support for a particular GPU architecture.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"What The Installer Does","type":"text"}]},{"type":"paragraph","content":[{"text":"The installer dispatches on the detected distribution family. On every supported family it adds NVIDIA's CUDA and Container Toolkit repositories (if missing), installs packages that satisfy the active minimums, optionally installs Docker, wires the NVIDIA Docker runtime, and adds the invoking user to the ","type":"text"},{"text":"docker","type":"text","marks":[{"type":"code_inline"}]},{"text":" group.","type":"text"}]},{"type":"paragraph","content":[{"text":"Common steps (all families):","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Adds NVIDIA's CUDA repository if missing (apt ","type":"text"},{"text":"cuda-keyring","type":"text","marks":[{"type":"code_inline"}]},{"text":" deb, ","type":"text"},{"text":"cuda-\u003cdistro>.repo","type":"text","marks":[{"type":"code_inline"}]},{"text":" for dnf/zypper).","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Adds NVIDIA's Container Toolkit repository if missing (","type":"text"},{"text":".list","type":"text","marks":[{"type":"code_inline"}]},{"text":" for apt, ","type":"text"},{"text":".repo","type":"text","marks":[{"type":"code_inline"}]},{"text":" for dnf/zypper).","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Installs the matching kernel header / devel package for the running kernel.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Installs the current open-driver and Container Toolkit packages from the configured repositories plus the CUDA Toolkit package selected by ","type":"text"},{"text":"--min-cuda-version","type":"text","marks":[{"type":"code_inline"}]},{"text":", then verifies all three against the active minimums.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For Docker backends and when Docker is missing, installs Docker (override / opt-out flags below), enables/starts the daemon, then runs ","type":"text"},{"text":"nvidia-ctk runtime configure --runtime=docker","type":"text","marks":[{"type":"code_inline"}]},{"text":" and restarts Docker when ","type":"text"},{"text":"systemctl","type":"text","marks":[{"type":"code_inline"}]},{"text":" is available.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Adds the invoking user (","type":"text"},{"text":"$SUDO_USER","type":"text","marks":[{"type":"code_inline"}]},{"text":" if available, else ","type":"text"},{"text":"$USER","type":"text","marks":[{"type":"code_inline"}]},{"text":") to the ","type":"text"},{"text":"docker","type":"text","marks":[{"type":"code_inline"}]},{"text":" group so subsequent shells can run ","type":"text"},{"text":"docker","type":"text","marks":[{"type":"code_inline"}]},{"text":" without ","type":"text"},{"text":"sudo","type":"text","marks":[{"type":"code_inline"}]},{"text":" — opt out with ","type":"text"},{"text":"--skip-docker-group","type":"text","marks":[{"type":"code_inline"}]},{"text":". ","type":"text"},{"text":"The new group membership does not take effect in the current shell","type":"text","marks":[{"type":"strong"}]},{"text":": log out and back in, or run ","type":"text"},{"text":"newgrp docker","type":"text","marks":[{"type":"code_inline"}]},{"text":" in each new shell.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Attempts ","type":"text"},{"text":"modprobe nvidia","type":"text","marks":[{"type":"code_inline"}]},{"text":" so verification can pass before reboot.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Family-specific package selections:","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Step","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"debian-family","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"rhel-family","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"suse-family","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Kernel headers","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"linux-headers-$(uname -r)","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"kernel-devel-$(uname -r)","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"kernel-headers-$(uname -r)","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"kernel-default-devel","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Driver","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"current ","type":"text"},{"text":"nvidia-open","type":"text","marks":[{"type":"code_inline"}]},{"text":" (override: ","type":"text"},{"text":"$NVIDIA_DRIVER_PACKAGE_DEBIAN","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"current ","type":"text"},{"text":"nvidia-driver-cuda","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"kmod-nvidia-open-dkms","type":"text","marks":[{"type":"code_inline"}]},{"text":" (override: ","type":"text"},{"text":"$NVIDIA_DRIVER_PACKAGE_RHEL","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"$NVIDIA_DRIVER_KMOD_RHEL","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"current ","type":"text"},{"text":"nvidia-open-driver-G06-signed-kmp-default","type":"text","marks":[{"type":"code_inline"}]},{"text":" (override: ","type":"text"},{"text":"$NVIDIA_DRIVER_PACKAGE_SUSE","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"CUDA toolkit","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"package derived from the active minimum, such as ","type":"text"},{"text":"cuda-toolkit-13-0","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"same","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"same","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Container Toolkit","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"current ","type":"text"},{"text":"nvidia-container-toolkit","type":"text","marks":[{"type":"code_inline"}]},{"text":" + base/tools/libs, then minimum-version validation","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"same","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"same","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Docker","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"docker.io","type":"text","marks":[{"type":"code_inline"}]},{"text":" (override: ","type":"text"},{"text":"$DOCKER_PACKAGE_DEBIAN","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"moby-engine","type":"text","marks":[{"type":"code_inline"}]},{"text":"+","type":"text"},{"text":"moby-cli","type":"text","marks":[{"type":"code_inline"}]},{"text":" on Fedora when available, else ","type":"text"},{"text":"docker-ce docker-ce-cli containerd.io","type":"text","marks":[{"type":"code_inline"}]},{"text":" from ","type":"text"},{"text":"download.docker.com","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"docker","type":"text","marks":[{"type":"code_inline"}]}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Verification","type":"text"}]},{"type":"paragraph","content":[{"text":"After installation, verify:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"nvidia-smi\nnvcc --version\ndocker info --format '{{json .Runtimes}}' | grep nvidia\nsudo docker run --rm --runtime=nvidia --gpus all \"$TAO_IMAGE\" nvidia-smi -L","type":"text"}]},{"type":"paragraph","content":[{"text":"The detected driver, CUDA Toolkit, and Container Toolkit versions must meet the active TAO-wide or model-specific minimums. Then run the selected image's GPU smoke test; version comparison alone is not sufficient compatibility proof.","type":"text"}]},{"type":"paragraph","content":[{"text":"For a Cosmos backend, extend that smoke with the backend contract's Python and entrypoint checks. Cosmos Framework must execute ","type":"text"},{"text":"/workspace/.venv/bin/python","type":"text","marks":[{"type":"code_inline"}]},{"text":" as a non-root UID, import ","type":"text"},{"text":"cosmos_framework.callbacks.tao_status","type":"text","marks":[{"type":"code_inline"}]},{"text":", find native torchrun, and verify the A100 PatchEmbed compatibility marker when the host reports compute capability 8.0. Cosmos-RL must resolve its requested action executable, import system PyAV for video workflows, resolve the restricted FFmpeg ","type":"text"},{"text":"h264_cuvid","type":"text","marks":[{"type":"code_inline"}]},{"text":" decoder, load ","type":"text"},{"text":"libnvcuvid.so.1","type":"text","marks":[{"type":"code_inline"}]},{"text":", verify the backward-safe linear Qwen3-VL PatchEmbed marker, and verify its checkpoint loader accepts the prepared ","type":"text"},{"text":"qwen3_vl","type":"text","marks":[{"type":"code_inline"}]},{"text":" directory. A container that only passes ","type":"text"},{"text":"nvidia-smi","type":"text","marks":[{"type":"code_inline"}]},{"text":" is not ready for Cosmos training.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Kubernetes Notes","type":"text"}]},{"type":"paragraph","content":[{"text":"For self-managed Kubernetes clusters, run the host installer on every GPU worker node or bake the same package set into the node image before installing the NVIDIA GPU Operator or device plugin.","type":"text"}]},{"type":"paragraph","content":[{"text":"The workflow check also warns if ","type":"text"},{"text":"kubectl","type":"text","marks":[{"type":"code_inline"}]},{"text":" is available but the cluster reports no ","type":"text"},{"text":"nvidia.com/gpu","type":"text","marks":[{"type":"code_inline"}]},{"text":" allocatable capacity. In that case, install/configure the NVIDIA GPU Operator after the worker host runtime is ready:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"helm repo add nvidia https://helm.ngc.nvidia.com/nvidia\nhelm repo update\nhelm install --wait gpu-operator -n gpu-operator --create-namespace nvidia/gpu-operator","type":"text"}]},{"type":"paragraph","content":[{"text":"Managed Kubernetes providers may own driver installation through node images or GPU Operator policy. Do not overwrite a provider-managed GPU node without user approval and a rollback plan.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Failure Modes","type":"text"}]},{"type":"paragraph","content":[{"text":"Unsupported distribution family","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"--install","type":"text","marks":[{"type":"code_inline"}]},{"text":" automates debian-, rhel-, and suse-family hosts. On Arch, Alpine, Gentoo, NixOS, FreeBSD, or anything without ","type":"text"},{"text":"/etc/os-release","type":"text","marks":[{"type":"code_inline"}]},{"text":" (e.g. macOS), the script exits with a clear error that lists the four version targets and the upstream NVIDIA install-guide URLs:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"https://docs.nvidia.com/cuda/cuda-installation-guide-linux/","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"https://docs.docker.com/engine/install/","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"paragraph","content":[{"text":"Install those four pieces using your distribution's package manager and rerun the script with ","type":"text"},{"text":"--check-only","type":"text","marks":[{"type":"code_inline"}]},{"text":" to verify. The check is universally portable — it only queries the binaries / package databases — so once the runtime is in place the workflow contract is satisfied regardless of the underlying distro.","type":"text"}]},{"type":"paragraph","content":[{"text":"Unsupported Ubuntu/Debian derivative","type":"text","marks":[{"type":"strong"}]},{"text":": When ","type":"text"},{"text":"ID","type":"text","marks":[{"type":"code_inline"}]},{"text":" is e.g. ","type":"text"},{"text":"pop","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"mint","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"zorin","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"raspbian","type":"text","marks":[{"type":"code_inline"}]},{"text":", or another debian-family derivative, the script maps the host onto the upstream Ubuntu/Debian CUDA repo via ","type":"text"},{"text":"UBUNTU_CODENAME","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"VERSION_CODENAME","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"focal","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"jammy","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"noble","type":"text","marks":[{"type":"code_inline"}]},{"text":" → Ubuntu 20.04/22.04/24.04; ","type":"text"},{"text":"bullseye","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"bookworm","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"trixie","type":"text","marks":[{"type":"code_inline"}]},{"text":" → Debian 11/12/12). If the host's codename doesn't match a known upstream release, ","type":"text"},{"text":"--install","type":"text","marks":[{"type":"code_inline"}]},{"text":" exits with the same manual-install guidance described above.","type":"text"}]},{"type":"paragraph","content":[{"text":"Docker not installed","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"--check-only","type":"text","marks":[{"type":"code_inline"}]},{"text":" reports ","type":"text"},{"text":"MISSING: Docker is not installed","type":"text","marks":[{"type":"code_inline"}]},{"text":" and prints the exact rerun command appropriate to the detected distro family. The default ","type":"text"},{"text":"--install","type":"text","marks":[{"type":"code_inline"}]},{"text":" path installs Docker (","type":"text"},{"text":"docker.io","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"moby-engine","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"docker-ce","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"docker","type":"text","marks":[{"type":"code_inline"}]},{"text":" depending on family), enables/starts the daemon, configures the NVIDIA runtime, and adds the invoking user to the ","type":"text"},{"text":"docker","type":"text","marks":[{"type":"code_inline"}]},{"text":" group. If you prefer to manage Docker yourself, install it before rerunning the script or pass ","type":"text"},{"text":"--skip-docker-install","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"Docker installed but ","type":"text","marks":[{"type":"strong"}]},{"text":"docker run","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" still needs sudo","type":"text","marks":[{"type":"strong"}]},{"text":": The script adds the invoking user to the ","type":"text"},{"text":"docker","type":"text","marks":[{"type":"code_inline"}]},{"text":" group, but Linux only refreshes group membership on a new login session. Log out and back in, or run ","type":"text"},{"text":"newgrp docker","type":"text","marks":[{"type":"code_inline"}]},{"text":" in each new shell, until the new membership is active.","type":"text"}]},{"type":"paragraph","content":[{"text":"Docker runtime still missing","type":"text","marks":[{"type":"strong"}]},{"text":": Restart Docker, then rerun ","type":"text"},{"text":"nvidia-ctk runtime configure --runtime=docker","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"Detected version is below the active minimum","type":"text","marks":[{"type":"strong"}]},{"text":": Rerun the same command with ","type":"text"},{"text":"--install","type":"text","marks":[{"type":"code_inline"}]},{"text":" after approval, preserving any model-specific ","type":"text"},{"text":"--min-*-version","type":"text","marks":[{"type":"code_inline"}]},{"text":" flags. Package-name environment overrides select distribution-specific driver packages but do not weaken the minimum-version checks.","type":"text"}]},{"type":"paragraph","content":[{"text":"Driver installed but ","type":"text","marks":[{"type":"strong"}]},{"text":"nvidia-smi","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" fails","type":"text","marks":[{"type":"strong"}]},{"text":": Load the module with ","type":"text"},{"text":"sudo modprobe nvidia","type":"text","marks":[{"type":"code_inline"}]},{"text":" or reboot. Secure Boot may require MOK enrollment on systems where it is enabled.","type":"text"}]},{"type":"paragraph","content":[{"text":"Kubernetes still has no GPU capacity","type":"text","marks":[{"type":"strong"}]},{"text":": Confirm the driver works on each GPU node with ","type":"text"},{"text":"nvidia-smi","type":"text","marks":[{"type":"code_inline"}]},{"text":", then check the GPU Operator/device plugin pods and node labels.","type":"text"}]}]},"metadata":{"date":"2026-11-02","name":"tao-setup-nvidia-gpu-host","tags":["setup","nvidia","cuda","docker","kubernetes"],"author":"@skillopedia","source":{"stars":3825,"repo_name":"skills","origin_url":"https://github.com/NVIDIA/skills/blob/HEAD/skills/tao-setup-nvidia-gpu-host/SKILL.md","repo_owner":"NVIDIA","body_sha256":"bc3b2a18db46a64dc6f60e604d115adfdedc764cb8c29777ab5a12bdaeff603e","cluster_key":"ff47ad8a7e1730b552bad5b12074a450beb1fde83d5069ef5cdffac2b16aea8e","clean_bundle":{"format":"clean-skill-bundle-v1","source":"NVIDIA/skills/skills/tao-setup-nvidia-gpu-host/SKILL.md","text_bytes":44523,"source_commit":"25340619336e89bcc40365c28bfe9e790430468e","skills_sh_rank":9414,"attachment_count":6,"text_attachments":6,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"skills/tao-setup-nvidia-gpu-host/SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"v1","category":"devops-infrastructure","category_label":"DevOps"},"exact_dupes_collapsed_into_this":0},"license":"Apache-2.0","version":"v1","category":"devops-infrastructure","metadata":{"author":"NVIDIA Corporation","version":"0.1.2"},"import_tag":"skills-sh-top20k-2026-11","description":"Host setup for TAO GPU backends. Checks and, after user approval, installs minimum-compatible NVIDIA driver, CUDA Toolkit, and NVIDIA Container Toolkit versions for Docker/local-Docker and Kubernetes GPU worker hosts. TAO-wide defaults can be overridden by the selected model's runtime profile. The `--check-only` path works on any Linux distribution; `--install` automates debian-family (Ubuntu/Debian/Pop!_OS/Mint/Zorin/Raspbian), rhel-family (Fedora/RHEL/Rocky/AlmaLinux), and suse-family (openSUSE/SLES) hosts, and prints actionable manual-install steps for everything else. Use when the user asks to \"set up an NVIDIA GPU host\", \"check TAO Docker GPU runtime\", or prepare a Kubernetes GPU worker for TAO.","allowed-tools":"Read Bash","compatibility":"Runs `--check-only` on any Linux distribution. `--install` automates Ubuntu 22.04/24.04 + Debian 12 (apt), Fedora + RHEL/Rocky/AlmaLinux 9/10 (dnf), and openSUSE Leap / SLES 15 (zypper). Requires sudo/root, internet access to NVIDIA package repositories (and download.docker.com on rhel-family), and an x86_64 or aarch64 (sbsa) host. Other distributions (Arch, Alpine, Gentoo, NixOS, …) get a clear error that names the version targets and the NVIDIA install-guide URL."}},"renderedAt":1793666202752}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.