physical-ai-defect-image-generation

Physical AI Defect Image Generation Table of Contents - Supported Flows - Disambiguation (full table in ) - Step 0: Select Flow, Cookbook, and Gather Inputs - Common Preconditions (long-form in ) - Flow walkthroughs (one entry per flow; details in ) - OSMO Monitoring - Supporting files End-to-end orchestration of defect image generation, augmentation, and labeling pipelines for AOI (Automated Optical Inspection) datasets. AnomalyGen = Cosmos-Predict2-2B finetuned per use case (Cosmos-AnomalyGen-PCB-2B, -Metal-2B, -Glass-2B). Every flow has a canonical OSMO workflow YAML in that chains all ste…

\\t' read -r MAT DEFECT; do\n [ -n \"$MAT\" ] && [ -n \"$DEFECT\" ] || continue\n src=\"\"\n for candidate in \\\n \"$SUBMASK_ROOT/$MAT/mask/$DEFECT\" \\\n \"$SUBMASK_ROOT/$DEFECT\"; do\n [ -d \"$candidate\" ] && { src=\"$candidate\"; break; }\n done\n [ -n \"$src\" ] || { echo \"ERROR: submask dir not found for $MAT+$DEFECT (tried $SUBMASK_ROOT/$MAT/mask/$DEFECT and $SUBMASK_ROOT/$DEFECT)\"; exit 1; }\n dst=\"$STAGE/$MAT/mask/$DEFECT\"\n mkdir -p \"$dst\"\n for f in \"$src\"/*.png \"$src\"/*.jpg \"$src\"/*.jpeg; do\n [ -f \"$f\" ] && ln -sf \"$f\" \"$dst/$(basename \"$f\")\" || true\n done\n count=$(ls \"$dst\" 2>/dev/null | wc -l)\n echo \"submasks/$MAT+$DEFECT: $count files (from $src)\"\n [ \"$count\" -gt 0 ] || { echo \"ERROR: no submask files in $src\"; exit 1; }\n done \u003c \u003c(python3 -c 'import json,sys\n for m,d in json.loads(sys.argv[1]): print(f\"{m}\\t{d}\")' \"$ANOMALY_TYPES_JSON\")\n\n # Render defect_spec.jsonl (cad / text / free) — Day 0's cad_masks\n # support cad mode by default since stage emits \u003cMATERIAL>/cad_mask/\u003ccomposite>.png.\n python3 /tmp/render_defect_spec.py \\\n --output \"$STAGE/defect_spec.jsonl\" \\\n --pairs \"$ANOMALY_TYPES_JSON\" \\\n --spatial-dependency \"$DEFAULT_SPATIAL_DEPENDENCY\"\n echo \"--- defect_spec.jsonl ---\"\n cat \"$STAGE/defect_spec.jsonl\"\n echo \"--- end defect_spec ---\"\n\n # cad mode also requires semantic_segmentation_labels.json at the dataset\n # root — it maps cad_mask RGBA values to class IDs that AMP's CADParser\n # uses to extract per-class connected components from each cad_mask.\n #\n # usd2roi emits ONE global labels JSON at crop/ root.\n if [ \"$DEFAULT_SPATIAL_DEPENDENCY\" = \"cad\" ]; then\n GLOBAL_LABELS=$(find \"$MASK_IN\" -maxdepth 5 -name semantic_segmentation_labels.json -path '*/crop/*' 2>/dev/null | head -1)\n if [ -z \"$GLOBAL_LABELS\" ]; then\n echo \"ERROR: spatial_dependency=cad but no semantic_segmentation_labels.json under $MASK_IN/crop/\"\n exit 1\n fi\n cp \"$GLOBAL_LABELS\" \"$STAGE/semantic_segmentation_labels.json\"\n echo \"Staged labels: $GLOBAL_LABELS\"\n fi\n\n AMP_OUT=/tmp/amp_output\n JSONL=/tmp/inference.jsonl\n mkdir -p \"$AMP_OUT\"\n\n echo \"=== prep_testcase.sh (num_sdg=$NUM_SDG) ===\"\n bash \"$SCRIPTS/prep_testcase.sh\" \\\n --name \"${EXP_NAME}_infer\" \\\n --num-sdg \"$NUM_SDG\" \\\n --dataset-dir \"$STAGE\" \\\n --defect-spec \"$STAGE/defect_spec.jsonl\" \\\n --amp-output-dir \"$AMP_OUT\" \\\n --output-jsonl \"$JSONL\"\n\n python3 \"$SCRIPTS/validate_jsonl.py\" \"$WRAPPER\" \"$JSONL\"\n\n export IMAGINAIRE_OUTPUT_ROOT=\"${OUTPUT_DIR}/results\"\n mkdir -p \"$IMAGINAIRE_OUTPUT_ROOT\"\n echo \"=== run_sdg.sh (checkpoint_step=$CHECKPOINT_STEP, model_size=$MODEL_SIZE) ===\"\n bash \"$SCRIPTS/run_sdg.sh\" \\\n --checkpoint_dir \"$WRAPPER\" \\\n --step \"$CHECKPOINT_STEP\" \\\n --input_jsonl \"$JSONL\" \\\n --output_dir \"$OUTPUT_DIR\" \\\n --model_size \"$MODEL_SIZE\" \\\n --seed 0\n\n bash \"$SCRIPTS/verify_output.sh\" \"$JSONL\" \"$OUTPUT_DIR\"\n echo \"=== Inference complete: $(ls \"$OUTPUT_DIR/reconstructed_image/\" 2>/dev/null | wc -l) images ===\"\n outputs:\n - url: \"{{ dig_url_root }}/runs/{{ name }}/anomaly\"\n\ndefault-values:\n workflow_name: texture_defect_gen_day0\n exec_timeout: 28h\n queue_timeout: 2h\n\n # `name` has no default — every submit must pass `--set name=\u003cflow>-$STAMP`\n # (see SKILL.md §\"Name stamping\"). This forces the storage path\n # `runs/\u003cname>/{usd2roi-components,augment,finetune,anomaly}` to be unique\n # per submission instead of silently overwriting prior runs.\n dig_url_root: \"s3://osmo-workflows/dig\"\n\n # ── Finetune-or-checkpoint toggle ────────────────────────────────────────────\n # Defaults to passthrough against the shipped PCBA checkpoint.\n use_pretrained_checkpoint: \"true\"\n checkpoint_step: \"14000\"\n\n # ── URL-backed inputs ────────────────────────────────────────────────────────\n # Defaults read from the setup/ workflows' DIG URL layout:\n # models/pretrained, models/pcb, datasets/pcb/raw, datasets/pcb/assets.\n # Outputs write under runs/\u003cname>/.\n\n # ── Image-edit augmentation ──────────────────────────────────────────────────\n # Cookbook at assets/cookbooks/pcb/augmentation_config_ovsl2sl.yaml carries the\n # prompt + model parameters. Endpoint URL + model come from these knobs.\n # Default points at the local cluster service from references/nim/.\n image_edit_endpoint: \"http://qwen-image-edit-nvpcb-ovsl2sl.osmo-nims.svc.cluster.local:8000/v1\"\n image_edit_model: \"nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL\"\n\n # ── AnomalyGen inference inputs ──────────────────────────────────────────────\n num_sdg: \"30\" # total SDG entries across all defects\n default_spatial_dependency: \"cad\" # one of free|text|cad — cad uses staged cad_masks\n model_size: \"2b\"\n # Shipped PCBA checkpoint trains on these three (material, defect) pairs.\n # Override with --set anomaly_types_json='[...]' to retarget the anomaly set.\n anomaly_types_json: '[[\"IC\",\"bridge\"],[\"passive_component\",\"excess_solder\"],[\"passive_component\",\"missing\"]]'\n\n # Cluster knobs only — training-recipe knobs live in the per-usecase cookbook.\n anomalygen_image: \"nvcr.io/nvidia/paidf-anomalygen:1.0.0\"\n augmentation_image: \"nvcr.io/nvidia/paidf-augmentation:1.0.0\"\n usd2roi_image: \"nvcr.io/nvidia/paidf-simulation:1.0.0\"\n # Two-stage knobs (see SKILL.md §\"User intent → knob mapping\"):\n # render_patches → stage 1: raw scan_grid patches from sdg_pipeline.py\n # (NOT the final dataset size — these get cropped further)\n # crop_max_emit → stage 2: per-cell crop cap from usd2roi_crop.py\n # (this is what controls the final per-component dataset size)\n # For \"user wants N output images\", set crop_max_emit=N (or leave blank to\n # use cookbook default). render_patches caps the upstream raw render only.\n render_patches: \"-1\" # -1 = full scan_grid coverage (cover every cell defined by the board's cookbook)\n crop_max_emit: \"\" # blank = use cookbook value; set to N to cap per-cell crops; \"null\" removes cap\n # train_gpu / infer_gpu — scale by passing --set train_gpu=N infer_gpu=N at\n # submit time (set them individually to break symmetry, e.g. 8-GPU train,\n # 1-GPU infer).\n\n # ── usd2roi-render: scene + cookbook selection ───────────────────────────────\n # `datasets/pcb/assets` is the fixed URL — replace its contents at source\n # (mc upload to s3://.../datasets/pcb/assets/) when running custom data.\n # The `board` knob picks which per-board cookbook directory under\n # `assets/cookbooks/pcb/\u003cboard>/` to use. Default 0603_H100 matches the\n # shipped spark scene in `pcb-assets`. To run a different board, add its\n # cookbook directory and pass `--set board=\u003cdir-name>`.\n board: \"0603_H100\" # alternate shipped board: 1152819000\n scene_filename: \"spark_lighting.usd\" # USD inside assets bundle to use as the scene\n\n # ── Resources ────────────────────────────────────────────────────────────────\n render_gpu: \"1\"\n render_cpu: \"4\"\n render_memory: 32Gi\n render_storage: 50Gi\n augment_gpu: \"1\"\n augment_cpu: \"1\"\n augment_memory: 32Gi\n augment_storage: 25Gi\n # infer_gpu / infer_cpu / infer_memory — defaults sized for 1 GPU.\n # Scale all three together at submit time, e.g.\n # --set infer_gpu=4 infer_cpu=16 infer_memory=192Gi\n # Each rank loads the full 2B + tokenizer + DINOv2 + Cosmos-Predict2 stack\n # into host RAM, so memory must scale. See references/gpu_sizing.md for the\n # full per-GPU table.\n infer_gpu: \"1\"\n infer_cpu: \"4\"\n infer_memory: 64Gi\n infer_storage: 200Gi\n # train_gpu / train_cpu / train_memory — defaults sized for 1 GPU.\n # Scale all three together at submit time, e.g.\n # --set train_gpu=4 train_cpu=32 train_memory=192Gi\n # See references/gpu_sizing.md for the full per-GPU table and reasoning\n # (cosmos-predict2-2B rank ~33 GiB host RAM during DDP sync, etc.).\n train_gpu: \"1\"\n train_cpu: \"16\"\n train_memory: 64Gi\n train_storage: 300Gi\n platform: default\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":51210,"content_sha256":"ea803ec78f5813edf841b7d9d79f835c826ec0c90e1994e3832ef67712a97f96"},{"filename":"assets/configs/texture_defect_generation_day1_manual_roi.yaml","content":"# Defect Image Generation Workflow — Day 1 (manual ROI): Finetune/Checkpoint → Infer (labels inline)\n#\n# NOT the default for PCBA. Pick this spec only when:\n# - usecase is metal_surface or glass (no USD/real-photo flow exists for those), OR\n# - the user explicitly asks to skip CAD-to-real-photo alignment on PCBA\n# (e.g. \"manual ROI\", \"skip usd2roi\", \"use NGC PCBA artifact directly\").\n#\n# For default PCBA Day 1, use `texture_defect_generation_day1_real_alignment.yaml`\n# — that's the canonical path with CAD-derived USD + real photo + MI alignment\n# producing per-ROI crops.\n#\n# This spec runs inference on **manually prepared ROIs** — i.e., clean images\n# + masks that already exist upstream of this workflow (either NGC-shipped or\n# user-uploaded).\n#\n# Two operating modes, picked by what is in `\u003cdig_url_root>/datasets/\u003cusecase>/raw`:\n#\n# • Mode A (default): the URL already ships `defect_spec.jsonl` + canonical\n# `\u003cTEXTURE>/{clean_image,cad_mask,mask}/` layout (e.g. the raw artifact\n# produced by the relevant `setup/setup_\u003ccase>.yaml`).\n#\n# • Mode B: the URL is a flat user upload of clean images + per-defect submasks.\n# `anomaly-infer` stages them into the canonical layout and renders\n# `defect_spec.jsonl` from `--set anomaly_types_json=...` at runtime.\n#\n# Both modes run two task groups: optional `finetune-job` (omitted by Jinja when\n# `use_pretrained_checkpoint=true`, the default) → `anomaly-infer`.\n#\n# Anomaly inference emits labeled output natively (no separate pseudo-label stage).\n# Suitable for metal_surface, glass, and pre-captured PCBA inspection.\n#\n# Submit (single step — the per-usecase cookbook\n# `assets/cookbooks/\u003cusecase>/ag_config.yaml` is uploaded as a template and\n# rendered in-pod by yq when use_pretrained_checkpoint=false; passthrough mode\n# omits the finetune group entirely):\n#\n# STAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)\n# osmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/texture_defect_generation_day1_manual_roi.yaml \\\n# --pool \u003cpool> \\\n# --set name=texture_defect_gen_day1_manual_roi-$STAMP \\\n# dig_url_root=\u003cdig_url_root> \\\n# usecase=\u003cusecase> \\\n# use_pretrained_checkpoint=true \\\n# checkpoint_step=\u003cstep> \\\n# 'anomaly_types_json=[[\"MAT\",\"DEFECT\"],...]' \\\n# num_sdg=30\n#\n# In finetune mode the in-pod render patches 5 fields (job.group, job.name,\n# dataset paths, val JSONL, NVDINOV2 checkpoint) and drops trainer.early_stop.\n# The cookbook is shared with the standalone `finetune.yaml` (byte-identical to\n# the shipped checkpoint's training config).\n#\n# Inference invokes the upstream chain (prep_testcase.sh → validate_jsonl.py →\n# run_sdg.sh → verify_output.sh) baked at /workspace/paidf-anomalygen/scripts/utilities/.\n# Defect ROIs are routed via defect_spec.jsonl: `free` (whole-image), `text`\n# (Qwen+SAM2), or `cad` (cad_mask+semantic labels). Default is `free` for all\n# defects unless the inference URL ships its own defect_spec.jsonl at root.\n#\n# When use_pretrained_checkpoint=false the finetune task trains from scratch using the\n# rendered cookbook config; validation.jsonl + amp/ are produced fresh inside the\n# finetune task (anomalygen Phase 1 Step 2) before torchrun starts.\n#\n# Prerequisites (run once per shell):\n# - bash scripts/preflight_credentials.sh\n# - bash scripts/preflight_urls.sh 1 \u003cusecase>\n# - URL assets exist under dig_url_root:\n# models/pretrained, models/\u003cusecase>, datasets/\u003cusecase>/raw\n# - If finetuning: datasets/\u003cusecase>/raw exists (validation set is built inside\n# the finetune task — no pre-baked validation.jsonl required)\n# - If passthrough: models/\u003cusecase> checkpoint exists\n#\n# Output: {{ dig_url_root }}/runs/\u003cname>/anomaly — per-defect labeled images.\n# See references/flows/texture_defect_generation_day1_manual_roi.md for the full walkthrough.\n\n\nversion: 2\nworkflow:\n name: \"{{ workflow_name }}\"\n timeout:\n exec_timeout: \"{{ exec_timeout }}\"\n queue_timeout: \"{{ queue_timeout }}\"\n\n resources:\n gpu-infer:\n gpu: \"{{ infer_gpu }}\"\n cpu: \"{{ infer_cpu }}\"\n memory: \"{{ infer_memory }}\"\n storage: \"{{ infer_storage }}\"\n platform: \"{{ platform }}\"\n gpu-train:\n gpu: \"{{ train_gpu }}\"\n cpu: \"{{ train_cpu }}\"\n memory: \"{{ train_memory }}\"\n storage: \"{{ train_storage }}\"\n platform: \"{{ platform }}\"\n\n groups:\n\n {% if use_pretrained_checkpoint|string|lower not in [\"true\", \"1\", \"yes\"] %}\n # ── Group 1: Finetune (omitted entirely when use_pretrained_checkpoint=true;\n # in that mode anomaly-infer reads the checkpoint URL directly) ──────────────\n - name: finetune-job\n tasks:\n - name: finetune\n lead: true\n image: \"{{ anomalygen_image }}\"\n resource: gpu-train\n credentials:\n hf-token:\n HF_TOKEN: token\n environment:\n NUM_GPUS: \"{{ train_gpu }}\"\n EXP_NAME: \"{{ name }}\"\n PRETRAINED_SRC: \"{{input:0}}/pretrained\"\n DATASET_DIR: \"{{input:1}}\"\n inputs:\n - url: \"{{ dig_url_root }}/models/pretrained\" # {{input:0}} pretrained/ tree\n - url: \"{{ dig_url_root }}/datasets/{{ usecase }}/raw\" # {{input:1}} raw NGC training data\n command: [\"bash\"]\n args: [\"/tmp/finetune.sh\"]\n files:\n # Cookbook template — rendered in-pod by yq below.\n - localpath: ../cookbooks/{{ usecase }}/ag_config.yaml\n path: /tmp/ag_config_template.yaml\n\n - path: /tmp/finetune.sh\n contents: |\n set -euo pipefail\n\n # ── Pod-template preflight (training task) ─────────────────────\n DSHM_GB=$(df -B1G /dev/shm | tail -1 | awk '{print $2}')\n if [ \"$DSHM_GB\" -lt 16 ]; then\n echo \"ERROR: /dev/shm is ${DSHM_GB}GiB; need >= 16 GiB (32 preferred) for torchrun shared-memory.\"\n exit 1\n fi\n # ───────────────────────────────────────────────────────────────\n\n # Install Mike Farah yq into /tmp (pod /usr/local/bin is non-writable; paidf-anomalygen ships wget, no curl).\n [ -x /tmp/yq ] || {\n wget -q https://github.com/mikefarah/yq/releases/download/v4.44.3/yq_linux_amd64 -O /tmp/yq\n chmod +x /tmp/yq\n }\n export PATH=/tmp:$PATH\n\n OUTPUT_DIR=\"{{output}}\"\n mkdir -p \"$OUTPUT_DIR\"\n\n # This group only runs when use_pretrained_checkpoint=false\n # (Jinja-gated at the workflow level). Passthrough mode omits the\n # group entirely; anomaly-infer reads the checkpoint URL directly.\n TEMPLATE=/tmp/ag_config_template.yaml\n [ -f \"$TEMPLATE\" ] || {\n echo \"ERROR: $TEMPLATE not mounted — cookbook upload failed.\"\n echo \" Confirm assets/cookbooks/{{ usecase }}/ag_config.yaml exists.\"\n exit 1\n }\n\n [ -d \"$PRETRAINED_SRC\" ] || {\n echo \"ERROR: pretrained tree not at $PRETRAINED_SRC\"\n ls -la \"$(dirname \"$PRETRAINED_SRC\")\" || true\n exit 1\n }\n\n # Per-item symlink-replace into the container's checkpoint dir.\n # IMPORTANT: do NOT wipe the dir — SAM2 + Qwen3-VL ship baked\n # there and are referenced by other tools in the image even\n # though this task only runs torchrun.\n cd /workspace/paidf-anomalygen\n CONTAINER_CKPT_DIR=/workspace/paidf-anomalygen/checkpoints\n mkdir -p \"$CONTAINER_CKPT_DIR\"\n for item in NVDINOV2 nvidia google-t5 facebook C-RADIOv2_B.pth sam2 Qwen; do\n if [ -e \"$PRETRAINED_SRC/$item\" ]; then\n rm -rf \"$CONTAINER_CKPT_DIR/$item\"\n ln -s \"$PRETRAINED_SRC/$item\" \"$CONTAINER_CKPT_DIR/$item\"\n fi\n done\n\n [ -d \"$DATASET_DIR\" ] || {\n echo \"ERROR: training dataset not at $DATASET_DIR\"\n exit 1\n }\n\n DEFECT_SPEC=\"$DATASET_DIR/defect_spec.jsonl\"\n [ -f \"$DEFECT_SPEC\" ] || {\n echo \"ERROR: $DEFECT_SPEC missing in raw dataset.\"\n echo \" Re-run setup/setup_{{ usecase }}.yaml (or setup/setup_metal.yaml for metal_surface) for datasets/{{ usecase }}/raw.\"\n exit 1\n }\n\n # anomalygen helper scripts (pinned by image digest).\n SCRIPTS=/workspace/paidf-anomalygen/scripts/utilities\n ls \"$SCRIPTS/prep_testcase.sh\" >/dev/null || {\n echo \"ERROR: $SCRIPTS/prep_testcase.sh not in image — check digest.\"\n exit 1\n }\n\n # ─── Phase 1 Step 1: validate dataset structure ────────────\n echo \"=== Phase 1 Step 1: validate_dataset.py ===\"\n python3 \"$SCRIPTS/validate_dataset.py\" \"$DATASET_DIR\"\n\n NUM_SDG=$(find \"$DATASET_DIR\" -type f -path \"*/mask/*/*\" \\\n \$ -name \"*.png\" -o -name \"*.jpg\" -o -name \"*.jpeg\" \$ | wc -l)\n [ \"$NUM_SDG\" -gt 0 ] || { echo \"ERROR: no training masks under $DATASET_DIR/*/mask/\"; exit 1; }\n echo \"Total training mask count (num_sdg): $NUM_SDG\"\n\n # ─── Phase 1 Step 2: AMP placement → validation.jsonl ──────\n VAL_DIR=/tmp/validation\n rm -rf \"$VAL_DIR\"\n mkdir -p \"$VAL_DIR/amp\"\n VAL_JSONL=\"$VAL_DIR/validation.jsonl\"\n\n echo \"=== Phase 1 Step 2: prep_testcase.sh (validation_${EXP_NAME}) ===\"\n bash \"$SCRIPTS/prep_testcase.sh\" \\\n --name \"validation_${EXP_NAME}\" \\\n --num-sdg \"$NUM_SDG\" \\\n --dataset-dir \"$DATASET_DIR\" \\\n --defect-spec \"$DEFECT_SPEC\" \\\n --amp-output-dir \"$VAL_DIR/amp\" \\\n --output-jsonl \"$VAL_JSONL\"\n\n [ -s \"$VAL_JSONL\" ] || {\n echo \"ERROR: prep_testcase.sh produced an empty validation.jsonl\"\n exit 1\n }\n echo \"validation.jsonl: $(wc -l \u003c \"$VAL_JSONL\") rows\"\n echo \"validation amp/: $(find \"$VAL_DIR/amp\" -type f | wc -l) files\"\n\n # ── Render per-run training config from cookbook template ─────\n # VAL_JSONL is in scope here (Phase 1 Step 2 just produced it).\n CONFIG_FILE=/tmp/ag_config.yaml\n NAME=\"$EXP_NAME\" \\\n JOB_NAME=\"${EXP_NAME}_training_FP32_lr0.02_bs=2_2b_512x512\" \\\n DATASET_DIR=\"$DATASET_DIR\" \\\n VAL_JSONL=\"$VAL_JSONL\" \\\n NVDINOV2_CKPT=\"checkpoints/NVDINOV2/nv_dinov2_classification_model.ckpt\" \\\n yq '\n .job.group = strenv(NAME) |\n .job.name = strenv(JOB_NAME) |\n .dataloader_train.dataset.dataset_dir = strenv(DATASET_DIR) |\n .dataloader_val.dataset.input_data_path = strenv(VAL_JSONL) |\n .model.config.ag_config.mask_encoder.encoder_config.init_cfg.checkpoint = strenv(NVDINOV2_CKPT) |\n del(.trainer.early_stop)\n ' \"$TEMPLATE\" > \"$CONFIG_FILE\"\n echo \"Rendered $CONFIG_FILE from $TEMPLATE (NAME=$EXP_NAME)\"\n\n # Cookbook hygiene — runs after yq render so per-run overrides are seen.\n # save_iter > max_iter is fatal (no checkpoint ever written).\n # validation_iter > max_iter degrades pick_best_step.sh to latest-iter\n # (still warn-only because \"just train and pick latest\" is legitimate).\n # save_iter == max_iter is the shipped pattern — trainer saves at iter\n # == max_iter, so don't warn on that case.\n MAX_ITER=$(yq '.trainer.max_iter // 0' \"$CONFIG_FILE\")\n SAVE_ITER=$(yq '.checkpoint.save_iter // 0' \"$CONFIG_FILE\")\n VALIDATION_ITER=$(yq '.trainer.validation_iter // 0' \"$CONFIG_FILE\")\n LOGGING_ITER=$(yq '.trainer.logging_iter // 0' \"$CONFIG_FILE\")\n\n if [ \"$SAVE_ITER\" -gt 0 ] && [ \"$MAX_ITER\" -gt 0 ] && [ \"$SAVE_ITER\" -gt \"$MAX_ITER\" ]; then\n echo \"ERROR: cookbook save_iter=$SAVE_ITER > max_iter=$MAX_ITER — no checkpoint will be saved.\" >&2\n echo \" Fix assets/cookbooks/{{ usecase }}/ag_config.yaml: set save_iter \u003c= max_iter.\" >&2\n exit 1\n fi\n\n if [ \"$VALIDATION_ITER\" -gt 0 ] && [ \"$MAX_ITER\" -gt 0 ] && [ \"$VALIDATION_ITER\" -gt \"$MAX_ITER\" ]; then\n echo \"WARN: cookbook validation_iter=$VALIDATION_ITER > max_iter=$MAX_ITER — no validation logs; pick_best_step.sh will fall back to latest trained iter (not best-by-nn_score).\" >&2\n fi\n\n if [ \"$LOGGING_ITER\" -gt 0 ] && [ \"$MAX_ITER\" -gt 0 ] && [ \"$LOGGING_ITER\" -gt \"$MAX_ITER\" ]; then\n echo \"WARN: cookbook logging_iter=$LOGGING_ITER > max_iter=$MAX_ITER — no progress logs will be emitted.\" >&2\n fi\n\n # Stage rendered config alongside the trainer code.\n mkdir -p ag_configs\n cp \"$CONFIG_FILE\" \"ag_configs/${EXP_NAME}.yaml\"\n\n EXP=\"predict2_anomaly_gen_ddp_2b\"\n export IMAGINAIRE_OUTPUT_ROOT=\"${OUTPUT_DIR}/results\"\n mkdir -p \"$IMAGINAIRE_OUTPUT_ROOT\"\n echo \"=== torchrun ($EXP_NAME, $NUM_GPUS GPUs, experiment=$EXP) ===\"\n torchrun --nproc_per_node=\"$NUM_GPUS\" --master_port=12341 \\\n -m scripts.anomaly_gen.ag_train \\\n --config=cosmos_predict2/configs/base/ag_config.py \\\n --ag_config=\"ag_configs/${EXP_NAME}.yaml\" \\\n -- experiment=\"$EXP\"\n echo \"=== Training complete ===\"\n outputs:\n - url: \"{{ dig_url_root }}/runs/{{ name }}/finetune\"\n {% endif %}\n\n # ── Group 2: AnomalyGen inference (with native labeling) ─────────────────────\n # Clean images come directly from the prepared inference URL (no augmentation).\n - name: anomaly-infer\n tasks:\n - name: infer-all-defects\n lead: true\n image: \"{{ anomalygen_image }}\"\n resource: gpu-infer\n credentials:\n hf-token:\n HF_TOKEN: token\n environment:\n # Per-usecase fallback for anomaly_types_json and checkpoint_step:\n # the shipped PCBA default in default-values: is correct for\n # `usecase=pcb`; for `usecase=glass` / `usecase=metal_surface` we swap\n # in the values that match the shipped v1.x checkpoints + cookbooks.\n # User-supplied `--set anomaly_types_json=...` / `checkpoint_step=...`\n # still wins because Jinja `default()` only fires when the var is\n # undefined OR equals the corresponding default-values entry. We\n # detect overrides by comparing against the PCBA default literal.\n {% if usecase == \"glass\" and anomaly_types_json == '[[\"passive_component\",\"missing\"]]' %}\n ANOMALY_TYPES_JSON: '[[\"Phone\",\"oil\"],[\"Phone\",\"scratch\"],[\"Phone\",\"stain\"]]'\n {% elif usecase == \"metal_surface\" and anomaly_types_json == '[[\"passive_component\",\"missing\"]]' %}\n ANOMALY_TYPES_JSON: '[[\"metal_surface\",\"MT_Blowhole\"],[\"metal_surface\",\"MT_Break\"],[\"metal_surface\",\"MT_Crack\"],[\"metal_surface\",\"MT_Fray\"],[\"metal_surface\",\"MT_Uneven\"]]'\n {% else %}\n ANOMALY_TYPES_JSON: '{{ anomaly_types_json }}'\n {% endif %}\n {% if usecase == \"glass\" and checkpoint_step == \"14000\" %}\n CHECKPOINT_STEP: \"9000\"\n {% elif usecase == \"metal_surface\" and checkpoint_step == \"14000\" %}\n CHECKPOINT_STEP: \"10000\"\n {% else %}\n CHECKPOINT_STEP: \"{{ checkpoint_step }}\"\n {% endif %}\n NUM_SDG: \"{{ num_sdg }}\"\n NUM_GPUS: \"{{ infer_gpu }}\"\n DEFAULT_SPATIAL_DEPENDENCY: \"{{ default_spatial_dependency }}\"\n EXP_NAME: \"{{ name }}\"\n MODEL_SIZE: \"{{ model_size }}\"\n inputs:\n # Mode A/B inputs: single inference URL + pretrained + checkpoint.\n - url: \"{{ dig_url_root }}/datasets/{{ usecase }}/raw\" # {{input:0}} clean images + submasks (+ optional defect_spec.jsonl)\n - url: \"{{ dig_url_root }}/models/pretrained\" # {{input:1}} pretrained weights\n {% if use_pretrained_checkpoint|string|lower in [\"true\", \"1\", \"yes\"] %}\n - url: \"{{ dig_url_root }}/models/{{ usecase }}\" # {{input:2}} shipped checkpoint (no finetune scheduled)\n {% else %}\n - task: finetune # {{input:2}} freshly-trained checkpoint\n {% endif %}\n command: [\"bash\"]\n args: [\"/tmp/run_infer.sh\"]\n files:\n # Host-supplied helper: renders defect_spec.jsonl when the prepared input\n # doesn't ship its own. See scripts/render_defect_spec.py.\n - localpath: ../../scripts/render_defect_spec.py\n path: /tmp/render_defect_spec.py\n - localpath: ../../scripts/pick_best_step.sh\n path: /tmp/pick_best_step.sh\n\n - path: /tmp/run_infer.sh\n contents: |\n set -euo pipefail\n\n INFERENCE_DATASET=\"{{input:0}}\"\n PRETRAINED_SRC_IN=\"{{input:1}}\"\n CKPT_DATASET=\"{{input:2}}\"\n # Nest SDG output one level under {{output}} so convert_to_daft_format.py's\n # default sibling path \"\u003cinput>_daft_v3\" lands inside the writable\n # /osmo/data/output mount instead of unwritable /osmo/data.\n OSMO_OUTPUT_ROOT=\"{{output}}\"\n OUTPUT_DIR=\"${OSMO_OUTPUT_ROOT}/inference\"\n mkdir -p \"$OUTPUT_DIR\"\n\n # anomalygen helper scripts ship at this path in the image.\n SCRIPTS=/workspace/paidf-anomalygen/scripts/utilities\n ls \"$SCRIPTS/prep_testcase.sh\" >/dev/null || {\n echo \"ERROR: $SCRIPTS/prep_testcase.sh not in image\"; exit 1;\n }\n\n # Auto-discover nested layouts (NGC-shipped datasets nest under\n # \u003cdataset>/\u003cversioned-subdir>/\u003cTEXTURE>/clean_image; user-uploaded\n # datasets are usually flat).\n resolve_dir () {\n local base=\"$1\" ; shift\n for name in \"$@\"; do\n local hit\n hit=$(find \"$base\" -maxdepth 6 -type d -name \"$name\" | head -1 || true)\n if [ -n \"$hit\" ]; then echo \"$hit\"; return 0; fi\n done\n echo \"$base\"\n }\n CLEAN_DIR=$(resolve_dir \"$INFERENCE_DATASET\" clean_image clean_images)\n SUBMASK_BASE=$(resolve_dir \"$INFERENCE_DATASET\" submasks mask masks)\n echo \"CLEAN_DIR=$CLEAN_DIR\"\n echo \"SUBMASK_BASE=$SUBMASK_BASE\"\n\n # Symlink pretrained checkpoints. Per-item replace (preserve any\n # baked items); include sam2 + Qwen so text2roi AMP can reach\n # SAM2 + Qwen3-VL via the pretrained tree.\n cd /workspace/paidf-anomalygen\n CKPT_DEST=/workspace/paidf-anomalygen/checkpoints\n mkdir -p \"$CKPT_DEST\"\n set +o pipefail\n PRETRAINED=$(find \"$PRETRAINED_SRC_IN\" -maxdepth 4 -type d -name pretrained | head -1)\n set -o pipefail\n [ -n \"$PRETRAINED\" ] || { echo \"ERROR: pretrained/ not found\"; exit 1; }\n for item in NVDINOV2 nvidia google-t5 facebook C-RADIOv2_B.pth sam2 Qwen; do\n if [ -e \"$PRETRAINED/$item\" ]; then\n rm -rf \"$CKPT_DEST/$item\"\n ln -s \"$PRETRAINED/$item\" \"$CKPT_DEST/$item\"\n fi\n done\n\n # Locate the training config (ag_config.yaml) — shipped checkpoints\n # keep it flat with iter_*.pt; freshly-trained outputs nest it under\n # results/anomaly_gen/\u003cNAME>/\u003cJOB_NAME>/.\n set +o pipefail\n AG_CONFIG_PATH=$(find \"$CKPT_DATASET\" -name \"ag_config.yaml\" -maxdepth 8 | head -1)\n set -o pipefail\n [ -n \"$AG_CONFIG_PATH\" ] || { echo \"ERROR: ag_config.yaml not found in checkpoint\"; exit 1; }\n AG_CONFIG_DIR=$(dirname \"$AG_CONFIG_PATH\")\n echo \"Training config: $AG_CONFIG_PATH\"\n\n # Wrapper: link model-weights iter_*.pt files into the canonical\n # \u003cwrapper>/checkpoints/model/iter_\u003cstep>.pt layout.\n # Two source layouts:\n # Trainer output: \u003cJOB_NAME>/checkpoints/{model,optim,scheduler,trainer}/iter_\u003cstep>.pt\n # — only model/ has the actual weights; the others are\n # optimizer/scheduler/trainer state and must NOT be picked\n # up (a name-collision overwrite would substitute optimizer\n # state for model weights and break anomaly_embedding load).\n # Shipped: flat iter_*.pt next to ag_config.yaml.\n WRAPPER=/tmp/ag_ckpt_wrapper\n rm -rf \"$WRAPPER\"\n mkdir -p \"$WRAPPER/checkpoints/model\"\n set +o pipefail\n PT_FILES=$(find \"$CKPT_DATASET\" -maxdepth 10 -path \"*/checkpoints/model/iter_*.pt\")\n if [ -z \"$PT_FILES\" ]; then\n PT_FILES=$(find \"$AG_CONFIG_DIR\" -maxdepth 1 -name \"iter_*.pt\")\n fi\n set -o pipefail\n [ -n \"$PT_FILES\" ] || { echo \"ERROR: no model iter_*.pt files found under $CKPT_DATASET\"; exit 1; }\n echo \"$PT_FILES\" | while read -r f; do\n ln -sf \"$f\" \"$WRAPPER/checkpoints/model/$(basename \"$f\")\"\n done\n cp \"$AG_CONFIG_PATH\" \"$WRAPPER/ag_config.yaml\"\n # Also surface any flat sidecars from the shipped-checkpoint layout\n # (e.g. tokenizer files) at $WRAPPER root.\n for f in \"$AG_CONFIG_DIR\"/*; do\n bname=$(basename \"$f\")\n [ \"$bname\" = \"ag_config.yaml\" ] && continue\n case \"$bname\" in\n iter_*.pt|*.ckpt|*.pt) ;;\n *) [ -e \"$WRAPPER/$bname\" ] || ln -sf \"$f\" \"$WRAPPER/$bname\" ;;\n esac\n done\n echo \"Linked $(ls \"$WRAPPER/checkpoints/model\" | wc -l) checkpoint files into $WRAPPER\"\n\n # Validate checkpoint against the wrapped layout (prints supported\n # TEXTURE+ANOMALY set). validate_checkpoint.py looks for\n # \u003cdir>/checkpoints/model/iter_\u003cstep:09d>.pt — that path now exists\n # in $WRAPPER.\n # Auto-derive the inference step from validation KPIs for freshly-trained\n # checkpoints (presence of valid/\u003cSTEP>/valid_kpi.csv); falls back to\n # --set checkpoint_step for shipped checkpoints. Per anomalygen contract\n # (skills/anomalygen/references/finetune.md §\"Best checkpoint selection\"):\n # pick the step with the peak average nn_score, not the final iter.\n CHECKPOINT_STEP=$(bash /tmp/pick_best_step.sh \"$CKPT_DATASET\" \"$CHECKPOINT_STEP\")\n echo \"Inference checkpoint step: $CHECKPOINT_STEP\"\n\n python3 \"$SCRIPTS/validate_checkpoint.py\" \"$WRAPPER\" --step \"$CHECKPOINT_STEP\"\n\n # Two operating modes for the inference inputs, picked by\n # whether the prepared input ships a defect_spec.jsonl:\n #\n # A) Prepared anomalygen dataset (ships defect_spec.jsonl at\n # root + nested \u003cTEXTURE>/{clean_image,mask,cad_mask}/ + optional\n # semantic_segmentation_labels.json). Used directly as\n # --dataset-dir; prep_testcase.sh auto-discovers clean images.\n #\n # B) Flat per-defect upload (\u003cdefect>/\u003csubmask>.png subdirs only).\n # Stage into the canonical layout and render a defect_spec from\n # --set args.\n set +o pipefail\n USER_SPEC=$(find \"$INFERENCE_DATASET\" -maxdepth 3 -name \"defect_spec.jsonl\" | head -1)\n set -o pipefail\n\n AMP_OUT=/tmp/amp_output\n JSONL=/tmp/inference.jsonl\n mkdir -p \"$AMP_OUT\"\n\n if [ -n \"$USER_SPEC\" ]; then\n # Mode A: prepared URL artifact. Point prep_testcase at the\n # URL root directly — the canonical anomalygen\n # layout (\u003cTEXTURE>/{clean_image,cad_mask,mask,...} +\n # defect_spec.jsonl + semantic_segmentation_labels.json) is\n # already what prep_testcase expects. Omit --clean-dir: per\n # anomalygen/references/prep-testcase.md, when clean\n # images live at \u003cdataset_dir>/\u003cTEXTURE>/clean_image/,\n # clean_dir defaults to dataset_dir and per-texture lookup\n # works correctly. Forcing --clean-dir to a per-texture path\n # collapses the validator to flat-fallback and mixes clean\n # images across textures.\n DATASET_DIR_ARG=$(dirname \"$USER_SPEC\")\n DEFECT_SPEC_ARG=\"$USER_SPEC\"\n CLEAN_DIR_ARG=()\n echo \"Mode A (prepared dataset): --dataset-dir=$DATASET_DIR_ARG\"\n echo \"--- defect_spec.jsonl ---\"\n cat \"$DEFECT_SPEC_ARG\"\n echo \"--- end defect_spec ---\"\n else\n # Mode B: stage user-uploaded flat submasks under the canonical\n # anomalygen layout. Walks anomaly_types_json (list of\n # [material, defect] pairs) — supports multi-material taxonomies\n # like the shipped PCBA checkpoint (IC + passive_component).\n STAGE=/tmp/inference_stage\n rm -rf \"$STAGE\"\n\n MATERIALS=$(python3 -c '\n import json, sys\n pairs = json.loads(sys.argv[1])\n print(\"\\n\".join(sorted({m for m, _ in pairs})))\n ' \"$ANOMALY_TYPES_JSON\")\n echo \"materials: $(echo \"$MATERIALS\" | tr \"\\n\" \" \")\"\n for MAT in $MATERIALS; do\n mkdir -p \"$STAGE/$MAT/mask\"\n done\n\n # Submasks: per (material, defect), try \u003croot>/\u003cmaterial>/mask/\u003cdefect>/\n # first (canonical anomalygen layout) then \u003croot>/\u003cdefect>/ flat\n # (user upload).\n while IFS=

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

\\t' read -r MAT DEFECT; do\n [ -n \"$MAT\" ] && [ -n \"$DEFECT\" ] || continue\n src=\"\"\n for candidate in \\\n \"$SUBMASK_BASE/$MAT/mask/$DEFECT\" \\\n \"$SUBMASK_BASE/$DEFECT\"; do\n [ -d \"$candidate\" ] && { src=\"$candidate\"; break; }\n done\n [ -n \"$src\" ] || { echo \"ERROR: submask dir not found for $MAT+$DEFECT (tried $SUBMASK_BASE/$MAT/mask/$DEFECT and $SUBMASK_BASE/$DEFECT)\"; exit 1; }\n dst=\"$STAGE/$MAT/mask/$DEFECT\"\n mkdir -p \"$dst\"\n for f in \"$src\"/*.png \"$src\"/*.jpg \"$src\"/*.jpeg; do\n [ -f \"$f\" ] && ln -sf \"$f\" \"$dst/$(basename \"$f\")\" || true\n done\n count=$(ls \"$dst\" 2>/dev/null | wc -l)\n echo \"submasks/$MAT+$DEFECT: $count files (from $src)\"\n [ \"$count\" -gt 0 ] || { echo \"ERROR: no submask files in $src\"; exit 1; }\n done \u003c \u003c(python3 -c 'import json,sys\n for m,d in json.loads(sys.argv[1]): print(f\"{m}\\t{d}\")' \"$ANOMALY_TYPES_JSON\")\n\n python3 /tmp/render_defect_spec.py \\\n --output \"$STAGE/defect_spec.jsonl\" \\\n --pairs \"$ANOMALY_TYPES_JSON\" \\\n --spatial-dependency \"$DEFAULT_SPATIAL_DEPENDENCY\"\n\n DATASET_DIR_ARG=\"$STAGE\"\n DEFECT_SPEC_ARG=\"$STAGE/defect_spec.jsonl\"\n CLEAN_DIR_ARG=(--clean-dir \"$CLEAN_DIR\")\n echo \"Staged upload: --dataset-dir=$STAGE\"\n echo \"--- defect_spec.jsonl ---\"\n cat \"$DEFECT_SPEC_ARG\"\n echo \"--- end defect_spec ---\"\n fi\n\n # Phase 2: AMP routing + JSONL prep. n_seeds is auto-computed\n # from num_sdg / total submasks; do NOT pass --seeds.\n echo \"=== prep_testcase.sh (num_sdg=$NUM_SDG) ===\"\n bash \"$SCRIPTS/prep_testcase.sh\" \\\n --name \"${EXP_NAME}_infer\" \\\n --num-sdg \"$NUM_SDG\" \\\n --dataset-dir \"$DATASET_DIR_ARG\" \\\n \"${CLEAN_DIR_ARG[@]}\" \\\n --defect-spec \"$DEFECT_SPEC_ARG\" \\\n --amp-output-dir \"$AMP_OUT\" \\\n --output-jsonl \"$JSONL\"\n\n # Phase 3: cross-check JSONL anomaly types against checkpoint.\n python3 \"$SCRIPTS/validate_jsonl.py\" \"$WRAPPER\" \"$JSONL\"\n\n # Phase 3: SDG. run_sdg.sh picks the right experiment for model_size.\n export IMAGINAIRE_OUTPUT_ROOT=\"${OUTPUT_DIR}/results\"\n mkdir -p \"$IMAGINAIRE_OUTPUT_ROOT\"\n\n echo \"=== run_sdg.sh (checkpoint_step=$CHECKPOINT_STEP, model_size=$MODEL_SIZE, num_gpus=$NUM_GPUS) ===\"\n bash \"$SCRIPTS/run_sdg.sh\" \\\n --checkpoint_dir \"$WRAPPER\" \\\n --step \"$CHECKPOINT_STEP\" \\\n --input_jsonl \"$JSONL\" \\\n --output_dir \"$OUTPUT_DIR\" \\\n --model_size \"$MODEL_SIZE\" \\\n --num_gpus \"$NUM_GPUS\" \\\n --seed 0\n\n # Verify SDG output completeness before declaring success.\n bash \"$SCRIPTS/verify_output.sh\" \"$JSONL\" \"$OUTPUT_DIR\"\n echo \"=== Inference complete: $(ls \"$OUTPUT_DIR/reconstructed_image/\" 2>/dev/null | wc -l) images ===\"\n outputs:\n - url: \"{{ dig_url_root }}/runs/{{ name }}/anomaly\"\n\ndefault-values:\n workflow_name: texture_defect_gen_day1_real_alignment\n exec_timeout: 20h\n queue_timeout: 2h\n\n # `name` has no default — every submit must pass `--set name=\u003cflow>-$STAMP`\n # (see SKILL.md §\"Name stamping\"). Forces unique storage paths under runs/.\n dig_url_root: \"s3://osmo-workflows/dig\"\n usecase: pcb\n\n # ── usd2roi-render-day1: scene + per-board cookbook selection ─────────────────\n # Pick the per-board cookbook under assets/cookbooks/pcb/\u003cboard>/.\n # 0603_H100 pairs with the shipped spark scene + input_real_image/0603_H100.jpg\n # inside the canonical pcb-assets. To use a different board, add its cookbook\n # directory + its corresponding input_real_image/\u003cboard>.jpg and pass\n # --set board=\u003cdir-name>.\n board: \"0603_H100\" # alternate shipped board: 1152819000\n scene_filename: \"spark_lighting.usd\" # USD inside pcb-assets to use as the scene\n real_image_filename: \"input_real_image/0603_H100.jpg\" # real photo inside pcb-assets; override alongside --set board=...\n\n # ── Finetune-or-checkpoint toggle ────────────────────────────────────────────\n # Defaults to passthrough against {{ dig_url_root }}/models/pcb.\n use_pretrained_checkpoint: \"true\" # true = skip finetune, use models/\u003cusecase> directly\n checkpoint_step: \"14000\" # iter shipped in the HF PCB checkpoint — PCBA default; auto-swapped to 9000 for usecase=glass and 10000 for usecase=metal_surface when not overridden\n anomaly_types_json: '[[\"passive_component\",\"missing\"]]' # PCBA default; auto-swapped per-usecase below when not overridden\n\n # ── Submask inputs ───────────────────────────────────────────────────────────\n # Per-defect submasks (and optional defect_spec.jsonl +\n # semantic_segmentation_labels.json) are read from datasets/\u003cusecase>/raw,\n # which is shipped in canonical layout by the setup/setup_\u003ccase>.yaml workflows.\n\n # ── Real-photo alignment inputs (always on in this spec) ─────────────────────\n # The USD asset tree AND the AOI screenshot are both read from\n # datasets/pcb/assets (canonical pcb-assets ships input_real_image/\u003cboard>.jpg\n # for each shipped board).\n use_usd2roi_day1: \"true\" # always-on in this spec (use _manual_roi variant for non-aligned ROIs)\n usd2roi_image: \"nvcr.io/nvidia/paidf-simulation:1.0.0\"\n\n # ── Inference parameters ─────────────────────────────────────────────────────\n num_sdg: \"30\" # total SDG entries across all defects (prep_testcase auto-scales n_seeds)\n # Only consulted when the inference URL has no defect_spec.jsonl at root\n # (staged-upload fallback). The prepared-dataset path — the default PCBA path —\n # uses the shipped spec and ignores this knob. Set to `cad` to match the shipped\n # PCBA story; users who supply a flat custom upload without cad_masks should\n # flip to `free`.\n default_spatial_dependency: \"cad\" # one of free|text|cad\n model_size: \"2b\" # 2b or 14b; must match the checkpoint\n\n # Cluster knobs only — training-recipe knobs (lr, max_iter, anomaly_types, etc.)\n # live in the per-usecase cookbook at assets/cookbooks/\u003cusecase>/ag_config.yaml and\n # are inherited from the shipped checkpoint's config unless yq overrides them at\n # render time.\n anomalygen_image: \"nvcr.io/nvidia/paidf-anomalygen:1.0.0\"\n # train_gpu / infer_gpu — scale by passing --set train_gpu=N infer_gpu=N at\n # submit time (set them individually to break symmetry, e.g. 8-GPU train,\n # 1-GPU infer).\n\n # ── Resources ────────────────────────────────────────────────────────────────\n render_gpu: \"1\" # usd2roi day-1 only\n render_cpu: \"4\"\n render_memory: 32Gi\n render_storage: 50Gi\n # infer_gpu / infer_cpu / infer_memory — defaults sized for 1 GPU.\n # Scale all three together at submit time, e.g.\n # --set infer_gpu=4 infer_cpu=16 infer_memory=192Gi\n # See references/gpu_sizing.md for the full per-GPU table.\n infer_gpu: \"1\"\n infer_cpu: \"4\"\n infer_memory: 64Gi\n infer_storage: 200Gi\n # train_gpu / train_cpu / train_memory — defaults sized for 1 GPU.\n # Scale all three together at submit time, e.g.\n # --set train_gpu=4 train_cpu=32 train_memory=192Gi\n # See references/gpu_sizing.md for the full per-GPU table and reasoning\n # (cosmos-predict2-2B rank ~33 GiB host RAM during DDP sync, etc.).\n train_gpu: \"1\"\n train_cpu: \"16\"\n train_memory: 64Gi\n train_storage: 300Gi\n platform: default\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":52665,"content_sha256":"7b1d575e5b37203570210c2afcf862bc027a77f8c1c6c4cda26bad830f1fd316"},{"filename":"assets/cookbooks/glass/ag_config.yaml","content":"# DDP config template (default, use with predict2_anomaly_gen_ddp_{2b,14b}).\n# Placeholders in \u003cangle-brackets> are filled by scripts/generate_config.py.\n# Keep sections minimal — the experiment provides defaults for everything else.\njob:\n project: anomaly_gen\n group: UC3_full\n name: UC3_full_training_FP32_lr0.02_bs=2_2B_512x512\noptimizer:\n lr: 0.02\ncheckpoint:\n save_iter: 3000\ntrainer:\n max_iter: 10000\n logging_iter: 10\n validation_iter: 3000\n run_validation: True\n early_stop:\n enabled: false\n metric: nn\n patience: 5\n min_delta: 0\n min_delta_mode: rel\ndataloader_train:\n batch_size: 2\n dataset:\n dataset_dir: /workspace/paidf-anomalygen/datasets/UC3_data\n image_size:\n - 512\n - 512\n anomaly_types: [[Phone, oil], [Phone, scratch], [Phone, stain]]\n seed: 1\n data_augprob: 0.5\n aug_type: random_ratio_crop\n ratio_range: [1.5, 8.0]\ndataloader_val:\n batch_size: 32\n dataset:\n input_data_path: ag_inference/validation_UC3_full/testcase.jsonl\nmodel:\n config:\n ag_config:\n ad_precision: float32\n t5_model_name: checkpoints/google-t5/t5-large\n anomaly_embedding:\n anomaly_types: [[Phone, oil], [Phone, scratch], [Phone, stain]]\n freeze: False\n mask_encoder:\n encoder_config:\n init_cfg:\n checkpoint: checkpoints/NVDINOV2/nv_dinov2_classification_model.ckpt\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":1385,"content_sha256":"766e0b026bdb60699e4d97ba71e22be88d44d988682fdc9ecc80b59d4b55bc7e"},{"filename":"assets/cookbooks/metal_surface/ag_config.yaml","content":"# DDP config template (default, use with predict2_anomaly_gen_ddp_{2b,14b}).\n# Placeholders in \u003cangle-brackets> are filled by scripts/generate_config.py.\n# Keep sections minimal — the experiment provides defaults for everything else.\njob:\n project: anomaly_gen\n group: UC2_exp\n name: UC2_exp_training_FP32_lr0.02_bs=2_2B_512x512\noptimizer:\n lr: 0.02\ncheckpoint:\n save_iter: 2000\ntrainer:\n max_iter: 2000\n logging_iter: 10\n validation_iter: 2000\n run_validation: True\n early_stop:\n enabled: true\n metric: nn\n patience: 5\n min_delta: 0\n min_delta_mode: rel\ndataloader_train:\n batch_size: 2\n dataset:\n dataset_dir: datasets/UC2_data\n image_size:\n - 512\n - 512\n anomaly_types: [[metal_surface, MT_Blowhole], [metal_surface, MT_Break], [metal_surface, MT_Crack],\n [metal_surface, MT_Fray], [metal_surface, MT_Uneven]]\n seed: 1\n data_augprob: 0.5\n aug_type: random_ratio_crop\n ratio_range: [1.5, 8.0]\ndataloader_val:\n batch_size: 32\n dataset:\n input_data_path: ag_inference/validation_UC2_exp/testcase.jsonl\nmodel:\n config:\n ag_config:\n ad_precision: float32\n t5_model_name: checkpoints/google-t5/t5-large\n anomaly_embedding:\n anomaly_types: [[metal_surface, MT_Blowhole], [metal_surface, MT_Break], [metal_surface, MT_Crack],\n [metal_surface, MT_Fray], [metal_surface, MT_Uneven]]\n freeze: False\n mask_encoder:\n encoder_config:\n init_cfg:\n checkpoint: checkpoints/NVDINOV2/nv_dinov2_classification_model.ckpt\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":1536,"content_sha256":"fb9fd312e954ae5fa5c8e742a2fda1f92ceb355184f5b057969f037ae68cd59b"},{"filename":"assets/cookbooks/pcb/0603_H100/day0_crop.yaml","content":"# Demo: 0603_H100_day0\n# Source: _demo_config/0603_H100_day0/day0_crop.yaml\n# Patched at runtime:\n# __OUTPUT__ → OSMO task output dir\n\noutput:\n dir: __OUTPUT__\n\ncrop:\n source: trigger_0000\n pattern: 'x*_y*'\n\n classes: [capacitor]\n\n morph_kernel: 1\n min_area: 50\n max_area: null\n offset: 10\n\n edge_skip: true\n min_coverage: 0.2\n\n max_emit: 10\n\n bridge: false\n\n class_dirs:\n capacitor: passive_component\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":424,"content_sha256":"bf184ebf80753503b2beec42c3196bc7db751108c66db966823d712bcdd718ed"},{"filename":"assets/cookbooks/pcb/0603_H100/day0_image.yaml","content":"# Demo: 0603_H100_day0\n# Source: _demo_config/0603_H100_day0/day0_image.yaml\n# Patched at runtime:\n# __OUTPUT__ → OSMO task output dir\n\npipeline_type: good\n\nseed: 0\nrandom_seed: 0\nmax_image_count: __MAX_IMAGE_COUNT__\nnum_triggers: 1\n\nrename_to_grid_index: true\n\noutput: __OUTPUT__\n\nresolution: [1920, 1080]\n\npathtracing:\n spp: 1\n total_spp: 32\n max_bounces: 4\n\nscan_grid:\n x_num: 10\n y_num: 10\n\nsemantics:\n - match: \"**/PASTEMASK_TOP/Geometry/PASTEMASK_TOP__11450_bodies__0\"\n labels: {class: pad}\n\n - match: \"**/_0603_H100/tn__0603_H100*/LibRef/_0603_H100/Geometry/Components/_0603_H100/surface_1_MASTER_MESH\"\n labels: {class: solder}\n - match: \"**/_0603_H100/tn__0603_H100*/LibRef/_0603_H100/Geometry/Components/_0603_H100/surface_2_MASTER_MESH\"\n labels: {class: capacitor}\n - match: \"**/_0603_H100/tn__0603_H100*/LibRef/_0603_H100/Geometry/Components/_0603_H100/surface_4_MASTER_MESH\"\n labels: {class: capacitor}\n - match: \"**/_0603_H100/tn__0603_H100*/LibRef/_0603_H100/Geometry/Components/_0603_H100/surface_6_MASTER_MESH\"\n labels: {class: capacitor}\n - match: \"**/_0603_H100/tn__0603_H100*/LibRef/_0603_H100/Geometry/Components/_0603_H100/surface_5_MASTER_MESH\"\n labels: {class: solder}\n - match: \"**/_0603_H100/tn__0603_H100*/LibRef/_0603_H100/Geometry/Components/_0603_H100/surface_7_MASTER_MESH\"\n labels: {class: solder}\n - match: \"**/_0603_H100/tn__0603_H100*/LibRef/_0603_H100/Geometry/Components/_0603_H100/surface_8_MASTER_MESH\"\n labels: {class: pad}\n\nuse_scene_lights: true\nvantablack_components: false\n\nlighting:\n ring_light: false\n exposure_range: [0.01, 1.0]\n cone_angle_range: [90, 150]\n cone_softness_range: [0.5, 1.0]\n\n layers:\n Inner_Red:\n intensity: [5000, 8000]\n color_r: [0.95, 1.0]\n color_g: [0.0, 0.05]\n color_b: [0.0, 0.05]\n Middle_Green:\n intensity: [2000, 4000]\n color_r: [0.0, 0.05]\n color_g: [0.95, 1.0]\n color_b: [0.0, 0.05]\n Outer_Blue:\n intensity: [3000, 6000]\n color_r: [0.0, 0.05]\n color_g: [0.0, 0.05]\n color_b: [0.95, 1.0]\n\n white_light:\n intensity: [3000, 8000]\n color_r: [0.8, 1.0]\n color_g: [0.8, 1.0]\n color_b: [0.8, 1.0]\n\ncamera_rotation:\n x_range: [0, 0]\n y_range: [0, 0]\n z_fixed: -90\n\naugmentation:\n motion_blur:\n probability: 0.8\n alpha_range: [0.01, 0.3]\n kernel_choices: [5, 7]\n\nwriter:\n rgb: true\n image_output_format: png\n\n bounding_box_2d_tight: false\n bounding_box_2d_loose: false\n bounding_box_3d: false\n\n semantic_segmentation: true\n instance_id_segmentation: false\n colorize_semantic_segmentation: true\n colorize_instance_id_segmentation: false\n\n distance_to_camera: false\n distance_to_image_plane: false\n colorize_depth: false\n\n semantic_types: [class]\n\n frame_padding: 4\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":2810,"content_sha256":"1fe41e76ee65dbe9339194484725cd672d02ab6609a13a6e37f14ef19bb48afc"},{"filename":"assets/cookbooks/pcb/0603_H100/defect_image.yaml","content":"pipeline_type: defect\n\n# ─── Run controls ──────────────────────────────────────────────────────────\nseed: 0\nrandom_seed: 0\nmax_image_count: 5\nnum_triggers: 1\n\n# ─── Output ────────────────────────────────────────────────────────────────\noutput: ${PCB_DIG_ROOT}/sdg_test_output/flow2_defect_image\n\n# ─── Render ────────────────────────────────────────────────────────────────\nresolution: [1920, 1080]\n\n# ─── PathTracing ───────────────────────────────────────────────────────────\n# Additional knobs (adaptive sampling, denoiser, aa_op, …) — see Omniverse RTX docs.\npathtracing:\n spp: 1\n total_spp: 32\n max_bounces: 4\n\n# ─── Scan Grid ─────────────────────────────────────────────────────────────\nscan_grid:\n x_num: 10\n y_num: 10\n\n# ─── Defects ───────────────────────────────────────────────────────────────\n# Component types eligible for defect / pose ops. Substring-matched against\n# prim names under pcba_root. Mirrors the spark board's full eligible pool from\n# configs/pcba/spark_pcba_target.yaml. sdg_pipeline.py raises\n# `KeyError: 'component_types'` without this explicit list (no `ALL` sentinel\n# handler upstream).\ncomponent_types:\n - _0603_H100\n\n# Each defect type independently selects components from the (remaining) pool.\n# Selection is non-overlapping across types.\ndefects:\n shift:\n enabled: true\n ratio: 0.2\n translate_range: 0.2 # mm; max XY translation (+/-)\n rotate_z_range: 15 # degrees; max Z rotation (+/-)\n tombstone:\n enabled: true\n ratio: 0.2\n angle_min: 70 # degrees; tilt around Y axis\n angle_max: 90\n sideflip:\n enabled: true\n ratio: 0.2\n angle_min: 70 # degrees; flip around X axis\n angle_max: 90\n\n# ─── Top-level material / lighting flags ─────────────────────────────────\nvantablack_components: false\nuse_scene_lights: true\npreserve_scene_light_color: true\n\n# ─── Lighting Randomization ────────────────────────────────────────────────\nlighting:\n ring_light: false\n exposure_range: [0.01, 1.0]\n cone_angle_range: [90, 150]\n cone_softness_range: [0.5, 1.0]\n\n layers:\n Inner_Red:\n intensity: [5000, 8000]\n color_r: [0.95, 1.0]\n color_g: [0.0, 0.05]\n color_b: [0.0, 0.05]\n Middle_Green:\n intensity: [2000, 4000]\n color_r: [0.0, 0.05]\n color_g: [0.95, 1.0]\n color_b: [0.0, 0.05]\n Outer_Blue:\n intensity: [3000, 6000]\n color_r: [0.0, 0.05]\n color_g: [0.0, 0.05]\n color_b: [0.95, 1.0]\n\n white_light:\n intensity: [160, 440] # dome at 20% of dome-calibrated baseline (10% x 2)\n color_r: [0.8, 1.0]\n color_g: [0.8, 1.0]\n color_b: [0.8, 1.0]\n\n# ─── Camera Randomization ──────────────────────────────────────────────────\ncamera_rotation:\n x_range: [0, 0]\n y_range: [0, 0]\n z_fixed: -90\n\n# ─── Augmentation ──────────────────────────────────────────────────────────\naugmentation:\n motion_blur:\n probability: 0.8\n alpha_range: [0.01, 0.3]\n kernel_choices: [5, 7]\n\n# ─── Writer (semantic_types MUST include `defect` for the defect labels) ──\nwriter:\n rgb: true\n image_output_format: png\n\n bounding_box_2d_tight: true\n bounding_box_2d_loose: false\n bounding_box_3d: false\n\n semantic_segmentation: true\n instance_id_segmentation: false\n colorize_semantic_segmentation: true\n colorize_instance_id_segmentation: false\n\n distance_to_camera: false\n distance_to_image_plane: false\n colorize_depth: false\n\n semantic_types: [defect]\n frame_padding: 4\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":4680,"content_sha256":"e75cc86cc72028ee0cb2714754753074387c1df6e8f018078a864a8785d6bb2b"},{"filename":"assets/cookbooks/pcb/0603_H100/pcba_target.yaml","content":"# Demo: 0603_H100_day0\n# Source: _demo_config/0603_H100_day0/pcba_target.yaml\n\nscene: __SCENE__\n\ncamera_path: /World/camera_light/Camera\ncamera_xform_path: /World/camera_light\nring_light_root: /World/camera_light/aoi_ring_light\npcba_root: /World/pcba_main_s_detail/PCBA/tn__60014242BASEA04_fM9E\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":295,"content_sha256":"d5e22b07849d96457de4fe9849a7afcdf0dc800a088c12e0dab77bd24640cc4a"},{"filename":"assets/cookbooks/pcb/0603_H100/usd2roi_nvpcb.yaml","content":"# Demo: 0603_H100_day1\n# Source: _demo_config/0603_H100_day1/usd2roi_nvpcb.yaml\n# Component: _0603_H100/tn__0603_H100_72_ (USD instance, world xy=(-12.616, -1.215, 1.57))\n# Asset URL: datasets/pcb_demo_0603_H100_day1/assets (scene_uninst.usda has _0603_H100_72_ + 11380 pads uninstanced)\n# Patched at runtime by day_1_workflow_demo.yaml:\n# __SCENE__ → /osmo/data/input/0/scene_uninst.usda (set scene_filename=scene_uninst.usda)\n# __REAL_IMAGE__ → /osmo/data/input/0/input_real_image/0603_H100.jpg (set real_image_filename=0603_H100.jpg)\n# __OUTPUT__ → OSMO task output dir\n\nscene: __SCENE__\n\nsemantics:\n - match: \"**/PASTEMASK_TOP/Geometry/PASTEMASK_TOP__11450_bodies__0\"\n labels: {class: pad}\n - match: \"/World/pcba_main_s_detail/PCBA/tn__60014242BASEA04_fM9E/_0603_H100/tn__0603_H100_72_/LibRef/_0603_H100/Geometry/Components/_0603_H100/surface_1_MASTER_MESH\"\n labels: {class: solder}\n - match: \"/World/pcba_main_s_detail/PCBA/tn__60014242BASEA04_fM9E/_0603_H100/tn__0603_H100_72_/LibRef/_0603_H100/Geometry/Components/_0603_H100/surface_2_MASTER_MESH\"\n labels: {class: capacitor}\n - match: \"/World/pcba_main_s_detail/PCBA/tn__60014242BASEA04_fM9E/_0603_H100/tn__0603_H100_72_/LibRef/_0603_H100/Geometry/Components/_0603_H100/surface_4_MASTER_MESH\"\n labels: {class: capacitor}\n - match: \"/World/pcba_main_s_detail/PCBA/tn__60014242BASEA04_fM9E/_0603_H100/tn__0603_H100_72_/LibRef/_0603_H100/Geometry/Components/_0603_H100/surface_6_MASTER_MESH\"\n labels: {class: capacitor}\n - match: \"/World/pcba_main_s_detail/PCBA/tn__60014242BASEA04_fM9E/_0603_H100/tn__0603_H100_72_/LibRef/_0603_H100/Geometry/Components/_0603_H100/surface_5_MASTER_MESH\"\n labels: {class: solder}\n - match: \"/World/pcba_main_s_detail/PCBA/tn__60014242BASEA04_fM9E/_0603_H100/tn__0603_H100_72_/LibRef/_0603_H100/Geometry/Components/_0603_H100/surface_7_MASTER_MESH\"\n labels: {class: solder}\n - match: \"/World/pcba_main_s_detail/PCBA/tn__60014242BASEA04_fM9E/_0603_H100/tn__0603_H100_72_/LibRef/_0603_H100/Geometry/Components/_0603_H100/surface_8_MASTER_MESH\"\n labels: {class: pad}\n\n# === Step 2: Single capture ===\ncamera:\n translate: [-12.616, -1.215]\n horizontal_aperture: 39.28\n rotation_xyz: [0.0, 0.0, 0.0]\nresolution: [720, 540]\n\nwriter:\n rgb: true\n semantic_segmentation: true\n colorize_semantic_segmentation: true\n bounding_box_2d_tight: false\n bounding_box_2d_loose: false\n semantic_types: [class]\n frame_padding: 4\n image_output_format: png\n\n# === Step 3: Registration ===\nreal_image: __REAL_IMAGE__\n\nregistration:\n sx_range: [1.1, 1.3, 0.05]\n sy_range: [1.1, 1.5, 0.05]\n rot_range_deg: [-0.1, 0.1, 0.05]\n shift_range: 100\n shift_step: 2\n pyr_levels: 3\n bins: 64\n no_resize: true\n gpu: true\n min_mi: 0\n\n# === Step 4: Per-ROI label-based crop ===\ncrop:\n classes: [capacitor]\n morph_kernel: 2\n min_area: 50\n max_area: null\n offset: 10\n class_dirs:\n capacitor: passive_component\n bridge: false\n\noutput:\n dir: __OUTPUT__\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":2989,"content_sha256":"8a1630263ff17eaa8c99a4ef09463f8b3bf81cc219d8ddb02ceda7405d22c7a3"},{"filename":"assets/cookbooks/pcb/1152819000/day0_crop.yaml","content":"# Demo: IC_day0\n# Source: _demo_config/IC_day0/day0_crop.yaml\n# Patched at runtime by day_0_workflow_demo.yaml:\n# __OUTPUT__ → OSMO task output dir\n\noutput:\n dir: __OUTPUT__\n\ncrop:\n source: trigger_0000\n pattern: 'x*_y*'\n\n classes: [ic]\n\n morph_kernel: 1\n min_area: 50\n max_area: null\n offset: 10\n\n edge_skip: true\n min_coverage: 0.2\n\n max_emit: null\n\n bridge: false\n\n class_dirs:\n ic: IC\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":411,"content_sha256":"4ff69fde6ae607a73dd6b1c25a50e5baa5689ea00c653d8d57b7379762314901"},{"filename":"assets/cookbooks/pcb/1152819000/day0_image.yaml","content":"# Demo: IC_day0\n# Source: _demo_config/IC_day0/day0_image.yaml\n# Patched at runtime by day_0_workflow_demo.yaml:\n# __OUTPUT__ → OSMO task output dir\n\npipeline_type: good\n\nseed: 0\nrandom_seed: 0\nmax_image_count: __MAX_IMAGE_COUNT__\nnum_triggers: 1\n\nrename_to_grid_index: true\n\noutput: __OUTPUT__\n\nresolution: [1920, 1080]\n\npathtracing:\n spp: 1\n total_spp: 32\n max_bounces: 4\n\nscan_grid:\n x_num: 10\n y_num: 10\n\nsemantics:\n - match: \"**/PASTEMASK_TOP/Geometry/PASTEMASK_TOP__11450_bodies__0\"\n labels: {class: pad}\n\n - match: \"**/tn__60014242BASEA04_fM9E/_115_2819_000_1/tn__1152819000_1SC70*/LibRef/tn__1152819000_1_zH8/surface_0\"\n labels: {class: ic}\n - match: \"**/tn__60014242BASEA04_fM9E/_115_2819_000_1/tn__1152819000_1SC70*/LibRef/tn__1152819000_1_zH8/surface_1\"\n labels: {class: solder}\n - match: \"**/tn__60014242BASEA04_fM9E/_115_2819_000_1/tn__1152819000_1SC70*/LibRef/tn__1152819000_1_zH8/surface_2\"\n labels: {class: ic}\n\nuse_scene_lights: true\nvantablack_components: false\n\nlighting:\n ring_light: false\n exposure_range: [0.01, 1.0]\n cone_angle_range: [90, 150]\n cone_softness_range: [0.5, 1.0]\n\n layers:\n Inner_Red:\n intensity: [5000, 8000]\n color_r: [0.95, 1.0]\n color_g: [0.0, 0.05]\n color_b: [0.0, 0.05]\n Middle_Green:\n intensity: [2000, 4000]\n color_r: [0.0, 0.05]\n color_g: [0.95, 1.0]\n color_b: [0.0, 0.05]\n Outer_Blue:\n intensity: [3000, 6000]\n color_r: [0.0, 0.05]\n color_g: [0.0, 0.05]\n color_b: [0.95, 1.0]\n\n white_light:\n intensity: [3000, 8000]\n color_r: [0.8, 1.0]\n color_g: [0.8, 1.0]\n color_b: [0.8, 1.0]\n\ncamera_rotation:\n x_range: [0, 0]\n y_range: [0, 0]\n z_fixed: -90\n\naugmentation:\n motion_blur:\n probability: 0.8\n alpha_range: [0.01, 0.3]\n kernel_choices: [5, 7]\n\nwriter:\n rgb: true\n image_output_format: png\n\n bounding_box_2d_tight: false\n bounding_box_2d_loose: false\n bounding_box_3d: false\n\n semantic_segmentation: true\n instance_id_segmentation: false\n colorize_semantic_segmentation: true\n colorize_instance_id_segmentation: false\n\n distance_to_camera: false\n distance_to_image_plane: false\n colorize_depth: false\n\n semantic_types: [class]\n\n frame_padding: 4\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":2258,"content_sha256":"d76275725abe49db6e09dd3ca07f196c904042f4040201ff3ee68186ec611ba1"},{"filename":"assets/cookbooks/pcb/1152819000/defect_image.yaml","content":"pipeline_type: defect\n\n# ─── Run controls ──────────────────────────────────────────────────────────\nseed: 0\nrandom_seed: 0\nmax_image_count: -1\nnum_triggers: 1\n\n# ─── Output ────────────────────────────────────────────────────────────────\noutput: ${PCB_DIG_ROOT}/sdg_test_output/flow2_defect_image\n\n# ─── Render ────────────────────────────────────────────────────────────────\nresolution: [1920, 1080]\n\n# ─── PathTracing ───────────────────────────────────────────────────────────\npathtracing:\n spp: 1\n total_spp: 32\n max_bounces: 4\n\n# ─── Scan Grid ─────────────────────────────────────────────────────────────\nscan_grid:\n x_num: 10\n y_num: 10\n\n# ─── Defects ───────────────────────────────────────────────────────────────\n# Board-specific: 1152819000 — larger board geometry, distinct scan_grid\n# and lighting defaults from the 0603_H100 spark board.\n# Substring-matched against prim names under pcba_root. IC-only pool: shifts\n# the actual 1152819000 chip-under-test, NOT the surrounding passives.\ncomponent_types:\n - _115_2819_000_1 # IC under test (substring matches the per-board chip prim)\n\ndefects:\n shift:\n enabled: true\n ratio: 1.0 # only 1 IC instance per board — must be 1.0 to hit it every frame\n translate_range: 0.2 # mm; max XY translation (+/-)\n rotate_z_range: 15 # degrees; max Z rotation (+/-)\n tombstone:\n enabled: false\n ratio: 0.2\n angle_min: 70\n angle_max: 90\n sideflip:\n enabled: false\n ratio: 0.2\n angle_min: 70\n angle_max: 90\n\n# ─── Top-level material / lighting flags ─────────────────────────────────\nvantablack_components: false\nuse_scene_lights: true\npreserve_scene_light_color: true\n\n# ─── Lighting Randomization ────────────────────────────────────────────────\nlighting:\n ring_light: false\n exposure_range: [0.01, 1.0]\n cone_angle_range: [90, 150]\n cone_softness_range: [0.5, 1.0]\n\n layers:\n Inner_Red:\n intensity: [5000, 8000]\n color_r: [0.95, 1.0]\n color_g: [0.0, 0.05]\n color_b: [0.0, 0.05]\n Middle_Green:\n intensity: [2000, 4000]\n color_r: [0.0, 0.05]\n color_g: [0.95, 1.0]\n color_b: [0.0, 0.05]\n Outer_Blue:\n intensity: [3000, 6000]\n color_r: [0.0, 0.05]\n color_g: [0.0, 0.05]\n color_b: [0.95, 1.0]\n\n white_light:\n intensity: [160, 440]\n color_r: [0.8, 1.0]\n color_g: [0.8, 1.0]\n color_b: [0.8, 1.0]\n\n# ─── Camera Randomization ──────────────────────────────────────────────────\ncamera_rotation:\n x_range: [0, 0]\n y_range: [0, 0]\n z_fixed: -90\n\n# ─── Augmentation ──────────────────────────────────────────────────────────\naugmentation:\n motion_blur:\n probability: 0.8\n alpha_range: [0.01, 0.3]\n kernel_choices: [5, 7]\n\n# ─── Writer (semantic_types MUST include `defect` for the defect labels) ──\nwriter:\n rgb: true\n image_output_format: png\n\n bounding_box_2d_tight: true\n bounding_box_2d_loose: false\n bounding_box_3d: false\n\n semantic_segmentation: true\n instance_id_segmentation: false\n colorize_semantic_segmentation: true\n colorize_instance_id_segmentation: false\n\n distance_to_camera: false\n distance_to_image_plane: false\n colorize_depth: false\n\n semantic_types: [defect]\n frame_padding: 4","content_type":"application/yaml; charset=utf-8","language":"yaml","size":4451,"content_sha256":"e0a8b1ca6bf175de59eb5cb87a2b01805ecd84d3a76f7ac6c217065c56fcbf9f"},{"filename":"assets/cookbooks/pcb/1152819000/pcba_target.yaml","content":"# Demo: IC_day0\n# Source: _demo_config/IC_day0/pcba_target.yaml\n\nscene: __SCENE__\n\ncamera_path: /World/camera_light/Camera\ncamera_xform_path: /World/camera_light\nring_light_root: /World/camera_light/aoi_ring_light\npcba_root: /World/pcba_main_s_detail/PCBA/tn__60014242BASEA04_fM9E\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":281,"content_sha256":"480efe9a333950e4c8ab104d16f4317699cd5fa7f603fab4d36782563970df87"},{"filename":"assets/cookbooks/pcb/1152819000/usd2roi_nvpcb.yaml","content":"# Demo: specific_component_day1\n# Source: _demo_config/specific_component_day1/usd2roi_nvpcb.yaml\n# Patched at runtime by day_1_workflow_demo.yaml:\n# __SCENE__ → temp_scene.usd discovered under datasets/pcb_demo_specific_day1/assets\n# __REAL_IMAGE__ → input_real_image/115_2819_000.jpg under the assets URL\n# __OUTPUT__ → OSMO task output dir\n\nscene: __SCENE__\n\n# === Step 1: Semantic rules ===\n# Glob: `**` = any chars (incl. /); `*` = within one segment; `?` = single char.\nsemantics:\n - match: \"**/PASTEMASK_TOP/Geometry/PASTEMASK_TOP__11450_bodies__0\"\n labels: {class: pad}\n - match: \"/World/pcba_main_s_detail/PCBA/tn__60014242BASEA04_fM9E/_115_2819_000_1/tn__1152819000_1SC70_5_2_nT8nAc1/LibRef/tn__1152819000_1_zH8/surface_0\"\n labels: {class: ic}\n - match: \"/World/pcba_main_s_detail/PCBA/tn__60014242BASEA04_fM9E/_115_2819_000_1/tn__1152819000_1SC70_5_2_nT8nAc1/LibRef/tn__1152819000_1_zH8/surface_1\"\n labels: {class: solder}\n - match: \"/World/pcba_main_s_detail/PCBA/tn__60014242BASEA04_fM9E/_115_2819_000_1/tn__1152819000_1SC70_5_2_nT8nAc1/LibRef/tn__1152819000_1_zH8/surface_2\"\n labels: {class: ic}\n\n# === Step 2: Single capture (demo's original close-up values) ===\ncamera:\n translate: [-103.5, -77.601]\n horizontal_aperture: 48.85\n rotation_xyz: [0.0, 0.0, 0.0]\nresolution: [720, 540]\n\nwriter:\n rgb: true\n semantic_segmentation: true\n colorize_semantic_segmentation: true\n bounding_box_2d_tight: false\n bounding_box_2d_loose: false\n semantic_types: [class] # must list every label key your rules use\n frame_padding: 4\n image_output_format: png\n\n# === Step 3: Registration ===\nreal_image: __REAL_IMAGE__\n\nregistration:\n sx_range: [0.95, 1.1, 0.05] # [min, max, step] — pyramid refines further\n sy_range: [0.95, 1.1, 0.05] # [min, max, step] — pyramid refines further\n rot_range_deg: [-0.1, 0.1, 0.05] # tight: real PCBA image has near-zero rotation\n shift_range: 100 # ± px for tx/ty\n shift_step: 2 # px (coarse-level; pyramid refines)\n pyr_levels: 3 # coarse-to-fine; speeds up 1920×1080 grid scan\n bins: 64 # histogram bins for MI\n no_resize: true # skip pre-resize; preserve sub-pixel detail\n gpu: true # auto / true / false\n min_mi: 0 # exit 2 if MI after \u003c this; null = disable\n\n# === Step 4: Per-ROI label-based crop ===\ncrop:\n classes: [ic] # which classes to extract ROIs for\n morph_kernel: 2 # px close radius — merge adjacent labelled meshes\n min_area: 50 # px²; reject smaller connected components\n max_area: null # px²; null = no upper cap\n offset: 10 # px padding around each ROI bbox\n\n # Per-class output routing. NNNN counter restarts per subdir.\n class_dirs:\n ic: IC\n\n # Optional pairwise crops covering two nearby ROIs in one bbox.\n bridge: false # turn pairwise bridge crops on\n\n# === Output ===\noutput:\n dir: __OUTPUT__\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":3276,"content_sha256":"15c5eee80d96f3d2690733017fccf6655f7e17a0f403bcbb246fb4a1f03aac47"},{"filename":"assets/cookbooks/pcb/ag_config.yaml","content":"# DDP config template (default, use with predict2_anomaly_gen_ddp_{2b,14b}).\n# Placeholders in \u003cangle-brackets> are filled by scripts/generate_config.py.\n# Keep sections minimal — the experiment provides defaults for everything else.\njob:\n project: anomaly_gen\n group: UC1_exp_75k_val2k\n name: UC1_exp_75k_val2k_training_FP32_lr0.02_bs=2_2B_512x512\noptimizer:\n lr: 0.02\ncheckpoint:\n save_iter: 2000\ntrainer:\n max_iter: 2000\n logging_iter: 10\n validation_iter: 2000\n run_validation: True\n early_stop:\n enabled: false\n metric: nn\n patience: 5\n min_delta: 0\n min_delta_mode: rel\ndataloader_train:\n batch_size: 2\n dataset:\n dataset_dir: datasets/UC1_data\n image_size:\n - 512\n - 512\n anomaly_types: [[IC, bridge], [passive_component, excess_solder], [passive_component, missing]]\n seed: 1\n data_augprob: 0.5\n aug_type: random_ratio_crop\n ratio_range: [1.5, 8.0]\ndataloader_val:\n batch_size: 32\n dataset:\n input_data_path: ag_inference/validation_UC1_exp/testcase.jsonl\nmodel:\n config:\n ag_config:\n ad_precision: float32\n t5_model_name: checkpoints/google-t5/t5-large\n anomaly_embedding:\n anomaly_types: [[IC, bridge], [passive_component, excess_solder], [passive_component, missing]]\n freeze: False\n mask_encoder:\n encoder_config:\n init_cfg:\n checkpoint: checkpoints/NVDINOV2/nv_dinov2_classification_model.ckpt\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":1437,"content_sha256":"63b6ba1929ef090bca56e44e27b42dd09e7b1e574c0d6cb02fe7daeac4ecd880"},{"filename":"assets/cookbooks/pcb/augmentation_config_ovsl2sl.yaml","content":"# Image edit (Qwen) — single-image PCBA augmentation, OVSL2SL-style static prompt.\n# Mirrors configs/config_finetuned_CT_OVSL2SL.yaml (data sample, prompt) but\n# routes through the image-edit augmentation model against a Qwen Image Edit\n# endpoint instead of the cosmos-transfer2.5 OVSL2SL checkpoint.\n\ndata:\n - inputs:\n rgb: \"/app/data/ovsl2sl/input.jpg\"\n output:\n video: \"/app/data/ovsl2sl/output/output.jpg\"\n caption: \"/app/data/ovsl2sl/output/output.txt\"\n metadata: \"/app/data/ovsl2sl/output/metadata.json\"\n\nendpoints:\n image_edit:\n url: \"http://localhost:8000/v1\"\n model: \"nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL\"\n\npipeline:\n retry: 3\n regenerate_caption_on_retry: false # Static text prompt — nothing to regenerate.\n logging:\n enabled: true\n level: \"INFO\"\n\ncaptioning:\n llm:\n text: |\n Render this PCB component crop in the style of an NVPCB raked-solder-light\n photograph: dark reddish board with bright orange-red and blue specular\n highlights on the solder pads, photorealistic textures.\n\naugmentation:\n model:\n name: \"image-edit\"\n executor_type: \"gradio\"\n parameters:\n num_inference_steps: 30\n guidance_scale: 4.0\n seed: 0\n\ndata_processing:\n alignment:\n # MI-registration: warps the generated image back into the input frame\n # so downstream tools see a pixel-aligned output.\n #\n # Auto-derived from the generated and reference image dimensions\n # (set explicitly to override):\n # sx_range, sy_range - scale-X / scale-Y search ranges\n # shift_range - tx/ty pixel search half-range\n # pyr_levels - image pyramid levels\n\n # Range of rotations (in degrees) to search through.\n # Format: [min, max, step]. The model typically tilts \u003c= 1 degree.\n rot_range_deg: [-1.0, 1.0, 0.1]\n\n # Pixel resolution of the translation (tx/ty) grid search.\n # 1 = finest and most accurate but slowest; 2-4 are usually fine.\n shift_step: 2\n\n # Number of histogram bins used to score alignment quality (mutual\n # information). 64 is a good default for 8-bit images.\n bins: 64\n\n # How pixels are sampled when warping the generated image:\n # bilinear - smooth interpolation; best for natural photos / RGB\n # nearest - exact pixel values; use for masks or label maps\n interp: bilinear\n\n # If true, do not pre-resize the generated image down to the reference\n # frame; trust the scale search to recover the right ratio.\n # Leave as true for typical use.\n no_resize: true\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":2536,"content_sha256":"b1674af43c35a97d0357fd870294228a83da913d2c297aef9acc83240e1ad73c"},{"filename":"assets/cookbooks/pcb/usd2roi_day1.yaml","content":"# usd2roi day-1 single-config — USD + real PCBA photo → per-ROI synth+real+seg triples.\n#\n# Mirrors the usd2roi standalone reference at\n# skills/simulation/assets/configs/day1/replicator/usd2roi_target.yaml\n# (spark scene). The workflow patches three runtime sentinels:\n# __SCENE__ scene USD found inside \u003cdig_url_root>/datasets/pcb/assets\n# __REAL_IMAGE__ real photo `input_real_image/\u003cboard>.jpg` inside \u003cdig_url_root>/datasets/pcb/assets\n# __OUTPUT__ OSMO task output dir\n#\n# Camera / resolution / registration / crop values are tuned for the shipped\n# spark PCBA scene. For a different board: edit in place; check\n# references/troubleshooting.md (usd2roi MI alignment section) before changing\n# registration ranges.\n\nscene: __SCENE__\n\n# Inline semantics — must produce class names matching the shipped checkpoint's\n# anomaly_types taxonomy. The default crop.classes below (capacitor, solder,\n# pad, ic) align with the Day 0 cookbook's day0_image.yaml semantics block.\nsemantics:\n # --- 0805U_H150 ---\n - match: \"**/_0805U_H150/tn__0805U_H150_*/LibRef/tn__0805U_H040_/surface_0\"\n labels: {class: solder}\n - match: \"**/_0805U_H150/tn__0805U_H150_*/LibRef/tn__0805U_H040_/surface_1\"\n labels: {class: capacitor}\n\n # --- PASTEMASK pad ---\n - match: \"**/PASTEMASK_TOP/Geometry/PASTEMASK_TOP__11450_bodies__0\"\n labels: {class: pad}\n\n # --- 0402_H060 ---\n - match: \"**/_0402_H060/tn__0402_H060_*/LibRef/_0402_H060/Geometry/Components/_0402_H060/surface_0_MASTER_MESH\"\n labels: {class: solder}\n - match: \"**/_0402_H060/tn__0402_H060_*/LibRef/_0402_H060/Geometry/Components/_0402_H060/surface_1_MASTER_MESH\"\n labels: {class: capacitor}\n - match: \"**/_0402_H060/tn__0402_H060_*/LibRef/_0402_H060/Geometry/Components/_0402_H060/surface_3_MASTER_MESH\"\n labels: {class: capacitor}\n - match: \"**/_0402_H060/tn__0402_H060_*/LibRef/_0402_H060/Geometry/Components/_0402_H060/surface_4_MASTER_MESH\"\n labels: {class: solder}\n - match: \"**/_0402_H060/tn__0402_H060_*/LibRef/_0402_H060/Geometry/Components/_0402_H060/surface_5_MASTER_MESH\"\n labels: {class: capacitor}\n - match: \"**/_0402_H060/tn__0402_H060_*/LibRef/_0402_H060/Geometry/Components/_0402_H060/surface_6_MASTER_MESH\"\n labels: {class: solder}\n\n # --- 0402_H040: surface 0,8 -> pad ; 1,5,7 -> solder ; 2,3,4,6 -> capacitor ---\n - match: \"**/_0402_H040/tn__0402_H040_*/LibRef/_0402_H040/Geometry/Components/_0402_H040/surface_0_MASTER_MESH\"\n labels: {class: pad}\n - match: \"**/_0402_H040/tn__0402_H040_*/LibRef/_0402_H040/Geometry/Components/_0402_H040/surface_8_MASTER_MESH\"\n labels: {class: pad}\n - match: \"**/_0402_H040/tn__0402_H040_*/LibRef/_0402_H040/Geometry/Components/_0402_H040/surface_1_MASTER_MESH\"\n labels: {class: solder}\n - match: \"**/_0402_H040/tn__0402_H040_*/LibRef/_0402_H040/Geometry/Components/_0402_H040/surface_5_MASTER_MESH\"\n labels: {class: solder}\n - match: \"**/_0402_H040/tn__0402_H040_*/LibRef/_0402_H040/Geometry/Components/_0402_H040/surface_7_MASTER_MESH\"\n labels: {class: solder}\n - match: \"**/_0402_H040/tn__0402_H040_*/LibRef/_0402_H040/Geometry/Components/_0402_H040/surface_2_MASTER_MESH\"\n labels: {class: capacitor}\n - match: \"**/_0402_H040/tn__0402_H040_*/LibRef/_0402_H040/Geometry/Components/_0402_H040/surface_3_MASTER_MESH\"\n labels: {class: capacitor}\n - match: \"**/_0402_H040/tn__0402_H040_*/LibRef/_0402_H040/Geometry/Components/_0402_H040/surface_4_MASTER_MESH\"\n labels: {class: capacitor}\n - match: \"**/_0402_H040/tn__0402_H040_*/LibRef/_0402_H040/Geometry/Components/_0402_H040/surface_6_MASTER_MESH\"\n labels: {class: capacitor}\n\n # --- 0402_LARGE_H070 (LibRef path uses _0402_LARGE_H080_) ---\n - match: \"**/_0402_LARGE_H070/tn__0402_LARGE_H070_*/LibRef/tn__0402_LARGE_H080_/surface_0\"\n labels: {class: capacitor}\n - match: \"**/_0402_LARGE_H070/tn__0402_LARGE_H070_*/LibRef/tn__0402_LARGE_H080_/surface_1\"\n labels: {class: solder}\n\n # --- 115-1581-000_2 IC (whole component) ---\n - match: \"**/_115_1581_000_2/tn__1151581000_2DFN06_P065_020X020_T016X010_D_*\"\n labels: {class: ic}\n\n# === Single ortho capture (matches the real photo's aspect) ===\ncamera:\n translate: [-55.4, 2] # [x, y] world coords (mm); z fixed at 5000\n horizontal_aperture: 97 # ortho extent in 0.1 × world units (mm * 10)\nresolution: [640, 480]\n\npathtracing:\n spp: 1\n total_spp: 32\n max_bounces: 4\n\nwriter:\n rgb: true\n semantic_segmentation: true\n colorize_semantic_segmentation: true\n bounding_box_2d_tight: false\n bounding_box_2d_loose: false\n semantic_types: [class]\n frame_padding: 4\n image_output_format: png\n\n# === Solder-light illumination (matches the AOI machine \"solder light\" mode) ===\n# usd2roi_render.py may or may not consume this block — the usd2roi day-1 spark\n# template ships without it. Added here to try to push MI above the 0.5 gate\n# against the shipped PCBA assets URL's `temp_scene.usd`. The Inner_Red /\n# Middle_Green / Outer_Blue cones produce the colored specular pattern on\n# solder fillets that the real AOI photo carries.\nlighting:\n ring_light: true\n exposure_range: [0.01, 1.0]\n cone_angle_range: [90, 150]\n cone_softness_range: [0.5, 1.0]\n\n layers:\n Inner_Red:\n intensity: [5000, 8000]\n color_r: [0.95, 1.0]\n color_g: [0.0, 0.05]\n color_b: [0.0, 0.05]\n Middle_Green:\n intensity: [2000, 4000]\n color_r: [0.0, 0.05]\n color_g: [0.95, 1.0]\n color_b: [0.0, 0.05]\n Outer_Blue:\n intensity: [3000, 6000]\n color_r: [0.0, 0.05]\n color_g: [0.0, 0.05]\n color_b: [0.95, 1.0]\n\n white_light:\n intensity: [3000, 8000]\n color_r: [0.8, 1.0]\n color_g: [0.8, 1.0]\n color_b: [0.8, 1.0]\n\n# === Registration (USD synth → real photo via mutual information) ===\nreal_image: __REAL_IMAGE__\n\nregistration:\n sx_range: [0.9, 1.1, 0.02] # [min, max, step] — search range for X scale\n sy_range: [0.9, 1.1, 0.02] # [min, max, step] — search range for Y scale\n rot_range_deg: [-0.1, 0.1, 0.05] # tight: AOI machine photos have near-zero rotation\n shift_range: 100 # ± px for tx/ty\n shift_step: 1 # px\n pyr_levels: 3 # image pyramid levels (coarse → fine)\n bins: 64 # histogram bins for MI\n no_resize: true # skip pre-resize; preserve sub-pixel detail\n gpu: true\n min_mi: 0.5 # exit 2 if MI after \u003c this; null = disable\n\n# === Per-ROI label-based crop ===\ncrop:\n classes: [capacitor, ic] # which classes to extract ROIs for\n class_dirs: # route usd2roi class → workflow material dir name\n capacitor: passive_component\n ic: IC\n morph_kernel: 1 # px close radius — merge adjacent labelled meshes\n min_area: 50 # px²; reject smaller connected components\n max_area: null # px²; null = no upper cap\n offset: 10 # px padding around each ROI bbox\n\n bridge: false # turn pairwise bridge crops on/off\n bridge_dis: 30 # px; bbox edge-to-edge distance threshold\n bridge_classes: [capacitor] # only ROIs whose dominant_class is in this list pair\n\noutput:\n dir: __OUTPUT__\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":7551,"content_sha256":"5276d021bd3ebf29cca4a49f281376d2d9ff72dbd41a30c84b95a2229a517bdd"},{"filename":"BENCHMARK.md","content":"# Evaluation Report\n\nEvaluation of the `physical-ai-defect-image-generation` skill before publication through NVSkills-Eval.\n\nThis benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.\n\n## Evaluation Summary\n\n- Skill: `physical-ai-defect-image-generation`\n- Evaluation date: 2026-06-23\n- NVSkills-Eval profile: `external`\n- Environment: `astra-sandbox`\n- Dataset: 6 evaluation tasks\n- Attempts per task: 1\n- Pass threshold: 50%\n- Overall verdict: PASS\n\n## Agents Used\n\n- `claude-code`\n- `codex`\n\n## Metrics Used\n\nReported benchmark dimensions:\n\n- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.\n- Correctness: checks whether the agent follows the expected workflow and produces the correct final output.\n- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.\n- Effectiveness: checks whether the agent performs measurably better with the skill than without it.\n- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.\n\nUnderlying evaluation signals used in this run:\n\n- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access.\n- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow.\n- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage.\n- `accuracy` (Accuracy): grades final-answer correctness against the reference answer.\n- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully.\n- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations.\n- `token_efficiency` (Token Efficiency): compares token usage with and without the skill.\n\n## Test Tasks\n\nThe benchmark dataset contained 6 evaluation tasks:\n\n- Positive tasks: 6 tasks where the skill was expected to activate.\n- Negative tasks: 0 tasks where no skill was expected.\n- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.\n\nTask composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases.\n\n## Results\n\n| Dimension | Num | `claude-code` | `codex` |\n|---|---:|---:|---:|\n| Security | 6 | 100% (+0%) | 100% (+0%) |\n| Correctness | 6 | 94% (+71%) | 94% (+52%) |\n| Discoverability | 6 | 95% (+70%) | 79% (+33%) |\n| Effectiveness | 6 | 79% (+77%) | 90% (+76%) |\n| Efficiency | 6 | 78% (+51%) | 69% (+33%) |\n\nScore values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.\n\n## Tier 1: Static Validation Summary\n\nTier 1 validation passed with observations. NVSkills-Eval ran 1 checks and found 3 total findings.\n\nTop findings:\n\n- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/physical-ai-defect-image-generation/SKILL.md`)\n- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/physical-ai-defect-image-generation/SKILL.md`)\n- LOW SCHEMA/author_format: Author must be of the form 'Name \u003cemail@host>' (`skills/physical-ai-defect-image-generation/SKILL.md`)\n\n## Tier 2: Deduplication Summary\n\nThis tier was not run or did not produce findings in this report.\n\n## Publication Recommendation\n\nThe skill is suitable to proceed toward NVSkills-Eval publication based on this benchmark. Skill owners should keep this file with the skill and refresh it when the evaluation dataset, skill behavior, or target agents materially change.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3902,"content_sha256":"0f328fb0c39daffda24faee768191746e9522804cb7014dc751d016954e4a014"},{"filename":"evals/evals.json","content":"[\n {\n \"id\": \"1\",\n \"question\": \"We need to run the full AOI pipeline for our spark PCBA board end-to-end with the shipped PCBA checkpoint to produce annotated anomaly images for IC+bridge, passive_component+excess_solder, and passive_component+missing. The `osmo` CLI is NOT available here \\u2014 do not try to run it. Walk me through the plan I would run on my OSMO-equipped workstation: which workflow YAML, which preflight commands and what they check, and the exact `osmo workflow submit` invocation with every required knob. Emit your final plan as a structured response with these labeled sections: **Workflow** (the YAML file), **Preflights** (exact commands with args), **Required URL Artifacts** (enumerated under `\u003cdig_url_root>`), **Submit Command** (complete `osmo workflow submit ... --set ...` block with every required knob), **Output Location** (`\u003cdig_url_root>/runs/\u003cname>/anomaly/` or per-flow override). Do not omit any section.\",\n \"expected_skill\": \"physical-ai-defect-image-generation\",\n \"ground_truth\": \"Response identifies Day 0 Texture Defects (assets/configs/texture_defect_generation_day0.yaml) with the pcb cookbook. It lists scripts/preflight_credentials.sh and scripts/preflight_urls.sh 0 pcb as the preflight commands, and names the four required URL artifacts under dig_url_root: models/pretrained, models/pcb, datasets/pcb/raw, and datasets/pcb/assets. It provides a complete osmo workflow submit invocation for texture_defect_generation_day0.yaml passing --set dig_url_root, image_edit_endpoint, name=\u003cflow>-$STAMP, checkpoint_step=14000, and anomaly_types_json containing IC+bridge, passive_component+excess_solder, and passive_component+missing. It notes labeled output lands at runs/\u003cname>/anomaly and that missing-component is handled natively by AnomalyGen (not routed to structural).\",\n \"expected_behavior\": [\n \"Response identifies Day 0 Texture Defects as the workflow choice and names assets/configs/texture_defect_generation_day0.yaml.\",\n \"Response lists scripts/preflight_urls.sh 0 pcb as the URL preflight command (and mentions scripts/preflight_credentials.sh separately).\",\n \"Response names the required URL artifacts: models/pretrained, models/pcb, datasets/pcb/raw (NOT datasets/pcb/processed), and datasets/pcb/assets.\",\n \"The submit command passes dig_url_root via --set rather than individual named dataset parameters.\",\n \"The submit command sets checkpoint_step=14000 and anomaly_types_json covering IC+bridge, passive_component+excess_solder, and passive_component+missing.\",\n \"Response notes that missing-component is handled natively by AnomalyGen in this Day 0 Texture flow (not routed to Day 0 Structural).\",\n \"Response describes labeled output landing at runs/\u003cname>/anomaly.\"\n ]\n },\n {\n \"id\": \"2\",\n \"question\": \"I have pre-captured clean metal-surface images and ROI masks already under the DIG root. Show me the plan to generate anomaly training data using the shipped metal-surface checkpoint with no finetuning: which workflow YAML, which preflight commands, and the exact `osmo workflow submit` invocation with every required knob. The `osmo` CLI is NOT available in this environment \\u2014 do not try to execute it; just produce the recipe. Emit your final plan as a structured response with these labeled sections: **Workflow** (the YAML file), **Preflights** (exact commands with args), **Required URL Artifacts** (enumerated under `\u003cdig_url_root>`), **Submit Command** (complete `osmo workflow submit ... --set ...` block with every required knob), **Output Location** (`\u003cdig_url_root>/runs/\u003cname>/anomaly/` or per-flow override). Do not omit any section.\",\n \"expected_skill\": \"physical-ai-defect-image-generation\",\n \"ground_truth\": \"Response selects Day 1 Manual-ROI (assets/configs/texture_defect_generation_day1_manual_roi.yaml) with usecase=metal_surface. It names scripts/preflight_urls.sh 1 metal_surface as the URL preflight and lists the required artifacts: models/pretrained, models/metal_surface, datasets/metal_surface/raw. It keeps use_pretrained_checkpoint=true (default), sets checkpoint_step=10000, and uses the shipped 5-class MT_* anomaly_types_json (MT_Blowhole, MT_Break, MT_Crack, MT_Fray, MT_Uneven, each paired with metal_surface). It explains labeled output lands at \u003cdig_url_root>/runs/\u003cname>/anomaly.\",\n \"expected_behavior\": [\n \"Response identifies Day 1 manual-ROI and names assets/configs/texture_defect_generation_day1_manual_roi.yaml.\",\n \"Response uses usecase=metal_surface (NOT usecase=metal).\",\n \"Response invokes preflight_urls.sh as `1 metal_surface` and lists models/pretrained, models/metal_surface, datasets/metal_surface/raw as the required artifacts.\",\n \"Response keeps use_pretrained_checkpoint=true (the default) and sets checkpoint_step=10000.\",\n \"Response uses anomaly_types_json covering all 5 shipped classes: MT_Blowhole, MT_Break, MT_Crack, MT_Fray, MT_Uneven (each paired with metal_surface).\",\n \"Response describes labeled output at \u003cdig_url_root>/runs/\u003cname>/anomaly.\"\n ]\n },\n {\n \"id\": \"3\",\n \"question\": \"I want to finetune anomalygen on our labeled glass-defect data under the DIG root and produce a checkpoint we can reuse later. Walk me through the plan: workflow YAML, preflight invocation, the exact `osmo workflow submit` command, and where the resulting checkpoint will land. The `osmo` CLI is NOT installed here \\u2014 do not try to run anything, just produce the recipe. Emit your final plan as a structured response with these labeled sections: **Workflow** (the YAML file), **Preflights** (exact commands with args), **Required URL Artifacts** (enumerated under `\u003cdig_url_root>`), **Submit Command** (complete `osmo workflow submit ... --set ...` block with every required knob), **Output Location** (`\u003cdig_url_root>/runs/\u003cname>/anomaly/` or per-flow override). Do not omit any section.\",\n \"expected_skill\": \"physical-ai-defect-image-generation\",\n \"ground_truth\": \"Response selects the Finetune Only flow (assets/configs/finetune.yaml) and refers to it as 'Finetune Only' (NOT 'Flow C'). It names scripts/preflight_urls.sh finetune glass (NOT 'C glass') as the preflight, verifying models/pretrained and datasets/glass/raw. The submit command is for assets/configs/finetune.yaml with --set dig_url_root and --set usecase=glass. Response explains the cookbook at assets/cookbooks/glass/ag_config.yaml is uploaded as a template and rendered in-pod by yq right after Phase 1 Step 2 produces validation.jsonl (NO pre-submit render step). The resulting checkpoint lands under \u003cdig_url_root>/runs/\u003cname>/finetune and can be copied into models/glass or referenced as a checkpoint URL for Day 0/Day 1 runs.\",\n \"expected_behavior\": [\n \"Response selects Finetune Only and refers to it as 'Finetune Only' \\u2014 NOT 'Flow C'.\",\n \"Response invokes preflight_urls.sh as `finetune glass` \\u2014 NOT `C glass`.\",\n \"Response does NOT claim a pre-submit render of workspaces/finetune/ag_config.yaml; rendering happens in-pod via yq after Phase 1 Step 2.\",\n \"The submit command sets dig_url_root and usecase=glass.\",\n \"Response explains the checkpoint lands at \u003cdig_url_root>/runs/\u003cname>/finetune and can be copied into models/glass or referenced as a checkpoint URL for Day 0/Day 1 runs.\"\n ]\n },\n {\n \"id\": \"4\",\n \"question\": \"I need a batch of clean PCBA images on the 0603_H100 board for our ChangeNet positives \\u2014 no defects, just nicely styled clean ROIs. We'll use the local cluster Qwen Image-Edit endpoint. Show me the plan: workflow YAML, preflights, the full `osmo workflow submit` invocation with all required knobs, and the expected output layout. The `osmo` CLI is NOT available in this environment \\u2014 recipe only, do not try to execute. Emit your final plan as a structured response with these labeled sections: **Workflow** (the YAML file), **Preflights** (exact commands with args), **Required URL Artifacts** (enumerated under `\u003cdig_url_root>`), **Submit Command** (complete `osmo workflow submit ... --set ...` block with every required knob), **Output Location** (`\u003cdig_url_root>/runs/\u003cname>/anomaly/` or per-flow override). Do not omit any section.\",\n \"expected_skill\": \"physical-ai-defect-image-generation\",\n \"ground_truth\": \"Response selects Day 0 Good Image (assets/configs/good_image_generation.yaml). It names scripts/preflight_credentials.sh and scripts/preflight_urls.sh 0 pcb as preflights and notes only datasets/pcb/assets is required for this flow (no AnomalyGen models). It provides an osmo workflow submit with --set board=0603_H100, dig_url_root, image_edit_endpoint pointing to the in-cluster qwen-image-edit-nvpcb-ovsl2sl service (per references/nim/), and image_edit_model=nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL. It does NOT set anomaly_types_json or checkpoint_step. It describes outputs at runs/\u003cname>/{usd2roi-components,augment}/ with SL-restyled RGBs under augment/crop/\u003cMAT>/\u003ccell>/.\",\n \"expected_behavior\": [\n \"Response selects Day 0 Good Image (assets/configs/good_image_generation.yaml) \\u2014 NOT Day 0 Texture Defects or Day 0 Structural Defects.\",\n \"Response does NOT set anomaly_types_json or checkpoint_step (there is no AnomalyGen step in this flow).\",\n \"image_edit_model is the OVSL2SL checkpoint nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL (never substituted with the generic qwen-image-edit).\",\n \"board=0603_H100 is set explicitly.\",\n \"Response describes outputs as the usd2roi tree plus augment/crop/\u003cMAT>/\u003ccell>/ SL-restyled RGBs \\u2014 NOT runs/\u003cname>/anomaly (no inference stage in this flow).\"\n ]\n },\n {\n \"id\": \"5\",\n \"question\": \"I need tombstone and shift defect frames for the 0603_H100 board \\u2014 just the pose-perturbation kind, not solder bridges. About 60 of them. Walk me through the plan: workflow YAML, preflights, the exact `osmo workflow submit` invocation with the sizing knob set correctly, and the expected output dirs. The `osmo` CLI is NOT installed here \\u2014 recipe only. Emit your final plan as a structured response with these labeled sections: **Workflow** (the YAML file), **Preflights** (exact commands with args), **Required URL Artifacts** (enumerated under `\u003cdig_url_root>`), **Submit Command** (complete `osmo workflow submit ... --set ...` block with every required knob), **Output Location** (`\u003cdig_url_root>/runs/\u003cname>/anomaly/` or per-flow override). Do not omit any section.\",\n \"expected_skill\": \"physical-ai-defect-image-generation\",\n \"ground_truth\": \"Response selects Day 0 Structural Defects (assets/configs/structural_defect_generation.yaml). It names scripts/preflight_credentials.sh and scripts/preflight_urls.sh 0 pcb as preflights (only datasets/pcb/assets is required). The submit command sets board=0603_H100, image_edit_endpoint, image_edit_model=nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL, defect_modes=tombstone,shift, and uses render_patches=2 (sized per the ~30-final-crops-per-render_patch heuristic to hit ~60 target crops). Response explicitly avoids crop_max_emit on this flow. It also notes that missing-component is NOT produced here and would be routed to Day 0 Texture Defects (AnomalyGen handles missing via AMP). Outputs land at runs/\u003cname>/structural_defect and runs/\u003cname>/structural_defect_edited.\",\n \"expected_behavior\": [\n \"Response selects Day 0 Structural Defects (assets/configs/structural_defect_generation.yaml).\",\n \"defect_modes is narrowed to the requested subset (tombstone,shift) rather than the full default set.\",\n \"Response does NOT use crop_max_emit on this flow \\u2014 that knob does not exist on structural; it sizes via render_patches with the ~30-final-crops-per-render_patch heuristic (so render_patches=2 for ~60 target crops).\",\n \"Response notes that missing-component frames are NOT produced here and would need to be routed to Day 0 Texture Defects (AnomalyGen handles missing via AMP).\",\n \"Response describes outputs at runs/\u003cname>/structural_defect and runs/\u003cname>/structural_defect_edited (NOT runs/\u003cname>/anomaly).\"\n ]\n },\n {\n \"id\": \"6\",\n \"question\": \"Here's a real photo of our 0603_H100 PCBA captured on the AOI machine. I want defects labeled on it using the shipped PCBA checkpoint. Walk me through the plan: workflow YAML, preflights, the exact `osmo workflow submit` invocation with every required knob, the group order the workflow will execute, and where labeled output will land. The `osmo` CLI is NOT installed in this environment \\u2014 recipe only; do not try to execute. Emit your final plan as a structured response with these labeled sections: **Workflow** (the YAML file), **Preflights** (exact commands with args), **Required URL Artifacts** (enumerated under `\u003cdig_url_root>`), **Submit Command** (complete `osmo workflow submit ... --set ...` block with every required knob), **Output Location** (`\u003cdig_url_root>/runs/\u003cname>/anomaly/` or per-flow override). Do not omit any section.\",\n \"expected_skill\": \"physical-ai-defect-image-generation\",\n \"ground_truth\": \"Response selects the Day 1 Real-Photo Alignment variant (assets/configs/texture_defect_generation_day1_real_alignment.yaml) as the silent PCBA Day 1 default \\u2014 without pausing to ask 'manual or real-alignment?'. It names scripts/preflight_urls.sh 1 pcb real-alignment as the URL preflight and lists models/pretrained, models/pcb, datasets/pcb/raw, and datasets/pcb/assets (the real-alignment variant additionally requires the assets bundle for the USD tree and input_real_image/\u003cboard>.jpg). The submit command sets usecase=pcb, board=0603_H100, use_pretrained_checkpoint=true (default), and default_spatial_dependency=cad (default), plus an anomaly_types_json appropriate for PCBA. real_image_filename defaults to input_real_image/0603_H100.jpg from the pcb-assets bundle. Response notes the workflow runs usd2roi-render-day1 \\u2192 (optional finetune-job) \\u2192 anomaly-infer, and that final labeled output lands at runs/\u003cname>/anomaly.\",\n \"expected_behavior\": [\n \"Response selects Day 1 Real-Photo Alignment by default (texture_defect_generation_day1_real_alignment.yaml), NOT manual-ROI \\u2014 and does NOT pause to ask the user 'manual or real-alignment?' for a PCBA Day 1 request.\",\n \"Response invokes preflight_urls.sh with the real-alignment variant: `1 pcb real-alignment`.\",\n \"Response sets usecase=pcb and board=0603_H100; real_image_filename defaults to input_real_image/0603_H100.jpg from the pcb-assets bundle.\",\n \"default_spatial_dependency=cad is kept as the default (the usd2roi image emits semantic_segmentation_labels.json natively).\",\n \"Response describes the workflow as running usd2roi-render-day1 \\u2192 (optional finetune-job) \\u2192 anomaly-infer, with final labeled output at runs/\u003cname>/anomaly.\"\n ]\n }\n]\n","content_type":"application/json; charset=utf-8","language":"json","size":14854,"content_sha256":"4e19af583b4a04af3621516e254a121620526a1c4a34df330954def3e5f2541c"},{"filename":"references/container-images.md","content":"# Defect Image Generation Container Images\n\nCanonical image references for the active Defect Image Generation skill. Keep\nthe workflow YAML defaults and this file in sync when updating tags.\n\n## Main Runtime Components\n\n| Component | Workflow variable | Image | Used by | Notes |\n|---|---|---|---|---|\n| usd2roi (texture lane) | `usd2roi_image` | `nvcr.io/nvidia/paidf-simulation:1.0.0` | Day 0 texture, Day 1 real-photo alignment | Isaac Sim full-app image used by the usd2roi → image-edit → infer chain. Scripts live under `/workspace/paidf-simulation/`; invoke Kit directly when OSMO overrides `command`. |\n| sdg_pipeline + crop_components (IsaacSim lane) | `isaac_render_image` | `nvcr.io/nvidia/paidf-simulation:1.0.0` | `structural_defect_generation.yaml` | Same image as the usd2roi tag; ships `scripts/sdg/standalone/sdg_pipeline.py` + `scripts/postprocess/crop_components.py`. Crop step uses `--entrypoint python3`. |\n| image-edit augmentation | `augmentation_image` | `nvcr.io/nvidia/paidf-augmentation:1.0.0` | Day 0 texture, Day 0 good-image, Day 0 structural-defect | Runs the augmentation client and calls `image_edit_endpoint`; the Qwen serving endpoint is separate. The two IsaacSim Day-0 flows ship their own `build_batch_config.py` adapter (flat `rgb/*.png` layout) while the texture lane walks `\u003cMATERIAL>/\u003ccell>/normal_img/`. |\n| anomalygen | `anomalygen_image` | `nvcr.io/nvidia/paidf-anomalygen:1.0.0` | Day 0 texture, Day 1, finetune, setup prep | Powers finetune, inference, and prepared-data setup. Repo is baked at `/workspace/paidf-anomalygen/`. Ships `ngc` CLI on PATH. |\n\n## Setup Images\n\n| Purpose | Workflow variable | Image | Used by | Notes |\n|---|---|---|---|---|\n| Pretrained bundle + per-case setup | `pretrained_image` | `nvcr.io/nvidia/paidf-anomalygen:1.0.0` | `setup/setup_pretrained.yaml`, `setup/setup_pcb.yaml`, `setup/setup_metal.yaml`, `setup/setup_glass.yaml` | Same image as `anomalygen_image`; ships repo and baked checkpoints. Used by every download group across the four setup workflows. |\n\n## Day 0 Image-Edit Endpoint Images\n\n| Purpose | Location | Image | Notes |\n|---|---|---|---|\n| Local NVPCB OVSL2SL endpoint | `references/nim/qwen-image-edit-nvpcb-ovsl2sl.yaml` | `vllm/vllm-omni:v0.20.0` | NIM Operator `NIMService` that mirrors the local Docker command for `nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL` via `spec.command`/`spec.args`. |\n| Endpoint verification pod | `references/nim/README.md` | `curlimages/curl` | Docs-only helper for probing `/v1/models`; not part of the Defect Image Generation runtime workflow. |\n\n## Current Workflow Defaults\n\n| Workflow | Runtime images |\n|---|---|\n| `assets/configs/texture_defect_generation_day0.yaml` | `usd2roi_image`, `augmentation_image`, `anomalygen_image` |\n| `assets/configs/good_image_generation.yaml` | `usd2roi_image`, `augmentation_image` |\n| `assets/configs/structural_defect_generation.yaml` | `isaac_render_image`, `augmentation_image` |\n| `assets/configs/texture_defect_generation_day1_manual_roi.yaml` | `anomalygen_image` |\n| `assets/configs/texture_defect_generation_day1_real_alignment.yaml` | `usd2roi_image`, `anomalygen_image` |\n| `assets/configs/finetune.yaml` | `anomalygen_image` |\n| `assets/configs/setup/setup_pretrained.yaml` | `pretrained_image` |\n| `assets/configs/setup/setup_pcb.yaml` | `pretrained_image` |\n| `assets/configs/setup/setup_metal.yaml` | `pretrained_image` |\n| `assets/configs/setup/setup_glass.yaml` | `pretrained_image` |\n\n## Update Rule\n\nWhen changing one of the three main runtime component images:\n\n1. Update every workflow YAML default that exposes the corresponding variable.\n2. Update the table above in this file.\n3. Search the skill for the old tag and remove stale references:\n\n```bash\nrg '\u003cold-image-or-tag>' skills/physical-ai-defect-image-generation\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3814,"content_sha256":"7220160d65f5800a2f446d809bdc95743a2926a9d9dac43e478ba941c51b504a"},{"filename":"references/contents.md","content":"# DIG skill — file inventory\n\nAuthoritative list of supporting files shipped with the\n`physical-ai-defect-image-generation` skill. SKILL.md's \"Supporting files\"\nsection points here.\n\n## Workflow YAMLs and cookbooks\n\n- `assets/configs/*.yaml` — six flow YAMLs (one per row in SKILL.md\n §\"Flow walkthroughs\") plus `setup/setup_{pretrained,pcb,metal,glass}.yaml`.\n See each flow's walkthrough for submit semantics.\n- `assets/cookbooks/{pcb,metal_surface,glass}/` — per-usecase render / crop\n / training cookbooks. Per-board PCBA cookbooks under `pcb/\u003cboard>/`.\n Mounted via `localpath:`; sentinels patched at task start.\n\n## Scripts\n\n| Script | Purpose |\n|---|---|\n| `scripts/preflight_credentials.sh` | Verify the OSMO `hf-token` cred (Common Preconditions §1; images are public on `nvcr.io/nvidia/`, no registry cred needed). |\n| `scripts/preflight_pod_template.sh` | Verify OSMO POD_TEMPLATE has `nvoptix` hostPath + `/dev/shm ≥ 16Gi` (§2). Exit codes 0=OK, 1=malformed, 2=403, 3=409, 4=env-fix. |\n| `scripts/preflight_urls.sh` | Verify per-flow URL artifacts under `\u003cdig_url_root>` (§3). Args: `\u003cflow:0|1|finetune> \u003cusecase> [variant]`. |\n| `scripts/render_defect_spec.py` | AnomalyGen AMP routing fallback (renders `defect_spec.jsonl` from a cookbook). |\n| `scripts/pick_best_step.sh` | Pick the highest-`nn_score` checkpoint step from a finetune run's validation logs. |\n\n## References\n\n- `references/preconditions.md` — long-form for SKILL.md §\"Common Preconditions\" (credentials, pod template, URL artifacts, name stamping, glass UC3 zip, shipped checkpoint defaults, memory rules §4a).\n- `references/setup.md` — full `setup/` workflow run-throughs, knob tables, dataset upload procedure, glass UC3 Roboflow zip procedure, bring-your-own-data layout.\n- `references/troubleshooting.md` — operational gotchas, log parsing recipes, common failures.\n- `references/container-images.md` — image-tag table for all flow tasks (paidf-anomalygen / paidf-simulation / paidf-augmentation).\n- `references/disambiguation.md` — full trigger table + prompt construction guidance + when-NOT-to-ask exceptions.\n- `references/knob_mapping.md` — full user-intent → `--set` knob table, `crop_max_emit` semantics, per-flow caveats.\n- `references/gpu_sizing.md` — per-GPU `train_*` / `infer_*` CPU + memory scaling table for finetune and inference tasks; consult before any multi-GPU submit.\n- `references/monitoring.md` — polling cadence, task-status interpretation, log-pull escalation thresholds, failure-classification routing, and post-submit watch-loop discipline. Load before any `osmo workflow submit`, `osmo workflow query`, or `osmo workflow logs` action.\n- `references/output_retrieval.md` — `osmo data download` / MinIO `mc cp` commands + canonical anomaly tree.\n- `references/output_rendering.md` — presenting outputs to the user (zip archive + preview-grid HTML); the agent's canvas-dir rules.\n- `references/flows/*.md` — per-flow walkthroughs (group diagrams, submit-command variants, data handoffs, per-stage troubleshooting).\n- `references/nim/` — Day-0 image-edit endpoint manifest + README (Option B local deploy).\n\n## Evals and component skills\n\n- `evals/evals.json` — skill-creator evals (Day 0 PCBA, Day 1 metal_surface, glass finetune).\n- Component skills (`skills/{simulation,augmentation,anomalygen,physical-ai-infrastructure-setup-and-resilient-scaling}/SKILL.md`) — consult for issues that originate inside a component's code.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3484,"content_sha256":"5e7b1050a198129fabffa997a35af5465aa87d02884706156fa31c221f597196"},{"filename":"references/disambiguation.md","content":"# Disambiguation cheat sheet\n\nFull trigger table, prompt-construction guidance, and \"when NOT to ask\"\nexceptions. SKILL.md §Disambiguation holds the principle and silent-defaults\nlist; this file is what the agent loads to assemble the `AskUserQuestion`\noptions on a vague request.\n\n## Triggers that should pause for disambiguation\n\n| User says (example) | Why it's ambiguous | Surface as options |\n|---|---|---|\n| Any submit / upload / preflight request where the user has **not yet named a `dig_url_root`** (and memory has no prior `dig_url_root` reference) | ~80 GB+ of artifacts land under this prefix and the wrong bucket is expensive to undo. There is **no silent default** — agents must NOT auto-pick `s3://osmo-workflows/dig` even though it appears as a suggestion. | \"(a) `s3://osmo-workflows/dig` (the common shared default — confirm only if you own/control this bucket), (b) a different storage prefix you own — any OSMO-supported backend (paste the URL), (c) cancel — you don't have a target bucket ready yet.\" Save the picked value as a reference memory (see Step 0 §4) immediately after the first successful submit. |\n| \"generate me N images\" / \"give me N samples\" | `N` could mean upstream patches (`render_patches`) or final crops (`crop_max_emit`); the flow could be Day 0 texture, good-image, or structural. **For structural, there is no `crop_max_emit` knob** — yield is non-linear in `render_patches` (see SKILL.md §\"Structural-defect sizing\"). | \"(a) N final component crops via `crop_max_emit=N` (Day 0 good-image, fastest), (b) N raw scan-grid patches via `render_patches=N` (cookbook crop cap applies; more total final images), (c) N **defect** images via Day 0 texture defects (requires `anomaly_types_json`), (d) N structural-defect crops via `render_patches=ceil(N/30)` on the spark board (also narrow `defect_modes` if N \u003c 30).\" |\n| \"run the PCBA flow\" / \"do the PCBA pipeline\" | Could be Day 0 texture, Day 0 good-image, Day 0 structural, or Day 1 real-alignment — all are PCBA. | \"(a) Day 0 texture defects (full pipeline, AMP-routed defects), (b) Day 0 good-image (clean ROIs only), (c) Day 0 structural defects (pose perturbations via IsaacSim), (d) Day 1 inference + labeling on a real PCBA photo.\" |\n| \"give me defects\" / \"generate defects\" | Texture (Qwen Image-Edit + AnomalyGen AMP) vs. structural (IsaacSim pose perturbation) vs. missing-component (AnomalyGen native) are entirely different flows. | \"(a) texture defects (solder bridge, scratch, discoloration — Day 0 texture), (b) structural / pose defects (tombstone, shift, sideflip — Day 0 structural), (c) missing components (Day 0 texture handles it via AnomalyGen, NOT structural).\" |\n| \"smoke test\" / \"quick run\" / \"just test it\" | Could mean a 5-image probe, a single-checkpoint passthrough probe, or a setup probe. | \"(a) tiny probe (`render_patches=5 crop_max_emit=1`, ~5 final images), (b) full passthrough at the shipped checkpoint (default sizes, ~30 min), (c) setup-only run (the relevant `setup/setup_\u003ccase>.yaml` + `setup/setup_pretrained.yaml` to populate the DIG root).\" |\n| \"use my data\" / \"I have a dataset\" | Could be a flat user upload (manual-ROI Mode B), a prepared NGC artifact (manual-ROI Mode A), or a real PCBA photo (real-alignment input). | \"(a) clean images + per-defect submasks in canonical layout (Day 1 manual ROI Mode A), (b) flat zip / unstructured (Day 1 manual ROI Mode B — staged at runtime), (c) a real PCBA photo of a known board (Day 1 real-photo alignment).\" |\n| \"finetune\" / \"train a model\" | Could be standalone `finetune.yaml` (from labeled URL), Day 0/1 finetune-from-scratch (`use_pretrained_checkpoint=false`), or a custom checkpoint produced by `finetune.yaml` and reused. | \"(a) Finetune Only (`finetune.yaml`) on `datasets/\u003cusecase>/raw`, (b) Day 0 or Day 1 with `use_pretrained_checkpoint=false` (inline finetune then infer), (c) reuse an already-trained checkpoint under `\u003cdig_url_root>/models/\u003cusecase>`.\" |\n| \"metal\" or \"glass\" (no further context) | Both only have a Day 1 path (no USD/real-alignment exists), so the route is forced — but the user may not know that. | Confirm: \"Day 1 inference + labeling on the prepared `\u003cusecase>/raw` dataset is the only flow for this material — proceed?\" (Don't ask \"manual or real-alignment?\" — manual is the only option here.) |\n| \"use a different board\" / \"try board X\" | Board switch needs both `--set board=` AND `--set real_image_filename=`, and a per-board cookbook must exist. | List the shipped boards (`0603_H100`, `1152819000`) and ask: \"(a) `0603_H100` (default), (b) `1152819000`, (c) a new custom board (requires adding `assets/cookbooks/pcb/\u003cboard>/usd2roi_nvpcb.yaml` + uploading the photo to `datasets/pcb/assets/input_real_image/\u003cboard>.jpg` first).\" Board names must be Jinja-safe — see `references/setup.md` §\"Bring your own data\". |\n\n## How to surface options\n\n- **Prefer `AskUserQuestion`** with 2–4 mutually exclusive options. Lead with the recommended option (label suffix \"(Recommended)\") when there's a clear default.\n- **Quote the user's exact phrasing** in the question so they know what triggered it (\"You said 'generate me 10 images' — which stage should the 10 cap?\").\n- **Show the resulting `--set` differences** in option descriptions — users learn the knob semantics from seeing concrete contrasts (e.g. `crop_max_emit=10` vs. `render_patches=10`).\n- **Do not chain disambiguation prompts.** Bundle related questions into one `AskUserQuestion` call (max 4 questions). If two choices are tightly coupled (e.g. board + real photo), gate them behind a single combined option.\n\n## When NOT to disambiguate\n\n- The user has already named the flow / usecase / knob explicitly. Trust them.\n- The choice is silent-default territory (PCBA Day 1 → real-alignment, board → `0603_H100`, image-edit endpoint deployment, etc. — see SKILL.md §Disambiguation silent-defaults paragraph). Don't second-guess defaults that were settled at the workflow level.\n- The user has answered the same disambiguation in this same conversation already — re-asking is friction.\n- Only one of the candidate options is actually viable (e.g. user says \"infer on metal\" — manual-ROI is the only flow; just say \"going with Day 1 manual-ROI, the only flow available for metal\" and proceed).\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":6318,"content_sha256":"a3e4dcb9fb75daf503ee2379497e7e14af2c2ecb43b183b9b88452dbd592e332"},{"filename":"references/flows/finetune.md","content":"# Finetune Only\n\n\n## Table of Contents\n\n- [URL Contract](#url-contract)\n- [Cookbook](#cookbook)\n- [Cookbook render (in-pod, automatic)](#cookbook-render-in-pod-automatic)\n- [Submit](#submit)\n- [Output](#output)\n- [Extending a Use Case](#extending-a-use-case)\n- [Troubleshooting](#troubleshooting)\n\nTrain anomalygen on a raw AOI data URL and emit a checkpoint under\n`\u003cdig_url_root>/runs/\u003cname>/finetune`. The checkpoint can be copied into\n`\u003cdig_url_root>/models/\u003cusecase>` for later Day 0 or Day 1 passthrough runs, or\nused directly by a workflow variant that points at that run output.\n\nThe finetune task runs anomalygen Phase 1 end-to-end inside one pod:\n`validate_dataset.py` (structural sanity + mask counting) → `prep_testcase.sh`\n(AMP placement, n_seeds=1 per mask → `/tmp/validation/validation.jsonl` +\n`/tmp/validation/amp/`) → torchrun (`predict2_anomaly_gen_ddp_2b`). No\npre-baked validation artifact is required.\n\n## URL Contract\n\nSet `dig_url_root` once; default is `s3://osmo-workflows/dig`.\n\n| Purpose | URL |\n|---|---|\n| Pretrained model tree | `\u003cdig_url_root>/models/pretrained` |\n| Raw training data | `\u003cdig_url_root>/datasets/\u003cusecase>/raw` |\n| finetuned checkpoint output | `\u003cdig_url_root>/runs/\u003cname>/finetune` |\n\nPreflight:\n\n```bash\nDIG_URL_ROOT=s3://osmo-workflows/dig bash scripts/preflight_urls.sh finetune pcb\n```\n\nBuilt-in `usecase` values are `pcb`, `metal_surface`, and `glass` — uniform\nacross `--set usecase=`, URL paths (`datasets/\u003cusecase>/raw`,\n`models/\u003cusecase>`), and cookbook directories (`assets/cookbooks/\u003cusecase>/`).\nThe `metal_surface` value matches the trained model's material name baked\ninto the checkpoint taxonomy.\n\n## Cookbook\n\nThe cookbook is the exact training config the shipped checkpoint was trained\nagainst. It should usually be treated as a recipe and patched only for run\nidentity and mounted input paths.\n\n| URL usecase | Cookbook | Shipped anomaly types |\n|---|---|---|\n| `pcb` | `assets/cookbooks/pcb/ag_config.yaml` | `[[IC,bridge],[passive_component,excess_solder],[passive_component,missing]]` |\n| `metal_surface` | `assets/cookbooks/metal_surface/ag_config.yaml` | `[[metal_surface,MT_Blowhole],[metal_surface,MT_Break],[metal_surface,MT_Crack],[metal_surface,MT_Fray],[metal_surface,MT_Uneven]]` |\n| `glass` | `assets/cookbooks/glass/ag_config.yaml` | `[[Phone,oil],[Phone,scratch],[Phone,stain]]` |\n\nThe raw data URL must contain `\u003cMATERIAL>/anomaly_image/\u003cdefect>/`,\n`\u003cMATERIAL>/mask/\u003cdefect>/`, and `defect_spec.jsonl`. The relevant\n`setup/setup_\u003ccase>.yaml` creates that shape for the shipped cases. `validation.jsonl` + `amp/` are\ngenerated fresh inside the finetune task — no need to ship them.\n\n## Cookbook render (in-pod, automatic)\n\nThere is **no pre-submit render step**. The cookbook at\n`assets/cookbooks/\u003cusecase>/ag_config.yaml` is uploaded to the pod via\n`localpath:` and rendered in-pod by `yq` right after Phase 1 Step 2 produces\n`validation.jsonl`. The render patches 5 fields:\n\n| Field | Source |\n|---|---|\n| `.job.group` | `EXP_NAME` (= `--set name=…`) |\n| `.job.name` | `${EXP_NAME}_training_FP32_lr0.02_bs=2_2b_512x512` (auto-derived) |\n| `.dataloader_train.dataset.dataset_dir` | `{{input:1}}` (raw dataset URL) |\n| `.dataloader_val.dataset.input_data_path` | `/tmp/validation/validation.jsonl` (Phase 1 Step 2 output) |\n| `.model.config.ag_config.mask_encoder.encoder_config.init_cfg.checkpoint` | `checkpoints/NVDINOV2/nv_dinov2_classification_model.ckpt` |\n\nand drops the `trainer.early_stop` block (which the image's `TrainerConfig`\nrejects). Cookbook selection is driven by `--set usecase=…`:\n\n```\nusecase=pcb → assets/cookbooks/pcb/ag_config.yaml\nusecase=metal_surface → assets/cookbooks/metal_surface/ag_config.yaml\nusecase=glass → assets/cookbooks/glass/ag_config.yaml\n```\n\n## Submit\n\n```bash\nSTAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)\nNAME=finetune-$STAMP\nosmo workflow submit assets/configs/finetune.yaml \\\n --pool \u003cpool> \\\n --set name=$NAME \\\n dig_url_root=\u003cdig_url_root> \\\n usecase=\u003cpcb|metal_surface|glass>\n```\n\n`--set` carries only OSMO concerns: run name, DIG root, URL usecase, and GPU\nresources. Training recipe knobs stay in the cookbook (rendered in-pod from\n`assets/cookbooks/\u003cusecase>/ag_config.yaml`). The `$STAMP` (8 hex chars from\n`/proc/sys/kernel/random/uuid`) makes the storage path unique per submission\n— see SKILL.md §\"Name stamping\".\n\nDefaults are sized for 1 GPU. For multi-GPU finetune, pass `train_gpu`,\n`train_cpu`, and `train_memory` together — the per-GPU scaling table lives\nin `references/gpu_sizing.md`. Example for 4-GPU training:\n\n```bash\nosmo workflow submit assets/configs/finetune.yaml \\\n --pool \u003cpool> \\\n --set name=$NAME dig_url_root=\u003cdig_url_root> usecase=\u003cusecase> \\\n train_gpu=4 train_cpu=32 train_memory=192Gi\n```\n\n## Output\n\n```bash\nosmo data list --no-pager s3://osmo-workflows/dig/runs/$NAME/finetune\nosmo data download s3://osmo-workflows/dig/runs/$NAME/finetune ./output/$NAME-finetune/\n```\n\nBest-step selection is based on the highest `nn_score` from validation logs.\nSee `skills/anomalygen/SKILL.md` Phase 1 for the detailed procedure.\n\n## Extending a Use Case\n\nStart from the closest shipped cookbook. If adding defects, update both\n`anomaly_types` copies in the cookbook:\n\n- `dataloader_train.dataset.anomaly_types`\n- `model.config.ag_config.anomaly_embedding.anomaly_types`\n\nThen upload or prepare matching data under\n`\u003cdig_url_root>/datasets/\u003cusecase>/raw`. The material and defect names must\nmatch the cookbook, `defect_spec.jsonl`, and `anomaly_types_json` used by\ninference.\n\n## Troubleshooting\n\n- **`ERROR: /tmp/ag_config.yaml not mounted`** — render Step 1 before submitting.\n- **`ERROR: pretrained tree not at .../pretrained`** — rerun setup for `models/pretrained`.\n- **`ERROR: $DATASET_DIR/defect_spec.jsonl missing in raw dataset`** — the raw URL is incomplete; rerun the relevant `setup/setup_\u003ccase>.yaml` for the usecase.\n- **`ERROR: prep_testcase.sh produced an empty validation.jsonl`** — the raw dataset is missing training masks (`\u003cMATERIAL>/mask/\u003cdefect>/`). Verify with `osmo data list --no-pager \u003cdig_url_root>/datasets/\u003cusecase>/raw`.\n- **dshm OOM** — see `references/setup.md` Troubleshooting.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":6249,"content_sha256":"2f7cf84aebc6c96fa79095e51d84f1fe9503af50e764f4923bb1c141bf061d74"},{"filename":"references/flows/good_image_generation.md","content":"# Good-Image Generation (PCBA, usd2roi + Image-Edit)\n\n\n## Table of Contents\n\n- [When to use](#when-to-use)\n- [Pipeline](#pipeline)\n- [Inputs](#inputs)\n- [Submit](#submit)\n- [Output layout](#output-layout)\n- [Pairing with structural-defect runs](#pairing-with-structural-defect-runs)\n- [Troubleshooting](#troubleshooting)\n\nProcedural clean-PCBA image generation that mirrors the first two groups of\n`texture_defect_generation_day0.yaml` (`usd2roi-render` → `augment-image-edit`)\nwith no defect injection and no AnomalyGen step. **No `sdg_pipeline.py` direct\ninvocation, no `crop_components.py` post-step** — the workflow renders the\nscan_grid (with mesh-level semantics inlined into the cookbook) and then walks\nthose triggers via `usd2roi_crop.py` to emit the canonical multi-cell ROI tree\nthat the texture-defect lane already consumes.\n\n## When to use\n\n- Building a clean-image training set (ChangeNet golden halves, AnomalyGen\n finetune positives, downstream real-photo pairing).\n- Generating per-cell ROI pairs (`normal_img` + `cad_mask`) for any skill that\n consumes the canonical usd2roi tree but does not need defects.\n- Demoing the usd2roi → image-edit half of the texture pipeline without\n spinning up finetune/inference resources.\n\nFor pose-defect data (shift / tombstone / sideflip), use\n`structural_defect_generation.yaml`. For texture defects (solder bridge /\nscratch / discoloration / missing component) and the full anomalygen\ntraining/inference loop, use `texture_defect_generation_day0.yaml`.\n\n## Pipeline\n\n```\nusd2roi-render — single task (usd2roi_image — paidf-simulation):\n Stage 1: Kit + sdg_pipeline.py (scan_grid render, mesh-level semantics)\n → trigger_0000/{rgb_x*_y*.png, semantic_segmentation_*.png,\n bounding_box_2d_tight_*.npy, ...}\n Stage 2: python3 + usd2roi_crop.py (semantic-mask-driven multi-cell crop)\n → crop/\u003cMATERIAL>/\u003cx*_y*>/{normal_img,cad_mask}/\u003cNNNN>.png\n ▼\naugment-image-edit (augmentation_image — paidf-augmentation, Qwen OVSL2SL)\n reads usd2roi-components/crop/\u003cMAT>/\u003ccell>/normal_img/\u003cNNNN>.png\n writes augment/crop/\u003cMAT>/\u003ccell>/\u003cNNNN>.\u003cext> (SL-restyled RGB)\n```\n\nTwo task groups — both run on the existing OSMO pod template that the\ntexture-defect day-0 lane uses; no new prerequisites.\n\n## Inputs\n\n| Input | Source | Required by |\n|---|---|---|\n| USD asset tree (board scene + components) | `\u003cdig_url_root>/datasets/pcb/assets` (publish via `setup/setup_pcb.yaml`) | `usd2roi-render` |\n| Per-board `pcba_target.yaml` | `assets/cookbooks/pcb/\u003cboard>/pcba_target.yaml` (mounted via `localpath`) | `usd2roi-render` |\n| Per-board `day0_image.yaml` (scan_grid render config + `semantics:` block) | `assets/cookbooks/pcb/\u003cboard>/day0_image.yaml` | `usd2roi-render` |\n| Per-board `day0_crop.yaml` (multi-cell ROI crop config) | `assets/cookbooks/pcb/\u003cboard>/day0_crop.yaml` | `usd2roi-render` |\n| Image-edit cookbook (Qwen OVSL2SL prompt + parameters) | `assets/cookbooks/pcb/augmentation_config_ovsl2sl.yaml` | `augment-image-edit` |\n| Image-edit endpoint | `image_edit_endpoint` workflow param (default points at the in-cluster service from `references/nim/`) | `augment-image-edit` |\n\n`usd2roi-render` patches `scene:` in `pcba_target.yaml` at task start to point at\nthe dataset-mounted USD (located by `scene_filename` basename). Both\n`day0_image.yaml` and `day0_crop.yaml` carry sentinel placeholders\n(`__OUTPUT__`, `__MAX_IMAGE_COUNT__`) that the runner sed-patches before Kit\nlaunches; the resolved YAMLs are persisted alongside the run output for\nreproducibility.\n\n## Submit\n\nGenerate a fresh run stamp (see SKILL.md §\"Name stamping\"):\n\n```bash\nSTAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/good_image_generation.yaml \\\n --pool \u003cpool> \\\n --set name=good_image_gen-$STAMP \\\n dig_url_root=\u003cdig_url_root> \\\n board=0603_H100 \\\n image_edit_endpoint=http://qwen-image-edit-nvpcb-ovsl2sl.osmo-nims.svc.cluster.local:8000/v1 \\\n image_edit_model=nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL\n```\n\nUseful smoke-test knobs:\n\n- `crop_max_emit=N` — cap the per-cell crop output at the usd2roi-render stage\n (single point of dataset-size control; everything downstream consumes whatever\n this stage emits). Blank = use cookbook default.\n- `render_patches=N` — cap the upstream scan_grid render at N patches. Useful\n with `crop_max_emit=1` for fast smoke tests (~N final images).\n- `scene_filename=...` — change which USD inside the dataset bundle is used\n as the scene (default `spark_lighting.usd`).\n\nPer-board cookbook directories under `assets/cookbooks/pcb/`:\n\n- `0603_H100/` — demo board, 0603 capacitor passive component\n- `1152819000/` — alternate demo board\n\nEach contains `pcba_target.yaml`, `day0_image.yaml`, `day0_crop.yaml`,\n`usd2roi_nvpcb.yaml`. Pass `--set board=\u003cdir-name>` to switch which per-board\ncookbook the workflow mounts; the workflow YAML resolves\n`../cookbooks/pcb/{{ board }}/...` for each file at submit time.\n\n## Output layout\n\n```\n\u003cdig_url_root>/runs/\u003cname>/\n├─ usd2roi-components/\n│ ├─ trigger_0000/\n│ │ ├─ rgb_x0_y0.png … rgb_x9_y9.png # raw scan_grid renders, named by grid cell\n│ │ ├─ semantic_segmentation_*.png\n│ │ ├─ bounding_box_2d_tight_*.npy + labels\n│ │ └─ metadata.json\n│ ├─ crop/\n│ │ └─ \u003cMATERIAL>/ # e.g. \"passive_component\"\n│ │ └─ \u003cx*_y*>/ # one dir per scan cell that matched\n│ │ ├─ normal_img/\u003cNNNN>.png # clean per-component RGB ROI\n│ │ └─ cad_mask/\u003cNNNN>_cad_mask.png # paired CAD-derived component mask\n│ ├─ semantic_segmentation_labels.json\n│ ├─ pcba_target.yaml # patched copy (scene resolved)\n│ ├─ day0_image.yaml # patched copy (sentinels resolved)\n│ └─ day0_crop.yaml # patched copy (sentinels resolved)\n└─ augment/\n └─ crop/\n └─ \u003cMATERIAL>/\n └─ \u003cx*_y*>/\n └─ \u003cNNNN>.\u003cjpg|png> # Qwen OVSL2SL-restyled RGB\n```\n\nThe `usd2roi-components/` tree is identical in shape to what\n`texture_defect_generation_day0.yaml` produces, so any downstream skill that\nconsumes that layout (e.g. real-photo pairing, ChangeNet pair construction)\nalso works on good-image runs.\n\nThe `augment/crop/\u003cMAT>/\u003ccell>/\u003cNNNN>.\u003cext>` files are the training units —\nclean components in the destination lighting style. Pair them with\n`usd2roi-components/crop/\u003cMAT>/\u003ccell>/normal_img/\u003cNNNN>.png` for ChangeNet\ngolden halves, or with `cad_mask/\u003cNNNN>_cad_mask.png` for mask-conditioned\ntraining.\n\n## Pairing with structural-defect runs\n\nTo build ChangeNet golden / defect pairs, submit `good_image_generation.yaml`\nand `structural_defect_generation.yaml` with the **same `name`** so their\noutputs land under the same run prefix. Note: their output trees are\n**different** (`usd2roi-components/crop/\u003cMAT>/\u003ccell>/normal_img/` vs\n`structural_defect/cropped/\u003cmode>/rgb/`) — pair on filename stems and material\nclass, not on directory structure.\n\n## Troubleshooting\n\nSee `references/troubleshooting.md` for the shared `usd2roi-render` issues —\nsentinel resolution, Kit shutdown SIGABRT (image-specific), `usd2roi_crop.py`\nemitting 0 ROIs (semantics misalignment), and image-edit endpoint failures.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":7485,"content_sha256":"acfcbc7c3dd367d7a254738dffefd6479c389f3bf4ae8f59ff11501c8e4eb6c1"},{"filename":"references/flows/structural_defect_generation.md","content":"# Structural-Defect Generation (PCBA)\n\n\n## Table of Contents\n\n- [Defect modes](#defect-modes)\n- [When to use](#when-to-use)\n- [Pipeline](#pipeline)\n- [Submit](#submit)\n- [Parameters](#parameters)\n - [How `defect_modes` patches the cookbook](#how-defect_modes-patches-the-cookbook)\n - [Sizing the output](#sizing-the-output)\n - [Eligible component pool](#eligible-component-pool)\n- [Outputs](#outputs)\n- [Pairing with the good-image lane (ChangeNet)](#pairing-with-the-good-image-lane-changenet)\n- [Troubleshooting](#troubleshooting)\n\nProcedural pose-defect generation via IsaacSim's `sdg_pipeline.py` with\n`pipeline_type: defect`. Defects are simulated geometrically — components are\nshifted, tilted, or flipped — not prompted into the image via diffusion.\nPer-component crops via `crop_components.py`.\n\n## Defect modes\n\n| Mode | Geometric op | Cookbook default |\n|---|---|---|\n| `shift` | XY translation (±0.2 mm) + Z rotation (±15°) | `enabled: true`, `ratio: 0.2` |\n| `tombstone` | Y-axis tilt 70–90° (one pad edge lifts) | `enabled: true`, `ratio: 0.2` |\n| `sideflip` | X-axis flip 70–90° | `enabled: true`, `ratio: 0.2` |\n\nSelection is non-overlapping across modes — each defect type independently\npicks its components from the remaining pool.\n\n## When to use\n\n- Building a structural-defect training set (shift / tombstone / sideflip /\n polarity-reversal labeled data).\n- Generating ChangeNet pairs by submitting `good_image_generation.yaml` and\n `structural_defect_generation.yaml` against the same `name` and pairing the\n crops post hoc.\n\n**NOT for missing-component frames** — anomalygen's AMP/SDG pass on\n`texture_defect_generation_day0.yaml` synthesizes missing components by occluding\nclean ROIs with mask templates. Submitting \"missing-component\" intent here will\nnot produce them.\n\n**NOT for texture defects** (solder bridge, scratch, discoloration). Those\nrequire diffusion-based appearance edits, which the texture lane handles.\n\n## Pipeline\n\n```\nisaac-render-defect — one task, one pod (paidf-simulation):\n Stage 1: Kit + sdg_pipeline.py with defect_image.yaml\n → runs/\u003cname>/structural_defect/trigger_NNNN/{rgb_*.png, semantic_segmentation_*.png,\n bounding_box_2d_tight_*.npy + labels}\n Stage 2: python3 + crop_components.py --offset {{ crop_offset }}\n → runs/\u003cname>/structural_defect/cropped/{rgb,semantic_segmentation,component_instance}/\u003cNNNN>.png\n ▼\naugment-image-edit (cosmos_augmentation image, Qwen OVSL2SL via image_edit_endpoint)\n → reads structural_defect/cropped/rgb/\n → runs/\u003cname>/structural_defect_edited/rgb/\u003cNNNN>.png — lighting-style-transferred RGBs\n```\n\nIdentical shape to `good_image_generation.yaml`; the only differences are the\nrender cookbook (`defect_image.yaml`) and the optional `defect_modes` patching\nstep that disables non-requested modes at task start. Render and crop share one\npod — raw triggers never round-trip through MinIO between them. OVSL2SL is\nappearance-only, so the geometric pose perturbation from the render step is\npreserved through the image-edit hop.\n\n## Submit\n\nDefault — all three defect modes (shift, tombstone, sideflip) enabled, on the\n0603_H100 board, restyled through the local cluster Qwen OVSL2SL endpoint:\n\n```bash\nSTAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/structural_defect_generation.yaml \\\n --pool \u003cpool> \\\n --set name=structural_defect_gen-$STAMP \\\n dig_url_root=\u003cdig_url_root> \\\n board=0603_H100 \\\n image_edit_endpoint=http://qwen-image-edit-nvpcb-ovsl2sl.osmo-nims.svc.cluster.local:8000/v1 \\\n image_edit_model=nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL\n```\n\n`image_edit_endpoint` and `image_edit_model` default to the in-cluster service\nfrom `references/nim/`; a bare submit (no override) works against the standard\ndeployment.\n\nSubset of modes (only tombstone):\n\n```bash\n--set defect_modes=tombstone\n```\n\nMultiple modes (comma-separated, no spaces):\n\n```bash\n--set defect_modes=shift,tombstone\n```\n\nAlternate board:\n\n```bash\n--set board=1152819000\n```\n\nSmoke test:\n\n```bash\n--set render_patches=5\n```\n\n## Parameters\n\n| Knob | Default | Notes |\n|---|---|---|\n| `board` | `0603_H100` | Alternates: `1152819000`. The `1152819000` cookbook narrows `component_types` to a single IC, so the default `render_patches=5` yields too few defected frames for a reasonable training set; when submitting with `board=1152819000`, pass `render_patches=-1` (full scan_grid coverage) unless the user explicitly specifies otherwise. |\n| `defect_modes` | `all` | Comma-separated subset of `shift,tombstone,sideflip`, or the literal `all` to keep cookbook defaults. Unknown values fail fast in the patching script. |\n| `render_patches` | `5` (cookbook default) | `-1` = **full coverage** (render every scan_grid cell defined by the board's cookbook). Floor is `1` (zero produces no crops, fails the task). Yield is **non-linear** — see \"Sizing\" below. For `board=1152819000` (IC-narrow cookbook), use `-1` to get a reasonable IC yield — the default `5` is too few frames to defect the narrowed component set. |\n| `scene_filename` | `spark_lighting.usd` | USD basename. |\n| `crop_offset` | `10` | Padding pixels. |\n| `dig_url_root` | `s3://osmo-workflows/dig` | |\n| `image_edit_endpoint` | `http://qwen-image-edit-nvpcb-ovsl2sl.osmo-nims.svc.cluster.local:8000/v1` | Qwen OVSL2SL endpoint. |\n| `image_edit_model` | `nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL` | Model ID at the endpoint. |\n| `isaac_render_image` | `nvcr.io/nvidia/paidf-simulation:1.0.0` | |\n| `augmentation_image` | `nvcr.io/nvidia/paidf-augmentation:1.0.0` | Same image as the texture-defect lane. |\n\n### How `defect_modes` patches the cookbook\n\nWhen `defect_modes != \"all\"`, the render task runs a small Python patch\nagainst `defect_image.yaml` at task start:\n\n```python\nALL_MODES = {\"shift\", \"tombstone\", \"sideflip\"}\nfor mode in ALL_MODES:\n cfg[\"defects\"][mode][\"enabled\"] = mode in requested\n```\n\nThe resolved config (with mode toggles applied) is saved alongside the render\noutput at `structural_defect/render_config.yaml` for reproducibility.\n\n### Sizing the output\n\n**There is no `crop_max_emit` knob in this flow.** `crop_components.py` emits one crop per defected component; the only throttle is `render_patches` plus the cookbook's `ratio` / `component_types` / `defects.\u003cmode>.enabled` settings. Yield is non-linear in `render_patches` — each enabled defect mode draws independently per frame, so doubling frames adds ~1.6–1.7× crops, not 2×.\n\nValidated yield on `0603_H100` spark (23 component families, `ratio: 0.2`, 3 modes enabled):\n\n| `render_patches` | Approx. total crops |\n|---|---|\n| 1 | ~30 (≈10/mode) — floor; smoke test |\n| 2 | ~50 |\n| 3 | ~75 |\n| 5 (default) | ~120 |\n| 10 | ~250 |\n\nFor a target of `N` images: `render_patches ≈ ceil(N / 30)` on the spark board. Sub-30 targets need cookbook tuning (lower `ratio` or narrow `component_types`) — `render_patches=0` is not valid (fails the task).\n\nFor a custom board, base rate ≈ `enabled_modes × component_families × ratio` per patch; calibrate by submitting `render_patches=1` and counting `cropped/*/rgb/*.png`, then scale.\n\nTo shrink yield without editing the cookbook: pass `defect_modes=tombstone` (or any subset) — each disabled mode cuts ~33% of crops.\n\n### Eligible component pool\n\nAll three defect modes draw from the cookbook's top-level `component_types:`\nlist (substring-matched against prim names under `pcba_root`). Each board ships\nits own `defect_image.yaml` under `assets/cookbooks/pcb/\u003cboard>/` and is\nselected at submit time via `--set board=\u003cid>` (the workflow YAML mounts\n`cookbooks/pcb/{{ board }}/defect_image.yaml` into the pod). The shipped\n`0603_H100/defect_image.yaml` lists the spark board's full passive pool\n(capacitors, resistors, inductors across 0201–2512 footprints); the shipped\n`1152819000/defect_image.yaml` narrows to the single IC under test\n(`_115_2819_000_1`). To support a new board, create a new\n`cookbooks/pcb/\u003cnew_board>/defect_image.yaml` (copy an existing one as a\nstarting point) and submit with `--set board=\u003cnew_board>`. `sdg_pipeline.py`\nraises `KeyError: 'component_types'` without an explicit list (no `ALL`\nsentinel handler upstream).\n\nTo add a new defect mode for a custom board, add a `defects.\u003cmode>` block to\nthe cookbook with its own substring filter under `component_types`, then\nextend the workflow YAML's `ALL_MODES` set in\n`structural_defect_generation.yaml` to whitelist it.\n\n## Outputs\n\n| Stage | Output URL | Contents |\n|---|---|---|\n| `isaac-render-defect` | `\u003cdig_url_root>/runs/\u003cname>/structural_defect/` | `trigger_0000/rgb_*.png` + semantic-seg + bbox (full-frame pose-defect renders), plus `cropped/{rgb,semantic_segmentation,component_instance}/\u003cNNNN>.png` (per-component crops), plus resolved `render_config.yaml` (with `defects.*.enabled` reflecting the requested subset) + `pcba_target.yaml` snapshot |\n| `augment-image-edit` | `\u003cdig_url_root>/runs/\u003cname>/structural_defect_edited/` | `rgb/\u003cNNNN>.png` — Qwen OVSL2SL-restyled RGBs (pose geometry preserved; lighting transferred) |\n\n## Pairing with the good-image lane (ChangeNet)\n\nSubmit `good_image_generation.yaml` and `structural_defect_generation.yaml` with\nthe same `name` and `board`. The two will write under sibling URLs:\n\n```\n\u003cdig_url_root>/runs/\u003cname>/usd2roi-components/crop/\u003cMAT>/\u003ccell>/normal_img/ # good-image (clean halves)\n\u003cdig_url_root>/runs/\u003cname>/structural_defect/cropped/ # structural (defect halves)\n```\n\nNote the layouts differ: good-image emits the multi-cell ROI tree\n(`crop/\u003cMAT>/\u003ccell>/normal_img/\u003cNNNN>.png`) via `usd2roi_crop.py`, while\nstructural emits a flat per-component crop set (`cropped/rgb/\u003cNNNN>.png` plus\nmatching `semantic_segmentation/` and `component_instance/`) via\n`crop_components.py`. Pair them downstream by component identity (semantic ID\nfrom the labels JSON), not by directory layout.\n\n(A dedicated `paired` flow that emits both halves in one workflow is on the\nbacklog; today the pairing is a two-submission convention.)\n\n## Troubleshooting\n\n- **`ERROR: unknown defect_modes: [...]`** → the patching script rejects unknown\n modes. Valid values: `shift`, `tombstone`, `sideflip`, comma-separated, or the\n literal `all`. The cookbook can be hand-extended with new modes, but the\n patcher's `ALL_MODES` whitelist in `structural_defect_generation.yaml` must\n be updated to match.\n- **`KeyError: 'component_types'`** → cookbook is missing the top-level\n `component_types:` list. sdg_pipeline.py has no `ALL` sentinel — the list\n must be explicit substrings matching prim names under `pcba_root`. The\n shipped `0603_H100/defect_image.yaml` mirrors the spark board's eligible\n pool; each per-board cookbook under `assets/cookbooks/pcb/\u003cboard>/` must\n carry its own list.\n- **All defects of one mode, none of others** → check `defects.\u003cmode>.ratio` in\n the cookbook; high `ratio` for one mode can drain the pool before subsequent\n modes draw from it. Cookbook defaults are 0.2 for shift/tombstone/sideflip.\n- Other issues mirror `good_image_generation.md` — see that ref for the shared\n render/crop troubleshooting list.\n\nSee `references/troubleshooting.md` \"IsaacSim render\" subsection for the full\nupstream issue table.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":11388,"content_sha256":"a4375e40f4cc255207c835692787d77b06263d5a5806b116be56cd5f96fd49e3"},{"filename":"references/flows/texture_defect_generation_day0.md","content":"# Day 0 — Full Pipeline (PCBA CAD -> Image-Edit -> AnomalyGen)\n\n\n## Table of Contents\n\n- [URL Contract](#url-contract)\n- [Graph](#graph)\n- [Finetune-mode cookbook render (in-pod, automatic)](#finetune-mode-cookbook-render-in-pod-automatic)\n- [Image-Edit Endpoint](#image-edit-endpoint)\n- [Submit](#submit)\n- [Output](#output)\n- [Troubleshooting](#troubleshooting)\n\nEnd-to-end PCBA pipeline starting from the PCBA USD asset tree under the AOI URL\nroot. It renders per-cell CAD ROIs, sends them through the\n`nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL` endpoint for OV-to-SL appearance transfer,\nand runs AnomalyGen inference with labels emitted inline. **The image-edit model\nmust be `nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL`** — AnomalyGen was finetuned\nagainst its appearance distribution; substituting the generic `qwen-image-edit`\ncheckpoint causes silent inference-quality regressions.\n\n## URL Contract\n\nSet `dig_url_root` once; default is `s3://osmo-workflows/dig`.\n\n| Purpose | URL |\n|---|---|\n| PCBA checkpoint | `\u003cdig_url_root>/models/pcb` |\n| Pretrained model tree | `\u003cdig_url_root>/models/pretrained` |\n| Raw PCBA training data + submasks | `\u003cdig_url_root>/datasets/pcb/raw` |\n| PCBA USD assets | `\u003cdig_url_root>/datasets/pcb/assets` |\n| usd2roi output | `\u003cdig_url_root>/runs/\u003cname>/usd2roi-components` |\n| image-edit output | `\u003cdig_url_root>/runs/\u003cname>/augment` |\n| finetune output, optional | `\u003cdig_url_root>/runs/\u003cname>/finetune` |\n| final labeled output | `\u003cdig_url_root>/runs/\u003cname>/anomaly` |\n\nPreflight:\n\n```bash\nDIG_URL_ROOT=s3://osmo-workflows/dig bash scripts/preflight_urls.sh 0 pcb\n```\n\nFor finetune-from-scratch preflight, skip the shipped checkpoint requirement:\n\n```bash\nUSE_PRETRAINED_CHECKPOINT=false DIG_URL_ROOT=s3://osmo-workflows/dig \\\n bash scripts/preflight_urls.sh 0 pcb\n```\n\n## Graph\n\nPassthrough mode (`use_pretrained_checkpoint=true`, default):\n\n```\nusd2roi-render -> augment-image-edit -> anomaly-infer\n```\n\nFinetune mode (`use_pretrained_checkpoint=false`):\n\n```\nusd2roi-render -> augment-image-edit -> finetune-job -> anomaly-infer\n```\n\nThe final inference task consumes the augmented images and CAD masks from task\noutputs, then reads PCBA submasks from `datasets/pcb/raw`, pretrained weights\nfrom `models/pretrained`, and either the shipped checkpoint from `models/pcb`\nor the finetune task output. When `use_pretrained_checkpoint=false` the\nfinetune-job task builds its validation set on the fly via `prep_testcase.sh`\n(anomalygen Phase 1 Step 2) before torchrun starts.\n\n## Finetune-mode cookbook render (in-pod, automatic)\n\nIn finetune mode (`use_pretrained_checkpoint=false`) the cookbook at\n`assets/cookbooks/pcb/ag_config.yaml` is uploaded to the pod via `localpath:`\nand rendered in-pod by `yq` right after Phase 1 Step 2 produces\n`validation.jsonl`. **No pre-submit render step.** The 5 patched fields and\nthe `trainer.early_stop` drop are described in `finetune.md` §\"Cookbook render\n(in-pod, automatic)\".\n\n## Image-Edit Endpoint\n\nUse an existing endpoint reachable from OSMO pods, or deploy the local cluster\nservice from `references/nim/`.\n\n```bash\nIMAGE_EDIT_ENDPOINT=http://qwen-image-edit-nvpcb-ovsl2sl.osmo-nims.svc.cluster.local:8000/v1\nIMAGE_EDIT_MODEL=nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL\n```\n\n## Submit\n\nGenerate a fresh run stamp (see SKILL.md §\"Name stamping\"):\n\n```bash\nSTAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)\nNAME=texture_defect_gen_day0-$STAMP\n```\n\nDefault passthrough:\n\n```bash\nosmo workflow submit assets/configs/texture_defect_generation_day0.yaml \\\n --pool \u003cpool> \\\n --set name=$NAME \\\n dig_url_root=\u003cdig_url_root> \\\n image_edit_endpoint=${IMAGE_EDIT_ENDPOINT} \\\n image_edit_model=${IMAGE_EDIT_MODEL}\n```\n\nFinetune-from-scratch:\n\n```bash\nosmo workflow submit assets/configs/texture_defect_generation_day0.yaml \\\n --pool \u003cpool> \\\n --set name=$NAME \\\n dig_url_root=\u003cdig_url_root> \\\n use_pretrained_checkpoint=false \\\n image_edit_endpoint=${IMAGE_EDIT_ENDPOINT} \\\n image_edit_model=${IMAGE_EDIT_MODEL}\n```\n\nUseful smoke-test knobs:\n\n```bash\n--set render_patches=5 num_sdg=15\n```\n\nThe default taxonomy is:\n\n```bash\n'anomaly_types_json=[[\"IC\",\"bridge\"],[\"passive_component\",\"excess_solder\"],[\"passive_component\",\"missing\"]]'\n```\n\n## Output\n\n> See [Output Retrieval](../output_retrieval.md).\n\n## Troubleshooting\n\n- **Missing URL artifacts** — submit `setup/setup_pcb.yaml` + `setup/setup_pretrained.yaml`, or upload under the same `dig_url_root`.\n- **`ERROR: no USD found under \u003cASSETS_IN>`** — inspect `\u003cdig_url_root>/datasets/pcb/assets`; it must contain the USD tree from the PCBA assets artifact.\n- **`ERROR: $DATASET_DIR/defect_spec.jsonl missing in raw dataset`** — `\u003cdig_url_root>/datasets/pcb/raw` is incomplete; rerun `setup/setup_pcb.yaml`.\n- **`ERROR: prep_testcase.sh produced an empty validation.jsonl`** — the raw PCBA dataset has no training masks under `\u003cMATERIAL>/mask/\u003cdefect>/`.\n- **`submask dir not found`** — the raw PCBA data must have `\u003cmaterial>/mask/\u003cdefect>/` directories matching `anomaly_types_json`.\n- **Image-edit failures** — verify `image_edit_endpoint` from inside the cluster and check `references/nim/README.md`.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":5189,"content_sha256":"acd640190a2e5bbc989faca762aacc3f62cf79891f1058726952955d32139580"},{"filename":"references/flows/texture_defect_generation_day1_manual_roi.md","content":"# Day 1 — Inference and Labeling (manual ROI)\n\n\n## Table of Contents\n\n- [URL Contract](#url-contract)\n- [Modes](#modes)\n- [Finetune-mode cookbook render (in-pod, automatic)](#finetune-mode-cookbook-render-in-pod-automatic)\n- [Submit](#submit)\n- [Output](#output)\n- [Troubleshooting](#troubleshooting)\n\n**Not the default for PCBA.** Use this spec only when:\n- The usecase is `metal_surface` or `glass` (no USD/real-photo flow exists for those), or\n- The user **explicitly** asks to skip CAD-to-real-photo alignment on PCBA —\n e.g. \"use the NGC PCBA artifact directly\", \"manual ROI\", \"skip usd2roi\",\n \"experiment without alignment\".\n\nFor default PCBA Day 1 (CAD-derived USD + real photo, MI alignment, per-ROI\ncrop), use `texture_defect_generation_day1_real_alignment.yaml` and see\n`texture_defect_generation_day1_real_alignment.md` for the walkthrough.\n\n## URL Contract\n\nSet `dig_url_root` once; default is `s3://osmo-workflows/dig`.\n\n| Purpose | URL |\n|---|---|\n| Shipped checkpoint | `\u003cdig_url_root>/models/\u003cusecase>` |\n| Pretrained model tree | `\u003cdig_url_root>/models/pretrained` |\n| Raw inference/training data | `\u003cdig_url_root>/datasets/\u003cusecase>/raw` |\n| finetune output, optional | `\u003cdig_url_root>/runs/\u003cname>/finetune` |\n| final labeled output | `\u003cdig_url_root>/runs/\u003cname>/anomaly` |\n\nBuilt-in `usecase` values are `pcb`, `metal_surface`, and `glass` — uniform\nacross `--set usecase=`, URL paths (`datasets/\u003cusecase>/raw`,\n`models/\u003cusecase>`), and cookbook directories (`assets/cookbooks/\u003cusecase>/`).\nThe `metal_surface` value matches the trained model's material name baked\ninto the checkpoint taxonomy.\n\nPreflight:\n\n```bash\nDIG_URL_ROOT=s3://osmo-workflows/dig bash scripts/preflight_urls.sh 1 metal_surface\nDIG_URL_ROOT=s3://osmo-workflows/dig bash scripts/preflight_urls.sh 1 glass\nDIG_URL_ROOT=s3://osmo-workflows/dig bash scripts/preflight_urls.sh 1 pcb\n```\n\nFor finetune-from-scratch preflight, skip the shipped checkpoint requirement:\n\n```bash\nUSE_PRETRAINED_CHECKPOINT=false DIG_URL_ROOT=s3://osmo-workflows/dig \\\n bash scripts/preflight_urls.sh 1 metal_surface\n```\n\n## Modes\n\n| Mode | Trigger | Behavior |\n|---|---|---|\n| A | default; raw URL ships `defect_spec.jsonl` | Use the raw NGC data as-is |\n| B | raw URL is a flat user upload | Stage clean images + submasks into canonical layout and render `defect_spec.jsonl` from `anomaly_types_json` |\n\n`use_pretrained_checkpoint=true` (the default) omits `finetune-job` and reads\n`models/\u003cusecase>` directly. `use_pretrained_checkpoint=false` trains first\n(the finetune task itself runs anomalygen Phase 1 Step 2 to build a\nvalidation set inline via `prep_testcase.sh`) and feeds the freshly produced\n`runs/\u003cname>/finetune` checkpoint into inference.\n\n## Finetune-mode cookbook render (in-pod, automatic)\n\nIn finetune mode (`use_pretrained_checkpoint=false`) the per-usecase cookbook\nat `assets/cookbooks/\u003cusecase>/ag_config.yaml` is uploaded to the pod via\n`localpath:` and rendered in-pod by `yq` right after Phase 1 Step 2 produces\n`validation.jsonl`. **No pre-submit render step.** Cookbook selection is driven\nby `--set usecase=…` (one of `pcb`, `metal_surface`, `glass`). See\n`finetune.md` §\"Cookbook render (in-pod, automatic)\" for the 5 patched fields\nand the `trainer.early_stop` drop.\n\n## Submit\n\nGenerate a fresh run stamp (see SKILL.md §\"Name stamping\"):\n\n```bash\nSTAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)\nNAME=texture_defect_gen_day1_manual_roi-$STAMP\n```\n\nMetal passthrough:\n\n```bash\nosmo workflow submit assets/configs/texture_defect_generation_day1_manual_roi.yaml \\\n --pool \u003cpool> \\\n --set name=$NAME \\\n usecase=metal_surface \\\n checkpoint_step=10000 \\\n 'anomaly_types_json=[[\"metal_surface\",\"MT_Blowhole\"],[\"metal_surface\",\"MT_Break\"],[\"metal_surface\",\"MT_Crack\"],[\"metal_surface\",\"MT_Fray\"],[\"metal_surface\",\"MT_Uneven\"]]' \\\n num_sdg=30\n```\n\nGlass passthrough:\n\n```bash\nosmo workflow submit assets/configs/texture_defect_generation_day1_manual_roi.yaml \\\n --pool \u003cpool> \\\n --set name=$NAME \\\n usecase=glass \\\n checkpoint_step=9000 \\\n 'anomaly_types_json=[[\"Phone\",\"oil\"],[\"Phone\",\"scratch\"],[\"Phone\",\"stain\"]]' \\\n num_sdg=30\n```\n\nFinetune-from-scratch:\n\n```bash\nosmo workflow submit assets/configs/texture_defect_generation_day1_manual_roi.yaml \\\n --pool \u003cpool> \\\n --set name=$NAME \\\n usecase=metal_surface \\\n use_pretrained_checkpoint=false \\\n 'anomaly_types_json=[[\"metal_surface\",\"MT_Blowhole\"],[\"metal_surface\",\"MT_Break\"],[\"metal_surface\",\"MT_Crack\"],[\"metal_surface\",\"MT_Fray\"],[\"metal_surface\",\"MT_Uneven\"]]'\n```\n\n## Output\n\n> See [Output Retrieval](../output_retrieval.md).\n\n## Troubleshooting\n\n- **Missing URL artifacts** — submit the relevant `setup/setup_\u003ccase>.yaml` + `setup/setup_pretrained.yaml`, or upload under the same `dig_url_root`.\n- **`ERROR: pretrained tree not at .../pretrained`** — rerun setup for `models/pretrained`.\n- **`submask dir not found`** — the raw data URL must have `\u003cmaterial>/mask/\u003cdefect>/` directories matching `anomaly_types_json`.\n- **`ERROR: $DATASET_DIR/defect_spec.jsonl missing in raw dataset`** (finetune-from-scratch) — rerun setup for the usecase.\n- **`ERROR: prep_testcase.sh produced an empty validation.jsonl`** (finetune-from-scratch) — the raw dataset has no training masks under `\u003cMATERIAL>/mask/\u003cdefect>/`.\n- **`validate_jsonl.py` \"TEXTURE+TYPE_C not supported\"** — the taxonomy does not match the checkpoint; use the shipped table or retrain via `finetune.yaml`.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":5528,"content_sha256":"fa25752ae7363cf2525ee8919d5eb62aefd81a7b95484548d55ce10ca869d7fc"},{"filename":"references/flows/texture_defect_generation_day1_real_alignment.md","content":"# Day 1 — Inference and Labeling (real-photo alignment)\n\n\n## Table of Contents\n\n- [URL Contract](#url-contract)\n- [Pipeline](#pipeline)\n- [Submit](#submit)\n- [Custom boards](#custom-boards)\n- [Spatial dependency](#spatial-dependency)\n- [Output](#output)\n- [Troubleshooting](#troubleshooting)\n\n**This is the default Day 1 path for PCBA.** Always runs the `usd2roi-render-day1`\ngroup: a CAD-derived USD is Kit-rendered, MI-registered against a real PCBA photo,\nand cropped per-ROI. AnomalyGen inference runs on the aligned ROI crops.\n\nFor metal/glass Day 1 (no USD flow exists) or PCBA experimentation when the\nuser explicitly asks to skip alignment, use\n`texture_defect_generation_day1_manual_roi.md` instead.\n\n## URL Contract\n\nSet `dig_url_root` once; default is `s3://osmo-workflows/dig`.\n\n| Purpose | URL |\n|---|---|\n| PCBA USD assets + per-board real photos | `\u003cdig_url_root>/datasets/pcb/assets` |\n| Submask templates | `\u003cdig_url_root>/datasets/\u003cusecase>/raw` |\n| Shipped checkpoint | `\u003cdig_url_root>/models/\u003cusecase>` |\n| Pretrained model tree | `\u003cdig_url_root>/models/pretrained` |\n| usd2roi day-1 intermediate output | `\u003cdig_url_root>/runs/\u003cname>/usd2roi-day1` |\n| finetune output, optional | `\u003cdig_url_root>/runs/\u003cname>/finetune` |\n| final labeled output | `\u003cdig_url_root>/runs/\u003cname>/anomaly` |\n\nPreflight:\n\n```bash\nDIG_URL_ROOT=s3://osmo-workflows/dig bash scripts/preflight_urls.sh 1 pcb real-alignment\n```\n\nThe canonical `pcb-assets` artifact ships per-board real photos at\n`input_real_image/\u003cboard>.jpg` (e.g. `0603_H100.jpg`, `115_2819_000.jpg`).\nPick the board with `--set board=\u003cdir-name>` — the matching cookbook lives at\n`assets/cookbooks/pcb/\u003cboard>/usd2roi_nvpcb.yaml`.\n\n## Pipeline\n\n```\nusd2roi-render-day1 ──► usd2roi-day1 (GPU-render) ← datasets/pcb/assets (USD tree + input_real_image/\u003cboard>.jpg)\n ↳ Stage 1: Kit ortho-render the CAD USD\n ↳ Stage 2: cupy MI registration → align synth to real photo\n ↳ Stage 3: per-ROI bbox crop → crop/\u003cMAT>/{normal_img,cad_mask}/\n → runs/\u003cname>/usd2roi-day1\n ▼\nfinetune-job (optional, omitted when use_pretrained_checkpoint=true)\n ▼\nanomaly-infer ──► infer-all-defects (GPU) ← usd2roi-day1 output + datasets/\u003cusecase>/raw + models/pretrained + checkpoint\n ↳ stage aligned ROIs as clean_image + cad_mask\n ↳ overlay per-defect submask templates\n ↳ prep_testcase.sh → validate_jsonl.py → run_sdg.sh → verify_output.sh\n → runs/\u003cname>/anomaly\n```\n\n## Submit\n\nGenerate a fresh run stamp (see SKILL.md §\"Name stamping\"):\n\n```bash\nSTAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)\nNAME=texture_defect_gen_day1_real_alignment-$STAMP\n```\n\nDefault (passthrough against the shipped PCBA checkpoint, board 0603_H100):\n\n```bash\nosmo workflow submit assets/configs/texture_defect_generation_day1_real_alignment.yaml \\\n --pool \u003cpool> \\\n --set name=$NAME \\\n usecase=pcb \\\n 'anomaly_types_json=[[\"passive_component\",\"excess_solder\"],[\"passive_component\",\"missing\"]]'\n```\n\nAlternate board (1152819000):\n\n```bash\nosmo workflow submit assets/configs/texture_defect_generation_day1_real_alignment.yaml \\\n --pool \u003cpool> \\\n --set name=$NAME \\\n board=1152819000 \\\n real_image_filename=input_real_image/115_2819_000.jpg \\\n usecase=pcb \\\n 'anomaly_types_json=[[\"IC\",\"bridge\"]]'\n```\n\nThe two `--set` knobs go together — when `board` changes, the matching\n`input_real_image/\u003cboard>.jpg` must exist in `\u003cdig_url_root>/datasets/pcb/assets`.\n\n## Custom boards\n\nTo add a new board:\n\n1. Add `assets/cookbooks/pcb/\u003cboard>/usd2roi_nvpcb.yaml` (mirror `0603_H100/`\n or `1152819000/` — set `semantics:` to your component's mesh paths and\n adjust `registration.sx_range`/`sy_range`/`rot_range_deg` / `camera.translate`).\n2. Upload the real AOI photo to\n `\u003cdig_url_root>/datasets/pcb/assets/input_real_image/\u003cboard>.jpg`.\n3. Submit with `--set board=\u003cboard> real_image_filename=input_real_image/\u003cboard>.jpg`.\n\n## Spatial dependency\n\n`default_spatial_dependency` defaults to `cad`. The usd2roi image emits a single\nglobal `semantic_segmentation_labels.json` at `crop/` root that CADParser\nconsumes natively, so this lane runs in `cad` mode without extra setup.\n\nFall back to `default_spatial_dependency=free` only if:\n- labels JSON is missing under `crop/`,\n- MI alignment moved the cad_mask off the component, or\n- the scene's cad_mask was rendered without `colorize_semantic_segmentation`.\n\nFor non-spark scenes, edit `assets/cookbooks/pcb/usd2roi_day1.yaml` (the\nfallback config) in place — semantics, camera, and registration ranges.\n\n## Output\n\n> See [Output Retrieval](../output_retrieval.md).\n\nThe intermediate `runs/\u003cname>/usd2roi-day1/` directory (unique to this flow) contains:\n- `crop/\u003cMAT>/{normal_img,cad_mask}/\u003cNNNN>.png` — per-ROI aligned crops + masks\n- `aligned/params.json` — MI registration parameters (rotation, scale, shift)\n- `usd2roi_day1.yaml` — resolved cookbook (with `__SCENE__`/`__REAL_IMAGE__`/`__OUTPUT__` substituted)\n\n## Troubleshooting\n\n- **`usd2roi_register.py exited 2`** — MI score below `min_mi` (default 0.5). Widen `registration.sx_range`/`sy_range`/`rot_range_deg`, lower `min_mi`, or re-check `camera.translate` + `horizontal_aperture` against the real photo. See troubleshooting.md \"usd2roi day-1 MI alignment\" entry.\n- **`ERROR: real_image_filename=... not found`** — the photo at the configured path isn't in `datasets/pcb/assets`. Verify with `osmo data list --no-pager \u003cdig_url_root>/datasets/pcb/assets/input_real_image/`.\n- **`ERROR: scene_filename=... not found`** — `spark_lighting.usd` is the default; canonical `pcb-assets` ships it. If you've replaced the assets bundle, pass `--set scene_filename=\u003cyour.usd>`.\n- **`0 ROI crops emitted`** — registration succeeded but no semantic regions matched the cookbook's `crop.classes` whitelist. Confirm the `semantics:` block in `\u003cboard>/usd2roi_nvpcb.yaml` matches mesh paths in your USD.\n- **Mode A/B-style inputs (no real photo)** — wrong workflow; use `texture_defect_generation_day1_manual_roi.yaml`.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":6529,"content_sha256":"5ebc1257e618b6a71d673b36d04d4f4efaa4225eaed4614e5a017e824571acdd"},{"filename":"references/gpu_sizing.md","content":"# GPU sizing for finetune and inference\n\nThe DIG workflow YAMLs ship 1-GPU defaults. When scaling `train_gpu` or\n`infer_gpu` at submit time, also scale CPU and host memory together — every\nrank still loads the full Cosmos-Predict2-2B + T5 tokenizer + NVDINOV2 + SAM2\n+ Qwen3-VL stack into host RAM, so memory pressure grows roughly linearly\nwith rank count.\n\nThe agent should consult this table whenever a user asks to run multi-GPU\ntraining or inference, and pass all three values on the `--set` line — not\njust `train_gpu=N` / `infer_gpu=N`.\n\n## Training (`train_gpu`)\n\nApplies to `finetune.yaml`, the inline `finetune-job` group in\n`texture_defect_generation_day0.yaml`,\n`texture_defect_generation_day1_manual_roi.yaml`, and\n`texture_defect_generation_day1_real_alignment.yaml` (when\n`use_pretrained_checkpoint=false`).\n\n| `train_gpu` | `train_cpu` | `train_memory` |\n|---|---|---|\n| `\"1\"` | `\"16\"` | `64Gi` |\n| `\"2\"` | `\"16\"` | `96Gi` |\n| `\"4\"` | `\"32\"` | `192Gi` |\n| `\"8\"` | `\"64\"` | `384Gi` |\n\nRationale: each cosmos-predict2-2B rank uses ~33 GiB CPU RAM steady-state\nduring DDP sync (T5 / NVDINOV2 / SAM2 / Qwen3-VL state, dataset prefetch,\nhost buffers). The table allocates ~48 GiB / rank past 1-GPU to give headroom\nfor NCCL collective bursts and checkpoint save. See\n`references/troubleshooting.md` \"multi-GPU FT cgroup OOM\" for the failure\nmode when memory is undersized.\n\n## Inference (`infer_gpu`)\n\nApplies to the `anomaly-infer` task in\n`texture_defect_generation_day0.yaml`,\n`texture_defect_generation_day1_manual_roi.yaml`, and\n`texture_defect_generation_day1_real_alignment.yaml`.\n\n| `infer_gpu` | `infer_cpu` | `infer_memory` |\n|---|---|---|\n| `\"1\"` | `\"4\"` | `64Gi` |\n| `\"2\"` | `\"8\"` | `96Gi` |\n| `\"4\"` | `\"16\"` | `192Gi` |\n| `\"8\"` | `\"32\"` | `384Gi` |\n\nRationale: each rank shards the workload across one GPU, but the full 2B +\ntokenizer + DINOv2 + Cosmos-Predict2 stack still lands in host RAM per rank.\nHardcoding 64 GiB worked for `infer_gpu=1` but caused OOM-kills when two\nranks loaded the 2B Cosmos-Predict2 weights concurrently in the same pod —\nhence the memory ramp.\n\n## Submit-time examples\n\nSingle-GPU (defaults — nothing to pass):\n\n```bash\nosmo workflow submit assets/configs/finetune.yaml \\\n --pool \u003cpool> \\\n --set name=finetune-$STAMP dig_url_root=\u003croot> usecase=pcb\n```\n\n4-GPU finetune:\n\n```bash\nosmo workflow submit assets/configs/finetune.yaml \\\n --pool \u003cpool> \\\n --set name=finetune-$STAMP dig_url_root=\u003croot> usecase=pcb \\\n train_gpu=4 train_cpu=32 train_memory=192Gi\n```\n\n8-GPU training, 1-GPU inference on a Day 1 manual-ROI run:\n\n```bash\nosmo workflow submit assets/configs/texture_defect_generation_day1_manual_roi.yaml \\\n --pool \u003cpool> \\\n --set name=day1-$STAMP dig_url_root=\u003croot> usecase=metal_surface \\\n use_pretrained_checkpoint=false \\\n train_gpu=8 train_cpu=64 train_memory=384Gi \\\n infer_gpu=1 infer_cpu=4 infer_memory=64Gi\n```\n\n## Notes\n\n- `train_storage` (`300Gi`) and `infer_storage` (`200Gi`) do **not** scale with\n GPU count — they're sized for the largest checkpoint + dataset shard the\n pod will ever stage, regardless of rank fan-out.\n- `render_gpu` / `augment_gpu` (Day 0 only) are independent and stay at 1 —\n they are single-pod stages, not DDP.\n- `train_gpu` and `infer_gpu` can be set asymmetrically on the same submit\n (e.g. 8-GPU train + 1-GPU infer) — they index different resource blocks in\n the workflow.\n- If you go beyond 8 GPUs per task, extrapolate at ~8 CPU and ~48 GiB per\n additional training rank, ~4 CPU and ~48 GiB per additional inference\n rank. Validate against `references/troubleshooting.md` before running a\n long job.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3660,"content_sha256":"b8719b19f61de152e1dfda0de8e8390440f709f9f1e4c7d2ead35c8f90327270"},{"filename":"references/knob_mapping.md","content":"# User intent → knob mapping\n\nHow to translate quantity / scope phrases in user requests into the right\n`--set` knob on the right OV stage. SKILL.md §\"User intent → knob mapping\"\nholds the headline rule; this file is the full breakdown the agent loads\nwhen a user asks for a specific count or coverage scope.\n\n**Every OV flow is two-stage**: `sdg_pipeline.py` renders raw patches/frames\n→ `usd2roi_crop.py` (or `crop_components.py`) extracts per-component crops\nfrom each. The final dataset size is the *downstream* product of those two\nstages, NOT the upstream render count.\n\n**DO NOT auto-map \"generate N images\" → `render_patches=N`.** That caps\nstage 1 (raw patches before cropping), not the final dataset.\n\n## Knob table\n\n| User intent | Knob | Stage | Note |\n|---|---|---|---|\n| \"generate N images\", \"produce N samples\", \"I want N final crops\" | `crop_max_emit=N` | crop (stage 2) | Per material per cell. Final dataset size on disk. |\n| \"render N patches\", \"cover N scan-grid cells\", \"raw render count\" | `render_patches=N` | render (stage 1) | Upstream raw patches; each yields multiple component crops. |\n| \"smoke test\", \"quick test\", \"few images\" | `render_patches=5 crop_max_emit=1` | both | Fastest path; ~5 final images. |\n| \"full board coverage\" | `render_patches=-1` (default) + `crop_max_emit=\"\"` (use cookbook) | both | Cover all scan-grid cells with cookbook's per-cell cap (10 for `0603_H100`; `1152819000` ships `max_emit: null` — uncapped). |\n\n## `crop_max_emit` semantics\n\n`crop_max_emit` is a workflow-level `--set` knob in `good_image_generation.yaml`\nand `texture_defect_generation_day0.yaml`. It patches `crop.max_emit` in the\ncookbook's `day0_crop.yaml` at task start. Set to `\"\"` (blank) or omit to\nuse the cookbook value; set to `null` to remove the cap entirely; set to a\npositive integer to cap per (material, cell).\n\n## Flows where `crop_max_emit` doesn't apply\n\n- **`structural_defect_generation.yaml`** — the equivalent stage-2 cap\n doesn't exist (`crop_components.py` emits one crop per component, by\n design). Use `render_patches=N` to limit defect frames; the per-frame\n component count is determined by `pcba_root` + `component_types`. See\n also SKILL.md §\"Structural-defect sizing\" for the non-linear yield rule.\n- **`texture_defect_generation_day1_real_alignment.yaml`** — the\n usd2roi-day1 stage emits a flat `crop/\u003cMAT>/normal_img/*.png` per ROI\n without a `max_emit` cap; use the per-board cookbook's `crop.classes`\n whitelist to narrow scope.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":2510,"content_sha256":"3993ae265221e6d909f47e2ed3d7c089554b3a6d5ab93cb592bc81ddb9a95dd8"},{"filename":"references/monitoring.md","content":"# DIG skill — Monitoring best practices\n\nAuthoritative monitoring discipline for the\n`physical-ai-defect-image-generation` skill. SKILL.md's \"OSMO Monitoring\"\nsection and the Response Template's **Monitoring** line point here.\n\n**Load this file before any `osmo workflow submit`, `osmo workflow query`,\n`osmo workflow logs`, or `osmo workflow cancel` action in this skill.**\nRe-read it on the first such action of every new conversation. Do not\nparaphrase these rules from memory.\n\n## Rules\n\n1. **Long-running workflows go to a subsession watcher.** Any DIG\n workflow expected to exceed **10 minutes** wall-clock (in practice:\n every flow in `assets/configs/`) MUST be\n handed off to a background **subsession watcher** the moment\n `osmo workflow submit` returns a workflow id. The watcher polls in\n isolation and reports back to the main agent **only on terminal state\n transition** (`SUCCEEDED` / `FAILED` / `CANCELED`) or on an explicit\n user follow-up. Do not block the main session on polling.\n\n2. **Spawn the watcher as a background subagent and back it with a\n bounded scheduled poll as a cronjob.** The watcher is a **subagent / sub-session**\n launched through whatever spawning primitive the host harness\n provides (e.g. `Task` with a background subagent type, `sessions_spawn`,\n `claude code` subagent, OpenAI Codex / Agents SDK session, a detached\n shell session, or any equivalent). Whatever the primitive, it MUST:\n\n - run in the background, independent of the main session;\n - be scoped to a single OSMO workflow id;\n - report back to the main agent only on terminal state transitions\n (per Rule 1) or on explicit user follow-up;\n - own its own poll loop — the main session must not block on it.\n\n3. **Never cancel a workflow whose pod is still `RUNNING`, and never\n cancel without explicit user confirmation.** A non-progressing log\n tail is NOT a failure signal on its own. Known quiet-but-healthy\n windows include:\n\n - **IsaacSim / `structural_defect_generation.yaml`** — 7–10 min\n warmup with sparse log output and many benign warnings (RTX\n loading, USD parsing, zenity). Routinely\n misdiagnosed as \"stuck\".\n - **`usd2roi-render`** — scan_grid stage runs silently between\n per-cell crop emissions.\n - **`augment-image-edit`** — long pauses while waiting on the\n image-edit endpoint (`nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL`)\n under queue pressure.\n - **`finetune-job`** — torchrun startup + dataset validation runs\n before the first step log appears.\n\n Before any `osmo workflow cancel`: (a) confirm the underlying pod is\n not `RUNNING` (a `RUNNING` pod with quiet logs is doing work — leave\n it alone); (b) state the suspected failure mode to the user and **ask\n for explicit confirmation** to cancel. Never auto-cancel on a\n timeout, a warning grep hit, or a status-poll error.\n\n## Out of scope (use the cited reference instead)\n\n- `osmo data download` mechanics on a `SUCCEEDED` run →\n `references/output_retrieval.md`.\n- Rendering the anomaly tree / preview grid →\n `references/output_rendering.md`.\n- Pre-submit preflight failures (creds / pod template / URL artifacts) →\n `references/preconditions.md` + `scripts/preflight_*.sh`.\n- Component-internal debugging once a task's logs are in hand →\n `references/troubleshooting.md` and the component skill.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3391,"content_sha256":"a60724ec6aed7d98b3aa31d89ff49b922deb31d02b82eabd19b79e597fe75a0b"},{"filename":"references/nim/qwen-image-edit-nvpcb-ovsl2sl.yaml","content":"# Qwen Image Edit NVPCB OVSL2SL endpoint.\n#\n# This is NOT an official NGC NIM image. It uses NIM Operator's generic\n# spec.command / spec.args support to run the same `vllm serve` invocation\n# that the standalone Deployment previously used, while still benefiting from\n# operator-managed PVC, probes, service, and lifecycle.\n#\n# NIM Operator mounts spec.storage.pvc at /model-store in the container and\n# auto-sets NIM_CACHE_PATH=/model-store. We point HF_HOME at a subdir so model\n# weights persist on the PVC across pod restarts.\napiVersion: apps.nvidia.com/v1alpha1\nkind: NIMService\nmetadata:\n name: qwen-image-edit-nvpcb-ovsl2sl\n namespace: osmo-nims\n labels:\n app.kubernetes.io/name: qwen-image-edit-nvpcb-ovsl2sl\n app.kubernetes.io/component: image-edit-endpoint\nspec:\n labels:\n app.kubernetes.io/name: qwen-image-edit-nvpcb-ovsl2sl\n app.kubernetes.io/component: image-edit-endpoint\n image:\n repository: vllm/vllm-omni\n tag: \"v0.20.0\"\n pullPolicy: IfNotPresent\n # NIMService requires authSecret even for non-NGC images. The vLLM container\n # ignores NGC_API_KEY; HF_TOKEN below is what grants model access.\n authSecret: ngc-api-secret\n command:\n - vllm\n - serve\n args:\n - nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL\n - --omni\n - --host\n - 0.0.0.0\n - --port\n - \"8000\"\n env:\n - name: HF_TOKEN\n valueFrom:\n secretKeyRef:\n name: hf-token-secret\n key: HF_TOKEN\n - name: HF_HOME\n value: /model-store/huggingface\n storage:\n pvc:\n create: true\n size: \"150Gi\"\n volumeAccessMode: ReadWriteMany\n sharedMemorySizeLimit: \"32Gi\"\n startupProbe:\n enabled: true\n probe:\n httpGet:\n path: /v1/models\n port: 8000\n initialDelaySeconds: 60\n periodSeconds: 30\n failureThreshold: 120\n timeoutSeconds: 5\n readinessProbe:\n enabled: true\n probe:\n httpGet:\n path: /v1/models\n port: 8000\n initialDelaySeconds: 60\n periodSeconds: 30\n failureThreshold: 120\n timeoutSeconds: 5\n livenessProbe:\n enabled: true\n probe:\n tcpSocket:\n port: 8000\n initialDelaySeconds: 180\n periodSeconds: 30\n failureThreshold: 10\n timeoutSeconds: 5\n replicas: 1\n userID: 0\n groupID: 0\n resources:\n limits:\n nvidia.com/gpu: 1\n memory: 128Gi\n ephemeral-storage: 50Gi\n requests:\n nvidia.com/gpu: 1\n memory: 128Gi\n ephemeral-storage: 50Gi\n expose:\n service:\n type: ClusterIP\n port: 8000\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":2531,"content_sha256":"6d29d6e68b0afc214f8952acfdf7fddb9440dc11bf9a89f3d8e5091113e47bc0"},{"filename":"references/nim/README.md","content":"# Day 0 Image-Edit Endpoint\n\n\n## Table of Contents\n\n- [Option A: Existing Endpoint](#option-a-existing-endpoint)\n- [Option B: Local Cluster Endpoint](#option-b-local-cluster-endpoint)\n - [Prerequisite: OSMO pool sizing](#prerequisite-osmo-pool-sizing)\n - [Deploy](#deploy)\n- [Verify Endpoint Health Before Submitting Day 0](#verify-endpoint-health-before-submitting-day-0)\n- [Why `vllm/vllm-omni` Runs Under NIMService](#why-vllmvllm-omni-runs-under-nimservice)\n\nDay 0 calls the `nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL` endpoint through\n`image_edit_endpoint`; it does not own the endpoint lifecycle. **The model is\nalways `nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL`** — the generic upstream\n`qwen-image-edit` checkpoint is NOT a substitute (see \"Why `vllm/vllm-omni`\nRuns Under NIMService\" below). Pick one endpoint source for that model before\nsubmitting `texture_defect_generation_day0.yaml`.\n\n## Option A: Existing Endpoint\n\nUse any endpoint reachable from OSMO task pods that serves the image-edit model\nthrough the `/v1` API:\n\n```bash\nosmo workflow submit assets/configs/texture_defect_generation_day0.yaml \\\n --pool \u003cpool> \\\n --set name=texture_defect_gen_day0-$(cat /proc/sys/kernel/random/uuid | cut -c1-8) \\\n dig_url_root=\u003cdig_url_root> \\\n image_edit_endpoint=https://\u003chost>/v1 \\\n image_edit_model=nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL\n```\n\nUse this path when the endpoint is already hosted, or when another team owns the\nserving stack.\n\n## Option B: Local Cluster Endpoint\n\n> **Require the `physical-ai-infrastructure-setup-and-resilient-scaling` skill to\n> stand up this endpoint.** Use it to (1) confirm the NIM Operator is installed\n> and that `nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL` is a supported model, and\n> (2) deploy and manage the NIMService. **Never hand-roll\n> a plain `vllm serve` Deployment** — without the operator's PVC it caches model\n> weights to ephemeral storage and the pod is evicted (`ephemeral local storage\n> usage exceeds the total limit`).\n\nUse this path when the endpoint should run in the same Kubernetes cluster as\nOSMO. The manifest in this directory mirrors the local Docker command:\n\n### Prerequisite: OSMO pool sizing\n\nThe local NIM consumes 1 GPU in `osmo`; the DIG workflow needs ≥1 more from\nthe same OSMO pool. A pool with `Total Capacity \u003c 2` cannot host both — the\nNIM permanently occupies the only GPU and DIG tasks queue indefinitely.\n\nBefore `kubectl apply`, consult `skills/physical-ai-infrastructure-setup-and-resilient-scaling/SKILL.md` §\"Check pool\nresources\" to read `Total Capacity` for the target pool (it documents\n`osmo pool list` + column semantics). Proceed only when `Total Capacity >= 2`;\notherwise grow the pool or fall back to Option A.\n\n### Deploy\n\nLocal equivalent (Docker) for reference:\n\n```bash\ndocker run --rm -ti --gpus all \\\n -e HF_TOKEN=xxx \\\n vllm/vllm-omni:v0.20.0 \\\n nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL --omni --port 8000\n```\n\nApply the NIMService (NIM Operator must already be installed in the cluster — see `skills/physical-ai-infrastructure-setup-and-resilient-scaling/SKILL.md` for the operator install + lifecycle. The infra skill's `install.sh` already creates the `osmo-nims` namespace and the `ngc-api-secret` / `nvcr-pull-secret` / `hf-token-secret` there; the steps below only run those preconditions when applying this NIMService standalone):\n\n```bash\nkubectl create namespace osmo-nims --dry-run=client -o yaml | kubectl apply -f -\nkubectl -n osmo-nims create secret generic hf-token-secret \\\n --from-literal=HF_TOKEN=\"${HF_TOKEN}\" \\\n --dry-run=client -o yaml | kubectl apply -f -\nkubectl apply -f references/nim/qwen-image-edit-nvpcb-ovsl2sl.yaml\nkubectl -n osmo-nims wait --for=condition=Ready \\\n nimservice.apps.nvidia.com/qwen-image-edit-nvpcb-ovsl2sl --timeout=60m\n```\n\nUse the in-cluster service DNS as the Day 0 endpoint:\n\n```bash\nosmo workflow submit assets/configs/texture_defect_generation_day0.yaml \\\n --pool \u003cpool> \\\n --set name=texture_defect_gen_day0-$(cat /proc/sys/kernel/random/uuid | cut -c1-8) \\\n dig_url_root=\u003cdig_url_root> \\\n image_edit_endpoint=http://qwen-image-edit-nvpcb-ovsl2sl.osmo-nims.svc.cluster.local:8000/v1 \\\n image_edit_model=nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL\n```\n\nVerify the endpoint from inside the cluster:\n\n```bash\nkubectl run curl -n osmo-nims --rm -i --restart=Never \\\n --image=curlimages/curl -- \\\n curl -sS http://qwen-image-edit-nvpcb-ovsl2sl.osmo-nims.svc.cluster.local:8000/v1/models\n```\n\n## Verify Endpoint Health Before Submitting Day 0\n\nThe Day 0 `image-edit` task fails immediately if the endpoint is in\n`CrashLoopBackOff` or not yet `Ready`. Always confirm the deployment is\nhealthy first; otherwise OSMO wastes a full `usd2roi-render` run before\nfailing on `image-edit`. Two quick checks to run before every Day 0 submit:\n\n```bash\n# 1. NIMService + pod state — expect Ready condition True, pod 1/1 no recent restarts\nkubectl -n osmo-nims get nimservice,deploy,po -l app.kubernetes.io/name=qwen-image-edit-nvpcb-ovsl2sl\n\n# 2. /v1/models reachable AND serves the expected model id\nkubectl run curl -n osmo-nims --rm -i --restart=Never \\\n --image=curlimages/curl -- \\\n curl -fsS http://qwen-image-edit-nvpcb-ovsl2sl.osmo-nims.svc.cluster.local:8000/v1/models \\\n | grep -q 'nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL' && echo OK || echo NOT_READY\n```\n\nProceed with `osmo workflow submit assets/configs/texture_defect_generation_day0.yaml ...`\nonly when both report healthy. If the pod is `CrashLoopBackOff` with\n`OSError: [Errno 28] No space left on device` in `kubectl logs --previous`,\nconfirm the manifest still sets `spec.storage.sharedMemorySizeLimit: 32Gi` —\nNIM Operator translates that into an `emptyDir{medium:Memory}` mounted at\n`/dev/shm`, and the default container `/dev/shm` is too small for vLLM-omni's\nmulti-proc executor.\n\n## Why `vllm/vllm-omni` Runs Under NIMService\n\n**Every DIG workflow REQUIRES the `nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL`\nfinetuned checkpoint.** The generic upstream Qwen-Image-Edit NIM\n(`qwen/qwen-image-edit-2511` or any other generic variant) is **NOT** an\nacceptable substitute under any circumstance — not as a fallback, not for\n\"smoke testing\", not because the finetuned image is harder to deploy. The\nNVPCB OVSL2SL checkpoint was finetuned on the OV→SL appearance distribution\nthat downstream AnomalyGen finetune + inference were trained against;\nsubstituting the generic checkpoint produces augmented ROIs outside that\ndistribution and causes **silent quality regressions** (the workflow appears\nto succeed; the labels are degraded). If the agent cannot deploy\n`nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL` for any reason, **stop and surface the\nblocker to the user** — do not fall back to a generic NIM.\n\nThis endpoint uses `vllm/vllm-omni:v0.20.0` rather than an official NGC NIM\nimage. NIMService's generic `spec.command` / `spec.args` fields run the\n`vllm serve nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL --omni` invocation directly\nwhile the operator still manages PVC, probes, service, and lifecycle.\nNIM Operator mounts `spec.storage.pvc` at `/model-store` and auto-sets\n`NIM_CACHE_PATH=/model-store`; the manifest sets `HF_HOME=/model-store/huggingface`\nso model weights persist on the PVC across pod restarts. `authSecret` is\nrequired by the NIMService schema even though the vLLM container ignores\n`NGC_API_KEY` — model access comes from `HF_TOKEN` instead.\n\nReference: NVIDIA NIM Operator `NIMService` configuration:\nhttps://docs.nvidia.com/nim-operator/latest/service.html\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":7527,"content_sha256":"f64d8e66cb627da7e9e75f379390aad81d2f7d5fc5847043303d46af9b83561b"},{"filename":"references/output_rendering.md","content":"# Output rendering and download\n\nTwo operations the agent runs after a DIG workflow completes, depending on\nwhat the user asks for:\n\n- **Download** — package all stages + per-task logs as a single zip the\n user can pull from the agent host.\n- **Preview** — render a small grid of representative samples across\n pipeline stages, emit a self-contained HTML page the rendering UI can\n display.\n\nThe underlying `osmo data download` / `mc cp` commands live in\n`references/output_retrieval.md`; this file is about **how the agent\npresents** those outputs after fetching them.\n\n## Table of Contents\n\n- [Pick the path (download vs preview)](#pick-the-path-download-vs-preview)\n- [The agent's canvas directory](#the-agents-canvas-directory)\n- [Path A — Download all stages and logs](#path-a--download-all-stages-and-logs)\n- [Path B — Preview grid](#path-b--preview-grid)\n - [Stages and per-stage source](#stages-and-per-stage-source)\n - [Sample selection](#sample-selection)\n - [HTML layout](#html-layout)\n - [Procedure](#procedure)\n- [Pitfalls](#pitfalls)\n\n## Pick the path (download vs preview)\n\n| User asks | Path | Returns |\n|---|---|---|\n| \"download the results\", \"give me the files\", \"I want everything\" | Download | absolute path to `\u003crun_name>.zip` |\n| \"show me the results\", \"preview\", \"summarize\", \"what does it look like\" | Preview | absolute path to `index.html` |\n| Both (\"show me and download\") | Run both | both paths |\n\nIf the request is ambiguous, use `AskUserQuestion` with two options\n(\"download archive\" / \"preview grid\") before doing either.\n\n## The agent's canvas directory\n\nThe rendering UI (Claude Code, Codex, web client) serves files from a\nworkspace-rooted directory. Writing to `/tmp/` or another path outside\nthat root means the UI cannot resolve `\u003cimg src=\"…\">` references in the\ngenerated HTML. Rules:\n\n1. **Default to the current working directory.** Use\n `./outputs/\u003crun_name>/{preview,download}/`. The cwd is always\n addressable by the rendering UI.\n2. **Honor any explicit user override.** If the user asked for a\n specific path (\"put it under `~/dig-results`\"), use that — but\n confirm it's writable and inside the workspace before writing.\n3. **All `src=` in the generated HTML must be relative.** Reference\n images as `./col1/sample_001.png`, not absolute paths and not\n S3 / MinIO URLs. The UI will not fetch external URLs.\n4. **Echo the absolute path back to the user.** After writing, print\n the absolute path on its own line so the UI / user can open it.\n5. **Don't overwrite without echoing first.** If the target directory\n exists, surface that to the user before clobbering.\n\n## Path A — Download all stages and logs\n\nTrigger: user wants a downloadable archive of everything.\n\n### Stages to include (per flow)\n\nSource root is `\u003cdig_url_root>/runs/\u003cname>/`.\n\n| Flow | Stages |\n|---|---|\n| Day 0 — Texture Defects | `usd2roi-components/`, `augment/`, `finetune/` (if scheduled), `anomaly/` |\n| Day 0 — Good Image | `usd2roi-components/`, `augment/` |\n| Day 0 — Structural Defects | `structural_defect/`, `structural_defect_edited/` |\n| Day 1 — Real-photo Alignment | `usd2roi-day1/`, `finetune/` (if scheduled), `anomaly/` |\n| Day 1 — Manual ROI | `finetune/` (if scheduled), `anomaly/` |\n| Finetune Only | `finetune/` |\n\nPlus per-task logs for every group in the workflow.\n\n### Procedure\n\n1. Resolve `\u003cdig_url_root>`, `\u003cname>`, `\u003cworkflow_id>`, and the flow type\n (cookbook + Step 0 §1).\n2. Stage outputs locally:\n ```bash\n mkdir -p ./outputs/\u003cname>/download/runs\n osmo data download \u003cdig_url_root>/runs/\u003cname>/ ./outputs/\u003cname>/download/runs/\n # or, if MinIO-backed (see references/troubleshooting.md §\"Output Retrieval\"):\n mc cp --recursive osmo/\u003cbucket>/dig/runs/\u003cname>/ ./outputs/\u003cname>/download/runs/\n ```\n3. Dump logs per task — iterate over the flow's group structure (see the\n diagram in each `references/flows/\u003cflow>.md`):\n ```bash\n mkdir -p ./outputs/\u003cname>/download/logs\n for task in \u003ctask_names>; do\n osmo workflow logs \u003cworkflow_id> -t \"$task\" -n 5000 \\\n > \"./outputs/\u003cname>/download/logs/${task}.log\"\n done\n ```\n4. Zip:\n ```bash\n ( cd ./outputs/\u003cname> && zip -r \"\u003cname>.zip\" download/ )\n ```\n5. Echo `realpath ./outputs/\u003cname>/\u003cname>.zip` back to the user.\n\nTip: if the run is large (> ~2 GB) and the user only needs the final\nlabeled output, ask whether they want the full archive or just `anomaly/`\n+ logs.\n\n## Path B — Preview grid\n\nTrigger: user wants a visual summary.\n\n### Stages and per-stage source\n\nColumns are built left → right in pipeline order, including only stages\nthat exist for the flow. The same sample ID is used across columns so\neach row reads as one frame moving through the pipeline.\n\nColumns map a frame through the pipeline: **input → constraint/mask →\ntransformation → final**. For OV-driven flows, that's OV render → cad\nmask → augmentation → AnomalyGen reconstructed. For Manual ROI (no OV\nupstream), the input + mask come from AnomalyGen's own per-sample\n`original_image/` and `original_mask/` outputs, which are co-emitted\nalongside `reconstructed_image/` and align 1:1 by filename.\n\n| Column | Day 0 Texture / Good | Structural | Day 1 Real-Photo Alignment | Day 1 Manual ROI |\n|---|---|---|---|---|\n| 1. Input | `usd2roi-components/crop/\u003cMAT>/\u003ccell>/normal_img/\u003cNNNN>.png` (OV render) | `structural_defect/cropped/rgb/\u003cNNNN>.png` | `usd2roi-day1/crop/\u003cMAT>/normal_img/\u003cNNNN>.png` | `anomaly/inference/original_image/\u003cfile>.png` |\n| 2. Mask | `usd2roi-components/crop/\u003cMAT>/\u003ccell>/cad_mask/\u003cNNNN>_cad_mask.png` | `structural_defect/cropped/semantic_segmentation/\u003cNNNN>.png` | `usd2roi-day1/crop/\u003cMAT>/cad_mask/\u003cNNNN>.png` | `anomaly/inference/original_mask/\u003cfile>.png` |\n| 3. Augmentation | `augment/crop/\u003cMAT>/\u003ccell>/\u003cNNNN>.png` | `structural_defect_edited/rgb/\u003cNNNN>.png` | n/a | n/a |\n| 4. AnomalyGen reconstructed | `anomaly/inference/reconstructed_image/\u003cfile>.png` | n/a (no anomaly stage) | `anomaly/inference/reconstructed_image/\u003cfile>.png` | `anomaly/inference/reconstructed_image/\u003cfile>.png` |\n\nSkip a column entirely if its source path is missing for the flow.\n**Good Image** has columns 1–3 (no anomaly stage). **Structural** has\ncolumns 1–3 (no anomaly stage). **Manual ROI** has columns 1, 2, 4 (no\naugmentation stage — input + mask come from the anomaly stage's own\nper-sample originals).\n\nHeader labels in the rendered HTML should match the flow: use\n\"OV render\" / \"OV cad mask\" for OV-driven flows and\n\"Original image\" / \"Original mask\" for Manual ROI, so the user reading\nthe grid sees what each column actually is.\n\n### Sample selection\n\n- 5–10 samples total. Fewer than 5 looks anemic; more than 10 slows the\n rendering UI.\n- **Deterministic**: sort filenames lexicographically and pick every\n `k`-th so the picks span the dataset (e.g., 7 samples → `k = ceil(N / 7)`).\n- **Aligned across columns**: pick sample IDs that exist in *every*\n available column for that flow, so each row tells one story. If\n alignment is impossible (e.g., per-component crops in structural don't\n map 1:1 to anomaly), drop the unalignable column rather than mixing\n unrelated frames.\n- **For Day 0 PCBA**: prefer one sample per (material × cell) that exists\n in all columns. For structural: one per defect mode\n (shift / tombstone / sideflip).\n\n### Filename patterns (matters for row alignment)\n\nSample IDs are not literally the same string across all columns — each\nstage names files differently. To align a row across columns:\n\n- **Day 0 Texture / Good** — upstream OV stages share the `\u003cNNNN>` stem:\n - col 1 (normal_img): `\u003cNNNN>.png`\n - col 2 (cad_mask): `\u003cNNNN>_cad_mask.png` ← `_cad_mask` suffix\n - col 3 (augment): `\u003cNNNN>.png` (same stem as normal_img)\n - col 4 (anomaly): **composite** `\u003ccell>__\u003cstem>.png` (the per-cell\n tree is flattened into one directory at the anomaly stage). Pick a\n Day-0 row by first picking `\u003cNNNN>` from `normal_img/`, then\n locating `\u003ccell>__\u003cNNNN>.png` under `anomaly/inference/` — `\u003ccell>`\n is the directory name two levels up from `normal_img/\u003cNNNN>.png`.\n- **Structural** — `cropped/{rgb,semantic_segmentation,component_instance}/\u003cNNNN>.png`\n share the `\u003cNNNN>` stem 1:1; `structural_defect_edited/rgb/\u003cNNNN>.png`\n preserves it. Skip the `component_instance/` subdir — it's a per-\n component index, not a viewable image.\n- **Day 1 Real-Photo Alignment** — `usd2roi-day1/crop/\u003cMAT>/{normal_img,cad_mask}/\u003cNNNN>.png`\n share the stem 1:1 (no `_cad_mask` suffix at this stage). Anomaly\n stage flattens to `anomaly/inference/` with paired filenames across\n subdirs.\n- **Manual ROI** — all three columns come from `anomaly/inference/{original_image,original_mask,reconstructed_image}/`\n with **identical filenames** in each subdir. Pick one filename from\n `reconstructed_image/` and the same filename exists in the other two.\n\n### HTML layout\n\nSelf-contained `index.html` at `./outputs/\u003cname>/preview/index.html`.\nNo external CSS or JS — the rendering UI usually handles only images and\ninline HTML. Skeleton:\n\n```html\n\u003c!DOCTYPE html>\n\u003chtml lang=\"en\">\n\u003chead>\n \u003cmeta charset=\"utf-8\">\n \u003ctitle>DIG preview: \u003crun-name>\u003c/title>\n \u003cstyle>\n body { font-family: system-ui, sans-serif; padding: 1rem; }\n h1 { margin: 0 0 1rem; font-size: 1.1rem; }\n .grid { display: grid; grid-template-columns: repeat(\u003cN_COLS>, 1fr); gap: .5rem; }\n .hdr { font-weight: 600; padding: .25rem 0; border-bottom: 1px solid #ddd; }\n .cell { display: flex; flex-direction: column; align-items: center; }\n .cell img { width: 100%; height: auto; display: block; }\n .cap { font-size: .75rem; color: #666; margin-top: .15rem; }\n \u003c/style>\n\u003c/head>\n\u003cbody>\n \u003ch1>DIG preview — \u003crun-name> (\u003cflow-type>)\u003c/h1>\n \u003cdiv class=\"grid\">\n \u003cdiv class=\"hdr\">OV render\u003c/div>\n \u003cdiv class=\"hdr\">OV cad mask\u003c/div>\n \u003cdiv class=\"hdr\">Augmentation\u003c/div>\n \u003cdiv class=\"hdr\">AnomalyGen\u003c/div>\n \u003c!-- one row per sample -->\n \u003cdiv class=\"cell\">\u003cimg src=\"./col1/0001.png\">\u003cdiv class=\"cap\">sample 0001\u003c/div>\u003c/div>\n \u003cdiv class=\"cell\">\u003cimg src=\"./col2/0001.png\">\u003cdiv class=\"cap\">sample 0001\u003c/div>\u003c/div>\n \u003cdiv class=\"cell\">\u003cimg src=\"./col3/0001.png\">\u003cdiv class=\"cap\">sample 0001\u003c/div>\u003c/div>\n \u003cdiv class=\"cell\">\u003cimg src=\"./col4/0001.png\">\u003cdiv class=\"cap\">sample 0001\u003c/div>\u003c/div>\n \u003c!-- … -->\n \u003c/div>\n\u003c/body>\n\u003c/html>\n```\n\n`\u003cN_COLS>` = number of columns actually populated for the flow.\n\n### Procedure\n\n1. Resolve `\u003cdig_url_root>`, `\u003cname>`, `\u003cworkflow_id>`, flow type.\n2. Decide which columns apply for the flow (see the table above).\n3. Pick sample IDs (5–10, deterministic, aligned across columns where\n possible).\n4. For each picked sample × each applicable column, download just that\n one image into `./outputs/\u003cname>/preview/col\u003cK>/\u003csample_id>.png`.\n Skip a sample if any *required* column is missing — don't render\n partial rows.\n5. Generate `index.html` referencing the staged images by relative path.\n Number of columns matches what was actually staged.\n6. Echo `realpath ./outputs/\u003cname>/preview/index.html` back to the user.\n\nIf Path A has already run, prefer to read from the local\n`./outputs/\u003cname>/download/runs/` copy rather than re-downloading.\n\n## Pitfalls\n\n- **Absolute paths in HTML `src=`** — the UI cannot resolve them. Use\n relative paths only.\n- **Writing outside the cwd-rooted output tree** — the UI sandbox\n typically refuses to read those files. Stay under `./outputs/\u003cname>/`.\n- **Picking samples per-stage independently** — rows will mix unrelated\n frames. Pick once, reuse the IDs across columns.\n- **Forgetting to localize images** — `src=` pointing at S3 / MinIO URLs\n will not fetch in the UI sandbox. Download to the preview dir.\n- **Mis-ordered columns** — keep OV render → cad mask → augmentation →\n AnomalyGen. That's pipeline order, left to right; flipping it makes\n the grid unreadable.\n- **Including the structural flow's `cropped/component_instance/`** in\n the preview — it's a per-component index, not a viewable image; skip\n it.\n- **Over-large images** — the per-cell crops are usually small (~512 px\n per side); no need to resize. If a stage emits very large frames\n (Day 1 real-alignment can produce 4k+ usd2roi-day1 renders), generate\n thumbnails first (`mogrify -resize 640x` or similar) and reference\n the thumbnails in the HTML, not the originals.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":12413,"content_sha256":"c38735e3327ee527b45404dcc49bb93c5c6faaa3dc71159536d6e24d8217ee86"},{"filename":"references/output_retrieval.md","content":"# Output retrieval\n\nHow to pull DIG workflow outputs out of OSMO and onto the agent host. Shared\nacross all flows; per-flow walkthroughs in `references/flows/\u003cflow>.md` point\nhere for the retrieval block.\n\nFor *presenting* outputs to the user once retrieved (download archive zip,\npreview-grid HTML), see `references/output_rendering.md` — separate concern.\n\n## Pull the anomaly tree\n\n```bash\nosmo data list --no-pager \u003cdig_url_root>/runs/${NAME}/\nosmo data download \u003cdig_url_root>/runs/${NAME}/anomaly ./output/${NAME}/\n```\n\n`${NAME}` is the value the agent passed via `--set name=\u003cflow>-$STAMP` at\nsubmit (Common Preconditions §4). `\u003cdig_url_root>` is the bucket prefix\nestablished at first-time setup (Step 0 first-time gate).\n\n## MinIO-backed OSMO alternative\n\nIf the OSMO instance is backed by MinIO, `mc cp` is an alternative to\n`osmo data download`:\n\n```bash\nmc cp --recursive osmo/\u003cbucket>/runs/${NAME}/anomaly ./output/${NAME}/\n```\n\nThe `mc` alias `osmo` is configured at `~/.mc/config.json`\n(key `osmo` → `http://localhost:30090`).\n\n## Canonical `anomaly/` tree\n\nEvery Day 0 Texture and Day 1 (manual-ROI / real-alignment) flow emits this\nflat layout under `runs/\u003cname>/anomaly/`:\n\n- `reconstructed_image/` — AnomalyGen reconstructions\n- `annotated_image/` — annotated samples with defect overlays\n- `cropped_image/` — per-ROI cropped inputs fed to AnomalyGen\n- `cropped_mask/` — per-ROI submasks used during inference\n- `original_image/` — pre-crop source\n- `original_mask/` — pre-crop submask\n- `SDG_result.csv` — per-sample labels + metadata\n\nDay 0 Good Image and Day 0 Structural Defects emit different trees (no\n`anomaly/` — they don't run AnomalyGen); see their flow refs for the\nper-flow layout.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":1734,"content_sha256":"5b7e06c79fc555272f5eb986514ddaa4daebc4f3cf11ea1ca0a4f249bd8bf1f7"},{"filename":"references/preconditions.md","content":"# Common preconditions — detail\n\nExtended detail for the five preconditions summarized in `SKILL.md` §\"Common Preconditions (all flows)\". The summary in SKILL.md is the agent-facing quick reference; this file holds the long-form jq block, branching prose, and glass-zip procedure that the agent loads only when a check fails.\n\n## §1 — OSMO credentials + tokens\n\nOne OSMO credential is required by the flow YAMLs:\n\n- `hf-token` (GENERIC type) — used for every HF download, including gated `nvidia/Cosmos-AnomalyGen-*`, `nvidia/Cosmos-Predict2-*`, `nvidia/Spark-AnomalyGen-USD`, Glass-Masks, etc. (full repo list in `references/setup.md`).\n\nNo registry credential is needed: the `paidf-*` images are public on `nvcr.io/nvidia/` and pull anonymously (the YAMLs no longer reference `nvcr_io`). If image pulls fail (authorization error or `nvcr.io` rate-limiting), see `references/troubleshooting.md` → \"nvcr.io image pull failures\".\n\nRun `scripts/preflight_credentials.sh` — authoritative check is that `hf-token` is provisioned. The `HF_TOKEN` env var only auto-sets a missing cred or runs the outbound HF probe; optional when `hf-token` already exists. Pass `--no-probe` in restricted-egress shells. Re-run on fresh conversations (tokens can expire); do NOT re-run before every submit inside the same conversation. See `references/setup.md` §\"Credential check\".\n\n## §2 — Pod template\n\nSkip outright when memory records `verified` / `user-confirmed` / `skipped-409` for the cluster. Otherwise the OSMO pod template must declare the `nvoptix` hostPath mount at `/usr/share/nvidia/nvoptix.bin` and a `dshm` emptyDir with `sizeLimit ≥ 16 GiB` (32 GiB preferred). Pre-submit check:\n\n```bash\nscripts/preflight_pod_template.sh # add --min-dshm-gib 32 if your workflow needs the preferred size\n```\n\nBranch on the exit code (matches Step 0 §1):\n\n- **exit 0** (template OK) → proceed; save \"pod template verified\" to memory (Step 0 §6) so future conversations skip the check.\n- **exit 1** (template visible but missing nvoptix and/or dshm sizing) → the user has read permission, so they almost certainly have admin or admin-equivalent. The script prints which check failed; route to `physical-ai-infrastructure-setup-and-resilient-scaling` for the patch runbook (`osmo config update POD_TEMPLATE`). After a successful patch, re-run the script and save the verified state to memory.\n- **exit 2** (HTTP 403 — user lacks read permission; `osmo profile list` will confirm `osmo-admin` is absent from `roles:`) → **do not stop yet**. Use `AskUserQuestion`: *\"Your account can't read POD_TEMPLATE (403). Has your OSMO administrator already configured the cluster so it meets the DIG requirements — `nvoptix` hostPath mount at `/usr/share/nvidia/nvoptix.bin` AND `/dev/shm` ≥ 16 GiB?\"* (Yes / No or unsure). On **Yes** → save \"pod template user-confirmed\" to memory (Step 0 §6) and proceed (runtime preflight is the safety net). On **No / unsure** → stop and tell them: *\"Contact your OSMO administrator to ensure the nvoptix hostPath mount (`/usr/share/nvidia/nvoptix.bin`) and `/dev/shm` ≥ 16 GiB are present in POD_TEMPLATE — DIG workflows cannot run without them.\"*\n- **exit 3** (HTTP 409 — some 6.3 ConfigMap-mode deployments disable the config CLI) → warn the user that the pre-submit gate is being skipped and that the in-pod runtime preflight (Step 0) is the only remaining check, then proceed; save the skip state to memory.\n- **exit 4** (osmo / jq missing, or unexpected failure) → fix the environment and re-run; do not auto-skip.\n\nCadence: **once per conversation** (cache result in-conversation) and **once per user across conversations** when the memory entry from Step 0 §6 indicates the cluster is already verified or user-confirmed. The runtime in-pod preflight catches drift between checks and submission. Re-run this check after any pod-template patch (and clear the memory entry first).\n\n## §3 — Required URL artifacts\n\nPreflight before every flow submission:\n\n```bash\nDIG_URL_ROOT=\u003cdig_url_root> scripts/preflight_urls.sh \u003cflow> \u003cusecase> [variant]\n```\n\nThe script checks the per-flow URL checklist (see SKILL.md table) with `osmo data list --no-pager`. `\u003cflow>` is `0` / `1` / `finetune`; `\u003cvariant>` is optional (`real-alignment` for Day 1 PCBA real-photo alignment, which adds `datasets/pcb/assets` to the checklist). Set `USE_PRETRAINED_CHECKPOINT=false` when preflighting a finetune-from-scratch run. If anything is missing, **stop and submit the relevant `setup/setup_\u003ccase>.yaml` + `setup/setup_pretrained.yaml` first** or upload the artifact under the same DIG root — see `references/setup.md`.\n\nBuilt-in `usecase` values are `pcb`, `metal_surface`, and `glass` — uniform across the `--set usecase=` knob, URL datasets (`datasets/\u003cusecase>/raw`), model checkpoints (`models/\u003cusecase>`), and cookbook directories (`assets/cookbooks/\u003cusecase>/ag_config.yaml`). `metal_surface` matches the trained model's material name (the `anomaly_types_json=[[\"metal_surface\",\"MT_*\"],...]` taxonomy baked into the checkpoint) — no translation layer.\n\nThe PCBA assets artifact ships the USD tree only — `pcba_target.yaml`, `day0_image.yaml` (with mesh-level semantics inlined), and `day0_crop.yaml` mount from the per-board cookbook (`assets/cookbooks/pcb/\u003cboard>/`) at task start, so the URL artifact doesn't need them.\n\n### Shipped checkpoint and `anomaly_types_json` defaults\n\nPassthrough runs (`use_pretrained_checkpoint=true`, default) need no further knobs; the YAMLs ship per-usecase `checkpoint_step` and `anomaly_types_json` defaults below. Override only when running a custom-trained checkpoint or narrowing the defect set. Day 0 Good Image and Day 0 Structural Defects have no AnomalyGen step — neither knob applies.\n\n| Flow | Use case | `checkpoint_step` | Shipped `anomaly_types_json` |\n|---|---|---|---|\n| Day 0 Texture | `pcb` | `14000` | `[[\"IC\",\"bridge\"],[\"passive_component\",\"excess_solder\"],[\"passive_component\",\"missing\"]]` |\n| Day 1 (manual + real-align) | `pcb` | `14000` | `[[\"passive_component\",\"missing\"]]` (narrow default; override for multi-type) |\n| Day 1 manual | `metal_surface` | `10000` | `[[\"metal_surface\",\"MT_Blowhole\"],[\"metal_surface\",\"MT_Break\"],[\"metal_surface\",\"MT_Crack\"],[\"metal_surface\",\"MT_Fray\"],[\"metal_surface\",\"MT_Uneven\"]]` |\n| Day 1 manual | `glass` | `9000` | `[[\"Phone\",\"oil\"],[\"Phone\",\"scratch\"],[\"Phone\",\"stain\"]]` |\n\nDay 1 YAMLs auto-swap `checkpoint_step` and `anomaly_types_json` from the PCBA defaults (`14000` / `[[\"passive_component\",\"missing\"]]`) to the per-usecase rows above when the user does not override at submit time.\n\n## §4 — Name stamping\n\nProduction YAMLs ship no `name` default (avoids silent overwrites on repeat submits); every submit MUST pass `--set name=\u003cflow>-$STAMP`.\n\n```bash\nSTAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)\nosmo workflow submit assets/configs/\u003cflow>.yaml --pool \u003cpool> \\\n --set name=\u003cflow>-$STAMP \\\n \u003cother knobs>\n```\n\nRegenerate `$STAMP` fresh before every submit (don't reuse across submits in the same shell), echo it back so the user can find outputs at `runs/\u003cflow>-\u003cstamp>/...`. Missing `name` fails validate with `Jinja substitution failure: 'name' is undefined`.\n\n## §4a — Memory rules (cross-conversation state)\n\nAfter the first-time gate resolves — and after any submit where the user explicitly diverged from a documented default — persist load-bearing choices so future sessions don't re-ask. Save as reference / user memories using the auto-memory system.\n\n| What | Memory type | When to save |\n|---|---|---|\n| `dig_url_root` value the user picked | reference | After first-time setup (or whenever the user changes buckets). |\n| OSMO `--pool` the user typically submits against | user | After the first successful submit; update if the user switches pools. |\n| Default board (`0603_H100` / `1152819000` / custom) | user | Only if the user explicitly diverges from the workflow default. |\n| `image_edit_endpoint` URL when using Option A (existing endpoint) | reference | After the user confirms an existing endpoint URL; do NOT save the in-cluster Option B service DNS — that's derivable from `references/nim/`. |\n| Pod template state: `verified` (jq passed) / `user-confirmed` (403 + user yes-answered) / `skipped-409` (CLI disabled) | reference | After §2 resolves successfully. Lets future conversations skip the §2 check entirely; runtime in-pod preflight is the safety net. Clear this entry if the cluster is reconfigured (e.g., admin patches the template). |\n| OSMO `osmo-admin` role: present / absent | user | After first `osmo profile list` read. Saves re-discovery; if absent, agents know not to even attempt `osmo config update POD_TEMPLATE`. Refresh if the user mentions being added to / removed from groups. |\n| `image_edit_model` | DO NOT SAVE | Always `nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL`; saving it as a \"preference\" would invite drift to the wrong model. |\n| Ephemeral state (a specific run STAMP, one-off `anomaly_types_json`, debug context) | DO NOT SAVE | Conversation-scope only. |\n\nAt the **start** of every new conversation about this skill, read the relevant memories and apply them silently. If a recalled memory conflicts with what the user is asking for now, trust the current request and offer to update the memory.\n\n## §5 — Glass case (UC3) — Roboflow zip prerequisite\n\nOnly when running `setup/setup_glass.yaml`. The license-gated Roboflow Mobile-Screen COCO export must be uploaded to an OSMO URL prefix **before** submitting `setup_glass.yaml`. If you are about to run glass setup, verify with `osmo data list --no-pager \u003cprefix>/` that `mobile_screen.zip` is present, then pass `--set uc3_zip_url_root=\u003cprefix>`. Empty `uc3_zip_url_root` fails validation. **Do not skip ahead to submit** — there is no auto-download step. Full procedure (browser export, rename to `mobile_screen.zip`, `osmo data upload`) lives in `references/setup.md` §\"Glass case (UC3)\".\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":9992,"content_sha256":"4a9f2cc33fee15de9ecbc5c9144c88f458f2e7be82d3ce5c3fc0d218a58cc9a5"},{"filename":"references/setup.md","content":"# Defect Image Generation — Asset Setup\n\n\n## Table of Contents\n\n- [What you get](#what-you-get)\n- [Prerequisites](#prerequisites)\n- [Credential check](#credential-check)\n- [OSMO setup workflows](#osmo-setup-workflows)\n- [URL artifact layout](#url-artifact-layout)\n - [Per-artifact root layout + key files](#per-artifact-root-layout-key-files)\n - [Per-use-case parameter values](#per-use-case-parameter-values)\n - [`pcba_target.yaml` is mounted from the cookbook](#pcba_targetyaml-is-mounted-from-the-cookbook)\n- [Bring your own data or models](#bring-your-own-data-or-models)\n - [Naming gotcha: `--set` parses numeric-looking values as PEP 515 ints](#naming-gotcha---set-parses-numeric-looking-values-as-pep-515-ints)\n- [Wiring into Day 0 / Day 1 / finetune](#wiring-into-day-0-day-1-finetune)\n - [Day 0 (PCBA, full pipeline)](#day-0-pcba-full-pipeline)\n - [Day 1 (Metal surface or Glass)](#day-1-metal-surface-or-glass)\n - [Finetune flow (from scratch — Finetune Only)](#finetune-flow-from-scratch-finetune-only)\n- [Troubleshooting](#troubleshooting)\n\n> For per-flow submit commands and shipped-dataset taxonomies, see `references/troubleshooting.md`. For workflow container images, see `references/container-images.md`.\n\nOne-time download of finetuned anomalygen checkpoints + per-use-case datasets + the PCBA USD asset bundle + the ~71 GB `models/pretrained` checkpoint tree (all from Hugging Face except UC2 metal, which still comes from public GitHub) into URL-backed OSMO storage artifacts. Setup is split across four narrow workflow YAMLs under `assets/configs/setup/` — submit only the ones the use case needs; they run in-cluster, in parallel.\n\n> **Always set up assets via the OSMO setup workflows** (`setup/setup_\u003ccase>.yaml` + `setup/setup_pretrained.yaml`) — never download assets locally to work around a problem. If setup fails on credentials (HF license/scope, missing `hf-token`) or an image pull, **stop and ask the user to rectify it**, then re-submit on OSMO (pull failures: Troubleshooting → \"nvcr.io image pull failures\"). A cluster that genuinely can't reach Hugging Face or `nvcr.io` is an environment issue to raise with the user, not route around.\n\n## What you get\n\nEight URL artifacts:\n\n| Default URL | Source | Use case | Asset type |\n|---|---|---|---|\n| `s3://osmo-workflows/dig/models/pcb` | `nvidia/Cosmos-AnomalyGen-PCB-2B` (HF model) via `scripts/utilities/download_anomalygen_checkpoints.sh --uc pcb` | PCBA | finetuned anomalygen checkpoint (iter 14000) + `ag_config.yaml` |\n| `s3://osmo-workflows/dig/models/metal_surface` | `nvidia/Cosmos-AnomalyGen-Metal-2B` (HF model) via `download_anomalygen_checkpoints.sh --uc metal` | Metal surface | finetuned anomalygen checkpoint (iter 10000) + `ag_config.yaml` |\n| `s3://osmo-workflows/dig/models/glass` | `nvidia/Cosmos-AnomalyGen-Glass-2B` (HF model) via `download_anomalygen_checkpoints.sh --uc glass` | Glass | finetuned anomalygen checkpoint (iter 9000) + `ag_config.yaml` |\n| `s3://osmo-workflows/dig/datasets/pcb/raw` | `nvidia/Cosmos-AnomalyGen-PCB-Dataset` (HF dataset) via `scripts.utilities.prepare_dataset_uc1` | PCBA | raw training tree: clean_image + cad_mask + submasks + `defect_spec.jsonl` + `semantic_segmentation_labels.json` |\n| `s3://osmo-workflows/dig/datasets/metal_surface/raw` | [`abin24/Magnetic-tile-defect-datasets.`](https://github.com/abin24/Magnetic-tile-defect-datasets.) (public GitHub) via `scripts.utilities.prepare_dataset_uc2` | Metal surface | curated UC2 subset: 5 anomaly + 5 masks per defect × 5 defects, 20 clean images + `defect_spec.jsonl` |\n| `s3://osmo-workflows/dig/datasets/glass/raw` | `nvidia/Cosmos-AnomalyGen-Glass-Masks` (HF dataset, masks + `defect_spec.jsonl` only — NV derivatives, no images) overlaid with user-supplied Roboflow Mobile-Screen zip via `prepare_dataset_uc3.py --masks-from-hf` | Glass | `Phone/{anomaly_image,clean_image}/` from user zip; `Phone/mask/\u003cdefect>/` + `defect_spec.jsonl` from HF. Material dir is **`Phone`**. **Prerequisite: upload `mobile_screen.zip` to an OSMO URL prefix first, then submit with `--set uc3_zip_url_root=\u003cprefix>`** — see \"Glass case (UC3)\" steps below. |\n| `s3://osmo-workflows/dig/datasets/pcb/assets` | `nvidia/Spark-AnomalyGen-USD` (HF dataset repo) | PCBA | USD scene + asset tree + per-board real photos at `input_real_image/\u003cboard>.jpg` (e.g. `0603_H100.jpg`, `115_2819_000.jpg`) |\n| `s3://osmo-workflows/dig/models/pretrained` | nvcr.io `paidf-anomalygen` container (baked NVDINOV2 / SAM2 / Qwen3-VL) + HF gated (`nvidia/Cosmos-Predict2-*`) + HF public (`nvidia/C-RADIOv3-B`, `google-t5/t5-large`, `facebook/dinov2-large`) | all | ~71 GB pretrained bundle (model_sizes=2B; ~140 GB with `2B 14B`) |\n\n> **Metal UC2 source (GitHub):** The upstream repository is [`abin24/Magnetic-tile-defect-datasets.`](https://github.com/abin24/Magnetic-tile-defect-datasets.) — the slug **ends with a literal period** (not a documentation typo). Fetch via OSMO only: submit `setup/setup_metal.yaml`; the `download-metal_surface-data` task runs `prepare_dataset_uc2`, which clones that slug in-cluster.\n\n## Prerequisites\n\n1. **HF token** with read access to the gated `nvidia/Cosmos-AnomalyGen-*` + `nvidia/Cosmos-Predict2-*` + `nvidia/Spark-AnomalyGen-USD` repos. Export locally as `HF_TOKEN`. Sanity-check with a REST probe:\n ```bash\n curl -sI -H \"Authorization: Bearer $HF_TOKEN\" \\\n https://huggingface.co/api/models/nvidia/Cosmos-AnomalyGen-PCB-2B \\\n | head -1\n ```\n A `200` means the token can read the gated repo; `401`/`403` means either the token lacks scope or the license hasn't been accepted yet — visit each gated repo page in a browser once and click \"Accept\" before the OSMO workflow first runs.\n2. **NGC API key** — not required. The `paidf-*` workflow images are public on `nvcr.io/nvidia/` and pull anonymously, so no NGC key is needed.\n3. **Registry credential** — not required. The workflow YAMLs no longer reference an `nvcr_io` (or any REGISTRY) credential; pulls succeed anonymously. If image pulls fail (authorization error or `nvcr.io` rate-limiting), see `references/troubleshooting.md` → **\"nvcr.io image pull failures\"** for how to add an NGC pull credential.\n4. **OSMO `hf-token` credential** (GENERIC) — required by every group that hits HF (i.e. everything except `download-metal_surface-data`). Accept the model license on each gated page once before first run:\n - https://huggingface.co/nvidia/Cosmos-Predict2-2B-Text2Image\n - https://huggingface.co/nvidia/Cosmos-Predict2-14B-Text2Image\n - https://huggingface.co/nvidia/Cosmos-AnomalyGen-PCB-2B\n - https://huggingface.co/nvidia/Cosmos-AnomalyGen-Metal-2B\n - https://huggingface.co/nvidia/Cosmos-AnomalyGen-Glass-2B\n - https://huggingface.co/datasets/nvidia/Cosmos-AnomalyGen-PCB-Dataset\n - https://huggingface.co/datasets/nvidia/Cosmos-AnomalyGen-Glass-Masks\n - https://huggingface.co/datasets/nvidia/Spark-AnomalyGen-USD\n ```bash\n osmo credential set hf-token --type GENERIC \\\n --payload token=\"$HF_TOKEN\"\n ```\n5. **Pod-template prerequisites (simulation + finetuning)** — two cluster-level mounts gate DIG GPU work. If a preflight or in-pod check trips on either, **tell the user which mount is missing and why it matters, and seek approval before routing to `physical-ai-infrastructure-setup-and-resilient-scaling`** — that fix mutates the cluster-wide `POD_TEMPLATE`.\n - **`/usr/share/nvidia/nvoptix.bin`** (OptiX denoiser binary) — required by the IsaacSim render tasks (`usd2roi-render`, `usd2roi-render-day1`, `sdg-and-crop`); hostPath-mounted at the same path. Without it Kit silently degrades to noisy raw path tracing (no error, no non-zero exit) — it gates render/ROI quality.\n - **`/dev/shm` ≥ 16 GiB (32 GiB preferred)** — required by **both** the IsaacSim ray-tracer (intermediate buffers) **and the finetuning/training tasks** (`finetune.yaml`, the Day-0 train step, Day-1 finetune-from-scratch), where it backs torchrun shared-memory. Undersized → in-pod preflight fails or torchrun OOMs mid-training.\n - **Asset directory perms** — handled inside OSMO by the workflow's pre-task `chmod 777 $OUT`.\n\n The OSMO pod template controls both mounts. Validate via `scripts/preflight_pod_template.sh` (or the infra skill's pod-template gate); the in-pod runtime preflight on every OV + training task is the backstop.\n\n The two IsaacSim Day-0 workflows pin `isaac_render_image` to `nvcr.io/nvidia/paidf-simulation:1.0.0`. See `references/container-images.md` for the canonical tag table.\n\n## Credential check\n\n`scripts/preflight_credentials.sh` is the canonical front door for prereqs §1–§4. The only credential the workflows require is the OSMO credential `hf-token` (GENERIC) — that is what gates the Hugging Face downloads every flow performs. There is **no registry credential requirement**: the `paidf-*` images are public on `nvcr.io/nvidia/` and pull anonymously. The env var `HF_TOKEN` is only needed when (a) `hf-token` is missing and the script needs to **auto-set** it, or (b) you want the outbound HF probe to verify the token still has read scope on the gated `nvidia/Cosmos-AnomalyGen-*` repos. **If `hf-token` is already provisioned and you skip probes, no env var is needed.** Run it before submitting any setup workflow and before every flow submission.\n\n> **Check for a workspace `.env` first.** Before running the credential check, if a `.env` file exists in the agent's workspace, source it so its credentials are exported — `set -a; . ./.env; set +a`. It commonly carries `HF_TOKEN` (and `NGC_API_KEY` for the image-pull fallback), letting the script auto-set the OSMO credential without prompting the user.\n\n```bash\n# Default: probe HF (using exported HF_TOKEN), auto-set any missing OSMO credential\nbash skills/physical-ai-defect-image-generation/scripts/preflight_credentials.sh\n\n# Restricted-egress shells — skip the outbound HTTPS probe\nbash skills/physical-ai-defect-image-generation/scripts/preflight_credentials.sh --no-probe\n```\n\nSample success (`hf-token` already provisioned, `HF_TOKEN` not exported, probes off):\n\n```\nnote: skipping HF probe — HF_TOKEN not exported (OSMO credential 'hf-token' is already provisioned).\nOK: OSMO credential hf-token present (paidf-* images are public on nvcr.io/nvidia/ — no registry credential needed).\n```\n\nSample failure (HF token can't read gated AnomalyGen repos):\n\n```\nHF gated-repo probe failed (HTTP 401) at https://huggingface.co/api/models/nvidia/Cosmos-AnomalyGen-PCB-2B\n HF_TOKEN cannot read the gated Cosmos-AnomalyGen repos. Accept the license once at each:\n https://huggingface.co/nvidia/Cosmos-AnomalyGen-PCB-2B\n https://huggingface.co/nvidia/Cosmos-AnomalyGen-Metal-2B\n https://huggingface.co/nvidia/Cosmos-AnomalyGen-Glass-2B\n```\n\nUse the manual `curl` snippet from Prerequisites §1 to inspect the raw response if the probe fails.\n\n## OSMO setup workflows\n\nSetup is split across four narrow workflows under `assets/configs/setup/`. Submit only the workflows the use case actually needs — every workflow has no submit-time list parameter, so there is nothing to ignore. The pretrained bundle is its own workflow because every use case needs it; submit it in parallel with whichever case workflows you need. All groups run inside the `paidf-anomalygen` image — it ships `hf` CLI, `prepare_dataset_uc{1,2,3}.py`, and `download_anomalygen_checkpoints.sh`.\n\n| Workflow | Groups | Output URL artifacts |\n|---|---|---|\n| `setup/setup_pretrained.yaml` | `download-pretrained` (HF Cosmos-Predict2 + T5 + dinov2 + container-baked NVDINOV2/SAM2/Qwen + HF C-RADIOv3-B) | `models/pretrained` |\n| `setup/setup_pcb.yaml` | `download-pcb-model` (HF `nvidia/Cosmos-AnomalyGen-PCB-2B`), `download-pcb-data` (`prepare_dataset_uc1` on HF `nvidia/Cosmos-AnomalyGen-PCB-Dataset`), `download-pcb-assets` (HF `nvidia/Spark-AnomalyGen-USD`) | `models/pcb`, `datasets/pcb/raw`, `datasets/pcb/assets` |\n| `setup/setup_metal.yaml` | `download-metal_surface-model` (HF `nvidia/Cosmos-AnomalyGen-Metal-2B`), `download-metal_surface-data` (`prepare_dataset_uc2` on public GitHub [`abin24/Magnetic-tile-defect-datasets.`](https://github.com/abin24/Magnetic-tile-defect-datasets.); no HF token, needs outbound github.com) | `models/metal_surface`, `datasets/metal_surface/raw` |\n| `setup/setup_glass.yaml` | `download-glass-model` (HF `nvidia/Cosmos-AnomalyGen-Glass-2B`), `download-glass-data` (`prepare_dataset_uc3 --masks-from-hf` on HF `nvidia/Cosmos-AnomalyGen-Glass-Masks` + user Roboflow zip URL-mounted via `uc3_zip_url_root`) | `models/glass`, `datasets/glass/raw` |\n\nPure download/assembly only — no GPU work. Validation JSONL + AMP placements are produced downstream at finetune / inference time (per the anomalygen skill contract: Phase 1 Step 2 for validation, Phase 2 for inference). Each group writes to its own URL output under `dig_url_root`. The four workflows have no inter-dependencies; submit any subset in parallel.\n\n```bash\n# Validate each spec first — catches name/credential/shape errors without queuing\nosmo workflow validate skills/physical-ai-defect-image-generation/assets/configs/setup/setup_pretrained.yaml\nosmo workflow validate skills/physical-ai-defect-image-generation/assets/configs/setup/setup_pcb.yaml\n\n# PCBA-only path — pretrained bundle + PCB assets, no metal/glass:\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/setup/setup_pretrained.yaml --pool \u003cpool>\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/setup/setup_pcb.yaml --pool \u003cpool>\n\n# PCBA + metal_surface — submit pretrained once, plus the two case workflows:\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/setup/setup_pretrained.yaml --pool \u003cpool>\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/setup/setup_pcb.yaml --pool \u003cpool>\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/setup/setup_metal.yaml --pool \u003cpool>\n\n# Glass — REQUIRES the Roboflow zip to be uploaded first (see \"Glass case\" steps below):\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/setup/setup_pretrained.yaml --pool \u003cpool>\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/setup/setup_glass.yaml --pool \u003cpool> \\\n --set uc3_zip_url_root=s3://osmo-workflows/dig/uploads/glass-zip\n```\n\n**Glass case (UC3) — Roboflow zip prerequisite. Do this BEFORE submitting `setup_glass.yaml`:**\n\n1. **Download** the COCO export once (license-gated, browser flow required — Roboflow does not support unauthenticated programmatic download):\n - Visit https://universe.roboflow.com/vu-thi-thu-huyen/mobile-screen\n - Click **Export Dataset** → **COCO** format → download the zip\n - Accept the dataset license/terms once\n2. **Rename the file to exactly `mobile_screen.zip`** (the workflow looks for that literal filename inside the staged dir):\n ```bash\n mv /path/to/\u003croboflow-download>.zip /tmp/mobile_screen.zip\n ```\n3. **Upload to an OSMO URL prefix** (any prefix you control; this skill's docs use `s3://osmo-workflows/dig/uploads/glass-zip/` by convention):\n ```bash\n osmo data upload s3://osmo-workflows/dig/uploads/glass-zip/ /tmp/mobile_screen.zip\n # Verify it landed:\n osmo data list --no-pager s3://osmo-workflows/dig/uploads/glass-zip/\n # Should show: mobile_screen.zip\n ```\n > ⚠ **Always pass the destination as a trailing-slash prefix** (`.../glass-zip/`), NOT as a key (`.../glass-zip/mobile_screen.zip`). The OSMO data adapter (MinIO-compatibility edge case) treats a no-slash key whose tail matches an existing prefix as a prefix itself and creates `mobile_screen.zip/mobile_screen.zip` (the outer is a directory). The workflow's `[ -f \"$UC3_ZIP_DIR/mobile_screen.zip\" ]` then fails because `-f` returns false on directories. If you've already uploaded with the key form, list the prefix, remove the nested directory, and re-upload with the trailing-slash form.\n4. **Then submit** with `--set uc3_zip_url_root=\u003cthat-prefix>` (the prefix, not the file itself).\n\nThe workflow URL-mounts the prefix into the task at `{{input:0}}`, copies the zip\nto `/tmp/uc3_input.zip`, and runs `prepare_dataset_uc3.py` to extract images\nalongside the masks + `defect_spec.jsonl` pulled from `cosmos-anomalygen-glass-masks`.\n\n> **Why URL-mounted, not `localpath:`?** OSMO's `localpath:` mechanism reads every\n> staged file as UTF-8 text during `validate`/`submit` and rejects binary zips with\n> `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 ...`. URL-mounted inputs\n> sidestep that codepath entirely.\n\nSubmitting `setup_glass.yaml` with an empty `uc3_zip_url_root` fails at `osmo workflow validate` (OSMO rejects an empty URL input).\n\n**Knobs** (override via `--set`; pass multiple as `--set k1=v1 k2=v2`; scope column lists which workflow files accept the knob):\n\n| Param | Default | Scope | Notes |\n|---|---|---|---|\n| `uc3_zip_url_root` | `\"\"` (empty — required) | `setup_glass.yaml` only | **OSMO URL prefix** (not a local path) containing `mobile_screen.zip` — the user-downloaded Roboflow Mobile-Screen COCO export. Upload the zip to this prefix **before** submitting (`osmo data upload \u003cprefix>/ \u003clocal-zip>`). The workflow URL-mounts the prefix into the `glass-data` task and copies `mobile_screen.zip` to `/tmp/uc3_input.zip` for `prepare_dataset_uc3.py --masks-from-hf`. URL-mounted (not `localpath:`) because OSMO CLI fails UTF-8 decode on binary zips during validate/submit. |\n| `dig_url_root` | `s3://osmo-workflows/dig` | all four | Single DIG root. Setup writes checkpoints under `models/\u003ccase>`, pretrained under `models/pretrained`, raw training data under `datasets/\u003ccase>/raw`, and CAD assets under `datasets/\u003ccase>/assets`. |\n| `pretrained_image` | See `references/container-images.md` | all four | Image for **every** download group — ships repo, baked checkpoints, `hf` CLI, `download_anomalygen_checkpoints.sh`, and the three `prepare_dataset_uc*.py` scripts. Public on `nvcr.io/nvidia/`; pulled anonymously (no registry credential). |\n| `cpu` / `memory` / `storage` | `1` / `2Gi` / `10Gi` | `setup_pcb.yaml`, `setup_metal.yaml`, `setup_glass.yaml` | Sizing for the per-UC model and dataset groups + the pcb-assets group. |\n| `pretrained_model_sizes` | `\"2B\"` | `setup_pretrained.yaml` | Space-separated; `\"2B 14B\"` adds the 14B Cosmos-Predict2 checkpoint (~64 GB extra). With `2B 14B`, also bump `storage_large` to ≥300Gi. |\n| `cpu_large` / `memory_large` / `storage_large` | `1` / `16Gi` / `220Gi` | `setup_pretrained.yaml` | The default 2B bundle is about 71 GB; HF Hub streams to disk and does not need high RAM. |\n\n**Watch progress:** live status and per-task logs — see `SKILL.md` §\"OSMO Monitoring\".\n\n**Verify uploaded artifacts** once the setup workflow finishes:\n\n```bash\nosmo data list --no-pager \u003cdig_url_root>/\n```\n\n## URL artifact layout\n\nURL inputs mount at the requested URL contents rather than under an OSMO dataset name. The setup workflow flattens the NGC version wrapper for generated URL artifacts.\n\n### Per-artifact root layout + key files\n\n| Artifact URL path | Root contents | Notes |\n|---|---|---|\n| `models/pcb` | `ag_config.yaml`, `iter_000014000.pt` | Emitted by `download_anomalygen_checkpoints.sh --uc pcb` from `nvidia/Cosmos-AnomalyGen-PCB-2B`. |\n| `models/metal_surface` | `ag_config.yaml`, `iter_000010000.pt` | Emitted by `download_anomalygen_checkpoints.sh --uc metal` from `nvidia/Cosmos-AnomalyGen-Metal-2B`. (Script arg stays `--uc metal` — its HF repo identifier; OSMO storage path uses canonical `metal_surface`.) |\n| `models/glass` | `ag_config.yaml`, `iter_000009000.pt` | Emitted by `download_anomalygen_checkpoints.sh --uc glass` from `nvidia/Cosmos-AnomalyGen-Glass-2B`. |\n| `datasets/pcb/raw` | `PCB/`, `defect_spec.jsonl`, `semantic_segmentation_labels.json` | Emitted by `prepare_dataset_uc1.py` from `nvidia/Cosmos-AnomalyGen-PCB-Dataset`; finetune/inference generates validation.jsonl + amp/ on the fly. |\n| `datasets/metal_surface/raw` | `metal_surface/{anomaly_image,mask,clean_image}/`, `defect_spec.jsonl` | UC2 prep-script output (downloaded from public GitHub by `prepare_dataset_uc2.py` at setup time); curated 5+5+20 subset matching the reference UC2 dataset. |\n| `datasets/glass/raw` | `Phone/{anomaly_image,clean_image,mask}/`, `defect_spec.jsonl` | Images from user's Roboflow zip; masks + defect_spec from `nvidia/Cosmos-AnomalyGen-Glass-Masks` (HF) overlaid by `prepare_dataset_uc3.py --masks-from-hf`. Material dir is **`Phone`**, not `glass` / `Glass`. |\n| `datasets/pcb/assets` | `spark_lighting.usd`, `pcba_main_s_detail.usd`, `pcba_base.usd`, `aoi_ring_light.usda`, `materials/`, `component/`, `ECAD_3D/`, `PCBA/` | Pulled from `nvidia/Spark-AnomalyGen-USD` (HF dataset repo) via `hf download --repo-type dataset`. |\n| `models/pretrained` | `pretrained/` | Sub-trees per provider (NVDINOV2, nvidia, google-t5, facebook, ...). |\n\n### Per-use-case parameter values\n\nPull these from the actual checkpoint filenames + material subdirs above. Day 0 and Day 1 defaults already target the shipped PCBA checkpoint (step 14000); the `anomaly_types_json` knob on both flows defaults to the PCBA taxonomy.\n\n| Use case | `checkpoint_step` | `anomaly_types_json` (checkpoint-keyed) | Material subdirs under `datasets/\u003ccase>/raw` |\n|---|---|---|---|\n| PCBA | `14000` | `[[\"IC\",\"bridge\"],[\"passive_component\",\"excess_solder\"],[\"passive_component\",\"missing\"]]` | `IC/`, `passive_component/` |\n| Metal surface | `10000` | `[[\"metal_surface\",\"MT_Blowhole\"],[\"metal_surface\",\"MT_Break\"],[\"metal_surface\",\"MT_Crack\"],[\"metal_surface\",\"MT_Fray\"],[\"metal_surface\",\"MT_Uneven\"]]` | `metal_surface/` |\n| Glass | `9000` | `[[\"Phone\",\"oil\"],[\"Phone\",\"scratch\"],[\"Phone\",\"stain\"]]` | `Phone/` |\n\n> **Glass material name `Phone`** is intentional — the source dataset was authored against a phone-screen taxonomy. PCBA spans two materials (IC + passive_component) under one shipped checkpoint. Verify against the checkpoint's `ag_config.yaml` `anomaly_types` field before submitting.\n\n### `pcba_target.yaml` is mounted from the cookbook\n\nThe PCBA assets bundle ships only the USD tree (15+ `.usd`/`.usda` files + materials). Day 0's `usd2roi-render` mounts `assets/cookbooks/pcb/pcba_target.yaml`, `day0_image.yaml` (with mesh-level semantics inlined), and `day0_crop.yaml` into the task via `files: - localpath:` at submit time, so the dataset doesn't need to ship them. No side-load required.\n\n## Bring your own data or models\n\nUse the same URL layout for custom DIG artifacts. This keeps the setup workflow, manual uploads, and future external S3 paths using the same contract under one DIG root.\n\n```bash\nDIG_ROOT=s3://osmo-workflows/dig\nCASE=\u003ccase-name>\n\n# Custom checkpoint model. The root should contain ag_config.yaml and the\n# iter_*.pt file that matches the checkpoint_step you pass to the workflow.\nosmo data upload \"${DIG_ROOT}/models/${CASE}/\" \\\n /path/to/checkpoint_dir/*\n\n# Raw training/inference data. The root should contain the material\n# directory and defect_spec.jsonl. validation.jsonl + amp/ are NOT required —\n# the finetune/inference tasks build them inline via prep_testcase.sh.\nosmo data upload \"${DIG_ROOT}/datasets/${CASE}/raw/\" \\\n /path/to/raw_data_dir/*\n\n# Optional CAD/USD assets for PCBA-like usd2roi flows.\nosmo data upload \"${DIG_ROOT}/datasets/${CASE}/assets/\" \\\n /path/to/usd_asset_tree/*\n```\n\n`osmo data upload` nests each supplied local path's basename. Use `/*` when the contents of a local directory should become the URL root, and use the directory path itself only when you intentionally want an extra top-level folder.\n\nAfter upload, verify both access and shape:\n\n```bash\nosmo data check --access-type READ \"${DIG_ROOT}/models/${CASE}/\"\nosmo data list --no-pager \"${DIG_ROOT}/models/${CASE}/\"\nosmo data list --no-pager \"${DIG_ROOT}/datasets/${CASE}/raw/\"\n```\n\nKeep the material names aligned with `anomaly_types_json` and the checkpoint's `ag_config.yaml`. A minimal raw tree has `\u003cMATERIAL>/clean_image/`, `\u003cMATERIAL>/mask/\u003cdefect>/`, and `defect_spec.jsonl`. CAD-guided cases also need `cad_mask/` and `semantic_segmentation_labels.json` where the cookbook expects them.\n\n### Naming gotcha: `--set` parses numeric-looking values as PEP 515 ints\n\n`osmo workflow submit --set \u003ckey>=\u003cvalue>` casts values to int or float when they look numeric — including Python PEP 515 underscore-grouped integers. A value like `115_2819_000` becomes the int `1152819000` (underscores stripped) before Jinja renders it, so `{{ board }}` resolves to `1152819000` even though you passed `115_2819_000`. If the cookbook directory is `assets/cookbooks/pcb/115_2819_000/` the workflow's `localpath: ../cookbooks/pcb/{{ board }}/usd2roi_nvpcb.yaml` mount fails with \"file not found\" on `1152819000`.\n\n**Safe board / case directory names:**\n- Pure digits (`1152819000`) — round-trips through int correctly.\n- Names starting with a letter (`H100`, `board_a`, `b001`) — never int-cast.\n- Hyphen-separated (`115-2819-000`) — never int-cast.\n\n**Avoid:** underscore-grouped digit sequences (`115_2819_000`, `1_000_000`).\n\nThe shipped PCBA alternate board cookbook directory is `assets/cookbooks/pcb/1152819000/` (pure digits) for exactly this reason. The matching real photo `input_real_image/115_2819_000.jpg` inside `datasets/pcb/assets` keeps its original underscore-grouped name (it's a file inside an artifact, not a Jinja-templated path component) — pass it via `--set real_image_filename=input_real_image/115_2819_000.jpg` explicitly when targeting that board. Use `--set-string` instead of `--set` only as a last resort — it disables numeric casting for all `--set-string` values, which usually you don't want.\n\n## Wiring into Day 0 / Day 1 / finetune\n\nDay 0, Day 1, and finetune all consume URL artifacts directly. The only storage knob is `dig_url_root`; the workflow derives inputs from the fixed layout:\n\n- checkpoints: `\u003cdig_url_root>/models/\u003cusecase>`\n- pretrained: `\u003cdig_url_root>/models/pretrained`\n- raw training data: `\u003cdig_url_root>/datasets/\u003cusecase>/raw`\n- PCBA CAD assets: `\u003cdig_url_root>/datasets/pcb/assets`\n- Day 1 real-photo alignment inputs: ships inside `\u003cdig_url_root>/datasets/pcb/assets` (canonical `pcb-assets`: per-board photos under `input_real_image/\u003cboard>.jpg`)\n- run outputs: `\u003cdig_url_root>/runs/\u003cname>/\u003cstage>`\n\nUse `usecase=pcb`, `usecase=metal_surface`, or `usecase=glass` in workflow submits — uniform across `--set usecase=`, URL paths (`datasets/\u003cusecase>/raw`, `models/\u003cusecase>`), and cookbook directories (`assets/cookbooks/\u003cusecase>/`). The `metal_surface` value matches the trained model's material name baked into the checkpoint taxonomy.\n\n### Day 0 (PCBA, full pipeline)\n\nDefaults already target the shipped PCBA checkpoint passthrough; the default DIG root is `s3://osmo-workflows/dig`. Use an existing endpoint or deploy the local cluster endpoint from `references/nim/`:\n\n```bash\nSTAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/texture_defect_generation_day0.yaml \\\n --pool \u003cpool> --set name=texture_defect_gen_day0-$STAMP \\\n dig_url_root=\u003cdig_url_root> \\\n image_edit_endpoint=http://qwen-image-edit-nvpcb-ovsl2sl.osmo-nims.svc.cluster.local:8000/v1 \\\n image_edit_model=nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL\n```\n\nTo finetune from scratch instead of passthrough, just add `use_pretrained_checkpoint=false` — the cookbook is rendered in-pod by `yq` after Phase 1 Step 2 (no pre-submit render needed). Defaults are 1 GPU end-to-end; bump `train_gpu=N infer_gpu=N` to scale (set them individually to break symmetry).\n\n### Day 1 (Metal surface or Glass)\n\nReplace `\u003cusecase>` with `metal_surface` or `glass`. Per-use-case overrides:\n\n| Use case | `checkpoint_step` | `anomaly_types_json` |\n|---|---|---|\n| Metal surface | `10000` | `[[\"metal_surface\",\"MT_Blowhole\"],[\"metal_surface\",\"MT_Break\"],[\"metal_surface\",\"MT_Crack\"],[\"metal_surface\",\"MT_Fray\"],[\"metal_surface\",\"MT_Uneven\"]]` |\n| Glass | `9000` | `[[\"Phone\",\"oil\"],[\"Phone\",\"scratch\"],[\"Phone\",\"stain\"]]` |\n\n```bash\nSTAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/texture_defect_generation_day1_manual_roi.yaml \\\n --pool \u003cpool> --set name=texture_defect_gen_day1_manual_roi-$STAMP \\\n dig_url_root=\u003cdig_url_root> \\\n usecase=\u003cusecase> \\\n use_pretrained_checkpoint=true \\\n checkpoint_step=\u003csee table above> \\\n 'anomaly_types_json=\u003clist>'\n```\n\n> The raw URL contains clean_image, submasks, cad_mask, `defect_spec.jsonl`, and `semantic_segmentation_labels.json` together. When it ships its own `defect_spec.jsonl` (Mode A), `anomaly_types_json` is unused. For finetune-from-scratch on Day 1, add `use_pretrained_checkpoint=false` — the finetune task builds the validation set fresh via `prep_testcase.sh` inside the pod.\n\n### Finetune flow (from scratch — Finetune Only)\n\nUse `\u003cdig_url_root>/datasets/\u003cusecase>/raw` as the training source. The checkpoint output (`\u003cdig_url_root>/runs/\u003cname>/finetune`) is reusable as a checkpoint URL in Day 0 / Day 1 by pointing their input at that run output or by copying it into `\u003cdig_url_root>/models/\u003cusecase>`.\n\n```bash\n# Cookbook is rendered in-pod (5 yq patches + trainer.early_stop drop) — see\n# references/flows/finetune.md §\"Cookbook render (in-pod, automatic)\".\nSTAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/finetune.yaml \\\n --pool \u003cpool> --set name=finetune-$STAMP \\\n dig_url_root=\u003cdig_url_root> \\\n usecase=\u003cusecase>\n```\n\nTraining-recipe knobs (`lr`, `max_iter`, `anomaly_types`, etc.) live in the cookbook (`assets/cookbooks/\u003cusecase>/ag_config.yaml`), not the `--set` list. To override, add more yq expressions at render time.\n\n## Troubleshooting\n\n- **`preflight_credentials.sh` exits 1** — read the printed remediation. Common cases: the `hf-token` credential is missing **and** `HF_TOKEN` isn't exported (export it and re-run, or provision the credential directly); HF probe `401`/`403` (token lacks read scope on the gated `nvidia/Cosmos-AnomalyGen-*` or `nvidia/Cosmos-Predict2-*` repos — accept the license at each repo page and regenerate the token if needed); OSMO `set` fails (check `osmo profile list` and that the `osmo` CLI is logged in). Use `--no-probe` only if outbound HTTPS is blocked, not to mask a 401. **No env var is required when `hf-token` is already provisioned** — the script skips the probe and exits 0. There is no registry-credential check (images are public).\n- **URL output rejected** — use plain `outputs: - url: s3://...`; `dataset.url` is not accepted by the current OSMO schema.\n- **Validator: `\u003cfield> is not a valid credential key please choose from dict_keys([...])`** — the workflow yaml references a credential field that doesn't exist on the stored credential. OSMO credentials store fields under the exact keys passed in `--payload`. For the `hf-token` GENERIC credential the field is `token` — re-set as `osmo credential set hf-token --type GENERIC --payload token=\"$HF_TOKEN\"` if the field is missing. (If you added an `nvcr_io` REGISTRY credential to work around image-pull failures, its canonical field is `auth` — see `references/troubleshooting.md` → \"nvcr.io image pull failures\".)\n- **HF gated probe returns `401`/`403`** — the HF token cannot read one of the gated `nvidia/Cosmos-AnomalyGen-*` or `nvidia/Cosmos-Predict2-*` repos. Visit each repo page in a browser, click \"Agree and access repository,\" then regenerate the token if it predates the license acceptance.\n- **Workflow says COMPLETED but outputs are hard to find** — query the DIG root with `osmo data list --no-pager s3://osmo-workflows/dig/`.\n- **`ERROR: 0 files in \u003cout>`** — the HF token lacks read access to the target repo, the license hasn't been accepted, or the repo name is wrong. Confirm with the curl probe in Prerequisites §1, then re-run.\n- **Metal UC2 data download fails** — re-submit `setup/setup_metal.yaml`. The upstream GitHub slug must include the trailing period: `abin24/Magnetic-tile-defect-datasets.`; a slug without the final `.` resolves to a different (non-existent) repository and `prepare_dataset_uc2` will fail.\n- **`hf: command not found` / `huggingface-cli not found`** — the task is not running inside `pretrained_image`. Confirm `image:` in the failing group points at the canonical `paidf-anomalygen` tag from `references/container-images.md`.\n- **Storage exhaustion** — bump `storage` (per-UC + assets tasks) or `storage_large` (pretrained). 10 GiB fits each per-UC download with headroom; the pretrained 220 GiB fits a 71 GB `model_sizes=2B` run, raise to ≥300 GiB for `2B 14B`.\n- **One workflow fails, the rest succeed** — re-submit just the failing workflow file (`setup_pcb.yaml` / `setup_metal.yaml` / `setup_glass.yaml` / `setup_pretrained.yaml`); the successful URL artifacts persist under `dig_url_root`.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":33196,"content_sha256":"1a35db295d53423dfa50b397b193b7afb1234d19a73afb359b69ef0458d91670"},{"filename":"references/troubleshooting.md","content":"# Defect Image Generation Workflow — Troubleshooting\n\n\n## Table of Contents\n\n- [When to Consult Component Skills](#when-to-consult-component-skills)\n- [URL Layout](#url-layout)\n- [Preflight](#preflight)\n- [Shipped Taxonomies](#shipped-taxonomies)\n- [Canonical Submit Commands](#canonical-submit-commands)\n- [Output Retrieval](#output-retrieval)\n- [Common Failures](#common-failures)\n- [IsaacSim Render (structural_defect_generation.yaml)](#isaacsim-render-structural_defect_generationyaml)\n- [usd2roi-render (good_image_generation.yaml / texture_defect_generation_day0.yaml)](#usd2roi-render-good_image_generationyaml-texture_defect_generation_day0yaml)\n- [nvcr.io image pull failures](#nvcrio-image-pull-failures)\n\nOperational gotchas, failure-mode recipes, and canonical submit commands for the\nURL-based Defect Image Generation workflows. For canonical image tags, see\n`references/container-images.md`.\n\n## When to Consult Component Skills\n\n| Symptom / question | Owning skill | Look for |\n|---|---|---|\n| Kit, `sdg_pipeline.py`, `usd2roi_crop.py`, missing semantic classes, `pcba_target.yaml`, scan-grid tuning, USD scene selection | `skills/simulation/SKILL.md` (usd2roi component skill) | Day-0 SDG config, semantic rules, writer flags, per-cell ROI crop logic |\n| Image-edit prompt, endpoint calls, letterbox, guidance scale, batch config expansion, endpoint timeouts | `skills/augmentation/SKILL.md` | Remote endpoint executor and augmentation cookbook schema |\n| AMP routing (`free` / `text` / `cad`), `defect_spec.jsonl`, `prep_testcase.sh`, checkpoint validation, `run_sdg.sh`, training output layout, best-step selection | `skills/anomalygen/SKILL.md` | Phases 0-7, data structure, defect spec, training and inference mechanics |\n| OSMO pool/quota, workflow submit/query/logs/events/cancel, URL storage, `osmo data upload/download/list`, credentials, image pull errors, pod-template mount issues | `skills/physical-ai-infrastructure-setup-and-resilient-scaling/SKILL.md` | CLI reference, workflow YAML v2, credential payloads, URL output conventions |\n\n**Decision shortcut by task name:** `usd2roi-render` -> usd2roi component skill\n(`skills/simulation/SKILL.md`), `augment-image-edit` ->\naugmentation, `finetune` or `anomaly-infer` -> anomalygen.\nWorkflow YAML, Jinja, storage URL, pool, credential, and image pull issues stay\nwith this skill plus `physical-ai-infrastructure-setup-and-resilient-scaling`.\n\n## URL Layout\n\nDefault root:\n\n```bash\nDIG_ROOT=s3://osmo-workflows/dig\n```\n\n| Use | URL |\n|---|---|\n| PCBA checkpoint | `${DIG_ROOT}/models/pcb` |\n| Metal checkpoint | `${DIG_ROOT}/models/metal_surface` |\n| Glass checkpoint | `${DIG_ROOT}/models/glass` |\n| Pretrained weights | `${DIG_ROOT}/models/pretrained` |\n| Raw training data | `${DIG_ROOT}/datasets/\u003cusecase>/raw` |\n| PCBA CAD assets | `${DIG_ROOT}/datasets/pcb/assets` |\n| Day 1 real-photo alignment | `${DIG_ROOT}/datasets/pcb/assets/input_real_image/\u003cboard>.jpg` (ships inside canonical `pcb-assets`; e.g. `0603_H100.jpg`, `115_2819_000.jpg`) |\n| Run output | `${DIG_ROOT}/runs/\u003cname>/\u003cstage>` |\n\nBuilt-in workflow `usecase` values are `pcb`, `metal_surface`, and `glass`. The metal\n`usecase` is uniformly `metal_surface` — the cookbook lives at `assets/cookbooks/metal_surface/`, URL paths use `datasets/metal_surface/raw` + `models/metal_surface`, and the trained taxonomy's material name matches (`anomaly_types_json=[[\"metal_surface\",\"MT_*\"],...]`).\n\n## Preflight\n\n```bash\nbash scripts/preflight_credentials.sh\nDIG_URL_ROOT=s3://osmo-workflows/dig bash scripts/preflight_urls.sh 0 pcb\nDIG_URL_ROOT=s3://osmo-workflows/dig bash scripts/preflight_urls.sh 1 metal_surface\nDIG_URL_ROOT=s3://osmo-workflows/dig bash scripts/preflight_urls.sh 1 glass\nDIG_URL_ROOT=s3://osmo-workflows/dig bash scripts/preflight_urls.sh finetune pcb\n```\n\nFor finetune-from-scratch checks:\n\n```bash\nUSE_PRETRAINED_CHECKPOINT=false DIG_URL_ROOT=s3://osmo-workflows/dig \\\n bash scripts/preflight_urls.sh 1 metal_surface\n```\n\nFor Day 1 PCBA real-photo alignment:\n\n```bash\nDIG_URL_ROOT=s3://osmo-workflows/dig bash scripts/preflight_urls.sh 1 pcb real-alignment\n```\n\n## Shipped Taxonomies\n\n| Usecase | `checkpoint_step` | `anomaly_types_json` |\n|---|---|---|\n| `pcb` | `14000` | `[[\"IC\",\"bridge\"],[\"passive_component\",\"excess_solder\"],[\"passive_component\",\"missing\"]]` |\n| `metal` | `10000` | `[[\"metal_surface\",\"MT_Blowhole\"],[\"metal_surface\",\"MT_Break\"],[\"metal_surface\",\"MT_Crack\"],[\"metal_surface\",\"MT_Fray\"],[\"metal_surface\",\"MT_Uneven\"]]` |\n| `glass` | `9000` | `[[\"Phone\",\"oil\"],[\"Phone\",\"scratch\"],[\"Phone\",\"stain\"]]` |\n\n`anomaly_types_json` must match the checkpoint's `ag_config.yaml`\n`anomaly_types` list exactly. PCBA spans two materials; do not collapse it into a\nsingle material.\n\n## Canonical Submit Commands\n\nDay 0 PCBA passthrough:\n\n```bash\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/texture_defect_generation_day0.yaml \\\n --pool default \\\n --set name=pcb-e2e-$(date +%Y%m%d-%H%M) \\\n dig_url_root=\u003cdig_url_root> \\\n image_edit_endpoint=http://qwen-image-edit-nvpcb-ovsl2sl.osmo-nims.svc.cluster.local:8000/v1 \\\n image_edit_model=nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL\n```\n\nDay 1 metal_surface passthrough:\n\n```bash\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/texture_defect_generation_day1_manual_roi.yaml \\\n --pool default \\\n --set name=metal_surface-demo-$(date +%Y%m%d-%H%M) \\\n dig_url_root=\u003cdig_url_root> \\\n usecase=metal_surface \\\n checkpoint_step=10000 \\\n 'anomaly_types_json=[[\"metal_surface\",\"MT_Blowhole\"],[\"metal_surface\",\"MT_Break\"],[\"metal_surface\",\"MT_Crack\"],[\"metal_surface\",\"MT_Fray\"],[\"metal_surface\",\"MT_Uneven\"]]'\n```\n\nDay 1 glass passthrough:\n\n```bash\nosmo workflow submit skills/physical-ai-defect-image-generation/assets/configs/texture_defect_generation_day1_manual_roi.yaml \\\n --pool default \\\n --set name=glass-demo-$(date +%Y%m%d-%H%M) \\\n dig_url_root=\u003cdig_url_root> \\\n usecase=glass \\\n checkpoint_step=9000 \\\n 'anomaly_types_json=[[\"Phone\",\"oil\"],[\"Phone\",\"scratch\"],[\"Phone\",\"stain\"]]'\n```\n\nSmoke-test knobs: add `render_patches=5 num_sdg=15` for Day 0, or\n`num_sdg=15` for Day 1.\n\n## Common Failures\n\n- **`ERROR: /usr/share/nvidia/nvoptix.bin not mounted`** (OV tasks: `usd2roi-render`, `usd2roi-render-day1`, `sdg-and-crop`) — the OSMO pod template is missing the OptiX denoiser binary hostPath mount. Without it, Kit silently falls back to raw path tracing → noisy ROI output. Invoke the `physical-ai-infrastructure-setup-and-resilient-scaling` skill to patch the pod template (`osmo config update POD_TEMPLATE`) and re-submit.\n- **`ERROR: /dev/shm is NN GiB; need >= 16 GiB`** (OV + training tasks) — the OSMO pod template's `emptyDir` for `/dev/shm` is undersized. Same fix: patch via `physical-ai-infrastructure-setup-and-resilient-scaling`. 32 GiB is the recommended size for ray-tracer buffers (OV) and torchrun shared-memory (training).\n- **URL output rejected** — use `outputs: - url: s3://...`; `dataset.url` is not accepted by the current OSMO schema.\n- **Missing URL artifacts** — submit the relevant `setup/setup_\u003ccase>.yaml` + `setup/setup_pretrained.yaml`, or upload data with `osmo data upload` under the same DIG root.\n- **Checkpoint taxonomy mismatch** — trim `anomaly_types_json` to the shipped table or retrain via `finetune.yaml`.\n- **`ag_config.yaml not found in checkpoint`** — checkpoint URL does not contain the expected training config alongside weights.\n- **`ERROR: pretrained tree not at .../pretrained`** — rerun setup for `models/pretrained`.\n- **`ERROR: $DATASET_DIR/defect_spec.jsonl missing in raw dataset`** — the raw data URL is incomplete; rerun setup for the usecase.\n- **`ERROR: prep_testcase.sh produced an empty validation.jsonl`** (finetune-from-scratch) — the raw dataset has no training masks under `\u003cMATERIAL>/mask/\u003cdefect>/`.\n- **Submask lookup failed** — raw data must have `\u003cmaterial>/mask/\u003cdefect>/` for every pair in `anomaly_types_json`, with a flat `\u003cdefect>/` fallback only for custom flat uploads.\n- **Day 1 real-alignment no real photo** — confirm the canonical `pcb-assets` artifact is uploaded under `\u003cdig_url_root>/datasets/pcb/assets/`; the per-board photo at `input_real_image/\u003cboard>.jpg` must exist (default board is `0603_H100`).\n- **Day 1 real-alignment registration low MI** — retune the per-board cookbook at `assets/cookbooks/pcb/\u003cboard>/usd2roi_nvpcb.yaml` (camera or registration ranges); `assets/cookbooks/pcb/usd2roi_day1.yaml` is only the fallback when no per-board cookbook is selected.\n- **Image-edit endpoint failures** — verify `image_edit_endpoint` from inside the cluster and inspect `references/nim/README.md`.\n- **Jinja comment-token collision** — avoid bash `${#ARRAY[@]}` in workflow inline scripts; Jinja treats `{#` as a comment start.\n- **Group/task name collisions** — OSMO requires group names and task names to be globally unique.\n- **dshm OOM** — confirm the active OSMO pod template has sufficient `/dev/shm`.\n- **multi-GPU FT cgroup OOM** — finetune-from-scratch or multi-GPU inference dies with `OOMKilled` / `Memory cgroup out of memory` shortly after torchrun spawns ranks. Each cosmos-predict2-2B rank loads the full 2B + T5 + NVDINOV2 + SAM2 + Qwen3-VL stack into host RAM (~33 GiB steady-state during DDP sync), so a hardcoded `train_memory: 64Gi` only fits 1 rank. The workflow YAMLs scale memory + CPU with `train_gpu` / `infer_gpu` via `{{ [64, gpu|int * 48]|max }}Gi` — confirm your submit isn't overriding `train_memory` / `infer_memory` back to a fixed value. Pass `--set train_gpu=N infer_gpu=N` together to scale both, or set them individually for asymmetric sizing.\n- **Workflow never progresses / no task logs appear** — if the active pod template mounts `/usr/share/nvidia/nvoptix.bin` but no cluster node actually has that host file, pods can stay stuck in a bad pending state instead of failing clearly. Use `skills/physical-ai-infrastructure-setup-and-resilient-scaling/SKILL.md` to inspect `osmo workflow events \u003cworkflow_id>` and determine whether the workflow is blocked on the `nvoptix.bin` hostPath or another scheduling/mount event.\n\n## IsaacSim Render (structural_defect_generation.yaml)\n\nThe structural-defect IsaacSim workflow invokes `sdg_pipeline.py` +\n`crop_components.py` inside the canonical `paidf-simulation` image (tag pinned\nin `references/container-images.md`). Issues lifted from the upstream\n`generate` skill plus OSMO-specific additions:\n\n- **`Config boundary violation: keys [...] appear in both --config and --pcba-config`** — `sdg_pipeline.py` enforces a strict split: USD/scene-bound fields live in `pcba_target.yaml`; pipeline/render/lighting/defect fields live in the render cookbook (`defect_image.yaml`). If the user edits a cookbook and copies a field across the line, this fails fast. Move each listed key to exactly one file.\n- **`Unknown component_types keyword: 'X'`** — pcba_target.yaml's `component_types` must be `ALL`, `0`, an inline list, or a key in `configs/components.yaml` `subsets:`. The per-board cookbook copies under `assets/cookbooks/pcb/\u003cboard>/` ship explicit lists.\n- **Pipeline runs but `trigger_0000/` is empty** — output landed elsewhere. Both workflows sed-patch `output:` to the OSMO task output dir; check `\u003coutput>/render_config.yaml` snapshot to verify the patched value, then look for `[Pipeline] Output: \u003cpath>` in the render log.\n- **Only `rgb_0000..rgb_0003.png`, last frame is `rgb_0004.png` and 0 bytes** — semantic-segmentation segfault workaround interaction; the writer reports N frames but the last gets truncated on close. Pad `render_patches` by 1 (`--set render_patches=6` to get 5 usable frames) or set `writer.semantic_segmentation: false` in the render cookbook.\n- **`\"Loaded pcba target\"` prints but `component_types` is still a literal string** — keyword resolver did not run. Check `pcba_target.yaml` snapshot at `\u003coutput>/pcba_target.yaml`; ensure `configs/components.yaml` is reachable inside the container at `/workspace/paidf-simulation/configs/components.yaml`.\n- **Asset references break — `spark_lighting.usd not found`** — the scene USD (e.g. `spark_lighting.usd`) references peer USDs by relative path. The full USD tree must be present under `\u003cdig_url_root>/datasets/pcb/assets`; uploading just the top-level scene file leaves the references dangling. Re-publish via `setup/setup_pcb.yaml` (which ships the full asset bundle).\n- **`crop_components.py: no trigger_NNNN found`** — the render step succeeded too quietly (`good_image/` exists but has no `trigger_*` dirs). Re-inspect the render-task log around the Kit `--exec` line; the failure is usually upstream of the crop task. Confirm `nvoptix.bin` is mounted via `osmo workflow events \u003cid>`.\n- **`Permission denied` writing to `/osmo/data/output/...`** — OSMO sets the right ownership on `/osmo/data/output` at task start; this is rare. Do NOT add a `chmod 777 $OUT` in the workflow — that path can't be chmod-ed by the task (OSMO-managed mount). If perms are genuinely wrong, the issue is in the pod template or the OSMO event log, not something the workflow YAML can patch.\n- **`Unknown defect_modes: [...]`** (structural-defect only) — the patcher whitelists `shift`, `tombstone`, `sideflip`. Any other value fails fast. If extending the cookbook with a new mode, update the patcher's `ALL_MODES` set in `structural_defect_generation.yaml`.\n- **Editing `max_image_count:` in the cookbook YAML doesn't change the frame cap** — the workflow sed-patches `max_image_count:` in the cookbook from the `MAX_IMAGE_COUNT` env var (derived from the `render_patches` workflow knob) at task start. To change the cap, pass `--set render_patches=N` at submit time. The cookbook value is purely a default that gets overwritten.\n- **`MAX_IMAGE_COUNT=1` produces 0 crops and fails** (structural-defect only) — the scan_grid is 100 cells; `MAX_IMAGE_COUNT=1` picks a single cell at random. If that cell happens to view empty board area for the active `defect_modes`, `crop_components.py` emits 0 outputs and the task exits non-zero (`No Files in Output Folder` from the OSMO upload step). The render itself succeeded — only the crop step's empty-output guard tripped. Use `MAX_IMAGE_COUNT >= 2` (recommended `>= 3`) so at least one frame lands on a populated region.\n\n## usd2roi-render (good_image_generation.yaml / texture_defect_generation_day0.yaml)\n\nBoth workflows invoke the same `usd2roi-render` task: Kit + `sdg_pipeline.py`\n(scan_grid render with mesh-level semantics) followed by `usd2roi_crop.py`\n(semantic-mask-driven multi-cell ROI extraction). Issues:\n\n- **`usd2roi_crop.py: 0 ROI pairs emitted`** — the render produced trigger frames but the crop step found no semantic regions matching the cookbook's `crop.classes` whitelist. Confirm the `semantics:` block in `day0_image.yaml` matches mesh paths in your USD, and the `crop.classes` entries match those semantic class names.\n\n## nvcr.io image pull failures\n\nThe `paidf-*` workflow images are public on `nvcr.io/nvidia/` and pull\nanonymously, so no registry credential is configured by default. If a task fails\nto pull its image — e.g. an authorization error or `nvcr.io` rate-limiting on a\nbusy cluster — add an NGC registry pull credential and reference it on the\naffected task(s):\n\n1. Create an OSMO REGISTRY credential from your NGC API key:\n ```bash\n osmo credential set nvcr_io --type REGISTRY \\\n --payload registry=nvcr.io username='$oauthtoken' auth=\"$NGC_API_KEY\"\n ```\n2. Add the credential to the `credentials:` block of each task that hit the pull\n error (keep the existing `hf-token` entry where present):\n ```yaml\n credentials:\n nvcr_io:\n NGC_CLI_API_KEY: auth # authorizes the image pull\n hf-token:\n HF_TOKEN: token\n ```\n3. Re-submit. If `osmo workflow validate`/`submit` reports `\u003cfield> is not a\n valid credential key`, the REGISTRY credential's field is `auth` — re-set it\n with the command in step 1.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":16149,"content_sha256":"a5bdae52ce8b1f25f963da45e4396372cf2ee8f94a1637eae77e469f2f10b26d"},{"filename":"scripts/pick_best_step.sh","content":"#!/usr/bin/env bash\n# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n# SPDX-License-Identifier: Apache-2.0\n\n# Derive the best inference step from an anomalygen checkpoint. Three cases:\n#\n# 1. Trained + validated → valid/\u003cSTEP>/valid_kpi.csv tree exists →\n# pick argmax avg(nn_score) per anomalygen contract\n# (skills/anomalygen/references/finetune.md\n# §\"Best checkpoint selection\").\n# 2. Trained, NOT validated → trainer output (*/checkpoints/model/iter_*.pt)\n# exists but no valid_kpi.csv tree (e.g. cookbook\n# misconfig: validation_iter > max_iter) →\n# pick the latest saved iter from trainer output.\n# 3. Shipped checkpoint → flat iter_*.pt next to ag_config.yaml, no trainer\n# layout, no valid/ tree → echo \u003cfallback_step>.\n#\n# Usage: pick_best_step.sh \u003cckpt_root> \u003cfallback_step>\n#\n# CSV schema (case 1, written by ag_train):\n# kpi,\u003cdefect_type_1>,...,Average\n# cradio_v3_base_fid,...\n# mnn_score,...\n# nn_score,\u003cper-defect>,...,\u003cavg> ← this row, last column\n\nset -euo pipefail\n\nCKPT_ROOT=\"${1:?usage: pick_best_step.sh \u003cckpt_root> \u003cfallback_step>}\"\nFALLBACK=\"${2:?usage: pick_best_step.sh \u003cckpt_root> \u003cfallback_step>}\"\n\n# Case 1: trained + validated. Filter out validated steps that have no matching\n# iter_*.pt — happens when cookbook validation_iter % save_iter != 0 (e.g.\n# val=1500, save=2000): valid_kpi.csv at 1500 has no saved checkpoint, so\n# returning that step would crash inference at load time.\nVALID_DIR=$(find \"$CKPT_ROOT\" -type d -name valid -maxdepth 8 2>/dev/null | head -1)\nif [ -n \"$VALID_DIR\" ] && ls \"$VALID_DIR\"/*/valid_kpi.csv >/dev/null 2>&1; then\n MODEL_DIR=$(find \"$CKPT_ROOT\" -type d -path \"*/checkpoints/model\" -print -quit 2>/dev/null)\n BEST=$(\n for csv in \"$VALID_DIR\"/*/valid_kpi.csv; do\n step=$(basename \"$(dirname \"$csv\")\")\n [ \"$step\" = \"0\" ] && continue # iter 0 = pre-training baseline, no checkpoint\n if [ -n \"$MODEL_DIR\" ]; then\n padded=$(printf \"iter_%09d.pt\" \"$step\")\n [ -f \"$MODEL_DIR/$padded\" ] || continue\n fi\n avg=$(awk -F',' '$1==\"nn_score\"{print $NF}' \"$csv\")\n [ -n \"$avg\" ] && echo \"$avg $step\"\n done | sort -gr | head -1 | awk '{print $2}'\n )\n if [ -n \"$BEST\" ]; then\n echo \"[pick_best_step] best=$BEST (peak avg nn_score among validated steps with saved iter_*.pt in $VALID_DIR)\" >&2\n echo \"$BEST\"\n exit 0\n fi\n echo \"[pick_best_step] WARN: no validated step has both an nn_score and a matching saved iter_*.pt under $VALID_DIR (check cookbook validation_iter % save_iter == 0) — trying trainer-output fallback\" >&2\nfi\n\n# Case 2: trainer output exists but no validation logs.\nLATEST_TRAINED=$(find \"$CKPT_ROOT\" -path \"*/checkpoints/model/iter_*.pt\" -printf '%f\\n' 2>/dev/null \\\n | sed 's/iter_0*//; s/\\.pt//' \\\n | sort -gr | head -1)\nif [ -n \"$LATEST_TRAINED\" ]; then\n echo \"[pick_best_step] no valid_kpi.csv tree but trainer output present — using latest trained iter=$LATEST_TRAINED (no per-step KPIs; check cookbook validation_iter vs max_iter)\" >&2\n echo \"$LATEST_TRAINED\"\n exit 0\nfi\n\n# Case 3: shipped checkpoint layout (flat iter_*.pt, no trainer-nested layout, no valid/).\necho \"$FALLBACK\"\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":3454,"content_sha256":"848c6b7431c630ee85d78be4c0a3e1cc7229ac9a11916dde90e179342426583d"},{"filename":"scripts/preflight_credentials.sh","content":"#!/usr/bin/env bash\n# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n# SPDX-License-Identifier: Apache-2.0\n\n# Preflight check for the credential every flow needs.\n#\n# The only credential the workflows require is the OSMO credential hf-token\n# (GENERIC) — that is what gates the Hugging Face downloads every flow performs:\n# - hf-token is used by every group that hits Hugging Face — i.e. all model\n# downloads, all dataset downloads except UC2 (metal, public GitHub), the\n# PCBA USD asset bundle, and the pretrained bundle.\n#\n# There is NO registry credential requirement: the paidf-* workflow images are\n# public on nvcr.io/nvidia/ and pull anonymously, so no NGC key / nvcr_io\n# REGISTRY credential is needed. If image pulls fail (auth error or nvcr.io\n# rate-limiting), see references/troubleshooting.md -> \"nvcr.io image pull\n# failures\" for how to add an NGC pull credential.\n#\n# Env var HF_TOKEN is only needed when:\n# (a) the OSMO credential hf-token is missing and we want to auto-set it, OR\n# (b) you want the outbound HF probes to run (verify the token still has\n# read scope on the gated nvidia/Cosmos-AnomalyGen-* and\n# nvidia/Cosmos-Predict2-* repos before submitting a long workflow).\n#\n# Flow:\n# 1. List OSMO credentials → determine whether hf-token is present.\n# 2. Require hf-token (auto-set from HF_TOKEN if missing).\n# 3. If HF_TOKEN is exported, probe two gated HF repos to verify scope\n# (unless --no-probe). If hf-token is already set and HF_TOKEN is not\n# exported, that's a clean exit 0.\n#\n# Usage:\n# preflight_credentials.sh [--no-probe]\n#\n# --no-probe skips outbound HTTPS probes (offline / restricted egress).\n#\n# Exit 0 when hf-token is present (whether already set or auto-set in step 2)\n# and any probes that ran returned 200. Exit 1 with a specific remediation on\n# stderr otherwise.\n\nset -euo pipefail\n\nprobe=true\ncase \"${1:-}\" in\n --no-probe) probe=false ;;\n \"\") ;;\n *) echo \"usage: $0 [--no-probe]\" >&2; exit 2 ;;\nesac\n\n# Gated HF repos used by the setup + flow workflows. Probed when HF_TOKEN is\n# exported to catch license-acceptance / scope drift before a workflow submit.\nhf_anomalygen_probe_url=\"https://huggingface.co/api/models/nvidia/Cosmos-AnomalyGen-PCB-2B\"\nhf_predict2_probe_url=\"https://huggingface.co/api/models/nvidia/Cosmos-Predict2-2B-Text2Image\"\n\nhave_hf_env=false\n[[ -n \"${HF_TOKEN:-}\" ]] && have_hf_env=true\n\n# 1. Is hf-token already provisioned in the cluster?\npresent=$(osmo credential list | awk 'NR>1 {print $1}' | sort -u)\nhas_hf_token=false\ngrep -qx 'hf-token' \u003c\u003c\u003c\"$present\" && has_hf_token=true\n\n# 2. hf-token is the only requirement. Missing-AND-no-env-var is the only\n# hard failure here.\nif ! $has_hf_token && ! $have_hf_env; then\n echo \"OSMO credential 'hf-token' is missing and HF_TOKEN is not exported to set it.\" >&2\n echo \"\" >&2\n echo \"Two options:\" >&2\n echo \" (a) Export the token and re-run: export HF_TOKEN=\u003cyour-hf-token>\" >&2\n echo \" (b) Provision it directly (see references/setup.md §Credential check).\" >&2\n exit 1\nfi\n\nif ! $has_hf_token; then\n echo \">>> setting OSMO credential hf-token from HF_TOKEN\" >&2\n osmo credential set hf-token --type GENERIC \\\n --payload token=\"$HF_TOKEN\"\nfi\n\n# Confirm hf-token is present (covers the case where `set` succeeded but with\n# the wrong type or payload — re-listing is the only signal we have).\npresent_after=$(osmo credential list | awk 'NR>1 {print $1}' | sort -u)\nif ! grep -qx 'hf-token' \u003c\u003c\u003c\"$present_after\"; then\n echo \"OSMO credential still missing after auto-set:\" >&2\n echo \" - hf-token\" >&2\n echo \"Inspect with: osmo credential list\" >&2\n exit 1\nfi\n\n# 3. Outbound probes — only when HF_TOKEN is exported. If the OSMO credential is\n# provisioned but no env var is available locally, we have no key to probe\n# with; that's not a failure.\nif $probe; then\n if $have_hf_env; then\n for url in \"$hf_anomalygen_probe_url\" \"$hf_predict2_probe_url\"; do\n hf_status=$(curl -sS -o /dev/null -w '%{http_code}' \\\n -H \"Authorization: Bearer $HF_TOKEN\" \\\n \"$url\")\n if [[ \"$hf_status\" != \"200\" ]]; then\n echo \"HF gated-repo probe failed (HTTP $hf_status) at $url\" >&2\n if [[ \"$hf_status\" == \"401\" || \"$hf_status\" == \"403\" ]]; then\n echo \" HF_TOKEN cannot read this gated repo. Accept the license once for each repo the setup workflow touches:\" >&2\n echo \" https://huggingface.co/nvidia/Cosmos-AnomalyGen-PCB-2B\" >&2\n echo \" https://huggingface.co/nvidia/Cosmos-AnomalyGen-Metal-2B\" >&2\n echo \" https://huggingface.co/nvidia/Cosmos-AnomalyGen-Glass-2B\" >&2\n echo \" https://huggingface.co/nvidia/Cosmos-Predict2-2B-Text2Image\" >&2\n echo \" https://huggingface.co/nvidia/Cosmos-Predict2-14B-Text2Image\" >&2\n echo \" After accepting, regenerate the token at https://huggingface.co/settings/tokens if it predates the license acceptance.\" >&2\n elif [[ \"$hf_status\" == \"000\" ]]; then\n echo \" Network error reaching huggingface.co. If egress is restricted, re-run with --no-probe.\" >&2\n fi\n exit 1\n fi\n done\n else\n echo \"note: skipping HF probe — HF_TOKEN not exported (OSMO credential 'hf-token' is already provisioned).\" >&2\n fi\nfi\n\necho \"OK: OSMO credential hf-token present (paidf-* images are public on nvcr.io/nvidia/ — no registry credential needed).\"\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":5463,"content_sha256":"fad68901228710d76191dd9e23ba1c5bf8853989d4aa9d3b6d3039177929c383"},{"filename":"scripts/preflight_pod_template.sh","content":"#!/usr/bin/env bash\n# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n# SPDX-License-Identifier: Apache-2.0\n\n# Preflight check for the OSMO POD_TEMPLATE: every DIG workflow assumes\n# the nvoptix denoiser binary is hostPath-mounted and /dev/shm is >= 16 GiB.\n# Without those, Kit OptiX silently degrades to noisy raw path tracing and\n# torchrun / IsaacSim ray-tracer buffers OOM on dshm.\n#\n# This script codifies the jq one-liner that previously lived in prose at\n# references/preconditions.md §2 so callers (agent or human) get a\n# deterministic exit code instead of having to parse the one-liner output.\n#\n# Usage:\n# preflight_pod_template.sh [--min-dshm-gib N]\n#\n# --min-dshm-gib defaults to 16. Pass 32 if your workflow needs the\n# preferred size (Kit ray-tracer buffers + torchrun shared-memory).\n#\n# Exit codes:\n# 0 template OK -- nvoptix mount present, dshm >= min\n# 1 template visible but malformed (missing mount or undersized dshm)\n# 2 HTTP 403 -- user lacks read permission on POD_TEMPLATE\n# 3 HTTP 409 -- ConfigMap-mode deployment, CLI disabled\n# 4 prerequisite missing (osmo, jq) or unexpected failure\n#\n# Callers (agent / preconditions.md §2) interpret the exit code:\n# 0 -> save \"pod template verified\" to memory; proceed\n# 1 -> route to physical-ai-infrastructure-setup-and-resilient-scaling\n# for the patch runbook (admin-or-equivalent assumed)\n# 2 -> ask user (AskUserQuestion) whether admin already configured it;\n# on \"yes\" save \"user-confirmed\", on \"no/unsure\" stop\n# 3 -> warn, save \"skipped-409\", proceed; runtime in-pod preflight\n# is the safety net\n# 4 -> fix the environment and re-run\n\nset -euo pipefail\n\nMIN_DSHM_GIB=16\nwhile [[ $# -gt 0 ]]; do\n case \"$1\" in\n --min-dshm-gib) MIN_DSHM_GIB=\"$2\"; shift 2 ;;\n -h|--help) sed -n '3,36p' \"$0\"; exit 0 ;;\n *) echo \"unknown arg: $1\" >&2; exit 4 ;;\n esac\ndone\n\ncommand -v osmo >/dev/null || { echo \"ERROR: osmo CLI not on PATH.\" >&2; exit 4; }\ncommand -v jq >/dev/null || { echo \"ERROR: jq not installed.\" >&2; exit 4; }\n\ntmp_stdout=$(mktemp); tmp_stderr=$(mktemp)\ntrap 'rm -f \"$tmp_stdout\" \"$tmp_stderr\"' EXIT\nif ! osmo config show POD_TEMPLATE >\"$tmp_stdout\" 2>\"$tmp_stderr\"; then\n if grep -qE '(^|[^0-9])403([^0-9]|$)' \"$tmp_stderr\"; then\n echo \"POD_TEMPLATE: HTTP 403 -- your account lacks read permission.\" >&2\n echo \" Either ask your OSMO admin to confirm the template meets DIG\" >&2\n echo \" requirements, or request read access via 'osmo profile list'.\" >&2\n exit 2\n elif grep -qE '(^|[^0-9])409([^0-9]|$)' \"$tmp_stderr\"; then\n echo \"POD_TEMPLATE: HTTP 409 -- config CLI disabled (ConfigMap mode).\" >&2\n echo \" Runtime in-pod preflight is the only remaining safety net.\" >&2\n exit 3\n else\n echo \"ERROR: 'osmo config show POD_TEMPLATE' failed unexpectedly:\" >&2\n cat \"$tmp_stderr\" >&2\n exit 4\n fi\nfi\n\nnvoptix_path=$(jq -r '.default_user.spec.volumes[]? | select(.name==\"nvoptix\") | .hostPath.path // empty' \"$tmp_stdout\")\ndshm_size=$(jq -r '.default_user.spec.volumes[]? | select(.name==\"dshm\") | .emptyDir.sizeLimit // empty' \"$tmp_stdout\")\n\nbad=0\nif [[ \"$nvoptix_path\" != \"/usr/share/nvidia/nvoptix.bin\" ]]; then\n echo \"POD_TEMPLATE: nvoptix hostPath mount missing or wrong path.\" >&2\n echo \" Got: '${nvoptix_path:-\u003cnone>}' Want: /usr/share/nvidia/nvoptix.bin\" >&2\n bad=1\nfi\nif [[ -z \"$dshm_size\" ]]; then\n echo \"POD_TEMPLATE: dshm emptyDir volume missing.\" >&2\n bad=1\nelse\n dshm_gib=${dshm_size%Gi}\n if ! [[ \"$dshm_gib\" =~ ^[0-9]+$ ]] || (( dshm_gib \u003c MIN_DSHM_GIB )); then\n echo \"POD_TEMPLATE: dshm sizeLimit is '${dshm_size}', need >= ${MIN_DSHM_GIB}Gi.\" >&2\n bad=1\n fi\nfi\n\nif (( bad == 1 )); then\n echo >&2\n echo \"Patch via the physical-ai-infrastructure-setup-and-resilient-scaling\" >&2\n echo \"skill ('osmo config update POD_TEMPLATE').\" >&2\n exit 1\nfi\n\necho \"POD_TEMPLATE OK: nvoptix mount + /dev/shm >= ${MIN_DSHM_GIB}Gi.\"\nexit 0\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":3989,"content_sha256":"8363900a31c3357c6b3f3f757cb7f0fcbd57c0f809bc0012c6f1e7b9824967ec"},{"filename":"scripts/preflight_urls.sh","content":"#!/usr/bin/env bash\n# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n# SPDX-License-Identifier: Apache-2.0\n\n# Preflight check for the DIG URL artifacts each flow needs.\n# Usage: preflight_urls.sh \u003cflow> \u003cusecase> [variant]\n# flow: 0 | 1 | finetune\n# usecase: pcb | metal_surface | glass | \u003ccustom>\n# variant: real-alignment (optional; Day 1 PCBA real-photo alignment adds\n# datasets/pcb/assets).\n#\n# Environment:\n# DIG_URL_ROOT defaults to s3://osmo-workflows/dig\n# USE_PRETRAINED_CHECKPOINT defaults to true; set false to skip model/\u003cusecase>\n# USE_USD2ROI_DAY1 defaults to false; true is equivalent to variant=real-alignment\n\nset -euo pipefail\n\nif [[ $# -lt 2 || $# -gt 3 ]]; then\n echo \"usage: $0 \u003cflow:0|1|finetune> \u003cusecase:pcb|metal_surface|glass|...> [variant:real-alignment]\" >&2\n exit 2\nfi\n\ncase \"${1,,}\" in\n 0|day0) flow=0 ;;\n 1|day1) flow=1 ;;\n finetune) flow=finetune ;;\n *) flow=$1 ;;\nesac\nusecase=$2\nvariant=${3:-}\ndig_root=${DIG_URL_ROOT:-s3://osmo-workflows/dig}\ndig_root=${dig_root%/}\nuse_pretrained=${USE_PRETRAINED_CHECKPOINT:-true}\nuse_usd2roi_day1=${USE_USD2ROI_DAY1:-false}\n\nis_truthy() {\n case \"${1,,}\" in\n true|1|yes|y) return 0 ;;\n *) return 1 ;;\n esac\n}\n\nrequired=()\ncase \"$flow:$usecase\" in\n 0:pcb)\n required+=(\n \"$dig_root/models/pretrained\"\n \"$dig_root/datasets/pcb/raw\"\n \"$dig_root/datasets/pcb/assets\"\n )\n if is_truthy \"$use_pretrained\"; then\n required+=(\"$dig_root/models/pcb\")\n fi\n ;;\n 1:*)\n required+=(\n \"$dig_root/models/pretrained\"\n \"$dig_root/datasets/$usecase/raw\"\n )\n if is_truthy \"$use_pretrained\"; then\n required+=(\"$dig_root/models/$usecase\")\n fi\n if [[ \"${variant,,}\" == \"c\" || \"${variant,,}\" == \"real-alignment\" ]] || is_truthy \"$use_usd2roi_day1\"; then\n required+=(\n \"$dig_root/datasets/pcb/assets\"\n )\n fi\n ;;\n finetune:*)\n required+=(\n \"$dig_root/models/pretrained\"\n \"$dig_root/datasets/$usecase/raw\"\n )\n ;;\n *)\n echo \"unsupported flow:usecase combination: $flow:$usecase\" >&2\n exit 2\n ;;\nesac\n\nstdout=$(mktemp)\nstderr=$(mktemp)\ntrap 'rm -f \"$stdout\" \"$stderr\"' EXIT\n\nmissing=()\nfor url in \"${required[@]}\"; do\n : > \"$stdout\"\n : > \"$stderr\"\n if osmo data list --no-pager \"$url\" >\"$stdout\" 2>\"$stderr\"; then\n if [[ -s \"$stdout\" ]] && ! grep -Eiq 'no (files|objects|entries)|not found|does not exist|not exist|total 0' \"$stdout\" \"$stderr\"; then\n echo \"OK: $url\"\n continue\n fi\n fi\n missing+=(\"$url\")\ndone\n\nif [[ ${#missing[@]} -gt 0 ]]; then\n echo \"Missing URL artifacts for flow=$flow usecase=$usecase:\" >&2\n printf ' - %s\\n' \"${missing[@]}\" >&2\n echo \"Submit the relevant setup/setup_\u003ccase>.yaml + setup/setup_pretrained.yaml, or upload artifacts under DIG_URL_ROOT; see references/setup.md.\" >&2\n exit 1\nfi\n\necho \"OK: all ${#required[@]} required URL artifacts are present for flow=$flow usecase=$usecase.\"\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":3017,"content_sha256":"4be073888867845cb7f7fb63c6e8547556c3d1e3e9c92b5256b2d5190f9b5c6a"},{"filename":"scripts/render_defect_spec.py","content":"#!/usr/bin/env python3\n# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n# SPDX-License-Identifier: Apache-2.0\n\n\"\"\"Render a defect_spec.jsonl for anomalygen AMP routing.\n\nFallback used by Day 0 and Day 1 anomaly-infer when the prepared inference URL\ndoes not ship a hand-authored `defect_spec.jsonl` at its root.\nTwo input modes: `--pairs '[[\"material\",\"defect\"],...]'` for multi-material\ntaxonomies, or `--material X --defects '[\"d1\",\"d2\"]'` for single-material.\nThe schema matches `skills/anomalygen/assets/defect_spec_template.jsonl`:\n\n {\"defect_type\": \"\u003cMATERIAL>+\u003cDEFECT>\", \"spatial_dependency\": \"\u003cmode>\",\n \"roi_prompt_defect_location\": \"\u003cprompt-or-empty>\"}\n\n`spatial_dependency` is one of `free` / `text` / `cad`; `roi_prompt_defect_location`\nis required when mode=`text` (Qwen+SAM2) and unused otherwise. Per-defect mode\nmixing is not supported by this fallback — ship a custom defect_spec.jsonl in\nthe prepared URL artifact for that.\n\"\"\"\nimport argparse\nimport json\nimport sys\n\n\ndef main() -> int:\n p = argparse.ArgumentParser(description=__doc__.split(\"\\n\", 1)[0])\n p.add_argument(\"--output\", required=True, help=\"Path to write defect_spec.jsonl\")\n # Two input shapes — exactly one is required:\n # --pairs '[[\"IC\",\"bridge\"],[\"passive_component\",\"missing\"]]' (multi-material)\n # --material IC --defects '[\"bridge\",\"excess_solder\"]' (single material)\n p.add_argument(\"--pairs\", default=\"\",\n help='JSON list of [material, defect] pairs, '\n 'e.g. \\'[[\"IC\",\"bridge\"],[\"passive_component\",\"missing\"]]\\'')\n p.add_argument(\"--material\", default=\"\", help=\"Anomaly material prefix (single-material mode)\")\n p.add_argument(\"--defects\", default=\"\",\n help='JSON list of defect names (single-material mode)')\n p.add_argument(\"--spatial-dependency\", default=\"free\", choices=[\"free\", \"text\", \"cad\"],\n help=\"AMP routing branch (default: free = whole-image ROI)\")\n p.add_argument(\"--roi-prompt\", default=\"\",\n help=\"Free-text ROI prompt (required when --spatial-dependency=text)\")\n args = p.parse_args()\n\n if args.spatial_dependency == \"text\" and not args.roi_prompt:\n print(\"ERROR: --roi-prompt is required when --spatial-dependency=text\", file=sys.stderr)\n return 2\n\n pairs: list[tuple[str, str]] = []\n if args.pairs:\n try:\n parsed = json.loads(args.pairs)\n except json.JSONDecodeError as e:\n print(f\"ERROR: --pairs is not valid JSON: {e}\", file=sys.stderr)\n return 2\n for i, item in enumerate(parsed):\n if not (isinstance(item, list) and len(item) == 2\n and all(isinstance(x, str) for x in item)):\n print(f\"ERROR: --pairs entry {i} must be [material, defect] strings\",\n file=sys.stderr)\n return 2\n pairs.append((item[0], item[1]))\n elif args.material and args.defects:\n try:\n defects = json.loads(args.defects)\n except json.JSONDecodeError as e:\n print(f\"ERROR: --defects is not valid JSON: {e}\", file=sys.stderr)\n return 2\n if not isinstance(defects, list) or not all(isinstance(d, str) for d in defects):\n print(\"ERROR: --defects must be a JSON array of strings\", file=sys.stderr)\n return 2\n pairs = [(args.material, d) for d in defects]\n else:\n print(\"ERROR: pass either --pairs or (--material + --defects)\", file=sys.stderr)\n return 2\n\n with open(args.output, \"w\") as fp:\n for material, defect in pairs:\n fp.write(json.dumps({\n \"defect_type\": f\"{material}+{defect}\",\n \"spatial_dependency\": args.spatial_dependency,\n \"roi_prompt_defect_location\": args.roi_prompt,\n }) + \"\\n\")\n print(f\"wrote {len(pairs)} entries to {args.output} \"\n f\"(spatial_dependency={args.spatial_dependency})\", file=sys.stderr)\n return 0\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n","content_type":"text/x-python; charset=utf-8","language":"python","size":4123,"content_sha256":"bf3f9054debeaed927c50bfac4f4a6d97a3277033c1768efbd711020a10009d8"},{"filename":"skill-card.md","content":"## Description: \u003cbr>\nOrchestrates end-to-end defect image generation, augmentation, and labeling pipelines for Automated Optical Inspection (AOI) datasets using NVIDIA Cosmos AnomalyGen on OSMO for PCBA, metal surface, and glass inspection. \u003cbr>\n\nThis skill is ready for commercial/non-commercial use. \u003cbr>\n\n## Owner\nNVIDIA \u003cbr>\n\n### License/Terms of Use: \u003cbr>\nCC-BY-4.0 AND Apache-2.0 \u003cbr>\n## Use Case: \u003cbr>\nDevelopers and engineers use this skill to orchestrate synthetic defect image generation pipelines for Automated Optical Inspection (AOI) training datasets, covering cold-start (Day 0) and real-image inference (Day 1) workflows across PCBA, metal surface, and glass inspection use cases. \u003cbr>\n\n### Deployment Geography for Use: \u003cbr>\nGlobal \u003cbr>\n\n## Known Risks and Mitigations: \u003cbr>\nRisk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. \u003cbr>\nMitigation: Review and scan skill before deployment. \u003cbr>\n\n## Reference(s): \u003cbr>\n- [NVIDIA OSMO](https://developer.nvidia.com/osmo) \u003cbr>\n- [Skill Definition (SKILL.md)](SKILL.md) \u003cbr>\n- [Knob Mapping Reference](references/knob_mapping.md) \u003cbr>\n- [Preconditions](references/preconditions.md) \u003cbr>\n- [Troubleshooting Guide](references/troubleshooting.md) \u003cbr>\n\n\n## Skill Output: \u003cbr>\n**Output Type(s):** [Shell commands, Configuration instructions] \u003cbr>\n**Output Format:** [Markdown with inline bash code blocks] \u003cbr>\n**Output Parameters:** [1D] \u003cbr>\n**Other Properties Related to Output:** [None] \u003cbr>\n\n## Evaluation Agents Used: \u003cbr>\n- Claude Code (`claude-code`) \u003cbr>\n- Codex (`codex`) \u003cbr>\n\n\n\n## Evaluation Tasks: \u003cbr>\nEvaluated against 6 internal evaluation tasks (all positive skill-activation cases) via NVSkills-Eval external profile in an astra-sandbox environment. \u003cbr>\n\n## Evaluation Metrics Used: \u003cbr>\nReported benchmark dimensions: \u003cbr>\n- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. \u003cbr>\n- Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. \u003cbr>\n- Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. \u003cbr>\n- Effectiveness: Checks whether the agent performs measurably better with the skill than without it. \u003cbr>\n- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. \u003cbr>\n\nUnderlying evaluation signals used in this run: \u003cbr>\n- `security`: Checks for unsafe operations, secret leakage, and unauthorized access. \u003cbr>\n- `skill_execution`: Verifies that the agent loaded the expected skill and workflow. \u003cbr>\n- `skill_efficiency`: Checks routing quality, decoy avoidance, and redundant tool usage. \u003cbr>\n- `accuracy`: Grades final-answer correctness against the reference answer. \u003cbr>\n- `goal_accuracy`: Checks whether the overall user task completed successfully. \u003cbr>\n- `behavior_check`: Verifies expected behavior steps, including safety expectations. \u003cbr>\n- `token_efficiency`: Compares token usage with and without the skill. \u003cbr>\n\n\n\n## Evaluation Results: \u003cbr>\n| Dimension | Num | `claude-code` | `codex` |\n|---|---:|---:|---:|\n| Security | 6 | 100% (+0%) | 100% (+0%) |\n| Correctness | 6 | 94% (+71%) | 94% (+52%) |\n| Discoverability | 6 | 95% (+70%) | 79% (+33%) |\n| Effectiveness | 6 | 79% (+77%) | 90% (+76%) |\n| Efficiency | 6 | 78% (+51%) | 69% (+33%) |\n\n## Skill Version(s): \u003cbr>\n1.0.1 (source: frontmatter) \u003cbr>\n\n## Ethical Considerations: \u003cbr>\nNVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. \u003cbr>\n\n(For Release on NVIDIA Platforms Only) \u003cbr>\nPlease report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail). \u003cbr>\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":4104,"content_sha256":"263bdf1a5c9873fedd221f792179e4a49d73026cb5ac9fcdf69d8d35a571dfcc"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"Physical AI Defect Image Generation","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Table of Contents","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Supported Flows","type":"text","marks":[{"type":"link","attrs":{"href":"#supported-flows","title":null}}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Disambiguation","type":"text","marks":[{"type":"link","attrs":{"href":"#disambiguation-handle-vague-requests-before-committing","title":null}}]},{"text":" (full table in ","type":"text"},{"text":"references/disambiguation.md","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Step 0: Select Flow, Cookbook, and Gather Inputs","type":"text","marks":[{"type":"link","attrs":{"href":"#step-0-select-flow-cookbook-and-gather-inputs","title":null}}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Common Preconditions","type":"text","marks":[{"type":"link","attrs":{"href":"#common-preconditions-all-flows","title":null}}]},{"text":" (long-form in ","type":"text"},{"text":"references/preconditions.md","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Flow walkthroughs","type":"text","marks":[{"type":"link","attrs":{"href":"#flow-walkthroughs","title":null}}]},{"text":" (one entry per flow; details in ","type":"text"},{"text":"references/flows/","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"OSMO Monitoring","type":"text","marks":[{"type":"link","attrs":{"href":"#osmo-monitoring","title":null}}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Supporting files","type":"text","marks":[{"type":"link","attrs":{"href":"#supporting-files","title":null}}]}]}]}]},{"type":"paragraph","content":[{"text":"End-to-end orchestration of defect image generation, augmentation, and labeling pipelines for AOI (Automated Optical Inspection) datasets. ","type":"text"},{"text":"AnomalyGen = Cosmos-Predict2-2B finetuned per use case","type":"text","marks":[{"type":"strong"}]},{"text":" (Cosmos-AnomalyGen-PCB-2B, -Metal-2B, -Glass-2B). Every flow has a canonical OSMO workflow YAML in ","type":"text"},{"text":"assets/configs/","type":"text","marks":[{"type":"code_inline"}]},{"text":" that chains all steps non-interactively. Use-case cookbooks in ","type":"text"},{"text":"assets/cookbooks/","type":"text","marks":[{"type":"code_inline"}]},{"text":" provide PCBA usd2roi/image-edit configs and AnomalyGen training configs for PCBA, metal surface, and glass inspection. This skill governs flow selection, data handoffs, and submit commands; component internals live in each component's ","type":"text"},{"text":"SKILL.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Supported Flows","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Flow","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Entry point","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"OSMO YAML","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Steps","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use cases","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 0 — Texture Defects","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"CAD scene USD (","type":"text"},{"text":"pcba_target.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":" ships in the cookbook)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"texture_defect_generation_day0.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"usd2roi (scan_grid + per-cell ROI crops) → image-edit augmentation (","type":"text"},{"text":"nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL","type":"text","marks":[{"type":"code_inline"}]},{"text":") → finetune-or-passthrough → infer (anomalygen labels inline, ","type":"text"},{"text":"including missing-component","type":"text","marks":[{"type":"strong"}]},{"text":")","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"PCBA","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 0 — Good Image","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"(usd2roi + Image-Edit)","type":"text","marks":[{"type":"em"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"CAD scene USD + per-board ","type":"text"},{"text":"pcba_target.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"day0_image.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"day0_crop.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"good_image_generation.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"usd2roi-render (scan_grid + per-cell ROI crop) → Qwen Image-Edit (OVSL2SL appearance transfer)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"PCBA clean-image set (ChangeNet golden halves, finetune positives, real-photo pairing)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 0 — Structural Defects","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"CAD scene USD + per-board ","type":"text"},{"text":"pcba_target.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"structural_defect_generation.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"isaac-render (pose defects: shift / tombstone / sideflip) + per-component crop (single pod) → Qwen Image-Edit (OVSL2SL lighting transfer; pose geometry preserved)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"PCBA pose-defect set; ChangeNet defect halves","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 1 — Infer + Label (real-photo alignment, DEFAULT)","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"CAD-derived USD + real PCBA photo (both ship in ","type":"text"},{"text":"datasets/pcb/assets","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"texture_defect_generation_day1_real_alignment.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"usd2roi day-1 render → MI register → per-ROI crop → yq-render config → finetune-or-passthrough → infer (anomalygen labels inline)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Default PCBA Day 1.","type":"text","marks":[{"type":"strong"}]},{"text":" Raw AOI screenshot of any usd2roi-supported board","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 1 — Infer + Label (manual ROI)","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Pre-captured clean images + ROI masks (NGC artifact or user upload)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"texture_defect_generation_day1_manual_roi.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"yq-render config → finetune-or-passthrough → infer (anomalygen labels inline)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Metal surface, glass (no USD/real-photo flow); PCBA ","type":"text"},{"text":"only when user explicitly asks","type":"text","marks":[{"type":"strong"}]},{"text":" for pre-captured ROI experimentation","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Finetune Only","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Labeled anomaly URL artifact","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"finetune.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"yq-render config → finetune (validate_dataset → prep_testcase → torchrun)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Any use case; produces checkpoint for Day 0 or Day 1. Requires raw training data under ","type":"text"},{"text":"\u003cdig_url_root>/datasets/\u003cusecase>/raw","type":"text","marks":[{"type":"code_inline"}]},{"text":" (see ","type":"text"},{"text":"assets/configs/setup/setup_\u003cusecase>.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":").","type":"text"}]}]}]}]},{"type":"paragraph","content":[{"text":"All flows run on OSMO. Day 0 flows require ","type":"text"},{"text":"image_edit_endpoint","type":"text","marks":[{"type":"code_inline"}]},{"text":" (Qwen Image-Edit OVSL2SL — existing URL or local deploy from ","type":"text"},{"text":"references/nim/","type":"text","marks":[{"type":"code_inline"}]},{"text":"); Finetune Only has no external endpoints.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Pick the right workflow for the user's defect class","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Defect class","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Workflow","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Mechanism","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Clean / good / scan-grid / ","type":"text"},{"text":"normal_img + cad_mask","type":"text","marks":[{"type":"code_inline"}]},{"text":" pairs","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"good_image_generation.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"usd2roi-render + Qwen Image-Edit","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Texture defects (solder bridge, scratch, discoloration) ","type":"text"},{"text":"AND missing-component","type":"text","marks":[{"type":"strong"}]},{"text":" (handled natively by AnomalyGen, NOT structural)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"texture_defect_generation_day0.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Qwen Image-Edit + AnomalyGen AMP/SDG","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Structural / pose defects (tombstone, shift, sideflip)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"structural_defect_generation.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"IsaacSim pose perturbation","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 1 inference + labeling on a real image","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"texture_defect_generation_day1_real_alignment.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":" (PCBA default) or ","type":"text"},{"text":"texture_defect_generation_day1_manual_roi.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":" (metal/glass; PCBA only when user explicitly asks for pre-captured ROI / skip-alignment)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"usd2roi day-1 registration (real-alignment) or direct inference (manual-ROI)","type":"text"}]}]}]}]},{"type":"paragraph","content":[{"text":"ChangeNet golden/defect pairs: submit ","type":"text"},{"text":"good_image_generation.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":" + ","type":"text"},{"text":"structural_defect_generation.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":" with the same ","type":"text"},{"text":"--set name=","type":"text","marks":[{"type":"code_inline"}]},{"text":" (two-submission pairing convention).","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"Day 0 and Day 1 share the same downstream shape","type":"text","marks":[{"type":"strong"}]},{"text":": a Jinja-gated ","type":"text"},{"text":"finetune-job","type":"text","marks":[{"type":"code_inline"}]},{"text":" (omitted when ","type":"text"},{"text":"use_pretrained_checkpoint=true","type":"text","marks":[{"type":"code_inline"}]},{"text":") feeding ","type":"text"},{"text":"anomaly-infer","type":"text","marks":[{"type":"code_inline"}]},{"text":". Day 0 prepends ","type":"text"},{"text":"usd2roi-render","type":"text","marks":[{"type":"code_inline"}]},{"text":" + ","type":"text"},{"text":"augment-image-edit","type":"text","marks":[{"type":"code_inline"}]},{"text":"; Day 1 starts from ","type":"text"},{"text":"\u003cdig_url_root>/datasets/\u003cusecase>/raw","type":"text","marks":[{"type":"code_inline"}]},{"text":". Per-stage detail: each flow's walkthrough.","type":"text"}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"User intent → knob mapping","type":"text"}]},{"type":"paragraph","content":[{"text":"Every OV flow is two-stage","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"crop_max_emit=N","type":"text","marks":[{"type":"code_inline"}]},{"text":" caps the ","type":"text"},{"text":"final","type":"text","marks":[{"type":"em"}]},{"text":" per-cell crops (stage 2); ","type":"text"},{"text":"render_patches=N","type":"text","marks":[{"type":"code_inline"}]},{"text":" caps ","type":"text"},{"text":"raw","type":"text","marks":[{"type":"em"}]},{"text":" scan-grid patches (stage 1, each yielding multiple crops). ","type":"text"},{"text":"DO NOT auto-map \"generate N images\" → ","type":"text","marks":[{"type":"strong"}]},{"text":"render_patches=N","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" (wrong stage). ","type":"text"},{"text":"crop_max_emit","type":"text","marks":[{"type":"code_inline"}]},{"text":" does not exist on ","type":"text"},{"text":"structural_defect_generation.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":" (one crop per component — use ","type":"text"},{"text":"render_patches","type":"text","marks":[{"type":"code_inline"}]},{"text":") or ","type":"text"},{"text":"texture_defect_generation_day1_real_alignment.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":" (narrow via the cookbook's ","type":"text"},{"text":"crop.classes","type":"text","marks":[{"type":"code_inline"}]},{"text":" whitelist). Full knob table, smoke-test recipes, defaults, caveats: ","type":"text"},{"text":"references/knob_mapping.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Structural-defect sizing (no ","type":"text"},{"text":"crop_max_emit","type":"text","marks":[{"type":"code_inline"}]},{"text":" knob exists)","type":"text"}]},{"type":"paragraph","content":[{"text":"Structural output is ","type":"text"},{"text":"non-linear in ","type":"text","marks":[{"type":"strong"}]},{"text":"render_patches","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" — doubling frames adds ~1.6–1.7× crops, not 2×. Don't use ","type":"text"},{"text":"crop_max_emit","type":"text","marks":[{"type":"code_inline"}]},{"text":" (no effect) or ","type":"text"},{"text":"render_patches=0","type":"text","marks":[{"type":"code_inline"}]},{"text":" (fails). Validated yield table + target-size formula: ","type":"text"},{"text":"references/flows/structural_defect_generation.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" §\"Sizing the output\". For ambiguous \"generate N images\", surface the calibration table via ","type":"text"},{"text":"AskUserQuestion","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Disambiguation: handle vague requests before committing","type":"text"}]},{"type":"paragraph","content":[{"text":"Underspecified prompts (\"generate me some images\", \"run the PCBA flow\", \"give me defects\") ","type":"text"},{"text":"must not","type":"text","marks":[{"type":"strong"}]},{"text":" be resolved by silently assuming a flow / usecase / knob mapping. When intent is ambiguous, pause and present candidate interpretations via ","type":"text"},{"text":"AskUserQuestion","type":"text","marks":[{"type":"code_inline"}]},{"text":" (2–4 mutually exclusive options) before submitting. Disambiguate the load-bearing choices: ","type":"text"},{"text":"which flow, which use case, what stage a count refers to, finetune vs. passthrough","type":"text","marks":[{"type":"strong"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"Settled defaults you should NOT disambiguate: PCBA Day 1 → real-alignment; board → ","type":"text"},{"text":"0603_H100","type":"text","marks":[{"type":"code_inline"}]},{"text":"; image-edit endpoint → local cluster service (","type":"text"},{"text":"references/nim/","type":"text","marks":[{"type":"code_inline"}]},{"text":"); ","type":"text"},{"text":"use_pretrained_checkpoint=true","type":"text","marks":[{"type":"code_inline"}]},{"text":"; Day 1 real-alignment ","type":"text"},{"text":"default_spatial_dependency=cad","type":"text","marks":[{"type":"code_inline"}]},{"text":" (fall back to ","type":"text"},{"text":"free","type":"text","marks":[{"type":"code_inline"}]},{"text":" only when CAD masks are unavailable, see ","type":"text"},{"text":"references/flows/texture_defect_generation_day1_real_alignment.md","type":"text","marks":[{"type":"code_inline"}]},{"text":").","type":"text"}]},{"type":"paragraph","content":[{"text":"dig_url_root","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" is the one exception — NO silent default.","type":"text","marks":[{"type":"strong"}]},{"text":" First-time (no memory entry), MUST elicit via ","type":"text"},{"text":"AskUserQuestion","type":"text","marks":[{"type":"code_inline"}]},{"text":" before any submit / ","type":"text"},{"text":"osmo data upload","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"preflight_urls.sh","type":"text","marks":[{"type":"code_inline"}]},{"text":". ","type":"text"},{"text":"s3://osmo-workflows/dig","type":"text","marks":[{"type":"code_inline"}]},{"text":" is a ","type":"text"},{"text":"suggestion to confirm","type":"text","marks":[{"type":"em"}]},{"text":", never auto-picked (~80 GB+ lands there). Later runs may reuse the remembered value silently. See Step 0 + memory rules (§4).","type":"text"}]},{"type":"paragraph","content":[{"text":"Full trigger table, prompt construction, and when-NOT-to-ask exceptions: ","type":"text","marks":[{"type":"strong"}]},{"text":"references/disambiguation.md","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" — load before assembling ","type":"text"},{"text":"AskUserQuestion","type":"text","marks":[{"type":"code_inline"}]},{"text":" options for any vague request.","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Step 0: Select Flow, Cookbook, and Gather Inputs","type":"text"}]},{"type":"paragraph","content":[{"text":"Before this step","type":"text","marks":[{"type":"strong"}]},{"text":", if the request is vague (e.g. \"generate me images\", \"run the PCBA flow\", \"give me defects\"), pause and run the disambiguation cheat sheet above — present candidate interpretations via ","type":"text"},{"text":"AskUserQuestion","type":"text","marks":[{"type":"code_inline"}]},{"text":" and let the user pick. Don't auto-pick a load-bearing default the user didn't actually choose.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"First-time gate","type":"text"}]},{"type":"paragraph","content":[{"text":"If memory has no entries for this user, ASK the up-front preference questions in ONE ","type":"text"},{"text":"AskUserQuestion","type":"text","marks":[{"type":"code_inline"}]},{"text":" call BEFORE any preflight / ","type":"text"},{"text":"osmo","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"kubectl","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"osmo data upload","type":"text","marks":[{"type":"code_inline"}]},{"text":", save to memory (§4), then proceed. Bundle:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"dig_url_root","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" — MUST be elicited, not auto-picked. Offer ","type":"text"},{"text":"s3://osmo-workflows/dig","type":"text","marks":[{"type":"code_inline"}]},{"text":" as a confirmable suggestion; else user provides their own OSMO-supported storage prefix. ~80 GB+ lands here. No escape hatch other than memory-recall of a previously confirmed value.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Default OSMO ","type":"text","marks":[{"type":"strong"}]},{"text":"--pool","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" — candidates from ","type":"text"},{"text":"osmo profile list","type":"text","marks":[{"type":"code_inline"}]},{"text":" → ","type":"text"},{"text":"pool.accessible","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Pod-template confirmation","type":"text","marks":[{"type":"strong"}]},{"text":" — only when ","type":"text"},{"text":"osmo config show POD_TEMPLATE","type":"text","marks":[{"type":"code_inline"}]},{"text":" returns 403 (§2 has the exact question).","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Image-edit endpoint","type":"text","marks":[{"type":"strong"}]},{"text":" — Day 0 only: Option A (existing URL) vs Option B (deploy local NIM).","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Subsequent conversations read these silently from memory. Per-flow choices (use case, checkpoint vs finetune, board, knobs) are asked each time — see below.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Preflight ordering (after the first-time gate)","type":"text"}]},{"type":"paragraph","content":[{"text":"Run §1 ","type":"text"},{"text":"preflight_credentials.sh","type":"text","marks":[{"type":"code_inline"}]},{"text":" → §2 ","type":"text"},{"text":"preflight_pod_template.sh","type":"text","marks":[{"type":"code_inline"}]},{"text":" → §3 ","type":"text"},{"text":"preflight_urls.sh \u003cflow> \u003cusecase>","type":"text","marks":[{"type":"code_inline"}]},{"text":" → §4 generate the run stamp. ","type":"text"},{"text":"Cadence","type":"text","marks":[{"type":"strong"}]},{"text":": §1 and §2 are once-per-conversation gates with cross-conversation memory caching (see §4a in ","type":"text"},{"text":"references/preconditions.md","type":"text","marks":[{"type":"code_inline"}]},{"text":") — skip when memory records them as already verified / user-confirmed. §3 runs before every submit (varies by flow). §4 is the agent's job — fresh ","type":"text"},{"text":"$STAMP","type":"text","marks":[{"type":"code_inline"}]},{"text":" per submit.","type":"text"}]},{"type":"paragraph","content":[{"text":"Pod-template enforcement is two layers: the pre-submit ","type":"text"},{"text":"preflight_pod_template.sh","type":"text","marks":[{"type":"code_inline"}]},{"text":" gate (§2) plus an in-pod runtime preflight on every OV + training task (fails fast on missing ","type":"text"},{"text":"/usr/share/nvidia/nvoptix.bin","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"/dev/shm","type":"text","marks":[{"type":"code_inline"}]},{"text":" \u003c 16 GiB). Runtime failure despite §2 passing → template was patched out → route to ","type":"text"},{"text":"physical-ai-infrastructure-setup-and-resilient-scaling","type":"text","marks":[{"type":"code_inline"}]},{"text":". Missing creds / URL artifacts → offer to submit ","type":"text"},{"text":"setup/setup_\u003ccase>.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":" + ","type":"text"},{"text":"setup/setup_pretrained.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":" first.","type":"text"}]},{"type":"paragraph","content":[{"text":"Then ask the user in one message — per-flow choices only (the first-time gate above already covered ","type":"text"},{"text":"dig_url_root","type":"text","marks":[{"type":"code_inline"}]},{"text":", pool, pod-template, and endpoint preferences; pull those from memory):","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Use case","type":"text","marks":[{"type":"strong"}]},{"text":" — PCBA (use Day 0 + pcb cookbook), metal surface (Day 1 + metal_surface cookbook), glass (Day 1 + glass cookbook), or custom?","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Checkpoint available?","type":"text","marks":[{"type":"strong"}]},{"text":" — If yes (","type":"text"},{"text":"use_pretrained_checkpoint=true","type":"text","marks":[{"type":"code_inline"}]},{"text":"), use ","type":"text"},{"text":"\u003cdig_url_root>/models/\u003cusecase>","type":"text","marks":[{"type":"code_inline"}]},{"text":" and provide ","type":"text"},{"text":"checkpoint_step","type":"text","marks":[{"type":"code_inline"}]},{"text":". If no, finetune from ","type":"text"},{"text":"\u003cdig_url_root>/datasets/\u003cusecase>/raw","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Local-NIM pool capacity check","type":"text","marks":[{"type":"strong"}]},{"text":" (Day 0 Option B only) — before ","type":"text"},{"text":"kubectl apply","type":"text","marks":[{"type":"code_inline"}]},{"text":", check ","type":"text"},{"text":"Total Capacity","type":"text","marks":[{"type":"code_inline"}]},{"text":" via ","type":"text"},{"text":"physical-ai-infrastructure-setup-and-resilient-scaling","type":"text","marks":[{"type":"code_inline"}]},{"text":". ","type":"text"},{"text":"Total Capacity \u003c 2","type":"text","marks":[{"type":"code_inline"}]},{"text":" cannot host NIM + DIG concurrently → ask user to add GPUs or switch to Option A. ","type":"text"},{"text":"image_edit_model","type":"text","marks":[{"type":"code_inline"}]},{"text":" is always ","type":"text"},{"text":"nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL","type":"text","marks":[{"type":"code_inline"}]},{"text":", never generic ","type":"text"},{"text":"qwen-image-edit","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Save user preferences to memory","type":"text","marks":[{"type":"strong"}]},{"text":" — after the first-time gate (and after any submit diverging from a documented default), persist load-bearing choices (","type":"text"},{"text":"dig_url_root","type":"text","marks":[{"type":"code_inline"}]},{"text":", OSMO pool, default board, image-edit endpoint, pod-template state, osmo-admin role). ","type":"text"},{"text":"Never save","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"image_edit_model","type":"text","marks":[{"type":"code_inline"}]},{"text":" (constant — saving invites drift) or ephemeral state (STAMP, one-off ","type":"text"},{"text":"anomaly_types_json","type":"text","marks":[{"type":"code_inline"}]},{"text":"). Full table: ","type":"text"},{"text":"references/preconditions.md","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" §4a \"Memory rules\"","type":"text","marks":[{"type":"strong"}]},{"text":". Read relevant memories at the start of every new conversation and apply silently.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Review the relevant flow reference before asking — most values have sensible defaults. Day 1 routing: PCBA defaults to ","type":"text"},{"text":"real_alignment","type":"text","marks":[{"type":"code_inline"}]},{"text":"; metal/glass have no USD flow so always ","type":"text"},{"text":"manual_roi","type":"text","marks":[{"type":"code_inline"}]},{"text":"; don't ask the user \"manual or real-alignment?\" for PCBA unless they explicitly ask to skip alignment.","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Common Preconditions (all flows)","type":"text"}]},{"type":"paragraph","content":[{"text":"Quick reference. Long-form: ","type":"text"},{"text":"references/preconditions.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"OSMO credentials + tokens","type":"text","marks":[{"type":"strong"}]},{"text":" — once per conversation. ","type":"text"},{"text":"If a ","type":"text","marks":[{"type":"strong"}]},{"text":".env","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" exists in the workspace, source it first","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"set -a; . ./.env; set +a","type":"text","marks":[{"type":"code_inline"}]},{"text":") so ","type":"text"},{"text":"HF_TOKEN","type":"text","marks":[{"type":"code_inline"}]},{"text":" is exported. Run ","type":"text"},{"text":"scripts/preflight_credentials.sh","type":"text","marks":[{"type":"code_inline"}]},{"text":"; authoritative check is the OSMO cred ","type":"text"},{"text":"hf-token","type":"text","marks":[{"type":"code_inline"}]},{"text":" is provisioned (images are public on ","type":"text"},{"text":"nvcr.io/nvidia/","type":"text","marks":[{"type":"code_inline"}]},{"text":" — no registry cred needed). Pass ","type":"text"},{"text":"--no-probe","type":"text","marks":[{"type":"code_inline"}]},{"text":" in restricted-egress shells. See ","type":"text"},{"text":"references/preconditions.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" §1.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Pod template","type":"text","marks":[{"type":"strong"}]},{"text":" — once per conversation, with cross-conversation memory caching (see Step 0 §6). Skip when memory records the cluster verified / user-confirmed / 409-skipped. Otherwise run ","type":"text"},{"text":"scripts/preflight_pod_template.sh","type":"text","marks":[{"type":"code_inline"}]},{"text":" and branch on exit code (0=verified / 1=patch via infra skill / 2=ask-user (HTTP 403) / 3=skip (HTTP 409) / 4=env-fix). Full branching prose and prompts in ","type":"text"},{"text":"references/preconditions.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" §2.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Required URL artifacts","type":"text","marks":[{"type":"strong"}]},{"text":" — before every submit. Run ","type":"text"},{"text":"DIG_URL_ROOT=\u003cdig_url_root> scripts/preflight_urls.sh \u003cflow> \u003cusecase> [variant]","type":"text","marks":[{"type":"code_inline"}]},{"text":". If anything is missing, ","type":"text"},{"text":"stop and submit the relevant ","type":"text","marks":[{"type":"strong"}]},{"text":"setup/setup_\u003ccase>.yaml","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" + ","type":"text","marks":[{"type":"strong"}]},{"text":"setup/setup_pretrained.yaml","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" first","type":"text","marks":[{"type":"strong"}]},{"text":" (the OSMO setup workflows) — see ","type":"text"},{"text":"references/setup.md","type":"text","marks":[{"type":"code_inline"}]},{"text":". ","type":"text"},{"text":"Never download assets locally to work around a problem; if setup fails on credentials, ask the user to rectify them and re-submit on OSMO.","type":"text","marks":[{"type":"strong"}]},{"text":" Per-flow checklist:","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Flow","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use case","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Required URL artifacts under ","type":"text"},{"text":"\u003cdig_url_root>","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 0 — Texture Defects","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"PCBA","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"models/pretrained","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"models/pcb","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"datasets/pcb/raw","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"datasets/pcb/assets","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 0 — Good Image","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"PCBA","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"datasets/pcb/assets","type":"text","marks":[{"type":"code_inline"}]},{"text":" only","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 0 — Structural Defects","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"PCBA","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"datasets/pcb/assets","type":"text","marks":[{"type":"code_inline"}]},{"text":" only","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 1","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Metal surface","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"models/pretrained","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"models/metal_surface","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"datasets/metal_surface/raw","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 1","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Glass","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"models/pretrained","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"models/glass","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"datasets/glass/raw","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 1 real-photo alignment","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"PCBA","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 1 PCBA plus ","type":"text"},{"text":"datasets/pcb/assets","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Finetune Only","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Any","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"models/pretrained","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"datasets/\u003cusecase>/raw","type":"text","marks":[{"type":"code_inline"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"Built-in ","type":"text"},{"text":"usecase","type":"text","marks":[{"type":"code_inline"}]},{"text":" values are ","type":"text"},{"text":"pcb","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"metal_surface","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"glass","type":"text","marks":[{"type":"code_inline"}]},{"text":". See ","type":"text"},{"text":"references/preconditions.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" §3.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Name stamping","type":"text","marks":[{"type":"strong"}]},{"text":" — regenerate ","type":"text"},{"text":"$STAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)","type":"text","marks":[{"type":"code_inline"}]},{"text":" before every submit and pass ","type":"text"},{"text":"--set name=\u003cflow>-$STAMP","type":"text","marks":[{"type":"code_inline"}]},{"text":". Production YAMLs ship no ","type":"text"},{"text":"name","type":"text","marks":[{"type":"code_inline"}]},{"text":" default. See ","type":"text"},{"text":"references/preconditions.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" §4.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Glass case (UC3) — Roboflow zip","type":"text","marks":[{"type":"strong"}]},{"text":" — only for ","type":"text"},{"text":"setup_glass.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":". Upload ","type":"text"},{"text":"mobile_screen.zip","type":"text","marks":[{"type":"code_inline"}]},{"text":" to an OSMO URL prefix first; pass ","type":"text"},{"text":"--set uc3_zip_url_root=\u003cprefix>","type":"text","marks":[{"type":"code_inline"}]},{"text":". Full procedure: ","type":"text"},{"text":"references/setup.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" §\"Glass case (UC3)\".","type":"text"}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Flow walkthroughs","type":"text"}]},{"type":"paragraph","content":[{"text":"Each flow's full walkthrough — group diagrams, prerequisites, submit-command variants, data handoffs, per-stage troubleshooting — lives under ","type":"text"},{"text":"references/flows/","type":"text","marks":[{"type":"code_inline"}]},{"text":". The agent should read the matching file before submitting any flow it hasn't run in the current conversation.","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Flow","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Workflow YAML","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Walkthrough","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 0 — Texture Defects (PCBA)","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"assets/configs/texture_defect_generation_day0.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/flows/texture_defect_generation_day0.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 0 — Good Image (PCBA)","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"assets/configs/good_image_generation.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/flows/good_image_generation.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 0 — Structural Defects (PCBA)","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"assets/configs/structural_defect_generation.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/flows/structural_defect_generation.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 1 — Infer + Label (real-photo alignment, default PCBA)","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"assets/configs/texture_defect_generation_day1_real_alignment.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/flows/texture_defect_generation_day1_real_alignment.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Day 1 — Infer + Label (manual ROI, metal/glass + PCBA experimentation)","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"assets/configs/texture_defect_generation_day1_manual_roi.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/flows/texture_defect_generation_day1_manual_roi.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Finetune Only","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"assets/configs/finetune.yaml","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/flows/finetune.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Cross-flow invariants","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"use_pretrained_checkpoint=true","type":"text","marks":[{"type":"code_inline"}]},{"text":" (default) → passthrough against ","type":"text"},{"text":"models/\u003cusecase>","type":"text","marks":[{"type":"code_inline"}]},{"text":". Set to ","type":"text"},{"text":"false","type":"text","marks":[{"type":"code_inline"}]},{"text":" to insert an in-pod ","type":"text"},{"text":"finetune-job","type":"text","marks":[{"type":"code_inline"}]},{"text":" group (cookbook yq-patched in-pod, no pre-submit render step).","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Day 0 emits per-cell ","type":"text"},{"text":"crop/\u003cMATERIAL>/\u003ccell>/...","type":"text","marks":[{"type":"code_inline"}]},{"text":" trees; Day 1 emits per-ROI crops registered against the USD; structural emits flat per-component crops.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Shipped per-usecase ","type":"text"},{"text":"checkpoint_step","type":"text","marks":[{"type":"code_inline"}]},{"text":" + ","type":"text"},{"text":"anomaly_types_json","type":"text","marks":[{"type":"code_inline"}]},{"text":" defaults: see ","type":"text"},{"text":"references/preconditions.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" §\"Shipped checkpoint and ","type":"text"},{"text":"anomaly_types_json","type":"text","marks":[{"type":"code_inline"}]},{"text":" defaults\".","type":"text"}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"OSMO Monitoring","type":"text"}]},{"type":"paragraph","content":[{"text":"Load ","type":"text","marks":[{"type":"strong"}]},{"text":"references/monitoring.md","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" before any ","type":"text","marks":[{"type":"strong"}]},{"text":"osmo workflow submit","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":", ","type":"text","marks":[{"type":"strong"}]},{"text":"osmo workflow query","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":", or ","type":"text","marks":[{"type":"strong"}]},{"text":"osmo workflow logs","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" action in this skill.","type":"text","marks":[{"type":"strong"}]},{"text":" It defines the polling cadence, task-status interpretation, log-pull escalation thresholds, failure-classification routing, and what to surface to the user vs. silently retry. Do not assemble a post-submit watch loop or status summary from memory — re-read it on the first such action of every conversation.","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"osmo workflow query \u003cworkflow_id> --format-type json | jq '{status, tasks: [.groups[].tasks[] | {name, status, exit_code}]}'\nosmo workflow logs \u003cworkflow_id> -t \u003ctask_name> -n 200\nosmo data download \u003cdig_url_root>/runs/\u003cname>/anomaly ./output/anomaly-\u003cname>/","type":"text"}]},{"type":"paragraph","content":[{"text":"Monitoring discipline: ","type":"text"},{"text":"references/monitoring.md","type":"text","marks":[{"type":"code_inline"}]},{"text":". Retrieval: ","type":"text"},{"text":"references/output_retrieval.md","type":"text","marks":[{"type":"code_inline"}]},{"text":". Presentation: ","type":"text"},{"text":"references/output_rendering.md","type":"text","marks":[{"type":"code_inline"}]},{"text":". Gotchas: ","type":"text"},{"text":"references/troubleshooting.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Response Template","type":"text"}]},{"type":"paragraph","content":[{"text":"For \"show me the plan / recipe\" requests, emit your final response with these labeled sections (so nothing truncates mid-recipe):","type":"text"}]},{"type":"paragraph","content":[{"text":"Workflow:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"\u003cflow name>","type":"text","marks":[{"type":"code_inline"}]},{"text":" → ","type":"text"},{"text":"assets/configs/\u003cyaml>","type":"text","marks":[{"type":"code_inline"}]}]},{"type":"paragraph","content":[{"text":"Preflights:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"scripts/preflight_credentials.sh","type":"text","marks":[{"type":"code_inline"}]},{"text":"; ","type":"text"},{"text":"scripts/preflight_urls.sh \u003c0|1|finetune> \u003cusecase> [variant]","type":"text","marks":[{"type":"code_inline"}]}]},{"type":"paragraph","content":[{"text":"Required URL Artifacts under ","type":"text","marks":[{"type":"strong"}]},{"text":"\u003cdig_url_root>","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":":","type":"text","marks":[{"type":"strong"}]},{"text":" enumerate per Common Preconditions §3 for the chosen flow.","type":"text"}]},{"type":"paragraph","content":[{"text":"Submit Command:","type":"text","marks":[{"type":"strong"}]}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"STAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)\nosmo workflow submit assets/configs/\u003cyaml> --pool \u003cpool> \\\n --set name=\u003cflow>-$STAMP dig_url_root=\u003croot> usecase=\u003cusecase> \\\n image_edit_endpoint=\u003cendpoint> image_edit_model=nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL \\\n checkpoint_step=\u003cstep> 'anomaly_types_json=\u003ctypes>'","type":"text"}]},{"type":"paragraph","content":[{"text":"Monitoring:","type":"text","marks":[{"type":"strong"}]},{"text":" load ","type":"text"},{"text":"references/monitoring.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" before running the submit; apply its polling cadence + log-pull thresholds after ","type":"text"},{"text":"osmo workflow submit","type":"text","marks":[{"type":"code_inline"}]},{"text":" returns a workflow id.","type":"text"}]},{"type":"paragraph","content":[{"text":"Output Location:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"\u003cdig_url_root>/runs/\u003cflow>-$STAMP/anomaly/","type":"text","marks":[{"type":"code_inline"}]},{"text":" (per-flow override: see flow walkthrough).","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Supporting files","type":"text"}]},{"type":"paragraph","content":[{"text":"Full inventory — workflow YAMLs, cookbooks, scripts table, references, evals, component skills — in ","type":"text"},{"text":"references/contents.md","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":". Top-level dirs: ","type":"text"},{"text":"assets/configs/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"assets/cookbooks/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"scripts/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"references/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"evals/","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},"metadata":{"date":"2026-11-02","name":"physical-ai-defect-image-generation","tags":["physical-ai","defect-image-generation","aoi","anomalygen","usd2roi","cosmos","cosmos-predict2","cosmos-anomalygen"],"tools":["Read","Shell"],"author":"@skillopedia","source":{"stars":3825,"repo_name":"skills","origin_url":"https://github.com/NVIDIA/skills/blob/HEAD/skills/physical-ai-defect-image-generation/SKILL.md","repo_owner":"NVIDIA","body_sha256":"02b77b4e1eedf523422a4f17fb6dd143cbfc044eca624ee8bbc716099c508828","cluster_key":"bcfd302880b6c0634f6f61f16cedec0420554524577c351c824665263e2f82db","clean_bundle":{"format":"clean-skill-bundle-v1","source":"NVIDIA/skills/skills/physical-ai-defect-image-generation/SKILL.md","text_bytes":458667,"source_commit":"25340619336e89bcc40365c28bfe9e790430468e","skills_sh_rank":8553,"attachment_count":52,"text_attachments":52,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"skills/physical-ai-defect-image-generation/SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"1.0.1","category":"ai-agent-development","category_label":"AI"},"exact_dupes_collapsed_into_this":0},"license":"CC-BY-4.0 AND Apache-2.0","version":"1.0.1","category":"ai-agent-development","metadata":{"tags":["physical-ai","defect-image-generation","aoi","anomalygen","usd2roi","cosmos","cosmos-predict2","cosmos-anomalygen"],"owner":"NVIDIA","author":"NVIDIA","service":"physical-ai-data-factory","version":"1.0.1","reviewed":"2026-06-23T00:00:00.000Z"},"import_tag":"skills-sh-top20k-2026-11","description":"Use when the user wants to orchestrate defect image generation with NVIDIA Cosmos AnomalyGen (Cosmos-Predict2-derived) on OSMO for PCBA, metal surface, and glass inspection. The Day 0 path handles cold-start with USD-to-ROI, image-edit augmentation, and AnomalyGen to create initial PCBA datasets. The Day 1 path performs inference and labeling on real images. This skill helps with first-time asset setup, creation of finetuning checkpoints, and configuring deployment.\nTrigger keywords: defect image generation, dig workflow, dig pipeline, defect image detection workflow, aoi pipeline, aoi anomalygen, usd2roi anomalygen, day 0 pcba, day 1 pcba, day 1 real-photo alignment, day 1 manual roi, metal surface anomaly, glass defect, anomalygen finetune, setup_pcb, setup_metal, setup_glass, setup_pretrained, dig setup, dig datasets, dig pretrained checkpoint, dig image-edit endpoint, cosmos defect generation, cosmos-predict2 defect, cosmos-anomalygen, cosmos predict2 finetune."}},"renderedAt":1793671765828}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.