debugging — Skillopedia

Debugging You are a hypothesis-driven debugger. Two disciplines apply regardless of language, runtime, or whether you have source: 1. Runtime truth beats code reading. Every claim about why the bug happens must come from observed state — never from a plausible story spun from reading code. 2. Leave no trace. Debugging creates artifacts. Every artifact is journaled and removed before you call the task done. The rest of this file is a map. The knowledge is in . This file cannot teach you how to debug — it can only tell you which reference will, for your exact situation. --- 🚨 READ THE REFERENC…

and data[i+1:i+2] == b'{':\n depth = 1; i += 2\n while i \u003c len(data) and depth > 0:\n cc = data[i:i+1]\n if cc == b'\\\\': i += 2; continue\n if cc == b'`':\n j = find_template_end(data, i+1)\n i = j + 1; continue\n if cc == b'{': depth += 1\n elif cc == b'}': depth -= 1\n elif cc in (b'\"', b\"'\"):\n q = cc; i += 1\n while i \u003c len(data) and data[i:i+1] != q:\n if data[i:i+1] == b'\\\\': i += 2\n else: i += 1\n i += 1; continue\n i += 1\n continue\n if c == b'`': return i\n i += 1\n return -1\n```\n\nFor function-body extraction, **track the parameter list separately** before tracking body braces. The naive approach mis-counts destructuring `function f({a, b, ...c})` as the body `{` and exits early.\n\n---\n\n## [5] Runtime verification\n\nYou usually cannot single-step JS inside a Bun-compiled binary the way you would with `node --inspect`. Workarounds:\n\n### Bun-compiled\n\nBun's inspector takes `--inspect[=\u003chost>:\u003cport>[/\u003cprefix>]]` on the command line. For env-var control of compiled binaries, the form is the same minus the leading `--`:\n\n```bash\n# Default port (6499) auto-prefix\n./target --inspect\n# → ws://localhost:6499/\u003cauto-prefix> (paste into https://debug.bun.sh)\n\n# Explicit host:port[/prefix]\n./target --inspect=localhost:9229/dbg\n# Or via env var (if --inspect cannot be passed)\nBUN_INSPECT=localhost:9229/dbg ./target\n```\n\n**For HTTP request tracing without an interactive debugger** (highest-value Bun-specific runtime evidence):\n\n```bash\n# Print every fetch() / node:http request as a curl command + full headers/body\nBUN_CONFIG_VERBOSE_FETCH=curl ./target ...\n\n# Or just print the request/response without curl-format\nBUN_CONFIG_VERBOSE_FETCH=true ./target ...\n```\n\nPlus generic env-var-based debug logging if the app supports it:\n\n```bash\nAPP_DEBUG=1 APP_LOG_LEVEL=debug APP_LOG_FILE=/tmp/trace.log ./target\n```\n\n### Node SEA / pkg / nexe\n```bash\n# These usually accept --inspect since they are real Node\n./target --inspect\n# Then chrome://inspect or node --inspect-brk\n```\n\n### Electron\n```bash\n./target.app/Contents/MacOS/target --inspect=9229 --remote-debugging-port=9223\n# Renderer process is at chrome://inspect, main process via the inspector port\n```\n\n### When you cannot make a real call\nThe target's API may require credentials, network access, or paid quota you don't have. **You are not stuck** — see [methodology/partial-runtime-evidence.md](../methodology/partial-runtime-evidence.md) for the fallback patterns.\n\n---\n\n## ⚠️ Gotchas — read these before extracting\n\n### G1. `strings -n N` silently drops short identifier interpolations\n\n`strings` outputs runs of printable characters of length **≥ N**. Default is 4 on most systems; many references (including older versions of `native-binary.md`) recommend `-n 8` for less noise.\n\n**With `-n 8`, short template-literal interpolations like `${x}`, `${i}`, `${R}` are silently dropped** because they are 4 chars surrounded by non-printable bytes (newlines or section padding). The result looks like:\n\n```text\nexpected: \u003cINSTRUCTIONS>\\n${x}\\n\u003c/INSTRUCTIONS>\nstrings: \u003cINSTRUCTIONS>\\n\u003c/INSTRUCTIONS> ← ${x} is gone, no warning\n```\n\nA consumer reading the strings output would conclude the template is empty.\n\n**Mitigation**:\n1. Use `strings` only for **fingerprinting** (Phase 1 triage), never as the source of extracted text.\n2. For actual extraction, **read the binary as bytes** with `python3 -c \"open('./target','rb').read()\"` and grep / parse from there.\n3. If you must use `strings`, try `strings -n 1 -t x ./target` and post-filter — but byte-level reads are still more reliable.\n\n### G2. Stale cached binary ≠ latest features\n\nBundled-app installers often check a remote version and skip download if a cached binary exists. If you reverse-engineered an old version and the user reports behavior you don't see in the source, **re-run the installer** (or fetch the version manifest manually) before assuming the source is current.\n\n```bash\n# Example pattern - varies by tool\ncurl -fsSL https://example.com/install.sh | head -50 # find version-fetch URL\ncurl -fsSL https://static.example.com/cli/cli-version.txt\n./your-tool --version\n# Compare. If different, re-install.\n```\n\n### G3. APFS / NTFS case-insensitivity silently overwrites files\n\nWhen extracting many minified function bodies (`cVR`, `CVR`, `dpr`, `DPR`, …) and saving each to its own file, **macOS APFS and Windows NTFS treat `cVR.txt` and `CVR.txt` as the same file**. The second write silently overwrites the first.\n\n**Mitigation**: prefix filenames with something case-distinguishing, e.g. `mode-cVR.txt`, `mode-CVR.txt`, or use a hash suffix.\n\n### G4. Bun's runtime adds 30-50 MB of unrelated symbols\n\nA 70 MB Bun-compiled binary is **mostly Bun runtime** (~50 MB) plus your app (~20 MB). When fingerprinting, you will see thousands of strings like `tree-sitter-typescript`, `react-native-stylex` etc. that the user's actual app doesn't use — these are package names baked into Bun's package-resolution data.\n\n**Mitigation**: when grepping for \"what does this app do?\", filter out runtime noise:\n```bash\nstrings -n 8 ./target | rg -v 'node_modules|@oven/bun|package-lock|tree-sitter|ffmpeg-installer' | head\n```\n\n### G5. Source maps usually NOT shipped\n\nBundled apps strip source maps for production. Variable names are minified to `T`, `R`, `a`, `r`, etc. Treat the bundle like an obfuscated codebase: identify constants by tracing assignments (`var T=\"actual-name\"`) and resolve interpolations manually.\n\n### G6. The \"extract\" file is not legally redistributable\n\nIf reverse-engineering proprietary software, the extracted source is the vendor's IP. Use it for understanding behavior, **never commit it to git**, never post snippets in public issues. Cleanup your `extracted-bundle.js` files in Phase 9.\n\n---\n\n## Silent-failure patterns specific to bundled JS\n\n| Pattern | Why it's silent |\n|---|---|\n| Bundle includes unreachable dead code from tree-shaking failures | You read code that never runs — verify with runtime trace |\n| `process.env.X` resolved at BUILD time, not RUNTIME | Setting the env var at runtime has no effect; the value is baked in |\n| `import.meta.url` in compiled binary returns `bun://...` not a real path | File-relative resolution silently breaks |\n| Worker threads spawn from embedded code, look for sub-bundle inside main bundle | Workers may have their own copy of dependencies |\n| Minified identifiers with case variants used in same module | Easy to confuse `cVR` with `CVR` when reading fast |\n\n---\n\n## Phase 9 cleanup specifics for bundled-JS work\n\n```bash\n# Remove extracted bundles — they may contain proprietary source\nrm -f /tmp/extracted-bundle.js /tmp/extracted-*.js\nrm -rf /tmp/asar-extracted/\nrm -rf /tmp/_MEI*\n\n# Remove strings dumps\nrm -f /tmp/*-strings.txt /tmp/*-strings-v*.txt\n\n# Remove Python helper scripts created for parsing\nrm -f /tmp/extract_bundled_js.py /tmp/parse_template.py\n\n# Verify the extraction directory is gone (if you used a workspace dir)\nls /Users/$USER/local-workspaces/*-extracted/ 2>/dev/null\n# rm -rf only after journal review confirms nothing important is there\n```\n\n---\n\n## When to escalate back to `native-binary.md`\n\nIf extraction reveals the \"bundle\" is actually compiled to v8 cached data (pkg with `--public-packages` or PyInstaller with bytecode-only mode), and decompilation is non-trivial, **switch back to `native-binary.md` workflow** (Ghidra against the runtime + careful tracing). Bundled-JS workflow only helps when the high-level source is recoverable as readable text.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":19353,"content_sha256":"f7f01bc4d95c24bd28d24a3ae240a037b0ded00d690d18a6900ddf63025fe4ec"},{"filename":"references/runtimes/go.md","content":"# Go Debugging\n\nCovers goroutines, `dlv` (Delve), `pprof`, the race detector, and the fact that Go's concurrency model means most bugs are about goroutines doing something quiet and wrong.\n\n---\n\n## Environment detection (Phase 0)\n\n```bash\ngo version\ncat go.mod | head -5\n\n# Delve installed?\nwhich dlv\ndlv version\n\n# Build constraints\ngrep -r '// +build\\|//go:build' cmd/ internal/ pkg/ 2>/dev/null | head\n\n# pprof wired up?\ngrep -r 'net/http/pprof\\|runtime/pprof' --include='*.go' | head -3\n```\n\n---\n\n## Delve (`dlv`) — the Go debugger\n\nGo's gc compiler emits DWARF, but plain gdb barely understands goroutines. **Use dlv, not gdb.** Plain gdb on a Go binary will miss goroutine state and print garbage for interface values.\n\n### The five `dlv` launch modes\n\n```bash\n# Build and launch under debugger (equivalent to `go run` + debug)\ndlv debug ./cmd/server -- --port=8080\n\n# Debug a test binary\ndlv test ./internal/handler/ # enters the test package under debug\n\n# Debug an existing binary (must be built with -gcflags=\"all=-N -l\" for best results)\ndlv exec ./bin/myserver\n\n# Attach to a running process\ndlv attach $(pgrep myserver)\n\n# Headless mode (IDE / remote attach) — default port 2345\ndlv debug --headless --listen=:2345 --api-version=2 ./cmd/server\n```\n\n### Building a debuggable binary\n\nThe compiler inlines and optimizes aggressively in normal builds, which makes stepping confusing. For serious debugging:\n\n```bash\ngo build -gcflags=\"all=-N -l\" -o ./bin/server ./cmd/server\n# -N disables optimization\n# -l disables inlining\n```\n\nThen `dlv exec ./bin/server`.\n\n### Essential dlv commands\n\n```\n(dlv) b main.main # breakpoint at function\n(dlv) b handler.go:42 # breakpoint at file:line\n(dlv) b pkg/foo.Bar # breakpoint at type method (Go path syntax)\n(dlv) c / continue # continue until next break\n(dlv) n / next # step over\n(dlv) s / step # step into\n(dlv) so / stepout # step out\n(dlv) bt / stack # stack trace of current goroutine\n(dlv) goroutines # list all goroutines\n(dlv) goroutine \u003cid> # switch to goroutine N\n(dlv) goroutine \u003cid> bt # stack of a specific goroutine\n(dlv) locals # all locals in frame\n(dlv) args # function args\n(dlv) p \u003cexpr> # print value (understands interfaces, maps, slices)\n(dlv) vars \u003cregex> # package vars matching regex\n(dlv) regs # registers (rare in Go debugging)\n(dlv) on \u003cbpid> print \u003cexpr> # auto-print on breakpoint hit (powerful!)\n(dlv) trace \u003clocation> # like breakpoint but just logs, doesn't stop\n```\n\nThe `trace` command is underused — it's like a logpoint, no stepping required.\n\n---\n\n## Goroutine-centric debugging\n\nGoroutine leaks and deadlocks are the most common Go bugs. `dlv`'s `goroutines` command is the starting point.\n\n```\n(dlv) goroutines -t # with truncated stack\n(dlv) goroutines -s # sorted by stack\n(dlv) goroutines -with user # filter user-spawned goroutines\n```\n\nCommon patterns:\n\n| You see in `goroutines` | Usually means |\n|---|---|\n| 100s of goroutines stuck at `chan receive` | Producer died; consumers leak |\n| 100s stuck at `semacquire` | Lock contention; a holder probably deadlocked |\n| One stuck at `select` with no default | Missing case or closed channel scenario |\n| Stuck at `netpoll` | External I/O not responding — not a Go bug, check downstream |\n| Growing count over time | Goroutine leak — need to find who's spawning without cleanup |\n\n### Panic signals in Go\n\n```go\n// Without recovery, panics crash the program with a stack trace of ALL goroutines\n// With recovery, they're silent unless explicitly logged:\ndefer func() {\n if r := recover(); r != nil {\n log.Printf(\"recovered panic: %v\\n%s\", r, debug.Stack()) // GOOD\n // log.Printf(\"recovered\") // BAD — silent\n }\n}()\n```\n\n**Always check for silent recovers** in Phase 8. Grep:\n```bash\nrg 'recover\$\$' --type go\n```\n\nAnd inspect each site for whether the panic is actually surfaced.\n\n---\n\n## Race detector — ALWAYS run when the bug is intermittent\n\n```bash\ngo test -race ./...\ngo run -race ./cmd/server\ngo build -race ./cmd/server\n```\n\nThe race detector wraps memory accesses and catches concurrent read/write without synchronization. **Run this before attaching dlv** if intermittency is involved — it often finds the bug directly.\n\nOutput shape:\n```\nWARNING: DATA RACE\nRead at 0x00c0001a0080 by goroutine 7:\n main.(*Counter).Value()\n /path/to/counter.go:14 +0x3c\nPrevious write at 0x00c0001a0080 by goroutine 6:\n main.(*Counter).Inc()\n /path/to/counter.go:10 +0x5f\n```\n\nBoth stacks. Both goroutines. The race is obvious from the line pair.\n\n---\n\n## pprof — for perf, memory, goroutine leaks\n\n### Wire it up (idempotent; usually already present)\n\n```go\nimport _ \"net/http/pprof\"\n\nfunc main() {\n go func() {\n log.Println(http.ListenAndServe(\"localhost:6060\", nil))\n }()\n // ... rest of your server\n}\n```\n\n### Queries\n\n```bash\n# CPU profile (30s)\ngo tool pprof http://localhost:6060/debug/pprof/profile?seconds=30\n\n# Heap snapshot\ngo tool pprof http://localhost:6060/debug/pprof/heap\n\n# Goroutine snapshot — find leaks\ngo tool pprof http://localhost:6060/debug/pprof/goroutine\n\n# Block profile — find blocking ops (needs runtime.SetBlockProfileRate)\ngo tool pprof http://localhost:6060/debug/pprof/block\n\n# Mutex profile — find lock contention (needs runtime.SetMutexProfileFraction)\ngo tool pprof http://localhost:6060/debug/pprof/mutex\n```\n\nInside pprof:\n```\n(pprof) top # top functions by self time\n(pprof) list main.handler # annotated source of a function\n(pprof) web # SVG callgraph in browser (requires graphviz)\n(pprof) traces # sample traces\n```\n\nFor goroutine leaks, **take two snapshots 30s apart** and diff:\n```bash\ngo tool pprof -base prof1.pb.gz prof2.pb.gz\n```\n\nGoroutines that appear in prof2 but not prof1 are new; if they stick around, they're leaking.\n\n---\n\n## `GODEBUG` — runtime-level observability\n\n```bash\nGODEBUG=gctrace=1 ./myserver # print GC stats\nGODEBUG=schedtrace=1000 ./myserver # scheduler trace every 1000ms\nGODEBUG=scheddetail=1,schedtrace=1000 # detailed scheduler state\nGODEBUG=allocfreetrace=1 ./myserver # every alloc/free (noisy!)\nGODEBUG=memprofilerate=1 ./myserver # profile every allocation\n```\n\nUseful for diagnosing GC pressure, goroutine starvation, or memory pattern issues.\n\n---\n\n## Silent-failure patterns in Go\n\n| Pattern | Why it's silent |\n|---|---|\n| `if err != nil { return err }` that returns to a caller that ignores | Error bubbles up, then gets discarded at the top |\n| `defer func() { recover() }()` — bare recover, no log | Panic swallowed, program continues with state corruption |\n| `_, _ = conn.Write(data)` | Intentionally discarded error |\n| Buffered channel send that blocks forever | Sender hangs; hard to see if no deadlock detection |\n| `time.Sleep` in a test | \"Works on my machine\"; test passes locally, fails in CI |\n| `go func() { ... }()` with no error path | Goroutine dies silently on panic unless recover+log |\n| Context canceled but operation continues | Ignored `ctx.Err()` check |\n| `json.Unmarshal` of zero-value struct field | Input missing the key; silently zero |\n| Closed channel read returning zero value | Consumer doesn't check `ok`; reads forever |\n\n---\n\n## Phase 9 cleanup specifics\n\n```bash\n# Kill dlv sessions\npkill -f 'dlv' || true\nlsof -iTCP:2345 -sTCP:LISTEN -nP 2>/dev/null # dlv default\n\n# Kill pprof HTTP endpoint if you started it just for this session\nlsof -iTCP:6060 -sTCP:LISTEN -nP 2>/dev/null\n\n# Revert any `fmt.Println(\"DEBUG: ...\")` or `log.Printf(\"DEBUG: ...\")` additions\ngit diff | grep -E '(fmt\\.Println\\(\"DEBUG|log\\.Printf\\(\"DEBUG|println!)'\ngit checkout \u003cfile>\n\n# Unset env vars\nunset GODEBUG\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":8181,"content_sha256":"284614797d02d1aee25f440a7daa145af59a3ff61e5ba79c28ad0542c28f4f00"},{"filename":"references/runtimes/native-binary.md","content":"# Native Binary Debugging (No Source / Reverse Engineering)\n\nFor binaries where you don't have trustworthy source: stripped production builds, third-party closed libs, malware, CTF challenges, firmware, vendored libs whose docs lie. The workflow is specific; doing it out of order wastes days.\n\nThis reference **coordinates** the triage and dynamic work. The heavy tools each have their own reference:\n- **Static decompilation** → [tools/ghidra.md](../tools/ghidra.md)\n- **Interactive debugging** → [tools/pwndbg.md](../tools/pwndbg.md)\n- **Scripted interaction / exploitation** → [tools/pwntools.md](../tools/pwntools.md)\n\nRead those before using them — especially Ghidra, which has a surprising amount of workflow that's not obvious.\n\n---\n\n## ⚠️ STOP — is this actually a stripped C/C++ binary?\n\nA growing share of \"binaries\" are actually **bundled high-level apps** — Bun SEA, Node SEA, Deno compile, pkg, nexe, Electron, Tauri, PyInstaller. Their workflow is completely different: the high-level source is recoverable with the right per-bundler tool (often plaintext, sometimes V8 cache / `.pyc` / eszip needing extra tooling), and Ghidra against the runtime VM wastes hours.\n\nQuick check:\n\n```bash\nfile ./target # Mach-O / ELF / PE - inconclusive\ndu -h ./target # 50 MB+ for a \"simple CLI\" → suspect bundled\nstrings -n 12 ./target | rg -iE 'bun|node_modules|webpack|esbuild|deno|pkg/lib|electron|pyinstaller|nexe|NODE_SEA_FUSE|tauri' | head -5\n```\n\n**If any hits** → close this file, open [bundled-js-binary.md](bundled-js-binary.md) instead. Following the Ghidra/pwndbg path on a bundled-app binary wastes hours decompiling the runtime VM while the app-level bundle is recoverable with the right per-bundler tool (plaintext for Bun/pkg/nexe/Electron-asar; eszip / V8-cache / `.pyc` for Deno / Node SEA / PyInstaller).\n\nIf `file` says \"Mach-O\" or \"ELF\", `du` is \u003c 20 MB, and the strings check is empty → continue here.\n\n---\n\n## The workflow (do these in order)\n\nEvery step's output is input to the next. Skipping steps means guessing later.\n\n```\n [1] Triage → what kind of binary is this?\n [2] Dynamic tracing → what syscalls / libcalls does it make?\n [3] Static analysis → what does it DO, in readable form? (Ghidra)\n [4] Dynamic debug → confirm hypotheses at runtime (pwndbg)\n [5] Scripted repro → lock the bug with a pwntools script\n [6] TDD + fix / report\n```\n\nSteps 1 and 2 are fast (minutes). Step 3 is slow (tens of minutes to hours depending on size). Don't skip 1-2 and go straight to Ghidra — the triage output tells you what to focus on inside Ghidra.\n\n---\n\n## [1] Triage — 5-minute fingerprint\n\n```bash\n# Basic identity\nfile ./target\n# elf, mach-o, pe? 32/64-bit? dynamically linked? stripped?\n\n# Architecture details\nreadelf -h ./target # ELF header: entry point, arch, type\nlipo -info ./target 2>/dev/null # macOS: universal binary?\n\n# Interesting strings (often leaks function names, error messages, URLs, API keys)\nstrings -n 8 ./target | head -100\nstrings -n 8 ./target | grep -iE '(http|/api/|error|debug|version)'\n\n# Imported symbols (what does it link against?)\nnm -D ./target 2>/dev/null # dynamic symbols\nobjdump -T ./target 2>/dev/null # same, alternate tool\nreadelf -d ./target # dynamic section (NEEDED libs)\nldd ./target 2>/dev/null # resolved library paths\n\n# Security posture (affects what exploits / bugs are possible)\nchecksec --file=./target # requires pwntools or installing checksec\n# NX, PIE, RELRO, stack canary, FORTIFY\n\n# Is it stripped?\nnm ./target 2>/dev/null | head # empty? stripped. full? not stripped.\nfile ./target # will say \"stripped\" or \"not stripped\"\n```\n\n### ⚠️ `strings -n N` silently drops short content\n\n`strings` prints runs of printable characters of length **≥ N**. With `-n 8`, **anything shorter than 8 chars sandwiched between non-printable bytes is dropped silently**. This includes:\n\n- Short identifier interpolations in templates (`${x}`, `${i}`, `${R}`)\n- Short embedded constants (`v3`, `null`, integer immediates as bytes)\n- Short error codes between binary padding\n\nReal example: a JavaScript template literal `\u003cINSTRUCTIONS>\\n${x}\\n\u003c/INSTRUCTIONS>` came out of `strings -n 8` as `\u003cINSTRUCTIONS>\\n\u003c/INSTRUCTIONS>` — the `${x}` (4 chars) was dropped. A consumer reading the dump would conclude the template was empty. It is not.\n\n**Use `strings` only for fingerprinting (Phase 1).** For any extraction whose correctness matters, **read bytes directly**:\n\n```bash\n# Count occurrences of a needle\nLC_ALL=C grep -aoc 'NEEDLE' ./target\n\n# Find offsets\nLC_ALL=C grep -aob 'NEEDLE' ./target | head\n\n# Or via Python for byte-precise context\npython3 -c \"\nimport sys\ndata = open('./target','rb').read()\nneedle = b'NEEDLE'\npos = data.find(needle)\nprint(repr(data[max(0,pos-100):pos+200]))\n\"\n```\n\nIf you must keep using `strings`, lower the threshold: `strings -n 1 -t x ./target | rg ...`. The signal-to-noise drops sharply but short content is preserved.\n\nWrite the triage summary to the journal:\n\n```markdown\n## Binary triage\n- Type: \u003cELF 64-bit, dynamically linked, stripped>\n- Arch: \u003cx86_64 | arm64 | ...>\n- Libs: \u003clibc, openssl, libcurl>\n- Security: \u003cNX, PIE, Partial RELRO, no canary>\n- Interesting strings: \u003cshort list>\n- First hypothesis surface: \u003cwhich function / area looks most relevant>\n```\n\n---\n\n## [2] Dynamic tracing — what does it actually call?\n\nThese are cheap — run them before Ghidra to orient yourself.\n\n### Linux: strace + ltrace\n\n```bash\n# System calls\nstrace -f -o trace.out ./target arg1 arg2\nstrace -f -e trace=network ./target # filter to network syscalls\nstrace -f -e trace=file ./target # filter to file ops\n\n# Library calls (less useful when stripped but still informative)\nltrace -f -o ltrace.out ./target\nltrace -f -e 'str*+mem*' ./target # filter to string/mem functions\n```\n\n### macOS: Mach-O specifics\n\n**SIP block reality check.** With System Integrity Protection enabled (default on every modern macOS), `dtruss` / `dtrace` will **silently fail** to attach to:\n- Anything in `/usr`, `/bin`, `/sbin`, `/System`\n- Apple-signed binaries (Xcode CLT, Homebrew formulae from Apple-distributed taps)\n- Notarized vendor binaries (Bun, Deno, Docker Desktop, etc.)\n\n`dtruss ./target` will appear to run but produce zero events. This is not a bug; it is the SIP design. Disabling SIP requires a Recovery Mode reboot — usually not worth it. Use the alternatives below.\n\n```bash\n# dtruss — works only when SIP allows it (your own unsigned binaries)\nsudo dtruss -f ./target 2>&1 | head -20 # equivalent to strace\n# If output is suspiciously empty → SIP blocked it. Switch to lldb or app-level logging.\n```\n\n**Mach-O metadata inspection (no SIP issues, no debugger needed):**\n\n```bash\n# Architecture and slices\nfile ./target # arm64 / x86_64 / universal\nlipo -info ./target # which architectures included\nlipo -thin arm64 ./target -output ./target-arm64 # extract one slice for analysis\n\n# Headers & load commands (segments, dylibs, code-signature pointer)\notool -h ./target # Mach header (cputype, ncmds, flags)\notool -l ./target | head -100 # load commands; entitlements live in code-signature blob, see codesign below\n\n# Dynamic library dependencies (macOS equivalent of ldd)\notool -L ./target # linked dylibs with versions\ndyld_info ./target # macOS 13+, more detailed than otool -L\n\n# Disassembly\notool -tv ./target | head -200 # quick disassembly without Ghidra\notool -tV ./target # with symbol-resolved branches\n\n# Imported / exported symbols (Apple `nm`, NOT GNU)\nnm -u ./target # undefined references = imports\nnm -gU ./target # external defined = exports\n# Note: GNU `-D`/dynamic flags are not honored on Apple `nm`; use the above forms.\nsymbols -fullSourcePath -onlyWithDebugInfo ./target # if any debug info survives\n\n# Code signature & entitlements (entitlements come from codesign, NOT otool)\ncodesign -dv --entitlements :- ./target 2>&1 # signature info + entitlements XML on stdout\nspctl --assess --type execute -vv ./target # Gatekeeper assessment\n\n# Cert chain — extract to a temp dir to avoid creating files named -0/-1 in cwd\ntmp=$(mktemp -d)\ncodesign -dvv --extract-certificates=\"$tmp/cert\" ./target 2>&1\nls -la \"$tmp\"\n# rm -rf \"$tmp\" # journal first, clean up later\n\n# Strings inside specific segments only (less noise than full-binary strings)\notool -s __TEXT __cstring ./target # C string section\notool -s __TEXT __const ./target # constants section\n```\n\n**Interactive debugging on macOS — use `lldb`, not `gdb`.**\n\nGDB on macOS requires a self-signed code-signing certificate (`codesign --entitlements gdb.entitlements --sign gdb-cert /opt/homebrew/bin/gdb`) and even then is unreliable on arm64. **Use `lldb` directly** — it ships with Xcode CLT and works without configuration.\n\n```bash\n# Start lldb\nlldb ./target\n\n# Set arguments\n(lldb) settings set target.run-args arg1 arg2\n\n# Run with breakpoints\n(lldb) breakpoint set --name function_name # symbol-based\n(lldb) breakpoint set --address 0x1000034c0 # address-based\n(lldb) breakpoint set --regex '.*decode.*' # regex over symbols\n\n# Run / step / inspect\n(lldb) run\n(lldb) bt # backtrace\n(lldb) frame variable # locals\n(lldb) register read # all registers\n(lldb) memory read --size 8 --format x --count 16 $sp # 16 qwords from stack\n(lldb) disassemble --frame # current function\n(lldb) image list # loaded modules\n(lldb) image lookup -a 0x1000034c0 # which module + symbol owns this address\n\n# Process attach to running process\n(lldb) process attach --pid 12345\n(lldb) process attach --name target # attach by name\n\n# Print Mach-O specific\n(lldb) image dump sections ./target\n(lldb) image dump symtab ./target\n```\n\n**Function interception via `DYLD_INSERT_LIBRARIES`** (macOS equivalent of `LD_PRELOAD`):\n\n```bash\n# Build a shim dylib that overrides specific functions\n# Then run target with it preloaded\nDYLD_INSERT_LIBRARIES=./shim.dylib DYLD_FORCE_FLAT_NAMESPACE=1 ./target\n```\n\nDYLD_INSERT works in the unrestricted case but is blocked in three distinct scenarios — distinguish them when diagnosing why your shim didn't load:\n\n1. **SIP / restricted process** (target has the `__RESTRICT,__restrict` section, is setuid/setgid, or is a platform/Apple-signed binary): dyld unconditionally strips all `DYLD_*` env vars before the process starts. Nothing you set will reach the target.\n2. **Hardened runtime + library validation** (`CS_RUNTIME` flag set, `com.apple.security.cs.disable-library-validation` entitlement absent): the process accepts `DYLD_INSERT_LIBRARIES` but **rejects** loading any dylib that isn't signed by the same Team ID or by Apple. Symptom: shim is found but not loaded; check `log show --predicate 'eventMessage CONTAINS \"library validation failed\"'`.\n3. **Notarization / Gatekeeper translocation**: the binary may be running from a translocated path; relative paths in `DYLD_INSERT_LIBRARIES` won't resolve. Use absolute paths.\n\nCheck each:\n\n```bash\n# Restrict segment present? (case 1)\notool -l ./target | grep -A2 __RESTRICT\n# Hardened runtime flag? (case 2)\ncodesign -d --verbose=4 ./target 2>&1 | grep -iE 'flags=|CodeDirectory'\n# Look for \"0x10000(runtime)\" or similar in the flags line.\n# Disable-library-validation entitlement?\ncodesign -d --entitlements :- ./target 2>&1 | grep disable-library-validation\n```\n\n**App-level debug logging (always works, ignores SIP):**\n\nWhen debugger attach is blocked, fall back to maximizing the app's own logging:\n\n```bash\n# Try common patterns\nAPP_DEBUG=1 APP_LOG_LEVEL=debug APP_LOG_FILE=/tmp/trace.log ./target\nNSDebugEnabled=YES ./target # Cocoa apps\nOS_ACTIVITY_MODE=debug ./target # os_log subsystem\n\n# Then read os_log unified logging stream live\nlog stream --predicate 'process == \"target\"' --level debug\n\n# Or extract historical logs\nlog show --predicate 'process == \"target\"' --last 1h --info --debug\n```\n\nThis is the **partial-runtime-evidence path** for macOS. See [methodology/partial-runtime-evidence.md](../methodology/partial-runtime-evidence.md) for how to combine app-level logs with static analysis when wire-level capture is blocked.\n\n**Network capture on macOS (TLS-decrypted):**\n\n```bash\n# 1. Find the active network service (don't assume \"Wi-Fi\"):\n# Map the default-route interface to the matching networksetup service name.\nnetworksetup -listallnetworkservices # show options\nDEFAULT_IF=$(route -n get default 2>/dev/null | awk '/interface:/ {print $2}')\necho \"Default-route interface: $DEFAULT_IF\"\n# Match the interface (en0, en1, ...) back to a service name:\nSERVICE=$(networksetup -listallhardwareports | awk -v iface=\"$DEFAULT_IF\" '\n /^Hardware Port:/ { hp = substr($0, index($0,$3)) }\n /^Device:/ { if ($2 == iface) print hp }\n')\nif [ -z \"$SERVICE\" ]; then\n echo \"Could not auto-detect active service. Pick one from -listallnetworkservices manually.\" >&2\n echo \"Aborting proxy setup.\" >&2\n false # signal failure but stay safe at top level\nelse\n echo \"Using service: $SERVICE\"\nfi\n\n# 2. JOURNAL the original proxy state before changing it (REQUIRED for safe rollback):\nnetworksetup -getwebproxy \"$SERVICE\" # save this output to journal\nnetworksetup -getsecurewebproxy \"$SERVICE\" # save this too\n\n# 3. Start mitmproxy with persistent CA at ~/.mitmproxy/\nmitmproxy --listen-host 127.0.0.1 --listen-port 8888 &\n\n# 4. Trust the mitmproxy CA system-wide if the target uses URLSession or any framework\n# that ignores HTTPS_PROXY/SSL_CERT_FILE (most macOS-native apps do):\nsudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ~/.mitmproxy/mitmproxy-ca-cert.pem\n\n# 5. Two routing options. Try env-var first; fall back to system proxy:\n# 5a. Apps that honor env vars (most CLIs):\nHTTPS_PROXY=http://127.0.0.1:8888 SSL_CERT_FILE=~/.mitmproxy/mitmproxy-ca-cert.pem ./target ...\n\n# 5b. Apps that use URLSession / system network config (most GUI apps, Bun, some CLIs):\nnetworksetup -setwebproxy \"$SERVICE\" 127.0.0.1 8888\nnetworksetup -setsecurewebproxy \"$SERVICE\" 127.0.0.1 8888\n\n# 6. Cleanup — RESTORE original state from journal, untrust CA:\nnetworksetup -setwebproxystate \"$SERVICE\" off\nnetworksetup -setsecurewebproxystate \"$SERVICE\" off\nsudo security delete-certificate -c \"mitmproxy\" /Library/Keychains/System.keychain\n```\n\n**Critical**: forgetting step 6 leaves all your subsequent traffic mis-routed and silently MITM-able. Journal every step.\n\n### What to look for\n\n| Observation | Hypothesis |\n|---|---|\n| `open(\"/etc/secret-config\", ...)` | Reads unexpected config; look at what it does with contents |\n| `connect(... 1.2.3.4:443)` | Phones home or depends on an external service |\n| `getenv(\"FOO\")` returning NULL | Env var expected but not set |\n| Repeated `poll`/`epoll_wait` with no progress | Stuck on I/O; check downstream |\n| `SIGSEGV` caught by signal handler | Custom crash recovery — often hides the real bug |\n| `dlopen(\"libfoo.so.42\")` | Dynamic plugin loading; check plugin path |\n\n---\n\n## [3] Static analysis with Ghidra\n\nWhen triage + tracing have narrowed you to \"something in function X\" or \"the crypto routine is weird\", open Ghidra.\n\n**Open [tools/ghidra.md](../tools/ghidra.md) before launching Ghidra** — the import / analyze / decompile workflow is not obvious and first-time users waste an hour figuring it out.\n\nGhidra's decompiler turns machine code into readable-ish C. That's usually what you want. Stay in the Decompiler view; drop to Listing (disassembly) only when the decompiler punts.\n\n---\n\n## [4] Dynamic debugging with pwndbg\n\nOnce static analysis gives you a hypothesis (\"this branch at 0x401234 is where the validation fails\"), confirm it at runtime with pwndbg.\n\n**Open [tools/pwndbg.md](../tools/pwndbg.md) before launching gdb.** Pwndbg gives you the context view (registers / stack / disasm / code all visible at once) which is essential for binary debugging.\n\nTypical pwndbg flow:\n\n```\n$ gdb ./target # pwndbg loads automatically if installed\npwndbg> break *0x401234 # break at the address static analysis flagged\npwndbg> run arg1 arg2\n# At the breakpoint:\npwndbg> context # registers + stack + disasm\npwndbg> telescope $rdi # walk pointers at $rdi\npwndbg> x/20xw $rsp # raw dump of stack\npwndbg> ni / si # step next / step instruction\n```\n\n---\n\n## [5] Scripted reproduction with pwntools\n\nOnce you have a hypothesis with a concrete repro input, lock it down with pwntools. This is the \"failing test\" equivalent for binaries.\n\n**Open [tools/pwntools.md](../tools/pwntools.md)** — the Process/Remote/ELF/context APIs are the foundation.\n\n```python\nfrom pwn import *\n\ncontext.binary = elf = ELF('./target')\n\np = process('./target')\np.sendlineafter(b'> ', b'\u003ctrigger input that reproduces the bug>')\nresult = p.recvall(timeout=3)\nassert b'expected-output-when-fixed' in result, f'bug repro: {result}'\n```\n\nThis script is now your \"red test\". When the fix is applied, the script should pass (or the assertion should be inverted for negative tests — e.g. \"the crash string should NOT appear\").\n\n---\n\n## [6] Fixing a binary bug you can't recompile\n\nThree options, in preference order:\n\n### Option A: Patch at the source (if you have it)\n\nIf the bug is in your own code and source is available, fix it there and rebuild. Standard TDD path.\n\n### Option B: Binary patch\n\nFor tiny fixes (one byte, one branch inversion):\n\n```bash\n# Identify the exact byte offset\n# e.g. Ghidra says the bug is at 0x401234 = file offset 0x1234\nprintf '\\x90\\x90' | dd of=./target bs=1 seek=$((0x1234)) conv=notrunc\n```\n\nJournal the exact `dd` command and the original bytes so you can revert.\n\n### Option C: Wrap / shim\n\nIf you can't patch the binary, write a shim library (LD_PRELOAD on Linux, DYLD_INSERT_LIBRARIES on macOS) that overrides the buggy function. pwntools has examples.\n\n### Option D: Report upstream\n\nIf it's a third-party binary and none of the above are feasible, the \"fix\" is a high-quality bug report with:\n- Full triage summary\n- Reproducible pwntools script\n- Ghidra decompilation of the buggy function\n- Hypothesis about the root cause\n- Recommended patch sketch (in C or pseudocode)\n\n---\n\n## Silent-failure patterns in native binaries\n\n| Pattern | Why it's silent |\n|---|---|\n| Ignored libc return codes (`read`, `write`, `malloc`) | Bug continues with garbage data; no check |\n| Signal handler swallows SIGSEGV | Crash converted to \"something didn't work\"; no log |\n| `setjmp`/`longjmp` unwinding over cleanup | Resources leak silently |\n| Thread-local error state never read (`errno`, `GetLastError`) | Error happened, nobody asked |\n| Recovered assertion failure in release build | `assert` compiled out; precondition violations silently corrupt |\n| Dangling pointer reads after free | Often looks like valid data until it doesn't |\n\n---\n\n## Phase 9 cleanup specifics\n\n```bash\n# Kill debugger sessions\npkill -f 'gdb' || true\npkill -f 'lldb' || true\n\n# Ghidra scratch projects (if made just for this session)\n# Named something like ~/ghidra-projects/debug-\u003ctimestamp>:\nls -la ~/ghidra-projects/ 2>/dev/null\n# rm -rf ~/ghidra-projects/debug-scratch # only if the journal says to\n\n# Core dumps left from crashes\nrm -f ./core ./core.* ~/core.*\n\n# strace/ltrace output files\nrm -f trace.out ltrace.out\n\n# If you made a binary patch (Option B above), confirm revert\n# The journal should have the original bytes — restore them:\n# printf '\u003coriginal-bytes>' | dd of=./target bs=1 seek=\u003coffset> conv=notrunc\n\n# Trace-output files\nrm -f /tmp/debug-*.bin /tmp/debug-*.strace /tmp/debug-*.ltrace\n\n# macOS-specific:\n# Restore proxy settings if you set them (CRITICAL — leaves system traffic mis-routed otherwise)\n# Use the SAME $SERVICE you used when enabling the proxy (read it from the journal).\n# Do NOT hardcode \"Wi-Fi\" — many machines route traffic over Ethernet, USB tether, or a VPN service.\n[ -n \"$SERVICE\" ] && {\n networksetup -setwebproxystate \"$SERVICE\" off 2>/dev/null\n networksetup -setsecurewebproxystate \"$SERVICE\" off 2>/dev/null\n}\n# Or restore explicitly from the journaled original state — see the proxy section above.\n\n# Stop mitmproxy\npkill -f 'mitmproxy' 2>/dev/null\n\n# Remove DYLD shim libraries you built\nrm -f /tmp/*-shim.dylib\n\n# Clear extracted strings dumps (these can be huge and may contain secrets)\nrm -f /tmp/*-strings*.txt\n\n# Verify hostname resolution returns to normal (mitmproxy can leave entries)\nscutil --dns | head -20\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":21367,"content_sha256":"520a767fadbd4d0b1b59fb6c652ded7f47a0ff5d742563f927334ae4537c2be0"},{"filename":"references/runtimes/node.md","content":"# Node.js / tsx / ts-node / Bun / Deno Debugging\n\nCovers Node 18+, tsx, ts-node, Bun, Deno. Launch recipes, inspector protocol usage, the `node inspect` CLI, and the **tsx source-map silent-failure** that costs people days.\n\n---\n\n## Environment detection (Phase 0)\n\n```bash\nnode --version\ncat package.json | head -40\n\n# Which JS runtime launches the app? (order them; the first match wins)\nls node_modules/.bin/tsx 2>/dev/null && echo 'has tsx'\nls node_modules/.bin/ts-node 2>/dev/null && echo 'has ts-node'\nls node_modules/.bin/vitest 2>/dev/null && echo 'has vitest'\nwhich bun 2>/dev/null && bun --version\nwhich deno 2>/dev/null && deno --version\n\n# Source-map situation\ngrep -E '\"sourceMap\"|\"inlineSources\"' tsconfig.json 2>/dev/null\ngrep -l '//# sourceMappingURL' dist/*.js 2>/dev/null | head -3\n\n# Debug-relevant ports\nlsof -iTCP:9229 -sTCP:LISTEN -nP 2>/dev/null\nlsof -iTCP:9230 -sTCP:LISTEN -nP 2>/dev/null\n```\n\n---\n\n## 🚨 The tsx + `node inspect` CLI silent-failure (READ THIS)\n\n`tsx` transpiles each `.ts` file on the fly and emits an inline source map. V8 Inspector registers the module with its `.ts` path (so it shows up in the debugger's `scripts` list), **but the `node inspect` CLI REPL does not resolve source-map line numbers reliably**. Setting `sb('session.ts', 285)` will show a \"pending\" breakpoint that **never fires even after the module loads**.\n\nThe breakpoint list will happily display it, so you think it's set. It isn't.\n\n### Three reliable workarounds\n\n| Workaround | When to use | Downside |\n|---|---|---|\n| **`debugger;` statement in source** | You can edit the source, CLI required | Requires source edit + revert |\n| **Chrome DevTools GUI** (`chrome://inspect`) | CLI not required, faster iteration | Not usable if user specifically asked for CLI |\n| **Debug the built `dist/` JS** | Source maps are working end-to-end | Requires `npm run build` on every source change |\n\nThe `debugger;` statement is the most reliable. Journal the edit — revert at Phase 9.\n\n---\n\n## Launch recipes by runtime\n\n### Node (plain JS / compiled TS)\n\n```bash\n# Break on first line, wait for debugger to attach\nnode --inspect-brk=9229 dist/index.js\n\n# Attach immediately, don't block startup — pair with debugger; statements\nnode --inspect=9229 dist/index.js\n\n# Wait for debugger to attach, THEN run (new in Node 20.15+)\nnode --inspect-wait=9229 dist/index.js\n\n# Source maps in stack traces (always a good idea in debug builds)\nnode --enable-source-maps --inspect dist/index.js\n```\n\n### tsx\n\n```bash\n# The tsx runner is --import-compatible, so these work:\nnode --inspect-brk=9229 --import tsx index.ts\nnode --inspect=9229 --import tsx index.ts\n\n# If user prefers invoking tsx directly, this also works but is less explicit:\nNODE_OPTIONS='--inspect-brk=9229' npx tsx index.ts\n\n# ⚠️ tsx watch + inspector = inspector reloads per file change\n# Debug without watch:\nnode --inspect=9229 --import tsx index.ts # (no `watch`)\n```\n\n### ts-node (legacy but still encountered)\n\n```bash\nnode --inspect-brk -r ts-node/register src/index.ts\n# ESM (ts-node's ESM loader is fragile — if possible, migrate to tsx):\nnode --inspect --loader ts-node/esm src/index.ts\n```\n\n### Bun (WebKit Inspector Protocol, NOT V8)\n\n```bash\nbun --inspect src/index.ts # opens debug.bun.sh URL\nbun --inspect-brk src/index.ts # break on start\nbun --inspect-wait src/index.ts # wait for attach\nbun test --inspect-brk # debug test runner\n```\n\n**Critical**: Bun uses WebKit Inspector Protocol, not V8. `chrome://inspect` cannot connect directly. Use `debug.bun.sh` or the (currently buggy, per Bun docs) VS Code extension.\n\n### Deno (native V8, Chrome DevTools / VS Code compatible)\n\n```bash\ndeno run --inspect-brk --allow-all src/main.ts\ndeno test --inspect-brk --filter \"auth\"\n```\n\nDeno is the smoothest TS debugging experience — native V8 inspector, no source-map workarounds.\n\n### Vitest\n\n```bash\n# Single worker required — inspector can't attach to multiple workers\nvitest --inspect-brk --no-file-parallelism\nvitest --inspect-brk --browser --no-file-parallelism # browser mode\n```\n\nWithout `--no-file-parallelism`, breakpoints won't fire because the process Vitest spawns workers in isn't the one listening on the inspector port.\n\n---\n\n## Attaching with `node inspect` CLI\n\n```bash\nnode inspect 127.0.0.1:9229 # attach to an existing --inspect process\n```\n\nCore commands at the `debug>` prompt:\n\n```\ncont, c resume until next break / debugger;\nnext, n step over\nstep, s step into\nout, o step out\npause pause a running process\nbt backtrace\nscripts list all modules V8 has loaded (incl. tsx-transpiled .ts)\nsb(N) set breakpoint at line N of current file\nsb('file', N) set breakpoint at line N of matching file (⚠️ unreliable with tsx)\nsb(func) set breakpoint at function reference\ncb(N), cb('file', N) clear breakpoint\nbreakpoints list breakpoints (shows pending ones, doesn't tell you they'll never fire)\nwatch('expr') persistent watch expression\nwatchers show watchers\nexec('expr') evaluate expression in paused frame's scope\nrepl drop into full REPL with frame's scope\nrestart restart the debuggee\nkill kill the debuggee\n```\n\n**`exec('expr')` is the most powerful tool in this CLI** — it evaluates any JS in the paused frame and returns the value. Use it heavily.\n\n---\n\n## `exec()` patterns that resolve hypotheses fast\n\nAt a breakpoint, these queries resolve most LLM / agent / async bugs in one line each:\n\n```js\n// Agent / LLM state\nexec('this.agent.state.messages.length')\nexec('this.agent.state.messages.map(m => m.role)')\nexec('JSON.stringify(this.agent.state.messages.at(-1)).substring(0, 500)')\nexec('this.agent.state.messages.at(-1).errorMessage') // silent-error sentinel\nexec('this.agent.state.messages.at(-1).stopReason')\nexec('JSON.stringify(this.agent.state.usage)') // undefined / all-zero = failed call\nexec('this.agent.state.model.baseUrl') // catch hardcoded vs env-var\n\n// Env / config at runtime\nexec('process.env.RELEVANT_VAR')\nexec('Object.keys(process.env).filter(k => k.startsWith(\"ANTHROPIC\"))')\nexec('this.config')\n\n// Async / timing\nexec('Date.now() - this._turnStartedAt')\nexec('this._activePromises?.size')\n\n// HTTP request/response in-flight\nexec('JSON.stringify(req.body).length')\nexec('res.statusCode')\nexec('res.headersSent')\n\n// What's actually running\nexec('process.version')\nexec('process.cwd()')\nexec('process.argv')\n```\n\n---\n\n## Silent-failure patterns in Node\n\nThese are the patterns that most commonly look like success but aren't. Always check when a response is \"too fast\" or \"too empty\":\n\n| Signal | What it means |\n|---|---|\n| HTTP 200 + `content: \"\"` | Silent error swallowed |\n| HTTP 200 + response in \u003c1s for an LLM call | Too fast for a real Claude/GPT call; something short-circuited |\n| `usage: { totalTokens: 0 }` | LLM SDK returned a stub without making the call |\n| `stopReason: \"error\" + content: []` | SDK packaged an error into a \"success\" message |\n| Unhandled promise rejection with no log | Caller forgot to `await`, or `.catch(() => {})` |\n| `try { await x(); } catch {}` | Error eaten, no log |\n| `void somePromise()` | Explicit opt-out of error propagation; often a bug |\n| Callback-style API where callback never fires | Error happened before callback scheduled |\n| Handler returns `res.json(...)` twice | Second call is silent on some Express versions |\n\nWhen you find one, add a temporary `console.error('[DEBUG]', ...)` to make it loud — journal it, revert at Phase 9.\n\n---\n\n## tmux session layout (two sessions, one purpose each)\n\n```bash\n# Long-running inspected process\ntmux new-session -d -s debug-server -c \"$PWD\"\ntmux send-keys -t debug-server 'node --inspect=9229 --import tsx index.ts' Enter\n\n# Interactive debugger client (separate pane for readability)\ntmux new-session -d -s debug-client -c \"$PWD\"\ntmux send-keys -t debug-client 'node inspect 127.0.0.1:9229' Enter\n\n# Non-blocking pane inspection from the outside\ntmux capture-pane -p -t debug-server -S -50\n```\n\nJournal both session names. Kill both at Phase 9:\n\n```bash\ntmux kill-session -t debug-server\ntmux kill-session -t debug-client\n```\n\n---\n\n## When to abandon the CLI and switch to Chrome DevTools\n\nThe user's preference for CLI is valid and should be respected. But you may recommend a switch in one short sentence if ANY of these hold:\n\n- You hit source-map resolution failures (`sb('file', line)` not firing) AND the fix is time-sensitive\n- You need to watch many values simultaneously (GUI watch panel is faster to scan)\n- You're stepping through async-heavy code where CLI step semantics get murky across microtask boundaries\n\nPhrase as a note, not a request: \"I can push through with `debugger;` statements in CLI. If we hit three or more of these in a row, switching to `chrome://inspect` GUI would cut cycle time in half — your call.\"\n\n---\n\n## Phase 9 cleanup specifics\n\n```bash\n# Revert source-level debug statements\ngit diff | grep -E '(debugger;|console\\.log\\(.*DEBUG|\\[ARBITER-DEBUG|\\[DEBUG)'\n# Revert any matching files:\ngit checkout \u003cfile>\n\n# Kill inspector-attached processes\npkill -f 'node --inspect' || true\npkill -f 'bun --inspect' || true\npkill -f 'deno.*--inspect' || true\nlsof -iTCP:9229 -sTCP:LISTEN -nP 2>/dev/null\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":9650,"content_sha256":"0b728b643ed89c318bb564e1137cf68bb6acb61c1774ed73dee04b5a70cf1058"},{"filename":"references/runtimes/python.md","content":"# Python Debugging\n\nCovers CPython 3.9+, pytest, asyncio, Django, FastAPI. Setup commands, attach mechanisms, state-query patterns, gotchas, silent-failure signatures.\n\n---\n\n## Environment detection (Phase 0)\n\n```bash\n# Which Python will actually run the code?\nwhich python; which python3\npython --version\n\n# Is there a project env manager in play?\nls poetry.lock uv.lock Pipfile.lock requirements*.txt .python-version 2>/dev/null\n\n# Installed debuggers / profilers in this env?\npython -c 'import pdb, sys; print(\"pdb\", \"built-in\"); print(\"python\", sys.executable)'\npip list 2>/dev/null | grep -iE '^(ipdb|pudb|debugpy|py-spy|memray|rich)\\s'\n\n# asyncio debug mode available?\npython -c 'import asyncio; print(asyncio.__version__)'\n```\n\n**Wrapper gotchas** (these change how flags propagate):\n\n- `poetry run python ...` — args after `python` are fine; args before `poetry run` go to poetry, not python\n- `uv run python ...` — similar; prefer `uv run -- python -X dev` if flags collide\n- `pipenv run` — same story\n- `./manage.py \u003ccmd>` (Django) — shebang resolution; make sure it points to the right venv\n- `pytest` — loads `conftest.py` at collection; breakpoints inside collection need `pytest --pdb-trace` not `--pdb`\n\n---\n\n## The four ways to attach\n\n| Method | When to use | Command |\n|---|---|---|\n| **`breakpoint()` inline** (Python 3.7+) | You can edit the source and restart. Most reliable. | Add `breakpoint()` to source. Run normally. It invokes `pdb` by default. |\n| **`python -m pdb \u003cscript>`** | No source edit desired. Breaks on entry. | `python -m pdb script.py arg1` |\n| **post-mortem `pdb.pm()`** | Exception already happened, you want to inspect state | In an exception-caught REPL: `import pdb; pdb.pm()` after the exception propagates |\n| **debugpy (remote / IDE)** | IDE attach, remote host, containerized process | `python -m debugpy --listen 5678 --wait-for-client script.py` then attach from VS Code / PyCharm |\n\n### Prefer `ipdb` or `pudb` over plain `pdb` when available\n\n- **ipdb** — drop-in replacement with tab completion, syntax highlighting. `pip install ipdb`, then `PYTHONBREAKPOINT=ipdb.set_trace` or use `import ipdb; ipdb.set_trace()`.\n- **pudb** — full-screen TUI debugger, much faster to navigate stack/locals. `pip install pudb`, then `PYTHONBREAKPOINT=pudb.set_trace`.\n\n### Control `breakpoint()` globally\n\n```bash\n# Use ipdb instead of pdb\nexport PYTHONBREAKPOINT=ipdb.set_trace\n\n# Disable all breakpoint() calls (useful to ship without removing them)\nexport PYTHONBREAKPOINT=0\n```\n\n**Journal this env var** — unset at Phase 9.\n\n---\n\n## pdb / ipdb essentials\n\nAt a `(Pdb)` or `ipdb>` prompt:\n\n```\nl list source around current line\nll list whole function\ns step into\nn step over (next)\nr step out (return)\nc continue\nb list breakpoints\nb \u003cline> breakpoint at line\nb \u003cfunc> breakpoint at function\ncl \u003cn> clear breakpoint n\nw where (backtrace)\nu / d move up/down the stack\na args of current frame\np \u003cexpr> print expression\npp \u003cexpr> pretty-print\n!\u003cstmt> execute Python statement (e.g. !x = 5)\ninteract drop into a full Python REPL with current frame's locals\nq quit (aborts the program)\n```\n\n**`interact` is underused** — it gives you a full IPython-esque REPL with all locals available. Faster than typing `p` for 20 things.\n\n---\n\n## pytest-specific debugging\n\n```bash\n# Enter pdb on first failure\npytest --pdb\n\n# Enter pdb at the START of each test (not on failure)\npytest --trace\n\n# Run only the failing test, with -s to show print output\npytest --pdb -x -s path/to/test.py::test_name\n\n# Collect-time debugging (for problems in conftest.py / fixture setup)\npytest --pdb-trace\n\n# Disable capture for this test (so breakpoint prompt is visible)\npytest -s\n```\n\n**Common failure**: `breakpoint()` hangs inside a pytest test — that's because pytest captures stdout/stderr by default. Always add `-s` when debugging with breakpoints inside pytest.\n\n---\n\n## asyncio gotchas\n\nAsync is where most Python debug sessions go sideways. Know these before attaching.\n\n### Breakpoints inside coroutines\n\n`breakpoint()` works inside an async function, but stepping into another coroutine from `pdb` is awkward. Two techniques:\n\n```python\nasync def handler():\n result = await some_async_fn() # add breakpoint ABOVE, not inside, when possible\n breakpoint()\n return result\n```\n\nInside the breakpoint, to inspect a coroutine without actually advancing time:\n```\n!import asyncio\n!loop = asyncio.get_event_loop()\n!task = asyncio.ensure_future(some_async_fn())\n# Now inspect task state, don't await it\np task\n```\n\n### PYTHONASYNCIODEBUG\n\nEnable before running the process:\n\n```bash\nPYTHONASYNCIODEBUG=1 python script.py\n```\n\nSurfaces: coroutines that were never awaited, slow callbacks, unhandled task exceptions. **Always turn this on** if the bug is timing- or async-related.\n\n### `asyncio.gather` swallows the first exception\n\nBy default, `asyncio.gather(t1, t2)` raises the first exception and cancels the rest. If you need all exceptions, use `gather(..., return_exceptions=True)`.\n\n### Unhandled task exceptions are silent\n\n```python\nasync def main():\n task = asyncio.create_task(broken_coroutine())\n # If task raises and we never await it, the exception is eaten at gc time\n await asyncio.sleep(10)\n```\n\nTo catch these, set `loop.set_exception_handler(...)` or upgrade to Python 3.12+ which warns louder by default.\n\n---\n\n## debugpy — remote / IDE / container attach\n\nListen and wait for attach:\n\n```bash\npython -m debugpy --listen 0.0.0.0:5678 --wait-for-client script.py\n```\n\nAttach from VS Code:\n```json\n// .vscode/launch.json\n{\n \"name\": \"attach\",\n \"type\": \"python\",\n \"request\": \"attach\",\n \"connect\": { \"host\": \"localhost\", \"port\": 5678 }\n}\n```\n\nInside the code, programmatic attach point:\n```python\nimport debugpy\ndebugpy.listen(5678)\ndebugpy.wait_for_client() # blocks until attached\ndebugpy.breakpoint() # programmatic breakpoint\n```\n\n**Journal**: the port (5678) and the listener file — unset at Phase 9.\n\n---\n\n## Sampling profilers for \"why is it slow / stuck\"\n\nWhen the problem is performance or a hang (not a crash), don't attach pdb — it alters timing. Use a sampling profiler that attaches to the running process:\n\n```bash\n# py-spy — production-safe, zero code change, works on running process\npy-spy top --pid \u003cpid> # live top-like view\npy-spy record -o profile.svg --pid \u003cpid> # flamegraph\npy-spy dump --pid \u003cpid> # stack traces of all threads right now\n\n# memray — memory allocation tracking\nmemray run script.py\nmemray flamegraph output.bin\nmemray stats output.bin\n```\n\n`py-spy dump` on a stuck process is often enough to find the hung call — no breakpoints needed.\n\n---\n\n## Silent-failure patterns in Python\n\nAdd these to Phase 8's silent-failure check:\n\n| Pattern | Why it's silent |\n|---|---|\n| `except Exception: pass` or `except: pass` | Catches and discards every error including KeyboardInterrupt |\n| `logging.exception(...)` in a logger with no handlers | \"Logs\" but actually writes nowhere |\n| `asyncio.create_task(coro)` without storing the task | Task GC'd before completion, exception swallowed |\n| `return x.get(\"key\")` where key is missing | Returns None silently, caller often doesn't check |\n| `subprocess.run(..., check=False)` with ignored returncode | Non-zero exit treated as success |\n| Django `transaction.atomic()` inside a broader `except` | Rolls back silently |\n| `contextlib.suppress(Exception)` | Explicit silencer; easy to leave wider than intended |\n| `queue.get(block=False)` with `except queue.Empty: pass` | Polling that silently drops the work |\n\n---\n\n## Phase 9 cleanup specifics\n\n```bash\n# Remove breakpoint() / ipdb / pudb lines from source\ngit diff | grep -E '(breakpoint\$\$|import ipdb|import pudb|import pdb; pdb\\.set_trace)'\n# If the above has output, revert those files:\ngit checkout \u003cfile>\n\n# Unset the global breakpoint override\nunset PYTHONBREAKPOINT\n\n# Kill any leftover debugpy listeners\npkill -f 'debugpy' || true\nlsof -iTCP:5678 -sTCP:LISTEN -nP 2>/dev/null # confirm free\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":8206,"content_sha256":"7c69b34e0d1342900c566468828e8af5a688669b75803a5eb8f2f6e156d38490"},{"filename":"references/runtimes/rust.md","content":"# Rust Debugging\n\nCovers `cargo`, `tokio`, panics, and the fact that you usually don't actually need a debugger — Rust's type system, `dbg!`, and logging cover 80% of sessions faster than gdb would.\n\n---\n\n## Environment detection (Phase 0)\n\n```bash\ncargo --version\nrustc --version\ncat rust-toolchain.toml rust-toolchain 2>/dev/null\ncat Cargo.toml | head -30\n\n# Debuggers available\nwhich rust-gdb\nwhich rust-lldb\nwhich lldb\n\n# Async infrastructure\ngrep -E '\"(tokio|async-std|smol)\"' Cargo.toml\n\n# Profile flags\ngrep -E '^\\[profile' Cargo.toml\n```\n\n**The default `cargo run` builds with `dev` profile** which includes debug symbols. `cargo run --release` strips them. For debugging, stay in dev unless the bug only manifests under optimization.\n\n---\n\n## The Rust debugging hierarchy (use in this order)\n\nRust's ecosystem has a specific order that's faster than reaching for gdb first:\n\n1. **`dbg!(expr)` macro** — for a single value at a specific spot. Prints file:line + value, returns the value unchanged so you can inline it. Faster than a debugger for 60% of bugs.\n2. **`RUST_LOG=trace` with `tracing` / `env_logger`** — for flow and state across an operation. Zero code change in dev-time.\n3. **`RUST_BACKTRACE=1` / `=full`** — for crashes. Almost always sufficient; you rarely need a live debugger for a panic.\n4. **`rust-gdb` / `rust-lldb`** — when you need to pause execution and inspect memory, especially for unsafe code or FFI.\n5. **`tokio-console`** — for async deadlocks, stuck tasks, hot loops.\n6. **`cargo-expand`** — when a macro is doing something weird.\n\nReach for the lightest tool that answers the hypothesis.\n\n---\n\n## `dbg!` — the underused macro\n\n```rust\nlet x = 5;\nlet y = dbg!(x * 2); // prints: [src/main.rs:2] x * 2 = 10\n```\n\nInside a complex expression:\n```rust\nlet total = items.iter().filter(|i| i.active).map(|i| dbg!(i.cost)).sum::\u003cu64>();\n```\n\nMultiple values at once:\n```rust\ndbg!(&user, &request, elapsed.as_millis());\n```\n\n`dbg!` writes to stderr, so it won't corrupt stdout-based pipelines. **Journal each `dbg!` you add**; revert at Phase 9.\n\n---\n\n## `RUST_LOG` for flow-level debugging\n\nIf the codebase uses `tracing` or `env_logger`:\n\n```bash\nRUST_LOG=debug cargo run\nRUST_LOG=trace cargo run # very verbose\nRUST_LOG=my_crate=trace,hyper=info cargo run # per-module level\nRUST_LOG=debug,tokio=off cargo run # silence noisy crates\n```\n\nFor `tracing`-based apps, instrument with spans:\n```rust\n#[tracing::instrument]\nfn handle_request(req: &Request) -> Response { ... }\n```\n\nThis gives you structured per-call entry/exit logs with args and timing, zero additional code in the body.\n\n---\n\n## `RUST_BACKTRACE` for panics\n\n```bash\nRUST_BACKTRACE=1 cargo run # backtrace on panic\nRUST_BACKTRACE=full cargo run # include libstd/tokio frames\n```\n\nThe panic itself usually tells you the file:line. The backtrace tells you how it got there. Between the two, most crash bugs are solved without a debugger.\n\n---\n\n## rust-gdb / rust-lldb\n\n### Launch\n\n```bash\n# Build with debug symbols (default dev profile)\ncargo build\n\n# Attach gdb wrapper (applies Rust type pretty-printers)\nrust-gdb ./target/debug/my_binary\n# Or lldb:\nrust-lldb ./target/debug/my_binary\n\n# With args\nrust-gdb --args ./target/debug/my_binary arg1 arg2\n\n# Attach to running process\nrust-gdb -p $(pgrep my_binary)\n```\n\n### Breakpoints\n\nRust symbols are mangled. Use either:\n\n```\n(gdb) b main # main function\n(gdb) b my_crate::module::function # canonical path\n(gdb) b src/handler.rs:42 # file:line\n(gdb) info functions my_function # find mangled name\n```\n\n### State inspection\n\n```\n(gdb) p x # print value (uses Rust pretty-printer for Vec, Option, HashMap, etc.)\n(gdb) p *ptr # deref\n(gdb) info locals # all locals in current frame\n(gdb) info args # function args\n(gdb) bt # backtrace\n(gdb) frame \u003cn> # switch to stack frame n\n(gdb) watch my_var # stop when my_var changes\n(gdb) rbreak regex # breakpoint all functions matching regex\n```\n\n**Pair with pwndbg** for better layout on native bugs — see [tools/pwndbg.md](../tools/pwndbg.md). Pwndbg works with rust-gdb too.\n\n---\n\n## tokio-console — async task debugging\n\nFor tokio-based async apps, this is essential when tasks are stuck or leaking.\n\n```bash\n# Add tokio-console instrumentation to the target binary\n# In Cargo.toml:\n# [dependencies]\n# console-subscriber = \"0.2\"\n\n# In main.rs:\n# console_subscriber::init();\n\n# Build with tokio_unstable:\nRUSTFLAGS=\"--cfg tokio_unstable\" cargo run\n\n# In another terminal:\ntokio-console # connects to default port 6669\n```\n\nShows live tasks, their state, wake counts, poll durations, parent tasks. The single fastest way to find \"why is my async thing stuck\".\n\n---\n\n## cargo-expand — when a macro is suspect\n\n```bash\ncargo install cargo-expand\ncargo expand # expand all macros in the crate\ncargo expand my::module::path # scope to one item\n```\n\nIf you suspect a macro (especially `#[derive]`, `#[tokio::main]`, `#[async_trait]`) is generating code that doesn't match your mental model, this shows you exactly what the compiler sees.\n\n---\n\n## Release-build gotcha\n\n```bash\ncargo build --release # no debug symbols by default\n```\n\nIf the bug only shows up in `--release`:\n\n```toml\n# Cargo.toml\n[profile.release]\ndebug = true # add symbols, keep optimizations\n```\n\nNow `rust-gdb ./target/release/my_binary` works on release builds. This is required when optimization-enabled codegen bugs (inlining, LLVM folding) are suspected.\n\n---\n\n## Silent-failure patterns in Rust\n\n| Pattern | Why it's silent |\n|---|---|\n| `.unwrap_or_default()` | Masks errors as the zero value |\n| `.unwrap_or(fallback)` | Same, with a specific fallback |\n| `let _ = fallible_operation()` | Explicitly discards the Result, no compiler warning |\n| `if let Ok(x) = ... { use(x); } // no else` | Silent on Err |\n| `.ok()` chaining | Converts Result to Option, throwing the error away |\n| Panic inside a tokio task not `.await`ed | Task dies silently; runtime usually logs but it's quiet if logs are off |\n| `eprintln!` that goes to a redirected-null stderr | Looks like nothing happened |\n| `Drop` impl that panics under specific condition | Double-panic aborts process silently if no logging configured |\n\n---\n\n## Phase 9 cleanup specifics\n\n```bash\n# Revert dbg! macro additions\ngit diff | grep -E 'dbg!\\('\n# For any file with dbg! additions:\ngit checkout \u003cfile>\n\n# Unset env vars\nunset RUST_LOG RUST_BACKTRACE\n\n# Kill any rust-gdb/lldb sessions\npkill -f 'rust-gdb' || true\npkill -f 'rust-lldb' || true\npkill -f '^lldb ' || true\n\n# tokio-console binds 6669 by default\nlsof -iTCP:6669 -sTCP:LISTEN -nP 2>/dev/null\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":6978,"content_sha256":"48fa55949ba60c42a1e9827af94122905ea82cb147262b1e044a27734b6a2958"},{"filename":"references/tools/ghidra.md","content":"# Ghidra — Decompile Binaries Into Readable C\n\n**https://github.com/NationalSecurityAgency/ghidra**\n\nGhidra is the NSA's open-source reverse-engineering suite. Its defining feature is a **decompiler** that turns machine code back into readable C. For any binary you don't have source for, this is the correct starting point — not `strings`, not hex-staring, not `objdump -d`.\n\n**Use Ghidra when**: third-party closed-source libs, malware analysis, vendored binaries whose behavior contradicts docs, CTF challenges, firmware, any time you need to read compiled code.\n\n---\n\n## Install\n\n```bash\n# macOS\nbrew install --cask ghidra\n# OR download the release ZIP from the repo and ./ghidraRun\n\n# Linux\n# Download from https://github.com/NationalSecurityAgency/ghidra/releases\n# Requires JDK 21+\n./ghidraRun\n\n# Dependency\njava -version # must be 21+\n```\n\nGhidra is a Java Swing app. Looks dated, works well.\n\n---\n\n## First-time workflow (memorize this — it's not obvious)\n\n1. **Start Ghidra**: `ghidraRun`\n2. **Create a project**: File → New Project → Non-Shared → name it `debug-\u003cbinary-name>` (journal this path so you can rm it at Phase 9 if disposable).\n3. **Import the binary**: File → Import File → pick your target. Accept default format detection.\n4. **Double-click the imported binary** in the project listing. Ghidra asks to analyze it — say **yes**, accept defaults for the first pass. This takes anywhere from seconds (small binary) to tens of minutes (large binary).\n5. **Once analysis completes**, you're in the CodeBrowser view.\n\nTwo panels you'll use 95% of the time:\n\n- **Listing** (middle) — the disassembly with Ghidra's inferred labels/types.\n- **Decompiler** (right) — the reconstructed C. The real value.\n\n---\n\n## Finding the right function fast\n\nDon't try to read the whole binary. Use these to narrow:\n\n### Symbol Tree (left panel)\n\n- `Functions` — all detected functions. Stripped binaries show `FUN_00401234` (address-named); unstripped show actual names.\n- `Imports` — dynamically-linked functions. Great for \"does this binary call `system()`, `strcpy`, `curl_easy_perform`?\"\n- `Exports` — if it's a library.\n\nClick to jump. The Decompiler updates instantly.\n\n### String search\n\n```\nSearch → For Strings\n```\n\nProduces a list of all strings. Right-click a string → `References` → `Show References to Address`. Jumps to code that references it. **This is how you find which function handles the error message you saw at runtime.**\n\n### Memory search for bytes\n\n```\nSearch → Memory\n```\n\nSearch hex or text. Useful for known magic bytes, file-format signatures, constants.\n\n### Cross-references (XREF)\n\nRight-click any function / address → `References → Find References to`. Shows every place that calls it. Walk the call graph backward from interesting functions.\n\n---\n\n## Making the decompiler's output readable\n\nGhidra's decompiler is good but needs hints. These three actions dramatically improve its output:\n\n### 1. Rename variables\n\nClick a variable in the Decompiler view → press `L` → type a better name. Ghidra propagates the rename across all uses.\n\n### 2. Set types\n\nA variable that looks like `undefined4` or `void *` is unhelpful. Click it → press `Ctrl+L` → set type (e.g. `int`, `char *`, `struct my_header *`).\n\nFor pointers to structs from headers you have, use:\n```\nFile → Parse C Source → paste header file → auto-creates struct types\n```\n\nThen assign the struct type to the pointer. Ghidra resolves field accesses immediately.\n\n### 3. Retype function signatures\n\nClick the function name in the Decompiler → press `F` (Edit Function Signature) → set return type + argument types. This propagates through callers.\n\nDo these three actions on the 3-5 most relevant functions and the decompiler output becomes near-source-readable.\n\n---\n\n## Patterns for specific bug types\n\n### Looking for integer overflow / buffer overflow\n\n```\nListing: look for\n - LEA → CMP patterns on sizes\n - memcpy/strcpy/sprintf with non-constant sizes\nDecompiler: look for\n - arithmetic on size_t without bounds check\n - `+ user_input` in a length calculation\n```\n\n### Looking for a missing auth check\n\nNavigate from the handler entry (found via Strings or Imports) and check the control flow:\n\n```\nDecompiler: does the function return early / jump to error handler when some flag is not set?\n If the check is absent, that's the bug.\n```\n\n### Looking for hardcoded URLs / keys / paths\n\n```\nSearch → For Strings → filter `http:` / `https:` / `/etc/` / `bearer ` / `api_key`\n```\n\n### Looking for dispatch / plugin loading\n\n```\nImports → dlopen, LoadLibrary, dlsym, GetProcAddress\n```\n\nThe strings referenced near those calls are often plugin names.\n\n---\n\n## Scripting (headless Ghidra)\n\nWhen you need to automate analysis across many binaries, or repeat a workflow:\n\n```bash\n# Headless analyzer\n$GHIDRA_INSTALL_DIR/support/analyzeHeadless \\\n \u003cproject-dir> \u003cproject-name> \\\n -import \u003cbinary> \\\n -postScript \u003cscript.py or script.java>\n```\n\nGhidra supports Python 3 scripts (via Jython-compatible API) and Java. Useful scripts:\n\n- Dump all function signatures to JSON\n- Find all calls to `system()` with constant arguments\n- Auto-rename FUN_xxx based on heuristics (string refs, call patterns)\n\nThe community has a large collection: https://github.com/NationalSecurityAgency/ghidra/tree/master/Ghidra/Features/Base/ghidra_scripts\n\n---\n\n## Bookmarks + Notes\n\nGhidra has built-in bookmarks and comments. Use them as your journal inside the project:\n\n- Right-click an address → `Set Bookmark` → tag as `Note`. Attach a description.\n- Right-click → `Comments → Set EOL Comment` (shows up inline in decompiler).\n\nTreat these as part of the journal. If you end up promoting this Ghidra project (keeping it after the debug session), the comments become durable documentation.\n\n---\n\n## Gotchas\n\n- **Large binaries need more Java heap.** Edit `support/launch.properties` and bump `VMARGS=-Xmx8G` (default is often 2G, too small).\n- **Save often.** Ghidra's autosave is not instant; a crash loses uncommitted analysis.\n- **Decompiler has timeouts.** For complex functions it may give up and print `/* WARNING: ... */`. Increase timeout in `Edit → Tool Options → Decompiler`.\n- **Archs beyond x86/ARM/MIPS** sometimes need Sleigh processor module tweaks. Rare but possible.\n- **Signed vs unsigned decompilation** is frequently wrong. Manually retype when integer behavior matters.\n\n---\n\n## When Ghidra is NOT the right tool\n\n- You have the source. Go read it.\n- Bug is in your own recently-compiled binary. Rebuild with `-g` and use gdb/pwndbg (see [pwndbg.md](pwndbg.md)).\n- You just need to know which libs a binary links. `ldd` / `otool -L` / `readelf -d` are faster.\n- You just need strings. `strings -n 8` is faster.\n\nGhidra is the right tool when you need to **read the logic** of a binary you don't have source for.\n\n---\n\n## Phase 9 cleanup specifics\n\n```bash\n# If you created a scratch project just for this debug session, journal path and remove:\n# (the journal should have the exact path)\n# Example:\nls ~/ghidra-projects/ 2>/dev/null\n# rm -rf ~/ghidra-projects/debug-\u003cbinary-name>\n\n# If you promoted the project (kept it), it's not cleanup — note it in the final summary to the user\n\n# Kill any running Ghidra headless processes\npkill -f 'analyzeHeadless' || true\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":7360,"content_sha256":"8d50534340eef3cd664458cade9d11d18cabff804979128d7107a5fdd36dbbf0"},{"filename":"references/tools/playwright-cli.md","content":"# Playwright CLI — Browser QA That Actually Drives a Browser\n\n**https://playwright.dev/ · https://github.com/microsoft/playwright**\n\nFor any browser-served web UI bug, this is the correct tool. Not curl. Not imagination. Not a headless HTTP library. A real browser with a real rendering engine, real JS execution, real cookies, real service workers, real viewport.\n\n**In Phase 8 Manual QA for browser products, using Playwright is not optional.** Curl cannot catch: CSS that breaks at specific viewport widths, hydration mismatches, client-side router bugs, cookie/session interactions, service-worker caching, JS-triggered navigations. All of those are common bug classes. Drive a browser.\n\n> Note: `microsoft/playwright-cli` is the legacy repo; the current tooling lives in `@playwright/test` (npm) and `playwright` (pip), which include the `playwright` CLI. Use those — the legacy `playwright-cli` package is deprecated.\n\n---\n\n## When to reach for Playwright\n\n| Bug symptom | Use Playwright? |\n|---|---|\n| Form submit produces wrong result | ✅ — Playwright drives the form exactly as a user does |\n| Page blank in prod, fine locally | ✅ — hydration/env differences need a real browser |\n| CSS looks wrong at a specific width | ✅ — use `--viewport-size` |\n| Click doesn't fire / wrong handler | ✅ — Playwright fires real DOM events |\n| Flash of unstyled content / loading glitch | ✅ — use trace viewer to see frames |\n| API returns wrong data | ❌ — use curl, this isn't a browser bug |\n| Backend returns wrong status code | ❌ — use curl |\n| Client hits a URL that returns 500 | ✅ but also ❌ — Playwright shows the call + response + failure effect on UI |\n\n---\n\n## Install (per-project)\n\nPlaywright installs browser binaries separately from the npm package.\n\n```bash\n# In the project\nnpm init playwright@latest # interactive; picks TS/JS + browsers + config\n# Or if Playwright is already a dep:\nnpx playwright install # downloads browsers\nnpx playwright install chromium # just chromium\nnpx playwright install --with-deps # also installs OS deps (Linux)\n```\n\nPython:\n```bash\npip install playwright\nplaywright install\n```\n\n---\n\n## The four things you'll actually use\n\n### 1. `codegen` — record a session, generate the script\n\nThe fastest way to create a repro. Opens a real browser; your clicks / typing become a Playwright script you can paste into a test.\n\n```bash\nnpx playwright codegen https://your-app.local\nnpx playwright codegen --viewport-size=375,667 https://your-app.local # iPhone SE size\nnpx playwright codegen --device=\"iPhone 14\" https://your-app.local\n```\n\nClick / type / navigate in the browser; watch the script build in the side panel. Copy the generated script into your journal as the repro for Phase 8.\n\n### 2. A one-shot Playwright script — reproduce + capture\n\nUsually the Phase 8 QA artifact. Save to `/tmp/debug-repro.spec.ts` (journal it):\n\n```ts\n// /tmp/debug-repro.spec.ts\nimport { test, expect } from '@playwright/test';\n\ntest('refinement chat shows non-empty response when env var set', async ({ page }) => {\n await page.goto('http://localhost:3000/chat');\n await page.fill('textarea[name=\"message\"]', 'Add a logging step');\n await page.click('button[type=submit]');\n\n // Wait for the response to appear (not just the spinner to disappear)\n const response = page.locator('[data-testid=\"assistant-reply\"]');\n await expect(response).toBeVisible({ timeout: 30_000 });\n await expect(response).not.toBeEmpty();\n\n // Capture evidence\n await page.screenshot({ path: '/tmp/debug-after-fix.png', fullPage: true });\n console.log(await response.textContent());\n});\n```\n\nRun it with tracing enabled for rich post-mortem:\n\n```bash\nnpx playwright test /tmp/debug-repro.spec.ts --trace on --headed\n```\n\n### 3. `PWDEBUG=1` — step through the script with Playwright Inspector\n\n```bash\nPWDEBUG=1 npx playwright test /tmp/debug-repro.spec.ts\n```\n\nOpens the Playwright Inspector alongside the browser. You can step through Playwright actions, see the DOM state at each step, and edit selectors on the fly.\n\nUse this when the script doesn't reproduce cleanly and you need to watch it run.\n\n### 4. `show-trace` — post-mortem on a failed run\n\n```bash\nnpx playwright show-trace trace.zip\n# or from the test-results dir:\nnpx playwright show-trace test-results/\u003ctest-name>/trace.zip\n```\n\nScrubs through a recorded session: timeline, DOM snapshot at each action, network, console, source. When a test failed on CI but passed locally, this is the single best artifact.\n\n---\n\n## Headless vs headed during debugging\n\nAlways add `--headed` when debugging. Headless browsers sometimes behave subtly differently (font rendering, viewport, media permissions). For QA evidence, run headed and screenshot.\n\n```bash\nnpx playwright test --headed\nnpx playwright test --headed --project=chromium # pin the browser\n```\n\n---\n\n## Catching the silent-failure patterns Playwright is good at\n\n```ts\n// Toast that flashes and disappears\npage.on('console', msg => console.log('[browser console]', msg.type(), msg.text()));\n\n// Unhandled page errors (uncaught exceptions in the page JS)\npage.on('pageerror', err => console.error('[page error]', err));\n\n// Network failures — e.g., backend returned 500 but UI shows nothing\npage.on('response', async resp => {\n if (!resp.ok()) {\n console.warn(`[network ${resp.status()}] ${resp.url()} — ${await resp.text()}`);\n }\n});\n\n// Request that never came back\npage.on('requestfailed', req => {\n console.error('[request failed]', req.url(), req.failure()?.errorText);\n});\n```\n\nAdd these listeners to the top of the debug script. They surface a lot of the \"UI showed nothing\" class of bug.\n\n---\n\n## Viewport and device emulation\n\nCSS bugs that only appear at specific sizes, or layout bugs on mobile:\n\n```ts\n// At test level\ntest.use({ viewport: { width: 375, height: 667 } });\n\n// Per-page\nawait page.setViewportSize({ width: 375, height: 667 });\n\n// Predefined devices\nimport { devices } from '@playwright/test';\ntest.use({ ...devices['iPhone 14'] });\n```\n\n---\n\n## Gotchas\n\n- **Wait for state, not for time.** `await page.waitForTimeout(2000)` is flaky. Use `await expect(locator).toBeVisible()` or `page.waitForResponse(urlPattern)`.\n- **Stale selectors re-resolve.** Playwright's locators re-find the element on each action, unlike Puppeteer's handles. Don't over-think it.\n- **Service workers persist across test runs in headed mode.** If you see cached behavior from a previous run, add `await context.clearCookies()` + clear storage before the test.\n- **Installing on CI requires `--with-deps`** on Linux images that lack the browser's shared-library deps.\n- **Parallel tests share a browser process by default**; if one test polls a debugger port, others may interfere. Use `workers: 1` for debugging.\n\n---\n\n## Phase 9 cleanup specifics\n\n```bash\n# Remove trace files from debug runs\nrm -rf playwright-report/ test-results/ trace.zip\n\n# Remove debug spec files from /tmp\nrm -f /tmp/debug-*.spec.ts\n\n# Remove screenshot captures\nrm -f /tmp/debug-*.png\n\n# If you installed browsers just for this session (rare):\n# Don't remove them — they're useful for future sessions. They live in ~/Library/Caches/ms-playwright (macOS) or ~/.cache/ms-playwright (Linux).\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":7300,"content_sha256":"1a56d1b4bea981d21362029af7275327855a26e34b47ec6d5ccc5d459ff43989"},{"filename":"references/tools/pwndbg.md","content":"# pwndbg — GDB With the Useful Views Always On\n\n**https://github.com/pwndbg/pwndbg**\n\npwndbg is a GDB plugin that turns GDB into something humans can actually use for binary debugging. It's strictly a superset of plain GDB — every vanilla GDB command still works, and pwndbg adds views and commands that make you productive.\n\n**If you'd reach for plain `gdb`, reach for pwndbg instead.** The only reason not to is if pwndbg isn't installed on the machine, and that's a 2-minute fix.\n\n---\n\n## Install\n\n```bash\n# macOS\nbrew install pwndbg\n# Or from source:\ngit clone https://github.com/pwndbg/pwndbg\ncd pwndbg && ./setup.sh\n\n# Linux\n# Most distros: apt/dnf/pacman install pwndbg (check availability)\n# Or the same git + ./setup.sh\n\n# Verify\ngdb --version\ngdb ./any-binary\n# At gdb prompt, you should see pwndbg banner + colorful context view\n```\n\nOnce installed, pwndbg auto-loads every time you start `gdb`. You don't source anything manually.\n\n---\n\n## The `context` view — the one feature that changes everything\n\nPlain GDB: you run `info registers`, then `bt`, then `x/10xw $rsp`, then `disas`. Four commands to see what's going on.\n\npwndbg: `context` (or it auto-shows at every break). One command. Everything on screen:\n\n```\n──── registers ────\n RAX 0x0\n RBX 0x7ffffffde158\n RCX 0x7fffff7abf10\n ...\n──── disasm ────\n ► 0x401234 mov rdi, rax\n 0x401237 call 0x401190\n ...\n──── stack ────\n 00:0000│ rsp 0x7ffffffde0a0 → 0x7fffff7c4000\n 01:0008│ 0x7ffffffde0a8 → 0x0\n ...\n──── backtrace ────\n ► f 0 0x401234 parse_input+0x3c\n f 1 0x401180 main+0x120\n f 2 0x7fffff7a5083 __libc_start_main+0xf3\n```\n\nYou always know where you are, what the CPU state is, what's on the stack, and how you got here. This is why pwndbg is the default.\n\n---\n\n## Launch recipes\n\n```bash\n# Debug an existing binary\ngdb ./target\n\n# With args\ngdb --args ./target arg1 arg2\n\n# Attach to a running process\ngdb -p $(pgrep target)\n\n# With a core dump\ngdb ./target ./core\n\n# Headless / remote (for automation or IDE attach)\ngdbserver :2345 ./target # on the target box\ngdb ./target # on your box\n(gdb) target remote \u003chost>:2345\n```\n\nAt the pwndbg prompt:\n\n---\n\n## Essential commands (pwndbg additions)\n\n### Layout / view\n\n```\ncontext # reprint the context view (usually auto)\ncontext regs stack # only show registers + stack sections\ntel $rsp 20 # telescope — walk pointers at $rsp for 20 slots (KEY COMMAND)\ntel $rdi 10 # walk pointers at $rdi (e.g. to dump a struct)\nstack 20 # 20 entries of stack\nvmmap # virtual memory map of the process\n```\n\n**`telescope` is pwndbg's killer command.** Given an address, it walks pointers recursively:\n```\n00:0000│ 0x7ffd... → 0x601010 (heap) → 0x2a (unknown, i.e. a number 42)\n01:0008│ 0x7ffd... → 0x7fff... (stack) → 'hello world'\n```\nThis single view resolves 80% of \"what is at this address\" questions.\n\n### Heap debugging\n\n```\nheap # overview of chunks\nbins # tcache / fastbin / unsorted / smallbin / largebin state\nmalloc_chunk \u003caddr> # inspect a specific chunk\nfind_fake_fast \u003caddr> # (exploit context) find fake-fast overlap candidates\nvis_heap_chunks # visualize heap layout\n```\n\nFor use-after-free / double-free / heap overflow hypotheses, `heap` + `bins` is usually sufficient to see the corruption.\n\n### Exploitation-adjacent (useful for bug understanding too)\n\n```\nchecksec # NX, PIE, RELRO, canary status\nrop --grep 'pop rdi' # find ROP gadgets\nnx # step over (aliased nicely)\nni # step over single instruction\nsi # step into single instruction\n```\n\n### Search\n\n```\nsearch -t byte 0x41 # find byte 0x41 anywhere in memory\nsearch -t string \"admin\" # find string\nsearch -p \u003caddr> # find pointers to \u003caddr>\n```\n\n---\n\n## Standard GDB commands still work\n\npwndbg doesn't replace GDB; it augments it. Everything you know still works:\n\n```\nbreak main # breakpoint at function\nb *0x401234 # breakpoint at address\nb file.c:42 # breakpoint at file:line\nc # continue\nn # next (source-level step over)\ns # step (source-level step into)\nfinish # step out\ninfo breakpoints # list breakpoints\ndelete \u003cn> # delete breakpoint\nwatch \u003cvar> # break on write to variable\nrwatch \u003cvar> # break on read\nawatch \u003cvar> # break on access\np \u003cexpr> # print expression\np/x \u003cexpr> # print in hex\nx/20xw \u003caddr> # examine 20 words as hex\nbt # backtrace\nframe \u003cn> # switch frame\ninfo registers # registers (but `context` is better)\ndisassemble \u003cfunc> # disasm a function\n```\n\n---\n\n## Python scripting inside GDB\n\npwndbg exposes a full Python API. Useful for automating observations across many breakpoints:\n\n```python\n(gdb) python\nimport gdb\ndef on_break():\n frame = gdb.selected_frame()\n pc = frame.read_register('pc')\n print(f'hit at {hex(int(pc))}')\n # Dump args, locals, anything\nend\n```\n\nOr scripted runs from outside:\n```bash\ngdb -batch -ex 'source script.gdb' -ex 'run' ./target\n```\n\n---\n\n## Common workflows by bug type\n\n### Segfault / crash\n\n```bash\ngdb ./target\n(gdb) run \u003cargs>\n# ... crash ...\n(gdb) context # see the crash site\n(gdb) bt # how did we get here?\n(gdb) info registers # what state\n(gdb) tel $rsp 20 # what's on the stack\n```\n\n### \"Function returns wrong value\"\n\n```bash\ngdb ./target\n(gdb) break \u003cfunction>\n(gdb) run \u003cargs>\n# At breakpoint:\n(gdb) finish # let it run to the return\n# pwndbg shows RAX (return value) in context\n```\n\n### \"Variable has unexpected value at point X\"\n\n```bash\ngdb ./target\n(gdb) break \u003cpoint-X>\n(gdb) run\n# At breakpoint:\n(gdb) p \u003cvar> # its value\n(gdb) watch \u003cvar> # set a watchpoint — break when it changes\n(gdb) c # continue; next stop is where it was modified\n```\n\n### \"Memory corruption / heap bug\"\n\n```bash\ngdb ./target\n(gdb) run\n# Crash at free():\n(gdb) heap # heap state\n(gdb) bins # bin state — often shows corruption here\n(gdb) vis_heap_chunks # visualize\n(gdb) malloc_chunk \u003csuspicious-addr>\n```\n\n---\n\n## Gotchas\n\n- **`bt` looks weird on stripped binaries** — function names become offsets. Use Ghidra's function labels to map back (see [ghidra.md](ghidra.md)).\n- **PIE binaries have randomized base addresses.** Addresses you see in Ghidra are unslid; addresses in pwndbg are slid. The `vmmap` command shows the base, and pwndbg's `piebase` command gives you the offset.\n- **Optimized builds inline functions.** You'll set a breakpoint on `my_function` and it won't hit because the function was inlined. Either disable optimizations or break on callers.\n- **Stack canaries trigger `__stack_chk_fail`.** If you see that in a backtrace, the bug caused a stack-smash; look one frame up.\n\n---\n\n## Phase 9 cleanup specifics\n\n```bash\n# Kill gdb / pwndbg sessions\npkill -f 'gdb' || true\npkill -f 'gdbserver' || true\n\n# Remove core dumps generated during session\nrm -f ./core ./core.* ~/core.*\n\n# Remove any scripted GDB files\nrm -f /tmp/debug-*.gdb\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":7867,"content_sha256":"11bd4dbe9f8d08816ddc4d169eb84fdd21cbded3721c63d5fb0346e40b875030"},{"filename":"references/tools/pwntools.md","content":"# pwntools — Scripted Binary / Network Interaction\n\n**https://docs.pwntools.com/en/stable/ · https://github.com/Gallopsled/pwntools**\n\npwntools is a Python framework for building reproducible interactions with binaries and network services. Originally built for CTF exploitation, it's the correct tool for any situation where you need:\n\n- A crafted input sent to a binary or network service, repeatably\n- A \"failing test\" equivalent for a bug that only manifests with specific byte-level input\n- A fuzz harness\n- An exploit PoC\n- Anything where you're tempted to use `echo ... | ./binary` but need more control than shell allows\n\n**Use pwntools for Phase 5 reproduction of binary bugs and Phase 7 tests against binaries.**\n\n---\n\n## Install\n\n```bash\npip install pwntools\n# Or in a venv:\npython -m venv .venv && source .venv/bin/activate\npip install pwntools\n\n# Verify\npython -c 'from pwn import *; print(\"ok\")'\n```\n\nOn some Linux distros you may need build deps: `apt install python3-dev libssl-dev`.\n\n---\n\n## The core API in five idioms\n\n### 1. Process / Remote — the same interface\n\n```python\nfrom pwn import *\n\n# Local process\np = process('./target')\n\n# Remote service\np = remote('example.com', 1337)\n\n# SSH (tunnel to a remote process)\nshell = ssh('user', 'host', password='...')\np = shell.process('./target', cwd='/tmp')\n\n# Same methods on all of the above — this is the value proposition\n```\n\n### 2. I/O — the only five methods you need\n\n```python\np.send(b'data') # send bytes\np.sendline(b'data') # send bytes + \\n\np.recv(n) # receive up to n bytes\np.recvuntil(b'> ') # receive until pattern (blocks)\np.recvline() # receive until \\n\np.interactive() # hand control to your terminal (for manual exploration)\n\n# Combined\np.sendlineafter(b'prompt> ', b'payload')\np.sendafter(b'key:', key)\n```\n\nTimeouts:\n```python\ntry:\n data = p.recvuntil(b'done', timeout=5)\nexcept pwnlib.exception.EOFError:\n print('process died')\nexcept TimeoutError:\n print('no response in 5s')\n```\n\n### 3. context — set arch/OS once, tools align\n\n```python\ncontext.binary = elf = ELF('./target') # auto-sets arch/os/endianness\n# or explicitly:\ncontext.update(arch='amd64', os='linux', endian='little', bits=64)\n```\n\nAfter setting context, helpers like `asm()`, `disasm()`, `cyclic()`, and `ROP()` produce correct output for that target automatically.\n\n### 4. ELF — parse without reverse-engineering by hand\n\n```python\nelf = ELF('./target')\n\nelf.symbols['main'] # address of main\nelf.plt['printf'] # address in PLT (dynamic linkage)\nelf.got['printf'] # GOT entry\nelf.address = 0x555555554000 # set base for PIE binaries\nelf.search(b'/bin/sh') # find string or bytes in the binary\nelf.functions['main'].address # same as elf.symbols['main']\nlist(elf.functions)[:10] # first 10 function names\n```\n\nFor the libc that's linked:\n```python\nlibc = ELF('/lib/x86_64-linux-gnu/libc.so.6')\nlibc.symbols['system']\n```\n\n### 5. cyclic — find offsets without counting\n\nFor \"where exactly does user input reach this variable\" bugs:\n\n```python\np = process('./target')\np.sendline(cyclic(256)) # send a De Bruijn pattern\n# Crash occurs; note the crash value (e.g. RIP = 0x6161616c)\noffset = cyclic_find(0x6161616c) # returns 12 (or wherever in the pattern)\n# Now you know: byte 12 of your input lands at RIP\n```\n\nSaves an hour of \"pad by N bytes then check\" iteration.\n\n---\n\n## Logging during debug\n\npwntools logs output by default. Configure level in the script:\n\n```python\ncontext.log_level = 'debug' # very verbose — shows sent/received bytes\ncontext.log_level = 'info' # default\ncontext.log_level = 'warning' # quiet\n```\n\nFor long scripts, log milestones:\n\n```python\nlog.info('Connected to target')\nlog.success('Bypassed the check')\nlog.failure('Canary corrupted')\nlog.progress('brute-forcing').status('attempt %d' % i)\n```\n\n---\n\n## Typical debug-session patterns\n\n### Reproduce a crash with a specific input\n\n```python\n# /tmp/debug-repro.py\nfrom pwn import *\ncontext.binary = './target'\n\np = process('./target')\np.sendlineafter(b'> ', b'\u003cbad input that crashes>')\np.wait()\n# If it crashed, p.poll() returns non-zero\nassert p.poll() is not None and p.poll() != 0, 'expected crash, got clean exit'\nlog.success(f'confirmed crash (exit {p.poll()})')\n```\n\nJournal this script path. Run it as your \"red test\":\n\n```bash\npython /tmp/debug-repro.py\n```\n\n### Fuzz harness for a suspected input class\n\n```python\n# /tmp/debug-fuzz.py\nfrom pwn import *\nimport random\n\ncontext.binary = './target'\ncontext.log_level = 'warning' # keep quiet in the loop\n\ncrashes = []\nfor i in range(1000):\n payload = bytes(random.randint(0, 255) for _ in range(random.randint(1, 100)))\n p = process('./target')\n p.sendline(payload)\n p.wait()\n if p.poll() is not None and p.poll() \u003c 0: # crashed by signal\n crashes.append((payload, p.poll()))\n log.success(f'iter {i}: crash sig={-p.poll()}')\n\nopen('/tmp/debug-crashes.txt', 'w').write(repr(crashes))\nlog.info(f'found {len(crashes)} crashes')\n```\n\n### Automated exploit harness (CTF or self-testing a known CVE)\n\n```python\nfrom pwn import *\ncontext.binary = elf = ELF('./target')\nlibc = elf.libc or ELF('/lib/x86_64-linux-gnu/libc.so.6')\n\np = process('./target')\n\n# Leak\np.sendline(b'A' * 64 + p64(elf.plt['puts']) + p64(elf.symbols['main']) + p64(elf.got['puts']))\nleak = u64(p.recv(6).ljust(8, b'\\x00'))\nlibc.address = leak - libc.symbols['puts']\nlog.success(f'libc base: {hex(libc.address)}')\n\n# Exploit\nrop = ROP(libc)\nrop.system(next(libc.search(b'/bin/sh')))\np.sendline(b'A' * 64 + rop.chain())\n\np.interactive()\n```\n\n---\n\n## Integration with gdb / pwndbg\n\npwntools can launch your process under gdb:\n\n```python\np = gdb.debug('./target', gdbscript='''\n break main\n continue\n''')\n```\n\nOr attach to a running pwntools-launched process:\n\n```python\np = process('./target')\ngdb.attach(p, gdbscript='break *0x401234')\n# continues in a new terminal window with gdb attached\np.sendline(b'trigger input')\n```\n\nThis is the best way to debug a specific crash repeatably — pwntools drives input, gdb/pwndbg observes runtime state.\n\n---\n\n## Gotchas\n\n- **Python version**: pwntools supports Python 3.8+. Very old distros may not have it.\n- **`p.interactive()` blocks.** It's for manual exploration; remove it from automated scripts.\n- **ASLR on local runs**: turn off for reproducibility during debugging: `echo 0 | sudo tee /proc/sys/kernel/randomize_va_space` (remember to revert — journal this!).\n- **`gdb.debug()` requires `gdb-multiarch`** for cross-arch binaries.\n- **Subprocess cleanup**: if your script crashes, orphan `./target` processes may linger. Kill them at Phase 9 or add `atexit` cleanup.\n\n---\n\n## Phase 9 cleanup specifics\n\n```bash\n# Remove pwntools debug scripts\nrm -f /tmp/debug-*.py\nrm -f /tmp/debug-crashes.txt\n\n# Kill orphan target processes from failed runs\npkill -f './target' || true # adjust to actual binary name\n\n# Restore ASLR if disabled\n# echo 2 | sudo tee /proc/sys/kernel/randomize_va_space # Linux default\n\n# Revert any binary patches applied for testing (see native-binary.md for details)\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":7345,"content_sha256":"0123de928e1e88186487adfee0b69d7a886c596fdd0c5132256cf9319fea466e"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"Debugging","type":"text"}]},{"type":"paragraph","content":[{"text":"You are a hypothesis-driven debugger. Two disciplines apply regardless of language, runtime, or whether you have source:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Runtime truth beats code reading.","type":"text","marks":[{"type":"strong"}]},{"text":" Every claim about why the bug happens must come from observed state — never from a plausible story spun from reading code.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Leave no trace.","type":"text","marks":[{"type":"strong"}]},{"text":" Debugging creates artifacts. Every artifact is journaled and removed before you call the task done.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"The rest of this file is a map. ","type":"text"},{"text":"The knowledge is in ","type":"text","marks":[{"type":"strong"}]},{"text":"references/","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":".","type":"text","marks":[{"type":"strong"}]},{"text":" This file cannot teach you how to debug — it can only tell you which reference will, for your exact situation.","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":1},"content":[{"text":"🚨 READ THE REFERENCES. THIS IS NOT OPTIONAL.","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"This skill is intentionally small.","type":"text","marks":[{"type":"strong"}]},{"text":" Ninety percent of what you need to know lives in ","type":"text"},{"text":"references/","type":"text","marks":[{"type":"code_inline"}]},{"text":". If you skim this file and start working without opening the references, you will reattach a debugger the wrong way, miss a silent-failure pattern you've never seen before, waste an hour on a source-map gotcha, or invent a worse version of a tool that already solves your problem.","type":"text"}]},{"type":"paragraph","content":[{"text":"Every reference below is mandatory when its scenario applies.","type":"text","marks":[{"type":"strong"}]},{"text":" \"I know this language\" is not an exemption. The references exist because every runtime and every specialist tool has at least one gotcha that silently wastes hours, and you will not know which gotcha until you read the file.","type":"text"}]},{"type":"paragraph","content":[{"text":"The gate rule","type":"text","marks":[{"type":"strong"}]},{"text":": before you run a command from a given reference's domain, you must have read that reference in this session. Re-reading across sessions is cheap. Guessing is expensive.","type":"text"}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Runtime Setup — MANDATORY READING BEFORE ATTACHING","type":"text"}]},{"type":"paragraph","content":[{"text":"The methodology is language-agnostic. The commands to launch, attach, breakpoint, and inspect are not. ","type":"text"},{"text":"Open the matching reference before Phase 0. Not during. Not after.","type":"text","marks":[{"type":"strong"}]}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Your runtime is…","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Open this before attaching anything","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Non-negotiable because…","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Python (CPython, pytest, asyncio, Django, FastAPI)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"📖 ","type":"text"},{"text":"references/runtimes/python.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/runtimes/python.md","title":null}},{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"pdb vs ipdb vs debugpy vs pytest --pdb all have different attach semantics. Async code needs special breakpoint handling. Wrappers like ","type":"text"},{"text":"poetry run","type":"text","marks":[{"type":"code_inline"}]},{"text":" swallow flags.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Node.js / tsx / ts-node / Bun / Deno (running source)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"📖 ","type":"text"},{"text":"references/runtimes/node.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/runtimes/node.md","title":null}},{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"tsx","type":"text","marks":[{"type":"code_inline"}]},{"text":" + ","type":"text"},{"text":"node inspect","type":"text","marks":[{"type":"code_inline"}]},{"text":" CLI has a ","type":"text"},{"text":"silent source-map failure","type":"text","marks":[{"type":"strong"}]},{"text":" — breakpoints by line number do not fire. You will not notice unless you read this first.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Rust (cargo, tokio, panics)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"📖 ","type":"text"},{"text":"references/runtimes/rust.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/runtimes/rust.md","title":null}},{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Release builds strip symbols. Tokio tasks need ","type":"text"},{"text":"tokio-console","type":"text","marks":[{"type":"code_inline"}]},{"text":". The borrow checker makes ","type":"text"},{"text":"dbg!","type":"text","marks":[{"type":"code_inline"}]},{"text":" the faster tool most of the time.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Go (goroutines, dlv, pprof, race)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"📖 ","type":"text"},{"text":"references/runtimes/go.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/runtimes/go.md","title":null}},{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Goroutine leaks and recovered panics are silent by default. ","type":"text"},{"text":"dlv","type":"text","marks":[{"type":"code_inline"}]},{"text":" has a specific port convention. ","type":"text"},{"text":"go test -race","type":"text","marks":[{"type":"code_inline"}]},{"text":" is the first thing to run, not the last.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Native binary / stripped C/C++ / no source","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"📖 ","type":"text"},{"text":"references/runtimes/native-binary.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/runtimes/native-binary.md","title":null}},{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"The workflow (triage → dynamic → static → scripted repro) is counterintuitive if you've never done it. ","type":"text"},{"text":"strings -n 8","type":"text","marks":[{"type":"code_inline"}]},{"text":" silently drops short interpolations like ","type":"text"},{"text":"${x}","type":"text","marks":[{"type":"code_inline"}]},{"text":" — read bytes directly for any extraction that matters. macOS adds SIP / Mach-O / lldb specifics that don't apply on Linux.","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Bundled-app binary","type":"text","marks":[{"type":"strong"}]},{"text":" (Bun SEA, Node SEA, Deno compile, pkg, nexe, Electron, Tauri, PyInstaller)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"📖 ","type":"text"},{"text":"references/runtimes/bundled-js-binary.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/runtimes/bundled-js-binary.md","title":null}},{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"These look like Mach-O / ELF but their ","type":"text"},{"text":"high-level","type":"text","marks":[{"type":"em"}]},{"text":" source is recoverable with the right per-bundler tool — Ghidra is overkill. Source-format reality varies: Bun/pkg/nexe/Electron-asar are usually plaintext; Node SEA with code-cache, PyInstaller ","type":"text"},{"text":".pyc","type":"text","marks":[{"type":"code_inline"}]},{"text":", and Deno eszip need extra tooling; Tauri's Rust core still needs native-binary.md. Workflow: identify bundler → locate bundle → extract with the bundler-specific tool → grep.","type":"text"}]}]}]}]},{"type":"paragraph","content":[{"text":"If you cannot honestly say you just opened the reference for your runtime, open it now.","type":"text","marks":[{"type":"strong"}]}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"🚨 ","type":"text"},{"text":"Native binary vs bundled binary — check before committing","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"file ./target","type":"text","marks":[{"type":"code_inline"}]},{"text":" calls them both Mach-O / ELF. The 30-second discriminator is ","type":"text"},{"text":"du -h ./target","type":"text","marks":[{"type":"code_inline"}]},{"text":" (50 MB+ suspect bundled) plus ","type":"text"},{"text":"strings -n 12 ./target | rg -iE 'bun|node_modules|webpack|esbuild|deno|pkg/lib|electron|pyinstaller|nexe|NODE_SEA_FUSE|tauri'","type":"text","marks":[{"type":"code_inline"}]},{"text":". If hits → bundled-js-binary.md. If clean → native-binary.md.","type":"text"}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Specialist Tools — ACTIVELY USE WHEN THE SCENARIO FITS","type":"text"}]},{"type":"paragraph","content":[{"text":"These are not \"optional extras\". They are the correct tool in their domain, and anything else is slower and less reliable. ","type":"text"},{"text":"If the bug fits the domain, you MUST use the tool. Read the reference first to know how.","type":"text","marks":[{"type":"strong"}]}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Tool","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use when","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Reference","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Playwright CLI","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Any browser-served web UI bug. Any flow that requires clicking/typing/navigating. Any \"works locally, breaks in prod\" where the browser or viewport is the variable. ","type":"text"},{"text":"For Phase 8 QA of any browser product, you MUST drive a real browser via Playwright — not curl, not imagination.","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"📖 ","type":"text"},{"text":"references/tools/playwright-cli.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/tools/playwright-cli.md","title":null}},{"type":"strong"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Ghidra","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Any binary without trustworthy source — third-party closed libs, malware, vendored binaries whose behavior contradicts docs, CTF, firmware. ","type":"text"},{"text":"Use Ghidra's decompiler before ","type":"text","marks":[{"type":"strong"}]},{"text":"strings","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":"/","type":"text","marks":[{"type":"strong"}]},{"text":"objdump","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" guessing. It turns machine code into readable C.","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"📖 ","type":"text"},{"text":"references/tools/ghidra.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/tools/ghidra.md","title":null}},{"type":"strong"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"pwndbg","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Any native binary debugging session. It is GDB with the useful views (registers, stack, disasm, heap) always visible. ","type":"text"},{"text":"If you'd reach for plain ","type":"text","marks":[{"type":"strong"}]},{"text":"gdb","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":", reach for ","type":"text","marks":[{"type":"strong"}]},{"text":"pwndbg","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" instead — it is strictly a superset.","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"📖 ","type":"text"},{"text":"references/tools/pwndbg.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/tools/pwndbg.md","title":null}},{"type":"strong"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"pwntools","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Any time you need a reproducible interaction with a binary or network service — crafted payloads, exploit automation, fuzz harness, CTF scripting.","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"📖 ","type":"text"},{"text":"references/tools/pwntools.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/tools/pwntools.md","title":null}},{"type":"strong"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"Failing to use these tools in their domain is a process failure, not a stylistic choice.","type":"text","marks":[{"type":"strong"}]},{"text":" If the bug is in a browser and you did Phase 8 without Playwright, you are doing it wrong. If the bug is in a stripped binary and you read hex with ","type":"text"},{"text":"xxd","type":"text","marks":[{"type":"code_inline"}]},{"text":", you are doing it wrong. The references tell you how. Read them.","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"The Phase Loop — READ THE REFERENCE FOR THE PHASE YOU ARE ENTERING","type":"text"}]},{"type":"paragraph","content":[{"text":"Each phase has exactly one reference. Read it as you enter the phase — not in advance, not from memory. The references are self-contained and short.","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"#","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Phase","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"📖 Open this when entering","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"0","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Environment assessment","type":"text","marks":[{"type":"strong"}]},{"text":" — know the runtime, ports, symbols, env vars, watchers before attaching","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/methodology/00-setup.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/methodology/00-setup.md","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"1","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Journal setup","type":"text","marks":[{"type":"strong"}]},{"text":" — single ","type":"text"},{"text":".debug-journal.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" tracks every artifact for guaranteed revert","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/methodology/00-setup.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/methodology/00-setup.md","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"2","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Hypothesis formation","type":"text","marks":[{"type":"strong"}]},{"text":" — minimum three, across orthogonal axes, each with distinguishing evidence","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/methodology/02-investigate.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/methodology/02-investigate.md","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"3","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Parallel investigation","type":"text","marks":[{"type":"strong"}]},{"text":" — team mode ","type":"text"},{"text":"debug-squad","type":"text","marks":[{"type":"code_inline"}]},{"text":" when enabled, async subagents otherwise","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/methodology/02-investigate.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/methodology/02-investigate.md","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"4","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Oracle Triple","type":"text","marks":[{"type":"strong"}]},{"text":" — after 2 consecutive failed rounds, spawn three Oracles with orthogonal framings and synthesize","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/methodology/04-oracle-triple.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/methodology/04-oracle-triple.md","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"5","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"User decision escalation","type":"text","marks":[{"type":"strong"}]},{"text":" — only when evidence exhausted and the call has policy implications","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/methodology/05-escalate.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/methodology/05-escalate.md","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"6","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Root cause confirmation","type":"text","marks":[{"type":"strong"}]},{"text":" — confirmed only when toggling the suspected cause toggles the bug","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/methodology/06-fix.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/methodology/06-fix.md","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"7","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"TDD fix","type":"text","marks":[{"type":"strong"}]},{"text":" — red test first, minimal green, no scope expansion","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/methodology/06-fix.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/methodology/06-fix.md","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"8","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Manual QA","type":"text","marks":[{"type":"strong"}]},{"text":" — actually use the system (tmux for CLI, Playwright for browser, real curl for API, real repro for binary)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/methodology/08-qa.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/methodology/08-qa.md","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"9","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Cleanup","type":"text","marks":[{"type":"strong"}]},{"text":" — walk the journal, revert every artifact, verify ","type":"text"},{"text":"git diff","type":"text","marks":[{"type":"code_inline"}]},{"text":" shows only fix + test","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/methodology/09-cleanup.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/methodology/09-cleanup.md","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"10","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Final verification","type":"text","marks":[{"type":"strong"}]},{"text":" — four evidence gates before declaring done","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/methodology/09-cleanup.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/methodology/09-cleanup.md","title":null}}]}]}]}]}]},{"type":"paragraph","content":[{"text":"Phase references are short by design.","type":"text","marks":[{"type":"strong"}]},{"text":" Reading one takes a minute. Skipping one costs an hour.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Cross-cutting methodology references","type":"text"}]},{"type":"paragraph","content":[{"text":"These are not phases — read them when the situation calls for them:","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Situation","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Reference","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"You cannot run the actual operation (paid API, blocked network, missing hardware) but still need runtime evidence","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"📖 ","type":"text"},{"text":"references/methodology/partial-runtime-evidence.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/methodology/partial-runtime-evidence.md","title":null}},{"type":"strong"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"You're about to declare an extraction / audit / reverse-engineering task done and want a skeptical pass","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"📖 ","type":"text"},{"text":"references/methodology/partial-runtime-evidence.md#verification-oracle-pattern-for-non-debug-tasks","type":"text","marks":[{"type":"link","attrs":{"href":"references/methodology/partial-runtime-evidence.md#verification-oracle-pattern-for-non-debug-tasks","title":null}},{"type":"strong"}]},{"text":" (Verification Oracle is ","type":"text"},{"text":"not","type":"text","marks":[{"type":"em"}]},{"text":" the same as Oracle Triple — read the file)","type":"text"}]}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Non-Negotiable Safety Invariants","type":"text"}]},{"type":"paragraph","content":[{"text":"\u003csafety>","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Runtime state is the only source of truth.","type":"text","marks":[{"type":"strong"}]},{"text":" A hypothesis without an observed value is a guess. Do not fix guesses.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Every debug artifact is journaled before it is created.","type":"text","marks":[{"type":"strong"}]},{"text":" Journal-then-modify, not modify-then-remember-maybe.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Never ship a fix without a failing-first test.","type":"text","marks":[{"type":"strong"}]},{"text":" Red→green transition required, or the fix is unverified.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Never declare done on type-check/compile alone.","type":"text","marks":[{"type":"strong"}]},{"text":" Types catch declaration bugs. Only running the actual user scenario catches the actual user bug.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Never ask the user a question that runtime evidence can already answer.","type":"text","marks":[{"type":"strong"}]},{"text":" Escalation is for genuine ambiguity.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Never silently swallow errors while debugging.","type":"text","marks":[{"type":"strong"}]},{"text":" If the system swallows errors, that is often the bug itself. Make them loud temporarily; restore at cleanup.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Never ","type":"text","marks":[{"type":"strong"}]},{"text":"git commit","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" from inside this skill.","type":"text","marks":[{"type":"strong"}]},{"text":" Commits belong to ","type":"text"},{"text":"/git-master","type":"text","marks":[{"type":"code_inline"}]},{"text":" after the user confirms the fix.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Never attach without having read the runtime reference.","type":"text","marks":[{"type":"strong"}]},{"text":" The gate rule. \u003c/safety>","type":"text"}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"What to Do Right Now","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Read the user's bug description.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Identify the runtime.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Open ","type":"text","marks":[{"type":"strong"}]},{"text":"references/runtimes/\u003cruntime>.md","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":".","type":"text","marks":[{"type":"strong"}]},{"text":" Read it.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Identify which specialist tools apply. ","type":"text"},{"text":"Open each matching ","type":"text","marks":[{"type":"strong"}]},{"text":"references/tools/*.md","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":".","type":"text","marks":[{"type":"strong"}]},{"text":" Read them.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Open ","type":"text"},{"text":"references/methodology/00-setup.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" and start Phase 0.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Follow the phase loop. Read each methodology reference as you enter the phase.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"The references are the skill. This file is an index.","type":"text","marks":[{"type":"strong"}]}]}]},"metadata":{"date":"2026-06-05","name":"debugging","author":"@skillopedia","source":{"stars":60584,"repo_name":"oh-my-opencode","origin_url":"https://github.com/code-yeongyu/oh-my-opencode/blob/HEAD/packages/shared-skills/skills/debugging/SKILL.md","repo_owner":"code-yeongyu","body_sha256":"45c0ed2d7c0d1a20cd6fc10a5263eb6f5eafe2403817fd979b21d701cf0133ac","cluster_key":"7b80ea78c38343030411ff4f4089faf7d07ee970183b0adcb18ae3a53d252c4f","clean_bundle":{"format":"clean-skill-bundle-v1","source":"code-yeongyu/oh-my-opencode/packages/shared-skills/skills/debugging/SKILL.md","attachments":[{"id":"ad7c2363-e8b9-5874-aea8-0a4e3596e32e","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/ad7c2363-e8b9-5874-aea8-0a4e3596e32e/attachment.md","path":"references/methodology/00-setup.md","size":5400,"sha256":"dc05ca5d38d870359b2ab8304ae32b31430892d864723d87f846e7a41cf8ed7f","contentType":"text/markdown; charset=utf-8"},{"id":"a541f694-9ae6-504b-8536-0ba955b255bf","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/a541f694-9ae6-504b-8536-0ba955b255bf/attachment.md","path":"references/methodology/02-investigate.md","size":7648,"sha256":"5ad08378f47e3b9ebe0fd1995e7d9b382d6cb84e1a32fa2e4c60789df8398fce","contentType":"text/markdown; charset=utf-8"},{"id":"8a81816b-3f03-502f-bf0b-15771f3201e2","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/8a81816b-3f03-502f-bf0b-15771f3201e2/attachment.md","path":"references/methodology/04-oracle-triple.md","size":6749,"sha256":"6282d2b26f5870d552cff39496f0d4cd428acce2a1b6fb226162e4e55c23c9a9","contentType":"text/markdown; charset=utf-8"},{"id":"8596921f-dc92-5342-865a-dfb3384d252d","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/8596921f-dc92-5342-865a-dfb3384d252d/attachment.md","path":"references/methodology/05-escalate.md","size":3211,"sha256":"33e2eb6af5e33e2b19df0cdee97abc71dca50c82a7e401181d91795e54718b39","contentType":"text/markdown; charset=utf-8"},{"id":"13008cf0-1a66-5afc-8d50-08615cfe7262","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/13008cf0-1a66-5afc-8d50-08615cfe7262/attachment.md","path":"references/methodology/06-fix.md","size":5529,"sha256":"3b7c3d901edb6cd04022da54bbe6d48a552c67fe834d3c38e3d07e8426e078ed","contentType":"text/markdown; charset=utf-8"},{"id":"779d1d21-c514-5aba-a772-5597dda196c9","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/779d1d21-c514-5aba-a772-5597dda196c9/attachment.md","path":"references/methodology/08-qa.md","size":6287,"sha256":"09f668880bfb868a46d937aa57ad67397438e6333897b8c0cb50a9c8282ce032","contentType":"text/markdown; charset=utf-8"},{"id":"0007069c-1457-5f66-97a0-77848a24a392","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/0007069c-1457-5f66-97a0-77848a24a392/attachment.md","path":"references/methodology/09-cleanup.md","size":6173,"sha256":"5675b3e72c38b9d0c87ab1b473f9f303aaea24047c8ef8d96e069539c8125679","contentType":"text/markdown; charset=utf-8"},{"id":"c5c671bf-7688-58f5-a63c-3d83f97dacf8","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/c5c671bf-7688-58f5-a63c-3d83f97dacf8/attachment.md","path":"references/methodology/partial-runtime-evidence.md","size":10697,"sha256":"820c1ce061a9cbef3145d3ede253e4e21c3fa384935ac2e76008b391945feb2a","contentType":"text/markdown; charset=utf-8"},{"id":"e2c006ec-4436-5151-ad61-1aaadf2bdd69","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/e2c006ec-4436-5151-ad61-1aaadf2bdd69/attachment.md","path":"references/runtimes/bundled-js-binary.md","size":19353,"sha256":"f7f01bc4d95c24bd28d24a3ae240a037b0ded00d690d18a6900ddf63025fe4ec","contentType":"text/markdown; charset=utf-8"},{"id":"f506d970-6614-5c2a-842a-90992182bb3a","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/f506d970-6614-5c2a-842a-90992182bb3a/attachment.md","path":"references/runtimes/go.md","size":8181,"sha256":"284614797d02d1aee25f440a7daa145af59a3ff61e5ba79c28ad0542c28f4f00","contentType":"text/markdown; charset=utf-8"},{"id":"88a66fab-b8b9-5f9b-8d57-7e9697126123","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/88a66fab-b8b9-5f9b-8d57-7e9697126123/attachment.md","path":"references/runtimes/native-binary.md","size":21367,"sha256":"520a767fadbd4d0b1b59fb6c652ded7f47a0ff5d742563f927334ae4537c2be0","contentType":"text/markdown; charset=utf-8"},{"id":"64181a5f-0a9e-52a5-8499-1543e3e30f04","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/64181a5f-0a9e-52a5-8499-1543e3e30f04/attachment.md","path":"references/runtimes/node.md","size":9650,"sha256":"0b728b643ed89c318bb564e1137cf68bb6acb61c1774ed73dee04b5a70cf1058","contentType":"text/markdown; charset=utf-8"},{"id":"1cd50921-96c9-5871-8134-8ffc36968947","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/1cd50921-96c9-5871-8134-8ffc36968947/attachment.md","path":"references/runtimes/python.md","size":8206,"sha256":"7c69b34e0d1342900c566468828e8af5a688669b75803a5eb8f2f6e156d38490","contentType":"text/markdown; charset=utf-8"},{"id":"01a7908c-ce5a-509d-bf92-5d70fb31b6ad","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/01a7908c-ce5a-509d-bf92-5d70fb31b6ad/attachment.md","path":"references/runtimes/rust.md","size":6978,"sha256":"48fa55949ba60c42a1e9827af94122905ea82cb147262b1e044a27734b6a2958","contentType":"text/markdown; charset=utf-8"},{"id":"089afaf9-095a-5391-8b1b-482d8151de1c","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/089afaf9-095a-5391-8b1b-482d8151de1c/attachment.md","path":"references/tools/ghidra.md","size":7360,"sha256":"8d50534340eef3cd664458cade9d11d18cabff804979128d7107a5fdd36dbbf0","contentType":"text/markdown; charset=utf-8"},{"id":"779ae0d1-c001-508d-b330-bbc6004b267f","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/779ae0d1-c001-508d-b330-bbc6004b267f/attachment.md","path":"references/tools/playwright-cli.md","size":7300,"sha256":"1a56d1b4bea981d21362029af7275327855a26e34b47ec6d5ccc5d459ff43989","contentType":"text/markdown; charset=utf-8"},{"id":"28ffbff6-422e-57b9-92d2-f12a34cd592e","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/28ffbff6-422e-57b9-92d2-f12a34cd592e/attachment.md","path":"references/tools/pwndbg.md","size":7867,"sha256":"11bd4dbe9f8d08816ddc4d169eb84fdd21cbded3721c63d5fb0346e40b875030","contentType":"text/markdown; charset=utf-8"},{"id":"a1d3c2b6-da34-5755-abdb-8caa3d0539d2","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/a1d3c2b6-da34-5755-abdb-8caa3d0539d2/attachment.md","path":"references/tools/pwntools.md","size":7345,"sha256":"0123de928e1e88186487adfee0b69d7a886c596fdd0c5132256cf9319fea466e","contentType":"text/markdown; charset=utf-8"}],"bundle_sha256":"b770b0305c116a9002c84095a5031d5760309b22fd514fed35c9a4abadd0b5ea","attachment_count":18,"text_attachments":18,"attachment_storage":"skillopedia-attachments-v1","binary_attachments":0,"excluded_attachments":[]},"cluster_size":4,"skill_md_path":"packages/shared-skills/skills/debugging/SKILL.md","import_metadata":{"date":"2026-06-05","author":"@skillopedia","version":"v1","category":"testing-qa","category_label":"Testing"},"exact_dupes_collapsed_into_this":3},"version":"v1","category":"testing-qa","import_tag":"clean-skills-v1","description":"MUST USE for any real runtime debugging across ANY language or binary — crashes, silent failures, wrong responses, stuck processes, memory leaks, async misbehavior, unexplained timing, reverse engineering. Runs a hypothesis-driven loop: form ≥3 hypotheses, investigate in parallel, after 2 failed rounds spawn Oracles from orthogonal angles, confirm root cause, lock with a failing test, fix minimally, QA by actually USING the system, scrub artifacts. The actual HOW lives in `references/` — READ THEM. Triggers: 'debug this', 'why is X not working', 'hanging', 'attach a debugger', 'reverse engineer', 'pwndbg', 'gdb', 'lldb', 'node inspect', 'tsx debug', 'pdb', 'dlv', 'delve', 'rust-gdb', 'set a breakpoint', 'context window exploded', 'why is the response empty', 'attach the debugger', 'debug it', 'why is this happening', 'trace this bug', 'reproduce and fix', 'silent failure', 'HTTP 200 but empty', 'why did it stop', 'inspect the binary', 'reverse engineering', 'playwright'."}},"renderedAt":1782979437773}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.