graphify — Skillopedia

/graphify Turn any folder of files into a navigable knowledge graph with community detection, an honest audit trail, and three outputs: interactive HTML, GraphRAG-ready JSON, and a plain-language GRAPH REPORT.md. Usage What graphify is for Drop any folder of code, docs, papers, images, or video into graphify and get a queryable knowledge graph. Persistent across sessions, honest audit trail (EXTRACTED/INFERRED/AMBIGUOUS), community detection surfaces cross-document connections you wouldn't think to ask about. When a graph is worth building: this applies when you are choosing a tool yourself;…

, re.IGNORECASE),\n # SSH/GPG private keys. Left boundary + IGNORECASE so `grid_rsa` (alpha before\n # `id_rsa`) and `ID_RSA` are handled correctly, not matched as a substring.\n re.compile(r'(^|[^A-Za-z0-9])(id_rsa|id_dsa|id_ecdsa|id_ed25519)(\\.pub)?

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

, re.IGNORECASE),\n re.compile(r'^secring(\\.(gpg|pgp))?

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

, re.IGNORECASE),\n # NOTE: aws_credentials/gcloud_credentials/service_account moved to the\n # boundary-checked Stage 3 keyword path (#2106). The old unbounded\n # `service.account` substring (regex `.` wildcard) matched real source like\n # google/oauth2/service_account.py and prose like aws_credentials_rotation.md.\n]\n\n# Committed dotenv / envrc templates — placeholders only, not live secrets.\n# Stage 2's `.env.` regex otherwise treats these like `.env.local` (#2184).\n_ENV_TEMPLATE_SUFFIXES = (\".example\", \".sample\", \".template\", \".dist\")\n\n\ndef _is_env_template(name: str) -> bool:\n \"\"\"True for `.env.example` / `.envrc.sample` style committed templates (#2184).\"\"\"\n lower = name.lower()\n if not lower.endswith(_ENV_TEMPLATE_SUFFIXES):\n return False\n # Basename must still be an .env* / .envrc* file (not e.g. secrets.example).\n return bool(re.match(r\"\\.(env|envrc)\\.\", lower))\n\n# Generic keyword patterns - these only count when the keyword is LOAD-BEARING\n# in the filename (see _generic_keyword_hit), because a keyword buried mid-phrase\n# in a long descriptive slug names a topic, not a credential store:\n# \"token-economics-of-recall.md\" is a note ABOUT tokens; \"api_token.txt\" IS one.\n# Uses lookarounds instead of \\b so underscore-prefixed names like api_token.txt\n# match. Both patterns use (?![a-zA-Z]) so that the trailing-underscore behavior\n# is consistent: \"secret_store.txt\" IS flagged, \"tokenizer.py\" is NOT (because\n# \"i\" after \"token\" is alpha and blocks the match).\n# `token` is kept separate because its longer suffix \"izer\"/\"ize\" is the only\n# common false-positive; other keywords have no such well-known derivatives.\n_GENERIC_KEYWORD_PATTERNS = [\n re.compile(r'(?\u003c![a-zA-Z0-9])(credential|secret|passwd|password|private_key)s?(?![a-zA-Z])', re.IGNORECASE),\n re.compile(r'(?\u003c![a-zA-Z0-9])tokens?(?![a-zA-Z])', re.IGNORECASE),\n # service_account / service-account / serviceaccount (GCP key files). In the\n # keyword path so `service_account.py` (real source) is spared while\n # `service-account.json` (a downloaded key) and bare names are still caught\n # (#2106; was an unbounded Stage 2 substring). aws_credentials/gcloud_credentials\n # are already covered by the `credential` keyword above.\n re.compile(r'(?\u003c![a-zA-Z0-9])service[._-]?account(?![a-zA-Z])', re.IGNORECASE),\n]\n\n# Prose/note formats: a heavily-linked wiki article whose topic slug ends in a\n# keyword (privacy-tokens.md, token-economics.md) is a document ABOUT the topic,\n# not a credential store, so it must not be silently dropped (#2106). A BARE\n# keyword name (secrets.md, token.md, passwords.md) still reads as a dump and\n# stays excluded — see _is_prose_note.\n_PROSE_EXTS = frozenset({\".md\", \".markdown\", \".rst\", \".org\", \".adoc\", \".tex\"})\n\n# Bare PLURAL \"tokens\" in a prose file (TOKENS.md) reads as a design-token\n# reference doc, not a credential dump — unlike every other generic keyword's\n# plural (secrets.md, passwords.md, credentials.md still read as dumps) and\n# unlike the singular \"token.md\", which stays excluded (#3527). Scoped to\n# prose extensions only: \"tokens.txt\" is still a plausible secret store.\n_BARE_PLURAL_TOKENS = re.compile(r'(?\u003c![a-zA-Z0-9])tokens(?![a-zA-Z])', re.IGNORECASE)\n\n# Data/serialization extensions that commonly ARE secret stores when their name\n# hits a generic keyword (credentials.json, secrets.yaml, token.toml) or they sit\n# in an ambiguous sensitive dir (secrets/db.json). These stay subject to the\n# Stage 1 ambiguous-dir drop and the Stage 3 keyword drop even though some route\n# through the CODE path for manifest parsing — only real programming-language\n# source is exempt (#1666, #1943).\n_SECRET_PRONE_DATA_EXTS = frozenset({\n \".json\", \".yaml\", \".yml\", \".toml\", \".ini\", \".cfg\", \".conf\", \".config\",\n \".xml\", \".properties\", \".env\", \".txt\",\n # .tfvars is Terraform's canonical VALUES store (routinely holds real\n # secrets), not source — keep it out of the graph even though it sits in\n # CODE_EXTENSIONS. .tf/.hcl are genuine infra source and stay graphable.\n \".tfvars\",\n})\n\n# Word separators for the load-bearing check (underscore intentionally included;\n# multi-word keywords like private_key are handled by the end-of-stem check,\n# which runs before word counting).\n_WORD_SPLIT = re.compile(r'[-_\\s.]+') # '.' included so `token.economics.notes` counts as 3 words (#2106)\n\n\ndef _is_prose_note(path: Path) -> bool:\n \"\"\"A prose/note file (.md/.rst/...) whose stem is a multi-word topic slug is\n exempt from the generic-keyword drop (#2106). A stem that IS exactly a bare\n keyword (secrets / token / passwords) is NOT exempt — that still reads as a\n credential dump. The one exception is bare plural \"tokens\" (TOKENS.md),\n which reads as a design-token reference doc rather than a secret store\n (#3527).\"\"\"\n if path.suffix.lower() not in _PROSE_EXTS:\n return False\n stem = Path(path.name).stem.lstrip('.') or Path(path.name).stem\n if _BARE_PLURAL_TOKENS.fullmatch(stem):\n return True\n return not any(p.fullmatch(stem) for p in _GENERIC_KEYWORD_PATTERNS)\n\n\ndef _generic_keyword_hit(name: str) -> bool:\n \"\"\"True if a generic secret keyword appears load-bearing in the filename.\n\n Secret-store files name their contents, and in English compounds the\n content noun is the head, which comes last: \"github-personal-access-token\",\n \"api_token\", \"oauth_token\". A keyword that is neither at the end of the\n stem nor in a short (\u003c=2 word) name is a topic word in a descriptive slug\n (\"token-economics-of-recall.md\", \"password-policy-discussion.md\") and must\n not cause the file to be silently dropped from the graph (#436, #718).\n \"\"\"\n # Stem = name minus only the FINAL extension (not up to the first dot), so a\n # multi-dot topic slug like `token.economics.notes.md` keeps all its words and\n # doesn't collapse to a bare `token` (#2106). Leading dots stripped so\n # dotfiles like `.token` keep their keyword.\n stem = Path(name).stem.lstrip('.') or Path(name).stem\n for pat in _GENERIC_KEYWORD_PATTERNS:\n hit = False\n for m in pat.finditer(stem):\n hit = True\n if m.end() == len(stem): # keyword ends the stem -> names the contents\n return True\n if hit and len([w for w in _WORD_SPLIT.split(stem) if w]) \u003c= 2:\n return True # short name like token_config.yaml / secret_handler.txt\n return False\n\n# Signals that a .md/.txt file is actually a converted academic paper\n_PAPER_SIGNALS = [\n re.compile(r'\\barxiv\\b', re.IGNORECASE),\n re.compile(r'\\bdoi\\s*:', re.IGNORECASE),\n re.compile(r'\\babstract\\b', re.IGNORECASE),\n re.compile(r'\\bproceedings\\b', re.IGNORECASE),\n re.compile(r'\\bjournal\\b', re.IGNORECASE),\n re.compile(r'\\bpreprint\\b', re.IGNORECASE),\n re.compile(r'\\\\cite\\{'), # LaTeX citation\n re.compile(r'\\[\\d+\\]'), # Numbered citation [1], [23] (inline)\n re.compile(r'\\[\\n\\d+\\n\\]'), # Numbered citation spread across lines (markdown conversion)\n re.compile(r'eq\\.\\s*\\d+|equation\\s+\\d+', re.IGNORECASE),\n re.compile(r'\\d{4}\\.\\d{4,5}'), # arXiv ID like 1706.03762\n re.compile(r'\\bwe propose\\b', re.IGNORECASE), # common academic phrasing\n re.compile(r'\\bliterature\\b', re.IGNORECASE), # \"from the literature\"\n]\n_PAPER_SIGNAL_THRESHOLD = 3 # need at least this many signals to call it a paper\n\n\ndef _is_graphable_source(path: Path) -> bool:\n \"\"\"True for genuine programming-language source — the only category exempt\n from the ambiguous-dir (Stage 1, #1943) and generic-keyword (Stage 3, #1666)\n drops. Data/serialization formats are NOT exempt even though some route\n through the CODE path for manifest parsing: credentials.json / secrets.yaml\n are exactly the stores those stages must keep catching.\n \"\"\"\n return classify_file(path) == FileType.CODE and path.suffix.lower() not in _SECRET_PRONE_DATA_EXTS\n\n\ndef _is_sensitive(path: Path) -> bool:\n \"\"\"Return True if this file likely contains secrets and should be skipped.\"\"\"\n # Stage 1: any PARENT directory is a known secrets dir (parts[:-1] excludes\n # the filename itself so a root-level file named \"credentials\" is not falsely\n # skipped — the name patterns in Stage 2 handle the filename). Dedicated\n # credential stores drop everything unconditionally; ambiguous bare-name dirs\n # (secrets/, credentials/) spare genuine source (#1943), which still falls\n # through so Stages 2-3 screen its filename like anywhere else.\n parents = path.parts[:-1]\n # Lowercase the segment comparison so `Secrets/`/`SECRETS/` (real on\n # case-insensitive macOS/Windows filesystems) are still caught (#2106).\n if any(part.lower() in _CREDENTIAL_STORE_DIRS for part in parents):\n return True\n if any(part.lower() in _AMBIGUOUS_SENSITIVE_DIRS for part in parents) and not _is_graphable_source(path):\n return True\n # Stage 2: filename pattern match. Template suffixes (.example/.sample/…)\n # on .env / .envrc are the usual \"safe to commit\" convention — keep them\n # in the graph without opening a broad Stage 2 allowlist (#2184 / #1921).\n name = path.name\n if any(p.search(name) for p in _SENSITIVE_PATTERNS) and not _is_env_template(name):\n return True\n # Stage 3: generic keywords, only when load-bearing in the name. Do NOT let a\n # bare name keyword silently drop a genuine programming-language source file:\n # a .rb/.py named device_token or passwords_controller is a module, not a secret\n # store (#1666). Data/config formats (.json, .yaml, .toml, ...) are deliberately\n # NOT exempt even though .json routes through the CODE path for manifest parsing,\n # because credentials.json / oauth_token.json / secrets.yaml are exactly the\n # secret stores this stage must catch. The specific Stage 2 patterns (.env, .pem,\n # id_rsa, ...) still apply to everything regardless of extension.\n if _generic_keyword_hit(name):\n # Genuine source AND multi-word prose notes are exempt; a bare-keyword\n # name (secrets.md, token.txt) still drops (#1666, #2106).\n return not (_is_graphable_source(path) or _is_prose_note(path))\n return False\n\n\ndef _looks_like_paper(path: Path) -> bool:\n \"\"\"Heuristic: does this text file read like an academic paper?\"\"\"\n try:\n # Only scan first 3000 chars for speed\n text = path.read_text(encoding=\"utf-8\", errors=\"ignore\")[:3000]\n hits = sum(1 for pattern in _PAPER_SIGNALS if pattern.search(text))\n return hits >= _PAPER_SIGNAL_THRESHOLD\n except Exception:\n return False\n\n\n_ASSET_DIR_MARKERS = {\".imageset\", \".xcassets\", \".appiconset\", \".colorset\", \".launchimage\"}\n\n\n_SHEBANG_CODE_INTERPRETERS = {\n \"python\", \"python3\", \"python2\",\n \"ruby\", \"perl\", \"node\", \"nodejs\",\n \"bash\", \"sh\", \"dash\", \"zsh\", \"fish\", \"ksh\", \"tcsh\",\n \"lua\", \"php\", \"julia\", \"Rscript\",\n}\n\n\ndef _split_env_s(value: str, rest: list[str]) -> list[str]:\n \"\"\"Re-tokenize an `env -S`/`--split-string` packed command, prepending the\n operand to any trailing args. Returns the unpacked argv.\"\"\"\n packed = \" \".join([value, *rest]).strip()\n return shlex.split(packed)\n\n\ndef _env_command_args(args: list[str], *, allow_split: bool = True) -> list[str]:\n \"\"\"Strip leading env(1) options and var assignments, return the trailing\n command argv. Covers macOS/BSD and GNU coreutils env documented spellings.\n\n POSIX/macOS short forms:\n env [-0iv] [-C workdir] [-P utilpath] [-S string]\n [-u name] [name=value ...] [utility [argument ...]]\n\n GNU coreutils long/compact forms additionally supported:\n --argv0=ARG / -a ARG / -aARG\n --unset=NAME / --unset NAME / -u NAME / -uNAME\n --chdir=DIR / --chdir DIR / -C DIR / -CDIR\n --split-string=STRING / --split-string STRING\n -S STRING / -SSTRING / -vS STRING / -vSSTRING\n --ignore-environment / --null / --debug / --list-signal-handling\n --default-signal[=SIG] / --ignore-signal[=SIG] / --block-signal[=SIG]\n\n `-S` / `--split-string` payloads are themselves env-style argument lists\n per the GNU shebang synopsis:\n #!/usr/bin/env -[v]S[option]... [name=value]... command [args]...\n so after splitting the payload we recursively re-parse it with\n `allow_split=False` (a nested -S inside a split payload is rejected to\n bound recursion).\n\n Unknown hyphen-prefixed args yield [] (we refuse to guess whether\n their next token is an interpreter or an operand).\n \"\"\"\n i = 0\n while i \u003c len(args):\n arg = args[i]\n\n if arg == \"--\":\n return args[i + 1:]\n\n # Split-string forms: tokenize the packed payload, then re-parse it\n # as env args (so leading assignments/flags inside the payload are\n # skipped before the interpreter is identified).\n if allow_split:\n if arg == \"-S\":\n if i + 1 >= len(args):\n return []\n return _env_command_args(\n _split_env_s(\" \".join(args[i + 1:]), []),\n allow_split=False,\n )\n if arg.startswith(\"-S\") and len(arg) > 2:\n return _env_command_args(\n _split_env_s(arg[2:], args[i + 1:]),\n allow_split=False,\n )\n if arg == \"-vS\":\n if i + 1 >= len(args):\n return []\n return _env_command_args(\n _split_env_s(\" \".join(args[i + 1:]), []),\n allow_split=False,\n )\n if arg.startswith(\"-vS\") and len(arg) > 3:\n return _env_command_args(\n _split_env_s(arg[3:], args[i + 1:]),\n allow_split=False,\n )\n if arg.startswith(\"--split-string=\"):\n return _env_command_args(\n _split_env_s(arg.split(\"=\", 1)[1], args[i + 1:]),\n allow_split=False,\n )\n if arg == \"--split-string\":\n if i + 1 >= len(args):\n return []\n return _env_command_args(\n _split_env_s(args[i + 1], args[i + 2:]),\n allow_split=False,\n )\n\n # Options with separate required operand\n if arg in {\"-u\", \"-C\", \"-P\", \"-a\", \"--unset\", \"--chdir\", \"--argv0\"}:\n if i + 2 > len(args):\n return []\n i += 2\n continue\n\n # Clumped short option + operand\n if (\n arg.startswith((\"-u\", \"-C\", \"-P\", \"-a\"))\n and len(arg) > 2\n and not arg.startswith(\"--\")\n ):\n i += 1\n continue\n\n # Long option with `=` operand\n if arg.startswith((\"--unset=\", \"--chdir=\", \"--argv0=\")):\n i += 1\n continue\n\n # No-operand flags\n if arg in {\"-\", \"-i\", \"-0\", \"-v\", \"--ignore-environment\", \"--null\",\n \"--debug\", \"--list-signal-handling\"}:\n i += 1\n continue\n\n # Signal-handling long flags (with or without =SIG operand — we treat\n # them as no-effect for interpreter-resolution purposes)\n if arg.startswith((\"--default-signal\", \"--ignore-signal\", \"--block-signal\")):\n i += 1\n continue\n\n # Unknown hyphen-prefixed: refuse to guess\n if arg.startswith(\"-\"):\n return []\n\n # Inline NAME=value assignment\n if \"=\" in arg:\n i += 1\n continue\n\n # First non-option, non-assignment token starts the command argv\n return args[i:]\n\n return []\n\n\ndef _strip_interpreter_version(interp: str) -> str:\n \"\"\"Map a version-suffixed interpreter onto the name it is a version of.\n\n Shebangs name a specific version as often as the generic one\n (``python3.12``, ``python2.7``, ``lua5.4``, ``php8.2``). Try the name with\n its major version (``python3``), then the bare name (``lua``), and keep the\n original when neither is a known code interpreter.\n \"\"\"\n if interp in _SHEBANG_CODE_INTERPRETERS:\n return interp\n m = re.fullmatch(r\"([A-Za-z]+)(\\d+)(?:\\.\\d+)*\", interp)\n if m:\n for candidate in (m.group(1) + m.group(2), m.group(1)):\n if candidate in _SHEBANG_CODE_INTERPRETERS:\n return candidate\n return interp\n\n\ndef _shebang_interpreter(path: Path) -> str | None:\n \"\"\"Return the interpreter name from a shebang line.\n\n Handles forms that a naive parser misses:\n - `#!/usr/bin/env -S python3 -u` (env -S split-args form, anywhere)\n - `#!/usr/bin/env -i bash` (no-operand env flags)\n - `#!/usr/bin/env -u VAR python3` (env options with operands)\n - `#!/usr/bin/env -C /tmp python3` (env -C workdir)\n - `#!/usr/bin/env -P /bin python3` (env -P utilpath)\n - `#!/usr/bin/env DEBUG=1 python3` (inline var assignment)\n - `#!\"/usr/local/bin/python with spaces\"` (shlex handles quotes)\n\n Returns the basename of the resolved interpreter, with a version suffix\n on a known interpreter dropped (``python3.12`` -> ``python3``), or None if\n there is no shebang / the file is unreadable / parsing fails.\n \"\"\"\n try:\n with path.open(\"rb\") as f:\n first = f.read(256)\n if not first.startswith(b\"#!\"):\n return None\n line = first.split(b\"\\n\")[0].decode(errors=\"replace\")[2:].strip()\n parts = shlex.split(line)\n if not parts:\n return None\n interp = Path(parts[0]).name\n if interp == \"env\":\n env_args = _env_command_args(parts[1:])\n if not env_args:\n return None\n interp = Path(env_args[0]).name\n return _strip_interpreter_version(interp)\n except (OSError, ValueError):\n return None\n\n\ndef _shebang_file_type(path: Path) -> FileType | None:\n \"\"\"Peek at the first line of an extensionless file for a shebang.\"\"\"\n interp = _shebang_interpreter(path)\n if interp in _SHEBANG_CODE_INTERPRETERS:\n return FileType.CODE\n return None\n\n\ndef classify_file(path: Path) -> FileType | None:\n # Package manifests (apm.yml, pyproject.toml, Cargo.toml, go.mod, pom.xml) are parsed\n # deterministically, so route them to the AST path (CODE) rather than the LLM\n # document path — otherwise apm.yml (a .yml \"document\") would be LLM-extracted\n # and a package would split into duplicate file-anchored nodes (#1377).\n from graphify.manifest_ingest import is_package_manifest_path\n if is_package_manifest_path(path):\n return FileType.CODE\n # Compound extensions must be checked before simple suffix lookup\n if path.name.lower().endswith(\".blade.php\"):\n return FileType.CODE\n # A project may declare what an ambiguous extension means to it\n # (.graphifyrc `language.inc=php`, #2961); classify by the declared one.\n ext = effective_suffix(path).lower()\n if not ext:\n return _shebang_file_type(path)\n if ext in CODE_EXTENSIONS:\n return FileType.CODE\n if ext in PAPER_EXTENSIONS:\n # PDFs inside Xcode asset catalogs are vector icons, not papers\n if any(part.endswith(tuple(_ASSET_DIR_MARKERS)) for part in path.parts):\n return None\n return FileType.PAPER\n if ext in IMAGE_EXTENSIONS:\n return FileType.IMAGE\n if ext in DOC_EXTENSIONS:\n # Check if it's a converted paper\n if _looks_like_paper(path):\n return FileType.PAPER\n return FileType.DOCUMENT\n if ext in OFFICE_EXTENSIONS:\n return FileType.DOCUMENT\n if ext in GOOGLE_WORKSPACE_EXTENSIONS:\n return FileType.DOCUMENT\n if ext in VIDEO_EXTENSIONS:\n return FileType.VIDEO\n return None\n\n\n# Missing-pypdf is a process-global condition, not per-file, so warn at most\n# once per run — a corpus of many PDFs must not print the same hint N times.\n_pypdf_missing_warned = False\n# Unreadable PDFs are per-file, so warn once per path: detect() counts words and\n# the semantic pass reads the text again, and the same file must not warn twice.\n_pdf_failed_warned: set[str] = set()\n\n\ndef extract_pdf_text(path: Path) -> str:\n \"\"\"Extract plain text from a PDF file using pypdf.\n\n Returns ``\"\"`` when the file cannot be read (malformed, encrypted, ...), the\n same as for a PDF with no text, but warns on stderr naming the file and the\n error first (#4527), so a failed read is not mistaken for an empty document.\n \"\"\"\n if not _file_within_size_cap(path):\n return \"\"\n try:\n from pypdf import PdfReader\n except ImportError:\n global _pypdf_missing_warned\n if not _pypdf_missing_warned:\n _pypdf_missing_warned = True\n print(\n \"[graphify] WARNING: PDF text extraction skipped: 'pypdf' is not \"\n \"installed. Install the pdf extra: uv tool install 'graphifyy[pdf]'\",\n file=sys.stderr,\n )\n return \"\"\n try:\n reader = PdfReader(str(path))\n pages = []\n for page in reader.pages:\n text = page.extract_text()\n if text:\n pages.append(text)\n return \"\\n\".join(pages)\n except Exception as exc:\n key = str(path)\n if key not in _pdf_failed_warned:\n _pdf_failed_warned.add(key)\n print(\n f\"[graphify] WARNING: could not extract text from {path} \"\n f\"({type(exc).__name__}: {exc}); it contributes no content to \"\n \"the graph.\",\n file=sys.stderr,\n )\n return \"\"\n\n\n_W = \"{http://schemas.openxmlformats.org/wordprocessingml/2006/main}\"\n_MC_FALLBACK = \"{http://schemas.openxmlformats.org/markup-compatibility/2006}Fallback\"\n# Content controls and custom XML wrap paragraphs and tables without changing them.\n_DOCX_WRAPPERS = frozenset({f\"{_W}sdt\", f\"{_W}customXml\"})\n# Text under these is not part of the text as the document reads: tracked deletions, text\n# moved away, the ruby guide over its base text, text boxes (read as paragraphs of their\n# own) and the fallback copy Word writes of every text box.\n_DOCX_SKIPPED = frozenset({f\"{_W}del\", f\"{_W}moveFrom\", f\"{_W}rt\", f\"{_W}txbxContent\", _MC_FALLBACK})\n# The run children python-docx's Run.text reads, each as the text it stands for.\n_DOCX_RUN_TEXT = tuple(f\"{_W}{tag}\" for tag in (\"br\", \"cr\", \"noBreakHyphen\", \"ptab\", \"t\", \"tab\"))\n\n\ndef _docx_inside(element, stop, tags) -> bool:\n node = element.getparent()\n while node is not None and node is not stop:\n if node.tag in tags:\n return True\n node = node.getparent()\n return False\n\n\ndef _docx_paragraph_text(p) -> str:\n \"\"\"Paragraph.text reads only the runs directly under the paragraph, so it drops tracked\n insertions, content controls, simple fields and smart tags.\"\"\"\n return \"\".join(\n str(element)\n for element in p.iter(*_DOCX_RUN_TEXT)\n if not _docx_inside(element, p, _DOCX_SKIPPED)\n )\n\n\ndef _docx_blocks(container):\n \"\"\"Yield the w:p and w:tbl elements of a container in document order, looking through\n content controls and custom XML, with each text box's content after its paragraph.\"\"\"\n for child in container:\n if child.tag == f\"{_W}p\":\n yield child\n for box in child.iter(f\"{_W}txbxContent\"):\n if not _docx_inside(box, child, _DOCX_SKIPPED):\n yield from _docx_blocks(box)\n elif child.tag == f\"{_W}tbl\":\n yield child\n elif child.tag in _DOCX_WRAPPERS:\n content = child.find(f\"{_W}sdtContent\")\n yield from _docx_blocks(child if content is None else content)\n\n\ndef _docx_children(parent, tag: str):\n for child in parent:\n if child.tag == f\"{_W}{tag}\":\n yield child\n elif child.tag in _DOCX_WRAPPERS:\n content = child.find(f\"{_W}sdtContent\")\n yield from _docx_children(child if content is None else content, tag)\n\n\ndef _docx_cell_markdown(tc) -> str:\n \"\"\"A cell's text on one line with pipes escaped, so it cannot break its table row.\"\"\"\n parts = []\n for block in _docx_blocks(tc):\n if block.tag == f\"{_W}p\":\n parts.append(_docx_paragraph_text(block))\n else: # a nested table, flattened into the cell\n parts.extend(\n _docx_cell_markdown(cell)\n for row in _docx_children(block, \"tr\")\n for cell in _docx_children(row, \"tc\")\n )\n return \" \".join(\" \".join(part.split()) for part in parts if part.strip()).replace(\"|\", \"\\\\|\")\n\n\ndef docx_to_markdown(path: Path) -> str:\n \"\"\"Convert a .docx file to markdown text using python-docx.\"\"\"\n if not _zip_within_caps(path):\n return \"\"\n try:\n from docx import Document\n from docx.table import Table\n from docx.text.paragraph import Paragraph\n doc = Document(str(path))\n lines = []\n # Walk the body in document order, so a table stays under the heading it belongs to.\n for block in _docx_blocks(doc.element.body):\n if block.tag == f\"{_W}tbl\":\n # python-docx's rows repeat a merged cell across the grid columns it spans,\n # which keeps the header and the rows the same width.\n rows = [[_docx_cell_markdown(cell._tc) for cell in row.cells] for row in Table(block, doc).rows]\n if not rows:\n continue\n header = \"| \" + \" | \".join(rows[0]) + \" |\"\n sep = \"| \" + \" | \".join(\"---\" for _ in rows[0]) + \" |\"\n lines.extend([header, sep])\n for row in rows[1:]:\n lines.append(\"| \" + \" | \".join(row) + \" |\")\n continue\n para = Paragraph(block, doc)\n style = para.style.name if para.style else \"\"\n text = _docx_paragraph_text(block).strip()\n if not text:\n lines.append(\"\")\n continue\n if style.startswith(\"Heading 1\"):\n lines.append(f\"# {text}\")\n elif style.startswith(\"Heading 2\"):\n lines.append(f\"## {text}\")\n elif style.startswith(\"Heading 3\"):\n lines.append(f\"### {text}\")\n elif style.startswith(\"List\"):\n lines.append(f\"- {text}\")\n else:\n lines.append(text)\n return \"\\n\".join(lines)\n except ImportError:\n return \"\"\n except Exception:\n return \"\"\n\n\ndef xlsx_to_markdown(path: Path) -> str:\n \"\"\"Convert an .xlsx file to markdown text using openpyxl.\"\"\"\n if not _zip_within_caps(path):\n return \"\"\n try:\n import openpyxl\n wb = openpyxl.load_workbook(str(path), read_only=True, data_only=True)\n sections = []\n for sheet_name in wb.sheetnames:\n ws = wb[sheet_name]\n rows = []\n for row in ws.iter_rows(values_only=True):\n if all(cell is None for cell in row):\n continue\n rows.append([str(cell) if cell is not None else \"\" for cell in row])\n if not rows:\n continue\n sections.append(f\"## Sheet: {sheet_name}\")\n if len(rows) >= 1:\n header = \"| \" + \" | \".join(rows[0]) + \" |\"\n sep = \"| \" + \" | \".join(\"---\" for _ in rows[0]) + \" |\"\n sections.extend([header, sep])\n for row in rows[1:]:\n sections.append(\"| \" + \" | \".join(row) + \" |\")\n wb.close()\n return \"\\n\".join(sections)\n except ImportError:\n return \"\"\n except Exception:\n return \"\"\n\n\ndef xlsx_extract_structure(path: Path) -> dict:\n \"\"\"Extract structural nodes (sheets, named tables, column headers) from an .xlsx file.\n\n Returns a nodes/edges dict compatible with the graphify extract pipeline.\n Used in addition to xlsx_to_markdown so Claude sees both structure and content.\n \"\"\"\n def _nid(*parts: str) -> str:\n return re.sub(r\"[^a-z0-9_]\", \"_\", \"_\".join(p.lower() for p in parts).strip(\"_\"))\n\n try:\n import openpyxl\n except ImportError:\n return {\"nodes\": [], \"edges\": []}\n\n try:\n wb = openpyxl.load_workbook(str(path), read_only=False, data_only=True)\n except Exception:\n return {\"nodes\": [], \"edges\": []}\n\n # F-035: typo fix — was `_re.sub` (NameError, but unreachable because the\n # whole xlsx codepath is currently behind a feature flag / not yet wired\n # into the dispatcher). Before re-enabling this path, re-audit it for\n # zip/XML bombs (openpyxl is built on top of zipfile and lxml-style XML\n # parsing — a malicious .xlsx can blow up memory at load_workbook time).\n stem = re.sub(r\"[^a-z0-9]\", \"_\", path.stem.lower())\n str_path = str(path)\n file_nid = _nid(str_path)\n nodes: list[dict] = [{\"id\": file_nid, \"label\": path.name, \"file_type\": \"document\",\n \"source_file\": str_path, \"source_location\": None}]\n edges: list[dict] = []\n seen: set[str] = {file_nid}\n\n def _add(nid: str, label: str) -> None:\n if nid not in seen:\n seen.add(nid)\n nodes.append({\"id\": nid, \"label\": label, \"file_type\": \"document\",\n \"source_file\": str_path, \"source_location\": None})\n\n def _edge(src: str, tgt: str, relation: str) -> None:\n edges.append({\"source\": src, \"target\": tgt, \"relation\": relation,\n \"confidence\": \"EXTRACTED\", \"source_file\": str_path,\n \"source_location\": None, \"weight\": 1.0})\n\n for sheet_name in wb.sheetnames:\n ws = wb[sheet_name]\n sheet_nid = _nid(stem, sheet_name)\n _add(sheet_nid, f\"{sheet_name} (sheet)\")\n _edge(file_nid, sheet_nid, \"contains\")\n\n # Named Excel Tables (ListObjects)\n if hasattr(ws, \"tables\"):\n for tbl in ws.tables.values():\n tbl_nid = _nid(stem, sheet_name, tbl.name)\n _add(tbl_nid, tbl.name)\n _edge(sheet_nid, tbl_nid, \"contains\")\n # Column headers from table header row\n ref = tbl.ref # e.g. \"A1:D10\"\n if ref:\n try:\n from openpyxl.utils import range_boundaries\n min_col, min_row, max_col, _ = range_boundaries(ref)\n header_row = list(ws.iter_rows(min_row=min_row, max_row=min_row,\n min_col=min_col, max_col=max_col,\n values_only=True))\n if header_row:\n for col_name in header_row[0]:\n if col_name:\n col_nid = _nid(stem, tbl.name, str(col_name))\n _add(col_nid, str(col_name))\n _edge(tbl_nid, col_nid, \"contains\")\n except Exception:\n pass\n else:\n # Fallback: first non-empty row as column headers\n for row in ws.iter_rows(max_row=1, values_only=True):\n for cell in row:\n if cell:\n col_nid = _nid(stem, sheet_name, str(cell))\n _add(col_nid, str(cell))\n _edge(sheet_nid, col_nid, \"contains\")\n break\n\n try:\n wb.close()\n except Exception:\n pass\n\n return {\"nodes\": nodes, \"edges\": edges}\n\n\ndef convert_office_file(path: Path, out_dir: Path, root: \"Path | None\" = None) -> Path | None:\n \"\"\"Convert a .docx or .xlsx to a markdown sidecar in out_dir.\n\n Returns the path of the converted .md file, or None if conversion failed\n or the required library is not installed.\n \"\"\"\n ext = path.suffix.lower()\n if ext == \".docx\":\n text = docx_to_markdown(path)\n elif ext == \".xlsx\":\n text = xlsx_to_markdown(path)\n else:\n return None\n\n if not text.strip():\n return None\n\n out_dir.mkdir(parents=True, exist_ok=True)\n # Use a stable name derived from the original path to avoid collisions.\n # Hash the path RELATIVE to the scan root, not the absolute path: the\n # absolute form salts the name with the checkout location, so the same\n # tracked .xlsx in two clones/worktrees emits two differently-named,\n # byte-identical sidecars — unbounded duplicates when graphify-out/ is\n # committed, each ingested as a distinct source doc (#2059). The relative\n # path still disambiguates same-stem files in different directories.\n # Normalize to NFC before hashing: on macOS (HFS+/APFS) os.walk/rglob return\n # filenames in NFD, while Python string literals and directly-constructed\n # Path objects are NFC, so the same source file would otherwise hash to\n # different sidecar names across runs — making --update treat every Office\n # file as new and re-extract it (#1226).\n import hashlib\n import unicodedata\n if root is None:\n # Default layout: out_dir is \u003croot>/\u003cgraphify-out>/converted.\n root = out_dir.parent.parent\n try:\n key = path.resolve().relative_to(Path(root).resolve()).as_posix()\n except (ValueError, OSError):\n # Not under the scan root (custom GRAPHIFY_OUT layouts, --include\n # sources, direct API callers): keep the previous absolute form rather\n # than guessing, so behavior is unchanged for those cases.\n key = str(path.resolve())\n normalized_path = unicodedata.normalize(\"NFC\", key)\n name_hash = hashlib.sha256(normalized_path.encode()).hexdigest()[:8]\n out_path = out_dir / f\"{path.stem}_{name_hash}.md\"\n # Skip re-writing only when the sidecar is present AND at least as new as the\n # source. detect_incremental tracks the SIDECAR (not the Office source), so a\n # sidecar that is never rewritten after the source changes leaves the doc\n # reported \"unchanged\" forever and freezes the graph (#1649). Re-converting\n # when the source is newer bumps the sidecar's mtime/content, which the\n # incremental hash check then correctly picks up. An unchanged source keeps\n # its (newer-or-equal) sidecar untouched so it never churns (#1226).\n try:\n if out_path.exists() and os.stat(_os_path(out_path)).st_mtime >= os.stat(_os_path(path)).st_mtime:\n return out_path\n except OSError:\n if out_path.exists():\n return out_path\n out_path.write_text(\n f\"\u003c!-- converted from {path.name} -->\\n\\n{text}\",\n encoding=\"utf-8\",\n )\n return out_path\n\n\ndef count_words(path: Path) -> int:\n try:\n ext = path.suffix.lower()\n if ext in _BINARY_MEDIA_EXTENSIONS:\n return 0\n if ext == \".pdf\":\n return len(extract_pdf_text(path).split())\n if ext == \".docx\":\n return len(docx_to_markdown(path).split())\n if ext == \".xlsx\":\n return len(xlsx_to_markdown(path).split())\n # Only regular files may be opened. A repository can contain named\n # pipes, sockets and device nodes, and `clone \u003cgithub-url>` exists to\n # point the scan at trees the operator did not write. open() on a FIFO\n # with no writer BLOCKS FOREVER — it never raises, so the except below\n # cannot help, and `graphify update` hangs with no output. os.stat\n # follows symlinks on purpose: a link pointing at a FIFO blocks exactly\n # like the FIFO itself.\n if not stat.S_ISREG(os.stat(_os_path(path)).st_mode):\n return 0\n with open(_os_path(path), encoding=\"utf-8\", errors=\"ignore\") as f:\n return len(f.read().split())\n except Exception:\n return 0\n\n\ndef _is_regular_file(path: Path) -> bool:\n \"\"\"True only for regular files (symlinks followed).\n\n Named pipes, sockets and device nodes must never reach a reader:\n ``open()`` on a FIFO with no writer blocks forever and never raises, so a\n single ``pipe.py`` in a scanned repository hangs the run with no output.\n A symlink is resolved deliberately — a link pointing at a FIFO blocks\n exactly like the FIFO itself. A path that cannot be stat'ed is treated as\n not readable rather than raising.\n \"\"\"\n try:\n return stat.S_ISREG(os.stat(path).st_mode)\n except OSError:\n return False\n\n\n# Directory names to always skip - venvs, caches, build artifacts, deps\n_SKIP_DIRS = {\n \"venv\", \".venv\", # \"env\"/\".env\"/\"*_env\" are gated on venv markers below (#2058)\n \"node_modules\", \"__pycache__\", \".git\",\n \"dist\", \"build\", \"target\", # bare \"out\" is gated on build-output evidence below (#3347)\n \"site-packages\", \"lib64\",\n \".pytest_cache\", \".mypy_cache\", \".ruff_cache\",\n \".tox\", \".nox\", \".eggs\", \"*.egg-info\", # nox is tox's successor, same .nox/ venv shape (#1804)\n \"graphify-out\", # never treat the default output as source input (#524)\n # Coverage/test-artefact dirs — generated, never architecturally meaningful\n \"lcov-report\", # Vitest/Istanbul/nyc HTML reports (#870);\n # bare \"coverage\" is gated on report\n # artefacts below (#2339)\n \"visual-tests\", \"visual-test\", # Playwright/visual-regression bundles (#869)\n \"__snapshots__\", # Jest/Vitest snapshot dir (unambiguous)\n \"storybook-static\", # Storybook production build output\n \"dist-protected\", # Protected dist variants (same noise as dist)\n # Framework cache/build dirs — generated, never architecturally meaningful (#873)\n \".next\", \".nuxt\", \".turbo\", \".angular\",\n \".idea\", \".cache\", \".parcel-cache\", \".svelte-kit\", \".terraform\", \".serverless\",\n \".graphify\", # graphify's own extraction cache — never index self-generated data\n \".obsidian\", \".smart-env\", # Obsidian vault metadata and plugin caches (#2493)\n \".worktrees\", # git worktree convention (#947) — sibling checkouts, always redundant\n}\n\n# Large generated files that are never useful to extract\n_SKIP_FILES = {\n \"package-lock.json\", \"yarn.lock\", \"pnpm-lock.yaml\",\n \"Cargo.lock\", \"poetry.lock\", \"Gemfile.lock\",\n \"composer.lock\", \"go.sum\", \"go.work.sum\",\n # Removed allowlist config (#2112) — no longer consumed, so keep a leftover\n # file out of the unclassified list instead of surfacing it as scan input.\n \".graphifyinclude\",\n}\n\n# Files a JS/TS toolchain reads to resolve modules: workspace package roots,\n# tsconfig `paths`/`baseUrl`, and the installed dependency tree. Changing one of\n# these can change a .ts/.tsx/.js result without touching a single source file, so\n# the no-change gate in watch._rebuild_code() has to see them (#4740).\n# tsconfig*.json / jsconfig.json / package.json are ordinary scan input (they land\n# in the \"code\" bucket and are stamped in the manifest like any other file). The\n# lock files are in _SKIP_FILES, so detect() records them separately instead.\n_RESOLUTION_INPUT_NAMES = frozenset({\n \"package.json\", \"package-lock.json\", \"pnpm-lock.yaml\", \"yarn.lock\",\n \"jsconfig.json\",\n})\n_TS_CONFIG_NAME_RE = re.compile(r\"^tsconfig(\\.[^/\\\\]+)?\\.json$\")\n\n\ndef is_resolution_input(path: \"Path | str\") -> bool:\n \"\"\"True when ``path`` steers JS/TS module resolution (#4740).\n\n Deliberately over-approximate: a false positive only costs a slower run, a\n false negative ships a stale graph.\n \"\"\"\n name = Path(path).name.lower()\n return name in _RESOLUTION_INPUT_NAMES or bool(_TS_CONFIG_NAME_RE.match(name))\n\n\n# A bare \"snapshots\" dir is a Jest/Vitest artifact only when it actually holds\n# snapshot files or lives directly under a JS test root. Elsewhere it is often a\n# real code namespace (e.g. Rails app/services/snapshots/), so pruning it by name\n# silently dropped legitimate source from the graph (#1666). \"__snapshots__\" stays\n# unconditionally pruned above; only the ambiguous bare name is gated here.\n_JS_SNAPSHOT_TEST_ROOTS = frozenset({\"__tests__\", \"__test__\"})\n\n# Platforms whose project-scope skills dir is one level below the hidden dir\n# (install.py: pi -> .pi/agent/skills, kilo -> .config/kilo/skills).\n_NESTED_SKILL_HOLDERS = frozenset({(\".pi\", \"agent\"), (\".config\", \"kilo\")})\n\n# Single files `graphify install` writes whole into a project (#4057): the\n# always-on rules/steering/workflow files and the opencode/kilo hook plugins.\n# Matched on the hidden holder dir plus the exact relative path, never on the\n# bare file name, so e.g. docs/graphify.md or src/plugins/graphify.js stay.\n# Files graphify only adds a section or entry to (AGENTS.md, CLAUDE.md,\n# settings.json, ...) belong to the user and are not listed here.\n_GRAPHIFY_INSTALLED_FILES = frozenset({\n (\".agents\", \"rules\", \"graphify.md\"), # antigravity\n (\".agents\", \"workflows\", \"graphify.md\"), # antigravity\n (\".cursor\", \"rules\", \"graphify.mdc\"), # cursor\n (\".kilo\", \"plugins\", \"graphify.js\"), # kilo\n (\".kiro\", \"steering\", \"graphify.md\"), # kiro\n (\".opencode\", \"plugins\", \"graphify.js\"), # opencode\n (\".windsurf\", \"rules\", \"graphify.md\"), # devin\n})\n\n\ndef _is_installed_graphify_file(path: \"Path\") -> bool:\n \"\"\"True for a single file `graphify install` wrote into the project (#4057).\"\"\"\n return path.parts[-3:] in _GRAPHIFY_INSTALLED_FILES\n\n# Files a coverage tool writes into its own output dir. Any one of them is proof\n# the directory is generated: lcov (lcov.info), nyc/Istanbul (coverage-final.json,\n# clover.xml, the lcov-report/ subtree), coverage.py (coverage.xml, .coverage),\n# JaCoCo/Cobertura (jacoco.xml, cobertura-coverage.xml).\n_COVERAGE_ARTIFACT_FILES = frozenset({\n \"lcov.info\", \"coverage-final.json\", \"coverage-summary.json\",\n \"clover.xml\", \"coverage.xml\", \"cobertura-coverage.xml\", \"jacoco.xml\",\n \".coverage\", \"index.html\",\n})\n_COVERAGE_ARTIFACT_DIRS = frozenset({\"lcov-report\", \"html-report\"})\n\n\ndef _has_coverage_artifacts(d: \"Path\") -> bool:\n \"\"\"True only when *d* holds files a coverage tool actually generated.\n\n ``coverage`` is a legitimate package name (a Python package, a Go/Rust module,\n a domain namespace), so pruning it by name alone silently drops real source —\n an entire 5-module package in #2339, with its dependents left in the graph so\n queries still returned plausible neighbours. Prune it only on real evidence,\n mirroring the ``snapshots``/``env`` gating (#1666/#2058): a coverage report\n file, or an Istanbul/lcov HTML report subtree.\n \"\"\"\n try:\n for name in _COVERAGE_ARTIFACT_FILES:\n if (d / name).is_file():\n return True\n for name in _COVERAGE_ARTIFACT_DIRS:\n if (d / name).is_dir():\n return True\n except OSError:\n pass\n return False\n\n\n# Files only a compiler/bundler writes. Any one of them inside an ``out/``\n# tree is proof the directory is generated output rather than source.\n_BUILD_OUTPUT_SUFFIXES = (\n \".class\", \".jar\", \".o\", \".obj\", \".exe\", \".dll\", \".pyc\",\n \".js.map\", \".css.map\", \".d.ts\",\n)\n\n\ndef _has_build_output_markers(d: \"Path\") -> bool:\n \"\"\"True only when *d* holds evidence a build tool generated it.\n\n ``out`` is a mainstream SOURCE convention in hexagonal / ports-and-adapters\n codebases — ``adapter/out/`` and ``port/out/`` hold the entire outbound\n layer (persistence adapters, entities, outbound ports) — so pruning it by\n name alone silently drops an architectural layer from the graph: 196 of\n 655 .java files on the #3347 corpus, exit code 0, no warning. Prune only\n on real generated-output evidence, mirroring the ``env``/``coverage``/\n ``snapshots`` gating (#1666/#2058/#2339): the IntelliJ ``out/production``\n layout, a Next.js static-export ``_next/`` tree, or compiled artifacts in\n the top two directory levels. An ``out`` that cannot be verified is kept —\n the same keep-on-doubt bias every other gated name has.\n \"\"\"\n try:\n if (d / \"production\").is_dir(): # IntelliJ IDEA compile output layout\n return True\n if (d / \"_next\").is_dir(): # `next export` static-site output\n return True\n subdirs: list = []\n with os.scandir(d) as it:\n for entry in it:\n if entry.is_file() and entry.name.lower().endswith(_BUILD_OUTPUT_SUFFIXES):\n return True\n if entry.is_dir():\n subdirs.append(entry.path)\n # Probe one more level across EVERY subdirectory (a wide TS outDir may\n # keep its compiled files only under later-sorted module dirs), bounded\n # by total entries scanned rather than by subdirectory count so a\n # pathological tree still costs O(1)-ish.\n budget = 4000\n for sub in subdirs:\n if budget \u003c= 0:\n break\n with os.scandir(sub) as it:\n for entry in it:\n budget -= 1\n if entry.is_file() and entry.name.lower().endswith(_BUILD_OUTPUT_SUFFIXES):\n return True\n if budget \u003c= 0:\n break\n except OSError:\n pass\n return False\n\n\ndef _has_venv_markers(d: \"Path\") -> bool:\n \"\"\"True only when *d* has actual virtualenv/conda structure on disk.\n\n ``env``/``.env``/``*_env`` is a real source-directory convention (UVM/ASIC\n verification trees, and others), so pruning it by name alone silently drops\n legitimate source with no trace (#2058). Prune it only on real evidence: a\n ``pyvenv.cfg``, an ``activate`` script, a ``lib/python*`` tree, or conda's\n ``conda-meta/`` (``conda create -p ./env`` writes no pyvenv.cfg).\n \"\"\"\n try:\n if (d / \"pyvenv.cfg\").is_file():\n return True\n if (d / \"bin\" / \"activate\").is_file() or (d / \"Scripts\" / \"activate\").is_file():\n return True\n if next(d.glob(\"lib/python*\"), None) is not None:\n return True\n if (d / \"conda-meta\").is_dir():\n return True\n except OSError:\n pass\n return False\n\n\n# Files only MSBuild/NuGet write. ``obj/`` gets the restore outputs (every\n# SDK-style restore writes project.assets.json) and, for legacy non-SDK\n# projects, the per-configuration FileListAbsolute.txt. ``bin/`` gets the\n# runtime manifests the SDK emits next to every executable, plus the PDBs.\n# .dll/.exe are deliberately absent: a checked-in tools/bin/ of vendored\n# binaries next to its wrapper scripts is source, not output.\n_MSBUILD_OBJ_MARKERS = (\n \"project.assets.json\", \".nuget.dgspec.json\", \".nuget.g.props\",\n \".filelistabsolute.txt\",\n)\n_MSBUILD_BIN_MARKERS = (\".deps.json\", \".runtimeconfig.json\", \".pdb\")\n\n\ndef _has_msbuild_markers(d: \"Path\", markers: tuple, max_depth: int) -> bool:\n \"\"\"True when a file ending in one of *markers* sits within *max_depth*\n directory levels below *d*.\n\n ``bin`` and ``obj`` are MSBuild's output and intermediate dirs, and an\n unignored C# project carries both after its first build: the generated\n GlobalUsings/AssemblyInfo sources land in the graph as file nodes with\n import edges to every implicit namespace, and the build-copied\n appsettings*.json are scanned twice. ``dotnet new`` writes no .gitignore,\n so nothing else stops them. Neither name is safe to prune alone — ``bin``\n is source in Rust's src/bin/, Ruby binstubs and script dirs — so prune\n only on MSBuild's own artifacts, the same keep-on-doubt gating ``out``\n gets (#3347). The walk is bounded by entries scanned, not just depth.\n \"\"\"\n budget = 4000\n pending = [(str(d), 0)]\n while pending:\n current, depth = pending.pop()\n try:\n with os.scandir(current) as it:\n for entry in it:\n budget -= 1\n if budget \u003c= 0:\n return False\n if entry.is_file():\n if entry.name.lower().endswith(markers):\n return True\n elif depth \u003c max_depth and entry.is_dir(follow_symlinks=False):\n pending.append((entry.path, depth + 1))\n except OSError:\n continue\n return False\n\n\ndef _is_noise_dir(part: str, parent: \"Path | None\" = None) -> bool:\n \"\"\"Return True if this directory name looks like a venv, cache, or dep dir.\"\"\"\n if part in _SKIP_DIRS:\n return True\n if part in (\"env\", \".env\") or part.endswith(\"_env\"):\n # Ambiguous: a real venv OR a real source dir. Prune only on actual venv\n # evidence, mirroring the \"snapshots\" gating (#1666/#2058).\n if parent is None:\n return False # cannot verify; keep a possibly-real code dir\n return _has_venv_markers(parent / part)\n if part == \"coverage\":\n # Ambiguous: a generated report dir OR a real package named coverage.\n # Prune only on actual coverage-artefact evidence (#2339).\n if parent is None:\n return False # cannot verify; keep a possibly-real code dir\n return _has_coverage_artifacts(parent / part)\n if part == \"out\":\n # Ambiguous: compiler/bundler output (IntelliJ, `next export`, a TS\n # outDir) OR the outbound layer of a hexagonal codebase\n # (adapter/out/, port/out/). Prune only on actual build-output\n # evidence (#3347).\n if parent is None:\n return False # cannot verify; keep a possibly-real code dir\n return _has_build_output_markers(parent / part)\n if part in (\"obj\", \"bin\"):\n # Ambiguous: MSBuild intermediate/output dirs OR real source (Rust's\n # src/bin/, script dirs). Depths reach a legacy obj/\u003cconfig>/ and a\n # RID-specific bin/\u003cconfig>/\u003ctfm>/\u003crid>/.\n if parent is None:\n return False # cannot verify; keep a possibly-real code dir\n if part == \"obj\":\n return _has_msbuild_markers(parent / part, _MSBUILD_OBJ_MARKERS, 1)\n return _has_msbuild_markers(parent / part, _MSBUILD_BIN_MARKERS, 3)\n if part == \"snapshots\":\n # Prune only when it looks like an actual JS/Vitest snapshot dir.\n if parent is None:\n return False # cannot verify; keep a possibly-real code dir\n snap_dir = parent / part\n if parent.name in _JS_SNAPSHOT_TEST_ROOTS:\n return True\n try:\n if next(snap_dir.glob(\"*.snap\"), None) is not None:\n return True\n except OSError:\n pass\n return False\n # Catch *_venv (unambiguous — \"venv\" is always a virtualenv signal). \"*_env\"\n # is gated on markers above (#2058), not pruned by name.\n if part.endswith(\"_venv\"):\n return True\n if part.endswith(\".egg-info\"):\n return True\n # worktrees/ nested inside a dotted dir (e.g. .claude/worktrees/, .git/worktrees/)\n if part == \"worktrees\" and parent is not None and parent.name.startswith(\".\"):\n return True\n # graphify's own skill folder, as `graphify install --project` writes it\n # (#4057): \u003chidden dir>/skills/graphify/ (.claude, .codex, .agents, ...),\n # .pi/agent/skills/graphify/, .config/kilo/skills/graphify/ and\n # .aider/graphify/. Matched on the whole path shape rather than the bare\n # names \"skills\"/\"graphify\" (#2479), so graphify's own source tree\n # (graphify/skills/\u003chost>/), a top-level skills/graphify/ and the user's\n # other skills under .claude/skills/ are still indexed.\n if part == \"graphify\" and parent is not None:\n if parent.name == \".aider\":\n return True\n if parent.name == \"skills\":\n holder = parent.parent\n if holder.name.startswith(\".\") and holder.name != \"..\":\n return True\n return (holder.parent.name, holder.name) in _NESTED_SKILL_HOLDERS\n return False\n\n\n_VCS_MARKERS = (\".git\", \".hg\", \".svn\", \"_darcs\", \".fossil\")\n\n\ndef _nfc(text: str) -> str:\n \"\"\"Normalize text to NFC so ignore matching survives Unicode form drift.\n\n macOS (APFS/HFS+) returns filenames in NFD: \"ç\" comes back as \"c\" +\n U+0327 COMBINING CEDILLA. Editors write ignore files in NFC, where the\n same \"ç\" is the single codepoint U+00E7. The two render identically and\n compare unequal, so a pattern like `Orçamento/` silently fails to exclude\n the directory it names — the files are scanned and, for docs/PDFs, sent\n to an LLM despite an explicit rule against it.\n\n Both sides are normalized to NFC before any fnmatch call. NFC is the form\n Linux and Windows already use, so this is a no-op there and only repairs\n the macOS mismatch.\n \"\"\"\n return unicodedata.normalize(\"NFC\", text)\n\n\ndef _parse_gitignore_line(raw: str) -> str:\n \"\"\"Parse one raw line from a .graphifyignore file per gitignore spec.\n\n - Strip newline chars\n - Strip inline comments (whitespace + # suffix), but only when # is\n preceded by whitespace — so path#with#hash.py is preserved\n - Unescape \\\\# to literal #\n - Remove trailing spaces unless escaped with backslash\n - Strip leading whitespace\n - Return empty string for blank lines and full-line comments\n \"\"\"\n line = raw.rstrip(\"\\n\\r\")\n line = line.lstrip()\n if not line or line.startswith(\"#\"):\n return \"\"\n # Strip inline comments: require whitespace before # (gitignore extension)\n line = re.sub(r\"\\s+#+[^\\\\].*$\", \"\", line)\n # Unescape \\# → literal #\n line = line.replace(\"\\\\#\", \"#\")\n # Remove unescaped trailing spaces (per gitignore spec)\n line = re.sub(r\"(?\u003c!\\\\) +$\", \"\", line)\n return _nfc(line)\n\n\ndef _is_valid_git_marker(dot_git: Path) -> bool:\n \"\"\"Whether ``dot_git`` is a ``.git`` that git itself would accept (#4444).\n\n A ``.git`` directory needs ``HEAD``, ``objects/`` and ``refs/``, the same\n entries git's own repository check looks for. A ``.git`` file (linked\n worktree, submodule) must start with ``gitdir:``. An empty or partial\n ``.git`` directory, e.g. one left in a home directory, is not a repo and\n must not become the ignore-walk ceiling. No git subprocess is used.\n \"\"\"\n try:\n if dot_git.is_dir():\n return (dot_git / \"HEAD\").is_file() and (dot_git / \"objects\").is_dir() and (dot_git / \"refs\").is_dir()\n if dot_git.is_file():\n with dot_git.open(\"rb\") as fh:\n return fh.read(7) == b\"gitdir:\"\n except OSError:\n pass\n return False\n\n\ndef _has_vcs_marker(directory: Path) -> bool:\n for marker in _VCS_MARKERS:\n path = directory / marker\n if marker == \".git\":\n if _is_valid_git_marker(path):\n return True\n elif path.exists():\n return True\n return False\n\n\ndef _find_vcs_root(start: Path) -> Path | None:\n \"\"\"Walk upward from start; return the first directory containing a VCS marker.\n\n ``.git`` only counts when it is a valid repo marker (#4444).\n \"\"\"\n current = start.resolve()\n home = Path.home()\n while True:\n if _has_vcs_marker(current):\n return current\n parent = current.parent\n if parent == current or current == home:\n return None\n current = parent\n\n\ndef _path_identity(path: Path) -> str:\n \"\"\"Portable comparison key for an existing filesystem path.\"\"\"\n return _nfc(os.path.normcase(os.path.abspath(os.fspath(path))))\n\n\ndef _git_tracked_path_keys(root: Path) -> tuple[set[str], set[str]]:\n \"\"\"Return tracked-file keys and their ancestor-directory keys under *root*.\n\n Gitignore rules do not apply to paths already present in Git's index. Ask\n Git once per scan/predicate construction with NUL-delimited output so every\n valid filename is preserved. Missing Git, a non-Git VCS marker, command\n failure, and malformed output all fail closed to the historical ignore\n behavior rather than making discovery fail (#2759).\n \"\"\"\n root = root.resolve()\n vcs_root = _find_vcs_root(root)\n if vcs_root is None or not (vcs_root / \".git\").exists():\n return set(), set()\n try:\n proc = subprocess.run(\n [\"git\", \"-C\", str(vcs_root), \"ls-files\", \"-z\", \"--cached\"],\n stdin=subprocess.DEVNULL,\n stdout=subprocess.PIPE,\n stderr=subprocess.DEVNULL,\n check=False,\n timeout=30,\n env={**os.environ, \"GIT_OPTIONAL_LOCKS\": \"0\"},\n )\n except (OSError, subprocess.SubprocessError):\n return set(), set()\n if proc.returncode != 0:\n return set(), set()\n\n tracked_files: set[str] = set()\n tracked_dirs: set[str] = set()\n for raw in proc.stdout.split(b\"\\0\"):\n if not raw:\n continue\n path = Path(os.path.abspath(vcs_root / os.fsdecode(raw)))\n try:\n path.relative_to(root)\n except ValueError:\n continue\n # Deleted index entries and submodule gitlinks are not discoverable\n # files. Symlinks to regular files remain eligible; the existing\n # in-root target guard still decides whether they may enter the corpus.\n if not _is_regular_file(path):\n continue\n tracked_files.add(_path_identity(path))\n parent = path.parent\n while parent != root:\n parent_key = _path_identity(parent)\n if parent_key in tracked_dirs:\n break # its ancestors were added with the first file below it\n tracked_dirs.add(parent_key)\n parent = parent.parent\n return tracked_files, tracked_dirs\n\n\ndef _git_info_exclude(vcs_root: Path) -> Path | None:\n \"\"\"Resolve ``$GIT_DIR/info/exclude`` for the repo rooted at ``vcs_root``.\n\n ``info/exclude`` is where git records local-only, uncommitted excludes — and\n where ``git worktree add`` writes nested worktree paths — so a repo can ignore\n a directory without any ``.gitignore`` entry. graphify only read\n ``.gitignore``/``.graphifyignore``, so it walked into those worktree copies and\n the graph exploded (#1810). Handles the linked-worktree/submodule case where\n ``.git`` is a file (``gitdir: \u003cpath>``) and the real excludes live in the\n shared common git dir. Returns None when there is no readable exclude file.\n \"\"\"\n dot_git = vcs_root / \".git\"\n git_dir: Path | None = None\n if dot_git.is_dir():\n git_dir = dot_git\n elif dot_git.is_file():\n try:\n content = dot_git.read_text(encoding=\"utf-8\", errors=\"ignore\").strip()\n except OSError:\n content = \"\"\n if content.startswith(\"gitdir:\"):\n gd = Path(content[len(\"gitdir:\"):].strip())\n if not gd.is_absolute():\n gd = (vcs_root / gd).resolve()\n git_dir = gd\n # A linked worktree's gitdir holds a `commondir` file pointing at the\n # shared git dir, where info/exclude actually lives.\n commondir = gd / \"commondir\"\n if commondir.exists():\n try:\n cd_raw = commondir.read_text(encoding=\"utf-8\", errors=\"ignore\").strip()\n except OSError:\n cd_raw = \"\"\n if cd_raw:\n cd = Path(cd_raw)\n git_dir = cd if cd.is_absolute() else (gd / cd).resolve()\n if git_dir is None:\n return None\n exclude = git_dir / \"info\" / \"exclude\"\n return exclude if exclude.is_file() else None\n\n\ndef _linked_worktree_main_root(directory: Path) -> Path | None:\n \"\"\"Return the main worktree's root for a linked-worktree dir, else None.\n\n ``git worktree add`` gives the linked checkout a ``.git`` FILE reading\n ``gitdir: \u003cmain>/.git/worktrees/\u003cname>``, and writes a ``commondir`` file\n inside that gitdir pointing back at the shared git dir ``\u003cmain>/.git`` —\n so the main checkout's root is the common dir's parent. A submodule's\n ``.git`` file points at ``\u003csuper>/.git/modules/\u003cname>``, which carries no\n ``commondir``; a bare repo's shared git dir is not named ``.git`` (it has\n no main working tree); and a moved/deleted main checkout leaves ``cd``\n dangling — all three fall through to None and stay scannable (#4683).\n No git subprocess is used.\n \"\"\"\n dot_git = directory / \".git\"\n try:\n if not dot_git.is_file():\n return None\n content = dot_git.read_text(encoding=\"utf-8\", errors=\"ignore\").strip()\n except OSError:\n return None\n if not content.startswith(\"gitdir:\"):\n return None\n gd = Path(content[len(\"gitdir:\"):].strip())\n if not gd.is_absolute():\n try:\n gd = (directory / gd).resolve()\n except (OSError, RuntimeError):\n return None\n try:\n cd_raw = (gd / \"commondir\").read_text(encoding=\"utf-8\", errors=\"ignore\").strip()\n except OSError:\n return None\n if not cd_raw:\n return None\n cd = Path(cd_raw)\n if not cd.is_absolute():\n try:\n cd = (gd / cd).resolve()\n except (OSError, RuntimeError):\n return None\n if cd.name != \".git\" or not cd.is_dir():\n return None\n return cd.parent\n\n\ndef _git_global_excludes(vcs_root: Path) -> Path | None:\n \"\"\"Resolve git's user-wide excludes file (``core.excludesFile``).\n\n This is where people keep machine-wide rules for local settings and secret\n files so they don't repeat them in every repo; git ignores those files but\n graphify read them (#4335). Asks git for the configured path (``--path``\n expands ``~``) and falls back to git's default,\n ``$XDG_CONFIG_HOME/git/ignore`` or ``~/.config/git/ignore``, when the key\n is unset or git is unavailable. Only applies inside a git repo, like git.\n Returns None when there is no readable file.\n \"\"\"\n if not (vcs_root / \".git\").exists():\n return None\n configured = \"\"\n try:\n proc = subprocess.run(\n [\"git\", \"-C\", str(vcs_root), \"config\", \"--path\", \"--get\", \"core.excludesFile\"],\n stdin=subprocess.DEVNULL,\n stdout=subprocess.PIPE,\n stderr=subprocess.DEVNULL,\n check=False,\n timeout=10,\n env={**os.environ, \"GIT_OPTIONAL_LOCKS\": \"0\"},\n )\n if proc.returncode == 0:\n configured = proc.stdout.decode(\"utf-8\", errors=\"replace\").strip()\n except (OSError, subprocess.SubprocessError):\n pass\n if configured:\n path = Path(configured).expanduser()\n if not path.is_absolute():\n path = vcs_root / path\n else:\n xdg = os.environ.get(\"XDG_CONFIG_HOME\")\n base = Path(xdg) if xdg and Path(xdg).is_absolute() else Path.home() / \".config\"\n path = base / \"git\" / \"ignore\"\n return path if path.is_file() else None\n\n\n_warned_ignore_encodings: set[str] = set()\n\n\ndef _read_ignore_text(path: Path) -> str:\n \"\"\"Read an ignore file, preferring UTF-8 but never silently dropping a rule.\n\n These files were read with ``errors=\"ignore\"``, which turns a mis-encoded\n byte into *no* byte. An ignore file saved in the host's ANSI codepage — the\n historical Notepad default on Windows, and still what ``Set-Content`` writes\n without ``-Encoding`` — is not valid UTF-8, so ``Or\\xe7amento/`` decoded to\n the pattern ``Oramento/``. That matches nothing, and nothing said so: the\n directory was scanned despite an explicit exclusion, which for a rule\n covering documents or PDFs means they reach the semantic pass anyway.\n\n So: UTF-8 (BOM-tolerant) first, since that is what the format should be and\n what every other reader here assumes. Only if that fails do we fall back to\n the host encoding, then to latin-1, which cannot fail and maps every byte to\n a codepoint — a rule spelled in some third encoding still comes out wrong,\n but it comes out *whole*, and the warning names the file so it is fixable.\n Decoding never raises, matching the previous contract.\n \"\"\"\n raw = path.read_bytes()\n try:\n return raw.decode(\"utf-8-sig\")\n except UnicodeDecodeError:\n pass\n # A BOM'd UTF-16 file (common from PowerShell `Set-Content` / Notepad \"Unicode\")\n # is not valid UTF-8, and latin-1 would map its interleaved NUL bytes to a\n # wall of \\x00, garbling every rule. Decode it as UTF-16 by its BOM first.\n if raw[:2] in (b\"\\xff\\xfe\", b\"\\xfe\\xff\"):\n try:\n return raw.decode(\"utf-16\")\n except UnicodeDecodeError:\n pass\n import locale\n import sys as _sys\n fallback = locale.getpreferredencoding(False) or \"latin-1\"\n for enc in (fallback, \"latin-1\"):\n try:\n text = raw.decode(enc)\n except (UnicodeDecodeError, LookupError):\n continue\n key = str(path)\n if key not in _warned_ignore_encodings:\n _warned_ignore_encodings.add(key)\n print(\n f\"[graphify] WARNING: {path} is not valid UTF-8; read it as \"\n f\"{enc} instead. Re-save it as UTF-8 — patterns with non-ASCII \"\n \"characters may not match as written.\",\n file=_sys.stderr,\n )\n return text\n return raw.decode(\"utf-8\", errors=\"ignore\")\n\n\ndef _load_dir_own_ignore(d: Path, *, gitignore: bool = True) -> list[tuple[Path, str]]:\n \"\"\"Read .gitignore/.graphifyignore directly inside *d* (not its ancestors).\n\n Merges .gitignore and .graphifyignore for this one directory (#1363):\n .gitignore is read first and .graphifyignore last, so .graphifyignore\n patterns (including `!` negations) win on conflict via last-match-wins;\n adding a .graphifyignore can only ever exclude MORE, never re-include a\n .gitignore-excluded file (#945 kept: a dir with only a .gitignore still\n gets sensible defaults).\n\n Shared by `_load_graphifyignore` (ancestor chain, loaded once before the\n scan) and the live os.walk loop in `detect()` (called per-directory as\n each descendant is visited), so nested ignore files *below* the scan\n root are honored too — previously only the scan root and its ancestors\n were read, so e.g. `vendor/sub/.gitignore` was silently ignored (#1206).\n \"\"\"\n patterns: list[tuple[Path, str]] = []\n for fname in ((\".gitignore\", \".graphifyignore\") if gitignore else (\".graphifyignore\",)):\n ignore_file = d / fname\n if ignore_file.exists():\n for raw in _read_ignore_text(ignore_file).splitlines():\n line = _parse_gitignore_line(raw)\n if line:\n patterns.append((d, line))\n return patterns\n\n\ndef _load_graphifyignore(root: Path, *, gitignore: bool = True) -> list[tuple[Path, str]]:\n \"\"\"Read .graphifyignore files and return (anchor_dir, pattern) pairs.\n\n Patterns are returned outer-first so that inner (closer) rules are\n appended last and win via last-match-wins semantics — matching gitignore\n behavior exactly.\n\n Walk ceiling: the nearest VCS root if inside a repo, otherwise the scan\n root itself (hermetic — no leakage across unrelated sibling projects).\n\n Covers the scan root and its ancestors only — directories *below* the\n scan root are picked up live during the os.walk in `detect()` instead,\n since they aren't known until the walk reaches them (#1206).\n \"\"\"\n root = root.resolve()\n ceiling = _find_vcs_root(root) or root\n\n # Collect ancestor dirs from ceiling down to root (outer → inner)\n dirs: list[Path] = []\n current = root\n while True:\n dirs.append(current)\n if current == ceiling:\n break\n current = current.parent\n dirs.reverse() # ceiling first, scan root last\n\n patterns: list[tuple[Path, str]] = []\n\n # git's user-wide excludes file ranks below info/exclude, so it goes first\n # of all, anchored at the VCS root like info/exclude (#4335).\n global_excludes = _git_global_excludes(ceiling) if gitignore else None\n if global_excludes is not None:\n for raw in _read_ignore_text(global_excludes).splitlines():\n line = _parse_gitignore_line(raw)\n if line:\n patterns.append((ceiling, line))\n\n # $GIT_DIR/info/exclude is repo-root-scoped and, per git, ranks below every\n # per-directory .gitignore/.graphifyignore — so load it first (lowest priority\n # under last-match-wins) anchored at the VCS root, letting a nearer `!`\n # re-include still override it (#1810).\n info_exclude = _git_info_exclude(ceiling) if gitignore else None\n if info_exclude is not None:\n for raw in _read_ignore_text(info_exclude).splitlines():\n line = _parse_gitignore_line(raw)\n if line:\n patterns.append((ceiling, line))\n\n for d in dirs:\n patterns.extend(_load_dir_own_ignore(d, gitignore=gitignore))\n return patterns\n\n\n# Parsed-pattern cache: raw pattern string -> (negated, directory_only,\n# path_relative, stripped_pattern). Ignore patterns are re-evaluated for every\n# walked entry; parsing the same strings per entry per scan was pure waste.\n# Plain dict (no LRU): the universe of keys is the distinct pattern lines in\n# the corpus's ignore files, which is small and bounded per scan. A long-lived\n# `graphify watch` process spanning many repos could still accumulate keys over\n# time, so cap it and clear wholesale on overflow (parsing is cheap, so a rare\n# full re-fill costs nothing that matters).\n_PARSED_PATTERN_CACHE: dict[str, tuple[bool, bool, bool, str]] = {}\n_PARSED_PATTERN_CACHE_MAX = 100_000\n\n\ndef _parse_ignore_pattern(pattern: str) -> tuple[bool, bool, bool, str]:\n \"\"\"Split one gitignore-style pattern into its matching flags, cached.\n\n Returns (negated, directory_only, path_relative, stripped). ``stripped``\n is empty for patterns that match nothing (bare \"!\", bare \"/\").\n \"\"\"\n got = _PARSED_PATTERN_CACHE.get(pattern)\n if got is None:\n negated = pattern.startswith(\"!\")\n raw = pattern[1:] if negated else pattern\n directory_only = raw.endswith(\"/\")\n path_relative = \"/\" in raw.rstrip(\"/\")\n got = (negated, directory_only, path_relative, raw.strip(\"/\"))\n if len(_PARSED_PATTERN_CACHE) >= _PARSED_PATTERN_CACHE_MAX:\n _PARSED_PATTERN_CACHE.clear()\n _PARSED_PATTERN_CACHE[pattern] = got\n return got\n\n\n# PurePath.relative_to compares casefolded parts on Windows only; mirror that\n# exactly, but skip the normcase pass entirely where it is a no-op (POSIX).\n_CASEFOLD_PATHS = os.path.normcase(\"Aa\") != \"Aa\"\n\n\ndef _lexical_relative(\n target: Path, target_parts: tuple[str, ...], anchor: Path\n) -> str | None:\n \"\"\"String-space equivalent of ``_nfc(str(target.relative_to(anchor)).replace(os.sep, \"/\"))``.\n\n Returns None where ``relative_to`` raises ValueError (target not under\n anchor). ``relative_to`` is purely lexical — a parts-prefix check — so this\n performs the same comparison on the already-parsed ``parts`` tuples instead\n of constructing a Path object (and, on 3.12+, walking ``parents``\n quadratically) per pattern per scanned entry. That construction was the\n dominant cost of large scans (see CHANGELOG: 76k-file vault, 50+ min).\n \"\"\"\n anchor_parts = anchor.parts\n if not anchor_parts:\n # Anchor without parts (Path(\".\")): defer to pathlib for its exact\n # \".\" / ValueError semantics. Real anchors are directories with parts,\n # so the hot path never lands here.\n try:\n return _nfc(str(target.relative_to(anchor)).replace(os.sep, \"/\"))\n except ValueError:\n return None\n n = len(anchor_parts)\n if len(target_parts) \u003c n:\n return None\n head = target_parts[:n]\n if head != anchor_parts and (\n not _CASEFOLD_PATHS\n or tuple(map(os.path.normcase, head))\n != tuple(map(os.path.normcase, anchor_parts))\n ):\n return None\n tail = target_parts[n:]\n if not tail:\n return \".\"\n return _nfc(\"/\".join(tail))\n\n\ndef _match_globstar_parts(\n path_parts: tuple[str, ...],\n pattern_parts: tuple[str, ...],\n path_idx: int,\n pattern_idx: int,\n memo: dict[tuple[int, int], bool],\n) -> bool:\n \"\"\"Recursive ``**``-aware segment match, memoized via an explicit dict.\n\n Lifted out of ``_match_anchored_ignore_pattern`` (was a per-call\n ``@lru_cache`` closure): the decorated inner closure referenced itself, so\n every call leaked a reference cycle for the GC to reclaim on this hot path.\n A plain dict passed in avoids both the cycle and the per-call cache setup.\n \"\"\"\n key = (path_idx, pattern_idx)\n cached = memo.get(key)\n if cached is not None:\n return cached\n\n if pattern_idx == len(pattern_parts):\n result = path_idx == len(path_parts)\n else:\n part = pattern_parts[pattern_idx]\n if part == \"**\":\n if pattern_idx == len(pattern_parts) - 1:\n result = path_idx \u003c len(path_parts)\n else:\n result = _match_globstar_parts(\n path_parts, pattern_parts, path_idx, pattern_idx + 1, memo\n ) or (\n path_idx \u003c len(path_parts)\n and _match_globstar_parts(\n path_parts, pattern_parts, path_idx + 1, pattern_idx, memo\n )\n )\n else:\n result = (\n path_idx \u003c len(path_parts)\n and fnmatch.fnmatchcase(path_parts[path_idx], part)\n and _match_globstar_parts(\n path_parts, pattern_parts, path_idx + 1, pattern_idx + 1, memo\n )\n )\n memo[key] = result\n return result\n\n\ndef _match_anchored_ignore_pattern(path: str, pattern: str) -> bool:\n \"\"\"Match an anchored gitignore pattern without letting ``*`` cross ``/``.\"\"\"\n path_parts = tuple(path.split(\"/\"))\n pattern_parts = tuple(pattern.split(\"/\"))\n # Fast path: with no ``**`` the match is a straight segment-wise fnmatch of\n # equal-length paths, so skip the recursive matcher and its memo entirely.\n if \"**\" not in pattern_parts:\n if len(path_parts) != len(pattern_parts):\n return False\n return all(\n fnmatch.fnmatchcase(pp, qp) for pp, qp in zip(path_parts, pattern_parts)\n )\n return _match_globstar_parts(path_parts, pattern_parts, 0, 0, {})\n\n\ndef _is_ignored(\n path: Path,\n root: Path,\n patterns: list[tuple[Path, str]],\n *,\n _cache: dict[Path, bool] | None = None,\n) -> bool:\n \"\"\"Return True if the path should be ignored per .graphifyignore patterns.\n\n Uses gitignore last-match-wins semantics: all patterns are evaluated in\n order; the final matching pattern determines the result. Negation patterns\n (starting with !) un-ignore a previously ignored path.\n\n Enforces gitignore's parent-exclusion rule: a ! pattern cannot re-include\n a file whose ancestor directory is already excluded.\n\n _cache: optional dict shared across calls within the same scan. Ancestor\n directory results are memoised so files under the same subtree don't\n re-evaluate the same patterns repeatedly.\n \"\"\"\n if not patterns:\n return False\n\n def _eval(target: Path) -> bool:\n \"\"\"Apply last-match-wins to a single target path.\n\n Everything derivable from ``target`` alone — its parts, its relative\n path per anchor, the NFC name, the is_dir() stat — is computed at most\n ONCE per call, no matter how many patterns are evaluated. The previous\n shape rebuilt pathlib objects (``relative_to``) and re-split strings\n per PATTERN per entry, which pinned real scans for tens of minutes\n (see CHANGELOG).\n \"\"\"\n if _cache is not None and target in _cache:\n return _cache[target]\n\n target_parts = target.parts\n target_name_nfc: str | None = None\n target_is_dir: bool | None = None\n # rel string (or None = outside anchor) per distinct anchor; patterns\n # overwhelmingly share a handful of anchors.\n rel_by_anchor: dict[Path, str | None] = {}\n\n def _matches(rel: str, p: str, path_relative: bool) -> bool:\n nonlocal target_name_nfc\n if path_relative:\n return _match_anchored_ignore_pattern(rel, p)\n # A pattern without a slash names one path component, so it is\n # compared with the target's own name only. Ancestors are tested\n # one by one by _is_ignored's parent-exclusion walk; matching them\n # again here let `*` cross \"/\" and left a directory re-included\n # with `!dir/` without its files.\n if target_name_nfc is None:\n target_name_nfc = _nfc(target.name)\n return fnmatch.fnmatch(target_name_nfc, p)\n\n result = False\n for anchor, pattern in patterns:\n negated, directory_only, path_relative, p = _parse_ignore_pattern(pattern)\n if not p:\n continue\n\n # gitignore semantics: patterns from A/.gitignore apply ONLY to paths\n # under A. Matching non-anchored patterns against root-relative paths\n # let e.g. .hypothesis/.gitignore's bare \"*\" ignore the ENTIRE repo\n # (detect() returned 0 files). The anchor dir itself is exempt — an\n # ignore file governs its directory's contents, not the directory.\n if anchor not in rel_by_anchor:\n rel_by_anchor[anchor] = _lexical_relative(target, target_parts, anchor)\n rel_anchor = rel_by_anchor[anchor]\n if rel_anchor is None:\n continue # target outside this pattern's anchor: cannot match\n matched = False\n if rel_anchor != \".\":\n matched = _matches(rel_anchor, p, path_relative=path_relative)\n if matched and directory_only:\n if target_is_dir is None:\n target_is_dir = target.is_dir()\n if not target_is_dir:\n matched = False\n\n if matched:\n result = not negated # last match wins; ! flips to un-ignore\n if _cache is not None:\n _cache[target] = result\n return result\n\n # Gitignore parent-exclusion rule: a ! re-include cannot rescue a file\n # whose ancestor directory is already excluded. Walk ancestors top-down;\n # if any ancestor is excluded, the file is excluded regardless of later\n # ! patterns targeting the file or a sub-path.\n try:\n rel_parts = path.relative_to(root).parts\n except ValueError:\n return _eval(path)\n\n ancestor = root\n for part in rel_parts[:-1]:\n ancestor = ancestor / part\n if _eval(ancestor):\n return True\n return _eval(path)\n\n\ndef _is_scan_ignored(\n path: Path,\n root: Path,\n patterns: list[tuple[Path, str]],\n explicit_patterns: list[tuple[Path, str]],\n tracked_files: set[str],\n tracked_dirs: set[str],\n *,\n cache: dict[Path, bool],\n explicit_cache: dict[Path, bool],\n) -> bool:\n \"\"\"Apply ignore rules while preserving Git-tracked paths.\n\n ``patterns`` combines Git and graph-specific rules. ``explicit_patterns``\n contains only .graphifyignore/--exclude rules, which remain authoritative\n even for tracked files. A tracked file's ancestor directories are preserved\n from Git-only pruning so the walk can reach the file (#2759).\n \"\"\"\n if not _is_ignored(path, root, patterns, _cache=cache):\n return False\n if _is_ignored(path, root, explicit_patterns, _cache=explicit_cache):\n return True\n identity = _path_identity(path)\n return identity not in tracked_files and identity not in tracked_dirs\n\n\ndef ignored_predicate(\n root: Path,\n *,\n extra_excludes: list[str] | None = None,\n gitignore: bool = True,\n) -> Callable[[Path], bool]:\n \"\"\"Build a per-path predicate answering \"would detect() exclude this path?\".\n\n Mirrors detect()'s ignore decisions for a single existing path WITHOUT\n re-walking the corpus, from the same machinery detect() uses: the ancestor\n .graphifyignore/.gitignore chain (_load_graphifyignore), CLI/persisted\n ``--exclude`` patterns appended last at the root anchor (#947), nested\n per-directory ignore files along the path's own lineage (#1206), the\n _is_noise_dir directory pruning, and _SKIP_FILES. The sensitive-file\n heuristic (_is_sensitive) is deliberately NOT included: callers use this\n predicate as positive evidence of a live ignore RULE (#2495), and a\n heuristic match is not user intent.\n\n Nested patterns are loaded lazily, once per directory, into one shared\n pattern list. That accumulation cannot cross-contaminate results — a\n pattern only ever matches paths under its anchor directory, so patterns\n from a sibling subtree are inert — which is the same invariant detect()'s\n live os.walk relies on, and it keeps the shared _is_ignored cache valid.\n \"\"\"\n root = root.resolve()\n patterns = _load_graphifyignore(root, gitignore=gitignore)\n explicit_patterns = _load_graphifyignore(root, gitignore=False)\n # Only shell out to git when .gitignore actually contributes patterns beyond\n # the explicit (.graphifyignore/--exclude) set: with no .gitignore in play,\n # nothing is dropped by gitignore and the tracked-file exemption is moot, so a\n # non-.gitignore corpus pays no `git ls-files` cost.\n tracked_files, tracked_dirs = (\n _git_tracked_path_keys(root)\n if gitignore and len(patterns) > len(explicit_patterns)\n else (set(), set())\n )\n if extra_excludes:\n for pat in extra_excludes:\n line = _parse_gitignore_line(pat)\n if line:\n patterns.append((root, line))\n explicit_patterns.append((root, line))\n cache: dict[Path, bool] = {}\n explicit_cache: dict[Path, bool] = {}\n # root's own ignore file is the last entry of _load_graphifyignore's chain.\n loaded_dirs: set[Path] = {root}\n\n def _ignored(path: Path) -> bool:\n path = Path(os.path.abspath(path))\n try:\n rel_parts = path.relative_to(root).parts\n except ValueError:\n return False # outside the scan root: detect() never considered it\n if path.name in _SKIP_FILES or _is_installed_graphify_file(path):\n return True\n # Noise-dir pruning: os.walk never descends these, so anything beneath\n # one is excluded from the corpus regardless of ignore patterns.\n parent = root\n for part in rel_parts[:-1]:\n if _is_noise_dir(part, parent):\n return True\n parent = parent / part\n # Load ignore files along this path's own lineage — detect()'s walk\n # would have loaded exactly these before reaching the file (#1206).\n ancestor = root\n for part in rel_parts[:-1]:\n ancestor = ancestor / part\n if ancestor not in loaded_dirs:\n loaded_dirs.add(ancestor)\n patterns.extend(_load_dir_own_ignore(ancestor, gitignore=gitignore))\n explicit_patterns.extend(\n _load_dir_own_ignore(ancestor, gitignore=False)\n )\n return _is_scan_ignored(\n path,\n root,\n patterns,\n explicit_patterns,\n tracked_files,\n tracked_dirs,\n cache=cache,\n explicit_cache=explicit_cache,\n )\n\n return _ignored\n\n\ndef _auto_follow_symlinks(root: Path) -> bool:\n \"\"\"Return whether ``root`` has any direct symlinked child.\n\n Kept for callers that import the private helper, but detection no longer\n enables symlink following automatically. Following symlinks is now an\n explicit opt-in, and out-of-root symlink targets are never indexed.\n \"\"\"\n try:\n for p in root.iterdir():\n if p.is_symlink():\n return True\n except (OSError, PermissionError):\n pass\n return False\n\n\ndef _resolves_under_root(path: Path, root: Path) -> bool:\n \"\"\"True when ``path`` resolves to a target inside ``root``.\"\"\"\n try:\n path.resolve().relative_to(root.resolve())\n except (OSError, RuntimeError, ValueError):\n return False\n return True\n\n\ndef detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True) -> dict:\n root = root.resolve()\n # The project's extension->language declarations (#2961) shape\n # classification for this scan.\n activate_language_overrides(root)\n configured_out_dir = root / GRAPHIFY_OUT\n configured_out_names = {configured_out_dir.name}\n try:\n configured_out_dir = configured_out_dir.resolve()\n except (OSError, RuntimeError):\n configured_out_dir = configured_out_dir.absolute()\n configured_out_names.add(configured_out_dir.name)\n # .graphifyinclude support was removed (#2112): its loader and matchers had\n # no consumers, so the file has been a silent no-op since dot directories\n # became indexed by default (#873). Surface that once per scan so a\n # leftover allowlist file is not a silent behavior change.\n if (root / \".graphifyinclude\").is_file():\n import sys as _sys\n print(\n \"[graphify] WARNING: .graphifyinclude is no longer supported \"\n \"(it has been non-functional since dot directories became indexed \"\n \"by default); to re-include ignored paths, use ! negation patterns \"\n \"in .graphifyignore.\",\n file=_sys.stderr,\n )\n if follow_symlinks is None:\n follow_symlinks = False\n google_workspace = google_workspace_enabled() if google_workspace is None else google_workspace\n files: dict[FileType, list[str]] = {\n FileType.CODE: [],\n FileType.DOCUMENT: [],\n FileType.PAPER: [],\n FileType.IMAGE: [],\n FileType.VIDEO: [],\n }\n total_words = 0\n\n def _wc(path: Path) -> int:\n # Cache word counts against each file's stat signature so unchanged\n # PDFs/docx aren't re-parsed on every run just to size the corpus (#1656).\n # cache_root (when given, e.g. from `extract --out`) keeps this cache out\n # of the scanned corpus (#1747).\n from graphify import cache as _cache\n return _cache.cached_word_count(path, root, count_words, cache_root=cache_root)\n\n skipped_sensitive: list[str] = []\n unclassified: list[str] = []\n # Files/dirs dropped by a .gitignore/.graphifyignore rule. Recorded so an\n # over-broad ignore (or a legitimately-ignored subtree) is visible instead\n # of silently vanishing from the graph (#1922). Directory-level entries keep\n # this bounded — a pruned `data/` is one entry, not one per contained file.\n ignored: list[str] = []\n pruned_noise: list[str] = []\n # JS/TS resolution inputs that _SKIP_FILES keeps out of the corpus. They are\n # not scan input, but they do steer resolution, so they are reported\n # separately for the no-change gate to compare (#4740).\n resolution_inputs: list[str] = []\n ignore_patterns = _load_graphifyignore(root, gitignore=gitignore)\n explicit_ignore_patterns = _load_graphifyignore(root, gitignore=False)\n # See ignored_predicate: skip the `git ls-files` subprocess when .gitignore\n # contributes no patterns, so a non-.gitignore corpus pays nothing for it.\n tracked_files, tracked_dirs = (\n _git_tracked_path_keys(root)\n if gitignore and len(ignore_patterns) > len(explicit_ignore_patterns)\n else (set(), set())\n )\n ignore_cache: dict[Path, bool] = {} # shared across all _is_ignored calls in this scan\n explicit_ignore_cache: dict[Path, bool] = {}\n # CLI --exclude patterns are anchored at the scan root and appended last\n # so they win over any .graphifyignore/.gitignore rules (#947).\n if extra_excludes:\n for pat in extra_excludes:\n line = _parse_gitignore_line(pat)\n if line:\n ignore_patterns.append((root, line))\n explicit_ignore_patterns.append((root, line))\n\n def _ignored_for_scan(path: Path) -> bool:\n return _is_scan_ignored(\n path,\n root,\n ignore_patterns,\n explicit_ignore_patterns,\n tracked_files,\n tracked_dirs,\n cache=ignore_cache,\n explicit_cache=explicit_ignore_cache,\n )\n\n # Always include graphify-out/memory/ - query results filed back into the graph\n memory_dir = root / GRAPHIFY_OUT / \"memory\"\n scan_paths = [root]\n if memory_dir.exists():\n scan_paths.append(memory_dir)\n\n seen: set[Path] = set()\n all_files: list[Path] = []\n\n # os.walk swallows os.scandir errors by default (no onerror -> the failing\n # directory subtree is silently skipped). That turns a transient\n # PermissionError, or a directory created/deleted mid-walk (e.g. concurrent\n # writes racing the scan), into a partial file list and, downstream, a\n # silently partial graph.json. Record and surface every skipped directory\n # so an incomplete enumeration is visible rather than silent.\n walk_errors: list[str] = []\n\n def _on_walk_error(err: OSError) -> None:\n import sys as _sys\n target = getattr(err, \"filename\", None) or \"\u003cunknown>\"\n walk_errors.append(f\"{target}: {err}\")\n print(\n f\"[graphify] WARNING: could not scan {target} ({err}); \"\n f\"its files are missing from this run's enumeration.\",\n file=_sys.stderr,\n )\n\n for scan_root in scan_paths:\n in_memory_tree = memory_dir.exists() and str(scan_root).startswith(str(memory_dir))\n for dirpath, dirnames, filenames in os.walk(\n scan_root, followlinks=follow_symlinks, onerror=_on_walk_error\n ):\n dp = Path(dirpath)\n if follow_symlinks and os.path.islink(dirpath):\n real = os.path.realpath(dirpath)\n parent_real = os.path.realpath(os.path.dirname(dirpath))\n if parent_real == real or parent_real.startswith(real + os.sep):\n dirnames.clear()\n continue\n if not in_memory_tree:\n # dp == root was already loaded by _load_graphifyignore (root is\n # the last entry in its ancestor chain); every other directory\n # reached by the walk is a descendant below the scan root, whose\n # own .gitignore/.graphifyignore is unknown until we get here.\n # Load it now, before pruning dp's children, so a nested ignore\n # file governs its own subtree the same way git honors it (#1206).\n if dp != root:\n ignore_patterns.extend(_load_dir_own_ignore(dp, gitignore=gitignore))\n explicit_ignore_patterns.extend(\n _load_dir_own_ignore(dp, gitignore=False)\n )\n # Prune noise dirs in-place so os.walk never descends into them.\n # Dot dirs are allowed — users often want .github/, .claude/, etc.\n # Framework caches (.next, .nuxt, …) are caught by _is_noise_dir.\n # Negations need no special-casing here: _is_ignored already applies\n # last-match-wins (so `!dir/` un-ignores a directory and it won't be\n # pruned) and the gitignore parent-exclusion rule (a `!` cannot rescue\n # a file beneath an excluded dir), so descending an ignored directory to\n # look for a re-included file is never necessary. The previous blanket\n # `has_negation` disabled directory pruning for EVERY ignored dir whenever\n # any `!` rule existed — e.g. a single `!docs/**` made the walk descend\n # bin/, obj/, wwwroot/, generated/, … : a pathological slowdown on large\n # repos for no correctness gain.\n kept_dirs: list[str] = []\n for d in dirnames:\n child = dp / d\n is_configured_out = False\n if d in configured_out_names:\n try:\n is_configured_out = child.resolve() == configured_out_dir\n except (OSError, RuntimeError):\n pass\n if is_configured_out:\n pruned_noise.append(str(child) + os.sep)\n continue\n if _is_noise_dir(d, dp):\n # Record pruned-as-noise dirs so a wrongly-pruned real\n # source dir is at least traceable in the output rather\n # than vanishing silently (#2058).\n pruned_noise.append(str(child) + os.sep)\n continue\n # A linked worktree whose main checkout is also inside the\n # scan root is a second copy of the same repo — every file\n # counted twice (#4683). Neither the name-based rules (the\n # worktree's name is arbitrary) nor info/exclude (the scan\n # root need not be a repo) reach this sibling layout, but\n # its .git file + commondir back-pointer identify it.\n _wt_main = _linked_worktree_main_root(child)\n if _wt_main is not None and _resolves_under_root(_wt_main, root):\n pruned_noise.append(str(child) + os.sep)\n continue\n # Directory-level pruning: ONE ignore evaluation excludes the\n # whole subtree — os.walk never descends, so a 29k-file\n # ignored dir costs one check, not one per contained file.\n if _ignored_for_scan(child):\n ignored.append(str(child) + os.sep)\n continue\n kept_dirs.append(d)\n dirnames[:] = kept_dirs\n if follow_symlinks:\n safe_dirs: list[str] = []\n for d in dirnames:\n child = dp / d\n if child.is_symlink() and not _resolves_under_root(child, root):\n skipped_sensitive.append(str(child) + \" [symlink target outside scan root]\")\n continue\n safe_dirs.append(d)\n dirnames[:] = safe_dirs\n for fname in filenames:\n if fname in _SKIP_FILES:\n if is_resolution_input(fname):\n # Not corpus input, but a JS/TS resolution input: record\n # it so the no-change gate can compare its content\n # (#4740). It stays out of the scan exactly as before.\n resolution_inputs.append(str(dp / fname))\n continue\n p = dp / fname\n if _is_installed_graphify_file(p):\n continue\n if p not in seen:\n seen.add(p)\n all_files.append(p)\n\n all_files.sort(key=lambda p: str(p))\n\n out_base = Path(cache_root).resolve() if cache_root is not None else root\n converted_dir = out_base / GRAPHIFY_OUT / \"converted\"\n\n for p in all_files:\n # For memory dir files, skip hidden/noise filtering\n in_memory = memory_dir.exists() and str(p).startswith(str(memory_dir))\n if not in_memory:\n # Skip files inside our own converted/ dir (avoid re-processing sidecars)\n if str(p).startswith(str(converted_dir)):\n continue\n if not in_memory and _ignored_for_scan(p):\n ignored.append(str(p))\n continue\n if not _resolves_under_root(p, root):\n skipped_sensitive.append(str(p) + \" [symlink target outside scan root]\")\n continue\n if not _is_regular_file(p):\n # A repository may contain named pipes, sockets and device nodes,\n # and `clone \u003cgithub-url>` exists precisely to point the scan at\n # trees the operator did not write.\n #\n # This has to be caught HERE, at the one place where a path is\n # admitted to the corpus, rather than at each read: the readers\n # number in the hundreds across the extractors, and open() on a\n # FIFO with no writer BLOCKS FOREVER — it never raises, so their\n # try/except cannot help and the whole run hangs with no output.\n skipped_sensitive.append(str(p) + \" [not a regular file]\")\n continue\n if _is_sensitive(p):\n skipped_sensitive.append(str(p))\n continue\n ftype = classify_file(p)\n if not ftype:\n # Considered but unclassifiable: an extension not in any supported set,\n # or an extensionless, non-shebang file (Dockerfile, Gemfile, Makefile,\n # Rakefile, LICENSE, ...). Previously these left no trace at all — not\n # counted, not listed — so a user couldn't tell they were seen (#1692).\n unclassified.append(str(p))\n continue\n if ftype:\n if p.suffix.lower() in GOOGLE_WORKSPACE_EXTENSIONS:\n if not google_workspace:\n skipped_sensitive.append(\n str(p)\n + \" [Google Workspace shortcut skipped - pass --google-workspace \"\n \"or set GRAPHIFY_GOOGLE_WORKSPACE=1]\"\n )\n continue\n try:\n md_path = convert_google_workspace_file(p, converted_dir, xlsx_to_markdown=xlsx_to_markdown, root=root)\n except Exception as exc:\n skipped_sensitive.append(str(p) + f\" [Google Workspace export failed: {exc}]\")\n continue\n if md_path:\n # #3504: the sidecar lands under converted_dir, which the\n # documented .gitignore advice puts inside a gitignored\n # graphify-out/ -- an ignore check here would reject the\n # tool's own output for the same reason it should be\n # gitignored in the first place, silently dropping the\n # source document from the corpus. The ignore check exists\n # to keep USER files out of the scan, not to filter output\n # this same pass just produced from an already-admitted file.\n if _ignored_for_scan(md_path) and not md_path.is_relative_to(converted_dir):\n continue\n files[ftype].append(str(md_path))\n total_words += _wc(md_path)\n else:\n skipped_sensitive.append(str(p) + \" [Google Workspace export produced no readable text]\")\n continue\n # Office files: convert to markdown sidecar so subagents can read them\n if p.suffix.lower() in OFFICE_EXTENSIONS:\n md_path = convert_office_file(p, converted_dir, root=root)\n if md_path:\n # #3504: see the matching comment in the Google Workspace\n # branch above -- same sidecar-under-a-gitignored-output-dir\n # trap, same exemption.\n if _ignored_for_scan(md_path) and not md_path.is_relative_to(converted_dir):\n continue\n files[ftype].append(str(md_path))\n total_words += _wc(md_path)\n else:\n # Conversion failed (library not installed) - skip with note\n skipped_sensitive.append(str(p) + \" [office conversion failed - pip install graphifyy[office]]\")\n continue\n files[ftype].append(str(p))\n # Bypass legacy cached counts derived from binary media bytes.\n if p.suffix.lower() not in _BINARY_MEDIA_EXTENSIONS:\n total_words += _wc(p)\n\n for ftype in files:\n files[ftype].sort()\n\n total_files = sum(len(v) for v in files.values())\n needs_graph = total_words >= CORPUS_WARN_THRESHOLD\n\n # Determine warning - lower bound, upper bound, or sensitive files skipped\n warning: str | None = None\n if not needs_graph:\n warning = (\n f\"Corpus is ~{total_words:,} words - fits in a single context window. \"\n f\"You may not need a graph.\"\n )\n elif total_words >= CORPUS_UPPER_THRESHOLD or total_files >= FILE_COUNT_UPPER:\n warning = (\n f\"Large corpus: {total_files} files · ~{total_words:,} words. \"\n f\"Semantic extraction will be expensive (many Claude tokens). \"\n f\"Consider running on a subfolder.\"\n )\n\n return {\n \"files\": {k.value: v for k, v in files.items()},\n \"total_files\": total_files,\n \"total_words\": total_words,\n \"needs_graph\": needs_graph,\n \"warning\": warning,\n \"skipped_sensitive\": skipped_sensitive,\n \"unclassified\": sorted(unclassified),\n \"walk_errors\": walk_errors,\n \"ignored\": sorted(ignored),\n \"pruned_noise_dirs\": sorted(pruned_noise),\n \"resolution_inputs\": sorted(set(resolution_inputs)),\n \"graphifyignore_patterns\": len(ignore_patterns),\n \"scan_root\": str(root.resolve()),\n }\n\n\n# Shared with cache.py (paths.os_long_path); the name stays for detect's callers.\n_os_path = os_long_path\n\n\ndef _md5_file(path: Path) -> str:\n \"\"\"MD5 of file contents streamed in 64KB chunks — for change detection only.\"\"\"\n import hashlib as _hl\n h = _hl.md5(usedforsecurity=False)\n try:\n with open(_os_path(path), \"rb\") as f:\n for chunk in iter(lambda: f.read(65536), b\"\"):\n h.update(chunk)\n except OSError:\n return \"\"\n return h.hexdigest()\n\n\ndef _stat_and_hash(path_str: str) -> tuple[str, float, str] | None:\n \"\"\"Stat + MD5 a single file; returns None on OSError (e.g. deleted mid-run).\"\"\"\n try:\n p = Path(path_str)\n return path_str, os.stat(_os_path(p)).st_mtime, _md5_file(p)\n except OSError:\n return None\n\n\ndef _nfc(s: str) -> str:\n \"\"\"NFC-normalize a path string used as a manifest key.\n\n On macOS, ``os.walk`` / ``getcwd`` yield NFD paths while path literals\n and many skill-substituted roots are NFC. Raw string compare then treats\n every file as both deleted and new, forcing a full re-extract (#2221).\n Same boundary as the Office sidecar hash fix (#1226).\n \"\"\"\n import unicodedata\n return unicodedata.normalize(\"NFC\", s)\n\n\ndef _to_relative_for_storage(key: str, root: Path) -> str:\n \"\"\"Return ``key`` as a forward-slash relative path from ``root``.\n\n Keys outside ``root`` (out-of-tree symlinked sources, external --include\n paths) and already-relative keys pass through unchanged — mirrors the\n fallback in :func:`graphify.watch._relativize_source_files` so the\n on-disk artifact survives the round-trip even when some paths cannot be\n portably encoded.\n\n Only ``root`` is resolved — the key itself is relativized symbolically\n so an in-root symlink (e.g. ``alias.py -> sub/target.py``) is stored\n under its own name. Resolving the key would point the stored entry at\n the symlink target, and the original key would then miss on reload and\n re-extract on every incremental run.\n\n Both sides of ``relpath`` are NFC'd first: stamped keys may already be\n NFC while ``Path(root).resolve()`` is NFD on macOS, and a mixed-form\n compare would mark an in-root file as ``../…`` and keep it absolute\n (#2221 / #777).\n \"\"\"\n p = Path(key)\n if not p.is_absolute():\n return key\n try:\n base = _nfc(str(Path(root).resolve()))\n rel = os.path.relpath(_nfc(str(p)), base)\n except (ValueError, OSError):\n return key # outside root (e.g. Windows cross-drive)\n # ``os.path.relpath`` happily produces ``../foo`` for paths outside\n # root; mirror the prior ``relative_to``-raises-ValueError semantics by\n # keeping out-of-root entries in their absolute form.\n if rel == \"..\" or rel.startswith(\"..\" + os.sep) or rel.startswith(\"../\"):\n return key\n return rel.replace(os.sep, \"/\")\n\n\n_DRIVE_LETTER_RE = re.compile(r\"^[A-Za-z]:[\\\\/]\")\n\n\ndef _looks_absolute(key: str) -> bool:\n \"\"\"True if ``key`` is absolute under ANY platform's path syntax, not\n just the current one.\n\n ``Path.is_absolute()`` only recognizes the CURRENT platform's own\n syntax: a POSIX-style ``/abs/path`` key loaded on Windows (or a\n ``C:\\\\...``/UNC key loaded on POSIX) is wrongly judged relative, and\n :func:`_to_absolute_from_storage` then joins it onto ``root`` instead of\n leaving it alone — silently producing a nonsense path rather than the\n unreachable-but-at-least-recognizable foreign one (#1964 review: a\n manifest genuinely can move between platforms, per this module's own\n \"mix of call sites/versions\" scope).\n \"\"\"\n return (\n Path(key).is_absolute()\n or key.startswith((\"/\", \"\\\\\"))\n or bool(_DRIVE_LETTER_RE.match(key))\n )\n\n\ndef _normpath_own_flavor(key: str) -> str:\n \"\"\"``normpath`` an absolute ``key`` under its OWN platform's syntax.\n\n ``os.path.normpath`` applies the current platform's rules, so on Windows\n it rewrites a POSIX key ``/home/u/foo.py`` to ``\\\\home\\\\u\\\\foo.py`` —\n a string POSIX can no longer read back, since ``posixpath`` does not\n treat ``\\\\`` as a separator. Pick the flavor from the key itself: a\n drive letter or leading backslash is Windows, a leading ``/`` that the\n current platform does not consider absolute is POSIX.\n \"\"\"\n if Path(key).is_absolute():\n return os.path.normpath(key)\n if key.startswith(\"\\\\\") or _DRIVE_LETTER_RE.match(key):\n return ntpath.normpath(key)\n return posixpath.normpath(key)\n\n\ndef _to_absolute_from_storage(key: str, root: Path) -> str:\n \"\"\"Inverse of :func:`_to_relative_for_storage`.\n\n Re-anchor a stored key against ``root``. Already-absolute keys\n (legacy manifests, out-of-root entries, or a foreign-platform key —\n see :func:`_looks_absolute`) are not re-anchored, so that newly-loaded\n manifests from before this change remain readable; they are only\n dot-segment normalized under their own platform's syntax\n (:func:`_normpath_own_flavor`).\n Uses ``Path(root).resolve()`` so the produced absolute path matches\n what :func:`detect` returns (which also resolves the scan root).\n NFC both sides so a relative key and an NFD-resolved root still join\n to the same string form the rest of the manifest path uses (#2221).\n\n The joined result is lexically normalized (``os.path.normpath``, not\n ``.resolve()``) so a stray ``..``/``.`` segment collapses to the same\n string an equivalent absolute key would use — otherwise two keys for\n the same file (one absolute, one relative-with-dot-segments, the kind\n of mismatch a foreign tool or an older graphify version can leave\n behind, per this function's own caller) fail to canonicalize to the\n same value and the duplicate collapse this function exists for misses\n them (#1964 review). Only the key's own dot segments are normalized,\n never symlinks — ``normpath`` never touches the filesystem, matching\n :func:`_to_relative_for_storage`'s own choice not to resolve the key.\n \"\"\"\n if _looks_absolute(key):\n return _normpath_own_flavor(key)\n # NFC the joined result so an NFD-resolved root + relative key lands on\n # the same form load_manifest / detect_incremental compare against.\n return _nfc(os.path.normpath(str(Path(root).resolve() / Path(key))))\n\n\ndef _collapse_manifest_duplicates(\n items: Iterable[tuple[str, Any]], key_fn: Callable[[str], str]\n) -> dict[str, Any]:\n \"\"\"Canonicalize each raw key via ``key_fn``, keeping the more recently\n observed entry when two distinct raw keys collapse to the same\n canonical one (#1964).\n\n A manifest written across a mix of graphify versions/call sites — some\n passing ``root`` to relativize keys, some not (an outdated installed\n skill runbook, for one) — can end up with both an absolute and a\n relative key for the same file, each carrying different data. A plain\n dict comprehension over such a manifest silently keeps whichever raw\n key happens to iterate last, an artifact of on-disk JSON key order that\n has nothing to do with which entry is actually current — so a fresher\n hash could be discarded in favor of a stale one purely because of how\n the keys happened to accumulate, letting detect_incremental() call a\n genuinely changed file unchanged.\n\n Ties are broken by each entry's own ``seen`` timestamp (when present on\n both sides and they disagree) rather than iteration order, since that\n field exists specifically to record when an entry was last confirmed\n current. Legacy scalar/partial entries (no ``seen``, or either side not\n a dict) fall back to the historical last-wins behavior, unchanged.\n \"\"\"\n result: dict[str, Any] = {}\n for k, v in items:\n canonical = key_fn(k)\n if canonical in result:\n prev = result[canonical]\n prev_seen = prev.get(\"seen\") if isinstance(prev, dict) else None\n new_seen = v.get(\"seen\") if isinstance(v, dict) else None\n if (\n isinstance(prev_seen, (int, float))\n and isinstance(new_seen, (int, float))\n and new_seen \u003c prev_seen\n ):\n continue\n result[canonical] = v\n return result\n\n\ndef _manifest_storage_anchor(manifest_path: str | Path | None, root: Path | None) -> Path | None:\n \"\"\"When a manifest lives under \u003crepo>/graphify-out/manifest.json and root\n is a subfolder of \u003crepo>, stored relative keys are anchored to \u003crepo> (#3785).\"\"\"\n if root is None or not manifest_path:\n return root\n try:\n mp = Path(manifest_path).resolve()\n if mp.parent.name in (\"graphify-out\", GRAPHIFY_OUT_NAME):\n manifest_base = mp.parent.parent\n root_resolved = Path(root).resolve()\n if root_resolved != manifest_base:\n try:\n root_resolved.relative_to(manifest_base)\n return manifest_base\n except ValueError:\n pass\n except (OSError, RuntimeError):\n pass\n return root\n\n\ndef load_manifest(\n manifest_path: str = _MANIFEST_PATH,\n *,\n root: Path | None = None,\n) -> dict:\n \"\"\"Load the manifest from a previous run. Returns {} on any error.\n\n When ``root`` is provided, stored relative keys are re-anchored against\n it so callers see absolute paths regardless of on-disk format. Legacy\n manifests with absolute keys pass through unchanged, so a graphify-out/\n written by an older version (or by a caller that didn't supply ``root``\n to :func:`save_manifest`) remains readable.\n\n Keys are NFC-normalized on load so a manifest written under one Unicode\n form still matches a scan that yields the other (#2221).\n \"\"\"\n try:\n raw = json.loads(Path(manifest_path).read_text(encoding=\"utf-8\"))\n except Exception:\n return {}\n if not isinstance(raw, dict):\n return raw\n if root is None:\n return _collapse_manifest_duplicates(raw.items(), _nfc)\n anchor = _manifest_storage_anchor(manifest_path, root) or root\n return _collapse_manifest_duplicates(\n raw.items(), lambda k: _nfc(_to_absolute_from_storage(k, anchor))\n )\n\n\ndef _checkout_prefix(abs_key: str, relative_keys: set[str]) -> str | None:\n \"\"\"Longest directory prefix of ``abs_key`` whose suffix is in ``relative_keys``.\n\n The suffix is forward-slash and NFC. ``src/c.py`` beats a shorter ``c.py``.\n Prefixes ``\"\"``, ``\"/\"``, and ``\"\\\\\"`` are skipped so a shorter suffix can\n still match. The prefix that remains must be a real directory.\n \"\"\"\n parts = _nfc(abs_key).replace(\"\\\\\", \"/\").split(\"/\")\n best: tuple[int, str] | None = None\n for i in range(1, len(parts)):\n suffix = \"/\".join(parts[i:])\n if suffix not in relative_keys:\n continue\n prefix = \"/\".join(parts[:i])\n if prefix in (\"\", \"/\", \"\\\\\"):\n continue\n if not Path(prefix).is_dir():\n continue\n if best is None or len(suffix) > best[0]:\n best = (len(suffix), prefix)\n return None if best is None else best[1]\n\n\ndef _previous_checkout_absolute_keys(\n existing_keys: Iterable[Any],\n storage_root: Path | None,\n relative_keys: set[str],\n) -> set[str]:\n \"\"\"Absolute keys that still name a previous checkout (#3581).\n\n A key is a candidate only when :func:`_to_relative_for_storage` keeps it\n absolute against ``storage_root`` (a sibling row that becomes relative,\n including #3785, is not a candidate). Candidates that share one directory\n prefix are a previous checkout when that group has at least two keys. One\n absolute key is an include row and stays. ``storage_root is None`` yields\n an empty set.\n \"\"\"\n if storage_root is None:\n return set()\n groups: dict[str, set[str]] = {}\n for key in existing_keys:\n if not isinstance(key, str) or not _looks_absolute(key):\n continue\n stored = _nfc(_to_relative_for_storage(key, storage_root))\n if not _looks_absolute(stored):\n continue\n prefix = _checkout_prefix(stored, relative_keys)\n if prefix is None:\n continue\n groups.setdefault(prefix, set()).add(key)\n stale: set[str] = set()\n for keys in groups.values():\n if len(keys) >= 2:\n stale.update(keys)\n return stale\n\n\ndef save_manifest(\n files: dict[str, list[str]],\n manifest_path: str = _MANIFEST_PATH,\n *,\n kind: str = \"both\",\n root: Path | None = None,\n scan_corpus: set[str] | list[str] | None = None,\n clear_semantic: set[str] | list[str] | None = None,\n clear_ast: set[str] | list[str] | None = None,\n ast_empty: set[str] | list[str] | None = None,\n) -> None:\n \"\"\"Save current file mtimes + content hashes for change detection.\n\n kind=\"ast\" — written by `graphify update` (AST-only rebuild). Stamps\n ast_hash; preserves an existing semantic_hash only when\n the file content is unchanged (mtime + hash match).\n kind=\"semantic\" — written by `graphify extract` after semantic extraction.\n Stamps semantic_hash; preserves existing ast_hash.\n kind=\"both\" — full pipeline: stamps both hashes (default).\n\n When ``root`` is provided, keys are relativized against it before write\n (forward-slash, posix-style) so the on-disk manifest is portable across\n machines and checkout locations (#777). Out-of-root entries are written\n as absolute so they continue to round-trip on the saving machine.\n When ``root`` is None the legacy absolute-keyed format is preserved.\n\n ``scan_corpus`` (#1908): full-scan callers pass the COMPLETE detect\n corpus (absolute paths) so seeded rows for in-root files that are still\n alive on disk but no longer part of the scan (newly excluded via\n .graphifyignore/.gitignore/--exclude) are dropped instead of surviving\n forever and masquerading as deletions in detect_incremental. It must be\n the RAW detect output, not a stamp-filtered subset — pruning to a\n filtered set would erase rows the filter merely omitted (failed chunks,\n --code-only doc rows). When this argument is set, two or more absolute\n keys that share a directory prefix and match relative keys this save writes\n are a previous checkout and are dropped (#3581). A single include row stays.\n Callers saving a SUBSET of files (changed_paths hooks, skill runbooks, #917)\n must leave this None so their untouched rows, including out-of-root rows,\n are preserved.\n\n ``clear_semantic`` (#1948): files that were dispatched this run but\n produced no stamped output (e.g. the LLM omitted their chunk on a\n --force re-run) are absent from ``files``, so the seed loop below would\n otherwise copy their prior semantic_hash verbatim — masking the omission\n and making detect_incremental(kind=\"semantic\") report them unchanged.\n Pass the set of such files (any path form ``scan_corpus`` accepts) to\n force their seeded semantic_hash to \"\" instead of inheriting it.\n\n ``clear_ast`` (#2543): same idea for AST failures (missing optional extra,\n zero-node anomalous extract). Blanks BOTH ``ast_hash`` and\n ``semantic_hash`` on the seeded row so either detect_incremental kind\n re-queues the file after the failure is fixed, without deleting\n graphify-out/.\n\n ``ast_empty`` (#2879): code files the AST extractor declined by design this\n run (data JSON, etc.) — stamped as up-to-date yet contributing zero nodes.\n Marks their row ``ast_empty: true`` so the #2543 heal can tell an intentional\n empty from a poisoned stamp and stop re-queuing them. Preserved across\n incrementals for unchanged rows; dropped once a file starts producing nodes.\n \"\"\"\n # Another root may have activated different overrides in this thread.\n if root is not None:\n activate_language_overrides(Path(root))\n existing = load_manifest(manifest_path, root=root)\n\n # Index both raw and NFC forms so scan/clear membership survives the\n # same NFC/NFD mismatch that breaks manifest lookups (#2221).\n def _path_index(paths: set[str] | list[str] | None) -> set[str] | None:\n if paths is None:\n return None\n indexed: set[str] = set()\n for p in paths:\n indexed.add(p)\n indexed.add(_nfc(p))\n return indexed\n\n scan_set = _path_index(scan_corpus)\n clear_set = _path_index(clear_semantic)\n clear_ast_set = _path_index(clear_ast)\n ast_empty_set = _path_index(ast_empty)\n try:\n root_res: Path | None = Path(root).resolve() if root is not None else None\n except (OSError, RuntimeError):\n root_res = Path(root) if root is not None else None\n\n def _in_scan(path_str: str) -> bool:\n if path_str in scan_set or _nfc(path_str) in scan_set:\n return True\n try:\n resolved = str(Path(path_str).resolve())\n return resolved in scan_set or _nfc(resolved) in scan_set\n except (OSError, RuntimeError):\n return False\n\n def _in_clear(path_str: str) -> bool:\n if clear_set is None:\n return False\n if path_str in clear_set or _nfc(path_str) in clear_set:\n return True\n try:\n resolved = str(Path(path_str).resolve())\n return resolved in clear_set or _nfc(resolved) in clear_set\n except (OSError, RuntimeError):\n return False\n\n def _in_clear_ast(path_str: str) -> bool:\n if clear_ast_set is None:\n return False\n if path_str in clear_ast_set or _nfc(path_str) in clear_ast_set:\n return True\n try:\n resolved = str(Path(path_str).resolve())\n return resolved in clear_ast_set or _nfc(resolved) in clear_ast_set\n except (OSError, RuntimeError):\n return False\n\n def _in_ast_empty(path_str: str) -> bool:\n if ast_empty_set is None:\n return False\n if path_str in ast_empty_set or _nfc(path_str) in ast_empty_set:\n return True\n try:\n resolved = str(Path(path_str).resolve())\n return resolved in ast_empty_set or _nfc(resolved) in ast_empty_set\n except (OSError, RuntimeError):\n return False\n\n def _in_root(path_str: str) -> bool:\n # Without a root we cannot tell in-root from out-of-root; fail open\n # (keep the row) so out-of-root corpora are never pruned by accident.\n if root_res is None:\n return False\n p = Path(path_str)\n try:\n p.relative_to(root_res)\n return True\n except ValueError:\n pass\n try:\n p.resolve().relative_to(root_res)\n return True\n except (ValueError, OSError, RuntimeError):\n return False\n\n def _normalise_entry(entry):\n if isinstance(entry, (int, float)):\n return {\"mtime\": entry, \"ast_hash\": \"\", \"semantic_hash\": \"\"}\n if isinstance(entry, dict) and \"hash\" in entry and \"ast_hash\" not in entry:\n return {\"mtime\": entry.get(\"mtime\", 0), \"ast_hash\": entry[\"hash\"], \"semantic_hash\": \"\"}\n if isinstance(entry, dict):\n return entry\n return None\n\n # Seed from the existing manifest so incremental callers passing a subset\n # of files don't silently erase entries for untouched files (#917).\n # Prune entries whose file no longer exists on disk — those are genuine\n # deletions that detect_incremental() should treat as gone. When the\n # caller supplied the full scan corpus, additionally prune in-root rows\n # the scan no longer covers: those files were excluded, not deleted, and\n # keeping the row makes them look deleted on every future run (#1908).\n all_files = [f for file_list in files.values() for f in file_list]\n storage_root = _manifest_storage_anchor(manifest_path, root) if root is not None else None\n # A subset save omits scan_corpus and must keep every live row it was not\n # given (#917), including an include row and a previous checkout. The\n # checkout drop runs only on a full scan.\n stale_checkout: set[str] = set()\n if scan_set is not None and storage_root is not None:\n new_rels: set[str] = set()\n for f in all_files:\n stored = _nfc(_to_relative_for_storage(_nfc(f), storage_root))\n if not _looks_absolute(stored):\n new_rels.add(stored)\n stale_checkout = _previous_checkout_absolute_keys(\n existing.keys(), storage_root, new_rels\n )\n manifest: dict[str, dict] = {}\n for f, entry in existing.items():\n normalised = _normalise_entry(entry)\n if normalised is None:\n continue\n try:\n if not Path(f).exists():\n continue\n except OSError:\n continue\n if f in stale_checkout:\n continue # previous checkout copied with graphify-out (#3581)\n if scan_set is not None and not _in_scan(f) and _in_root(f):\n continue # excluded-but-alive: drop the stale row (#1908)\n if clear_ast_set is not None and _in_clear_ast(f):\n # AST failure this run (missing extra / zero nodes, #2543): blank\n # both hashes so either detect_incremental kind re-queues. A genuine\n # failure is not an intentional empty, so drop any stale marker too.\n normalised = {k: v for k, v in normalised.items() if k != \"ast_empty\"}\n normalised = {**normalised, \"ast_hash\": \"\", \"semantic_hash\": \"\"}\n elif clear_set is not None and _in_clear(f):\n # Dispatched-but-omitted this run: don't inherit the stale\n # semantic_hash, or detect_incremental would call it unchanged (#1948).\n normalised = {**normalised, \"semantic_hash\": \"\"}\n manifest[f] = normalised\n\n with ThreadPoolExecutor() as pool:\n raw = pool.map(_stat_and_hash, all_files)\n hashed: dict[str, tuple[float, str]] = {\n r[0]: (r[1], r[2]) for r in raw if r is not None\n }\n\n for f in all_files:\n if f not in hashed:\n continue # file deleted between detect() and manifest write\n mtime, h = hashed[f]\n key = _nfc(f)\n prev = _normalise_entry(existing.get(key, {})) or {}\n # Hash workers do not inherit this thread's language overrides.\n current_lang = cache_salt(f)\n lang_changed = current_lang != prev.get(\"language\")\n if kind in (\"ast\", \"both\"):\n ast_h = h\n elif lang_changed:\n ast_h = \"\"\n else:\n ast_h = prev.get(\"ast_hash\", \"\")\n if kind in (\"semantic\", \"both\"):\n sem_h = h\n elif lang_changed:\n sem_h = \"\"\n else:\n # Preserve semantic_hash only when content is unchanged\n sem_h = prev.get(\"semantic_hash\", \"\") if h == prev.get(\"ast_hash\", \"\") else \"\"\n\n # Preserve previous seen timestamp if the entry's mtime and target hash(es)\n # are genuinely unchanged and no clear was requested for this file.\n prev_seen = prev.get(\"seen\")\n is_unchanged = (\n isinstance(prev_seen, (int, float))\n and mtime == prev.get(\"mtime\")\n and (ast_h == prev.get(\"ast_hash\", \"\") if kind in (\"ast\", \"both\") else True)\n and (sem_h == prev.get(\"semantic_hash\", \"\") if kind in (\"semantic\", \"both\") else True)\n and not lang_changed\n and not _in_clear_ast(f)\n and not _in_clear(f)\n )\n entry: dict = {\n \"mtime\": mtime,\n \"seen\": prev_seen if is_unchanged else time.time(),\n \"ast_hash\": ast_h,\n \"semantic_hash\": sem_h,\n }\n if current_lang is not None:\n # No override keeps legacy rows unchanged.\n entry[\"language\"] = current_lang\n # Intentional-empty marker (#2879): set when the extractor declined the\n # file this run; otherwise inherited across an unchanged incremental the\n # same way ast_hash is, and dropped once the file produces nodes.\n if _in_ast_empty(f):\n entry[\"ast_empty\"] = True\n elif not lang_changed and prev.get(\"ast_empty\") and ast_h == prev.get(\"ast_hash\", \"\"):\n entry[\"ast_empty\"] = True\n manifest[key] = entry\n if storage_root is not None:\n manifest = _collapse_manifest_duplicates(\n manifest.items(), lambda k: _nfc(_to_relative_for_storage(k, storage_root))\n )\n else:\n manifest = _collapse_manifest_duplicates(manifest.items(), _nfc)\n\n # Avoid rewriting manifest.json when the serialized payload is identical (#2838).\n manifest_p = Path(manifest_path)\n if manifest_p.is_file():\n try:\n disk_raw = json.loads(manifest_p.read_text(encoding=\"utf-8\"))\n if isinstance(disk_raw, dict) and disk_raw == manifest:\n return\n except Exception:\n pass\n\n from graphify.paths import write_json_atomic\n # Atomic write: a crash mid-write must not leave a truncated manifest that\n # detect_incremental then fails to parse.\n write_json_atomic(manifest_path, manifest, indent=2)\n\n\ndef _mtime_may_hide_a_rewrite(current_mtime: float, stored: dict) -> bool:\n \"\"\"Was this manifest row written in the same tick as the file it describes?\n\n The incremental gate treats \"mtime unchanged\" as proof the content is\n unchanged. That is only true while the filesystem can distinguish the two\n writes: an edit keeping the file the same length and landing in the same\n timestamp tick moves neither size nor mtime, so the file silently skips\n re-extraction and the graph keeps serving the old content.\n\n ``seen`` records when the row was stamped. If the file's mtime falls inside\n the same tick, this row cannot prove currency and the caller pays for one\n MD5. Every other row — the whole settled corpus, and any manifest written\n by an earlier run — keeps the free stat-only fastpath.\n\n Rows predating ``seen`` are treated as safe: they necessarily come from an\n earlier process, where a later write would have had to move mtime.\n \"\"\"\n seen = stored.get(\"seen\")\n if not isinstance(seen, (int, float)):\n return False\n delta = float(seen) - float(current_mtime)\n if delta \u003c 0:\n return False # file is newer than the row; the mtime check already fired\n # Derive granularity from the timestamp: a whole-second mtime means the\n # filesystem cannot separate writes inside that second.\n coarse = float(current_mtime).is_integer()\n return delta \u003c (_MTIME_COARSE_S if coarse else _MTIME_SUBSECOND_S)\n\n\ndef classify_manifest_files(\n file_buckets: dict[str, list[str]],\n manifest: dict,\n *,\n kind: str = \"ast\",\n unstamped_is_changed: bool = True,\n) -> tuple[dict[str, list[str]], dict[str, list[str]]]:\n \"\"\"Split ``file_buckets`` into (changed, unchanged) against ``manifest``.\n\n This is the comparison ``detect_incremental`` has always used: mtime first\n (free, no disk IO beyond stat), content hash only when mtime moved or when\n the racily-clean window needs it. A missing row, a row with no usable hash,\n a language change, or an unknown row format all count as changed, so a caller\n can only ever over-report change.\n\n Factored out so the ``graphify update`` no-change gate in\n :func:`graphify.watch._rebuild_code` reaches the same verdict from an\n already-computed ``detect()`` result instead of paying for a second scan\n (#4740). ``file_buckets`` is not required to come from ``detect()[\"files\"]``:\n the gate passes a synthetic bucket holding the JS/TS resolution inputs that\n the manifest stamps but the scan skips.\n\n ``unstamped_is_changed=False`` is for that gate. A file with no row (or a row\n whose hash was blanked after a failed extraction, #2543) is then treated as\n unchanged: it contributed no nodes, so re-extraction cannot change the graph,\n and only the gate's own file-set digest can tell \"same file, still failing\"\n from \"new file\". The incremental path keeps the default, where such a file is\n re-queued until it extracts.\n \"\"\"\n new_files: dict[str, list[str]] = {k: [] for k in file_buckets}\n unchanged_files: dict[str, list[str]] = {k: [] for k in file_buckets}\n\n for ftype, file_list in file_buckets.items():\n for f in file_list:\n # Manifest keys are NFC; scan paths may arrive NFD (#2221).\n stored = manifest.get(_nfc(f))\n try:\n current_mtime = os.stat(_os_path(Path(f))).st_mtime\n except Exception:\n current_mtime = 0\n\n # Legacy manifest: plain float value stores only mtime.\n # Compare with `!=` so backwards mtime motion (git checkout of an\n # older commit, tarball restore, rsync --times) still triggers a\n # re-extract; the previous `>` silently kept the stale cache and\n # the graph drifted from disk (#1859). No stored hash means we\n # cannot verify content — any mtime delta forces a re-extract,\n # and the next save promotes the entry into the dict schema.\n # A language change invalidates even identical bytes and legacy rows.\n stored_lang = stored.get(\"language\") if isinstance(stored, dict) else None\n if stored is None:\n changed = unstamped_is_changed\n elif cache_salt(f) != stored_lang:\n changed = True\n elif isinstance(stored, (int, float)):\n changed = current_mtime != stored\n elif isinstance(stored, dict):\n # Normalise legacy {mtime, hash} to new schema\n if \"hash\" in stored and \"ast_hash\" not in stored:\n stored = {\"mtime\": stored.get(\"mtime\", 0), \"ast_hash\": stored[\"hash\"], \"semantic_hash\": \"\"}\n hash_key = \"semantic_hash\" if kind == \"semantic\" else \"ast_hash\"\n stored_hash = stored.get(hash_key, \"\")\n # Missing semantic_hash means update ran but extract hasn't — always re-extract\n if not stored_hash:\n changed = unstamped_is_changed\n else:\n stored_mtime = stored.get(\"mtime\")\n # Schema-drift guard (#1163): tolerate a nested {mtime: ...}\n # dict or any non-numeric value without crashing.\n if isinstance(stored_mtime, dict):\n stored_mtime = stored_mtime.get(\"mtime\")\n if not isinstance(stored_mtime, (int, float)):\n stored_mtime = None\n if stored_mtime is None or current_mtime != stored_mtime:\n # mtime bumped — verify with content hash before re-extracting\n changed = _md5_file(Path(f)) != stored_hash\n elif _mtime_may_hide_a_rewrite(current_mtime, stored):\n # mtime is unchanged, but it was recorded in the same\n # filesystem tick the file was written in — a later\n # same-length edit lands in that tick without moving\n # mtime, and the file silently skips re-extraction\n # while the graph keeps serving the old content.\n # Only this narrow window pays for a content hash.\n changed = _md5_file(Path(f)) != stored_hash\n else:\n changed = False\n else:\n changed = True # unknown format, re-extract to be safe\n\n if changed:\n new_files[ftype].append(f)\n else:\n unchanged_files[ftype].append(f)\n\n return new_files, unchanged_files\n\n\ndef detect_incremental(\n root: Path,\n manifest_path: str = _MANIFEST_PATH,\n *,\n follow_symlinks: bool | None = None,\n google_workspace: bool | None = None,\n kind: str = \"semantic\",\n extra_excludes: list[str] | None = None,\n gitignore: bool = True,\n cache_root: Path | None = None,\n) -> dict:\n \"\"\"Like detect(), but returns only new or modified files since the last run.\n\n kind=\"semantic\" (default for extract): a file is \"changed\" when its\n semantic_hash is missing or its content has changed since the last\n semantic extraction pass. Use this for `graphify extract` so that\n files touched by `graphify update` (AST-only) are re-extracted\n semantically.\n kind=\"ast\": a file is \"changed\" when its ast_hash is missing or its\n content has changed. Use this for `graphify update`.\n\n Fast path: mtime unchanged + hash matches → unchanged (free, no disk IO\n beyond stat). Slow path: mtime bumped → compare MD5 against the relevant\n hash field before re-extracting.\n\n Backwards compatible with legacy manifests storing plain float mtime values\n or {mtime, hash} dicts (treated as ast_hash only; semantic_hash = miss).\n\n The ``follow_symlinks`` flag is forwarded to :func:`detect` so in-root\n symlinked sub-trees are scanned consistently between full and incremental\n runs. ``None`` (default) does not follow symlinked directories; callers must\n opt in explicitly, and resolved targets outside the scan root are skipped.\n\n ``cache_root`` is forwarded to :func:`detect`'s own word-count cache the\n same way the fresh-scan (non-incremental) call site already does. Without\n it, an incremental run with a ``--out`` destination outside the scan root\n falls back to anchoring the cache at the scan root itself, leaking\n ``graphify-out/cache/stat-index.json`` there even though the fresh-scan\n path stays clean (#3847).\n \"\"\"\n full = detect(\n root,\n follow_symlinks=follow_symlinks,\n google_workspace=google_workspace,\n extra_excludes=extra_excludes,\n gitignore=gitignore,\n cache_root=cache_root,\n )\n # Pass ``root`` so a manifest written with relative keys (post-#777) is\n # re-anchored to the absolute form the rest of this function compares\n # against. Legacy absolute-keyed manifests pass through unchanged.\n manifest = load_manifest(manifest_path, root=root)\n\n if not manifest:\n # No previous run - treat everything as new\n full[\"incremental\"] = True\n full[\"new_files\"] = full[\"files\"]\n full[\"unchanged_files\"] = {k: [] for k in full[\"files\"]}\n full[\"new_total\"] = full[\"total_files\"]\n full[\"deleted_files\"] = []\n full[\"excluded_files\"] = []\n return full\n\n new_files, unchanged_files = classify_manifest_files(full[\"files\"], manifest, kind=kind)\n\n # Manifest rows that left the corpus, split by disk existence (#1908):\n # a row whose file is gone from DISK is a genuine deletion (its cached\n # nodes are ghosts); a row whose file still exists but is out of the\n # current scan was EXCLUDED (ignore rules / --exclude changed) and must\n # not be reported as deleted. Mirrors the watch-side excluded-vs-deleted\n # distinction (#1795).\n try:\n root_res: Path | None = Path(root).resolve() if root is not None else None\n except (OSError, RuntimeError):\n root_res = Path(root) if root is not None else None\n\n def _in_root(path_str: str) -> bool:\n if root_res is None:\n return True\n p = Path(path_str)\n try:\n p.relative_to(root_res)\n return True\n except ValueError:\n pass\n try:\n p.resolve().relative_to(root_res)\n return True\n except (ValueError, OSError, RuntimeError):\n return False\n\n current_files = {_nfc(f) for flist in full[\"files\"].values() for f in flist}\n # Resolution inputs carry manifest rows but are deliberately kept out of\n # detect()'s corpus. Without this they read as rows that \"left the scan\" and\n # would be reported as excluded on every run (#4740).\n current_files |= {_nfc(f) for f in full.get(\"resolution_inputs\", [])}\n deleted_files: list[str] = []\n excluded_files: list[str] = []\n for f in manifest:\n if not _in_root(f):\n continue\n if _nfc(f) in current_files:\n continue\n try:\n alive = Path(f).exists()\n except OSError:\n alive = False\n (excluded_files if alive else deleted_files).append(f)\n\n new_total = sum(len(v) for v in new_files.values())\n full[\"incremental\"] = True\n full[\"new_files\"] = new_files\n full[\"unchanged_files\"] = unchanged_files\n full[\"new_total\"] = new_total\n full[\"deleted_files\"] = deleted_files\n full[\"excluded_files\"] = excluded_files\n return full\n","content_type":"text/x-python; charset=utf-8","language":"python","size":146012,"content_sha256":"40338956e13286b0ce51f8f7fa44512caa89f9bdf48a6e02de016c1ac7616c75"},{"filename":"diagnostics.py","content":"\"\"\"Read-only diagnostics for MultiDiGraph readiness.\"\"\"\n\nfrom __future__ import annotations\n\nimport json\nimport re\nfrom collections import Counter, defaultdict\nfrom copy import deepcopy\nfrom pathlib import Path\nfrom typing import Any\n\nimport networkx as nx\n\n\n_SUPPRESSION_DECL_RE = re.compile(r\"^\\s*(?P\u003cname>seen_[A-Za-z0-9_]+)\\s*[:=]\")\n_TYPE_TUPLE_RE = re.compile(r\"set\\[tuple\\[(?P\u003cinside>[^\\]]+)\\]\\]\")\n\n\ndef _safe_text(value: Any) -> str:\n if value is None:\n return \"\"\n if isinstance(value, (str, int, float, bool)):\n return str(value)\n return json.dumps(value, sort_keys=True, default=str, ensure_ascii=False)\n\n\ndef _edge_list(extraction: dict[str, Any]) -> list[Any]:\n edges = extraction.get(\"edges\")\n if edges is None:\n edges = extraction.get(\"links\")\n return edges if isinstance(edges, list) else []\n\n\ndef _node_ids(extraction: dict[str, Any]) -> set[str]:\n nodes = extraction.get(\"nodes\", [])\n if not isinstance(nodes, list):\n return set()\n return {\n str(node[\"id\"])\n for node in nodes\n if isinstance(node, dict) and \"id\" in node and node.get(\"id\") is not None\n }\n\n\ndef _canonical_edge(edge: Any) -> dict[str, str]:\n if not isinstance(edge, dict):\n return {\n \"source\": \"\",\n \"target\": \"\",\n \"relation\": \"\",\n \"confidence\": \"\",\n \"source_file\": \"\",\n \"source_location\": \"\",\n \"context\": \"\",\n \"_invalid\": \"non_object_edge\",\n }\n source = edge.get(\"source\", edge.get(\"from\"))\n target = edge.get(\"target\", edge.get(\"to\"))\n return {\n \"source\": _safe_text(source),\n \"target\": _safe_text(target),\n \"relation\": _safe_text(edge.get(\"relation\")),\n \"confidence\": _safe_text(edge.get(\"confidence\")),\n \"source_file\": _safe_text(edge.get(\"source_file\")),\n \"source_location\": _safe_text(edge.get(\"source_location\")),\n \"context\": _safe_text(edge.get(\"context\")),\n \"_invalid\": \"\",\n }\n\n\ndef _exact_signature(edge: Any) -> str:\n if not isinstance(edge, dict):\n return \"\u003cnon-object>\"\n normalized = dict(edge)\n if \"source\" not in normalized and \"from\" in normalized:\n normalized[\"source\"] = normalized[\"from\"]\n if \"target\" not in normalized and \"to\" in normalized:\n normalized[\"target\"] = normalized[\"to\"]\n normalized.pop(\"from\", None)\n normalized.pop(\"to\", None)\n return json.dumps(\n normalized,\n sort_keys=True,\n default=str,\n ensure_ascii=False,\n separators=(\",\", \":\"),\n )\n\n\ndef _count_extra(counter: Counter[Any]) -> int:\n return sum(count - 1 for count in counter.values() if count > 1)\n\n\ndef _variant_group_count(\n grouped_edges: dict[tuple[str, str], list[dict[str, str]]],\n field: str,\n *,\n relation_sensitive: bool = False,\n) -> int:\n groups = 0\n for edges in grouped_edges.values():\n if relation_sensitive:\n by_relation: dict[str, set[str]] = defaultdict(set)\n for edge in edges:\n by_relation[edge[\"relation\"]].add(edge[field])\n groups += sum(1 for values in by_relation.values() if len(values) > 1)\n elif len({edge[field] for edge in edges}) > 1:\n groups += 1\n return groups\n\n\ndef _tuple_arity_from_annotation(line: str) -> int:\n match = _TYPE_TUPLE_RE.search(line)\n if not match:\n return 0\n inside = match.group(\"inside\").strip()\n if not inside:\n return 0\n return inside.count(\",\") + 1\n\n\ndef scan_producer_suppression_sites(path: str | Path) -> dict[str, Any]:\n \"\"\"Find likely `seen_*` producer-suppression sets in an extractor file.\"\"\"\n source_path = Path(path)\n if not source_path.exists():\n return {\n \"path\": str(source_path),\n \"total_sites\": 0,\n \"sites\": [],\n \"error\": \"file not found\",\n }\n\n sites: list[dict[str, Any]] = []\n lines = source_path.read_text(encoding=\"utf-8\").splitlines()\n for lineno, line in enumerate(lines, start=1):\n match = _SUPPRESSION_DECL_RE.match(line)\n if not match:\n continue\n sites.append(\n {\n \"line\": lineno,\n \"name\": match.group(\"name\"),\n \"tuple_arity\": _tuple_arity_from_annotation(line),\n \"sample\": line.strip()[:120],\n }\n )\n\n return {\n \"path\": str(source_path),\n \"total_sites\": len(sites),\n \"sites\": sites,\n \"error\": \"\",\n }\n\n\ndef diagnose_extraction(\n extraction: dict[str, Any],\n *,\n directed: bool = True,\n root: str | Path | None = None,\n max_examples: int = 5,\n extract_path: str | Path | None = None,\n) -> dict[str, Any]:\n \"\"\"Summarize graph-integrity risks for one JSON graph/extraction dict.\n\n Missing ``ref_*`` targets are expected external references produced for\n dependencies outside the scanned corpus. They are counted separately from\n actionable internal dangling endpoints.\n \"\"\"\n from graphify.build import build_from_json\n\n node_ids = _node_ids(extraction)\n raw_edges = _edge_list(extraction)\n canonical_edges = [_canonical_edge(edge) for edge in raw_edges]\n\n # Code-typed semantic nodes the extractor could not verify against the source\n # it read (#1949): likely-inferred (or hallucinated) symbols surfaced from a\n # document. Count them so the flag on graph.json nodes is actually surfaced.\n unverified_node_count = sum(\n 1 for n in extraction.get(\"nodes\", [])\n if isinstance(n, dict) and n.get(\"verification\") == \"unverified\"\n )\n\n exact_counts: Counter[str] = Counter(_exact_signature(edge) for edge in raw_edges)\n directed_pairs: Counter[tuple[str, str]] = Counter()\n undirected_pairs: Counter[tuple[str, str]] = Counter()\n grouped: dict[tuple[str, str], list[dict[str, str]]] = defaultdict(list)\n\n non_object_edges = 0\n missing_endpoint_edges = 0\n dangling_endpoint_edges = 0\n external_reference_edges = 0\n self_loop_edges = 0\n valid_candidate_edges = 0\n\n for edge in canonical_edges:\n if edge[\"_invalid\"]:\n non_object_edges += 1\n continue\n source = edge[\"source\"]\n target = edge[\"target\"]\n if not source or not target:\n missing_endpoint_edges += 1\n continue\n if source not in node_ids or target not in node_ids:\n # Two intentionally-distinct populations of \"external\": a `ref_`\n # target here is a JSON/config out-of-corpus reference ($ref /\n # extends / references), whereas import-family externals are minted\n # as declared `external=True` stub nodes at build time (#2873) and\n # therefore never reach this branch. Don't try to unify them.\n if source in node_ids and target.startswith(\"ref_\"):\n external_reference_edges += 1\n else:\n dangling_endpoint_edges += 1\n continue\n if source == target:\n self_loop_edges += 1\n valid_candidate_edges += 1\n directed_pair = (source, target)\n undirected_pair = (source, target) if source \u003c= target else (target, source)\n directed_pairs[directed_pair] += 1\n undirected_pairs[undirected_pair] += 1\n grouped[directed_pair].append(edge)\n\n examples: list[dict[str, Any]] = []\n if max_examples > 0:\n for (source, target), count in directed_pairs.most_common():\n if count \u003c 2:\n continue\n edges = grouped[(source, target)]\n examples.append(\n {\n \"source\": source,\n \"target\": target,\n \"edge_count\": count,\n \"relations\": sorted({edge[\"relation\"] for edge in edges}),\n \"source_files\": sorted({edge[\"source_file\"] for edge in edges}),\n \"source_locations\": sorted({edge[\"source_location\"] for edge in edges}),\n \"contexts\": sorted({edge[\"context\"] for edge in edges}),\n }\n )\n if len(examples) >= max_examples:\n break\n\n build_error = \"\"\n graph_type = \"\"\n post_build_edge_count: int | None = None\n post_build_node_count: int | None = None\n try:\n graph_input = deepcopy(extraction)\n graph: nx.Graph = build_from_json(graph_input, directed=directed, root=root)\n graph_type = type(graph).__name__\n post_build_edge_count = graph.number_of_edges()\n post_build_node_count = graph.number_of_nodes()\n except Exception as exc:\n build_error = f\"{type(exc).__name__}: {exc}\"\n\n suppression_path = (\n Path(extract_path) if extract_path else Path(__file__).with_name(\"extract.py\")\n )\n\n return {\n \"node_count\": len(node_ids),\n \"unverified_node_count\": unverified_node_count,\n \"raw_edge_count\": len(raw_edges),\n \"non_object_edges\": non_object_edges,\n \"missing_endpoint_edges\": missing_endpoint_edges,\n \"dangling_endpoint_edges\": dangling_endpoint_edges,\n \"external_reference_edges\": external_reference_edges,\n \"self_loop_edges\": self_loop_edges,\n \"valid_candidate_edges\": valid_candidate_edges,\n \"exact_duplicate_edges\": _count_extra(exact_counts),\n \"directed_unique_endpoint_pairs\": len(directed_pairs),\n \"directed_same_endpoint_collapsed_edges\": _count_extra(directed_pairs),\n \"undirected_unique_endpoint_pairs\": len(undirected_pairs),\n \"undirected_same_endpoint_collapsed_edges\": _count_extra(undirected_pairs),\n \"same_endpoint_group_count\": sum(1 for count in directed_pairs.values() if count > 1),\n \"relation_variant_groups\": _variant_group_count(grouped, \"relation\"),\n \"source_file_variant_groups\": _variant_group_count(\n grouped, \"source_file\", relation_sensitive=True\n ),\n \"source_location_variant_groups\": _variant_group_count(\n grouped, \"source_location\", relation_sensitive=True\n ),\n \"context_variant_groups\": _variant_group_count(grouped, \"context\", relation_sensitive=True),\n \"post_build_graph_type\": graph_type,\n \"post_build_node_count\": post_build_node_count,\n \"post_build_edge_count\": post_build_edge_count,\n \"post_build_error\": build_error,\n \"producer_suppression\": scan_producer_suppression_sites(suppression_path),\n \"examples\": examples,\n }\n\n\ndef _read_json_file(path: str | Path) -> dict[str, Any]:\n \"\"\"Read a JSON graph after applying Graphify's graph-load size cap.\"\"\"\n from graphify.security import check_graph_file_size_cap\n\n json_path = Path(path)\n check_graph_file_size_cap(json_path)\n try:\n data = json.loads(json_path.read_text(encoding=\"utf-8\"))\n except (json.JSONDecodeError, OSError) as exc:\n raise RuntimeError(\n f\"Cannot parse {json_path}: {exc}. \"\n \"The file may be corrupted — re-run 'graphify extract'.\"\n ) from exc\n if not isinstance(data, dict):\n raise ValueError(\"diagnostic input must be a JSON object\")\n return data\n\n\ndef diagnose_file(\n path: str | Path,\n *,\n directed: bool | None = None,\n root: str | Path | None = None,\n max_examples: int = 5,\n extract_path: str | Path | None = None,\n) -> dict[str, Any]:\n \"\"\"Diagnose a graph/extraction JSON file without mutating it.\n\n When `directed` is None, the JSON's \"directed\" flag is honored. Raw\n extraction JSON that has no \"directed\" flag defaults to directed analysis.\n \"\"\"\n data = _read_json_file(path)\n if directed is None:\n raw_directed = data.get(\"directed\")\n effective_directed = raw_directed if isinstance(raw_directed, bool) else True\n else:\n effective_directed = directed\n\n summary = diagnose_extraction(\n data,\n directed=effective_directed,\n root=root,\n max_examples=max_examples,\n extract_path=extract_path,\n )\n summary[\"input_path\"] = str(path)\n summary[\"effective_directed\"] = effective_directed\n return summary\n\n\ndef format_diagnostic_json(summary: dict[str, Any]) -> dict[str, Any]:\n return {\n \"schema_version\": 1,\n \"summary\": {\n key: value\n for key, value in summary.items()\n if key not in {\"examples\", \"producer_suppression\"}\n },\n \"examples\": summary.get(\"examples\", []),\n \"producer_suppression\": summary.get(\"producer_suppression\", {}),\n \"notes\": [\n \"Diagnostics are read-only.\",\n \"A normal graph.json is already post-build and cannot recover raw producer edges.\",\n \"Producer suppression sites are heuristic source-code evidence.\",\n ],\n }\n\n\ndef format_diagnostic_report(summary: dict[str, Any]) -> str:\n suppression = summary.get(\"producer_suppression\", {})\n lines = [\n \"[graphify] MultiDiGraph edge-collapse diagnostic\",\n f\"input: {summary.get('input_path', '\u003cin-memory>')}\",\n \"input_stage: provided JSON (normal graph.json is post-build)\",\n f\"effective_directed: {summary.get('effective_directed', '\u003cdirect-call>')}\",\n f\"nodes: {summary['node_count']}\",\n f\"unverified_code_nodes: {summary.get('unverified_node_count', 0)}\",\n f\"raw_edges: {summary['raw_edge_count']}\",\n f\"valid_candidate_edges: {summary['valid_candidate_edges']}\",\n f\"missing_endpoint_edges: {summary['missing_endpoint_edges']}\",\n f\"dangling_endpoint_edges: {summary['dangling_endpoint_edges']}\",\n f\"external_reference_edges: {summary.get('external_reference_edges', 0)}\",\n f\"self_loop_edges: {summary['self_loop_edges']}\",\n f\"exact_duplicate_edges: {summary['exact_duplicate_edges']}\",\n f\"directed_unique_endpoint_pairs: {summary['directed_unique_endpoint_pairs']}\",\n (\n \"directed_same_endpoint_collapsed_edges: \"\n f\"{summary['directed_same_endpoint_collapsed_edges']}\"\n ),\n f\"undirected_unique_endpoint_pairs: {summary['undirected_unique_endpoint_pairs']}\",\n (\n \"undirected_same_endpoint_collapsed_edges: \"\n f\"{summary['undirected_same_endpoint_collapsed_edges']}\"\n ),\n f\"same_endpoint_group_count: {summary['same_endpoint_group_count']}\",\n f\"relation_variant_groups: {summary['relation_variant_groups']}\",\n f\"source_file_variant_groups: {summary['source_file_variant_groups']}\",\n f\"source_location_variant_groups: {summary['source_location_variant_groups']}\",\n f\"context_variant_groups: {summary['context_variant_groups']}\",\n f\"post_build_graph_type: {summary['post_build_graph_type']}\",\n f\"post_build_edges: {summary['post_build_edge_count']}\",\n f\"producer_suppression_sites: {suppression.get('total_sites', 0)}\",\n ]\n if summary.get(\"post_build_error\"):\n lines.append(f\"post_build_error: {summary['post_build_error']}\")\n if suppression.get(\"error\"):\n lines.append(f\"producer_suppression_error: {suppression['error']}\")\n if suppression.get(\"sites\"):\n lines.append(\"producer_suppression_examples:\")\n for site in suppression[\"sites\"][:8]:\n lines.append(\n f\" - L{site['line']} {site['name']} arity={site['tuple_arity'] or 'unknown'}\"\n )\n if summary.get(\"examples\"):\n lines.append(\"examples:\")\n for example in summary[\"examples\"]:\n lines.append(\n \" - \"\n f\"{example['source']} -> {example['target']} \"\n f\"edges={example['edge_count']} \"\n f\"relations={example['relations']} \"\n f\"locations={example['source_locations']} \"\n f\"contexts={example['contexts']}\"\n )\n lines.append(\n \"note: normal graph.json is post-build; raw producer loss must be measured earlier.\"\n )\n return \"\\n\".join(lines)\n","content_type":"text/x-python; charset=utf-8","language":"python","size":15990,"content_sha256":"2032df8c35a192c60c33cbc2eaf8e4516936bba301b8f25acf407ac54d333278"},{"filename":"export.py","content":"# write graph to HTML, JSON, SVG, GraphML, Obsidian vault, and Neo4j Cypher\nfrom __future__ import annotations\nimport hashlib\nimport html as _html\nimport json\nimport math\nimport os\nimport re\nimport shutil\nimport sys\nfrom collections import Counter\nfrom datetime import date\nfrom pathlib import Path\nimport networkx as nx\nfrom networkx.readwrite import json_graph\nfrom graphify.security import sanitize_label\nfrom graphify.analyze import _node_community_map\nfrom graphify.build import edge_datas, has_parallel_links\nfrom graphify.paths import (\n os_replace_with_fallback,\n stem_filename_budget,\n write_json_atomic,\n write_text_atomic,\n write_text_atomic_if_changed,\n)\n\nfrom graphify.exporters.graphdb import push_to_falkordb, push_to_neo4j # noqa: E402,F401\n\n\n# Artifacts worth preserving across rebuilds (non-regenerable without LLM or curation).\n_BACKUP_ARTIFACTS = [\n \"graph.json\",\n \"GRAPH_REPORT.md\",\n \".graphify_labels.json\",\n \".graphify_analysis.json\",\n \"manifest.json\",\n \".graphify_semantic_marker\",\n \"cost.json\",\n]\n\n\ndef backup_if_protected(out_dir: Path) -> \"Path | None\":\n \"\"\"Snapshot graph artifacts to a dated subfolder before an overwrite.\n\n Triggers when graph.json exists AND either:\n - .graphify_semantic_marker is present (graph cost real LLM tokens), or\n - .graphify_labels.json contains at least one non-default community label\n (graph has been curated by a human or skill).\n\n Returns the backup folder path, or None if no backup was taken.\n Never raises — backup failure prints a warning but never blocks the write.\n Set GRAPHIFY_NO_BACKUP=1 to disable.\n \"\"\"\n if os.environ.get(\"GRAPHIFY_NO_BACKUP\"):\n return None\n out = Path(out_dir)\n if not (out / \"graph.json\").exists():\n return None\n\n is_semantic = (out / \".graphify_semantic_marker\").exists()\n is_curated = False\n labels_file = out / \".graphify_labels.json\"\n if labels_file.exists():\n try:\n labels = json.loads(labels_file.read_text(encoding=\"utf-8\"))\n is_curated = any(v != f\"Community {k}\" for k, v in labels.items())\n except Exception:\n pass\n\n if not is_semantic and not is_curated:\n return None\n\n reason = \"+\".join(filter(None, [\"semantic\" if is_semantic else \"\", \"curated\" if is_curated else \"\"]))\n today = date.today().isoformat()\n backup_dir = out / today\n graph_src = out / \"graph.json\"\n\n try:\n # Skip re-copying if today's backup already holds every artifact with an\n # identical graph.json. graph.json alone is not proof of completeness: a\n # snapshot missing a sidecar (an earlier copy attempt failed part-way)\n # must fall through and retry the copy instead of reporting itself\n # complete forever (#3769). If content differs, overwrite the backup in\n # place — one folder per day, always the latest pre-overwrite state.\n if backup_dir.exists() and (backup_dir / \"graph.json\").exists():\n src_hash = hashlib.sha256(graph_src.read_bytes()).hexdigest()\n bak_hash = hashlib.sha256((backup_dir / \"graph.json\").read_bytes()).hexdigest()\n if src_hash == bak_hash and all(\n (backup_dir / name).exists()\n for name in _BACKUP_ARTIFACTS\n if (out / name).exists()\n ):\n return backup_dir # identical content, nothing to do\n backup_dir.mkdir(parents=True, exist_ok=True)\n copied = 0\n failed: list[str] = []\n for name in _BACKUP_ARTIFACTS:\n src = out / name\n if src.exists():\n try:\n shutil.copy2(src, backup_dir / name)\n copied += 1\n except Exception:\n failed.append(name)\n if failed:\n # A partial snapshot reported as a successful backup lets a stale\n # graph.json survive the next overwrite unnoticed (#3769). Best-effort\n # is unchanged: warn, report no backup, do not block the write.\n print(\n f\"[graphify] warning: backup incomplete - could not copy \"\n f\"{', '.join(failed)} -> {backup_dir.name}/ - continuing with overwrite\",\n file=sys.stderr,\n )\n return None\n if copied:\n print(f\"[graphify] backed up {reason} graph ({copied} files) -> {backup_dir.name}/\")\n return backup_dir\n except Exception as exc:\n print(f\"[graphify] warning: backup failed ({exc}) - continuing with overwrite\", file=sys.stderr)\n return None\n\ndef _obsidian_tag(name: str) -> str:\n r\"\"\"Sanitize a community name for use as an Obsidian tag.\n\n Obsidian tags accept letters from any language plus digits, hyphens,\n underscores and slashes; spaces and most punctuation are not allowed, and a\n tag cannot be digits-only. ``\\w`` is Unicode-aware in Python 3, so Hangul,\n CJK, Cyrillic and accented Latin survive instead of being stripped (#2862):\n an ASCII-only filter collapsed every non-Latin community label to\n underscores, so every note in that community carried the same tag.\n \"\"\"\n tag = re.sub(r\"[^\\w\\-/]\", \"\", name.replace(\" \", \"_\"))\n if not tag.strip(\"_-/\"):\n return \"unnamed\" # label was punctuation only\n if tag.isdigit():\n return f\"c{tag}\" # Obsidian ignores digits-only tags\n return tag\n\n\ndef _strip_diacritics(text: str | None) -> str:\n import unicodedata\n if not isinstance(text, str):\n text = \"\" if text is None else str(text)\n nfkd = unicodedata.normalize(\"NFKD\", text)\n return \"\".join(c for c in nfkd if not unicodedata.combining(c))\n\n\ndef _yaml_str(s: str) -> str:\n \"\"\"Escape a value for safe embedding in a YAML double-quoted scalar (F-009).\n\n See `graphify.ingest._yaml_str` for the full rationale; duplicated here to\n avoid pulling the URL-fetching `ingest` module into export's dependency\n graph. Handles backslash, double-quote, all line breaks (\\\\n, \\\\r,\n U+2028, U+2029), tab, NUL, and other C0/DEL control characters that\n would otherwise let a hostile `source_file` / `community` / etc. break\n out of the YAML scalar and inject sibling keys.\n \"\"\"\n if s is None:\n return \"\"\n out: list[str] = []\n for ch in str(s):\n cp = ord(ch)\n if ch == \"\\\\\":\n out.append(\"\\\\\\\\\")\n elif ch == '\"':\n out.append('\\\\\"')\n elif ch == \"\\n\":\n out.append(\"\\\\n\")\n elif ch == \"\\r\":\n out.append(\"\\\\r\")\n elif ch == \"\\t\":\n out.append(\"\\\\t\")\n elif ch == \"\\0\":\n out.append(\"\\\\0\")\n elif cp == 0x2028:\n out.append(\"\\\\L\")\n elif cp == 0x2029:\n out.append(\"\\\\P\")\n elif cp \u003c 0x20 or cp == 0x7F:\n out.append(f\"\\\\x{cp:02x}\")\n else:\n out.append(ch)\n return \"\".join(out)\n\n\nfrom graphify.exporters.base import COMMUNITY_COLORS # noqa: E402,F401\n\nfrom graphify.exporters.html import to_html # noqa: E402,F401\n\n\n# Increment when the persisted graph structure changes incompatibly for consumers.\nGRAPH_SCHEMA_VERSION = 1\n\n\ndef _graphify_version() -> str | None:\n \"\"\"Return the installed graphify version for graph provenance.\"\"\"\n try:\n from importlib.metadata import version\n\n return version(\"graphifyy\")\n except Exception:\n return None\n\n\n# Fallback scores for an edge that carries a confidence tier but no\n# confidence_score. The INFERRED default was 0.5, which references/extraction-spec.md\n# rules out in as many words — \"never omit it, never use 0.5 as a default\" — and\n# which is not in the discrete INFERRED set {0.55, 0.65, 0.75, 0.85, 0.95} either.\n# It is now the bottom of that set: a missing score is an absence of evidence\n# about strength, so the honest fallback is the weakest value the rubric allows,\n# not a midpoint that reads as a coin flip (#2813). Every AST emission site now\n# supplies its own score, so this is a backstop rather than a routine path.\n_CONFIDENCE_SCORE_DEFAULTS = {\"EXTRACTED\": 1.0, \"INFERRED\": 0.55, \"AMBIGUOUS\": 0.2}\n\n\ndef attach_hyperedges(G: nx.Graph, hyperedges: list) -> None:\n \"\"\"Store hyperedges in the graph's metadata dict.\"\"\"\n existing = G.graph.get(\"hyperedges\", [])\n # Skip id-less persisted entries when seeding the dedup set (#2775): the\n # semantic extractor emits hyperedges with no `id` and build.py persists them\n # verbatim, so a prior graph.json can contain id-less hyperedges. A hard\n # `h[\"id\"]` here raised `KeyError: 'id'` on every incremental re-extract,\n # symmetric with the `.get(\"id\")` guard the loop below already applies to the\n # incoming set.\n seen_ids = {h[\"id\"] for h in existing if h.get(\"id\")}\n for h in hyperedges:\n if h.get(\"id\") and h[\"id\"] not in seen_ids:\n existing.append(h)\n seen_ids.add(h[\"id\"])\n G.graph[\"hyperedges\"] = existing\n\n\ndef _git_head(cwd: \"str | Path | None\" = None) -> str | None:\n \"\"\"Return git HEAD for the repo containing ``cwd``, or None outside a repo.\n\n ``cwd`` selects the repository to ask, exactly as in watch._git_head\n (#2316). Without it the command inherits the caller's working directory,\n which stamps the *invoking* repo's commit when the graph being written\n describes a different repo — provenance must come from the repo the graph\n describes, so callers pass the graph's own location.\n \"\"\"\n import subprocess as _sp\n try:\n r = _sp.run(\n [\"git\", \"rev-parse\", \"HEAD\"], stdin=_sp.DEVNULL,\n capture_output=True, text=True, timeout=3,\n cwd=str(cwd) if cwd is not None else None,\n )\n return r.stdout.strip() if r.returncode == 0 else None\n except Exception:\n return None\n\n\n# Sentinel: an existing graph.json is present and non-empty but cannot be parsed\n# into a node count (corrupt, mid-write, or structurally wrong). The caller must\n# fail CLOSED on this — the same way to_json's #479 guard refuses to overwrite\n# such a file — because we cannot prove the new graph isn't a silent shrink.\nMALFORMED_GRAPH = object()\n\n\ndef existing_graph_node_count(path: \"str | Path\"):\n \"\"\"Node count of an existing graph.json.\n\n Returns:\n - an ``int`` node count when the file parses;\n - ``None`` when there is verifiably nothing to protect — absent, empty, or\n over the size cap (matching how :func:`to_json` lets the new graph\n replace an empty/oversized file);\n - :data:`MALFORMED_GRAPH` when the file is present and non-empty but\n unparseable — the caller must treat this as fail-closed (refuse to\n overwrite), mirroring to_json's #479 handling of a corrupt/mid-write file.\n\n The raw ``--no-cluster`` write path uses this to apply the same #479 shrink\n guard that :func:`to_json` applies inline for the clustered path.\n \"\"\"\n p = Path(path)\n if not p.exists():\n return None\n from graphify.security import check_graph_file_size_cap\n try:\n check_graph_file_size_cap(p)\n except Exception:\n # Oversized: reading it to compare would be the DoS the cap guards against.\n return None\n try:\n raw = p.read_text(encoding=\"utf-8\")\n except Exception:\n # Present but unreadable: fail closed if it has bytes, else nothing to lose.\n try:\n return MALFORMED_GRAPH if p.stat().st_size > 0 else None\n except Exception:\n return None\n if not raw.strip():\n return None\n try:\n data = json.loads(raw)\n except Exception:\n return MALFORMED_GRAPH\n nodes = data.get(\"nodes\") if isinstance(data, dict) else None\n return len(nodes) if isinstance(nodes, list) else MALFORMED_GRAPH\n\n\ndef to_json(G: nx.Graph, communities: dict[int, list[str]], output_path: str, *, force: bool = False, built_at_commit: str | None = None, community_labels: dict[int, str] | None = None, original_links: \"list[dict] | None\" = None) -> bool:\n # Drop #3774 accounting before any write. Extract pops these first; every\n # other caller of to_json (the documented build_merge persist path) does not.\n from graphify.build import take_shrink_accounting\n take_shrink_accounting(G)\n # Safety check: refuse to silently shrink an existing graph (#479)\n existing_path = Path(output_path)\n if not force and existing_path.exists():\n from graphify.security import check_graph_file_size_cap\n try:\n check_graph_file_size_cap(existing_path)\n except Exception:\n # Existing graph.json trips the size cap; reading it to compare would\n # be the very DoS the cap guards against. Can't verify — let the new\n # graph replace the oversized file.\n oversized = True\n else:\n oversized = False\n if not oversized:\n try:\n raw = existing_path.read_text(encoding=\"utf-8\")\n except Exception:\n raw = \"\"\n if not raw.strip():\n # Empty/whitespace existing file (e.g. a freshly touched path):\n # no nodes to lose, so any new graph is a growth — proceed.\n existing_n = 0\n existing_nodes = []\n else:\n try:\n existing_data = json.loads(raw)\n existing_nodes = existing_data.get(\"nodes\", [])\n if not isinstance(existing_nodes, list):\n existing_nodes = []\n existing_n = len(existing_nodes)\n except Exception as exc:\n # Non-empty but unparseable existing graph (corrupt or a\n # mid-write): we cannot verify the new graph is not a silent\n # shrink. Fail SAFE — refuse rather than overwrite. A\n # fail-OPEN here (the prior behavior) is the silent data-loss\n # path #479 exists to prevent: a transiently unreadable\n # graph.json would let a partial rebuild clobber a good one.\n import sys as _sys\n print(\n f\"[graphify] WARNING: existing {existing_path} could not be \"\n f\"read to verify the new graph is not smaller ({exc}). \"\n f\"Refusing to overwrite; pass force=True to override.\",\n file=_sys.stderr,\n )\n return False\n new_n = G.number_of_nodes()\n from graphify.build import _lost_saved_semantic_nodes\n new_node_list = [\n {\"id\": node_id, **{key: value for key, value in attrs.items() if key != \"id\"}}\n for node_id, attrs in G.nodes(data=True)\n ]\n folded: dict[str, str] = {}\n missing = _lost_saved_semantic_nodes(existing_nodes, new_node_list, set(), folded)\n if missing:\n import sys as _sys\n ids = \", \".join(str(node.get(\"id\")) for node in missing)\n print(\n \"[graphify] WARNING: Refusing to overwrite — saved semantic \"\n f\"node(s) are missing from the new graph: {ids}.\",\n file=_sys.stderr,\n )\n return False\n # A saved semantic node merged into an AST twin that was already\n # saved is one node, not two; a reload-and-recluster folds it (#4389).\n # A twin new to this run does not offset some other node's loss.\n existing_ids = {n.get(\"id\") for n in existing_nodes if isinstance(n, dict)}\n merged = sum(twin in existing_ids for twin in folded.values())\n if new_n \u003c existing_n - merged:\n import sys as _sys\n print(\n f\"[graphify] WARNING: new graph has {new_n} nodes but existing \"\n f\"graph.json has {existing_n} (net -{existing_n - new_n}). \"\n f\"Refusing to overwrite. Possible causes: missing chunk files from \"\n f\"a previous session, or fuzzy dedup collapsed same-named symbols \"\n f\"across files during an --update on an already-current graph. \"\n f\"Run a full rebuild (/graphify .) to be safe, or pass force=True \"\n f\"only if you have verified the reduction is legitimate.\",\n file=_sys.stderr,\n )\n return False\n\n node_community = _node_community_map(communities)\n _labels: dict[int, str] = {int(k): v for k, v in (community_labels or {}).items()}\n try:\n data = json_graph.node_link_data(G, edges=\"links\")\n except TypeError:\n data = json_graph.node_link_data(G)\n\n def _json_sort_key(item: dict) -> str:\n return json.dumps(item, ensure_ascii=False, sort_keys=True, separators=(\",\", \":\"))\n\n for node in data[\"nodes\"]:\n cid = node_community.get(node[\"id\"])\n node[\"community\"] = cid\n if cid is not None and _labels:\n node[\"community_name\"] = _labels.get(cid, f\"Community {cid}\")\n node[\"norm_label\"] = _strip_diacritics(node.get(\"label\", \"\")).lower()\n if original_links is not None:\n # A simple Graph keeps one edge per node pair, so re-deriving the link\n # list from G after a reload-and-recluster (cluster-only/label) would\n # silently drop a second edge on a pair that stayed connected by\n # another (e.g. an `imports` and a `calls` edge between the same two\n # nodes, #3999). Community is purely a node attribute, so clustering\n # never needs to add or remove an edge; write back every original\n # link verbatim instead, dropping only one whose endpoint no longer\n # exists in G. This also sidesteps any endpoint-order canonicalization\n # undirected storage would otherwise apply when deriving from G.\n #\n # The endpoint-existence filter below assumes original_links carry the\n # SAME ids as G's post-load nodes. That holds because build_from_json\n # rewrites a legacy node id to its canonical stem (_semantic_id_remap /\n # _doc_twin_remap) IN PLACE on the shared link dicts the caller then\n # passes here, so a remapped endpoint already matches node_ids rather\n # than being silently dropped (regression-tested in test_cli_export.py:\n # test_cluster_only_preserves_parallel_edges_across_an_id_remap).\n node_ids = {n[\"id\"] for n in data[\"nodes\"]}\n data[\"links\"] = [\n dict(link) for link in original_links\n if isinstance(link, dict)\n and link.get(\"source\") in node_ids and link.get(\"target\") in node_ids\n ]\n # The flag above was derived from the in-memory simple graph, but the\n # restored links may hold parallel edges (#3999); keep the serialized\n # metadata consistent with the links or readers collapse them (#4500).\n if has_parallel_links(data[\"links\"], directed=bool(data.get(\"directed\", True))):\n data[\"multigraph\"] = True\n for link in data[\"links\"]:\n if \"confidence_score\" not in link:\n conf = link.get(\"confidence\", \"EXTRACTED\")\n link[\"confidence_score\"] = _CONFIDENCE_SCORE_DEFAULTS.get(conf, 1.0)\n # Restore original edge direction. Undirected NetworkX storage may\n # canonicalize endpoint order, flipping `calls` and other directional\n # edges in graph.json. The build path stashes the true endpoints in\n # _src/_tgt for exactly this purpose (#563).\n true_src = link.pop(\"_src\", None)\n true_tgt = link.pop(\"_tgt\", None)\n if true_src is not None and true_tgt is not None:\n link[\"source\"] = true_src\n link[\"target\"] = true_tgt\n # Canonicalize the key order WITHIN each node/link dict. node_link_data always\n # appends the node key (`id`) at the end, so a node whose `id` was an inline\n # attribute on a cold build (position varies) lands last after a read-rebuild\n # (build_from_json consumes `id` as the pure node key). The values are\n # identical either way, but the field order churns, so a byte-diff of two\n # equivalent graph.json files is noisy and any position-sensitive consumer\n # sees a spurious change on every round-trip. Emit a stable order — the\n # identity keys first, then the remaining keys sorted — so the serialized\n # form is invariant regardless of how the attribute was stored in memory.\n def _canonical(item: dict, lead: tuple[str, ...]) -> dict:\n leading = [k for k in lead if k in item]\n rest = sorted(k for k in item if k not in leading)\n return {k: item[k] for k in (*leading, *rest)}\n\n data[\"nodes\"] = [_canonical(n, (\"id\", \"label\")) for n in data[\"nodes\"]]\n data[\"links\"] = [_canonical(link, (\"source\", \"target\", \"relation\")) for link in data[\"links\"]]\n data[\"nodes\"].sort(key=_json_sort_key)\n data[\"links\"].sort(key=_json_sort_key)\n if \"hyperedges\" not in getattr(G, \"graph\", {}):\n # Hardening (#2485): a graph with NO hyperedges key at all was built by\n # a path that never engaged hyperedge metadata — distinct from an\n # intentional empty set ([], which build_from_json now stores\n # explicitly after a full-wipeout revalidation). If the file on disk\n # already holds a non-empty set, emptying it without a trace is silent\n # data loss; warn loudly so the wipeout is attributable. We still write\n # the graph's truth rather than preserving the stale set — resurrecting\n # hyperedges whose members may no longer exist would reintroduce the\n # dangling-member shape #1916 removed.\n _prev_hyperedges = None\n try:\n if existing_path.exists():\n from graphify.security import check_graph_file_size_cap\n check_graph_file_size_cap(existing_path)\n _prev = json.loads(existing_path.read_text(encoding=\"utf-8\"))\n if isinstance(_prev, dict):\n _prev_hyperedges = _prev.get(\"hyperedges\")\n except Exception:\n _prev_hyperedges = None\n if _prev_hyperedges:\n print(\n f\"[graphify] WARNING: graph carries no hyperedge metadata but \"\n f\"{existing_path} already holds {len(_prev_hyperedges)} \"\n f\"hyperedge(s); writing an empty set. Rebuild from the original \"\n f\"extraction if this is unexpected.\",\n file=sys.stderr,\n )\n hyperedges = sorted(getattr(G, \"graph\", {}).get(\"hyperedges\", []), key=_json_sort_key)\n if isinstance(data.get(\"graph\"), dict) and \"hyperedges\" in data[\"graph\"]:\n data[\"graph\"][\"hyperedges\"] = hyperedges\n data[\"hyperedges\"] = hyperedges\n graph_metadata = data.setdefault(\"graph\", {})\n graph_metadata[\"schema_version\"] = GRAPH_SCHEMA_VERSION\n graphify_version = _graphify_version()\n if graphify_version is not None:\n graph_metadata[\"graphify_version\"] = graphify_version\n # Fallback provenance comes from the repo the graph is being written INTO\n # (output_path lives in \u003ctarget>/graphify-out/), never the shell's cwd —\n # the same cwd-anchoring mistake #2316 fixed for `update`.\n commit = built_at_commit if built_at_commit is not None else _git_head(Path(output_path).resolve().parent)\n if commit:\n data[\"built_at_commit\"] = commit\n from graphify.paths import write_json_atomic\n # Atomic write: a crash/ENOSPC mid-write must not truncate a good graph.json.\n write_json_atomic(output_path, data, compact=True)\n return True\n\n\ndef prune_dangling_edges(graph_data: dict) -> tuple[dict, int]:\n \"\"\"Remove edges whose source or target node is not in the node set.\n\n Returns the cleaned graph_data dict and the number of pruned edges.\n \"\"\"\n node_ids = {n[\"id\"] for n in graph_data[\"nodes\"]}\n links_key = \"links\" if \"links\" in graph_data else \"edges\"\n before = len(graph_data[links_key])\n graph_data[links_key] = [\n e for e in graph_data[links_key]\n if e[\"source\"] in node_ids and e[\"target\"] in node_ids\n ]\n return graph_data, before - len(graph_data[links_key])\n\n\ndef _cypher_escape(s: str) -> str:\n \"\"\"Escape a string for safe embedding in a Cypher single-quoted literal.\n\n Handles all characters that could prematurely terminate the literal or\n inject control sequences:\n - `\\\\` and `'` (literal terminators)\n - newlines/CRs (would break the per-line statement framing)\n - NUL/control bytes (defensive — Neo4j errors on raw NULs)\n\n Also strips any leading/trailing whitespace that would let an attacker\n break the `;`-terminated statement boundary used by `cypher-shell`.\n Closing `}` and `)` are NOT special inside a single-quoted Cypher string,\n so escaping the quote and backslash correctly is sufficient (a `}` inside\n a properly-closed `'...'` literal is just a character) — but we previously\n missed `\\\\n` / `\\\\r` which DO let a payload break out of the statement\n line and inject a fresh MATCH/DELETE on the following line. See F-008.\n \"\"\"\n # First normalise: drop NUL and other C0 control chars except tab.\n s = \"\".join(ch for ch in s if ch >= \" \" or ch == \"\\t\")\n return (\n s.replace(\"\\\\\", \"\\\\\\\\\")\n .replace(\"'\", \"\\\\'\")\n .replace(\"\\n\", \"\\\\n\")\n .replace(\"\\r\", \"\\\\r\")\n )\n\n\n# Restrict identifier-position values (labels and relationship types are NOT\n# quoted in Cypher and so cannot be safely escaped — they must be allowlisted).\n_CYPHER_IDENT_RE = re.compile(r\"[^A-Za-z0-9_]\")\n\n\ndef _cypher_label(raw: str, fallback: str) -> str:\n \"\"\"Sanitise a value used in identifier position (node label / rel type).\n\n Cypher does not provide a way to escape `:Foo` label syntax, so we must\n strip everything except `[A-Za-z0-9_]` and require the result to start\n with a letter; otherwise we fall back to a safe constant.\n \"\"\"\n cleaned = _CYPHER_IDENT_RE.sub(\"\", raw or \"\")\n if not cleaned or not cleaned[0].isalpha():\n return fallback\n return cleaned\n\n\ndef to_cypher(G: nx.Graph, output_path: str) -> None:\n lines = [\"// Neo4j Cypher import - generated by /graphify\", \"\"]\n for node_id, data in G.nodes(data=True):\n label = _cypher_escape(data.get(\"label\", node_id))\n node_id_esc = _cypher_escape(node_id)\n ftype = _cypher_label(\n (data.get(\"file_type\", \"unknown\") or \"unknown\").capitalize(),\n \"Entity\",\n )\n lines.append(f\"MERGE (n:{ftype} {{id: '{node_id_esc}'}}) SET n.label = '{label}';\")\n lines.append(\"\")\n for u, v, data in G.edges(data=True):\n u, v = data.get(\"_src\", u), data.get(\"_tgt\", v)\n rel = _cypher_label(\n (data.get(\"relation\", \"RELATES_TO\") or \"RELATES_TO\").upper(),\n \"RELATES_TO\",\n )\n conf = _cypher_escape(data.get(\"confidence\", \"EXTRACTED\"))\n u_esc = _cypher_escape(u)\n v_esc = _cypher_escape(v)\n lines.append(\n f\"MATCH (a {{id: '{u_esc}'}}), (b {{id: '{v_esc}'}}) \"\n f\"MERGE (a)-[:{rel} {{confidence: '{conf}'}}]->(b);\"\n )\n with open(output_path, \"w\", encoding=\"utf-8\") as f: # nosec\n f.write(\"\\n\".join(lines))\n\n\n# Keep backward-compatible alias - skill.md calls generate_html\ngenerate_html = to_html\n\n\n# Characters XML 1.0 cannot carry: the C0 controls except tab, LF and CR.\n_XML_ILLEGAL_RE = re.compile(\"[\\x00-\\x08\\x0b\\x0c\\x0e-\\x1f]\")\n\n\ndef _strip_xml_illegal(s: str) -> str:\n \"\"\"Drop characters XML 1.0 cannot represent, leaving tab/LF/CR intact.\n\n ``nx.write_graphml`` raises ``ValueError(\"All strings must be XML\n compatible: Unicode or ASCII, no NULL bytes or control characters\")`` on any\n of them and aborts the whole export over a single label. Labels arrive\n unfiltered from the corpus, so this is ordinary content rather than hostile\n input: an ANSI escape in a markdown heading pasted from a terminal capture,\n or the form feed some Python/Emacs sources use as a section separator\n (#2897).\n \"\"\"\n return _XML_ILLEGAL_RE.sub(\"\", s)\n\n\n# C0 controls and DEL, folded to a space when building a filename stem. Windows\n# rejects them in a path outright with OSError EINVAL, so one of them in a label\n# aborted a whole Obsidian vault export; POSIX would accept the name but leave a\n# note nothing can comfortably open (#2897).\n_CONTROL_TO_SPACE_RE = re.compile(\"[\\x00-\\x1f\\x7f]\")\n\n\ndef _cap_filename(s: str, limit: int = 200) -> str:\n \"\"\"Cap a filename stem to ``limit`` UTF-8 bytes so it stays under the 255-byte\n filesystem limit even after the ``.md`` extension and dedup suffix are added\n (#1094). The cap is on BYTES, not chars, because a label of multibyte\n characters (CJK, accented) can exceed 255 bytes well under 255 chars. When\n truncation happens, an 8-char hash of the full label is appended so two\n distinct labels sharing a long prefix produce distinct, deterministic\n filenames instead of colliding.\"\"\"\n b = s.encode(\"utf-8\")\n if len(b) \u003c= limit:\n return s\n digest = hashlib.sha1(s.encode(\"utf-8\")).hexdigest()[:8] # nosec - not security\n keep = limit - 9 # \"_\" + 8 hex chars\n truncated = b[:keep].decode(\"utf-8\", \"ignore\") # \"ignore\" drops a split trailing char\n return f\"{truncated}_{digest}\"\n\n\n# A frontmatter tag entry in graphify's own namespace, e.g. \" - graphify/document\".\n_GRAPHIFY_TAG_RE = re.compile(r\"^\\s*-\\s+graphify/\\S\")\n\n# Frontmatter sits at the very top of a note; reading this much is enough to see\n# the whole block without pulling a large note into memory.\n_NOTE_FRONTMATTER_PROBE_BYTES = 4096\n\n# Community notes carry no frontmatter; graphify identifies its own by the\n# Dataview query it writes into every one of them.\n_COMMUNITY_QUERY_MARKER = \"FROM #community/\"\n\n\ndef _is_graphify_note(path: Path) -> bool:\n \"\"\"Whether a vault note carries graphify's own frontmatter signature.\n\n Every note graphify writes opens with a YAML frontmatter block tagging it in\n the ``graphify/`` namespace::\n\n ---\n source_file: \"d0.md\"\n tags:\n - graphify/document\n - graphify/EXTRACTED\n ---\n\n Only that block is inspected, and only a tag entry inside it counts — a\n user's note that merely mentions graphify in its prose is not adopted.\n\n Community overview notes are recognised separately: they carry no\n frontmatter at all, so they are identified by graphify's own filename prefix\n together with the Dataview query it writes into the body. Requiring both\n keeps a user's own ``_COMMUNITY_*.md`` from being adopted on the name alone.\n \"\"\"\n try:\n with path.open(\"r\", encoding=\"utf-8\", errors=\"replace\") as fh:\n head = fh.read(_NOTE_FRONTMATTER_PROBE_BYTES)\n except OSError:\n return False\n if path.name.startswith(_COMMUNITY_PREFIX) and _COMMUNITY_QUERY_MARKER in head:\n return True\n if not head.startswith(\"---\"):\n return False\n for line in head.splitlines()[1:]:\n if line.strip() == \"---\":\n return False # frontmatter closed without a graphify tag\n if _GRAPHIFY_TAG_RE.match(line):\n return True\n return False\n\n\ndef _adopt_pre_manifest_notes(out: Path) -> set[str]:\n \"\"\"Names of notes in *out* that graphify itself wrote before manifests existed.\n\n Deliberately limited to top-level ``*.md``: those are the only files graphify\n can identify as its own from their content. ``.obsidian/graph.json`` is NOT\n adopted — graphify writes one, but so does Obsidian, and with no manifest\n there is no way to tell whose it is. Leaving it unowned keeps the\n conservative behaviour for the one file where guessing wrong would cost the\n user their own vault configuration.\n \"\"\"\n try:\n candidates = sorted(out.glob(\"*.md\"))\n except OSError:\n return set()\n return {p.name for p in candidates if _is_graphify_note(p)}\n\n\ndef _obsidian_safe_stem(label: str, limit: int = 200) -> str:\n \"\"\"Filename stem for an Obsidian note / canvas card from a node label.\n\n Strips filesystem-unsafe characters, a trailing ``.md``-family extension\n (so ``CLAUDE.md`` does not become ``CLAUDE.md.md``), and a leading ``.`` —\n Obsidian hides every note whose name starts with a dot, so ``.env.md``\n would be written but invisible in the UI (#2205). The ``dot-`` prefix keeps\n the name recognizable; H1 / frontmatter still carry the true label.\n \"\"\"\n cleaned = re.sub(\n r'[\\\\/*?:\"\u003c>|#^[\\]]',\n \"\",\n # CR/LF were already folded to spaces here; every other C0 control now\n # goes the same way. They are not merely awkward in a filename — Windows\n # rejects them outright, so a single one aborted the whole vault export\n # rather than spoiling one note (#2897).\n _CONTROL_TO_SPACE_RE.sub(\" \", label),\n ).strip()\n cleaned = re.sub(r\"\\.(md|mdx|qmd|markdown)$\", \"\", cleaned, flags=re.IGNORECASE)\n # Obsidian treats a leading-dot filename as a hidden file (#2205). Only\n # prefix when something nameable remains after the dots: an all-dots label\n # like \"...\" would otherwise become the meaningless stem \"dot-\" instead of\n # falling through to the \"unnamed\" guard below (#1409).\n if cleaned.startswith(\".\") and re.search(r\"\\w\", cleaned.lstrip(\".\"), flags=re.UNICODE):\n cleaned = \"dot-\" + cleaned.lstrip(\".\")\n # A stem of only punctuation (e.g. \"@\", \"*\", \"#\") survives the unsafe-char\n # strip above but is empty once a downstream tool re-slugs on word chars\n # (e.g. qmd's handelize() reduces \"@\" -> \"\" and raises, aborting the whole\n # `qmd update`). Require at least one word char; else fall back so we never\n # emit a \"@.md\"-style filename. (#1409)\n if not re.search(r\"\\w\", cleaned, flags=re.UNICODE):\n return \"unnamed\"\n return _cap_filename(cleaned, limit)\n\n\n# Room _dedup_node_filenames / the community loop need for a collision suffix\n# (\"_1\" … \"_9999\") appended AFTER the stem was capped. The suffix is technically\n# unbounded, but 5 chars (\"_\" + 4 digits) covers ~10k identical stems, far past\n# anything real; sizing it to 3 digits let a 1000th collision overrun MAX_PATH.\n_DEDUP_SUFFIX_RESERVE = 5\n\n# Prefix the community overview notes carry (\"_COMMUNITY_Backend.md\").\n_COMMUNITY_PREFIX = \"_COMMUNITY_\"\n\n\ndef _dedup_node_filenames(G: nx.Graph, safe_name) -> dict[str, str]:\n \"\"\"Map each node_id to a unique note filename, appending a numeric suffix on\n collision. The collision set is keyed on the lowercased name so two labels\n differing only by case (e.g. \"References\" vs \"references\") still get distinct\n filenames - on case-insensitive filesystems (macOS/APFS, Windows/NTFS) they\n would otherwise resolve to one path and silently overwrite each other on disk.\n The suffixed candidate is itself re-checked, so a generated \"base_1\" never\n silently overwrites a node whose literal label is already \"base_1\".\"\"\"\n node_filenames: dict[str, str] = {}\n used: set[str] = set()\n for node_id, data in G.nodes(data=True):\n base = safe_name(data.get(\"label\", node_id))\n candidate = base\n n = 1\n while candidate.lower() in used:\n candidate = f\"{base}_{n}\"\n n += 1\n used.add(candidate.lower())\n node_filenames[node_id] = candidate\n return node_filenames\n\n\ndef to_obsidian(\n G: nx.Graph,\n communities: dict[int, list[str]],\n output_dir: str,\n community_labels: dict[int, str] | None = None,\n cohesion: dict[int, float] | None = None,\n) -> int:\n \"\"\"Export graph as an Obsidian vault - one .md file per node with [[wikilinks]],\n plus one _COMMUNITY_name.md overview note per community (sorted to top by underscore prefix).\n\n Open the output directory as a vault in Obsidian to get an interactive\n graph view with community colors and full-text search over node metadata.\n\n Returns the number of node notes + community notes written.\n \"\"\"\n out = Path(output_dir)\n out.mkdir(parents=True, exist_ok=True)\n\n # #1506: when the export target is an existing Obsidian vault (a user pointed\n # --obsidian-dir at one), we must not clobber the user's own notes or their\n # .obsidian/ config. Track the files graphify owns in a manifest; a pre-existing\n # file NOT in the manifest is the user's and is never overwritten.\n _manifest_path = out / \".graphify_obsidian_manifest.json\"\n try:\n _owned: set[str] = set(json.loads(_manifest_path.read_text(encoding=\"utf-8\")).get(\"files\", []))\n _manifest_existed = True\n except (OSError, ValueError):\n _owned = set()\n _manifest_existed = False\n if not _manifest_existed:\n # A vault written before the manifest existed has no record of what\n # graphify owns, so every note it wrote last time reads as the user's and\n # is skipped. The re-export then writes fresh notes BESIDE the stale ones\n # and the vault carries two generations, with a warning claiming graphify\n # \"did not create\" files it did (#2863). Adopt the notes that carry\n # graphify's own frontmatter, once, so the manifest starts out honest.\n _owned |= _adopt_pre_manifest_notes(out)\n _written: list[str] = []\n _skipped: list[str] = []\n\n def _owned_write(rel_name: str, content: str) -> bool:\n \"\"\"Write a graphify-owned file, refusing to overwrite a pre-existing file\n graphify didn't create. Returns True if the note is owned/current (whether\n or not it was physically written).\n\n The disk write is skipped when the content is byte-identical to what is\n already there (#3060) — an export re-runs on every graph.json change, and\n rewriting an unchanged note churns disk and fires inotify / re-index /\n sync for nothing. The note is still recorded in ``_written`` so it stays\n owned and is neither pruned as stale nor dropped from the manifest.\"\"\"\n target = out / rel_name\n if target.exists() and rel_name not in _owned:\n _skipped.append(rel_name)\n return False\n target.parent.mkdir(parents=True, exist_ok=True)\n write_text_atomic_if_changed(target, content)\n _written.append(rel_name)\n return True\n\n node_community = _node_community_map(communities)\n\n # Cap stems against THIS vault's path, not just NAME_MAX: on Windows the\n # 200-byte default plus an ordinary vault directory overruns MAX_PATH and\n # every note write raises FileNotFoundError (#2655). No-op on POSIX.\n _stem_limit = stem_filename_budget(out, reserve=_DEDUP_SUFFIX_RESERVE)\n\n # Map node_id → safe filename so wikilinks stay consistent.\n # Deduplicate: if two nodes produce the same filename, append a numeric suffix.\n node_filename = _dedup_node_filenames(\n G, lambda label: _obsidian_safe_stem(label, _stem_limit)\n )\n\n # Helper: compute dominant confidence for a node across all its edges\n def _dominant_confidence(node_id: str) -> str:\n confs = []\n for u, v, edata in G.edges(node_id, data=True):\n confs.append(edata.get(\"confidence\", \"EXTRACTED\"))\n if not confs:\n return \"EXTRACTED\"\n return Counter(confs).most_common(1)[0][0]\n\n # Map file_type → graphify tag\n _FTYPE_TAG = {\n \"code\": \"graphify/code\",\n \"document\": \"graphify/document\",\n \"paper\": \"graphify/paper\",\n \"image\": \"graphify/image\",\n }\n\n # Write one .md file per node\n node_notes_written = 0\n for node_id, data in G.nodes(data=True):\n label = data.get(\"label\", node_id)\n cid = node_community.get(node_id)\n community_name = (\n community_labels.get(cid, f\"Community {cid}\")\n if community_labels and cid is not None\n else f\"Community {cid}\"\n )\n\n # Build tags for this node\n ftype = data.get(\"file_type\", \"\")\n ftype_tag = _FTYPE_TAG.get(ftype, f\"graphify/{_obsidian_tag(ftype)}\" if ftype else \"graphify/document\")\n dom_conf = _dominant_confidence(node_id)\n conf_tag = f\"graphify/{_obsidian_tag(dom_conf)}\"\n comm_tag = f\"community/{_obsidian_tag(community_name)}\"\n node_tags = [ftype_tag, conf_tag, comm_tag]\n\n lines: list[str] = []\n\n # YAML frontmatter - readable in Obsidian's properties panel.\n # All scalars pass through _yaml_str so a hostile source_file or\n # community label cannot break out and inject sibling keys (F-009).\n lines += [\n \"---\",\n f'source_file: \"{_yaml_str(data.get(\"source_file\", \"\"))}\"',\n f'type: \"{_yaml_str(ftype)}\"',\n f'community: \"{_yaml_str(community_name)}\"',\n ]\n if data.get(\"source_location\"):\n lines.append(f'location: \"{_yaml_str(str(data[\"source_location\"]))}\"')\n # Add tags list to frontmatter\n lines.append(\"tags:\")\n for tag in node_tags:\n lines.append(f\" - {tag}\")\n lines += [\"---\", \"\", f\"# {label}\", \"\"]\n\n # Outgoing edges as wikilinks\n neighbors = list(G.neighbors(node_id))\n if neighbors:\n lines.append(\"## Connections\")\n for neighbor in sorted(neighbors, key=lambda n: G.nodes[n].get(\"label\", n)):\n neighbor_label = node_filename[neighbor]\n # One bullet per parallel edge, not per neighbour: graph.json is\n # reloaded as a MultiGraph when it carries parallel edges\n # (#4500), and `edge_data()` returns only the FIRST of them — so\n # a pair joined by both `imports` and `calls` lost one relation\n # from the note. Sorted so the note is byte-stable regardless of\n # the order the parallel edges were inserted in.\n for edata in sorted(\n edge_datas(G, node_id, neighbor),\n key=lambda d: (str(d.get(\"relation\", \"\")), str(d.get(\"confidence\", \"EXTRACTED\"))),\n ):\n relation = edata.get(\"relation\", \"\")\n confidence = edata.get(\"confidence\", \"EXTRACTED\")\n lines.append(f\"- [[{neighbor_label}]] - `{relation}` [{confidence}]\")\n lines.append(\"\")\n\n # Inline tags at bottom of note body (for Obsidian tag panel)\n inline_tags = \" \".join(f\"#{t}\" for t in node_tags)\n lines.append(inline_tags)\n\n fname = node_filename[node_id] + \".md\"\n if _owned_write(fname, \"\\n\".join(lines)):\n node_notes_written += 1\n\n # Write one _COMMUNITY_name.md overview note per community\n # Build inter-community edge counts for \"Connections to other communities\"\n inter_community_edges: dict[int, dict[int, int]] = {}\n for cid in communities:\n inter_community_edges[cid] = {}\n for u, v in G.edges():\n cu = node_community.get(u)\n cv = node_community.get(v)\n if cu is not None and cv is not None and cu != cv:\n inter_community_edges.setdefault(cu, {})\n inter_community_edges.setdefault(cv, {})\n inter_community_edges[cu][cv] = inter_community_edges[cu].get(cv, 0) + 1\n inter_community_edges[cv][cu] = inter_community_edges[cv].get(cu, 0) + 1\n\n # Precompute per-node community reach (number of distinct communities a node connects to)\n def _community_reach(node_id: str) -> int:\n neighbor_cids = {\n node_community[nb]\n for nb in G.neighbors(node_id)\n if nb in node_community and node_community[nb] != node_community.get(node_id)\n }\n return len(neighbor_cids)\n\n def _community_name(cid) -> str:\n return (\n community_labels.get(cid, f\"Community {cid}\")\n if community_labels and cid is not None\n else f\"Community {cid}\"\n )\n\n # One case-folded-deduped filename per community, computed once so the note we\n # write and every [[_COMMUNITY_...]] cross-reference resolve to the same file.\n # Two community labels differing only by case (e.g. LLM labels \"API\" vs \"Api\")\n # would otherwise overwrite each other on case-insensitive filesystems - and\n # this path had no dedup at all, so even same-case duplicate labels collided.\n community_filename: dict = {}\n used_community: set[str] = set()\n # The community stem carries the \"_COMMUNITY_\" prefix on top of the dedup\n # suffix, so it gets that much less of the MAX_PATH window (#2655).\n _community_stem_limit = stem_filename_budget(\n out, reserve=_DEDUP_SUFFIX_RESERVE + len(_COMMUNITY_PREFIX)\n )\n for cid in communities:\n base = f\"{_COMMUNITY_PREFIX}{_obsidian_safe_stem(_community_name(cid), _community_stem_limit)}\"\n candidate = base\n n = 1\n while candidate.lower() in used_community:\n candidate = f\"{base}_{n}\"\n n += 1\n used_community.add(candidate.lower())\n community_filename[cid] = candidate\n\n community_notes_written = 0\n for cid, all_members in communities.items():\n community_name = _community_name(cid)\n # A community's member list can contain ids with no backing node in G\n # (e.g. pruned nodes, stale community assignments from a prior run, or\n # synthesized/merge-artifact ids). Dereferencing those via G.nodes[n] or\n # node_filename[n] raises KeyError and aborts the whole vault export, so\n # skip dangling members rather than crashing (issue #1236).\n members = [m for m in all_members if m in G and m in node_filename]\n n_members = len(members)\n coh_value = cohesion.get(cid) if cohesion else None\n\n lines: list[str] = []\n\n # YAML frontmatter\n lines.append(\"---\")\n lines.append(\"type: community\")\n if coh_value is not None:\n lines.append(f\"cohesion: {coh_value:.2f}\")\n lines.append(f\"members: {n_members}\")\n lines.append(\"---\")\n lines.append(\"\")\n lines.append(f\"# {community_name}\")\n lines.append(\"\")\n\n # Cohesion + member count summary\n if coh_value is not None:\n cohesion_desc = (\n \"tightly connected\" if coh_value >= 0.7\n else \"moderately connected\" if coh_value >= 0.4\n else \"loosely connected\"\n )\n lines.append(f\"**Cohesion:** {coh_value:.2f} - {cohesion_desc}\")\n lines.append(f\"**Members:** {n_members} nodes\")\n lines.append(\"\")\n\n # Members section\n lines.append(\"## Members\")\n for node_id in sorted(members, key=lambda n: G.nodes[n].get(\"label\", n)):\n data = G.nodes[node_id]\n node_label = node_filename[node_id]\n ftype = data.get(\"file_type\", \"\")\n source = data.get(\"source_file\", \"\")\n entry = f\"- [[{node_label}]]\"\n if ftype:\n entry += f\" - {ftype}\"\n if source:\n entry += f\" - {source}\"\n lines.append(entry)\n lines.append(\"\")\n\n # Dataview live query (improvement 2)\n comm_tag_name = _obsidian_tag(community_name)\n lines.append(\"## Live Query (requires Dataview plugin)\")\n lines.append(\"\")\n lines.append(\"```dataview\")\n lines.append(f\"TABLE source_file, type FROM #community/{comm_tag_name}\")\n lines.append(\"SORT file.name ASC\")\n lines.append(\"```\")\n lines.append(\"\")\n\n # Connections to other communities\n cross = inter_community_edges.get(cid, {})\n if cross:\n lines.append(\"## Connections to other communities\")\n for other_cid, edge_count in sorted(cross.items(), key=lambda x: -x[1]):\n other_fname = community_filename.get(other_cid) or (\n f\"{_COMMUNITY_PREFIX}\"\n f\"{_obsidian_safe_stem(_community_name(other_cid), _community_stem_limit)}\"\n )\n lines.append(f\"- {edge_count} edge{'s' if edge_count != 1 else ''} to [[{other_fname}]]\")\n lines.append(\"\")\n\n # Top bridge nodes - highest degree nodes that connect to other communities\n bridge_nodes = [\n (node_id, G.degree(node_id), _community_reach(node_id))\n for node_id in members\n if _community_reach(node_id) > 0\n ]\n bridge_nodes.sort(key=lambda x: (-x[2], -x[1]))\n top_bridges = bridge_nodes[:5]\n if top_bridges:\n lines.append(\"## Top bridge nodes\")\n for node_id, degree, reach in top_bridges:\n node_label = node_filename[node_id]\n lines.append(\n f\"- [[{node_label}]] - degree {degree}, connects to {reach} \"\n f\"{'community' if reach == 1 else 'communities'}\"\n )\n\n fname = community_filename[cid] + \".md\"\n if _owned_write(fname, \"\\n\".join(lines)):\n community_notes_written += 1\n\n # Improvement 4: write .obsidian/graph.json to color nodes by community in graph\n # view — but never clobber an existing .obsidian/graph.json graphify doesn't own\n # (the user's graph-view settings live there). _owned_write handles that and\n # creates the .obsidian/ dir only when it actually writes.\n graph_config = {\n \"colorGroups\": [\n {\n # Same sanitizer as the note tags (#2862): built from the raw\n # label, the canvas colour group queried a tag that no note\n # carries whenever the label held non-ASCII or punctuation.\n \"query\": f\"tag:#community/{_obsidian_tag(label)}\",\n \"color\": {\"a\": 1, \"rgb\": int(COMMUNITY_COLORS[cid % len(COMMUNITY_COLORS)].lstrip('#'), 16)}\n }\n for cid, label in sorted((community_labels or {}).items())\n ]\n }\n _owned_write(\".obsidian/graph.json\", json.dumps(graph_config, indent=2))\n\n # #1896: prune notes for nodes that dropped out of the graph. Only files the\n # manifest says graphify owns are candidates, and anything written or skipped\n # this run is excluded — so a user's own note is never touched (foreign files\n # land in _skipped, never _owned). Guard each path to stay inside the vault in\n # case a corrupt/hostile manifest contains `../` entries.\n stale = _owned - set(_written) - set(_skipped)\n pruned = 0\n for rel_name in sorted(stale):\n target = (out / rel_name).resolve()\n if out.resolve() not in target.parents:\n continue\n try:\n target.unlink(missing_ok=True)\n pruned += 1\n except OSError:\n pass\n if pruned:\n print(\n f\"[graphify] pruned {pruned} note(s) for nodes no longer in the graph\",\n file=sys.stderr,\n )\n\n # Persist the manifest of files graphify owns, so a re-run can safely update its\n # own notes while still refusing to touch the user's. Warn (once, aggregated)\n # about anything skipped to avoid clobbering a pre-existing file.\n try:\n _manifest_path.write_text(json.dumps({\"files\": sorted(set(_written))}, indent=2), encoding=\"utf-8\")\n except OSError:\n pass\n if _skipped:\n shown = \", \".join(_skipped[:5]) + (f\" (+{len(_skipped) - 5} more)\" if len(_skipped) > 5 else \"\")\n print(\n f\"[graphify] WARNING: skipped {len(_skipped)} pre-existing file(s) graphify \"\n f\"did not create, to avoid overwriting your notes: {shown}. \"\n f\"Export into an empty directory (or the default graphify-out/obsidian) \"\n f\"to get the full vault.\",\n file=sys.stderr,\n )\n\n return node_notes_written + community_notes_written\n\n\ndef to_canvas(\n G: nx.Graph,\n communities: dict[int, list[str]],\n output_path: str,\n community_labels: dict[int, str] | None = None,\n node_filenames: dict[str, str] | None = None,\n) -> None:\n \"\"\"Export graph as an Obsidian Canvas file - communities as groups, nodes as cards.\n\n Generates a structured layout: communities arranged in a grid, nodes within\n each community arranged in rows. Edges shown between connected nodes.\n Opens in Obsidian as an infinite canvas with community groupings visible.\n \"\"\"\n # Obsidian canvas color codes (cycle through for communities)\n CANVAS_COLORS = [\"1\", \"2\", \"3\", \"4\", \"5\", \"6\"] # red, orange, yellow, green, cyan, purple\n\n # Build node_filenames if not provided (same dedup logic as to_obsidian).\n # The CLI calls to_canvas without passing the map, so it must derive the\n # SAME stem budget to keep card links pointing at the notes to_obsidian\n # wrote — hence budgeting against the canvas's own directory, which is the\n # vault directory (#2655).\n _stem_limit = stem_filename_budget(\n Path(output_path).parent, reserve=_DEDUP_SUFFIX_RESERVE\n )\n if node_filenames is None:\n node_filenames = _dedup_node_filenames(\n G, lambda label: _obsidian_safe_stem(label, _stem_limit)\n )\n\n # Fallback: with no community data (e.g. --no-cluster builds or a missing\n # analysis sidecar) the grid below produces nothing and the canvas is written\n # as an empty 32-byte shell on an otherwise populated graph. Emit every node\n # into one synthetic community so the canvas always reflects the graph (#1324).\n if not communities and G.number_of_nodes() > 0:\n communities = {0: [str(n) for n in G.nodes()]}\n\n num_communities = len(communities)\n cols = math.ceil(math.sqrt(num_communities)) if num_communities > 0 else 1\n rows = math.ceil(num_communities / cols) if num_communities > 0 else 1\n\n canvas_nodes: list[dict] = []\n canvas_edges: list[dict] = []\n\n # Lay out communities in a grid\n gap = 80\n group_x_offsets: list[int] = []\n group_y_offsets: list[int] = []\n\n # Precompute group sizes so we can calculate offsets.\n # inner_cols is the per-community grid width; the box dimensions AND the node\n # placement loop below both derive from it, so the cards always fill the box\n # instead of wrapping into a narrow strip inside an oversized box.\n sorted_cids = sorted(communities.keys())\n group_sizes: dict[int, tuple[int, int]] = {}\n group_cols: dict[int, int] = {}\n for cid in sorted_cids:\n # Skip dangling community members with no backing node / filename, so box\n # sizing matches the cards actually laid out and `G.nodes[m]` never\n # KeyErrors below — mirrors the to_obsidian guard (#1236).\n members = [m for m in communities[cid] if m in G and m in node_filenames]\n n = len(members)\n inner_cols = max(1, math.ceil(math.sqrt(n)))\n w = max(600, 220 * inner_cols)\n h = max(400, 100 * math.ceil(n / inner_cols) + 120)\n group_sizes[cid] = (w, h)\n group_cols[cid] = inner_cols\n\n # Compute cumulative row heights and col widths for grid placement\n # Each grid cell uses the max width/height in its col/row\n col_widths: list[int] = []\n row_heights: list[int] = []\n for col_idx in range(cols):\n max_w = 0\n for row_idx in range(rows):\n linear = row_idx * cols + col_idx\n if linear \u003c len(sorted_cids):\n cid = sorted_cids[linear]\n w, _ = group_sizes[cid]\n max_w = max(max_w, w)\n col_widths.append(max_w)\n\n for row_idx in range(rows):\n max_h = 0\n for col_idx in range(cols):\n linear = row_idx * cols + col_idx\n if linear \u003c len(sorted_cids):\n cid = sorted_cids[linear]\n _, h = group_sizes[cid]\n max_h = max(max_h, h)\n row_heights.append(max_h)\n\n # Map from cid → (group_x, group_y, group_w, group_h)\n group_layout: dict[int, tuple[int, int, int, int]] = {}\n for idx, cid in enumerate(sorted_cids):\n col_idx = idx % cols\n row_idx = idx // cols\n gx = sum(col_widths[:col_idx]) + col_idx * gap\n gy = sum(row_heights[:row_idx]) + row_idx * gap\n gw, gh = group_sizes[cid]\n group_layout[cid] = (gx, gy, gw, gh)\n\n # Build set of all node_ids in canvas for edge filtering\n all_canvas_nodes: set[str] = set()\n for members in communities.values():\n all_canvas_nodes.update(members)\n\n # Generate group and node canvas entries\n for idx, cid in enumerate(sorted_cids):\n members = communities[cid]\n community_name = (\n community_labels.get(cid, f\"Community {cid}\")\n if community_labels and cid is not None\n else f\"Community {cid}\"\n )\n gx, gy, gw, gh = group_layout[cid]\n canvas_color = CANVAS_COLORS[idx % len(CANVAS_COLORS)]\n\n # Group node\n canvas_nodes.append({\n \"id\": f\"g{cid}\",\n \"type\": \"group\",\n \"label\": community_name,\n \"x\": gx,\n \"y\": gy,\n \"width\": gw,\n \"height\": gh,\n \"color\": canvas_color,\n })\n\n # Node cards inside the group - laid out in the same ceil(sqrt(n))-column\n # grid the box was sized for (group_cols[cid]), so cards fill the box.\n inner_cols = group_cols[cid]\n # Same dangling-member guard as the sizing loop and to_obsidian (#1236):\n # a community id absent from G / node_filenames would KeyError the sort.\n members = [m for m in members if m in G and m in node_filenames]\n sorted_members = sorted(members, key=lambda n: G.nodes[n].get(\"label\", n))\n for m_idx, node_id in enumerate(sorted_members):\n col = m_idx % inner_cols\n row = m_idx // inner_cols\n nx_x = gx + 20 + col * (180 + 20)\n nx_y = gy + 80 + row * (60 + 20)\n fname = node_filenames.get(\n node_id,\n _obsidian_safe_stem(G.nodes[node_id].get(\"label\", node_id), _stem_limit),\n )\n canvas_nodes.append({\n \"id\": f\"n_{node_id}\",\n \"type\": \"file\",\n \"file\": f\"{fname}.md\",\n \"x\": nx_x,\n \"y\": nx_y,\n \"width\": 180,\n \"height\": 60,\n })\n\n # Generate edges - only between nodes both in canvas, cap at 200 highest-weight\n all_edges_weighted: list[tuple[float, str, str, str]] = []\n for u, v, edata in G.edges(data=True):\n u, v = edata.get(\"_src\", u), edata.get(\"_tgt\", v)\n if u in all_canvas_nodes and v in all_canvas_nodes:\n weight = edata.get(\"weight\", 1.0)\n relation = edata.get(\"relation\", \"\")\n conf = edata.get(\"confidence\", \"EXTRACTED\")\n label = f\"{relation} [{conf}]\" if relation else f\"[{conf}]\"\n all_edges_weighted.append((weight, u, v, label))\n\n all_edges_weighted.sort(key=lambda x: -x[0])\n for weight, u, v, label in all_edges_weighted[:200]:\n canvas_edges.append({\n \"id\": f\"e_{u}_{v}\",\n \"fromNode\": f\"n_{u}\",\n \"toNode\": f\"n_{v}\",\n \"label\": label,\n })\n\n canvas_data = {\"nodes\": canvas_nodes, \"edges\": canvas_edges}\n write_json_atomic(output_path, canvas_data, indent=2)\n\n\ndef to_graphml(\n G: nx.Graph,\n communities: dict[int, list[str]],\n output_path: str,\n) -> None:\n \"\"\"Export graph as GraphML - opens in Gephi, yEd, and any GraphML-compatible tool.\n\n Community IDs are written as a node attribute so Gephi can colour by community.\n Edge confidence (EXTRACTED/INFERRED/AMBIGUOUS) is preserved as an edge attribute.\n \"\"\"\n H = G.copy()\n node_community = _node_community_map(communities)\n for node_id in H.nodes():\n H.nodes[node_id][\"community\"] = node_community.get(node_id, -1)\n # Drop internal markers (e.g. the AST-provenance \"_origin\" tag, #1116, and\n # the \"_src\"/\"_tgt\" direction markers) — they are persistence/runtime details,\n # not graph data, and should not leak into the exported file.\n for _, attrs in H.nodes(data=True):\n for k in [k for k in attrs if k.startswith(\"_\")]:\n del attrs[k]\n for _, _, attrs in H.edges(data=True):\n for k in [k for k in attrs if k.startswith(\"_\")]:\n del attrs[k]\n # nx.write_graphml only accepts scalar attribute values: None raises, and a\n # dict/list value (e.g. a per-node `metadata` dict, or the graph-level\n # `hyperedges` list set by attach_hyperedges()) raises\n # \"GraphML does not support type \u003cclass 'dict'/'list'> as data values\" (#1831).\n # Coerce None -> \"\" and non-scalars -> a JSON string, across all three scopes.\n def _graphml_safe(val):\n if val is None:\n return \"\"\n if isinstance(val, bool) or isinstance(val, (int, float)):\n return val # GraphML-native scalars pass through unchanged\n if isinstance(val, str):\n # Scalar, but still has to be XML-representable — see\n # _strip_xml_illegal. This is the line that turns \"one label carried\n # an ANSI escape\" from a lost export into a lost escape character.\n return _strip_xml_illegal(val)\n try:\n return _strip_xml_illegal(json.dumps(val, default=str, sort_keys=True))\n except (TypeError, ValueError):\n return _strip_xml_illegal(str(val))\n\n # Node IDs become the `id` attribute of every \u003cnode> and edge endpoint, so\n # they must be XML-representable too. Normalised ids never carry a control\n # character, but a caller can hand us a hand-built graph, and a crash here\n # loses the export just as completely as one in the values.\n _id_remap = {n: _strip_xml_illegal(n) for n in H.nodes if isinstance(n, str)}\n _id_remap = {k: v for k, v in _id_remap.items() if k != v}\n if _id_remap:\n H = nx.relabel_nodes(H, _id_remap, copy=True)\n\n for key, val in list(H.graph.items()):\n H.graph[key] = _graphml_safe(val)\n for node_id in H.nodes():\n for key, val in list(H.nodes[node_id].items()):\n H.nodes[node_id][key] = _graphml_safe(val)\n # edges(data=True) rather than H.edges[u, v]: graph.json is loaded as a\n # MultiGraph when it carries parallel edges (#4500), and a MultiGraph edge\n # view needs a key, so the (u, v) form crashed the whole export.\n for *_, attrs in H.edges(data=True):\n for key, val in list(attrs.items()):\n attrs[key] = _graphml_safe(val)\n\n # Write atomically: a mid-serialization error otherwise leaves a 0-byte\n # .graphml on disk that downstream tooling mistakes for a completed export\n # (#1831). Write to a sibling temp file, then replace on success.\n out = Path(output_path)\n tmp = out.with_name(out.name + \".tmp\")\n try:\n nx.write_graphml(H, str(tmp))\n os_replace_with_fallback(str(tmp), str(out))\n finally:\n if tmp.exists():\n try:\n tmp.unlink()\n except OSError:\n pass\n\n\ndef to_svg(\n G: nx.Graph,\n communities: dict[int, list[str]],\n output_path: str,\n community_labels: dict[int, str] | None = None,\n figsize: tuple[int, int] = (20, 14),\n) -> None:\n \"\"\"Export graph as an SVG file using matplotlib + spring layout.\n\n Lightweight and embeddable - works in Obsidian notes, Notion, GitHub READMEs,\n and any markdown renderer. No JavaScript required.\n\n Node size scales with degree. Community colors match the HTML output.\n \"\"\"\n try:\n import matplotlib\n matplotlib.use(\"Agg\")\n import matplotlib.pyplot as plt\n import matplotlib.patches as mpatches\n except ImportError as e:\n raise ImportError(\"matplotlib not installed. Run: pip install matplotlib\") from e\n\n node_community = _node_community_map(communities)\n\n fig, ax = plt.subplots(figsize=figsize, facecolor=\"#1a1a2e\")\n ax.set_facecolor(\"#1a1a2e\")\n ax.axis(\"off\")\n\n pos = nx.spring_layout(G, seed=42, k=2.0 / (G.number_of_nodes() ** 0.5 + 1))\n\n degree = dict(G.degree())\n max_deg = max(degree.values(), default=1) or 1\n\n node_colors = [COMMUNITY_COLORS[node_community.get(n, 0) % len(COMMUNITY_COLORS)] for n in G.nodes()]\n node_sizes = [300 + 1200 * (degree.get(n, 1) / max_deg) for n in G.nodes()]\n\n # Draw edges - dashed for non-EXTRACTED\n for u, v, data in G.edges(data=True):\n conf = data.get(\"confidence\", \"EXTRACTED\")\n style = \"solid\" if conf == \"EXTRACTED\" else \"dashed\"\n alpha = 0.6 if conf == \"EXTRACTED\" else 0.3\n x0, y0 = pos[u]\n x1, y1 = pos[v]\n ax.plot([x0, x1], [y0, y1], color=\"#aaaaaa\", linewidth=0.8,\n linestyle=style, alpha=alpha, zorder=1)\n\n nx.draw_networkx_nodes(G, pos, ax=ax, node_color=node_colors,\n node_size=node_sizes, alpha=0.9)\n nx.draw_networkx_labels(G, pos, ax=ax,\n labels={n: _strip_xml_illegal(str(G.nodes[n].get(\"label\", n)))\n for n in G.nodes()},\n font_size=7, font_color=\"white\")\n\n # Legend\n if community_labels:\n patches = [\n mpatches.Patch(\n color=COMMUNITY_COLORS[cid % len(COMMUNITY_COLORS)],\n label=_strip_xml_illegal(f\"{label} ({len(communities.get(cid, []))})\"),\n )\n for cid, label in sorted(community_labels.items())\n ]\n ax.legend(handles=patches, loc=\"upper left\", framealpha=0.7,\n facecolor=\"#2a2a4e\", labelcolor=\"white\", fontsize=8)\n\n plt.tight_layout()\n plt.savefig(output_path, format=\"svg\", bbox_inches=\"tight\",\n facecolor=fig.get_facecolor())\n plt.close(fig)\n","content_type":"text/x-python; charset=utf-8","language":"python","size":65765,"content_sha256":"942b901eeb269c5fd5abc8766a3b136a5941784bc4b069db71df435d09110529"},{"filename":"exporters/__init__.py","content":"\"\"\"exporters package.\"\"\"\n","content_type":"text/x-python; charset=utf-8","language":"python","size":25,"content_sha256":"0a1e96c29d7601843980a62a1069a95c05cb9f07d6f431a054feb8664abf21e1"},{"filename":"file_slice.py","content":"\"\"\"Intra-file slicing for oversized text documents (#1369).\n\nThe extraction packer (`_pack_chunks_by_tokens`) treats each file as atomic and\n`_read_files` caps every file at ``_FILE_CHAR_CAP`` characters, so a document\nlarger than that cap had everything past the cap silently dropped — the model\nnever saw it, and nothing in the adaptive-retry path could recover it (\"a single\nfile larger than the budget ... packing can't shrink one big file\").\n\nThis module splits an oversized *splittable text* document (Markdown, plain\ntext, reStructuredText) into contiguous ``FileSlice`` units at heading /\nparagraph / line boundaries so the whole file gets extracted across several\nunits. Every slice of a file reports the **parent file path** as its source, so\nthe resulting nodes are never fragmented per-slice — they merge by source_file\nexactly as if the file had been extracted in one pass.\n\nOnly plain-text documents are sliced: code files need whole-symbol context, and\nPDFs/images are read through their own extractors and have no char-offset model.\n\"\"\"\n\nfrom __future__ import annotations\n\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom graphify.extractors.base import _read_source_text\n\n# Plain-text document types where boundary-based slicing is meaningful and where\n# `_file_to_text` reads them with ``_read_source_text``, the same reader as\n# `unit_source_text` (so a char range matches the text the model is shown).\n# Deliberately excludes code (.py, .ts, ...) and binary docs (.pdf) — those are\n# never sliced.\n#\n# This set has to keep pace with ``detect.DOC_EXTENSIONS``: anything classified\n# as a document reaches the semantic pass, and anything the pass sees that is\n# NOT listed here is silently cut at ``_FILE_CHAR_CAP`` by ``_read_files``. The\n# two lists drifted as DOC_EXTENSIONS grew — .qmd, .skill, .html, .yaml and .yml\n# were documents that never got sliced, so a 38k-character one reached the model\n# as its first 20k with no warning and no partial marker (#2900).\n# ``tests/test_oversized_document_slicing.py`` pins the relationship so a future\n# addition to DOC_EXTENSIONS fails loudly instead of quietly losing content.\n_SPLITTABLE_TEXT_SUFFIXES = frozenset({\n \".md\", \".mdx\", \".markdown\", \".txt\", \".rst\",\n \".qmd\", \".skill\", \".html\",\n \".j2\", \".jinja\", \".jinja2\",\n})\n\n# Document types whose BYTES are not what the model is shown. `llm._file_to_text`\n# routes these through a converter, so a character range has to be taken over the\n# converted text, never over the file. Kept separate from the set above because\n# they are splittable for a different reason and via a different reader.\n_CONVERTED_TEXT_SUFFIXES = frozenset({\".pdf\"})\n\n\ndef _pdf_text(path: Path) -> str:\n \"\"\"Extracted text of a PDF — the same string `llm._file_to_text` builds.\n\n Imported lazily from ``detect`` so this module keeps no import-time\n dependency on the extraction stack (``llm`` imports *this* module, so the\n reverse direction would be a cycle).\n \"\"\"\n from graphify.detect import extract_pdf_text\n return extract_pdf_text(path)\n\n\n# Slicing a PDF means extracting its text, and the slicing pass asks for the same\n# file several times: once to measure it, then once per slice as the prompt is\n# built. Memoised on (path, size, mtime_ns) so a corpus of papers is parsed once\n# rather than once per slice, and so a file rewritten mid-run is re-read instead\n# of served a stale body. Bounded: the entries are whole documents, and a large\n# corpus should not pin all of them in memory.\n# Resolve the path so relative names in different corpus roots cannot share text.\n_CONVERTED_TEXT_CACHE: \"dict[tuple, str]\" = {}\n_CONVERTED_TEXT_CACHE_MAX = 64\n\n\ndef unit_source_text(path: Path) -> str:\n \"\"\"The text a unit contributes to the prompt, whatever its container.\n\n Plain-text files are read directly; converted types (PDF) go through their\n converter. Both `expand_oversized_files` and `read_slice_text` use this, so\n the offsets a slice carries always index the same string the model sees.\n \"\"\"\n if path.suffix.lower() not in _CONVERTED_TEXT_SUFFIXES:\n return _read_source_text(path, warn=False)\n try:\n st = path.stat()\n key = (str(path.resolve()), st.st_size, st.st_mtime_ns)\n except OSError:\n return _pdf_text(path)\n hit = _CONVERTED_TEXT_CACHE.get(key)\n if hit is not None:\n return hit\n text = _pdf_text(path)\n if len(_CONVERTED_TEXT_CACHE) >= _CONVERTED_TEXT_CACHE_MAX:\n _CONVERTED_TEXT_CACHE.clear()\n _CONVERTED_TEXT_CACHE[key] = text\n return text\n\n# Boundary preferences, strongest first. A Markdown heading (``\\n#``) keeps a\n# section with its title; a blank line keeps a paragraph intact; a bare newline\n# avoids cutting mid-line. If none is found in the window we hard-cut.\n_BOUNDARY_SEPARATORS = (\"\\n#\", \"\\n\\n\", \"\\n\")\n\n\n@dataclass(frozen=True)\nclass FileSlice:\n \"\"\"A contiguous ``[start, end)`` character range of a splittable text file.\n\n ``index``/``total`` are for logging only. ``path`` is the real file on disk;\n the slice always reports ``path`` as its source so slices don't fragment the\n graph.\n \"\"\"\n\n path: Path\n start: int\n end: int\n index: int\n total: int\n\n\n# A unit of extraction work: either a whole file (``Path``) or one slice of one.\nUnit = \"Path | FileSlice\"\n\n\ndef unit_path(unit: \"Path | FileSlice\") -> Path:\n \"\"\"The on-disk path a unit belongs to (the parent file for a slice).\"\"\"\n return unit.path if isinstance(unit, FileSlice) else unit\n\n\ndef is_splittable_text(path: Path) -> bool:\n \"\"\"True for document types that may be sliced.\n\n Covers plain text read straight off disk and converted types (PDF) whose\n text is produced by a converter. Both are sliceable because\n :func:`unit_source_text` gives the slicing pass the same string the prompt\n will carry; what disqualifies a type is having no text at all (an image) or\n text the reader cannot address by character offset.\n \"\"\"\n suffix = path.suffix.lower()\n return suffix in _SPLITTABLE_TEXT_SUFFIXES or suffix in _CONVERTED_TEXT_SUFFIXES\n\n\ndef _best_cut(text: str, start: int, end: int) -> int:\n \"\"\"Return a cut index in ``(start, end]`` at the strongest nearby boundary.\n\n Searches the window ``text[start:end]`` for the latest heading, then blank\n line, then newline, and returns the index just *after* it (a heading cuts\n just *before* the ``#`` so the heading leads the next slice). Falls back to a\n hard cut at ``end`` when the window has no usable boundary, which still makes\n forward progress because ``end > start``.\n \"\"\"\n window = text[start:end]\n for sep in _BOUNDARY_SEPARATORS:\n idx = window.rfind(sep)\n if idx > 0: # a boundary strictly inside the window (non-empty prev slice)\n if sep == \"\\n#\":\n return start + idx + 1 # keep the newline with the previous slice\n return start + idx + len(sep)\n return end\n\n\ndef slice_boundaries(text: str, max_chars: int) -> list[tuple[int, int]]:\n \"\"\"Contiguous ``(start, end)`` ranges covering all of ``text``, each ≤ max_chars.\n\n Ranges are gap-free and non-overlapping, so concatenating the slices\n reproduces ``text`` exactly — no content is dropped.\n \"\"\"\n n = len(text)\n if n \u003c= max_chars:\n return [(0, n)]\n bounds: list[tuple[int, int]] = []\n pos = 0\n while pos \u003c n:\n hard = min(pos + max_chars, n)\n end = _best_cut(text, pos, hard) if hard \u003c n else n\n if end \u003c= pos: # defensive: never stall\n end = hard\n bounds.append((pos, end))\n pos = end\n return bounds\n\n\ndef expand_oversized_files(\n files: list[Path], max_chars: int\n) -> list[\"Path | FileSlice\"]:\n \"\"\"Replace each oversized splittable-text file with a list of ``FileSlice``s.\n\n Files at or below ``max_chars`` (and all non-splittable files) pass through\n unchanged as ``Path``, so behaviour is identical for everything that already\n fit. Unreadable files pass through untouched (the reader handles the error).\n \"\"\"\n out: list[\"Path | FileSlice\"] = []\n for f in files:\n if not is_splittable_text(f):\n out.append(f)\n continue\n try:\n # The CONVERTED text for a PDF, so the boundaries below index the\n # same string read_slice_text will later slice and the prompt will\n # carry — not the container's bytes (#2906).\n text = unit_source_text(f)\n except OSError:\n out.append(f)\n continue\n if len(text) \u003c= max_chars:\n out.append(f)\n continue\n ranges = slice_boundaries(text, max_chars)\n total = len(ranges)\n for i, (s, e) in enumerate(ranges):\n out.append(FileSlice(path=f, start=s, end=e, index=i, total=total))\n return out\n\n\ndef read_slice_text(fs: FileSlice) -> str:\n \"\"\"Read just this slice's characters from its parent file.\n\n Goes through :func:`unit_source_text`, so a PDF slice indexes the extracted\n text rather than the container's bytes — the offsets `expand_oversized_files`\n computed and the string the prompt carries are then the same string (#2906).\n \"\"\"\n return unit_source_text(fs.path)[fs.start:fs.end]\n\n\ndef bisect_slice(fs: FileSlice) -> tuple[FileSlice, FileSlice] | None:\n \"\"\"Split a slice into two halves at a newline near its midpoint, or None.\n\n Used by the adaptive-retry path when a single slice still overflows the\n model's output: halving it produces a smaller response. Returns None when the\n slice is already too small to split meaningfully.\n \"\"\"\n if fs.end - fs.start \u003c= 1:\n return None\n # Index the SAME string the slice offsets were computed against and the\n # prompt carries: for a PDF that is the extracted text via unit_source_text,\n # not the raw container bytes. Reading the container here (the old behavior)\n # searched for the newline cut in binary coordinates, so a compressed PDF\n # slice could cut mid-line or past the text end (#2906). Any converter/read\n # failure means we cannot split, so fall back to None (treated as atomic).\n try:\n text = unit_source_text(fs.path)\n except Exception:\n return None\n mid = (fs.start + fs.end) // 2\n nl = text.find(\"\\n\", mid, fs.end)\n cut = nl + 1 if (nl != -1 and fs.start \u003c nl + 1 \u003c fs.end) else mid\n if not (fs.start \u003c cut \u003c fs.end):\n return None\n left = FileSlice(fs.path, fs.start, cut, fs.index, fs.total)\n right = FileSlice(fs.path, cut, fs.end, fs.index, fs.total)\n return left, right\n","content_type":"text/x-python; charset=utf-8","language":"python","size":10638,"content_sha256":"95eef66dcf28445212043aa88078d5a0412451da2404e03deb232cc9c8e983a9"},{"filename":"global_graph.py","content":"from __future__ import annotations\nimport json\nimport hashlib\nimport sys\nfrom datetime import datetime, timezone\nfrom pathlib import Path\nimport networkx as nx\nfrom networkx.readwrite import json_graph as _jg\n\n_GLOBAL_DIR = Path.home() / \".graphify\"\n_GLOBAL_GRAPH = _GLOBAL_DIR / \"global-graph.json\"\n_GLOBAL_MANIFEST = _GLOBAL_DIR / \"global-manifest.json\"\n\n\ndef _load_manifest() -> dict:\n if _GLOBAL_MANIFEST.exists():\n try:\n return json.loads(_GLOBAL_MANIFEST.read_text(encoding=\"utf-8\"))\n except Exception as exc:\n # Don't silently wipe the user's manifest on a parse error: that\n # deletes every tracked repo. Back the bad file up and surface the\n # error so the user can recover or report it.\n backup = _GLOBAL_MANIFEST.with_suffix(\n _GLOBAL_MANIFEST.suffix + f\".corrupt.{int(datetime.now(timezone.utc).timestamp())}\"\n )\n try:\n _GLOBAL_MANIFEST.rename(backup)\n print(\n f\"[graphify global] manifest at {_GLOBAL_MANIFEST} failed to parse ({exc}); \"\n f\"moved to {backup} and starting fresh. Restore from the backup if this was \"\n f\"unexpected.\",\n file=sys.stderr,\n )\n except Exception as rename_exc:\n print(\n f\"[graphify global] manifest at {_GLOBAL_MANIFEST} failed to parse ({exc}) \"\n f\"and could not be backed up ({rename_exc}). Starting fresh.\",\n file=sys.stderr,\n )\n return {\"version\": 1, \"repos\": {}}\n\n\ndef _save_manifest(manifest: dict) -> None:\n _GLOBAL_DIR.mkdir(parents=True, exist_ok=True)\n from graphify.paths import write_json_atomic\n write_json_atomic(_GLOBAL_MANIFEST, manifest, indent=2)\n\n\ndef _load_global_graph() -> nx.Graph:\n if _GLOBAL_GRAPH.exists():\n from graphify.security import check_graph_file_size_cap\n check_graph_file_size_cap(_GLOBAL_GRAPH)\n data = json.loads(_GLOBAL_GRAPH.read_text(encoding=\"utf-8\"))\n if \"links\" not in data and \"edges\" in data:\n data = dict(data, links=data[\"edges\"])\n try:\n return _jg.node_link_graph(data, edges=\"links\")\n except TypeError:\n return _jg.node_link_graph(data)\n return nx.Graph()\n\n\ndef _save_global_graph(G: nx.Graph) -> None:\n _GLOBAL_DIR.mkdir(parents=True, exist_ok=True)\n try:\n data = _jg.node_link_data(G, edges=\"links\")\n except TypeError:\n data = _jg.node_link_data(G)\n from graphify.paths import write_json_atomic\n write_json_atomic(_GLOBAL_GRAPH, data, compact=True)\n\n\ndef _file_hash(path: Path) -> str:\n h = hashlib.sha256()\n h.update(path.read_bytes())\n return h.hexdigest()[:16]\n\n\ndef global_add(source_path: Path, repo_tag: str) -> dict:\n \"\"\"Add or update a project graph in the global graph.\n\n Returns a summary dict with keys: repo_tag, nodes_added, nodes_removed, skipped,\n cross_repo_calls.\n Skipped=True means the source graph hasn't changed since last add.\n \"\"\"\n from graphify.build import prefix_graph_for_global, prune_repo_from_graph\n\n if not source_path.exists():\n raise FileNotFoundError(f\"graph not found: {source_path}\")\n\n manifest = _load_manifest()\n src_hash = _file_hash(source_path)\n\n existing = manifest[\"repos\"].get(repo_tag, {})\n existing_path = existing.get(\"source_path\", \"\")\n if existing_path and existing_path != str(source_path.resolve()):\n print(\n f\"[graphify global] warning: repo tag '{repo_tag}' previously pointed to \"\n f\"{existing_path!r}, now updating to {str(source_path.resolve())!r}. \"\n f\"Use --as \u003ctag> to give it a different name.\",\n file=sys.stderr,\n )\n if existing.get(\"source_hash\") == src_hash:\n return {\"repo_tag\": repo_tag, \"nodes_added\": 0, \"nodes_removed\": 0, \"skipped\": True,\n \"cross_repo_calls\": 0}\n\n # Load source graph\n from graphify.security import check_graph_file_size_cap\n check_graph_file_size_cap(source_path)\n data = json.loads(source_path.read_text(encoding=\"utf-8\"))\n if \"links\" not in data and \"edges\" in data:\n data = dict(data, links=data[\"edges\"])\n from graphify.paths import restore_arc_direction\n data = restore_arc_direction(data)\n try:\n src_G = _jg.node_link_graph(data, edges=\"links\")\n except TypeError:\n src_G = _jg.node_link_graph(data)\n\n # Prefix IDs for cross-project isolation\n prefixed = prefix_graph_for_global(src_G, repo_tag)\n\n # Load global graph and prune stale nodes for this repo\n G = _load_global_graph()\n removed = prune_repo_from_graph(G, repo_tag)\n\n # Merge external-library nodes (no source_file) by label to avoid duplication\n external_labels = {\n d.get(\"label\", \"\"): n\n for n, d in G.nodes(data=True)\n if not d.get(\"source_file\") and d.get(\"label\")\n }\n # Map each deduplicated external onto the existing global node so that\n # edges incident to it can be rewired instead of dropped.\n remap = {}\n for node, data in prefixed.nodes(data=True):\n if not data.get(\"source_file\") and data.get(\"label\") in external_labels:\n remap[node] = external_labels[data[\"label\"]]\n\n # Compose: add prefixed nodes (except deduplicated externals) into global graph\n for node, data in prefixed.nodes(data=True):\n if node not in remap:\n G.add_node(node, **data)\n for u, v, data in prefixed.edges(data=True):\n u = remap.get(u, u)\n v = remap.get(v, v)\n # The direction markers name endpoints too; rewire them with u/v so\n # they never point at a deduplicated (absent) external node.\n for key in (\"_src\", \"_tgt\"):\n if data.get(key) in remap:\n data = {**data, key: remap[data[key]]}\n if u != v: # don't introduce self-loops via remapping\n G.add_edge(u, v, **data)\n\n added = prefixed.number_of_nodes() - len(remap)\n # A member call parked on a caller node (#3152) may be answered by a repo\n # already in the global graph, or by this one for a repo added earlier. The\n # pass recomputes its own output, so adding repos one at a time lands where a\n # single merge-graphs of the same inputs would.\n from graphify.cross_repo_calls import link_cross_repo_member_calls\n\n cross_repo_calls = link_cross_repo_member_calls(G)\n _save_global_graph(G)\n\n manifest[\"repos\"][repo_tag] = {\n \"added_at\": datetime.now(timezone.utc).isoformat(),\n \"source_path\": str(source_path.resolve()),\n \"node_count\": added,\n \"edge_count\": prefixed.number_of_edges(),\n \"source_hash\": src_hash,\n }\n _save_manifest(manifest)\n\n return {\"repo_tag\": repo_tag, \"nodes_added\": added, \"nodes_removed\": removed,\n \"skipped\": False, \"cross_repo_calls\": cross_repo_calls}\n\n\ndef global_remove(repo_tag: str) -> int:\n \"\"\"Remove all nodes for repo_tag from the global graph. Returns count removed.\"\"\"\n from graphify.build import prune_repo_from_graph\n\n manifest = _load_manifest()\n if repo_tag not in manifest[\"repos\"]:\n raise KeyError(f\"repo '{repo_tag}' not in global graph\")\n\n G = _load_global_graph()\n removed = prune_repo_from_graph(G, repo_tag)\n _save_global_graph(G)\n\n del manifest[\"repos\"][repo_tag]\n _save_manifest(manifest)\n return removed\n\n\ndef global_list() -> dict:\n \"\"\"Return the manifest repos dict.\"\"\"\n return _load_manifest().get(\"repos\", {})\n\n\ndef global_path() -> Path:\n return _GLOBAL_GRAPH\n","content_type":"text/x-python; charset=utf-8","language":"python","size":7642,"content_sha256":"d928c1e4f5de47b8cdf8f6bf36f2c95d722ad699af51d77eb12238b18ccb0b62"},{"filename":"google_workspace.py","content":"\"\"\"Optional Google Workspace shortcut export support.\n\nGoogle Drive for desktop stores native Docs, Sheets, and Slides as small JSON\nshortcut files (.gdoc, .gsheet, .gslides). Those files are pointers, not the\ndocument content. This module exports them to Markdown sidecars via the\ngoogleworkspace CLI (`gws`) so Graphify can extract their actual contents.\n\"\"\"\nfrom __future__ import annotations\n\nimport hashlib\nimport json\nimport os\nimport re\nimport shutil\nimport subprocess\nimport tempfile\nimport urllib.parse\nfrom pathlib import Path\nfrom typing import Callable, Any\n\n\nGOOGLE_WORKSPACE_EXTENSIONS = {\".gdoc\", \".gsheet\", \".gslides\"}\n\n\ndef google_workspace_enabled(value: str | None = None) -> bool:\n \"\"\"Return True when Google Workspace shortcut export is enabled.\"\"\"\n raw = value if value is not None else os.environ.get(\"GRAPHIFY_GOOGLE_WORKSPACE\", \"\")\n return raw.strip().lower() in {\"1\", \"true\", \"yes\", \"on\"}\n\n\ndef _safe_yaml_str(value: str) -> str:\n \"\"\"Escape a value for a YAML double-quoted scalar.\n\n Same escaping as graphify.ingest._yaml_str (F-009): backslash, quote,\n tab, NUL and other C0/DEL controls, plus the Unicode line breaks\n U+2028, U+2029 and U+0085, so a hostile shortcut url or file id cannot\n break out of the scalar and inject sibling frontmatter keys.\n \"\"\"\n if value is None:\n return \"\"\n out: list[str] = []\n for ch in str(value):\n cp = ord(ch)\n if ch == \"\\\\\":\n out.append(\"\\\\\\\\\")\n elif ch == '\"':\n out.append('\\\\\"')\n elif ch == \"\\n\":\n out.append(\"\\\\n\")\n elif ch == \"\\r\":\n out.append(\"\\\\r\")\n elif ch == \"\\t\":\n out.append(\"\\\\t\")\n elif ch == \"\\0\":\n out.append(\"\\\\0\")\n elif cp == 0x2028:\n out.append(\"\\\\L\")\n elif cp == 0x2029:\n out.append(\"\\\\P\")\n elif cp == 0x0085:\n out.append(\"\\\\x85\")\n elif cp \u003c 0x20 or cp == 0x7F:\n out.append(f\"\\\\x{cp:02x}\")\n else:\n out.append(ch)\n return \"\".join(out)\n\n\ndef _extract_file_id_from_url(url: str) -> str | None:\n \"\"\"Extract a Drive file ID from common Google Docs/Drive URL shapes.\"\"\"\n if not url:\n return None\n parsed = urllib.parse.urlparse(url)\n query = urllib.parse.parse_qs(parsed.query)\n if query.get(\"id\"):\n return query[\"id\"][0]\n match = re.search(r\"/(?:document|spreadsheets|presentation|file)/d/([^/?#]+)\", parsed.path)\n if match:\n return match.group(1)\n return None\n\n\ndef _extract_resource_key(url: str, data: dict[str, Any]) -> str | None:\n for key in (\"resource_key\", \"resourceKey\"):\n value = data.get(key)\n if value:\n return str(value)\n if not url:\n return None\n parsed = urllib.parse.urlparse(url)\n query = urllib.parse.parse_qs(parsed.query)\n if query.get(\"resourcekey\"):\n return query[\"resourcekey\"][0]\n return None\n\n\ndef read_google_shortcut(path: Path) -> dict[str, str | None]:\n \"\"\"Read a .gdoc/.gsheet/.gslides shortcut and return export metadata.\"\"\"\n try:\n data = json.loads(path.read_text(encoding=\"utf-8\"))\n except Exception as exc:\n raise RuntimeError(f\"could not read Google Workspace shortcut {path}: {exc}\") from exc\n\n url = str(data.get(\"url\") or \"\")\n file_id = (\n data.get(\"doc_id\")\n or data.get(\"file_id\")\n or data.get(\"fileId\")\n or data.get(\"id\")\n or _extract_file_id_from_url(url)\n )\n if not file_id:\n resource_id = str(data.get(\"resource_id\") or \"\")\n if \":\" in resource_id:\n file_id = resource_id.split(\":\", 1)[1]\n\n if not file_id:\n raise RuntimeError(f\"Google Workspace shortcut {path} does not include a Drive file ID\")\n\n return {\n \"file_id\": str(file_id),\n \"url\": url or None,\n \"resource_key\": _extract_resource_key(url, data),\n \"account\": str(data.get(\"email\")) if data.get(\"email\") else None,\n }\n\n\ndef _run_gws_export(file_id: str, mime_type: str, output: Path, resource_key: str | None = None) -> None:\n exe = shutil.which(\"gws\")\n if not exe:\n raise RuntimeError(\n \"gws is required for Google Workspace export. Install it from \"\n \"https://github.com/googleworkspace/cli and run `gws auth login -s drive`.\"\n )\n\n params: dict[str, str] = {\"fileId\": file_id, \"mimeType\": mime_type}\n # Drive resource keys are sent via X-Goog-Drive-Resource-Keys. The current\n # gws export command has no custom-header flag, so do not pass resourceKey\n # as an unsupported query parameter.\n _ = resource_key\n output = output.resolve()\n output.parent.mkdir(parents=True, exist_ok=True)\n timeout = int(os.environ.get(\"GRAPHIFY_GOOGLE_WORKSPACE_TIMEOUT\", \"120\"))\n result = subprocess.run(\n [exe, \"drive\", \"files\", \"export\", \"--params\", json.dumps(params), \"-o\", output.name],\n capture_output=True,\n cwd=output.parent,\n text=True,\n timeout=timeout,\n )\n if result.returncode != 0:\n stderr = (result.stderr or result.stdout or \"\").strip()\n if len(stderr) > 1200:\n stderr = stderr[:1200] + \"...\"\n raise RuntimeError(f\"gws export failed for {file_id}: {stderr}\")\n\n\ndef _sidecar_path(path: Path, out_dir: Path, root: \"Path | None\" = None) -> Path:\n # Hash the scan-root-relative, NFC-normalized path — not the absolute path.\n # The absolute form salts the sidecar name with the checkout location, so the\n # same shortcut in two clones/worktrees emits differently-named byte-identical\n # sidecars, each ingested as a distinct source doc when graphify-out/ is\n # committed (#2059; mirrors convert_office_file). NFC guards macOS NFD drift\n # (#1226). The relative path still disambiguates same-stem files.\n import unicodedata\n if root is None:\n root = out_dir.parent.parent\n try:\n key = path.resolve().relative_to(Path(root).resolve()).as_posix()\n except (ValueError, OSError):\n key = str(path.resolve())\n name_hash = hashlib.sha256(unicodedata.normalize(\"NFC\", key).encode()).hexdigest()[:8]\n return out_dir / f\"{path.stem}_{name_hash}.md\"\n\n\ndef _with_frontmatter(path: Path, shortcut: dict[str, str | None], body: str, exported_mime_type: str) -> str:\n source_url = shortcut.get(\"url\") or \"\"\n account = shortcut.get(\"account\") or \"\"\n account_line = \"\"\n if account:\n account_hash = hashlib.sha256(account.encode()).hexdigest()[:12]\n account_line = f'google_account_hash: \"{account_hash}\"\\n'\n return (\n \"---\\n\"\n f'source_file: \"{_safe_yaml_str(str(path))}\"\\n'\n 'source_type: \"google_workspace\"\\n'\n f'google_file_id: \"{_safe_yaml_str(shortcut[\"file_id\"] or \"\")}\"\\n'\n f'google_export_mime_type: \"{_safe_yaml_str(exported_mime_type)}\"\\n'\n f'source_url: \"{_safe_yaml_str(source_url)}\"\\n'\n f\"{account_line}\"\n \"---\\n\\n\"\n f\"\u003c!-- converted from Google Workspace shortcut: {path.name} -->\\n\\n\"\n f\"{body.strip()}\\n\"\n )\n\n\ndef convert_google_workspace_file(\n path: Path,\n out_dir: Path,\n *,\n xlsx_to_markdown: Callable[[Path], str] | None = None,\n root: \"Path | None\" = None,\n) -> Path | None:\n \"\"\"Export a Google Workspace shortcut to a Markdown sidecar.\n\n Returns the converted Markdown path, or None when conversion is unsupported\n or produced no readable content.\n \"\"\"\n ext = path.suffix.lower()\n if ext not in GOOGLE_WORKSPACE_EXTENSIONS:\n return None\n\n shortcut = read_google_shortcut(path)\n out_dir.mkdir(parents=True, exist_ok=True)\n out_path = _sidecar_path(path, out_dir, root=root)\n\n if ext == \".gdoc\":\n with tempfile.NamedTemporaryFile(\"w+b\", suffix=\".md\", delete=False, dir=out_dir) as tmp:\n tmp_path = Path(tmp.name)\n try:\n _run_gws_export(shortcut[\"file_id\"] or \"\", \"text/markdown\", tmp_path, shortcut.get(\"resource_key\"))\n body = tmp_path.read_text(encoding=\"utf-8\", errors=\"replace\")\n finally:\n tmp_path.unlink(missing_ok=True)\n if not body.strip():\n return None\n out_path.write_text(_with_frontmatter(path, shortcut, body, \"text/markdown\"), encoding=\"utf-8\")\n return out_path\n\n if ext == \".gslides\":\n with tempfile.NamedTemporaryFile(\"w+b\", suffix=\".txt\", delete=False, dir=out_dir) as tmp:\n tmp_path = Path(tmp.name)\n try:\n _run_gws_export(shortcut[\"file_id\"] or \"\", \"text/plain\", tmp_path, shortcut.get(\"resource_key\"))\n body = tmp_path.read_text(encoding=\"utf-8\", errors=\"replace\")\n finally:\n tmp_path.unlink(missing_ok=True)\n if not body.strip():\n return None\n out_path.write_text(_with_frontmatter(path, shortcut, body, \"text/plain\"), encoding=\"utf-8\")\n return out_path\n\n if ext == \".gsheet\":\n if xlsx_to_markdown is None:\n raise RuntimeError(\"Google Sheets export requires the office extra: pip install graphifyy[office,google]\")\n with tempfile.NamedTemporaryFile(\"w+b\", suffix=\".xlsx\", delete=False, dir=out_dir) as tmp:\n tmp_path = Path(tmp.name)\n try:\n _run_gws_export(\n shortcut[\"file_id\"] or \"\",\n \"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet\",\n tmp_path,\n shortcut.get(\"resource_key\"),\n )\n body = xlsx_to_markdown(tmp_path)\n finally:\n tmp_path.unlink(missing_ok=True)\n if not body.strip():\n return None\n out_path.write_text(\n _with_frontmatter(\n path,\n shortcut,\n body,\n \"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet\",\n ),\n encoding=\"utf-8\",\n )\n return out_path\n\n return None\n","content_type":"text/x-python; charset=utf-8","language":"python","size":9939,"content_sha256":"806c47a1c26e19c14a55afc06233f54221bf3b263a38d96f7912a1a9ec47c501"},{"filename":"hooks.py","content":"# git hook integration - install/uninstall graphify post-commit and post-checkout hooks\nfrom __future__ import annotations\nimport os\nimport re\nimport sys\nfrom pathlib import Path\n\n_HOOK_MARKER = \"# graphify-hook-start\"\n_HOOK_MARKER_END = \"# graphify-hook-end\"\n_CHECKOUT_MARKER = \"# graphify-checkout-hook-start\"\n_CHECKOUT_MARKER_END = \"# graphify-checkout-hook-end\"\n_MERGE_HOOK_MARKER = \"# graphify-merge-hook-start\"\n_MERGE_HOOK_MARKER_END = \"# graphify-merge-hook-end\"\n_REWRITE_HOOK_MARKER = \"# graphify-rewrite-hook-start\"\n_REWRITE_HOOK_MARKER_END = \"# graphify-rewrite-hook-end\"\n\n# __PINNED_PYTHON__ is replaced at install time with the absolute path of the\n# Python interpreter that ran `graphify hook install`. For uv-tool and pipx\n# installs the interpreter lives inside an isolated venv, so the launcher on\n# PATH is the only entry point — and GUI git clients / CI runners often have a\n# minimal PATH that omits ~/.local/bin. Pinning sys.executable at install time\n# makes the hook work regardless of PATH at git-trigger time.\n_PYTHON_DETECT = \"\"\"\\\n# Detect the correct Python interpreter (handles uv tool, pipx, venv, system installs).\n# _PINNED was recorded at hook-install time; tried first so the hook works even\n# when the graphify launcher is not on PATH (common in GUI clients and CI).\n#\n# Probes check availability with importlib.util.find_spec instead of importing\n# the package: a probe that imports graphify wholesale executes the full package\n# import (10s+ cold on machines with AV-scanned or large site-packages) and used\n# to run up to FOUR times synchronously, stalling every commit before the\n# detached launch even started. find_spec locates the package without executing\n# it, so each probe costs interpreter startup only. The detached rebuild still\n# fails loudly in the log if the package is broken under that interpreter.\n_GFY_PROBE=\"import importlib.util, sys; sys.exit(0 if importlib.util.find_spec('graphify') else 1)\"\nGRAPHIFY_PYTHON=\"\"\n_PINNED='__PINNED_PYTHON__'\nif [ -n \"$_PINNED\" ] && [ -x \"$_PINNED\" ] && \"$_PINNED\" -c \"$_GFY_PROBE\" 2>/dev/null; then\n GRAPHIFY_PYTHON=\"$_PINNED\"\nfi\n# Second probe: read graphify-out/.graphify_python (written by the skill and\n# CLI; survives uv-tool reinstalls and is the same source the README documents).\nif [ -z \"$GRAPHIFY_PYTHON\" ]; then\n _GFY_PYTHON_FILE=\"graphify-out/.graphify_python\"\n if [ -f \"$_GFY_PYTHON_FILE\" ]; then\n _FROM_FILE=$(cat \"$_GFY_PYTHON_FILE\" 2>/dev/null | tr -d '[:space:]')\n case \"$_FROM_FILE\" in\n *[!a-zA-Z0-9/_.@:\\\\\\\\-]*) _FROM_FILE=\"\" ;; # allowlist (covers Windows paths)\n esac\n if [ -n \"$_FROM_FILE\" ] && [ -x \"$_FROM_FILE\" ] && \"$_FROM_FILE\" -c \"$_GFY_PROBE\" 2>/dev/null; then\n GRAPHIFY_PYTHON=\"$_FROM_FILE\"\n fi\n fi\nfi\n# Third probe: resolve via the graphify launcher on PATH.\nif [ -z \"$GRAPHIFY_PYTHON\" ]; then\n GRAPHIFY_BIN=$(command -v graphify 2>/dev/null)\n if [ -n \"$GRAPHIFY_BIN\" ]; then\n # Windows pip layout: Scripts/graphify(.exe) sits beside ..\\\\python.exe\n # (or .\\\\python.exe inside a venv's Scripts dir). NOTE: command -v may\n # return the launcher path WITHOUT the .exe suffix, so this cannot key\n # on the extension.\n _GFY_BINDIR=$(dirname \"$GRAPHIFY_BIN\")\n if [ -x \"$_GFY_BINDIR/../python.exe\" ] && \"$_GFY_BINDIR/../python.exe\" -c \"$_GFY_PROBE\" 2>/dev/null; then\n GRAPHIFY_PYTHON=\"$_GFY_BINDIR/../python.exe\"\n elif [ -x \"$_GFY_BINDIR/python.exe\" ] && \"$_GFY_BINDIR/python.exe\" -c \"$_GFY_PROBE\" 2>/dev/null; then\n GRAPHIFY_PYTHON=\"$_GFY_BINDIR/python.exe\"\n fi\n fi\n if [ -z \"$GRAPHIFY_PYTHON\" ] && [ -n \"$GRAPHIFY_BIN\" ]; then\n # POSIX launcher: parse the shebang. head -c + tr strip NUL bytes first —\n # when the launcher is a Windows binary reached without its .exe suffix,\n # a raw `head -1` reads binary into the command substitution and the\n # shell warns about ignored null bytes on every commit. Gate on a\n # leading '#!': a launcher can also be a binary trampoline with no\n # shebang at all (uv tool installs on Windows), and its bytes must\n # never reach the shebang parse (#2852).\n case \"$GRAPHIFY_BIN\" in\n *.exe) _GFY_HEAD=\"\" ;;\n *) _GFY_HEAD=$(head -c 256 \"$GRAPHIFY_BIN\" 2>/dev/null | tr -d '\\\\000') ;;\n esac\n case \"$_GFY_HEAD\" in\n '#!'*) _SHEBANG=$(printf '%s\\\\n' \"$_GFY_HEAD\" | head -n 1 | sed 's/^#![[:space:]]*//') ;;\n *) _SHEBANG=\"\" ;;\n esac\n case \"$_SHEBANG\" in\n */env\\\\ *) GRAPHIFY_PYTHON=\"${_SHEBANG#*/env }\" ;;\n *) GRAPHIFY_PYTHON=\"$_SHEBANG\" ;;\n esac\n # Allowlist: only keep characters valid in a filesystem path to prevent\n # injection if the shebang contains shell metacharacters.\n case \"$GRAPHIFY_PYTHON\" in\n *[!a-zA-Z0-9/_.@:\\\\\\\\-]*) GRAPHIFY_PYTHON=\"\" ;;\n esac\n if [ -n \"$GRAPHIFY_PYTHON\" ] && ! \"$GRAPHIFY_PYTHON\" -c \"$_GFY_PROBE\" 2>/dev/null; then\n GRAPHIFY_PYTHON=\"\"\n fi\n fi\nfi\n# Fourth probe: uv tool environments. `uv tool install` (the README's\n# recommended method) puts graphify in an isolated venv that no ambient\n# python can import, and on Windows its launcher on PATH is a binary\n# trampoline with no shebang to parse — so the probes above can all miss a\n# healthy install and the hook dies at the last-resort fallback (#2852).\n# Scan the uv tool envs directly; UV_TOOL_DIR overrides the default\n# location. A tool env is adopted only if its python passes the probe, so a\n# co-installed tool without graphify never satisfies it.\n#\n# The snap roots matter because an install made from inside a snap-confined\n# editor lands in that snap's private HOME, which the plain $HOME roots above\n# never see once the hook runs from an ordinary shell. Revisions are globbed\n# rather than pinned: snap rotates them on update, which is exactly what makes\n# a pinned path unsafe (see _is_rotating_prefix).\nif [ -z \"$GRAPHIFY_PYTHON\" ]; then\n for _GFY_TOOLS in \\\n \"${UV_TOOL_DIR:-}\" \\\n \"$HOME/.local/share/uv/tools\" \\\n \"$HOME/AppData/Roaming/uv/tools\" \\\n \"$HOME\"/snap/*/current/.local/share/uv/tools \\\n \"$HOME\"/snap/*/[0-9]*/.local/share/uv/tools; do\n [ -n \"$_GFY_TOOLS\" ] || continue\n for _GFY_CAND in \"$_GFY_TOOLS\"/*/bin/python \"$_GFY_TOOLS\"/*/Scripts/python.exe; do\n [ -x \"$_GFY_CAND\" ] || continue\n if \"$_GFY_CAND\" -c \"$_GFY_PROBE\" 2>/dev/null; then\n GRAPHIFY_PYTHON=\"$_GFY_CAND\"\n break 2\n fi\n done\n done\nfi\n# Last resort: try python3 / python (works for system/venv installs on PATH).\nif [ -z \"$GRAPHIFY_PYTHON\" ]; then\n if command -v python3 >/dev/null 2>&1 && python3 -c \"$_GFY_PROBE\" 2>/dev/null; then\n GRAPHIFY_PYTHON=\"python3\"\n elif command -v python >/dev/null 2>&1 && python -c \"$_GFY_PROBE\" 2>/dev/null; then\n GRAPHIFY_PYTHON=\"python\"\n else\n echo \"[graphify hook] could not locate a Python with graphify installed. Add the graphify bin dir to PATH or re-run 'graphify hook install' from the env where graphify lives.\" >&2\n exit 0\n fi\nfi\n\"\"\"\n\n_GIT_C_ESCAPES = {\n \"a\": 7, \"b\": 8, \"t\": 9, \"n\": 10, \"v\": 11, \"f\": 12, \"r\": 13, '\"': 34, \"\\\\\": 92,\n}\n\n\ndef _unquote_git_path(name: str) -> str:\n \"\"\"Decode a path as printed by ``git diff --name-only``.\n\n Even with ``core.quotePath=false`` git wraps a name containing a double\n quote, backslash or control character in double quotes and C-escapes it,\n spelling raw bytes as ``\\\\ooo`` octal. Unquoted names are returned as-is.\n \"\"\"\n if len(name) \u003c 2 or not (name.startswith('\"') and name.endswith('\"')):\n return name\n body = name[1:-1]\n out = bytearray()\n i = 0\n while i \u003c len(body):\n ch = body[i]\n if ch != \"\\\\\" or i + 1 >= len(body):\n out += ch.encode(\"utf-8\")\n i += 1\n continue\n nxt = body[i + 1]\n if nxt in \"0123\" and i + 3 \u003c len(body) and all(c in \"01234567\" for c in body[i + 1:i + 4]):\n out.append(int(body[i + 1:i + 4], 8))\n i += 4\n elif nxt in _GIT_C_ESCAPES:\n out.append(_GIT_C_ESCAPES[nxt])\n i += 2\n else:\n out += ch.encode(\"utf-8\")\n i += 1\n return os.fsdecode(bytes(out))\n\n\n# The Python that the rebuild runs, shared by both hooks. Embedded verbatim into\n# the launcher below and re-executed in the detached child. Must not contain the\n# double-quote, $, backtick or backslash characters: it is carried inside a\n# shell double-quoted `-c \"...\"` argument (see _detached_launch).\n_REBUILD_BODY_COMMIT = \"\"\"\\\nimport os, signal, sys, threading, multiprocessing\nfrom pathlib import Path\n\nfrom graphify.hooks import _unquote_git_path\n\nchanged_raw = os.environ.get('GRAPHIFY_CHANGED', '')\nchanged = [Path(_unquote_git_path(f.strip())) for f in changed_raw.strip().splitlines() if f.strip()]\n\nif not changed:\n sys.exit(0)\n\nprint(f'[graphify hook] {len(changed)} file(s) changed - rebuilding graph...')\n\ntry:\n from graphify.watch import _rebuild_code, _apply_resource_limits\n _apply_resource_limits()\n _timeout = int(os.environ.get('GRAPHIFY_REBUILD_TIMEOUT', '600'))\n if _timeout > 0:\n if hasattr(signal, 'SIGALRM'):\n def _sigalrm_bail(*_a):\n # Killing here, before the exception unwinds, matters: once\n # TimeoutError starts propagating it passes straight through\n # the ProcessPoolExecutor with-block's own __exit__, which\n # calls shutdown(wait=True) and blocks until every worker\n # exits -- forever, for a worker stuck the way #3341 was,\n # since the alarm firing never actually stops it. Killing the\n # workers first means shutdown has nothing left to wait for.\n for _child in multiprocessing.active_children():\n _child.kill()\n raise TimeoutError(f'graphify rebuild exceeded {_timeout}s')\n signal.signal(signal.SIGALRM, _sigalrm_bail)\n signal.alarm(_timeout)\n else:\n def _bail():\n print(f'[graphify hook] graphify rebuild exceeded {_timeout}s', flush=True)\n for _child in multiprocessing.active_children():\n _child.kill()\n os._exit(1)\n _watchdog = threading.Timer(_timeout, _bail)\n _watchdog.daemon = True\n _watchdog.start()\n _force = os.environ.get('GRAPHIFY_FORCE', '').lower() in ('1', 'true', 'yes')\n _root = Path('.')\n from graphify.paths import GRAPHIFY_OUT as _out\n _saved = Path(_out) / '.graphify_root'\n if _saved.exists():\n _txt = _saved.read_text(encoding='utf-8-sig').strip()\n if _txt:\n _candidate = Path(_txt)\n try:\n _cwd = Path.cwd().resolve()\n _resolved = _candidate.resolve()\n # Python 3.13 no longer raises on a symlink loop (resolve returns\n # the path unresolved), so require a real directory: a loop or a\n # dangling target is not a dir and correctly falls back.\n _in_repo = (_resolved == _cwd or _cwd in _resolved.parents) and _resolved.is_dir()\n except (OSError, RuntimeError):\n _in_repo = False\n if _in_repo:\n _root = _candidate\n else:\n print(f'[graphify hook] ignoring out-of-repo .graphify_root: {_txt}')\n _rebuild_code(_root, changed_paths=changed, force=_force)\n # Refresh the work-memory lessons doc when saved Q&A outcomes exist\n # (best-effort; never fails the hook).\n try:\n _md = (_root / _out) / 'memory'\n if _md.is_dir() and any(_md.glob('*.md')):\n from graphify.reflect import reflect as _reflect\n _gj = (_root / _out) / 'graph.json'\n _reflect(memory_dir=_md, out_path=(_root / _out) / 'reflections' / 'LESSONS.md',\n graph_path=_gj if _gj.exists() else None)\n except Exception:\n pass\nexcept TimeoutError as exc:\n print(f'[graphify hook] {exc}')\n sys.exit(1)\nexcept Exception as exc:\n print(f'[graphify hook] Rebuild failed: {exc}')\n sys.exit(1)\n\"\"\"\n\n_REBUILD_BODY_CHECKOUT = \"\"\"\\\nfrom graphify.watch import _rebuild_code, _apply_resource_limits\nfrom pathlib import Path\nimport os, signal, sys, threading, multiprocessing\ntry:\n _apply_resource_limits()\n _timeout = int(os.environ.get('GRAPHIFY_REBUILD_TIMEOUT', '600'))\n if _timeout > 0:\n if hasattr(signal, 'SIGALRM'):\n def _sigalrm_bail(*_a):\n # Killing here, before the exception unwinds, matters: once\n # TimeoutError starts propagating it passes straight through\n # the ProcessPoolExecutor with-block's own __exit__, which\n # calls shutdown(wait=True) and blocks until every worker\n # exits -- forever, for a worker stuck the way #3341 was,\n # since the alarm firing never actually stops it. Killing the\n # workers first means shutdown has nothing left to wait for.\n for _child in multiprocessing.active_children():\n _child.kill()\n raise TimeoutError(f'graphify rebuild exceeded {_timeout}s')\n signal.signal(signal.SIGALRM, _sigalrm_bail)\n signal.alarm(_timeout)\n else:\n def _bail():\n print(f'[graphify] graphify rebuild exceeded {_timeout}s', flush=True)\n for _child in multiprocessing.active_children():\n _child.kill()\n os._exit(1)\n _watchdog = threading.Timer(_timeout, _bail)\n _watchdog.daemon = True\n _watchdog.start()\n _force = os.environ.get('GRAPHIFY_FORCE', '').lower() in ('1', 'true', 'yes')\n # post-checkout: branch switch can touch arbitrary files; full rebuild path\n # (no changed_paths) is correct here. The flock inside _rebuild_code still\n # prevents pile-ups when commit + checkout fire back-to-back.\n _root = Path('.')\n from graphify.paths import GRAPHIFY_OUT as _out\n _saved = Path(_out) / '.graphify_root'\n if _saved.exists():\n _txt = _saved.read_text(encoding='utf-8-sig').strip()\n if _txt:\n _candidate = Path(_txt)\n try:\n _cwd = Path.cwd().resolve()\n _resolved = _candidate.resolve()\n # Python 3.13 no longer raises on a symlink loop (resolve returns\n # the path unresolved), so require a real directory: a loop or a\n # dangling target is not a dir and correctly falls back.\n _in_repo = (_resolved == _cwd or _cwd in _resolved.parents) and _resolved.is_dir()\n except (OSError, RuntimeError):\n _in_repo = False\n if _in_repo:\n _root = _candidate\n else:\n print(f'[graphify] ignoring out-of-repo .graphify_root: {_txt}')\n _rebuild_code(_root, force=_force)\n # Refresh the work-memory lessons doc when saved Q&A outcomes exist\n # (best-effort; never fails the hook).\n try:\n _md = (_root / _out) / 'memory'\n if _md.is_dir() and any(_md.glob('*.md')):\n from graphify.reflect import reflect as _reflect\n _gj = (_root / _out) / 'graph.json'\n _reflect(memory_dir=_md, out_path=(_root / _out) / 'reflections' / 'LESSONS.md',\n graph_path=_gj if _gj.exists() else None)\n except Exception:\n pass\nexcept TimeoutError as exc:\n print(f'[graphify] {exc}')\n sys.exit(1)\nexcept Exception as exc:\n print(f'[graphify] Rebuild failed: {exc}')\n sys.exit(1)\n\"\"\"\n\n# Cross-platform detached-launch shim (#1161). The hooks used to background the\n# rebuild with `nohup \"$GRAPHIFY_PYTHON\" -c \"...\" &`, but Git for Windows' bundled\n# MSYS shell ships no nohup (nor setsid), so that line died with\n# 'nohup: command not found' and the rebuild silently never ran — git commit/pull\n# still returned 0, so the graph just went stale with no signal. graphify already\n# requires Python, so we let Python do the detaching: a tiny outer process spawns\n# the real rebuild fully detached and returns immediately, so the hook never\n# blocks. POSIX uses start_new_session (the setsid equivalent); Windows uses\n# CREATE_NO_WINDOW | CREATE_NEW_PROCESS_GROUP, breaking away from any job object\n# when allowed. Do NOT 'simplify' CREATE_NO_WINDOW back into DETACHED_PROCESS:\n# on Windows 11 with Windows Terminal as the default console host, a\n# console-less python still gets a VISIBLE console allocated when its runtime\n# touches the console API during startup (ctrl-handler installation), popping\n# an empty Terminal window over whatever the user is doing - once per commit,\n# for the whole rebuild. Reproduced via GetConsoleWindow(): DETACHED_PROCESS\n# child reports a visible hwnd, CREATE_NO_WINDOW child reports none (3bac3df).\n# This payload is carried inside a shell double-quoted -c argument,\n# so it deliberately uses only single-quoted Python strings (no \", $, ` or \\\\).\n_LAUNCHER_TEMPLATE = \"\"\"\\\nimport os, subprocess, sys\n_src = '''\n__REBUILD_BODY__\n'''\n_log = os.environ.get('GRAPHIFY_REBUILD_LOG') or os.path.join(os.path.expanduser('~'), '.cache', 'graphify-rebuild.log')\ntry:\n os.makedirs(os.path.dirname(_log), exist_ok=True)\n _out = open(_log, 'a', buffering=1, encoding='utf-8', errors='replace')\nexcept OSError:\n _out = subprocess.DEVNULL\n_kw = dict(stdout=_out, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, cwd=os.getcwd(), close_fds=True)\n_cmd = [sys.executable, '-c', _src]\nif os.name == 'nt':\n _flags = 0x08000000 | 0x00000200 # CREATE_NO_WINDOW | CREATE_NEW_PROCESS_GROUP\n try:\n subprocess.Popen(_cmd, creationflags=_flags | 0x01000000, **_kw) # + CREATE_BREAKAWAY_FROM_JOB\n except OSError:\n subprocess.Popen(_cmd, creationflags=_flags, **_kw)\nelse:\n subprocess.Popen(_cmd, start_new_session=True, **_kw)\n\"\"\"\n\n\ndef _detached_launch(rebuild_body: str) -> str:\n \"\"\"Return a POSIX-sh line that runs ``rebuild_body`` as a detached background\n Python process via ``$GRAPHIFY_PYTHON``.\n\n Replaces the old ``nohup ... &`` form, which failed on Git for Windows'\n shell (no nohup/setsid) and let the rebuild silently never run (#1161).\n The launcher writes the child's output to ``$GRAPHIFY_REBUILD_LOG`` and\n returns the instant the child is spawned, so the git hook never blocks.\n \"\"\"\n launcher = _LAUNCHER_TEMPLATE.replace(\"__REBUILD_BODY__\", rebuild_body)\n return '\"$GRAPHIFY_PYTHON\" -c \"' + launcher + '\"\\n'\n\n\n# Skip the rebuild inside a linked worktree (git worktree add), shared by both\n# hooks. With core.hooksPath shared across worktrees a commit in any worktree\n# fires these hooks; the canonical graphify-out/ belongs to the primary checkout,\n# so rebuilding from a worktree is wasteful, writes a rogue delta-only graph the\n# user never asked for, and races deploy/CI `git clean` against the detached\n# rebuild (\"failed to remove graphify-out/: Directory not empty\") (#1809, #1806).\n# A linked worktree has git-dir != git-common-dir. Both are resolved to absolute\n# via `cd ... && pwd` before comparing: git's exported GIT_DIR / --git-dir can be\n# absolute while --git-common-dir is the relative \".git\", and a raw compare would\n# false-positive on the PRIMARY checkout and wrongly skip it.\n_WORKTREE_GUARD = \"\"\"\\\n_GFY_GITDIR=$(cd \"$(git rev-parse --git-dir 2>/dev/null)\" 2>/dev/null && pwd)\n_GFY_COMMONDIR=$(cd \"$(git rev-parse --git-common-dir 2>/dev/null)\" 2>/dev/null && pwd)\nif [ -n \"$_GFY_COMMONDIR\" ] && [ \"$_GFY_GITDIR\" != \"$_GFY_COMMONDIR\" ]; then\n exit 0\nfi\n\"\"\"\n\n\n# Run the rest of each hook from the checkout's top level, shared by all hooks\n# and placed right after _WORKTREE_GUARD (it reuses _GFY_GITDIR/_GFY_COMMONDIR).\n# The rebuild reads graphify-out/ and its .graphify_root relative to the cwd,\n# but git does not always start hooks at the top: with GIT_DIR exported and no\n# GIT_WORK_TREE (a nested git call from another hook or tool), git takes the\n# caller's cwd as the work tree, and `--show-toplevel` echoes that same\n# subdirectory. A stray graphify-out/ left there by an earlier run then roots the\n# rebuild at the subdirectory - for .claude/worktrees/ that is every linked\n# worktree (#3258). In the standard layout the work tree is the parent of the\n# common .git dir; GIT_WORK_TREE and --separate-git-dir layouts keep git's own\n# answer. GIT_DIR is rewritten to its absolute form (not newly exported) so the\n# git calls below still resolve after the cd.\n_TOPLEVEL_ANCHOR = \"\"\"\\\nif [ -z \"${GIT_WORK_TREE:-}\" ] && [ \"${_GFY_COMMONDIR##*/}\" = \".git\" ]; then\n _GFY_TOP=${_GFY_COMMONDIR%/*}\nelse\n _GFY_TOP=$(git rev-parse --show-toplevel 2>/dev/null)\nfi\n[ -n \"$_GFY_TOP\" ] && cd \"$_GFY_TOP\" || exit 0\n[ -n \"$_GFY_GITDIR\" ] && GIT_DIR=$_GFY_GITDIR\n\"\"\"\n\n\n# Both hook bodies run inside a subshell `( ... )`. The generated block is\n# appended to whatever post-commit / post-checkout already exists, and other\n# tools chain their own logic after it; every skip condition in the block is a\n# bare `exit 0`, which in a flat script ends the WHOLE hook, silently dropping\n# anything after graphify's end marker - on every root commit (HEAD~1 does\n# not exist), every rebase/merge, every linked worktree, every\n# GRAPHIFY_SKIP_HOOK=1 (#2986). Inside the subshell an `exit` ends only\n# graphify's section; the detached rebuild launch is unaffected, and the\n# hook's own exit status stays 0 as before.\n_HOOK_SCRIPT = \"\"\"\\\n# graphify-hook-start\n# Auto-rebuilds the knowledge graph after each commit (code files only, no LLM needed).\n# Installed by: graphify hook install\n(\n\n# Deterministic clustering: networkx louvain iterates string-keyed sets whose\n# order is randomized per-process by PYTHONHASHSEED, so community assignments\n# churn run-to-run. Pinning it makes graphify-out reproducible.\nexport PYTHONHASHSEED=0\n__VIZ_LIMIT_EXPORT__\n# Git for Windows/MSYS hooks can inherit fragile pipe handles from GUI clients\n# and agent shells. Keep hook-triggered rebuilds sequential by default there;\n# explicit GRAPHIFY_MAX_WORKERS still wins for users who want parallelism.\nif [ -n \"${WINDIR:-}\" ] || [ -n \"${MSYSTEM:-}\" ]; then\n export GRAPHIFY_MAX_WORKERS=\"${GRAPHIFY_MAX_WORKERS:-1}\"\nfi\n\n# Skip during a rebase or merge to avoid blocking --continue with unstaged changes\n# git exports GIT_DIR to hooks; the rev-parse fallback only runs when invoked by\n# hand (each git exec costs 1s+ on AV-scanned Windows machines).\nGIT_DIR=${GIT_DIR:-$(git rev-parse --git-dir 2>/dev/null)}\n[ -d \"$GIT_DIR/rebase-merge\" ] && exit 0\n[ -d \"$GIT_DIR/rebase-apply\" ] && exit 0\n[ -f \"$GIT_DIR/MERGE_HEAD\" ] && exit 0\n[ \"${GRAPHIFY_SKIP_HOOK:-0}\" = \"1\" ] && exit 0\n\n\"\"\" + _WORKTREE_GUARD + _TOPLEVEL_ANCHOR + \"\"\"\n# CHERRY_PICK_HEAD can't be the skip signal: git leaves it on disk while post-commit\n# runs, so every pick was silenced (#4490, same trap as MERGE_HEAD in #4463). For a\n# range pick or revert, skip every step but the last and rebuild once from\n# sequencer/head, so a tracked graph.json isn't rewritten between steps. sequencer/ is\n# trusted only when its first todo line is the commit just made; a stale one left by\n# `git reset --hard` would otherwise silence every later commit.\n_GFY_SEQ_BASE=\"\"\nif [ -f \"$GIT_DIR/sequencer/todo\" ]; then\n IFS= read -r _GFY_FIRST \u003c \"$GIT_DIR/sequencer/todo\"\n _GFY_VERB=${_GFY_FIRST%% *}\n _GFY_REST=${_GFY_FIRST#* }\n case \"$_GFY_REST\" in\n *\" \"*) _GFY_SUBJ=${_GFY_REST#* } ;;\n *) _GFY_SUBJ=\"\" ;;\n esac\n _GFY_SEQ_LIVE=0\n _GFY_Q='\"'\n case \"$_GFY_VERB\" in\n pick) [ \"$(git log -1 --format=%s 2>/dev/null)\" = \"$_GFY_SUBJ\" ] && _GFY_SEQ_LIVE=1 ;;\n revert) [ \"$(git log -1 --format=%s 2>/dev/null)\" = \"Revert ${_GFY_Q}${_GFY_SUBJ}${_GFY_Q}\" ] && _GFY_SEQ_LIVE=1 ;;\n esac\n if [ \"$_GFY_SEQ_LIVE\" = \"1\" ]; then\n _GFY_PENDING=0\n while IFS= read -r _GFY_STEP || [ -n \"$_GFY_STEP\" ]; do\n [ -n \"$_GFY_STEP\" ] && _GFY_PENDING=$((_GFY_PENDING + 1))\n done \u003c \"$GIT_DIR/sequencer/todo\"\n [ \"$_GFY_PENDING\" -gt 1 ] && exit 0\n [ -f \"$GIT_DIR/sequencer/head\" ] && IFS= read -r _GFY_SEQ_BASE \u003c \"$GIT_DIR/sequencer/head\"\n fi\nfi\n\n# core.quotePath=false: by default git prints a non-ASCII path as a quoted\n# octal-escaped string, which matches no file on disk, so the edit was silently\n# left out of the rebuild. Names git still quotes are decoded in the rebuild.\nCHANGED=$(git -c core.quotePath=false diff --name-only HEAD~1 HEAD 2>/dev/null || git -c core.quotePath=false diff --name-only HEAD 2>/dev/null)\n# Last pick of a range: cover every commit picked since the sequence began, not only this one.\nif [ -n \"$_GFY_SEQ_BASE\" ]; then\n CHANGED=$(git -c core.quotePath=false diff --name-only \"$_GFY_SEQ_BASE\" HEAD 2>/dev/null)\nfi\nif [ -z \"$CHANGED\" ]; then\n exit 0\nfi\n\n# Skip when only output-dir artifacts changed (avoids rebuild loop when graph\n# outputs are tracked in git). The dir is whatever GRAPHIFY_OUT names, the same\n# source the rebuild body reads (#1423): a literal graphify-out/ here let a\n# commit touching only a renamed output dir's graph.json trigger a full rebuild.\n_GFY_OUT=\"${GRAPHIFY_OUT:-graphify-out}\"\n_GFY_OUT=\"${_GFY_OUT%/}\"\n# The leading ( on each pattern is POSIX and keeps bash 3.2 (macOS /bin/sh)\n# from mis-parsing the pattern's ) as the end of the $(...) substitution.\n_NON_GRAPH=$(printf '%s\\n' \"$CHANGED\" | while IFS= read -r _GFY_F; do\n case \"$_GFY_F\" in\n (\"$_GFY_OUT\"/*) ;;\n (*) printf '%s\\n' \"$_GFY_F\" ;;\n esac\ndone)\nif [ -z \"$_NON_GRAPH\" ]; then\n exit 0\nfi\n\n\"\"\" + _PYTHON_DETECT + \"\"\"\nexport GRAPHIFY_CHANGED=\"$CHANGED\"\n\n# Run the rebuild detached so git commit returns immediately. Full-repo rebuilds\n# can take hours; blocking the post-commit hook stalls the shell. The Python\n# launcher below detaches the child cross-platform, so it works on Git for\n# Windows' shell too (which lacks the coreutils backgrounding tools) (#1161).\n_GRAPHIFY_LOG=\"${HOME}/.cache/graphify-rebuild.log\"\nmkdir -p \"$(dirname \"$_GRAPHIFY_LOG\")\"\nexport GRAPHIFY_REBUILD_LOG=\"$_GRAPHIFY_LOG\"\necho \"[graphify hook] launching background rebuild (log: $_GRAPHIFY_LOG)\"\n\"\"\" + _detached_launch(_REBUILD_BODY_COMMIT) + \"\"\")\n# graphify-hook-end\n\"\"\"\n\n\n_CHECKOUT_SCRIPT = \"\"\"\\\n# graphify-checkout-hook-start\n# Auto-rebuilds the knowledge graph (code only) when switching branches.\n# Installed by: graphify hook install\n(\n\n# Deterministic clustering: networkx louvain iterates string-keyed sets whose\n# order is randomized per-process by PYTHONHASHSEED, so community assignments\n# churn run-to-run. Pinning it makes graphify-out reproducible.\nexport PYTHONHASHSEED=0\n__VIZ_LIMIT_EXPORT__\n# Git for Windows/MSYS hooks can inherit fragile pipe handles from GUI clients\n# and agent shells. Keep hook-triggered rebuilds sequential by default there;\n# explicit GRAPHIFY_MAX_WORKERS still wins for users who want parallelism.\nif [ -n \"${WINDIR:-}\" ] || [ -n \"${MSYSTEM:-}\" ]; then\n export GRAPHIFY_MAX_WORKERS=\"${GRAPHIFY_MAX_WORKERS:-1}\"\nfi\n\nPREV_HEAD=$1\nNEW_HEAD=$2\nBRANCH_SWITCH=$3\n\n# Only run on branch switches, not file checkouts\nif [ \"$BRANCH_SWITCH\" != \"1\" ]; then\n exit 0\nfi\n\n# A no-op checkout (e.g. `git checkout -b` with no start point) reports a\n# branch switch but leaves the tree unchanged ΓÇö nothing to rebuild (#2421).\n[ \"$PREV_HEAD\" = \"$NEW_HEAD\" ] && exit 0\n\n# Only run if the output dir exists (graph has been built before). Resolve it\n# from GRAPHIFY_OUT like the rebuild body does (#1423): a literal graphify-out/\n# here made the branch-switch rebuild a silent no-op for every renamed output dir.\n_GFY_OUT=\"${GRAPHIFY_OUT:-graphify-out}\"\nif [ ! -d \"${_GFY_OUT%/}\" ]; then\n exit 0\nfi\n\n# Skip during rebase/merge/cherry-pick\n# git exports GIT_DIR to hooks; the rev-parse fallback only runs when invoked by\n# hand (each git exec costs 1s+ on AV-scanned Windows machines).\nGIT_DIR=${GIT_DIR:-$(git rev-parse --git-dir 2>/dev/null)}\n[ -d \"$GIT_DIR/rebase-merge\" ] && exit 0\n[ -d \"$GIT_DIR/rebase-apply\" ] && exit 0\n[ -f \"$GIT_DIR/MERGE_HEAD\" ] && exit 0\n[ -f \"$GIT_DIR/CHERRY_PICK_HEAD\" ] && exit 0\n\n# Honor the same opt-out as post-commit: without this, GRAPHIFY_SKIP_HOOK=1\n# suppressed commit-triggered rebuilds but not branch-switch ones (#1809).\n[ \"${GRAPHIFY_SKIP_HOOK:-0}\" = \"1\" ] && exit 0\n\n\"\"\" + _WORKTREE_GUARD + _TOPLEVEL_ANCHOR + _PYTHON_DETECT + \"\"\"\n_GRAPHIFY_LOG=\"${HOME}/.cache/graphify-rebuild.log\"\nmkdir -p \"$(dirname \"$_GRAPHIFY_LOG\")\"\nexport GRAPHIFY_REBUILD_LOG=\"$_GRAPHIFY_LOG\"\necho \"[graphify] Branch switched - launching background rebuild (log: $_GRAPHIFY_LOG)\"\n\"\"\" + _detached_launch(_REBUILD_BODY_CHECKOUT) + \"\"\")\n# graphify-checkout-hook-end\n\"\"\"\n\n\n_MERGE_SCRIPT = \"\"\"\\\n# graphify-merge-hook-start\n# Auto-rebuilds the knowledge graph (code only) after a merge, unscoped.\n# The git merge driver graphify registers for graph.json does a pure union\n# of both sides, which can resurrect a symbol that was deleted on one side\n# (#2418). Nothing else ever re-extracts that symbol's file unless it is\n# touched again, so the phantom node would otherwise persist indefinitely;\n# an unscoped rebuild here re-derives the graph from the merged tree and\n# drops it like any other stale node.\n# Installed by: graphify hook install\n(\n\n# Deterministic clustering: networkx louvain iterates string-keyed sets whose\n# order is randomized per-process by PYTHONHASHSEED, so community assignments\n# churn run-to-run. Pinning it makes graphify-out reproducible.\nexport PYTHONHASHSEED=0\n__VIZ_LIMIT_EXPORT__\n# Git for Windows/MSYS hooks can inherit fragile pipe handles from GUI clients\n# and agent shells. Keep hook-triggered rebuilds sequential by default there;\n# explicit GRAPHIFY_MAX_WORKERS still wins for users who want parallelism.\nif [ -n \"${WINDIR:-}\" ] || [ -n \"${MSYSTEM:-}\" ]; then\n export GRAPHIFY_MAX_WORKERS=\"${GRAPHIFY_MAX_WORKERS:-1}\"\nfi\n\n# Only run if graphify-out/ exists (graph has been built before)\nif [ ! -d \"graphify-out\" ]; then\n exit 0\nfi\n\n# git leaves MERGE_HEAD on disk while post-merge runs and deletes it only\n# after the hook returns, so it must not be checked here: that skipped every\n# merge that makes a merge commit (#4463). The checks below are belt and\n# braces for any quirk that lands here mid rebase or cherry-pick.\n# git exports GIT_DIR to hooks; the rev-parse fallback only runs when invoked\n# by hand (each git exec costs 1s+ on AV-scanned Windows machines).\nGIT_DIR=${GIT_DIR:-$(git rev-parse --git-dir 2>/dev/null)}\n[ -d \"$GIT_DIR/rebase-merge\" ] && exit 0\n[ -d \"$GIT_DIR/rebase-apply\" ] && exit 0\n[ -f \"$GIT_DIR/CHERRY_PICK_HEAD\" ] && exit 0\n\n# Honor the same opt out the commit and checkout hooks already respect.\n[ \"${GRAPHIFY_SKIP_HOOK:-0}\" = \"1\" ] && exit 0\n\n\"\"\" + _WORKTREE_GUARD + _TOPLEVEL_ANCHOR + _PYTHON_DETECT + \"\"\"\n_GRAPHIFY_LOG=\"${HOME}/.cache/graphify-rebuild.log\"\nmkdir -p \"$(dirname \"$_GRAPHIFY_LOG\")\"\nexport GRAPHIFY_REBUILD_LOG=\"$_GRAPHIFY_LOG\"\necho \"[graphify] Merge completed - launching background rebuild (log: $_GRAPHIFY_LOG)\"\n\"\"\" + _detached_launch(_REBUILD_BODY_CHECKOUT) + \"\"\")\n# graphify-merge-hook-end\n\"\"\"\n\n\n_REWRITE_SCRIPT = \"\"\"\\\n# graphify-rewrite-hook-start\n# Auto-rebuilds the knowledge graph (code only) when a rebase finishes, including\n# `git pull --rebase`. post-commit cannot cover it: it skips while rebase-merge/\n# exists, and every replayed commit is made under it. git runs post-rewrite once,\n# after the last commit, with the tree in its final state (#4490).\n# Installed by: graphify hook install\n(\n\n# Deterministic clustering: networkx louvain iterates string-keyed sets whose\n# order is randomized per-process by PYTHONHASHSEED, so community assignments\n# churn run-to-run. Pinning it makes graphify-out reproducible.\nexport PYTHONHASHSEED=0\n__VIZ_LIMIT_EXPORT__\n# Git for Windows/MSYS hooks can inherit fragile pipe handles from GUI clients\n# and agent shells. Keep hook-triggered rebuilds sequential by default there;\n# explicit GRAPHIFY_MAX_WORKERS still wins for users who want parallelism.\nif [ -n \"${WINDIR:-}\" ] || [ -n \"${MSYSTEM:-}\" ]; then\n export GRAPHIFY_MAX_WORKERS=\"${GRAPHIFY_MAX_WORKERS:-1}\"\nfi\n\n# git writes the old/new sha pairs of the rewritten commits to stdin; drain them so\n# git never blocks on a full pipe. Skipped when run by hand, where stdin is a terminal.\n[ -t 0 ] || cat >/dev/null\n\n# $1 is \"rebase\" or \"amend\". An amend already rebuilt through post-commit.\n[ \"${1:-}\" = \"rebase\" ] || exit 0\n\n# Only run if the output dir exists (graph has been built before). Resolve it from\n# GRAPHIFY_OUT like the rebuild body does (#1423), not a literal graphify-out/.\n_GFY_OUT=\"${GRAPHIFY_OUT:-graphify-out}\"\nif [ ! -d \"${_GFY_OUT%/}\" ]; then\n exit 0\nfi\n\n# No rebase-merge/ or rebase-apply/ check, unlike post-commit and post-checkout: git\n# removes rebase-merge/ only after post-rewrite returns, so it is still on disk here\n# and would skip every rebase (the same trap as MERGE_HEAD in post-merge, #4463).\n\n# Honor the same opt out the commit and checkout hooks already respect.\n[ \"${GRAPHIFY_SKIP_HOOK:-0}\" = \"1\" ] && exit 0\n\n\"\"\" + _WORKTREE_GUARD + _TOPLEVEL_ANCHOR + _PYTHON_DETECT + \"\"\"\n_GRAPHIFY_LOG=\"${HOME}/.cache/graphify-rebuild.log\"\nmkdir -p \"$(dirname \"$_GRAPHIFY_LOG\")\"\nexport GRAPHIFY_REBUILD_LOG=\"$_GRAPHIFY_LOG\"\necho \"[graphify] Rebase completed - launching background rebuild (log: $_GRAPHIFY_LOG)\"\n\"\"\" + _detached_launch(_REBUILD_BODY_CHECKOUT) + \"\"\")\n# graphify-rewrite-hook-end\n\"\"\"\n\n\ndef _load_graphifyrc(root: Path) -> dict:\n \"\"\"Load ``\u003croot>/.graphifyrc``; the parser lives in :mod:`graphify.rcfile`.\n\n Kept as the hooks-side name so callers and tests need not move. Returns\n ``viz_node_limit`` (used here) alongside any other option the file sets.\n \"\"\"\n from graphify.rcfile import load_graphifyrc\n return load_graphifyrc(root)\n\n\ndef _git_root(path: Path) -> Path | None:\n \"\"\"Walk up to find .git directory.\"\"\"\n current = path.resolve()\n for parent in [current, *current.parents]:\n if (parent / \".git\").exists():\n return parent\n return None\n\n\n_WINDOWS_DRIVE_RE = re.compile(r\"^[A-Za-z]:[\\\\/]\")\n\n\ndef _reject_windows_path(value: str, source: str) -> None:\n \"\"\"Raise if a hooks path looks like a Windows absolute path (#1385).\n\n On POSIX/WSL ``Path(\"C:\\\\Users\\\\...\").is_absolute()`` is False, so an absolute\n Windows hooks path gets joined under the repo root and mkdir'd as a literal\n junk directory (backslashes and all), while install reports success and the\n real ``.git/hooks`` gets nothing. Fail loudly instead so the user can fix it.\n \"\"\"\n if os.name == \"nt\":\n return\n if _WINDOWS_DRIVE_RE.match(value) or \"\\\\\" in value:\n raise RuntimeError(\n f\"git hooks path from {source} looks like a Windows path: {value!r}. \"\n f\"On WSL/POSIX this can't resolve to a real directory. Unset it with \"\n f\"`git config --local --unset core.hooksPath`, or set a POSIX path.\"\n )\n\n\ndef _is_within(child: Path, parent: Path) -> bool:\n \"\"\"True if `child` is `parent` or a path underneath it.\"\"\"\n try:\n child.resolve().relative_to(parent.resolve())\n except (ValueError, OSError):\n return False\n return True\n\n\ndef _rev_parse_path(root: Path, flag: str) -> Path | None:\n \"\"\"Resolve one ``git rev-parse`` path flag against ``root``.\n\n ``-c core.hooksPath=`` is not used: an empty value makes ``--git-path hooks``\n print ``./`` instead of the real hooks directory.\n \"\"\"\n import subprocess as _sp\n try:\n res = _sp.run(\n [\"git\", \"-C\", str(root), \"rev-parse\", flag],\n capture_output=True, text=True, encoding=\"utf-8\",\n )\n except (OSError, FileNotFoundError):\n return None\n if res.returncode != 0:\n return None\n raw = res.stdout.strip()\n if not raw or any(c in raw for c in (\"\\n\", \"\\r\", \"\\x00\")):\n return None\n path = Path(raw)\n if not path.is_absolute():\n path = root / path\n return path.resolve()\n\n\ndef _builtin_hooks_dir(root: Path) -> Path | None:\n \"\"\"Git's own hooks directory, ignoring core.hooksPath.\n\n ``--git-path hooks`` follows core.hooksPath, so the default is derived from\n the common git dir. A linked worktree's hooks live in the main repo's\n ``.git/hooks``, which is outside the worktree root.\n \"\"\"\n for flag in (\"--git-common-dir\", \"--git-dir\"):\n git_dir = _rev_parse_path(root, flag)\n if git_dir is not None:\n return (git_dir / \"hooks\").resolve()\n return None\n\n\ndef _hooks_path_allowed(root: Path, candidate: Path) -> bool:\n \"\"\"Allow in-repo hook dirs (Husky) and git's own hooks dir only.\n\n A core.hooksPath that resolves anywhere else is repository-controlled\n local config. Honoring it makes `hook install` write outside the repo (#3869).\n \"\"\"\n if _is_within(candidate, root):\n return True\n builtin = _builtin_hooks_dir(root)\n return builtin is not None and candidate.resolve() == builtin.resolve()\n\n\ndef _hooks_dir(root: Path) -> Path:\n \"\"\"Return the git hooks directory, respecting core.hooksPath if set (e.g. Husky).\n\n Asks git itself via ``rev-parse --git-path hooks`` rather than parsing\n ``.git/config`` with configparser: git legally allows duplicate keys and\n sections (VS Code writes such configs), which a strict configparser rejects\n with DuplicateOptionError/DuplicateSectionError, so every hook command\n printed a spurious \"could not read core.hooksPath\" warning (#1907). git\n resolves core.hooksPath, includeIf, and linked worktrees (where .git is a\n file, not a directory) correctly in one place. Genuinely corrupt configs\n are still surfaced: git itself fails on them, and its stderr is printed.\n \"\"\"\n # NOTE: do NOT pass --path-format=absolute — added in git 2.31; older git\n # echoes it back as a literal argument, contaminating stdout and causing a\n # phantom directory to be created (#907). git -C \u003croot> already returns an\n # absolute path for worktree/external-gitdir cases, and a path relative to\n # \u003croot> for normal repos — anchoring on root covers both.\n import subprocess as _sp\n try:\n res = _sp.run(\n [\"git\", \"-C\", str(root), \"rev-parse\", \"--git-path\", \"hooks\"],\n capture_output=True, text=True, encoding=\"utf-8\",\n )\n if res.returncode != 0:\n # git failing here is a real signal (corrupt .git/config, tampering,\n # permission flips by another tool). Surface git's own stderr rather\n # than silently falling through to the default hooks directory.\n err = (res.stderr or \"\").strip()\n print(\n f\"[graphify hooks] git could not resolve the hooks path for \"\n f\"{root}: {err or f'git exited with code {res.returncode}'}\",\n file=sys.stderr,\n )\n else:\n raw = res.stdout.strip()\n # A valid hooks path can never contain newlines or NUL. Their presence\n # means git echoed an unrecognised flag back (old git behaviour).\n if raw and not any(c in raw for c in (\"\\n\", \"\\r\", \"\\x00\")):\n _reject_windows_path(raw, \"git rev-parse --git-path hooks\")\n d = (root / raw).resolve()\n if _hooks_path_allowed(root, d):\n d.mkdir(parents=True, exist_ok=True)\n return d\n print(\n f\"[graphify hooks] refusing hooks path {d}: it is outside \"\n f\"{root.resolve()}. Installing into the default git hooks \"\n f\"directory instead (#3869).\",\n file=sys.stderr,\n )\n default = _builtin_hooks_dir(root)\n if default is not None:\n default.mkdir(parents=True, exist_ok=True)\n return default\n except (OSError, FileNotFoundError):\n pass\n d = root / \".git\" / \"hooks\"\n d.mkdir(parents=True, exist_ok=True)\n return d\n\n\ndef _install_hook(\n hooks_dir: Path,\n name: str,\n script: str,\n marker: str,\n marker_end: str = \"\",\n) -> str:\n \"\"\"Install a single git hook, appending if an existing hook is present, or updating\n an existing graphify block in-place.\"\"\"\n hook_path = hooks_dir / name\n if hook_path.exists():\n content = hook_path.read_text(encoding=\"utf-8\")\n if marker in content:\n if marker_end and marker_end in content:\n start_idx = content.find(marker)\n end_idx = content.find(marker_end)\n if start_idx != -1 and end_idx != -1 and end_idx >= start_idx:\n end_idx += len(marker_end)\n new_content = content[:start_idx] + script.rstrip() + content[end_idx:]\n if new_content == content:\n return f\"already installed at {hook_path}\"\n hook_path.write_text(new_content, encoding=\"utf-8\", newline=\"\\n\")\n return f\"updated existing {name} hook at {hook_path}\"\n return f\"already installed at {hook_path}\"\n hook_path.write_text(content.rstrip() + \"\\n\\n\" + script, encoding=\"utf-8\", newline=\"\\n\")\n return f\"appended to existing {name} hook at {hook_path}\"\n hook_path.write_text(\"#!/bin/sh\\n\" + script, encoding=\"utf-8\", newline=\"\\n\")\n hook_path.chmod(0o755)\n return f\"installed at {hook_path}\"\n\n\ndef _uninstall_hook(hooks_dir: Path, name: str, marker: str, marker_end: str) -> str:\n \"\"\"Remove graphify section from a git hook using start/end markers.\"\"\"\n hook_path = hooks_dir / name\n if not hook_path.exists():\n return f\"no {name} hook found - nothing to remove.\"\n content = hook_path.read_text(encoding=\"utf-8\")\n if marker not in content:\n return f\"graphify hook not found in {name} - nothing to remove.\"\n new_content = re.sub(\n rf\"{re.escape(marker)}.*?{re.escape(marker_end)}\\n?\",\n \"\",\n content,\n flags=re.DOTALL,\n ).strip()\n if not new_content or new_content in (\"#!/bin/bash\", \"#!/bin/sh\"):\n hook_path.unlink()\n return f\"removed {name} hook at {hook_path}\"\n hook_path.write_text(new_content + \"\\n\", encoding=\"utf-8\", newline=\"\\n\")\n return f\"graphify removed from {name} at {hook_path} (other hook content preserved)\"\n\n\ndef _pinned_python() -> str:\n \"\"\"Return sys.executable if its path is shell-safe, else an empty string.\n\n Applies the same allowlist used in _PYTHON_DETECT: rejects any character\n that is not a valid plain filesystem path character, preventing $(...),\n backtick, double-quote, semicolon, etc. from being injected into generated\n shell scripts or the merge-driver command line. The allowlist includes ':'\n and '\\\\' so Windows paths (C:\\\\...) are accepted, and a plain space so\n Windows profile paths (C:\\\\Users\\\\First Last\\\\...) are too — a space cannot\n start a substitution or a new command, and every consumer quotes the value:\n the hook scripts embed it as '$_PINNED' (single-quoted, then referenced as\n \"$_PINNED\") and _register_merge_driver double-quotes it (#2166). Before that\n a space rejected the whole path, so hooks installed under any Windows user\n whose profile name contains a space silently pinned nothing. An empty return\n means callers must fall back to the `graphify` launcher on PATH — safe\n degradation.\n \"\"\"\n if re.search(r\"[^a-zA-Z0-9/_.@: \\\\-]\", sys.executable):\n return \"\"\n if _is_rotating_prefix(sys.executable):\n return \"\"\n return sys.executable\n\n\n# ``~/snap/\u003capp>/\u003crevision>/`` — snap swaps \u003crevision> on every package update and\n# prunes the old tree, so anything under it is a path with an expiry date.\n_ROTATING_PREFIX_RE = re.compile(r\"/snap/[^/]+/(\\d+|current)/\")\n\n\ndef _is_rotating_prefix(path: str) -> bool:\n \"\"\"True if `path` lives under a directory the packaging system rotates.\n\n A pin is only worth writing if it will still resolve tomorrow. An interpreter\n inside a snap revision will not: the revision number changes on update and the\n old tree is removed, which silently kills every hook pinned to it — observed\n across 15 repositories at once when an editor snap moved past its revision.\n\n Returning \"\" here is the documented safe degradation: the hook falls through to\n its other probes, including the uv-tools scan, which searches snap-confined\n homes too.\n \"\"\"\n return bool(_ROTATING_PREFIX_RE.search(path.replace(\"\\\\\", \"/\")))\n\n\ndef _merge_attr_line() -> str:\n \"\"\"The .gitattributes line assigning the graphify merge driver to graph.json.\n\n The graph lives under the configured output directory (graphify.paths,\n GRAPHIFY_OUT env override). gitattributes patterns are repo-relative, so an\n absolute output-dir override cannot be expressed there — fall back to the\n default name in that case.\n \"\"\"\n from graphify.paths import GRAPHIFY_OUT\n out = GRAPHIFY_OUT\n if not out or Path(out).is_absolute() or \"\\\\\" in out:\n out = \"graphify-out\"\n return f\"{out.rstrip('/')}/graph.json merge=graphify\"\n\n\ndef _has_merge_attr(content: str) -> bool:\n \"\"\"True if a (non-comment) `\u003c...>graph.json ... merge=graphify` line exists.\"\"\"\n for raw in content.splitlines():\n line = raw.strip()\n if not line or line.startswith(\"#\"):\n continue\n fields = line.split()\n if fields and fields[0].endswith(\"graph.json\") and \"merge=graphify\" in fields[1:]:\n return True\n return False\n\n\ndef _register_merge_driver(root: Path) -> str:\n \"\"\"Register the graph.json union merge driver in git config + .gitattributes (#1902).\n\n README and CHANGELOG 0.7.0 document `graphify merge-driver` as being set up\n by `hook install`, but install never actually registered it. Writes go\n through `git config` (never hand-edit .git/config — in a linked worktree the\n effective config is not at root/.git/config). The interpreter is pinned the\n same way the hook scripts pin it, so the driver works even when the graphify\n launcher is not on PATH at merge time.\n \"\"\"\n import subprocess as _sp\n pinned = _pinned_python()\n if pinned:\n # Double-quoted: the allowlist in _pinned_python() permits a space (Windows\n # profile paths), and git runs this driver string through a shell, so an\n # unquoted \"C:\\\\Users\\\\First Last\\\\...\\\\python.exe\" would split into two\n # words and the driver would never run (#2166). The same allowlist keeps\n # '

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

and backticks out, so double quotes cannot introduce expansion.\n driver = f'\"{pinned}\" -m graphify merge-driver %O %A %B'\n else:\n driver = \"graphify merge-driver %O %A %B\"\n try:\n for key, value in (\n (\"merge.graphify.name\", \"graphify graph.json union merge\"),\n (\"merge.graphify.driver\", driver),\n ):\n _sp.run(\n [\"git\", \"-C\", str(root), \"config\", key, value],\n check=True, capture_output=True, text=True,\n )\n except (OSError, _sp.CalledProcessError) as exc:\n return f\"not registered (git config failed: {exc})\"\n\n line = _merge_attr_line()\n attrs = root / \".gitattributes\"\n if attrs.exists():\n content = attrs.read_text(encoding=\"utf-8\")\n if _has_merge_attr(content):\n return f\"already registered ({line})\"\n # Never clobber other entries; preserve a trailing newline.\n if content and not content.endswith(\"\\n\"):\n content += \"\\n\"\n attrs.write_text(content + line + \"\\n\", encoding=\"utf-8\", newline=\"\\n\")\n else:\n attrs.write_text(line + \"\\n\", encoding=\"utf-8\", newline=\"\\n\")\n return f\"registered ({line})\"\n\n\ndef _unregister_merge_driver(root: Path) -> str:\n \"\"\"Remove the merge-driver git config keys and the .gitattributes line.\"\"\"\n import subprocess as _sp\n for key in (\"merge.graphify.name\", \"merge.graphify.driver\"):\n try:\n # --unset exits nonzero if the key is absent; that is fine.\n _sp.run(\n [\"git\", \"-C\", str(root), \"config\", \"--unset\", key],\n capture_output=True, text=True,\n )\n except OSError:\n pass\n attrs = root / \".gitattributes\"\n if not attrs.exists():\n return \"not registered - nothing to remove.\"\n content = attrs.read_text(encoding=\"utf-8\")\n kept = [\n raw for raw in content.splitlines()\n if not _has_merge_attr(raw)\n ]\n if kept == content.splitlines():\n return \"gitattributes entry not found - nothing to remove.\"\n if kept:\n # Other entries survive; the file stays.\n attrs.write_text(\"\\n\".join(kept) + \"\\n\", encoding=\"utf-8\", newline=\"\\n\")\n return \"removed from .gitattributes (other entries preserved)\"\n attrs.unlink()\n return \"removed (.gitattributes deleted - no other entries)\"\n\n\ndef _merge_driver_status(root: Path) -> str:\n \"\"\"Report whether the merge driver is registered (config + gitattributes).\"\"\"\n import subprocess as _sp\n try:\n res = _sp.run(\n [\"git\", \"-C\", str(root), \"config\", \"--get\", \"merge.graphify.driver\"],\n capture_output=True, text=True,\n )\n cfg_ok = res.returncode == 0 and bool(res.stdout.strip())\n except OSError:\n cfg_ok = False\n attrs = root / \".gitattributes\"\n attr_ok = attrs.exists() and _has_merge_attr(attrs.read_text(encoding=\"utf-8\"))\n if cfg_ok and attr_ok:\n return \"registered\"\n if cfg_ok:\n return \"partially registered (git config set, .gitattributes line missing)\"\n if attr_ok:\n return \"partially registered (.gitattributes line set, git config missing)\"\n return \"not registered\"\n\n\ndef _user_hooks_dir(hooks_dir: Path) -> Path:\n \"\"\"Return the user-editable hooks directory.\n\n Husky 9 sets core.hooksPath to .husky/_ (wrapper scripts auto-generated by\n Husky), while user-editable hooks live in the parent .husky/. Return the\n parent when the resolved dir ends in '_' so install/status/uninstall target\n the correct location (#987).\n \"\"\"\n if hooks_dir.name == \"_\":\n return hooks_dir.parent\n return hooks_dir\n\n\ndef install(path: Path = Path(\".\")) -> str:\n \"\"\"Install graphify's post-commit, post-checkout, post-merge, and post-rewrite hooks in the nearest git repo.\"\"\"\n root = _git_root(path)\n if root is None:\n raise RuntimeError(f\"No git repository found at or above {path.resolve()}\")\n\n hooks_dir = _user_hooks_dir(_hooks_dir(root))\n\n cfg = _load_graphifyrc(root)\n viz_limit = cfg.get(\"viz_node_limit\")\n if viz_limit is not None:\n # Use the `:-` default form (like GRAPHIFY_MAX_WORKERS below) so an\n # explicit `GRAPHIFY_VIZ_NODE_LIMIT=... git commit` still wins over the\n # baked project default — persisting config must not clobber a per-run\n # override.\n viz_export = f'export GRAPHIFY_VIZ_NODE_LIMIT=\"${{GRAPHIFY_VIZ_NODE_LIMIT:-{viz_limit}}}\"\\n'\n else:\n viz_export = \"\"\n\n pinned = _pinned_python()\n hook = _HOOK_SCRIPT.replace(\"__PINNED_PYTHON__\", pinned).replace(\"__VIZ_LIMIT_EXPORT__\", viz_export)\n checkout = _CHECKOUT_SCRIPT.replace(\"__PINNED_PYTHON__\", pinned).replace(\"__VIZ_LIMIT_EXPORT__\", viz_export)\n merge_hook = _MERGE_SCRIPT.replace(\"__PINNED_PYTHON__\", pinned).replace(\"__VIZ_LIMIT_EXPORT__\", viz_export)\n rewrite_hook = _REWRITE_SCRIPT.replace(\"__PINNED_PYTHON__\", pinned).replace(\"__VIZ_LIMIT_EXPORT__\", viz_export)\n\n commit_msg = _install_hook(hooks_dir, \"post-commit\", hook, _HOOK_MARKER, _HOOK_MARKER_END)\n checkout_msg = _install_hook(hooks_dir, \"post-checkout\", checkout, _CHECKOUT_MARKER, _CHECKOUT_MARKER_END)\n post_merge_msg = _install_hook(\n hooks_dir, \"post-merge\", merge_hook, _MERGE_HOOK_MARKER, _MERGE_HOOK_MARKER_END\n )\n post_rewrite_msg = _install_hook(\n hooks_dir, \"post-rewrite\", rewrite_hook, _REWRITE_HOOK_MARKER, _REWRITE_HOOK_MARKER_END\n )\n merge_msg = _register_merge_driver(root)\n\n return (\n f\"post-commit: {commit_msg}\\npost-checkout: {checkout_msg}\\n\"\n f\"post-merge: {post_merge_msg}\\npost-rewrite: {post_rewrite_msg}\\nmerge driver: {merge_msg}\"\n )\n\n\ndef uninstall(path: Path = Path(\".\")) -> str:\n \"\"\"Remove graphify's post-commit, post-checkout, post-merge, and post-rewrite hooks.\"\"\"\n root = _git_root(path)\n if root is None:\n raise RuntimeError(f\"No git repository found at or above {path.resolve()}\")\n\n hooks_dir = _user_hooks_dir(_hooks_dir(root))\n commit_msg = _uninstall_hook(hooks_dir, \"post-commit\", _HOOK_MARKER, _HOOK_MARKER_END)\n checkout_msg = _uninstall_hook(hooks_dir, \"post-checkout\", _CHECKOUT_MARKER, _CHECKOUT_MARKER_END)\n post_merge_msg = _uninstall_hook(\n hooks_dir, \"post-merge\", _MERGE_HOOK_MARKER, _MERGE_HOOK_MARKER_END\n )\n post_rewrite_msg = _uninstall_hook(\n hooks_dir, \"post-rewrite\", _REWRITE_HOOK_MARKER, _REWRITE_HOOK_MARKER_END\n )\n merge_msg = _unregister_merge_driver(root)\n\n return (\n f\"post-commit: {commit_msg}\\npost-checkout: {checkout_msg}\\n\"\n f\"post-merge: {post_merge_msg}\\npost-rewrite: {post_rewrite_msg}\\nmerge driver: {merge_msg}\"\n )\n\n\n# The two per-install values in a graphify block: the interpreter that ran\n# `hook install` and the .graphifyrc viz limit (status checks that one itself).\n_PINNED_LINE_RE = re.compile(r\"^_PINNED='[^'\\n]*'$\", re.MULTILINE)\n_VIZ_EXPORT_LINE_RE = re.compile(r\"^export GRAPHIFY_VIZ_NODE_LIMIT=.*\\n\", re.MULTILINE)\n\n\ndef _comparable_block(text: str, marker: str, marker_end: str) -> str | None:\n \"\"\"The graphify block in *text* with its per-install values blanked out, or\n None when the block has no end marker (install cannot rewrite those either).\"\"\"\n start = text.find(marker)\n end = text.find(marker_end, start) if start != -1 else -1\n if end == -1:\n return None\n block = text[start:end + len(marker_end)]\n return _VIZ_EXPORT_LINE_RE.sub(\"\", _PINNED_LINE_RE.sub(\"_PINNED=''\", block))\n\n\ndef status(path: Path = Path(\".\")) -> str:\n \"\"\"Check if graphify hooks are installed.\"\"\"\n root = _git_root(path)\n if root is None:\n return \"Not in a git repository.\"\n hooks_dir = _user_hooks_dir(_hooks_dir(root))\n # status is a read-only diagnostic: a malformed .graphifyrc must not turn it\n # into a traceback. Report the config problem and continue with no limit.\n try:\n cfg = _load_graphifyrc(root)\n except ValueError as exc:\n cfg = {}\n print(f\" warning: {exc}\")\n cfg_limit = cfg.get(\"viz_node_limit\")\n\n def _check(name: str, marker: str, marker_end: str, script: str) -> str:\n p = hooks_dir / name\n if not p.exists():\n return \"not installed\"\n text = p.read_text(encoding=\"utf-8\")\n if marker not in text:\n return \"not installed (hook exists but graphify not found)\"\n if cfg_limit is not None:\n # Baked as `\"${GRAPHIFY_VIZ_NODE_LIMIT:-\u003cn>}\"` so a per-run override\n # wins; match the default \u003cn>, and still accept the older bare\n # `\"\u003cn>\"` form from hooks installed before that change.\n m = re.search(\n r'export GRAPHIFY_VIZ_NODE_LIMIT=\"(?:\\$\\{GRAPHIFY_VIZ_NODE_LIMIT:-(\\d+)\\}|(\\d+))\"',\n text,\n )\n installed_limit = int(m.group(1) or m.group(2)) if m else None\n if installed_limit != cfg_limit:\n return (\n f\"installed (out of date: hook has limit \"\n f\"{installed_limit if installed_limit is not None else 'unset'}, \"\n f\".graphifyrc has {cfg_limit})\"\n )\n # Upgrading the package does not touch hooks already on disk, so a block\n # from an older release keeps running its old script (#3771).\n installed = _comparable_block(text, marker, marker_end)\n current = _comparable_block(script.replace(\"__VIZ_LIMIT_EXPORT__\", \"\"), marker, marker_end)\n if installed is not None and installed != current:\n return \"installed (out of date: run `graphify hook install` to refresh it)\"\n return \"installed\"\n\n commit = _check(\"post-commit\", _HOOK_MARKER, _HOOK_MARKER_END, _HOOK_SCRIPT)\n checkout = _check(\"post-checkout\", _CHECKOUT_MARKER, _CHECKOUT_MARKER_END, _CHECKOUT_SCRIPT)\n post_merge = _check(\"post-merge\", _MERGE_HOOK_MARKER, _MERGE_HOOK_MARKER_END, _MERGE_SCRIPT)\n post_rewrite = _check(\"post-rewrite\", _REWRITE_HOOK_MARKER, _REWRITE_HOOK_MARKER_END, _REWRITE_SCRIPT)\n merge = _merge_driver_status(root)\n\n res = (\n f\"post-commit: {commit}\\npost-checkout: {checkout}\\n\"\n f\"post-merge: {post_merge}\\npost-rewrite: {post_rewrite}\\nmerge driver: {merge}\"\n )\n if cfg_limit is not None:\n res += f\"\\nviz node limit: {cfg_limit}\"\n return res\n","content_type":"text/x-python; charset=utf-8","language":"python","size":57190,"content_sha256":"e2c4e86e01f8a1667c6266bae97e3093a1131cb4397f8d9f0c2aefbc460fc35f"},{"filename":"ids.py","content":"\"\"\"Single source of truth for node-ID normalization.\n\nThree independent producers must agree on node IDs or the graph splits a single\nentity into disconnected ghost nodes:\n\n1. The AST extractor (``extract._make_id``) — deterministic, per-language.\n2. The semantic subagents (LLM) — follow the node-ID spec in the skill prompt.\n3. The graph builder (``build._normalize_id``) — reconciles edge endpoints when\n the LLM emits IDs with slightly different punctuation or casing than the AST.\n\nHistorically the normalization recipe was copy-pasted into ``extract._make_id``\nand ``build._normalize_id`` and kept in sync only by mirrored docstrings, which\nis exactly how the recurring ID-drift bug class crept in (#811 Unicode collapse,\n#550 same-filename collisions, #1033 AST-vs-LLM file-node mismatch, #1104). This\nmodule exists so the recipe lives in one place and the two callers can no longer\ndiverge.\n\nThe recipe: iterate ``casefold`` then NFKC-normalize to a fixpoint (casefold can\n*expand* a character into a base letter plus a combining mark — ``İ`` -> ``i`` +\nU+0307 — and NFKC then recomposes what can be recomposed; because the two do not\ncommute and neither is a fixpoint of the other, a single pass is not\ncaseless-stable), then replace runs of non-word characters with a single\nunderscore (``re.UNICODE`` so CJK/Cyrillic/Arabic/accented-Latin letters survive\ninstead of collapsing to a per-file node), collapse repeated underscores, then\nstrip leading/trailing underscores.\n\nCasefolding runs BEFORE the non-word filter, not after. With it last, the\ncombining marks casefold introduces were never filtered: ``İslemYap`` produced\n``i̇slemyap`` — an id containing U+0307, which is not a ``\\\\w`` character — and a\nsecond pass collapsed it to ``i_slemyap``, so the function was not idempotent\nand the builder's re-normalization disagreed with the extractor's ``make_id``\nfor any Turkish identifier (#2614).\n\nCasefolding runs in a FIXPOINT LOOP, not once. A single ``NFKC(casefold(...))``\nleft ``normalize_id(s) != normalize_id(s.casefold())`` for some combining-mark\nsequences (Greek ypogegrammeni U+0345 followed by a combining accent):\npre-casefolding turns U+0345 into ``ι``, which NFKC composes with the accent into\na precomposed char the single pass never reached. Iterating to a fixpoint —\ncasefold first, on the raw input — makes the result caseless-stable regardless of\nhow many times the caller has already casefolded.\n\"\"\"\nfrom __future__ import annotations\n\nimport hashlib\nimport re\nimport unicodedata\n\n__all__ = [\"normalize_id\", \"make_id\", \"decl_salt\", \"DeclIdRegistry\"]\n\n\ndef normalize_id(s: str) -> str:\n r\"\"\"Normalize a single ID string to its canonical form.\n\n Guarantees, all enforced by tests:\n\n - Idempotent: ``normalize_id(normalize_id(s)) == normalize_id(s)``.\n - The result contains only ``\\w`` characters and ``_``.\n - Caseless-stable: ``normalize_id(s) == normalize_id(s.casefold())``.\n\n casefold and NFKC do not commute, and neither is a fixpoint of the other:\n casefolding a char can expand it into a base letter plus a combining mark\n (``İ`` -> ``i`` + U+0307), and NFKC can then recompose that mark with an\n adjacent one into a different precomposed char. A single ``NFKC(casefold(...))``\n pass therefore left ``normalize_id(s) != normalize_id(s.casefold())`` for some\n combining-mark sequences (e.g. Greek ypogegrammeni U+0345 followed by a\n combining accent): pre-casefolding turned U+0345 into ``ι`` which NFKC then\n composed with the accent, reaching a form the single-pass recipe never saw.\n\n So iterate ``casefold`` then ``NFKC`` to a fixpoint (casefold FIRST, on the\n raw input, so a caller that pre-casefolds lands on the same fixpoint). The\n loop is bounded — Unicode caseless folding converges in one or two steps —\n with a hard cap as a termination guard. Only then apply the ``[^\\w]+`` filter,\n so every combining mark casefold introduced has been fully normalized before\n it is filtered (#2614 and its combining-mark follow-on).\n \"\"\"\n cur = s\n for _ in range(6):\n nxt = unicodedata.normalize(\"NFKC\", cur.casefold())\n if nxt == cur:\n break\n cur = nxt\n cur = re.sub(r\"[^\\w]+\", \"_\", cur, flags=re.UNICODE)\n cur = re.sub(r\"_+\", \"_\", cur)\n return cur.strip(\"_\")\n\n\ndef decl_salt(name: str) -> str:\n \"\"\"Stable 12-hex digest appended when a declaration id collides (#4854).\"\"\"\n return hashlib.sha1(\n name.encode(\"utf-8\", \"replace\"), usedforsecurity=False\n ).hexdigest()[:12]\n\n\nclass DeclIdRegistry:\n \"\"\"Assign unique ids to declarations in one file.\n\n Ownership is keyed by the id the declaration would have used, so two\n declarations whose names normalize alike (``Foo`` / ``_Foo``, or ``Base`` in\n two namespaces) stay distinct. A reference lookup by name only answers when\n exactly one declaration in the file spells that name; an ambiguous name\n keeps the caller's existing fallback.\n \"\"\"\n\n def __init__(self, reserved_ids=()):\n self._assigned: dict[tuple[str, str], str] = {}\n self._owners: dict[str, tuple[str, str]] = {\n rid: (\"file\", rid) for rid in reserved_ids\n }\n self._by_name: dict[tuple[str, str], set[str]] = {}\n self._salted: set[str] = set()\n self._spelled: dict[str, str] = {}\n\n def resolve(self, nid: str, name: str, kind: str, taken) -> str:\n key = (kind, nid)\n owner = self._owners.get(nid)\n if owner == key:\n return self._assigned[key]\n if nid not in taken and owner is None:\n self._owners[nid] = key\n self._assigned[key] = nid\n self._spelled[nid] = name\n self._by_name.setdefault((kind, normalize_id(name)), set()).add(nid)\n return nid\n assigned = self._assigned.get(key)\n if assigned is not None:\n return assigned\n # An id already used by a real declaration is a duplicate reference to\n # that declaration (prototype owner, same name re-declared); reuse it\n # instead of minting a second node. Only a salted id, whose owner stole\n # the spelling from this declaration, forces a counter-salt (#4854). A\n # different kind only counts as a duplicate when spelled the same (TS\n # `class Foo` + `namespace Foo`): `class Foo` + `function foo` are two (#4904).\n if (nid in taken and owner is not None and owner[0] != \"file\"\n and nid not in self._salted\n and (owner[0] == kind or self._spelled.get(nid) == name)):\n return nid\n base = make_id(nid, decl_salt(name))\n candidate = base\n counter = 2\n while candidate in taken or candidate in self._owners:\n candidate = make_id(base, str(counter))\n counter += 1\n self._owners[candidate] = key\n self._assigned[key] = candidate\n self._salted.add(candidate)\n self._by_name.setdefault((kind, normalize_id(name)), set()).add(candidate)\n return candidate\n\n def lookup(self, name: str, kinds=(\"type\", \"fn\", \"var\", \"ns\", \"alias\")) -> str | None:\n norm = normalize_id(name)\n for kind in kinds:\n ids = self._by_name.get((kind, norm))\n if ids:\n return next(iter(ids)) if len(ids) == 1 else None\n return None\n\n\ndef make_id(*parts: str) -> str:\n \"\"\"Build a canonical node ID from one or more name parts.\n\n Parts are joined with ``_`` (after stripping stray ``_``/``.`` edges from each\n part) and then run through :func:`normalize_id`, so the result is identical to\n what the builder produces from the joined string.\n \"\"\"\n return normalize_id(\"_\".join(p.strip(\"_.\") for p in parts if p))\n","content_type":"text/x-python; charset=utf-8","language":"python","size":7774,"content_sha256":"10df21a09e0347cf4d8933a60607784db85685930c900cf0812d0411514e8cf7"},{"filename":"ingest.py","content":"# fetch URLs (tweet/arxiv/pdf/web) and save as annotated markdown\nfrom __future__ import annotations\nimport json\nimport re\nimport uuid\nimport urllib.error\nimport urllib.parse\nfrom datetime import datetime, timezone\nfrom pathlib import Path\n\nfrom graphify.security import safe_fetch, safe_fetch_text, validate_url\n\n\ndef _yaml_str(s: str) -> str:\n \"\"\"Escape a string for embedding in a YAML double-quoted scalar.\n\n Handles every YAML 1.1/1.2 line-break and control character that could\n let a hostile value (e.g. a fetched page title) break out of the quoted\n scalar and inject sibling YAML keys (F-009 / F-019). The previous\n implementation missed `\\\\t`, `\\\\0`, the unicode line-separator U+2028 and\n paragraph-separator U+2029 — all of which YAML treats as line breaks.\n\n We intentionally do not depend on PyYAML (not in pyproject deps) and\n instead emit safely-escaped double-quoted scalars by hand: the YAML\n double-quoted form recognises `\\\\\\\\`, `\\\\\"`, `\\\\n`, `\\\\r`, `\\\\t`, `\\\\0`,\n `\\\\L` (U+2028), `\\\\P` (U+2029), and `\\\\xNN`/`\\\\uNNNN` numeric escapes.\n \"\"\"\n if s is None:\n return \"\"\n out: list[str] = []\n for ch in str(s):\n cp = ord(ch)\n if ch == \"\\\\\":\n out.append(\"\\\\\\\\\")\n elif ch == '\"':\n out.append('\\\\\"')\n elif ch == \"\\n\":\n out.append(\"\\\\n\")\n elif ch == \"\\r\":\n out.append(\"\\\\r\")\n elif ch == \"\\t\":\n out.append(\"\\\\t\")\n elif ch == \"\\0\":\n out.append(\"\\\\0\")\n elif cp == 0x2028:\n out.append(\"\\\\L\")\n elif cp == 0x2029:\n out.append(\"\\\\P\")\n elif cp \u003c 0x20 or cp == 0x7F:\n out.append(f\"\\\\x{cp:02x}\")\n else:\n out.append(ch)\n return \"\".join(out)\n\n\ndef _safe_filename(url: str, suffix: str) -> str:\n \"\"\"Turn a URL into a safe filename.\"\"\"\n parsed = urllib.parse.urlparse(url)\n name = parsed.netloc + parsed.path\n name = re.sub(r\"[^\\w\\-]\", \"_\", name).strip(\"_\")\n name = re.sub(r\"_+\", \"_\", name)[:80]\n return name + suffix\n\n\ndef _host_is(host: str, *domains: str) -> bool:\n \"\"\"True when *host* is one of *domains* or a subdomain of one.\"\"\"\n return any(host == domain or host.endswith(f\".{domain}\") for domain in domains)\n\n\ndef _detect_url_type(url: str) -> str:\n \"\"\"Classify the URL for targeted extraction.\"\"\"\n parsed = urllib.parse.urlparse(url)\n # Match the host, not the URL text: dropbox.com and netflix.com contain \"x.com\".\n host = (parsed.hostname or \"\").rstrip(\".\")\n if _host_is(host, \"twitter.com\", \"x.com\"):\n return \"tweet\"\n if _host_is(host, \"arxiv.org\"):\n return \"arxiv\"\n if _host_is(host, \"github.com\"):\n return \"github\"\n if _host_is(host, \"youtube.com\", \"youtu.be\"):\n return \"youtube\"\n path = parsed.path.lower()\n if path.endswith(\".pdf\"):\n return \"pdf\"\n if any(path.endswith(ext) for ext in (\".png\", \".jpg\", \".jpeg\", \".webp\", \".gif\")):\n return \"image\"\n return \"webpage\"\n\n\ndef _fetch_html(url: str) -> str:\n return safe_fetch_text(url)\n\n\ndef _html_to_markdown(html: str, url: str) -> str:\n \"\"\"Convert HTML to clean markdown. Uses markdownify if available, else basic strip.\"\"\"\n # Always pre-strip script/style so their text content never leaks into output\n html = re.sub(r\"\u003cscript[^>]*>.*?\u003c/script>\", \"\", html, flags=re.DOTALL | re.IGNORECASE)\n html = re.sub(r\"\u003cstyle[^>]*>.*?\u003c/style>\", \"\", html, flags=re.DOTALL | re.IGNORECASE)\n try:\n from markdownify import markdownify\n return markdownify(html, heading_style=\"ATX\", bullets=\"-\", strip=[\"img\"])\n except ImportError:\n # Fallback: basic tag strip\n text = re.sub(r\"\u003c[^>]+>\", \" \", html)\n text = re.sub(r\"\\s+\", \" \", text).strip()\n return text[:8000]\n\n\ndef _fetch_tweet(url: str, author: str | None, contributor: str | None) -> tuple[str, str]:\n \"\"\"Fetch a tweet URL. Returns (content, filename).\"\"\"\n # Normalize to twitter.com for oEmbed. Rewrite the host only: a plain\n # text replace also rewrote \"x.com\" wherever it appeared in the path or query.\n parts = urllib.parse.urlsplit(url)\n host = (parts.hostname or \"\").rstrip(\".\")\n if _host_is(host, \"x.com\"):\n host = host[: -len(\"x.com\")] + \"twitter.com\"\n netloc = f\"{host}:{parts.port}\" if parts.port else host\n parts = parts._replace(netloc=netloc)\n oembed_url = urllib.parse.urlunsplit(parts)\n oembed_api = f\"https://publish.twitter.com/oembed?url={urllib.parse.quote(oembed_url)}&omit_script=true\"\n try:\n data = json.loads(safe_fetch_text(oembed_api))\n tweet_text = re.sub(r\"\u003c[^>]+>\", \"\", data.get(\"html\", \"\")).strip()\n tweet_author = data.get(\"author_name\", \"unknown\")\n except Exception:\n # oEmbed failed - save URL stub\n tweet_text = f\"Tweet at {url} (could not fetch content)\"\n tweet_author = \"unknown\"\n\n now = datetime.now(timezone.utc).isoformat()\n content = f\"\"\"---\nsource_url: \"{_yaml_str(url)}\"\ntype: tweet\nauthor: \"{_yaml_str(tweet_author)}\"\ncaptured_at: {now}\ncontributor: \"{_yaml_str(contributor or author or 'unknown')}\"\n---\n\n# Tweet by @{tweet_author}\n\n{tweet_text}\n\nSource: {url}\n\"\"\"\n filename = _safe_filename(url, \".md\")\n return content, filename\n\n\ndef _fetch_webpage(url: str, author: str | None, contributor: str | None) -> tuple[str, str]:\n \"\"\"Fetch a generic webpage and convert to markdown.\"\"\"\n html = _fetch_html(url)\n # Extract title\n title_match = re.search(r\"\u003ctitle[^>]*>(.*?)\u003c/title>\", html, re.IGNORECASE | re.DOTALL)\n title = re.sub(r\"\\s+\", \" \", title_match.group(1)).strip() if title_match else url\n\n markdown = _html_to_markdown(html, url)\n now = datetime.now(timezone.utc).isoformat()\n content = f\"\"\"---\nsource_url: \"{_yaml_str(url)}\"\ntype: webpage\ntitle: \"{_yaml_str(title)}\"\ncaptured_at: {now}\ncontributor: \"{_yaml_str(contributor or author or 'unknown')}\"\n---\n\n# {title}\n\nSource: {url}\n\n---\n\n{markdown[:12000]}\n\"\"\"\n filename = _safe_filename(url, \".md\")\n return content, filename\n\n\ndef _fetch_arxiv(url: str, author: str | None, contributor: str | None) -> tuple[str, str]:\n \"\"\"Fetch arXiv abstract page.\"\"\"\n # Convert /abs/ or /pdf/ to abs for the API\n arxiv_id = re.search(r\"(\\d{4}\\.\\d{4,5})\", url)\n if arxiv_id:\n api_url = f\"https://export.arxiv.org/abs/{arxiv_id.group(1)}\"\n try:\n html = _fetch_html(api_url)\n abstract_match = re.search(r'class=\"abstract[^\"]*\"[^>]*>(.*?)\u003c/blockquote>', html, re.DOTALL | re.IGNORECASE)\n abstract = re.sub(r\"\u003c[^>]+>\", \"\", abstract_match.group(1)).strip() if abstract_match else \"\"\n title_match = re.search(r'class=\"title[^\"]*\"[^>]*>(.*?)\u003c/h1>', html, re.DOTALL | re.IGNORECASE)\n title = re.sub(r\"\u003c[^>]+>\", \" \", title_match.group(1)).strip() if title_match else arxiv_id.group(1)\n authors_match = re.search(r'class=\"authors\"[^>]*>(.*?)\u003c/div>', html, re.DOTALL | re.IGNORECASE)\n paper_authors = re.sub(r\"\u003c[^>]+>\", \"\", authors_match.group(1)).strip() if authors_match else \"\"\n except Exception:\n title, abstract, paper_authors = arxiv_id.group(1), \"\", \"\"\n else:\n return _fetch_webpage(url, author, contributor)\n\n now = datetime.now(timezone.utc).isoformat()\n content = f\"\"\"---\nsource_url: \"{_yaml_str(url)}\"\narxiv_id: \"{_yaml_str(arxiv_id.group(1) if arxiv_id else '')}\"\ntype: paper\ntitle: \"{_yaml_str(title)}\"\npaper_authors: \"{_yaml_str(paper_authors)}\"\ncaptured_at: {now}\ncontributor: \"{_yaml_str(contributor or author or 'unknown')}\"\n---\n\n# {title}\n\n**Authors:** {paper_authors}\n**arXiv:** {arxiv_id.group(1) if arxiv_id else url}\n\n## Abstract\n\n{abstract}\n\nSource: {url}\n\"\"\"\n filename = f\"arxiv_{arxiv_id.group(1).replace('.', '_')}.md\" if arxiv_id else _safe_filename(url, \".md\")\n return content, filename\n\n\ndef _free_capture_path(target_dir: Path, filename: str) -> Path:\n \"\"\"Pick *filename*, or the first free numbered variant, in *target_dir*.\n\n Uses lstat so no existing directory entry is replaced, including a\n dangling symlink that a write would otherwise follow out of the corpus.\n Selection and writing are separate; concurrent captures are not atomic.\n \"\"\"\n stem, suffix = Path(filename).stem, Path(filename).suffix\n out_path = target_dir / filename\n counter = 1\n while True:\n try:\n out_path.lstat()\n except FileNotFoundError:\n return out_path\n out_path = target_dir / f\"{stem}_{counter}{suffix}\"\n counter += 1\n\n\ndef _download_binary(url: str, suffix: str, target_dir: Path) -> Path:\n \"\"\"Save a sequential capture without replacing an existing directory entry.\"\"\"\n out_path = _free_capture_path(target_dir, _safe_filename(url, suffix))\n out_path.write_bytes(safe_fetch(url))\n return out_path\n\n\ndef ingest(url: str, target_dir: Path, author: str | None = None, contributor: str | None = None) -> Path:\n \"\"\"\n Fetch a URL and save it into target_dir as a graphify-ready file.\n\n Returns the path of the saved file.\n \"\"\"\n target_dir.mkdir(parents=True, exist_ok=True)\n url_type = _detect_url_type(url)\n\n try:\n validate_url(url)\n except ValueError as exc:\n raise ValueError(f\"ingest: {exc}\") from exc\n\n try:\n if url_type == \"pdf\":\n out = _download_binary(url, \".pdf\", target_dir)\n print(f\"Downloaded PDF: {out.name}\")\n return out\n\n if url_type == \"image\":\n suffix = Path(urllib.parse.urlparse(url).path).suffix or \".jpg\"\n out = _download_binary(url, suffix, target_dir)\n print(f\"Downloaded image: {out.name}\")\n return out\n\n if url_type == \"youtube\":\n from graphify.transcribe import download_audio\n out = download_audio(url, target_dir)\n print(f\"Downloaded audio: {out.name}\")\n return out\n\n if url_type == \"tweet\":\n content, filename = _fetch_tweet(url, author, contributor)\n elif url_type == \"arxiv\":\n content, filename = _fetch_arxiv(url, author, contributor)\n else:\n content, filename = _fetch_webpage(url, author, contributor)\n except (urllib.error.HTTPError, urllib.error.URLError, OSError) as exc:\n raise RuntimeError(f\"ingest: failed to fetch {url!r}: {exc}\") from exc\n\n out_path = _free_capture_path(target_dir, filename)\n out_path.write_text(content, encoding=\"utf-8\")\n print(f\"Saved {url_type}: {out_path.name}\")\n return out_path\n\nOUTCOMES = (\"useful\", \"dead_end\", \"corrected\")\n\n\ndef save_query_result(\n question: str,\n answer: str,\n memory_dir: Path,\n query_type: str = \"query\",\n source_nodes: list[str] | None = None,\n outcome: str | None = None,\n correction: str | None = None,\n) -> Path:\n \"\"\"Save a Q&A result as markdown so it gets extracted into the graph on next --update.\n\n Files are stored in memory_dir (typically graphify-out/memory/) with YAML frontmatter\n that graphify's extractor reads as node metadata. This closes the feedback loop:\n the system grows smarter from both what you add AND what you ask.\n\n ``outcome`` (one of :data:`OUTCOMES`) and ``correction`` are optional work-memory\n signals: they are written both to the frontmatter (so `graphify reflect` can\n aggregate them deterministically) and to an ``## Outcome`` body section (so the\n signal round-trips into the graph on the next semantic re-extraction).\n \"\"\"\n if outcome is not None and outcome not in OUTCOMES:\n raise ValueError(f\"outcome must be one of {OUTCOMES}, got {outcome!r}\")\n\n memory_dir = Path(memory_dir)\n memory_dir.mkdir(parents=True, exist_ok=True)\n\n now = datetime.now(timezone.utc)\n slug = re.sub(r\"[^\\w]\", \"_\", question.lower())[:50].strip(\"_\")\n # A second-granularity stamp plus a 50-char slug is not unique: two saves in\n # the same second whose questions share a prefix resolve to one path, and the\n # later write_text silently replaces the earlier one (#3301). The short uuid\n # makes every save its own file; the query_ prefix and .md suffix are kept.\n filename = f\"query_{now.strftime('%Y%m%d_%H%M%S')}_{uuid.uuid4().hex[:8]}_{slug}.md\"\n\n frontmatter_lines = [\n \"---\",\n f'type: \"{query_type}\"',\n f'date: \"{now.isoformat()}\"',\n f'question: \"{_yaml_str(question)}\"',\n 'contributor: \"graphify\"',\n ]\n if outcome:\n frontmatter_lines.append(f'outcome: \"{_yaml_str(outcome)}\"')\n if correction:\n frontmatter_lines.append(f'correction: \"{_yaml_str(correction)}\"')\n if source_nodes:\n nodes_str = \", \".join(f'\"{_yaml_str(n)}\"' for n in source_nodes[:10])\n frontmatter_lines.append(f\"source_nodes: [{nodes_str}]\")\n frontmatter_lines.append(\"---\")\n\n body_lines = [\n \"\",\n f\"# Q: {question}\",\n \"\",\n \"## Answer\",\n \"\",\n answer,\n ]\n if outcome or correction:\n body_lines += [\"\", \"## Outcome\", \"\"]\n if outcome:\n body_lines.append(f\"- Signal: {outcome}\")\n if correction:\n body_lines.append(f\"- Correction: {correction}\")\n if source_nodes:\n body_lines += [\"\", \"## Source Nodes\", \"\"]\n body_lines += [f\"- {n}\" for n in source_nodes]\n\n content = \"\\n\".join(frontmatter_lines + body_lines)\n out_path = memory_dir / filename\n out_path.write_text(content, encoding=\"utf-8\")\n return out_path\n\n\nif __name__ == \"__main__\":\n import argparse\n parser = argparse.ArgumentParser(description=\"Fetch a URL into a graphify /raw folder\")\n parser.add_argument(\"url\", help=\"URL to fetch\")\n parser.add_argument(\"target_dir\", nargs=\"?\", default=\"./raw\", help=\"Target directory (default: ./raw)\")\n parser.add_argument(\"--author\", help=\"Your name (stored as node metadata)\")\n parser.add_argument(\"--contributor\", help=\"Contributor name for team graphs\")\n args = parser.parse_args()\n out = ingest(args.url, Path(args.target_dir), author=args.author, contributor=args.contributor)\n print(f\"Ready for graphify: {out}\")\n","content_type":"text/x-python; charset=utf-8","language":"python","size":14114,"content_sha256":"16a861230f079619cfff49964ad98cd491d719c66544f86e9f83ae12a90ac0aa"},{"filename":"interface_dispatch.py","content":"\"\"\"Member-level dispatch from an interface requirement to its implementer(s).\n\nA call through an injected dependency lands on the interface's method node,\nbecause that is what the call site names: `_report.Build()` where `_report` is\nan `IReport` resolves to `IReport.Build()`. The implementing method is a\nseparate node, and nothing joins the two, so a directed walk stops at the\ninterface and every chain through an injected dependency is cut at that point.\n\nThis resolver runs after all files are extracted, with the merged corpus\navailable, and links the interface's method to every implementing method:\n\n ireport_ireport_build --dispatches_to--> report_report_build\n ireport_ireport_build --dispatches_to--> alternatereport_alternatereport_build\n\nFor a static structural graph the question a call through the interface asks\nis \"what can this reach\", not \"which one does it reach at runtime\" — with N\nimplementers the answer is all N, not none. Each implementer is still subject\nto the single-owner guard `resolve_pascal_inherited_calls` and\n`resolve_ruby_member_calls` already use: an implementer that itself declares\nmore than one method of the same name is ambiguous and is skipped on its own,\nindependently of its siblings, unless the overloads pair up one-to-one by\nsignature or parameter count (C#, #4584).\n\nThe join is per method rather than per call site, so one edge per implementer\nreconnects every call that reaches the interface method. Confidence is\n`INFERRED` and falls as the candidate set grows, on the extraction rubric's own\ntiers: 0.85 (\"strong inference\") for a single implementer — the target is\nforced, the source text just never names it — 0.75 for two, and a 0.65 floor\n(\"weak inference\") from three on, rather than inventing finer-grained\nprecision the rubric does not have.\n\nThe mechanism knows no language. Each language registers a thin wrapper\n(`graphify.csharp_dispatch` for C# interfaces, #3003) that fixes the file\nsuffixes both ends of a pair must be declared in: `implements` is resolved by\nname, so a same-named type from another language is a collision, not an\nimplementation. A language whose interfaces have the same shape (Swift\nprotocols, for one) adds a wrapper, not a second copy of the algorithm.\n\"\"\"\nfrom __future__ import annotations\n\nfrom graphify.rcfile import effective_suffix\n\nDISPATCH_RELATION = \"dispatches_to\"\n\n# Confidence falls as the implementer count grows, snapped to the extraction\n# rubric's discrete INFERRED tiers (0.95/0.85/0.75/0.65/0.55) rather than a\n# continuous function of count, which the rubric has no basis for (#2813\n# forbids inventing precision a formula does not have).\n_DISPATCH_CONFIDENCE_BY_IMPLEMENTER_COUNT = {1: 0.85, 2: 0.75}\n_DISPATCH_CONFIDENCE_FLOOR = 0.65\n\n\ndef _dispatch_confidence(implementer_count: int) -> float:\n \"\"\"INFERRED confidence for a dispatch edge, given how many implementers exist.\n\n One implementer is forced — \"strong inference\". Three or more floors at\n \"weak inference\" instead of continuing to fall per extra implementer.\n \"\"\"\n return _DISPATCH_CONFIDENCE_BY_IMPLEMENTER_COUNT.get(implementer_count, _DISPATCH_CONFIDENCE_FLOOR)\n\n\ndef declared_in(node: dict | None, suffixes: tuple[str, ...]) -> bool:\n \"\"\"True when the node is a declaration that lives in a file with one of ``suffixes``.\n\n Every end of a dispatch pair has to pass this. A sourceless stub minted for\n a dangling reference carries no members worth dispatching to, and a\n cross-language pair is a name collision rather than an implementation. The\n suffix is the effective one, so a `.graphifyrc` language declaration counts\n (#2961).\n \"\"\"\n if not node:\n return False\n source_file = node.get(\"source_file\")\n return bool(source_file) and effective_suffix(str(source_file)) in suffixes\n\n\ndef method_label(node: dict) -> str:\n \"\"\"Return a method node's bare name, for matching.\n\n Case is kept: C# and Swift are case sensitive, and an implementing member\n must spell the interface member exactly, so folding case could only pair a\n declaration with a member that does not implement it.\n\n The name is cut at the first parenthesis rather than by stripping a trailing\n `()`. Today the extractors label every method `.Name()`, so the two are the\n same, but the pair match is keyed on this string and a label that ever carried\n a signature would silently stop matching instead of failing visibly.\n \"\"\"\n label = str(node.get(\"label\", \"\")).strip().removeprefix(\".\")\n return label.split(\"(\", 1)[0]\n\n\ndef resolve_interface_dispatch(\n per_file: list[dict],\n all_nodes: list[dict],\n all_edges: list[dict],\n *,\n suffixes: tuple[str, ...],\n) -> None:\n \"\"\"Link each interface method to every implementation that owns it unambiguously.\n\n Purely additive: the existing call edge to the interface method is left in\n place, since the call site really does name the interface. ``suffixes`` is\n the language gate: the interface and each implementer must both be\n declared in a file with one of these extensions.\n \"\"\"\n if not any(\n declared_in(result, suffixes)\n for result in per_file\n if isinstance(result, dict)\n ) and not any(\n declared_in(n, suffixes) for n in all_nodes\n ):\n return\n\n node_by_id = {n.get(\"id\"): n for n in all_nodes}\n\n implementers: dict[str, set[str]] = {}\n methods_of: dict[str, dict[str, set[str]]] = {}\n for e in all_edges:\n rel = e.get(\"relation\")\n if rel == \"implements\":\n implementers.setdefault(e.get(\"target\"), set()).add(e.get(\"source\"))\n elif rel == \"method\":\n owner, method_nid = e.get(\"source\"), e.get(\"target\")\n mnode = node_by_id.get(method_nid)\n if mnode is None or not declared_in(mnode, suffixes):\n continue\n name = method_label(mnode)\n if name:\n # A set, so the same method arriving on two `method` edges cannot\n # look like two same-named methods and trip the guard below.\n methods_of.setdefault(owner, {}).setdefault(name, set()).add(method_nid)\n\n if not implementers or not methods_of:\n return\n\n # Scoped to this relation: another edge between the two members, whatever it\n # is, says nothing about whether the dispatch link is already there.\n existing_pairs = {\n (e.get(\"source\"), e.get(\"target\"))\n for e in all_edges\n if e.get(\"relation\") == DISPATCH_RELATION\n }\n new_edges: list[dict] = []\n\n for interface_nid, impls in implementers.items():\n interface_node = node_by_id.get(interface_nid)\n if interface_node is None or not declared_in(interface_node, suffixes):\n continue\n interface_methods = methods_of.get(interface_nid, {})\n if not interface_methods:\n continue\n\n # Gate first, then count: an `implements` edge from a sourceless stub\n # (a dangling/external reference) or a cross-language name collision\n # never produces an edge below, so it must not count toward \"how many\n # implementers is this split between\" either — that would understate\n # confidence for the implementers that actually get a dispatch edge.\n gated_impls: list[tuple[str, dict]] = []\n for impl_nid in impls:\n impl_node = node_by_id.get(impl_nid)\n if impl_node is not None and declared_in(impl_node, suffixes):\n gated_impls.append((impl_nid, impl_node))\n if not gated_impls:\n continue\n score = _dispatch_confidence(len(gated_impls))\n\n for impl_nid, impl_node in gated_impls:\n impl_methods = methods_of.get(impl_nid, {})\n for name, declared in interface_methods.items():\n for source, target in _overload_pairs(\n declared, impl_methods.get(name, set()), node_by_id\n ):\n if source == target or (source, target) in existing_pairs:\n continue\n existing_pairs.add((source, target))\n new_edges.append({\n \"source\": source,\n \"target\": target,\n \"relation\": DISPATCH_RELATION,\n \"context\": \"call\",\n \"confidence\": \"INFERRED\",\n \"confidence_score\": score,\n \"source_file\": str(impl_node.get(\"source_file\", \"\")),\n \"source_location\": impl_node.get(\"source_location\"),\n \"weight\": 1.0,\n })\n\n all_edges.extend(new_edges)\n\n\ndef _overload_pairs(\n declared: set[str], candidates: set[str], node_by_id: dict\n) -> list[tuple[str, str]]:\n \"\"\"Pair an interface's methods of one name with an implementer's.\n\n One of each pairs up as always. Otherwise (C# overloads, #4584) each\n interface method pairs with the one implementer method of the same declared\n signature, else of the same parameter count among those no signature\n claimed; no unique match, no edge. An implementer method takes at most one\n interface method, so an overload implemented on a base class is not pinned\n on a sibling of the same count.\n \"\"\"\n if len(declared) == 1 and len(candidates) == 1:\n return [(next(iter(declared)), next(iter(candidates)))]\n\n def _meta(nid: str) -> dict:\n meta = (node_by_id.get(nid) or {}).get(\"metadata\")\n return meta if isinstance(meta, dict) else {}\n\n pairs: dict[str, str] = {}\n for key in (\"signature\", \"arity\"):\n taken = set(pairs.values())\n for source in sorted(declared):\n want = _meta(source).get(key)\n if source in pairs or want is None:\n continue\n hits = [c for c in candidates if c not in taken and _meta(c).get(key) == want]\n if len(hits) == 1:\n pairs[source] = hits[0]\n # Two interface methods on one implementer method: neither is certain.\n targets = list(pairs.values())\n return [(s, t) for s, t in pairs.items() if targets.count(t) == 1]\n","content_type":"text/x-python; charset=utf-8","language":"python","size":10188,"content_sha256":"ecbb05cb0b8cc9f3fa46ad79928a349dcfbddd89a6d77f13bf03e05338bfd879"},{"filename":"manifest_ingest.py","content":"\"\"\"Deterministic package-manifest ingestion (#1377).\n\nPackage manifests (``apm.yml``, ``pyproject.toml``, ``Cargo.toml``, ``go.mod``,\n``pom.xml``) declare a package and its dependencies. Left to the LLM document path, the same\npackage gets a different file-anchored node id from its own manifest than from\neach dependent's dependency reference, so it splits into duplicate nodes. This\nmodule parses manifests deterministically and emits ONE canonical package node\nper package -- keyed by NAME via :func:`graphify.ids.make_id` -- plus\n``depends_on`` edges, so a package referenced from N manifests collapses to a\nsingle hub node (the dependency stub and the package's own definition node share\nthe canonical id and merge at build time).\n\nMirrors ``mcp_ingest``: recognized by filename, routed to the deterministic AST\npath (never the LLM), so a manifest is extracted exactly once.\n\"\"\"\nfrom __future__ import annotations\n\nimport re\nimport xml.etree.ElementTree as ET\nfrom pathlib import Path\nfrom typing import Any\n\nfrom graphify.ids import make_id\n\n__all__ = [\"is_package_manifest_path\", \"extract_package_manifest\", \"PACKAGE_MANIFEST_NAMES\"]\n\n# manifest filename (lowercased) -> ecosystem tag\nPACKAGE_MANIFEST_NAMES: dict[str, str] = {\n \"apm.yml\": \"apm\",\n \"apm.yaml\": \"apm\",\n \"pyproject.toml\": \"python\",\n \"cargo.toml\": \"cargo\",\n \"go.mod\": \"go\",\n \"pom.xml\": \"maven\",\n}\n\n_MAX_MANIFEST_BYTES = 2_000_000 # 2 MB cap — manifests are small; this rejects junk\n\n_TOMLI_REQUIRED = (\n \"Package-manifest ingestion on Python \u003c 3.11 needs tomli. \"\n \"Install with: pip install 'tomli' \"\n \"(or reinstall graphifyy, which declares tomli for python_version \u003c '3.11').\"\n)\n\n\ndef _load_toml_module():\n \"\"\"Return a tomllib-compatible module, or raise ImportError (#3283).\n\n Returning ``None`` used to look identical to a virtual workspace root with\n nothing to emit, so missing ``tomli`` on Python 3.10 silently dropped every\n ``Cargo.toml`` / ``pyproject.toml``. Raise instead: the caller\n (``extract_package_manifest``) surfaces this as a visible per-manifest error\n rather than dropping the file silently. In practice the runtime ``tomli``\n dependency (python_version \u003c '3.11') keeps this path unreachable for a\n standard install.\n \"\"\"\n try:\n import tomllib as _toml # type: ignore[import-not-found]\n return _toml\n except ImportError:\n try:\n import tomli as _toml # type: ignore[import-not-found,no-redef]\n return _toml\n except ImportError as exc:\n raise ImportError(_TOMLI_REQUIRED) from exc\n\n\ndef is_package_manifest_path(path: Path) -> bool:\n \"\"\"True if ``path`` is a recognized package manifest (by filename).\"\"\"\n return path.name.lower() in PACKAGE_MANIFEST_NAMES\n\n\ndef _pkg_id(name: str) -> str:\n \"\"\"Canonical package node id, keyed by package NAME so every reference to the\n same package -- its own manifest and any dependent's dependency line -- maps\n to one node.\"\"\"\n return make_id(\"pkg\", name)\n\n\ndef extract_package_manifest(path: Path) -> dict[str, Any]:\n \"\"\"Parse a package manifest into a canonical package node + ``depends_on`` edges.\"\"\"\n try:\n if path.stat().st_size > _MAX_MANIFEST_BYTES:\n return {\"nodes\": [], \"edges\": [], \"error\": \"manifest too large to index\"}\n text = path.read_text(encoding=\"utf-8\", errors=\"replace\")\n except OSError as exc:\n return {\"nodes\": [], \"edges\": [], \"error\": f\"manifest read error: {exc}\"}\n\n eco = PACKAGE_MANIFEST_NAMES[path.name.lower()]\n try:\n info = _PARSERS[eco](text)\n except Exception as exc: # noqa: BLE001 — a malformed manifest must not abort extraction\n return {\"nodes\": [], \"edges\": [], \"error\": f\"manifest parse error: {exc}\"}\n if isinstance(info, dict) and info.get(\"skipped\"):\n return {\"nodes\": [], \"edges\": [], \"skipped\": info[\"skipped\"]}\n if not info or not info.get(\"name\"):\n return {\"nodes\": [], \"edges\": []}\n\n name = info[\"name\"]\n str_path = str(path)\n pkg_nid = _pkg_id(name)\n node: dict[str, Any] = {\n \"id\": pkg_nid,\n \"label\": name,\n \"file_type\": \"code\", # valid schema type; `type` distinguishes packages\n \"type\": \"package\",\n \"ecosystem\": eco,\n \"source_file\": str_path,\n \"source_location\": \"L1\",\n }\n if info.get(\"version\"):\n node[\"version\"] = info[\"version\"]\n nodes: list[dict] = [node]\n edges: list[dict] = []\n\n seen: set[str] = set()\n for dep in info.get(\"deps\", []):\n if not dep:\n continue\n dep_nid = _pkg_id(dep)\n if dep_nid == pkg_nid or dep_nid in seen:\n continue\n seen.add(dep_nid)\n # The edge targets the dependency's canonical package id. If that package's\n # own manifest is in the corpus, the edge resolves to its (single) node; if\n # the dependency is external, build_from_json prunes the dangling edge. We\n # deliberately do NOT emit a stub node — a stub with an empty source_file\n # would risk clobbering the real node's source_file under id-dedup.\n edges.append({\n \"source\": pkg_nid,\n \"target\": dep_nid,\n \"relation\": \"depends_on\",\n \"context\": \"dependency\",\n \"confidence\": \"EXTRACTED\",\n \"confidence_score\": 1.0,\n \"source_file\": str_path,\n \"source_location\": \"L1\",\n \"weight\": 1.0,\n })\n return {\"nodes\": nodes, \"edges\": edges}\n\n\n# ── per-ecosystem parsers: text -> {\"name\", \"version\"?, \"deps\": [str]} | None ──\n\ndef _coerce_deps(value: Any) -> list[str]:\n \"\"\"A dependency block may be a list of names or a name->spec map.\"\"\"\n if isinstance(value, dict):\n return [str(k) for k in value]\n if isinstance(value, list):\n out: list[str] = []\n for item in value:\n if isinstance(item, str):\n out.append(item)\n elif isinstance(item, dict) and item:\n out.append(str(next(iter(item))))\n return out\n return []\n\n\ndef _parse_apm(text: str) -> dict | None:\n try:\n import yaml\n except ImportError:\n return _parse_apm_fallback(text)\n data = yaml.safe_load(text)\n if not isinstance(data, dict):\n return None\n return {\n \"name\": data.get(\"name\"),\n \"version\": data.get(\"version\"),\n \"deps\": _coerce_deps(data.get(\"dependencies\")),\n }\n\n\ndef _parse_apm_fallback(text: str) -> dict | None:\n \"\"\"Minimal line parser for apm.yml when PyYAML is unavailable: a top-level\n ``name:``/``version:`` plus a simple ``dependencies:`` block (list items or\n a name map).\"\"\"\n name = None\n version = None\n deps: list[str] = []\n in_deps = False\n for line in text.splitlines():\n if not in_deps:\n m = re.match(r'^name:\\s*[\"\\']?([^\"\\'\\s#]+)', line)\n if m:\n name = m.group(1)\n continue\n # `version` is part of the manifest contract the YAML path already\n # returns; dropping it here made a package node lose its version\n # on every machine without PyYAML installed.\n m = re.match(r'^version:\\s*[\"\\']?([^\"\\'\\s#]+)', line)\n if m:\n version = m.group(1)\n continue\n if re.match(r'^dependencies:\\s*

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

, line):\n in_deps = True\n continue\n if in_deps:\n dm = (re.match(r'^\\s*-\\s*[\"\\']?([^\"\\'\\s#:]+)', line)\n or re.match(r'^\\s{2,}([A-Za-z0-9._/@-]+)\\s*:', line))\n if dm:\n deps.append(dm.group(1))\n elif re.match(r'^\\S', line): # next top-level key ends the block\n in_deps = False\n return {\"name\": name, \"version\": version, \"deps\": deps} if name else None\n\n\ndef _pep508_name(spec: str) -> str:\n \"\"\"`requests>=2.0` -> `requests`; `pkg[extra]==1; python_version\u003c'3.9'` -> `pkg`.\"\"\"\n return re.split(r'[\\s\u003c>=!~;\\[\\(]', spec.strip(), maxsplit=1)[0]\n\n\ndef _parse_pyproject(text: str) -> dict | None:\n _toml = _load_toml_module()\n data = _toml.loads(text)\n proj = data.get(\"project\", {}) if isinstance(data.get(\"project\"), dict) else {}\n poetry = (data.get(\"tool\", {}) or {}).get(\"poetry\", {}) if isinstance(data.get(\"tool\"), dict) else {}\n name = proj.get(\"name\") or (poetry.get(\"name\") if isinstance(poetry, dict) else None)\n if not name:\n return None\n deps: list[str] = [_pep508_name(s) for s in (proj.get(\"dependencies\") or []) if isinstance(s, str)]\n if isinstance(poetry, dict):\n for dep in (poetry.get(\"dependencies\") or {}):\n if str(dep).lower() != \"python\":\n deps.append(str(dep))\n return {\"name\": name, \"version\": proj.get(\"version\") or (poetry.get(\"version\") if isinstance(poetry, dict) else None), \"deps\": deps}\n\n\ndef _parse_cargo(text: str) -> dict | None:\n \"\"\"Cargo.toml: name/version from ``[package]``, runtime deps from\n ``[dependencies]`` plus every ``[target.\u003ccfg>.dependencies]`` table (mirrors\n ``_parse_pyproject``'s runtime-only scope; dev-/build-dependencies excluded).\"\"\"\n _toml = _load_toml_module()\n data = _toml.loads(text)\n pkg = data.get(\"package\", {}) if isinstance(data.get(\"package\"), dict) else {}\n name = pkg.get(\"name\")\n # A virtual workspace root (``[workspace]``, no ``[package]``) declares no\n # package of its own — emit nothing rather than a fabricated node. ``name`` is\n # never workspace-inheritable in Cargo, but guard on the type anyway.\n if not isinstance(name, str) or not name:\n if isinstance(data.get(\"workspace\"), dict):\n return {\"name\": None, \"skipped\": \"virtual workspace root Cargo.toml\"}\n return None\n # ``version`` may be workspace-inherited (``version.workspace = true``), which\n # parses to a table; keep only a concrete string version.\n version = pkg.get(\"version\")\n if not isinstance(version, str):\n version = None\n # A dependency value is a bare version string or an inline table; either way\n # _coerce_deps keys it by the dependency NAME (the table/map key).\n deps = _coerce_deps(data.get(\"dependencies\"))\n # Platform-conditional deps live under ``[target.\u003ccfg>.dependencies]``; fold\n # them in so a crate whose deps are entirely cfg-gated still emits its edges.\n targets = data.get(\"target\")\n if isinstance(targets, dict):\n for cfg in targets.values():\n if isinstance(cfg, dict):\n deps += _coerce_deps(cfg.get(\"dependencies\"))\n return {\"name\": name, \"version\": version, \"deps\": deps}\n\n\ndef _parse_gomod(text: str) -> dict | None:\n name = None\n deps: list[str] = []\n in_block = False\n for line in text.splitlines():\n s = line.strip()\n if name is None:\n m = re.match(r'^module\\s+(\\S+)', s)\n if m:\n name = m.group(1)\n continue\n if re.match(r'^require\\s*\\(', s):\n in_block = True\n continue\n if in_block:\n if s.startswith(')'):\n in_block = False\n continue\n dm = re.match(r'^(\\S+)\\s+v\\S+', s)\n if dm:\n deps.append(dm.group(1))\n else:\n dm = re.match(r'^require\\s+(\\S+)\\s+v\\S+', s)\n if dm:\n deps.append(dm.group(1))\n return {\"name\": name, \"version\": None, \"deps\": deps} if name else None\n\n\ndef _parse_pom(text: str) -> dict | None:\n # Drop the default namespace so findtext/findall don't need the {uri} prefix.\n text = re.sub(r'\\sxmlns=\"[^\"]*\"', '', text, count=1)\n root = ET.fromstring(text)\n aid = root.findtext(\"artifactId\")\n gid = root.findtext(\"groupId\") or root.findtext(\"parent/groupId\")\n version = root.findtext(\"version\") or root.findtext(\"parent/version\")\n if not aid:\n return None\n props: dict[str, str] = {}\n for prop in root.findall(\"properties/*\"):\n if isinstance(prop.tag, str) and prop.text:\n props[prop.tag] = prop.text.strip()\n for key, value in (\n (\"project.groupId\", gid),\n (\"project.artifactId\", aid),\n (\"project.version\", version),\n (\"project.parent.groupId\", root.findtext(\"parent/groupId\")),\n (\"project.parent.version\", root.findtext(\"parent/version\")),\n ):\n if value:\n props.setdefault(key, value.strip())\n\n def _resolve(value: str | None) -> str | None:\n if not value:\n return value\n return re.sub(r\"\\$\\{([^}]+)\\}\", lambda m: props.get(m.group(1), m.group(0)), value.strip())\n\n gid = _resolve(gid)\n version = _resolve(version)\n name = f\"{gid}:{aid}\" if gid else aid\n deps: list[str] = []\n # Only the project's own \u003cdependencies> (and those of its profiles) are\n # dependencies. \u003cdependencyManagement> only pins versions, and a plugin's\n # \u003cdependencies> belong to the build tool.\n for dep in root.findall(\"dependencies/dependency\") + root.findall(\"profiles/profile/dependencies/dependency\"):\n da = _resolve(dep.findtext(\"artifactId\"))\n dg = _resolve(dep.findtext(\"groupId\"))\n if da:\n deps.append(f\"{dg}:{da}\" if dg else da)\n return {\"name\": name, \"version\": version, \"deps\": deps}\n\n\n_PARSERS = {\n \"apm\": _parse_apm,\n \"python\": _parse_pyproject,\n \"cargo\": _parse_cargo,\n \"go\": _parse_gomod,\n \"maven\": _parse_pom,\n}\n","content_type":"text/x-python; charset=utf-8","language":"python","size":13425,"content_sha256":"3f646619085ac44238b92e62385969a18930462982c627116d3c2907e0644b65"},{"filename":"manifest.py","content":"# re-export manifest helpers from detect for backwards compatibility\nfrom graphify.detect import save_manifest, load_manifest, detect_incremental\n\n__all__ = [\"save_manifest\", \"load_manifest\", \"detect_incremental\"]\n","content_type":"text/x-python; charset=utf-8","language":"python","size":214,"content_sha256":"c865a2c577d4bf618d19d512d1f687ea0c961b8c09879b40d414536f184f7da2"},{"filename":"markdown_resolution.py","content":"\"\"\"Cross-file resolution of code-span mentions in Markdown documents.\n\n``extract_markdown`` reports every inline code span that reads as a symbol\nname (``Widget``, ``render()``, ``pkg.Widget``, ``src/mod.py::Widget::render``)\nas a ``raw_calls`` entry tagged ``language: \"markdown\"`` instead of guessing a\ntarget id: the symbol is defined in some other file, and only once every file\nis extracted and the id-remap passes have run can the mention be matched to\nthe node that actually exists. Riding the ``raw_calls`` channel means those\npasses rewrite the mention's ``caller_nid`` alongside every other raw call,\nand the extraction cache round-trips it for free.\n\nThis resolver runs from the language-resolver registry (see\n``resolver_registry``) with the merged corpus in hand and turns each mention\ninto a ``references`` edge from the heading (or page) that cites the symbol\nto the code node that defines it:\n\n* a path-qualified mention resolves against the cited file only. The file is\n matched by its path relative to the document, then as a segment-aligned\n suffix of a scanned file, and must be unique; the symbol chain is walked\n through ``contains`` / ``method`` edges so ``mod.py::Widget::render`` is\n the ``render`` owned by ``Widget``. The document names the file and the\n symbol verbatim, so the edge is EXTRACTED;\n* a bare mention resolves only when exactly one callable or type in the\n corpus carries that label (case-sensitive; language built-ins excluded).\n Only nodes the extractors marked ``_callable`` qualify, for the reason the\n indirect-call pass gives (#1566): a by-name match must land on a real\n function, method or class, never on a same-named data symbol such as a\n JSON key. A dotted mention (``pkg.Widget``, ``Widget.render``) keeps its\n qualifiers, and a candidate survives only when every qualifier is a label\n on its ``contains`` / ``method`` owner chain or a segment (or stem) of its\n source path: ``time.sleep`` never lands on a repo's own ``sleep``, and\n ``Widget.render`` picks the ``render`` that ``Widget`` owns. No other\n tie-breaking: an ambiguous name is a guess, and a guess is not an edge.\n A bare, parenthesis-less method name is held to one more test: a word like\n ``redis`` is prose as often as it is a reference, so a method match survives\n only when the citing document also names the class that owns it (#4609).\n The trailing ``()``, a dotted form, or a class/function label carries enough\n evidence on its own.\n The match is INFERRED (0.95, a named cross-file reference).\n\nAn explicit relative cite (``./`` or ``../``) resolves against the document's\ndirectory only; it never falls through to the suffix rule, so a path that\nescapes the corpus cannot land on an unrelated copy of the file elsewhere.\n\nThe shared cross-file call pass in ``extract`` skips ``markdown`` raw calls,\nso a mention never surfaces as a ``calls`` edge.\n\"\"\"\nfrom __future__ import annotations\n\nimport os\nimport re\nfrom typing import Any\n\nfrom graphify.extractors.base import _LANGUAGE_BUILTIN_GLOBALS\n\nMARKDOWN_MENTION_SUFFIXES = frozenset({\".md\", \".mdx\", \".qmd\", \".skill\"})\n\n_CONTAINMENT_RELATIONS = frozenset({\"contains\", \"method\"})\n\n#: Leading ``./`` and ``../`` segments of a cited path, stripped as whole\n#: segments (``lstrip(\"./\")`` would eat the dot of ``.github/...``).\n_RELATIVE_PREFIX_RE = re.compile(r\"^(\\.\\.?/)+\")\n\n\ndef _symbol_label(node: dict[str, Any]) -> str:\n \"\"\"A node's label as a document would write it: no ``()``, no leading dot.\"\"\"\n return str(node.get(\"label\", \"\")).strip(\"()\").lstrip(\".\")\n\n\ndef _is_file_node(node: dict[str, Any]) -> bool:\n label = str(node.get(\"label\", \"\"))\n source_file = str(node.get(\"source_file\", \"\"))\n return bool(source_file) and label == os.path.basename(source_file)\n\n\ndef _posix(path: str) -> str:\n return path.replace(\"\\\\\", \"/\")\n\n\ndef _match_cited_file(cited: str, doc_file: str, source_files: set[str]) -> str | None:\n \"\"\"The scanned source file a ``path::Name`` mention names, or None.\n\n Tries the path relative to the citing document first (``../src/mod.py``\n from ``docs/guide.md``), then an exact match, then a unique\n segment-aligned suffix (``src/mod.py`` naming ``/repo/src/mod.py``). A\n cite that is explicitly relative (``./x`` or ``../x``) stops after the\n first step: it names one location, and a suffix match elsewhere would be a\n different file.\n \"\"\"\n cited_posix = _posix(cited)\n doc_dir = os.path.dirname(doc_file)\n relative = _posix(os.path.normpath(os.path.join(doc_dir, cited_posix)))\n if relative in source_files:\n return relative\n if cited_posix.startswith((\"./\", \"../\")):\n return None\n if cited_posix in source_files:\n return cited_posix\n stripped = _RELATIVE_PREFIX_RE.sub(\"\", cited_posix)\n matches = [sf for sf in source_files\n if sf == stripped or sf.endswith(\"/\" + stripped)]\n return matches[0] if len(matches) == 1 else None\n\n\ndef _evidence(node_id: str, nodes_by_id: dict[str, dict[str, Any]],\n parents: dict[str, set[str]]) -> set[str]:\n \"\"\"Labels a dotted mention may qualify ``node_id`` with.\n\n The labels of every node on its ``contains`` / ``method`` owner chain\n (``Widget`` for ``Widget.render``) plus each segment and stem of its\n source path (``pkg`` and ``mod`` for ``pkg.mod.Widget``).\n \"\"\"\n evidence: set[str] = set()\n frontier = {node_id}\n seen: set[str] = set()\n while frontier:\n nid = frontier.pop()\n if nid in seen:\n continue\n seen.add(nid)\n owners = parents.get(nid, set())\n for owner in owners:\n node = nodes_by_id.get(owner)\n if node is not None:\n evidence.add(_symbol_label(node))\n frontier |= owners\n node = nodes_by_id.get(node_id, {})\n for segment in _posix(str(node.get(\"source_file\", \"\"))).split(\"/\"):\n if segment:\n evidence.add(segment)\n evidence.add(os.path.splitext(segment)[0])\n return evidence\n\n\ndef _markdown_raw_calls(per_file: list[dict]) -> list[dict]:\n calls: list[dict] = []\n for result in per_file:\n if not isinstance(result, dict):\n continue\n for rc in result.get(\"raw_calls\", []) or []:\n if isinstance(rc, dict) and rc.get(\"language\") == \"markdown\":\n calls.append(rc)\n return calls\n\n\ndef resolve_markdown_mentions(\n per_file: list[dict],\n all_nodes: list[dict],\n all_edges: list[dict],\n) -> None:\n \"\"\"Turn ``language: \"markdown\"`` raw calls into ``references`` edges in place.\"\"\"\n mentions = _markdown_raw_calls(per_file)\n if not mentions:\n return\n\n code_nodes = [\n n for n in all_nodes\n if n.get(\"id\") and n.get(\"file_type\") == \"code\" and not _is_file_node(n)\n ]\n by_label: dict[str, list[str]] = {}\n by_file: dict[str, dict[str, list[str]]] = {}\n for n in code_nodes:\n label = _symbol_label(n)\n if not label:\n continue\n if n.get(\"_callable\"):\n by_label.setdefault(label, []).append(n[\"id\"])\n source_file = _posix(str(n.get(\"source_file\", \"\")))\n if source_file:\n by_file.setdefault(source_file, {}).setdefault(label, []).append(n[\"id\"])\n source_files = set(by_file)\n\n parents: dict[str, set[str]] = {}\n for e in all_edges:\n if e.get(\"relation\") in _CONTAINMENT_RELATIONS:\n parents.setdefault(str(e.get(\"target\")), set()).add(str(e.get(\"source\")))\n nodes_by_id = {n[\"id\"]: n for n in all_nodes if n.get(\"id\")}\n node_ids = set(nodes_by_id)\n existing = {\n (e.get(\"source\"), e.get(\"target\"))\n for e in all_edges if e.get(\"relation\") == \"references\"\n }\n\n def _resolve_chain(file_labels: dict[str, list[str]], names: list[str]) -> str | None:\n candidates = list(file_labels.get(names[-1], []))\n for depth, owner_name in enumerate(reversed(names[:-1]), start=1):\n owners = set(file_labels.get(owner_name, []))\n narrowed = []\n for c in candidates:\n chain = {c}\n for _ in range(depth):\n chain = {p for nid in chain for p in parents.get(nid, ())}\n if chain & owners:\n narrowed.append(c)\n candidates = narrowed\n return candidates[0] if len(candidates) == 1 else None\n\n def _target(rc: dict) -> tuple[str | None, str, float]:\n \"\"\"The node a mention resolves to, plus the confidence of that match.\"\"\"\n names = list(rc.get(\"qualifiers\") or []) + [str(rc.get(\"callee\", \"\")).strip()]\n cited = rc.get(\"path\")\n if cited:\n source_file = _match_cited_file(\n str(cited), _posix(str(rc.get(\"source_file\", \"\"))), source_files\n )\n if source_file is None:\n return None, \"EXTRACTED\", 1.0\n return _resolve_chain(by_file[source_file], names), \"EXTRACTED\", 1.0\n callee = names[-1]\n if callee in _LANGUAGE_BUILTIN_GLOBALS:\n return None, \"INFERRED\", 0.95\n qualifiers = set(names[:-1])\n candidates = [\n c for c in by_label.get(callee, [])\n if qualifiers \u003c= _evidence(c, nodes_by_id, parents)\n ]\n return (candidates[0] if len(candidates) == 1 else None), \"INFERRED\", 0.95\n\n def _method_owner(node_id: str) -> str | None:\n \"\"\"The class that owns *node_id*, when it is a method and not a function.\"\"\"\n for parent in parents.get(node_id, ()):\n node = nodes_by_id.get(parent)\n if node is not None and node.get(\"_callable_class\"):\n return parent\n return None\n\n #: Targets already cited by each document, used as evidence in pass 2b.\n cited_targets: dict[str, set[str]] = {}\n\n def _add(rc: dict, target: str | None, confidence: str, score: float) -> None:\n caller = str(rc.get(\"caller_nid\", \"\"))\n if target is None or target == caller:\n return\n doc = _posix(str(rc.get(\"source_file\", \"\")))\n if (caller, target) in existing:\n # The edge is already in the graph, but this document still CITES that\n # target, so it stays on record as owner evidence for a later method\n # mention in the same document (#4609 review finding).\n cited_targets.setdefault(doc, set()).add(target)\n return\n existing.add((caller, target))\n all_edges.append({\n \"source\": caller,\n \"target\": target,\n \"relation\": \"references\",\n \"context\": rc.get(\"context\", \"\"),\n \"confidence\": confidence,\n \"confidence_score\": score,\n \"source_file\": rc.get(\"source_file\", \"\"),\n \"source_location\": rc.get(\"source_location\"),\n \"weight\": 1.0,\n })\n cited_targets.setdefault(doc, set()).add(target)\n\n # Pass 1: explicit evidence resolves on its own — a cited path, a dotted\n # qualifier chain, or a trailing (). Record those targets per document.\n # Pass 2 handles the bare, parenthesis-less, single-name spans, where the\n # word alone (`redis`, `data`, `message`) is indistinguishable from prose.\n bare: list[dict] = []\n for rc in mentions:\n caller = str(rc.get(\"caller_nid\", \"\"))\n callee = str(rc.get(\"callee\", \"\")).strip()\n if not caller or not callee or caller not in node_ids:\n continue\n names = list(rc.get(\"qualifiers\") or []) + [callee]\n span = str(rc.get(\"context\") or \"\")\n if rc.get(\"path\") or len(names) > 1 or \"(\" in span:\n _add(rc, *_target(rc))\n else:\n bare.append(rc)\n\n # Pass 2a: bare mentions that do not name a method. A class or free function\n # label is specific enough to keep matching, and the class a later method\n # mention needs must be on record before pass 2b runs.\n method_mentions: list[tuple[dict, str, str, float]] = []\n for rc in bare:\n target, confidence, score = _target(rc)\n if target is not None and _method_owner(target) is not None:\n method_mentions.append((rc, target, confidence, score))\n else:\n _add(rc, target, confidence, score)\n\n # Pass 2b: a bare method name (#4609) is only evidence when the same\n # document already cites the class that owns it. `redis` the server does not\n # link to `Cache.redis` unless the document names `Cache`;\n # `successWithWarnings` survives because `HandlerResult` is cited next to it.\n for rc, target, confidence, score in method_mentions:\n doc = _posix(str(rc.get(\"source_file\", \"\")))\n if _method_owner(target) in cited_targets.get(doc, ()):\n _add(rc, target, confidence, score)\n","content_type":"text/x-python; charset=utf-8","language":"python","size":12757,"content_sha256":"6a7e6f54c183a9522191d624cd05dbc273162fb484a8f5abe3b1ed583fe4db46"},{"filename":"mcp_ingest.py","content":"\"\"\"mcp_ingest.py — Extract MCP (Model Context Protocol) server configuration files.\n\nReads `.mcp.json` / `claude_desktop_config.json` / `mcp.json` / `mcp_servers.json`\nand turns the `mcpServers` map into Graphify nodes and edges.\n\nSymmetry with `serve.py`: Graphify exposes itself AS an MCP server. This module\nindexes MCP servers AS a corpus type, completing the loop — an agent that runs\ngraphify with `--mcp` can now query its own configured MCP layer.\n\nEntry point:\n extract_mcp_config(path: Path) -> dict[str, list[dict]]\n\n Returns `{\"nodes\": [...], \"edges\": [...]}` compatible with Graphify's\n extraction-result format. Returns `{\"nodes\": [...], \"edges\": [...], \"error\": \"...\"}`\n when the file is malformed, too large, or has no `mcpServers` map — the empty\n result keeps it indistinguishable from \"no MCP config here\" for downstream\n callers.\n\nDetected filenames (case-sensitive, matched on basename):\n - .mcp.json (Claude Code project config)\n - claude_desktop_config.json (Claude Desktop)\n - mcp.json (generic / per-tool)\n - mcp_servers.json (alternate naming)\n\nSchema emitted:\n Node kinds:\n - file the config file itself (label = filename)\n - mcp_server one per entry under mcpServers\n - mcp_command executable (npx, uvx, node, python, ...) — global ID\n - mcp_package npm / pypi package id parsed from args — global ID\n - env_var env variable NAME only — global ID. VALUES ARE NEVER READ.\n\n Edge relations:\n - contains file -> mcp_server\n - references mcp_server -> mcp_command\n - references mcp_server -> mcp_package\n - requires_env mcp_server -> env_var (new relation; distinguishes\n env dependencies from generic refs)\n\nSecurity:\n - Env var VALUES are never read, persisted, labelled, or surfaced. Only env\n var NAMES become nodes. (`env: {\"API_KEY\": \"sk-...\"}` -> node \"API_KEY\" only.)\n - File size capped at 1 MiB (matches extract_json).\n - All labels go through `sanitize_label` (control characters stripped, length\n capped) before emission.\n - Args are NOT persisted as nodes/edges to avoid leaking paths or secrets that\n some servers embed as positional args.\n\nCross-config emergent edges:\n Because `mcp_command`, `mcp_package`, and `env_var` nodes use global IDs (no\n per-file stem prefix), the same package or env var across two MCP configs\n produces shared nodes — naturally surfacing \"what configs depend on this\n thing?\" via graph traversal. Server nodes ARE stem-scoped so two configs\n declaring different servers under the same key (e.g., both have \"filesystem\")\n do not collide.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport json\nimport re\nimport unicodedata\nfrom pathlib import Path\nfrom typing import Any\n\nfrom graphify.ids import make_id as _shared_make_id\nfrom graphify.security import sanitize_label\n\n\nMCP_CONFIG_FILENAMES: frozenset[str] = frozenset({\n \".mcp.json\",\n \"claude_desktop_config.json\",\n \"mcp.json\",\n \"mcp_servers.json\",\n})\n\n_MAX_BYTES = 1_048_576 # 1 MiB — same cap as extract_json\n_MAX_SERVERS_PER_FILE = 200 # generous; flags pathological configs\n\n\ndef is_mcp_config_path(path: Path) -> bool:\n \"\"\"Return True when ``path`` is a recognised MCP config filename.\"\"\"\n return path.name in MCP_CONFIG_FILENAMES\n\n\ndef extract_mcp_config(path: Path) -> dict[str, Any]:\n \"\"\"Parse an MCP config file into Graphify nodes and edges.\n\n Behaviour matches other extractors in `extract.py`:\n - returns ``{\"nodes\": [...], \"edges\": [...]}`` on success\n - returns ``{\"nodes\": [], \"edges\": [], \"error\": \"\u003creason>\"}`` on parse\n failure, oversize file, or missing ``mcpServers`` map\n \"\"\"\n try:\n with path.open(\"rb\") as fh:\n raw = fh.read(_MAX_BYTES + 1)\n except OSError as exc:\n return {\"nodes\": [], \"edges\": [], \"error\": f\"mcp_ingest read error: {exc}\"}\n\n if len(raw) > _MAX_BYTES:\n return {\"nodes\": [], \"edges\": [], \"error\": \"mcp config too large to index\"}\n\n try:\n text = raw.decode(\"utf-8\")\n except UnicodeDecodeError as exc:\n return {\"nodes\": [], \"edges\": [], \"error\": f\"mcp_ingest decode error: {exc}\"}\n\n try:\n doc = json.loads(text)\n except json.JSONDecodeError as exc:\n return {\"nodes\": [], \"edges\": [], \"error\": f\"mcp_ingest json error: {exc}\"}\n\n if not isinstance(doc, dict):\n return {\"nodes\": [], \"edges\": [], \"error\": \"mcp_ingest: root is not an object\"}\n\n servers = doc.get(\"mcpServers\")\n if not isinstance(servers, dict):\n # Some tools nest the map (e.g., {\"mcp\": {\"servers\": {...}}}). Try one\n # well-known alternate shape but do not search exhaustively.\n nested = doc.get(\"mcp\")\n if isinstance(nested, dict):\n servers = nested.get(\"servers\")\n if not isinstance(servers, dict):\n return {\"nodes\": [], \"edges\": [], \"error\": \"mcp_ingest: no mcpServers map\"}\n\n str_path = str(path)\n file_nid = _make_id(str_path)\n nodes: list[dict[str, Any]] = []\n edges: list[dict[str, Any]] = []\n seen_node_ids: set[str] = set()\n seen_edge_keys: set[tuple[str, str, str]] = set()\n\n _add_node(\n nodes, seen_node_ids,\n nid=file_nid,\n label=path.name,\n kind=\"mcp_config_file\",\n source_file=str_path,\n line=1,\n )\n\n file_stem = _file_stem(path)\n server_count = 0\n for server_name, spec in servers.items():\n if not isinstance(server_name, str) or not server_name:\n continue\n if not isinstance(spec, dict):\n # Skip non-object server entries silently — the broken entry is\n # the user's, not ours.\n continue\n if server_count >= _MAX_SERVERS_PER_FILE:\n break\n server_count += 1\n _emit_server(\n server_name=server_name,\n spec=spec,\n file_nid=file_nid,\n file_stem=file_stem,\n source_file=str_path,\n nodes=nodes,\n edges=edges,\n seen_node_ids=seen_node_ids,\n seen_edge_keys=seen_edge_keys,\n )\n\n return {\"nodes\": nodes, \"edges\": edges}\n\n\ndef _emit_server(\n *,\n server_name: str,\n spec: dict[str, Any],\n file_nid: str,\n file_stem: str,\n source_file: str,\n nodes: list[dict[str, Any]],\n edges: list[dict[str, Any]],\n seen_node_ids: set[str],\n seen_edge_keys: set[tuple[str, str, str]],\n) -> None:\n \"\"\"Emit nodes/edges for one entry under ``mcpServers``.\"\"\"\n server_nid = _make_id(file_stem, \"mcp_server\", server_name)\n _add_node(\n nodes, seen_node_ids,\n nid=server_nid,\n label=server_name,\n kind=\"mcp_server\",\n source_file=source_file,\n line=1, # JSON doesn't expose line numbers without a parser pass\n )\n _add_edge(\n edges, seen_edge_keys,\n source=file_nid,\n target=server_nid,\n relation=\"contains\",\n source_file=source_file,\n line=1,\n )\n\n command = spec.get(\"command\")\n if isinstance(command, str) and command.strip():\n cmd_label = command.strip()\n cmd_nid = _make_id(\"mcp_command\", cmd_label)\n _add_node(\n nodes, seen_node_ids,\n nid=cmd_nid,\n label=cmd_label,\n kind=\"mcp_command\",\n source_file=source_file,\n line=1,\n )\n _add_edge(\n edges, seen_edge_keys,\n source=server_nid,\n target=cmd_nid,\n relation=\"references\",\n source_file=source_file,\n line=1,\n context=\"command\",\n )\n\n args = spec.get(\"args\")\n if isinstance(args, list):\n package = _detect_package_from_args(args)\n if package:\n pkg_nid = _make_id(\"mcp_package\", package)\n _add_node(\n nodes, seen_node_ids,\n nid=pkg_nid,\n label=package,\n kind=\"mcp_package\",\n source_file=source_file,\n line=1,\n )\n _add_edge(\n edges, seen_edge_keys,\n source=server_nid,\n target=pkg_nid,\n relation=\"references\",\n source_file=source_file,\n line=1,\n context=\"package\",\n )\n\n env = spec.get(\"env\")\n if isinstance(env, dict):\n # ONLY KEYS. Values may contain secrets and are never read here.\n for env_name in env.keys():\n if not isinstance(env_name, str) or not env_name:\n continue\n env_nid = _make_id(\"env_var\", env_name)\n _add_node(\n nodes, seen_node_ids,\n nid=env_nid,\n label=env_name,\n kind=\"env_var\",\n source_file=source_file,\n line=1,\n )\n _add_edge(\n edges, seen_edge_keys,\n source=server_nid,\n target=env_nid,\n relation=\"requires_env\",\n source_file=source_file,\n line=1,\n )\n\n\n# ── Package detection from args ───────────────────────────────────────────────\n\n# Patterns observed in real MCP server configs:\n# [\"-y\", \"@modelcontextprotocol/server-filesystem\", \"/data\"] (npx)\n# [\"-y\", \"@org/[email protected]\"]\n# [\"mcp-server-fetch\"] (uvx / python)\n# [\"mcp-server-time\", \"--local-timezone=UTC\"]\n# [\"@scoped/some-mcp\"] (pnpx)\n# [\"mcp-server-fetch\"] (uvx direct)\n_NPM_PKG_RE = re.compile(r\"^@[a-z0-9][a-z0-9._-]*/[a-z0-9][a-z0-9._-]*(?:@[\\w.\\-+]+)?$\")\n_PY_MCP_PKG_RE = re.compile(r\"^[a-z0-9][a-z0-9._-]*-mcp(?:-[a-z0-9._-]+)?$|^mcp-[a-z0-9][a-z0-9._-]*$\")\n_ARG_FLAG_RE = re.compile(r\"^-{1,2}\\w\")\n\n\ndef _detect_package_from_args(args: list[Any]) -> str | None:\n \"\"\"Return the first arg that looks like an npm or pypi package id, else None.\n\n Skips short flags (-y, --yes) and option arguments (--local-timezone=UTC).\n \"\"\"\n for raw in args:\n if not isinstance(raw, str):\n continue\n arg = raw.strip()\n if not arg or _ARG_FLAG_RE.match(arg):\n continue\n if _NPM_PKG_RE.match(arg):\n return _strip_version(arg)\n if _PY_MCP_PKG_RE.match(arg):\n return arg\n return None\n\n\ndef _strip_version(pkg: str) -> str:\n \"\"\"Drop the ``@version`` suffix from an npm package id, preserving the scope.\n\n Scoped: ``@scope/name`` or ``@scope/[email protected]`` — there are at most two\n ``@`` chars; the second is the version separator.\n Unscoped: ``name`` or ``[email protected]``.\n \"\"\"\n if pkg.startswith(\"@\"):\n version_at = pkg.find(\"@\", 1)\n return pkg if version_at == -1 else pkg[:version_at]\n version_at = pkg.find(\"@\")\n return pkg if version_at == -1 else pkg[:version_at]\n\n\n# ── Node / edge construction (Graphify schema) ────────────────────────────────\n\n\ndef _add_node(\n nodes: list[dict[str, Any]],\n seen: set[str],\n *,\n nid: str,\n label: str,\n kind: str,\n source_file: str,\n line: int,\n) -> None:\n \"\"\"Append a node if not already present. ``kind`` is metadata, not file_type.\"\"\"\n if not nid or nid in seen:\n return\n seen.add(nid)\n nodes.append({\n \"id\": nid,\n \"label\": sanitize_label(label),\n \"file_type\": \"code\",\n \"source_file\": source_file,\n \"source_location\": f\"L{line}\",\n \"metadata\": {\"mcp_kind\": kind},\n })\n\n\ndef _add_edge(\n edges: list[dict[str, Any]],\n seen: set[tuple[str, str, str]],\n *,\n source: str,\n target: str,\n relation: str,\n source_file: str,\n line: int,\n context: str | None = None,\n) -> None:\n \"\"\"Append an edge if (source, target, relation) is not already present.\"\"\"\n if not source or not target or source == target:\n return\n key = (source, target, relation)\n if key in seen:\n return\n seen.add(key)\n edge: dict[str, Any] = {\n \"source\": source,\n \"target\": target,\n \"relation\": relation,\n \"confidence\": \"EXTRACTED\",\n \"confidence_score\": 1.0,\n \"source_file\": source_file,\n \"source_location\": f\"L{line}\",\n \"weight\": 1.0,\n }\n if context:\n edge[\"context\"] = context\n edges.append(edge)\n\n\n# ── ID helpers (kept local; mirror extract.py shape) ──────────────────────────\n\n\ndef _make_id(*parts: str) -> str:\n \"\"\"Build a stable node ID via the single shared recipe (#1378).\"\"\"\n return _shared_make_id(*parts)\n\n\n# Canonical recipe imported directly (no import cycle: extractors.base imports\n# only graphify.ids), so this can no longer drift from extract._file_stem.\nfrom graphify.extractors.base import _file_stem # noqa: E402\n","content_type":"text/x-python; charset=utf-8","language":"python","size":13184,"content_sha256":"7553845a7cae7c310803bf37d992b695b27d21e1b827c1c33dbed2b15971be61"},{"filename":"multigraph_compat.py","content":"\"\"\"Runtime compatibility probe for Graphify MultiDiGraph mode.\n\nVerifies that the current NetworkX runtime supports the behaviors a future\nopt-in --multigraph build will rely on. The probe is BEHAVIOR-based, not\nversion-based — both NX 3.4.2 (Py 3.10 lane) and NX 3.6.1+ (Py 3.11+ lane)\npass. The probe result is cached for the process lifetime via lru_cache.\n\nNo call sites added yet; downstream multigraph PRs will gate on\nrequire_multigraph_capabilities() before enabling MDG mode.\n\"\"\"\n\nfrom __future__ import annotations\n\nfrom collections.abc import Callable\nfrom dataclasses import dataclass\nfrom functools import lru_cache\nimport sys\nfrom typing import Any\n\nimport networkx as nx\nfrom networkx.readwrite import json_graph\n\n\n@dataclass(frozen=True)\nclass CapabilityCheck:\n name: str\n ok: bool\n detail: str\n\n\n@dataclass(frozen=True)\nclass MultigraphCapabilityResult:\n python_version: str\n networkx_version: str\n checks: tuple[CapabilityCheck, ...]\n\n @property\n def ok(self) -> bool:\n return all(check.ok for check in self.checks)\n\n @property\n def failed(self) -> tuple[CapabilityCheck, ...]:\n return tuple(check for check in self.checks if not check.ok)\n\n def error_message(self) -> str:\n if self.ok:\n return (\n \"Graphify MultiDiGraph capability probe passed \"\n f\"(Python {self.python_version}, NetworkX {self.networkx_version}).\"\n )\n failed = \"; \".join(f\"{check.name}: {check.detail}\" for check in self.failed)\n return (\n \"error: --multigraph requires NetworkX keyed MultiDiGraph node-link \"\n \"round-trip support. \"\n f\"Detected Python {self.python_version}, NetworkX {self.networkx_version}. \"\n f\"Failed capability check(s): {failed}. \"\n \"Default simple graph mode remains available.\"\n )\n\n\ndef _check(name: str, func: Callable[[], bool | str]) -> CapabilityCheck:\n try:\n detail = func()\n except Exception as exc:\n return CapabilityCheck(name, False, f\"{type(exc).__name__}: {exc}\")\n if detail is True:\n return CapabilityCheck(name, True, \"ok\")\n if isinstance(detail, str):\n return CapabilityCheck(name, False, detail)\n return CapabilityCheck(name, False, f\"unexpected result {detail!r}\")\n\n\ndef _build_probe_graph() -> nx.MultiDiGraph:\n graph = nx.MultiDiGraph()\n graph.add_node(\"a\", label=\"A\")\n graph.add_node(\"b\", label=\"B\")\n graph.add_edge(\"a\", \"b\", key=\"calls:a.py:L1\", relation=\"calls\", source_file=\"a.py\")\n graph.add_edge(\"a\", \"b\", key=\"imports:a.py:L2\", relation=\"imports\", source_file=\"a.py\")\n return graph\n\n\ndef _probe_keyed_parallel_edges() -> bool | str:\n graph = _build_probe_graph()\n if not graph.is_multigraph() or not graph.is_directed():\n return f\"probe graph type was {type(graph).__name__}\"\n if graph.number_of_edges(\"a\", \"b\") != 2:\n return f\"expected 2 keyed parallel edges, got {graph.number_of_edges('a', 'b')}\"\n keys = set(graph[\"a\"][\"b\"].keys())\n expected = {\"calls:a.py:L1\", \"imports:a.py:L2\"}\n if keys != expected:\n return f\"expected keys {sorted(expected)}, got {sorted(keys)}\"\n return True\n\n\ndef _probe_node_link_round_trip() -> bool | str:\n graph = _build_probe_graph()\n data = json_graph.node_link_data(graph, edges=\"links\")\n if data.get(\"multigraph\") is not True:\n return f\"serialized multigraph flag was {data.get('multigraph')!r}\"\n if data.get(\"directed\") is not True:\n return f\"serialized directed flag was {data.get('directed')!r}\"\n links = data.get(\"links\")\n if not isinstance(links, list) or len(links) != 2:\n length = 0 if not isinstance(links, list) else len(links)\n return f\"serialized links length was {length}\"\n serialized_keys: set[str] = set()\n for edge in links:\n if isinstance(edge, dict):\n edge_key = edge.get(\"key\")\n if isinstance(edge_key, str):\n serialized_keys.add(edge_key)\n expected = {\"calls:a.py:L1\", \"imports:a.py:L2\"}\n if serialized_keys != expected:\n return f\"serialized keys {sorted(serialized_keys)} did not match {sorted(expected)}\"\n loaded = json_graph.node_link_graph(data, edges=\"links\")\n if not isinstance(loaded, nx.MultiDiGraph):\n return f\"round-trip graph type was {type(loaded).__name__}\"\n if loaded.number_of_edges(\"a\", \"b\") != 2:\n return f\"round-trip edge count was {loaded.number_of_edges('a', 'b')}\"\n loaded_keys = set(loaded[\"a\"][\"b\"].keys())\n if loaded_keys != expected:\n return f\"round-trip keys {sorted(loaded_keys)} did not match {sorted(expected)}\"\n return True\n\n\ndef _probe_duplicate_key_overwrite_semantics() -> bool | str:\n graph = nx.MultiDiGraph()\n graph.add_edge(\"x\", \"y\", key=\"same\", marker=\"first\")\n graph.add_edge(\"x\", \"y\", key=\"same\", marker=\"second\")\n edges = list(graph.edges(keys=True, data=True))\n if len(edges) != 1:\n return f\"expected one edge after duplicate-key add, got {len(edges)}\"\n if edges[0][3].get(\"marker\") != \"second\":\n return f\"expected second attr overwrite, got {edges[0][3].get('marker')!r}\"\n return True\n\n\ndef _probe_reserved_key_attr_rejected() -> bool | str:\n \"\"\"Verify the Python language guarantee that NetworkX add_edge inherits.\n\n Python forbids passing the same keyword argument twice — once explicitly\n and once via **kwargs. This probe confirms that protection still applies\n to nx.MultiDiGraph.add_edge: a future loader that builds attrs from JSON\n will be reliably protected from accidentally setting `key` via attrs while\n also passing `key=` explicitly.\n\n The probe always passes on any Python 3.x version. Its purpose is to\n document the invariant explicitly in the probe suite so that if a future\n Python version relaxes this rule (extremely unlikely), the probe surfaces\n the regression.\n \"\"\"\n graph = nx.MultiDiGraph()\n attrs: dict[str, Any] = {\"key\": \"attr-key\", \"relation\": \"calls\"}\n try:\n graph.add_edge(\"a\", \"b\", key=\"schema-key\", **attrs)\n except TypeError:\n return True\n return \"add_edge accepted duplicate key keyword and attr; loader must not rely on this\"\n\n\ndef _probe_remove_edges_from_two_tuple_semantics() -> bool | str:\n graph = nx.MultiDiGraph()\n graph.add_edge(\"a\", \"b\", key=\"one\")\n graph.add_edge(\"a\", \"b\", key=\"two\")\n graph.remove_edges_from([(\"a\", \"b\")])\n remaining = graph.number_of_edges(\"a\", \"b\")\n if remaining != 1:\n return f\"expected one remaining edge after two-tuple removal, got {remaining}\"\n return True\n\n\ndef _probe_to_undirected_preserves_multigraph_type() -> bool | str:\n graph = _build_probe_graph()\n undirected = graph.to_undirected()\n undirected_view = graph.to_undirected(as_view=True)\n if not isinstance(undirected, nx.MultiGraph):\n return f\"to_undirected() returned {type(undirected).__name__}\"\n if not isinstance(undirected_view, nx.MultiGraph):\n return f\"to_undirected(as_view=True) returned {type(undirected_view).__name__}\"\n return True\n\n\n@lru_cache(maxsize=1)\ndef probe_multigraph_capabilities() -> MultigraphCapabilityResult:\n checks = (\n _check(\"keyed_parallel_edges\", _probe_keyed_parallel_edges),\n _check(\"node_link_edges_links_round_trip\", _probe_node_link_round_trip),\n _check(\"duplicate_key_overwrite_semantics\", _probe_duplicate_key_overwrite_semantics),\n _check(\"reserved_key_attr_rejected\", _probe_reserved_key_attr_rejected),\n _check(\n \"remove_edges_from_two_tuple_semantics\",\n _probe_remove_edges_from_two_tuple_semantics,\n ),\n _check(\n \"to_undirected_preserves_multigraph_type\",\n _probe_to_undirected_preserves_multigraph_type,\n ),\n )\n return MultigraphCapabilityResult(\n python_version=(\n f\"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}\"\n ),\n networkx_version=nx.__version__,\n checks=checks,\n )\n\n\ndef require_multigraph_capabilities() -> MultigraphCapabilityResult:\n result = probe_multigraph_capabilities()\n if not result.ok:\n raise RuntimeError(result.error_message())\n return result\n","content_type":"text/x-python; charset=utf-8","language":"python","size":8262,"content_sha256":"73abcd5293c91f2584ea29539774a126fe99a783e3672887d69b5d70033c798b"},{"filename":"pascal_resolution.py","content":"\"\"\"Cross-file resolution for Pascal/Delphi calls to inherited methods.\n\nThe per-file Pascal/Delphi extractors (``extract_pascal``,\n``_extract_pascal_regex``) resolve calls to a method on the caller's own\nclass, its ancestor chain, or a file-level free function -- but only within\nthe single file being extracted (each file is extracted independently; see\n``_resolve_pascal_callee_factory`` in ``extract.py``). Real Delphi/MTM-style\ncodebases very commonly split a class across two files -- a code-generator\nbase class (e.g. Sistec's ``Th0Xxx``) and a manual descendant that extends it\nin a separate unit (``Th5Xxx``) -- so a call from the manual descendant to a\nmethod it inherits from the generated base falls outside any one file's own\nscope. The per-file pass reports these as ``raw_calls`` instead of guessing.\n\nThis resolver runs after all files are extracted (registered in\n``graphify.resolver_registry``), with the full merged node/edge corpus\navailable, so it can walk an ``inherits`` chain across file boundaries. It\nintentionally does NOT fall back to a global by-name match the way the\nper-file pass's final tier does -- an unqualified call resolving to a\nspecific ancestor mirrors Delphi's actual method-lookup semantics (nearest\nancestor in the chain), so walking `inherits` is a structurally justified\nresolution, not a heuristic guess; guessing by name across an entire\nmulti-thousand-file corpus is not the same bet.\n\"\"\"\nfrom __future__ import annotations\n\n_PASCAL_SUFFIXES = (\".pas\", \".pp\", \".dpr\", \".dpk\", \".inc\")\n\n\ndef _pascal_raw_calls(per_file: list[dict]) -> list[dict]:\n calls: list[dict] = []\n for result in per_file:\n if not isinstance(result, dict):\n continue\n for rc in result.get(\"raw_calls\", []):\n if not isinstance(rc, dict):\n continue\n if str(rc.get(\"source_file\", \"\")).endswith(_PASCAL_SUFFIXES):\n calls.append(rc)\n return calls\n\n\ndef resolve_pascal_inherited_calls(\n per_file: list[dict],\n all_nodes: list[dict],\n all_edges: list[dict],\n) -> None:\n \"\"\"Resolve Pascal/Delphi calls to a method inherited across file boundaries.\n\n Purely additive: only emits edges for raw calls the per-file pass could\n not resolve locally (see module docstring). Each emission requires a\n single owning class at the nearest matching level of the caller's\n ``inherits`` chain (god-node guard, same principle as\n ``resolve_ruby_member_calls``) -- an ambiguous or unresolved name\n produces no edge rather than a guess.\n \"\"\"\n node_by_id = {n.get(\"id\"): n for n in all_nodes}\n\n class_bases: dict[str, list[str]] = {}\n # method_nid -> its owning class nid, so a raw call's caller_nid (a method\n # or free-function nid) can be mapped to the CLASS whose inherits chain\n # should be walked. Derived from `all_edges` (already remapped/finalized\n # by the id-disambiguation passes that run before resolvers, same as\n # caller_nid itself) rather than carried as a separate field on the raw\n # call -- a field the generic id-remap machinery would not know to update.\n owner_of: dict[str, str] = {}\n class_procs: dict[str, dict[str, list[str]]] = {}\n for e in all_edges:\n if e.get(\"relation\") == \"inherits\":\n class_bases.setdefault(e[\"source\"], []).append(e[\"target\"])\n elif e.get(\"relation\") == \"method\":\n owner, method_nid = e.get(\"source\"), e.get(\"target\")\n owner_of[method_nid] = owner\n mnode = node_by_id.get(method_nid)\n if mnode is None:\n continue\n name_lower = str(mnode.get(\"label\", \"\")).removesuffix(\"()\").lower()\n # Count DISTINCT methods, not edge multiplicity: the tree-sitter\n # Pascal extractor emits one `method` edge for the interface\n # declaration and one for the implementation, so the same\n # method_nid arrives twice. Deduping keeps the single-owner\n # god-node guard below (`len(candidates) == 1`) measuring real\n # same-name collisions across classes, not the same method\n # double-counted -- otherwise every inherited call looks ambiguous.\n bucket = class_procs.setdefault(owner, {}).setdefault(name_lower, [])\n if method_nid not in bucket:\n bucket.append(method_nid)\n\n existing_pairs = {(e.get(\"source\"), e.get(\"target\")) for e in all_edges}\n\n def _resolve(owner: str, name_lower: str) -> str | None:\n seen_bases: set[str] = set()\n queue = list(class_bases.get(owner, []))\n while queue:\n base = queue.pop(0)\n if base in seen_bases:\n continue\n seen_bases.add(base)\n candidates = class_procs.get(base, {}).get(name_lower)\n if candidates:\n return candidates[0] if len(candidates) == 1 else None\n queue.extend(class_bases.get(base, []))\n return None\n\n for rc in _pascal_raw_calls(per_file):\n caller = rc.get(\"caller_nid\")\n name_lower = rc.get(\"callee\")\n if not caller or not name_lower:\n continue\n owner = owner_of.get(caller)\n if not owner:\n continue\n target = _resolve(str(owner), str(name_lower))\n if not target or target == caller:\n continue\n pair = (caller, target)\n if pair in existing_pairs:\n continue\n existing_pairs.add(pair)\n all_edges.append({\n \"source\": caller,\n \"target\": target,\n \"relation\": \"calls\",\n \"context\": \"call\",\n \"confidence\": \"EXTRACTED\",\n \"confidence_score\": 1.0,\n \"source_file\": rc.get(\"source_file\", \"\"),\n \"source_location\": rc.get(\"source_location\"),\n \"weight\": 1.0,\n })\n","content_type":"text/x-python; charset=utf-8","language":"python","size":5843,"content_sha256":"e4e9400ee20e6b410e521ad24bfe4c7e09742590ba1b0964cc84b49158d9318f"},{"filename":"paths.py","content":"\"\"\"Single source of truth for the graphify output-directory name.\n\nThe output directory is ``graphify-out`` by default and overridable with the\n``GRAPHIFY_OUT`` env var (worktrees or shared-output setups, #686). It accepts a\nrelative name (``\"graphify-out-feature\"``) or an absolute path\n(``\"/shared/graphify-out\"``).\n\nThis used to be duplicated as an identical ``_GRAPHIFY_OUT`` constant in\n``__main__``, ``cache``, and ``watch``, while ``security`` and ``callflow_html``\nhardcoded the literal ``\"graphify-out\"`` and silently ignored the override\n(#1423). Centralising it here keeps the name in one place. The value is read\nonce at import time, matching the previous per-module constants — set\n``GRAPHIFY_OUT`` before the process starts (the normal worktree/shared-output\nflow) and every reader honours it.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport errno\nimport functools\nimport json\nimport os\nimport re\nimport stat\nimport sys\nimport tempfile\nfrom pathlib import Path, PurePosixPath, PureWindowsPath\n\n\ndef _out_from_env() -> str:\n \"\"\"``GRAPHIFY_OUT``, or ``\"graphify-out\"`` when it is unset, empty, or names\n the project itself / a parent of it.\n\n ``os.environ.get(key, default)`` only applies the default when the variable\n is missing, so ``GRAPHIFY_OUT=`` (a blank ``.env`` entry) used to yield\n ``\"\"``: output landed in the project root and ``uninstall --purge`` deleted\n the whole project. ``.`` and ``..`` are never a valid output dir either.\n \"\"\"\n value = (os.environ.get(\"GRAPHIFY_OUT\") or \"\").strip()\n if not value:\n return \"graphify-out\"\n norm = os.path.normpath(value)\n if norm == \".\" or all(part == \"..\" for part in Path(norm).parts):\n print(\n f\"warning: GRAPHIFY_OUT={value!r} points at the project or a parent \"\n \"of it; using 'graphify-out' instead\",\n file=sys.stderr,\n )\n return \"graphify-out\"\n return value\n\n\nGRAPHIFY_OUT = _out_from_env()\n\n\ndef os_long_path(path: \"str | Path\") -> str:\n r\"\"\"Return an OS path string safe for open()/stat() on Windows long paths.\n\n On win32, paths longer than the legacy MAX_PATH (260 chars) are rejected by\n the plain file APIs unless prefixed with the extended-length marker ``\\\\?\\``\n (which also requires a fully-qualified path). Without it, _md5_file /\n save_manifest / count_words silently fail to hash deeply-nested files, so\n their manifest entry never stabilizes and detect_incremental re-flags them\n as changed on every run (#1655), and cache entries under a deep root cannot\n be written at all. cache._normalize_path strips this prefix for stable\n KEYS; this adds it for I/O. Non-win32 and already-prefixed paths pass\n through unchanged.\n \"\"\"\n if sys.platform != \"win32\":\n return str(path)\n s = str(path)\n if s.startswith(\"\\\\\\\\?\\\\\"):\n return s\n try:\n s = os.path.abspath(s) # \\\\?\\ requires a fully-qualified path\n except Exception:\n return str(path)\n if s.startswith(\"\\\\\\\\\"):\n # UNC share \\\\server\\share -> \\\\?\\UNC\\server\\share\n return \"\\\\\\\\?\\\\UNC\\\\\" + s[2:]\n return \"\\\\\\\\?\\\\\" + s\n\n\ndef os_replace_with_fallback(src: \"str | Path\", dst: \"str | Path\") -> None:\n \"\"\"``os.replace(src, dst)``, falling back to a copy for a known set of\n Windows quirks (#3508) that raise even when ``src``/``dst`` are the same\n directory on the same drive: ``PermissionError`` (WinError 5/32 --\n destination briefly locked by another handle, antivirus, an open reader)\n and WinError 17 (\"cannot move to a different disk drive\", observed on some\n Windows/filesystem combinations despite textbook same-volume semantics).\n WinError 17 maps to a plain ``OSError`` in Python, not ``PermissionError``,\n so it's checked via ``winerror`` rather than the exception type. Any other\n failure is a real one and is re-raised.\n\n ``os.replace`` atomically swaps whatever sits at ``dst`` -- including a\n symlink, which it REPLACES in place rather than following (some callers,\n e.g. install.py's managed skill symlinks, #3286, rely on exactly this).\n A naive ``shutil.copy2(src, dst)`` does the opposite when ``dst`` is a\n symlink: opening it for writing follows the link and overwrites its\n TARGET's content instead. So the fallback copies to a fresh temp file in\n ``dst``'s directory first, renames whatever is currently at ``dst`` (link\n or file) aside as a backup rather than deleting it outright, and only\n then renames the temp copy into ``dst``'s place -- matching replace's\n \"whatever was there is gone, a plain file replaces it\" semantics. If that\n final rename fails, the backup is renamed straight back so a mid-swap\n failure leaves the original in place rather than leaving ``dst`` missing.\n \"\"\"\n try:\n os.replace(src, dst)\n return\n except OSError as exc:\n if not isinstance(exc, PermissionError) and getattr(exc, \"winerror\", None) != 17:\n raise\n import shutil\n dst = os.fspath(dst)\n if os.path.normcase(os.path.abspath(os.fspath(src))) == os.path.normcase(os.path.abspath(dst)):\n # Replacing a path with itself needs no swap at all; the rename-aside-\n # then-back sequence below would rename src out from under itself via\n # the \"back up dst\" step and then crash unlinking a path that no\n # longer exists at the end.\n return\n dst_dir = os.path.dirname(dst) or \".\"\n fd, tmp_copy = tempfile.mkstemp(dir=dst_dir, prefix=\".gfy-replace-\", suffix=\".tmp\")\n os.close(fd)\n try:\n shutil.copy2(src, tmp_copy)\n backup = None\n if os.path.lexists(dst):\n bfd, backup = tempfile.mkstemp(dir=dst_dir, prefix=\".gfy-replace-bak-\", suffix=\".tmp\")\n os.close(bfd)\n os.unlink(backup) # reserve the name only; rename needs it free on Windows\n os.rename(dst, backup) # a plain rename moves a symlink itself, never its target\n try:\n os.rename(tmp_copy, dst)\n except BaseException:\n if backup is not None:\n try:\n os.rename(backup, dst)\n except OSError:\n pass # best-effort restore; the swap failure below still propagates\n raise\n if backup is not None:\n try:\n os.unlink(backup)\n except OSError:\n pass\n except BaseException:\n try:\n os.unlink(tmp_copy)\n except OSError:\n pass\n raise\n os.unlink(src)\n\n\ndef _atomic_replace(path: \"str | Path\", write_fn) -> None:\n \"\"\"Atomically replace ``path`` with content written by ``write_fn(f)``.\n\n Writes a temp file in the SAME directory, then ``os.replace``s it into place\n (an atomic rename on one filesystem). A process kill (SIGKILL/Ctrl-C), OOM, or\n ENOSPC mid-write leaves the previous file intact — the destination is\n untouched until the rename. This is NOT a power-loss durability guarantee:\n there is no fsync (matching the rest of the codebase), so an OS/hardware crash\n right after the rename can still expose unflushed bytes on some filesystems.\n The temp file is removed if the write fails.\n\n A symlinked destination is resolved first so the write goes THROUGH the link\n to its target (rather than replacing the link with a regular file), keeping\n the shared-output/worktree symlink setups this module documents working.\n \"\"\"\n # Resolve symlinks so the temp lands on the target's filesystem (same-fs\n # atomic rename) and the replace writes through the link, not over it.\n real = Path(os.path.realpath(str(path)))\n real.parent.mkdir(parents=True, exist_ok=True)\n fd, tmp = tempfile.mkstemp(dir=str(real.parent), prefix=\".gfy-\", suffix=\".tmp\")\n try:\n with os.fdopen(fd, \"w\", encoding=\"utf-8\") as f:\n write_fn(f)\n # mkstemp creates the temp file 0600; match the destination's existing\n # mode (or the umask default for a new file) so an atomic replace never\n # silently tightens a previously group/world-readable output to\n # owner-only. Best-effort — a chmod failure must not fail the write.\n try:\n mode = stat.S_IMODE(os.stat(real).st_mode)\n except OSError:\n umask = os.umask(0)\n os.umask(umask)\n mode = 0o666 & ~umask\n try:\n os.chmod(tmp, mode)\n except OSError:\n pass\n # On Windows os.replace refuses a read-only destination and the fallback\n # would rename it aside and clobber it. Refuse here instead, for this\n # generic write path only: install/cache callers of the fallback must\n # still be able to replace their own (possibly read-only) copies.\n if os.name == \"nt\" and real.is_file() and not os.access(real, os.W_OK):\n raise PermissionError(errno.EACCES, \"destination is read-only\", str(real))\n os_replace_with_fallback(tmp, str(real))\n except BaseException:\n try:\n os.unlink(tmp)\n except OSError:\n # The temp was chmod'd to match the destination above, so when the\n # destination is read-only the temp is too — and Windows refuses to\n # unlink a read-only file. Clear the bit and retry, or every failed\n # write leaks a `.gfy-*.tmp` into the output directory.\n try:\n os.chmod(tmp, stat.S_IWRITE)\n os.unlink(tmp)\n except OSError:\n pass\n raise\n\n\ndef write_text_atomic(path: \"str | Path\", text: str) -> None:\n \"\"\"Atomically write ``text`` (UTF-8) to ``path``. See :func:`_atomic_replace`.\"\"\"\n _atomic_replace(path, lambda f: f.write(text))\n\n\ndef write_text_atomic_if_changed(path: \"str | Path\", text: str) -> bool:\n \"\"\"Atomically write ``text`` only if it differs from what is already on disk;\n return ``True`` if a write happened, ``False`` if the file was left untouched.\n\n Exporters regenerate their whole page set on every ``graph.json`` change and\n used to rewrite every file unconditionally — tens of thousands of identical\n pages per run, which also fires inotify / re-index / sync on unchanged\n content (#3060). Skipping the atomic replace when nothing changed avoids the\n rename entirely, so mtime and inode are preserved.\n\n The comparison is on DECODED text, never bytes: :func:`write_text_atomic`\n writes in text mode and (on Windows) translates ``\\\\n`` to ``\\\\r\\\\n`` while\n :func:`Path.read_text` translates it back, so a byte compare would report\n every file as changed. A missing or non-UTF-8 destination counts as changed.\"\"\"\n try:\n if Path(os.path.realpath(str(path))).read_text(encoding=\"utf-8\") == text:\n return False\n except (OSError, UnicodeDecodeError):\n pass\n write_text_atomic(path, text)\n return True\n\n\ndef write_json_atomic(\n path: \"str | Path\", obj, *, indent: \"int | None\" = None, ensure_ascii: bool = True, compact: bool = False\n) -> None:\n \"\"\"Atomically write ``obj`` as JSON to ``path``, streaming the encode into the\n temp file rather than materializing the whole string first (matters for very\n large graphs). ``ensure_ascii`` mirrors ``json.dump`` so callers that emit raw\n UTF-8 (non-ASCII labels/paths) keep byte-for-byte output. ``compact`` writes one\n line with no padding after ``,``/``:`` (graph.json, #3413). See :func:`_atomic_replace`.\"\"\"\n separators = (\",\", \":\") if compact else None\n _atomic_replace(\n path, lambda f: json.dump(obj, f, indent=indent, ensure_ascii=ensure_ascii, separators=separators)\n )\n\n# Directory segments that, when they appear as a whole path component, mark the\n# whole path as a test location. Matched against path *segments* (not raw\n# substrings) so \"src/contest.py\" / \"latest/x.py\" / \"src/greatest/x.py\" do NOT\n# match — only a segment that *equals* one of these names (case-insensitively).\n_TEST_DIR_SEGMENTS = frozenset({\"tests\", \"test\", \"spec\", \"specs\", \"__tests__\"})\n\n# Filename patterns marking a file as a test, matched against the *filename*\n# only (case-insensitive). These are conventions across ecosystems:\n# test_*.py pytest / unittest\n# *_test.* Go / Python / Rust\n# *.test.* JS/TS (jest, vitest)\n# *.spec.* / *_spec.* Jasmine / RSpec / Karma\n# *.Tests.ps1 PowerShell Pester\n# *Test.java / *Tests.cs (case-sensitive convention, handled below)\n_TEST_FILENAME_PATTERNS = (\n re.compile(r\"^test_.*\", re.IGNORECASE),\n re.compile(r\".*_test\\..+$\", re.IGNORECASE),\n re.compile(r\".*\\.test\\..+$\", re.IGNORECASE),\n re.compile(r\".*\\.spec\\..+$\", re.IGNORECASE),\n re.compile(r\".*_spec\\..+$\", re.IGNORECASE),\n re.compile(r\".*\\.tests\\.ps1$\", re.IGNORECASE),\n # Java `FooTest.java` / `FooTests.java`, C# `FooTests.cs` style. Require an\n # uppercase-led `Test`/`Tests` immediately before the extension so plain\n # words like \"greatest\"/\"contest.cs\" do not match.\n re.compile(r\".*Test\\.java$\"),\n re.compile(r\".*Tests\\.java$\"),\n re.compile(r\".*Tests\\.cs$\"),\n)\n\n\ndef _is_test_path(path: str) -> bool:\n \"\"\"Classify a source path as a test path (case-insensitive, segment-aware).\n\n Shared by extract.py and symbol_resolution.py so cross-file call resolution\n treats test mocks/stubs identically. A path is a test path when:\n * any whole path segment equals a known test dir name\n (``tests``/``test``/``spec``/``specs``/``__tests__``), or\n * the filename matches a known test-file naming convention.\n\n Conservative on purpose: matches segments/filenames, never raw substrings,\n so ``latest.py``, ``src/contest.py`` and ``src/greatest/x.py`` are NON-test.\n \"\"\"\n if not path:\n return False\n # Accept both POSIX and Windows separators regardless of host OS so the\n # classifier is stable across the mixed paths that flow through extraction.\n norm = str(path).replace(\"\\\\\", \"/\")\n pure = PurePosixPath(norm)\n segments = list(pure.parts)\n # Strip a leading drive/anchor segment (e.g. \"C:/\") that PureWindowsPath\n # would surface; with the manual \"\\\\\"->\"/\" swap above PurePosixPath keeps\n # the path body intact, but guard against a Windows drive embedded as a\n # segment just in case.\n for segment in segments:\n if segment.lower() in _TEST_DIR_SEGMENTS:\n return True\n # A drive-letter colon segment like \"c:\" is never a test dir.\n filename = pure.name\n if not filename:\n return False\n for pattern in _TEST_FILENAME_PATTERNS:\n if pattern.match(filename):\n return True\n return False\n\n\[email protected]_cache(maxsize=65536)\ndef _parent_parts(source_file: str) -> tuple[str, ...]:\n \"\"\"Segments of ``source_file``'s parent directory, POSIX-normalized.\n\n `_path_proximity_winner` compares every candidate's directory against the\n call site's, and an ambiguous name (``run``, ``get``) brings the same\n candidate files back on every call site, so the parse is memoized. Equal\n tuples here mean equal ``PurePosixPath(...).parent`` values.\n \"\"\"\n return PurePosixPath(source_file.replace(\"\\\\\", \"/\")).parent.parts\n\n\ndef _path_proximity_winner(call_site_file: str, candidate_files: dict[str, str]) -> str | None:\n \"\"\"Pick the candidate whose source file is closest to the call site.\n\n ``candidate_files`` maps candidate id -> its source_file. Returns a single\n winning candidate id, or ``None`` when no proximity tier yields a unique\n winner. Tiers, in order:\n\n 1. same file as the call site,\n 2. same directory,\n 3. longest common path-prefix (must be a strict, unique maximum).\n\n Used only as a secondary tie-break after the test/non-test filter, so the\n god-node guard still holds when proximity is genuinely ambiguous.\n \"\"\"\n if not call_site_file:\n return None\n call_norm = str(call_site_file).replace(\"\\\\\", \"/\")\n call_parts = _parent_parts(call_norm)\n\n # Tier 1: exact same file.\n same_file = [cid for cid, f in candidate_files.items()\n if str(f).replace(\"\\\\\", \"/\") == call_norm]\n if len(same_file) == 1:\n return same_file[0]\n if len(same_file) > 1:\n return None # genuinely ambiguous within one file; bail\n\n # Tier 2: same directory.\n same_dir = [cid for cid, f in candidate_files.items()\n if _parent_parts(str(f)) == call_parts]\n if len(same_dir) == 1:\n return same_dir[0]\n if len(same_dir) > 1:\n return None\n\n # Tier 3: longest common path-prefix, computed over path segments. The\n # winner must be a strict unique maximum, else we bail (guard holds).\n def _common_prefix_len(f: str) -> int:\n parts = _parent_parts(str(f))\n n = 0\n for a, b in zip(call_parts, parts):\n if a != b:\n break\n n += 1\n return n\n\n scored = sorted(\n ((cid, _common_prefix_len(f)) for cid, f in candidate_files.items()),\n key=lambda kv: kv[1],\n reverse=True,\n )\n if not scored:\n return None\n best = scored[0][1]\n winners = [cid for cid, score in scored if score == best]\n if len(winners) == 1 and best > 0:\n return winners[0]\n return None\n\n\ndef disambiguate_ambiguous_candidates(\n candidates: list[str],\n candidate_files: dict[str, str],\n call_site_file: str,\n) -> str | None:\n \"\"\"Resolve an ambiguous bare-name call to one candidate, or ``None``.\n\n Shared god-node tie-breaker (#1553) used by both the inline cross-file call\n pass in ``extract.py`` and ``symbol_resolution.resolve_cross_file_raw_calls``\n so the heuristics stay aligned across languages. ``candidates`` is the list\n of node ids sharing the callee's name; ``candidate_files`` maps each id ->\n its source_file. Returns the surviving candidate id only when exactly one\n survives; otherwise ``None`` (caller keeps the god-node guard / ``continue``).\n\n Tie-breakers, in order:\n 1. NON-TEST preference. Classify the call site and each candidate as\n test/non-test. When the call site is NON-test, drop test candidates.\n When the call site IS a test file, prefer test-local candidates\n (same file first, then any test candidate); fall back to the full set\n only if no test candidate exists.\n 2. PATH PROXIMITY over whatever survived step 1.\n \"\"\"\n if not candidates:\n return None\n if len(candidates) == 1:\n return candidates[0]\n\n call_is_test = _is_test_path(call_site_file)\n test_cands = [c for c in candidates if _is_test_path(candidate_files.get(c, \"\"))]\n nontest_cands = [c for c in candidates if c not in set(test_cands)]\n\n if call_is_test:\n # Prefer a test-local definition (same file) first.\n call_norm = str(call_site_file).replace(\"\\\\\", \"/\")\n same_file_test = [\n c for c in test_cands\n if str(candidate_files.get(c, \"\")).replace(\"\\\\\", \"/\") == call_norm\n ]\n if len(same_file_test) == 1:\n return same_file_test[0]\n if test_cands:\n survivors = test_cands\n else:\n survivors = nontest_cands or candidates\n else:\n # Non-test call site: drop test mocks/stubs entirely.\n survivors = nontest_cands\n\n if len(survivors) == 1:\n return survivors[0]\n if not survivors:\n return None\n\n # Step 2: path proximity over the survivors.\n return _path_proximity_winner(\n call_site_file,\n {c: candidate_files.get(c, \"\") for c in survivors},\n )\n\n# Bare directory name even when GRAPHIFY_OUT is an absolute path. Used by path\n# guards that walk parents looking for the output directory by name.\nGRAPHIFY_OUT_NAME = os.path.basename(os.path.normpath(GRAPHIFY_OUT))\n\n\ndef out_path(*parts: str) -> Path:\n \"\"\"A path inside the configured output dir, e.g. ``out_path(\"cache\")``.\n\n ``Path(GRAPHIFY_OUT) / ...`` resolves correctly for both a relative name\n (\"graphify-out\") and an absolute override (\"/shared/graphify-out\").\n \"\"\"\n return Path(GRAPHIFY_OUT, *parts)\n\n\ndef default_graph_json() -> str:\n \"\"\"Default ``graph.json`` path under the configured output dir.\n\n The package-wide fallback used by serve/build/benchmark/prs and the CLI read\n commands so a ``GRAPHIFY_OUT`` override is honoured everywhere, not just where\n the path is passed explicitly (#1423).\n \"\"\"\n return str(out_path(\"graph.json\"))\n\n\ndef is_absolute_any_platform(p: \"str | Path | None\") -> bool:\n \"\"\"Whether *p* is absolute under POSIX **or** Windows rules.\n\n ``Path.is_absolute()`` and ``os.path.isabs()`` answer for the HOST os only,\n which is the wrong question for a path that was *stored* — a ``source_file``\n in ``graph.json``, a ``prune_sources`` entry, a cache key. Those travel\n between machines (build in Docker/CI, update on a Windows workstation, or\n the reverse), so the host's rules do not describe the string in hand:\n\n - On Windows, ``WindowsPath(\"/home/ci/repo/docs/a.md\").is_absolute()`` is\n False — no drive letter — so a Linux-built graph's absolute paths read as\n relative and get baked into node IDs or joined under the scan root (#2618).\n - On POSIX, ``PosixPath(\"C:/Users/u/a.md\").is_absolute()`` is False for the\n mirror-image reason (#2197, #1789).\n\n ``os.path.isabs`` is additionally not stable across supported interpreters:\n Python 3.13 changed ``ntpath.isabs`` so a path starting with a single slash\n is no longer absolute, where 3.10–3.12 said it was. The project supports\n >=3.10, so a guard written on it silently means different things per version.\n\n Answering for both platforms is the conservative choice for stored paths:\n treating a path as absolute at worst declines to relativize it (the string is\n kept as-is), whereas treating an absolute path as relative corrupts identity.\n Covers drive-letter, UNC, and POSIX-root forms with either separator.\n\n NOTE: this is for STORED/portable paths. Code resolving a path against the\n real local filesystem (``cli``, ``detect``, ``hooks``) must keep using\n ``Path.is_absolute()`` — there the host's rules are exactly right.\n \"\"\"\n if not p:\n return False\n s = str(p)\n return PurePosixPath(s).is_absolute() or PureWindowsPath(s).is_absolute()\n\n\n# Legacy Windows path ceiling. Unless long-path support is enabled *and* every\n# consumer opts in, the ENTIRE path — drive, directories, filename, and the\n# terminating NUL — must fit in MAX_PATH (260) characters, so the usable budget\n# is 259. POSIX has no equivalent whole-path ceiling in practice; its limit is\n# per-component (NAME_MAX, conventionally 255 bytes).\n_WINDOWS_MAX_PATH = 260\n\n# Floor for the stem budget below. A directory deep enough to push the budget\n# under this cannot host readable filenames anyway; keep enough room for\n# _cap_filename's \"_\" + 8-char digest so a truncated stem stays collision-safe\n# and deterministic rather than degenerating into a bare prefix.\n_MIN_STEM_BUDGET = 16\n\n\ndef stem_filename_budget(output_dir: \"str | Path\", *, reserve: int = 0, limit: int = 200) -> int:\n \"\"\"Largest filename stem an exporter may write directly into ``output_dir``.\n\n Exporters cap note/article filenames so they stay under the filesystem's\n per-component limit (conventionally NAME_MAX=255 bytes, hence the 200\n default). That is the right question on POSIX and the wrong one on Windows,\n where the constraint is on the WHOLE path, not the component: a 200-char\n stem under a perfectly ordinary vault directory such as\n ``C:\\\\Users\\\\me\\\\projects\\\\svc\\\\graphify-out\\\\obsidian`` exceeds MAX_PATH and\n the write dies with ``FileNotFoundError``, aborting the export mid-vault.\n\n Returns ``limit`` unchanged on POSIX, so existing output is byte-for-byte\n stable there. On Windows it returns the smaller of ``limit`` and whatever\n still fits inside MAX_PATH once ``output_dir``, the separator, ``reserve``\n (room for caller-added prefixes/collision suffixes) and the ``.md``\n extension are accounted for.\n\n The budget is a CHARACTER count, but callers that cap UTF-8 BYTES may pass\n it straight through: a string's UTF-8 length is never below its character\n length, so a byte-capped stem always satisfies the character ceiling too.\n \"\"\"\n if os.name != \"nt\":\n return limit\n try:\n base = os.path.abspath(str(output_dir))\n except (OSError, ValueError):\n return limit\n # An extended-length path (\"\\\\?\\C:\\...\", \"\\\\?\\UNC\\...\") opts out of MAX_PATH\n # entirely, so nothing needs shrinking.\n if base.startswith(\"\\\\\\\\?\\\\\"):\n return limit\n budget = (_WINDOWS_MAX_PATH - 1) - len(base) - len(os.sep) - reserve - len(\".md\")\n return max(_MIN_STEM_BUDGET, min(limit, budget))\n\n\ndef nfc(s: str) -> str:\n \"\"\"NFC-normalize a path string.\n\n macOS (HFS+/APFS) reports filenames in NFD while manifests, graph\n ``source_file`` entries and user input are typically NFC. Comparing raw\n strings makes the same file look like two different paths, so any path\n membership test must normalize BOTH sides (#2210, #2221/#2224).\n \"\"\"\n import unicodedata\n return unicodedata.normalize(\"NFC\", s)\n\n\ndef restore_arc_direction(data: dict) -> dict:\n \"\"\"Stamp each link's stored direction as ``_src``/``_tgt`` before loading.\n\n graph.json is written ``directed: false`` but carries true direction in arc\n order (#563), or in ``_src``/``_tgt`` markers on legacy canonicalized files.\n An undirected ``node_link_graph`` load re-orders endpoints by node-list\n position, so readers recover direction from these markers. Existing markers\n win (#2309). Same idiom as the query and merge-graphs loaders (#2261).\n \"\"\"\n links = data.get(\"links\")\n if not isinstance(links, list):\n return data\n return dict(data, links=[\n {**link, \"_src\": link.get(\"_src\", link.get(\"source\")), \"_tgt\": link.get(\"_tgt\", link.get(\"target\"))}\n if isinstance(link, dict) else link\n for link in links\n ])\n\n\ndef load_node_link_graph(path_or_data):\n \"\"\"Load a graphify graph.json into a networkx graph, accepting both writers.\n\n The clustered writer stores edges under ``links`` (networkx's node-link\n default); the raw ``--no-cluster`` writer stores them under ``edges``.\n Consumers that call ``node_link_graph(data, edges=\"links\")`` directly\n raise ``KeyError: 'links'`` on a raw graph (#2212) — the ``except\n TypeError`` fallback only covers old networkx without the ``edges``\n kwarg, not the missing key. Normalize before parsing, same idiom as\n affected.py/serve.py.\n\n Accepts a path (size-cap-checked via the security module, then parsed)\n or an already-parsed dict (no size check — the caller owns any cap).\n \"\"\"\n from networkx.readwrite import json_graph\n data = path_or_data\n if not isinstance(data, dict):\n p = Path(data)\n from graphify.security import check_graph_file_size_cap # lazy: security imports paths\n check_graph_file_size_cap(p)\n data = json.loads(p.read_text(encoding=\"utf-8\"))\n if isinstance(data, dict) and \"links\" not in data and \"edges\" in data:\n data = dict(data, links=data[\"edges\"])\n if isinstance(data, dict):\n data = restore_arc_direction(data)\n try:\n return json_graph.node_link_graph(data, edges=\"links\")\n except TypeError: # networkx too old for the edges kwarg; default is \"links\"\n return json_graph.node_link_graph(data)\n","content_type":"text/x-python; charset=utf-8","language":"python","size":27613,"content_sha256":"84c6897a518d217cb98e66a54244ada1d203b7ee28aaa1ee67e430e8d47d2bb0"},{"filename":"pg_introspect.py","content":"from __future__ import annotations\nfrom pathlib import Path, PurePosixPath\nfrom graphify.extract import extract_sql\n\n\ndef _quote_ident(name: str) -> str:\n \"\"\"Double-quote a PostgreSQL identifier, escaping embedded double-quotes.\"\"\"\n return '\"' + name.replace('\"', '\"\"') + '\"'\n\n\ndef introspect_postgres(dsn: str | None = None) -> dict:\n \"\"\"Connect to PostgreSQL, reconstruct DDL, and extract via extract_sql().\"\"\"\n try:\n import psycopg\n except ModuleNotFoundError:\n raise ImportError(\n \"psycopg is required for --postgres. \"\n \"Install with: pip install 'graphifyy[postgres]'\"\n )\n\n try:\n conn = psycopg.connect(dsn or \"\") # empty string = PG* env vars\n except psycopg.OperationalError as exc:\n # Sanitize: strip the DSN/credentials that psycopg may embed in the\n # OperationalError message (e.g. \"connection to server … failed: …\\nDETAIL: …\")\n msg = str(exc).split(\"\\n\")[0]\n raise ConnectionError(f\"could not connect to PostgreSQL: {msg}\") from None\n\n try:\n conn.execute(\"SET TRANSACTION ISOLATION LEVEL SERIALIZABLE READ ONLY DEFERRABLE\")\n\n # 1. Query tables\n with conn.cursor() as cur:\n cur.execute(\"\"\"\n SELECT table_schema, table_name, table_type\n FROM information_schema.tables\n WHERE table_schema NOT IN ('pg_catalog', 'information_schema')\n ORDER BY table_schema, table_name;\n \"\"\")\n tables = cur.fetchall()\n\n # 2. Query views\n cur.execute(\"\"\"\n SELECT table_schema, table_name, view_definition\n FROM information_schema.views\n WHERE table_schema NOT IN ('pg_catalog', 'information_schema')\n ORDER BY table_schema, table_name;\n \"\"\")\n views = cur.fetchall()\n\n # 3. Query routines (functions/procedures), including language\n cur.execute(\"\"\"\n SELECT routine_schema, routine_name, routine_type,\n routine_definition, external_language\n FROM information_schema.routines\n WHERE routine_schema NOT IN ('pg_catalog', 'information_schema')\n ORDER BY routine_schema, routine_name;\n \"\"\")\n routines = cur.fetchall()\n\n # 4. Query foreign keys — grouped by constraint to handle composites.\n # Read pg_catalog.pg_constraint, NOT information_schema.referential_\n # constraints: that view only shows constraints where the current\n # user has WRITE access to the referencing table (owner or a\n # privilege other than SELECT), so a read-only introspection role\n # sees zero FK rows while tables/views/routines all appear — the\n # graph then silently loses every 'references' edge (#1746).\n # pg_constraint is not privilege-filtered. It also keys constraints\n # by oid rather than by name (constraint names are only unique per\n # table, so the old name-based key_column_usage joins could\n # cross-match same-named constraints on sibling tables).\n cur.execute(\"\"\"\n SELECT\n con.conname AS constraint_name,\n ns.nspname AS table_schema,\n rel.relname AS table_name,\n (SELECT ARRAY_AGG(att.attname ORDER BY k.ord)\n FROM UNNEST(con.conkey) WITH ORDINALITY AS k(attnum, ord)\n JOIN pg_catalog.pg_attribute att\n ON att.attrelid = con.conrelid AND att.attnum = k.attnum\n ) AS columns,\n fns.nspname AS foreign_table_schema,\n frel.relname AS foreign_table_name,\n (SELECT ARRAY_AGG(att.attname ORDER BY k.ord)\n FROM UNNEST(con.confkey) WITH ORDINALITY AS k(attnum, ord)\n JOIN pg_catalog.pg_attribute att\n ON att.attrelid = con.confrelid AND att.attnum = k.attnum\n ) AS foreign_columns\n FROM pg_catalog.pg_constraint con\n JOIN pg_catalog.pg_class rel ON rel.oid = con.conrelid\n JOIN pg_catalog.pg_namespace ns ON ns.oid = rel.relnamespace\n JOIN pg_catalog.pg_class frel ON frel.oid = con.confrelid\n JOIN pg_catalog.pg_namespace fns ON fns.oid = frel.relnamespace\n WHERE con.contype = 'f'\n AND ns.nspname NOT IN ('pg_catalog', 'information_schema')\n ORDER BY ns.nspname, rel.relname, con.conname;\n \"\"\")\n fks = cur.fetchall()\n finally:\n conn.close()\n\n ddl = []\n\n # Tables — quote identifiers to handle reserved words, hyphens, mixed-case\n for schema, name, ttype in tables:\n if ttype == \"BASE TABLE\":\n ddl.append(f\"CREATE TABLE {_quote_ident(schema)}.{_quote_ident(name)} (id INT);\")\n\n # Views — real body if available, stub if NULL (permission denied)\n for schema, name, body in views:\n if body:\n ddl.append(f\"CREATE VIEW {_quote_ident(schema)}.{_quote_ident(name)} AS {body};\")\n else:\n ddl.append(f\"CREATE VIEW {_quote_ident(schema)}.{_quote_ident(name)} AS SELECT 1;\")\n\n # FK edges — one ALTER TABLE per constraint (handles composite FKs correctly).\n # Emitted BEFORE the function DDL: routine bodies the grammar can't parse\n # (notably C-language extension functions, whose \"body\" is just the C symbol\n # name) put tree-sitter into error recovery that consumes the statements\n # after them — with FKs last, every 'references' edge is silently lost on\n # any DB with a common extension installed (#1854). FK statements only\n # reference tables, which are emitted first, so this order is always safe.\n for constraint_name, t_schema, t_name, cols, r_schema, r_name, r_cols in fks:\n col_list = \", \".join(_quote_ident(c) for c in cols)\n ref_col_list = \", \".join(_quote_ident(c) for c in r_cols)\n ddl.append(\n f\"ALTER TABLE {_quote_ident(t_schema)}.{_quote_ident(t_name)} \"\n f\"ADD CONSTRAINT {_quote_ident(constraint_name)} \"\n f\"FOREIGN KEY ({col_list}) REFERENCES {_quote_ident(r_schema)}.{_quote_ident(r_name)}({ref_col_list});\"\n )\n\n # Functions & Procedures — real body if available, stub if NULL\n # Use $gfx$ as the dollar-quote tag to avoid collision with $ inside bodies.\n # Use external_language from the catalog; fall back to plpgsql if NULL/blank.\n for schema, name, rtype, body, ext_lang in routines:\n lang = (ext_lang or \"plpgsql\").lower()\n fn_sig = f\"{_quote_ident(schema)}.{_quote_ident(name)}()\"\n stub_body = \"BEGIN SELECT 1; END;\"\n if rtype in (\"FUNCTION\", \"PROCEDURE\"):\n actual_body = body if body else stub_body\n # Represent PROCEDUREs as FUNCTION so tree-sitter-sql can parse them\n ddl.append(\n f\"CREATE FUNCTION {fn_sig} RETURNS void\"\n f\" AS $gfx$ {actual_body} $gfx$ LANGUAGE {lang};\"\n )\n\n ddl_string = \"\\n\".join(ddl)\n\n # Determine host/dbname for virtual path DSN sanitization\n info = psycopg.conninfo.conninfo_to_dict(dsn or \"\")\n host = info.get(\"host\", \"localhost\")\n dbname = info.get(\"dbname\", \"db\")\n virtual_path = PurePosixPath(f\"postgresql://{host}/{dbname}\")\n\n # Pass virtual path and in-memory DDL content to extract_sql\n result = extract_sql(virtual_path, content=ddl_string)\n # extract_sql() reports a missing grammar as an 'error' key with empty\n # nodes/edges; surface it instead of letting the CLI print\n # \"PostgreSQL: 0 nodes\" as if the database were empty.\n if result.get(\"error\"):\n raise ImportError(\n f\"{result['error']} (required by --postgres; \"\n \"install with: pip install 'graphifyy[postgres]')\"\n )\n return result","content_type":"text/x-python; charset=utf-8","language":"python","size":8036,"content_sha256":"dadb7ca401ff2d03bc5ec50e7c1c394353aff298420bab618ba7b9c674978c63"},{"filename":"pi/package.json","content":"{\n \"name\": \"graphify-pi\",\n \"private\": true,\n \"description\": \"Native Pi (and Oh My Pi) bridge to Graphify's installed hook-guard CLI\",\n \"license\": \"Apache-2.0\",\n \"type\": \"module\",\n \"pi\": {\n \"extensions\": [\"./index.ts\"]\n }\n}\n","content_type":"application/json; charset=utf-8","language":"json","size":233,"content_sha256":"a475ce1e81586d4161db72f00a486fadde2be7a9b122880060cf7a6425ce6a17"},{"filename":"querylog.py","content":"\"\"\"Query logging for graphify — append-only JSONL, fail-silent.\"\"\"\nfrom __future__ import annotations\n\nimport json\nimport os\nimport re\nimport time\nfrom datetime import datetime, timezone\nfrom pathlib import Path\nfrom typing import Any\n\n_NODES_RE = re.compile(r\"(\\d+)\\s+nodes?\\s+found\")\n\n\ndef _log_path() -> Path | None:\n # Opt-in only (#1797). The log records every query/path/explain question and\n # corpus path (and full responses if GRAPHIFY_QUERY_LOG_RESPONSES) in a\n # plaintext file under ~/.cache — outside any repo's .gitignore/retention. A\n # default-on record of proprietary queries contradicts graphify's on-device,\n # no-telemetry posture, so it is OFF unless explicitly enabled:\n # GRAPHIFY_QUERY_LOG=\u003cpath> log to that path, or\n # GRAPHIFY_QUERY_LOG_ENABLE=1 log to ~/.cache/graphify-queries.log.\n # GRAPHIFY_QUERY_LOG_DISABLE=1 still forces it off (back-compat, wins).\n if os.environ.get(\"GRAPHIFY_QUERY_LOG_DISABLE\", \"\").lower() in (\"1\", \"true\", \"yes\"):\n return None\n override = os.environ.get(\"GRAPHIFY_QUERY_LOG\", \"\").strip()\n if override:\n return Path(override).expanduser()\n if os.environ.get(\"GRAPHIFY_QUERY_LOG_ENABLE\", \"\").lower() in (\"1\", \"true\", \"yes\"):\n return Path.home() / \".cache\" / \"graphify-queries.log\"\n return None\n\n\ndef _log_responses() -> bool:\n return os.environ.get(\"GRAPHIFY_QUERY_LOG_RESPONSES\", \"\").lower() in (\"1\", \"true\", \"yes\")\n\n\ndef nodes_from_result(result: str) -> int | None:\n m = _NODES_RE.search(result or \"\")\n return int(m.group(1)) if m else None\n\n\ndef log_query(\n *,\n kind: str,\n question: str,\n corpus: str,\n result: str | None = None,\n nodes_returned: int | None = None,\n duration_ms: float | None = None,\n **extra: Any,\n) -> None:\n \"\"\"Append one JSONL record to the query log. Never raises.\"\"\"\n try:\n path = _log_path()\n if path is None:\n return\n if nodes_returned is None and result is not None:\n nodes_returned = nodes_from_result(result)\n rec: dict[str, Any] = {\n \"ts\": datetime.now(timezone.utc).isoformat(),\n \"kind\": kind,\n \"question\": question,\n \"corpus\": corpus,\n \"nodes_returned\": nodes_returned,\n }\n if result is not None:\n rec[\"result_chars\"] = len(result)\n if duration_ms is not None:\n rec[\"duration_ms\"] = round(duration_ms, 3)\n for k, v in extra.items():\n if v is not None:\n rec[k] = v\n if result is not None and _log_responses():\n rec[\"response\"] = result\n path.parent.mkdir(parents=True, exist_ok=True)\n with path.open(\"a\", encoding=\"utf-8\") as fh:\n fh.write(json.dumps(rec, ensure_ascii=False) + \"\\n\")\n except Exception:\n pass\n","content_type":"text/x-python; charset=utf-8","language":"python","size":2841,"content_sha256":"953ad5fece9d380795a70e7dda4b72069ed7a57f56d8648fd339fea959a150a4"},{"filename":"rcfile.py","content":"\"\"\"Project-level configuration: ``\u003croot>/.graphifyrc``.\n\nOne ``key=value`` per line, ``#`` comments, blank lines ignored. Keys:\n\n``viz_node_limit=\u003cint>``\n Baked into the generated git hooks (see :mod:`graphify.hooks`).\n\n``language.\u003cext>=\u003clanguage | .ext>``\n Treat files with extension ``\u003cext>`` as the named language for\n classification, extractor dispatch and cross-file resolution (#2961).\n ``\u003cext>`` may be written with or without its leading dot; the value is\n either a language name from :data:`LANGUAGE_ALIASES` or an explicit\n extension graphify already knows (``.php``, ``.pas``, ``.sql``, ...)::\n\n # pfSense: every .inc under this repo is PHP, not Pascal\n language.inc=php\n # a repo whose templates are plain TypeScript\n language.tpl=.ts\n\n An ambiguous extension (``.inc`` is PHP, Pascal, SQL or assembly\n depending on the project; ``.m`` is Objective-C or MATLAB; ``.h`` is C or\n C++) has no single correct global mapping, so the project declares it.\n\nThe parser imports :mod:`graphify.detect` lazily only when it must validate\nan explicit extension. This avoids a module-level import cycle while keeping\ntypos from silently disabling classification.\n\"\"\"\nfrom __future__ import annotations\n\nimport sys\nimport threading\nfrom pathlib import Path\n\nRC_FILENAME = \".graphifyrc\"\n\n# Language name -> the canonical extension graphify dispatches it under.\n# Deliberately a short list of names a user is likely to type; any known\n# extension can be given directly instead (``language.inc=.php``).\nLANGUAGE_ALIASES: dict[str, str] = {\n \"php\": \".php\",\n \"pascal\": \".pas\", \"delphi\": \".pas\", \"freepascal\": \".pas\",\n \"sql\": \".sql\",\n \"python\": \".py\",\n \"javascript\": \".js\", \"js\": \".js\",\n \"typescript\": \".ts\", \"ts\": \".ts\",\n \"c\": \".c\",\n \"cpp\": \".cpp\", \"c++\": \".cpp\", \"cxx\": \".cpp\",\n \"objc\": \".mm\", \"objective-c\": \".mm\", \"objectivec\": \".mm\",\n \"java\": \".java\",\n \"kotlin\": \".kt\",\n \"scala\": \".scala\",\n \"groovy\": \".groovy\",\n \"go\": \".go\", \"golang\": \".go\",\n \"rust\": \".rs\",\n \"ruby\": \".rb\",\n \"csharp\": \".cs\", \"c#\": \".cs\",\n \"swift\": \".swift\",\n \"lua\": \".lua\",\n \"zig\": \".zig\",\n \"elixir\": \".ex\",\n \"julia\": \".jl\",\n \"dart\": \".dart\",\n \"r\": \".r\",\n \"fortran\": \".f90\",\n \"shell\": \".sh\", \"bash\": \".sh\", \"sh\": \".sh\",\n \"powershell\": \".ps1\",\n \"verilog\": \".v\", \"systemverilog\": \".sv\",\n \"terraform\": \".tf\", \"hcl\": \".hcl\",\n \"ocaml\": \".ml\",\n \"lisp\": \".lisp\", \"commonlisp\": \".lisp\",\n \"markdown\": \".md\",\n \"json\": \".json\",\n \"yaml\": \".yaml\",\n \"html\": \".html\",\n}\n\n\ndef _normalise_ext(raw: str) -> str:\n ext = raw.strip().lower()\n if not ext.startswith(\".\"):\n ext = \".\" + ext\n return ext\n\n\ndef parse_language_value(value: str) -> str:\n \"\"\"Resolve the right-hand side of ``language.\u003cext>=`` to a canonical suffix.\n\n Accepts a name from :data:`LANGUAGE_ALIASES` (case-insensitive) or an\n explicit dotted extension. Raises ``ValueError`` for anything else.\n \"\"\"\n v = value.strip()\n if not v:\n raise ValueError(\"empty value\")\n if v.startswith(\".\"):\n ext = _normalise_ext(v)\n if len(ext) \u003c 2 or any(ch.isspace() for ch in ext):\n raise ValueError(f\"{value!r} is not an extension\")\n from graphify.detect import CODE_EXTENSIONS, DOC_EXTENSIONS\n if ext not in CODE_EXTENSIONS and ext not in DOC_EXTENSIONS:\n raise ValueError(f\"unknown extension {value!r}\")\n return ext\n try:\n return LANGUAGE_ALIASES[v.lower()]\n except KeyError:\n known = \", \".join(sorted(LANGUAGE_ALIASES))\n raise ValueError(\n f\"unknown language {value!r} (use one of: {known}; \"\n f\"or an explicit extension such as .php)\"\n ) from None\n\n\ndef load_graphifyrc(root: Path) -> dict:\n \"\"\"Parse ``\u003croot>/.graphifyrc``. Returns ``{}`` when absent.\n\n Returned keys: ``viz_node_limit`` (int) and ``languages``\n (``{\".inc\": \".php\", ...}``) — each present only when the file sets it.\n Unknown keys are ignored so a newer graphify's options do not break an\n older one. Malformed lines raise ``ValueError`` naming the line.\n \"\"\"\n rc_path = Path(root) / RC_FILENAME\n if not rc_path.is_file():\n return {}\n\n cfg: dict = {}\n languages: dict[str, str] = {}\n # utf-8-sig: Windows PowerShell 5.1 (`Set-Content -Encoding UTF8`) and older\n # Notepad save UTF-8 with a BOM. Read as plain utf-8, the BOM became part of\n # the first key, which then matched nothing and was ignored without a word.\n content = rc_path.read_text(encoding=\"utf-8-sig\")\n for line_num, raw in enumerate(content.splitlines(), 1):\n line = raw.strip()\n if not line or line.startswith(\"#\"):\n continue\n if \"=\" not in line:\n raise ValueError(f\"Invalid line {line_num} in {rc_path}: {raw!r} (expected key=value)\")\n key, val = line.split(\"=\", 1)\n key = key.strip()\n val = val.strip()\n if key == \"viz_node_limit\":\n try:\n parsed_val = int(val)\n if parsed_val \u003c 0:\n raise ValueError(\"must be a non-negative integer\")\n cfg[\"viz_node_limit\"] = parsed_val\n except ValueError as exc:\n raise ValueError(\n f\"Invalid viz_node_limit in {rc_path} at line {line_num}: {val!r}. \"\n f\"Must be a non-negative integer.\"\n ) from exc\n elif key.startswith(\"language.\"):\n ext_part = key[len(\"language.\"):].strip()\n if not ext_part or any(ch.isspace() for ch in ext_part):\n raise ValueError(\n f\"Invalid language key in {rc_path} at line {line_num}: {key!r} \"\n f\"(expected language.\u003cext>=\u003clanguage>)\"\n )\n try:\n target = parse_language_value(val)\n except ValueError as exc:\n raise ValueError(\n f\"Invalid {key} in {rc_path} at line {line_num}: {exc}\"\n ) from None\n languages[_normalise_ext(ext_part)] = target\n if languages:\n cfg[\"languages\"] = languages\n return cfg\n\n\n# ---------------------------------------------------------------------------\n# Active overrides for this process\n# ---------------------------------------------------------------------------\n#\n# detect()/extract() activate the scan root's overrides for the run; the\n# extraction pool forwards them to its workers (they do not inherit module\n# state under ``spawn``). Stored lower-cased on both sides.\n\n# Thread-local, not module-global: two extractions can run concurrently in one\n# process (an MCP server, a threaded harness) with different roots, and a shared\n# dict let one scan's `activate_language_overrides` clobber the other's while\n# `effective_suffix`/`classify_file`/`cache_salt` read it — cross-contaminating\n# which extractor a `.inc`/`.m`/`.h` file gets (#2961 review). Each thread now\n# carries its own overrides. The dominant paths are unaffected: the CLI runs on\n# the main thread, and a spawn pool worker sets its own copy in `_worker_init`\n# on its main thread, the same thread that then runs `_extract_single_file`.\n_local = threading.local()\n_warned_roots: set[str] = set() # warning-dedup only; harmless to share\n\n\ndef _active() -> dict[str, str]:\n got = getattr(_local, \"active\", None)\n if got is None:\n got = {}\n _local.active = got\n return got\n\n\ndef set_language_overrides(mapping: dict[str, str] | None) -> None:\n \"\"\"Replace this thread's extension overrides (``{\".inc\": \".php\"}``).\"\"\"\n active = _active()\n active.clear()\n if mapping:\n for ext, target in mapping.items():\n active[_normalise_ext(ext)] = _normalise_ext(target)\n\n\ndef get_language_overrides() -> dict[str, str]:\n return dict(_active())\n\n\ndef activate_language_overrides(root: Path) -> dict[str, str]:\n \"\"\"Load ``\u003croot>/.graphifyrc`` and make its language overrides active.\n\n A malformed file is reported once per root on stderr and treated as\n having no overrides — a scan must not die on a config typo, but the\n user must hear about it, or their ``.inc`` silently stays Pascal.\n Returns the mapping now active.\n \"\"\"\n try:\n cfg = load_graphifyrc(Path(root))\n except (ValueError, OSError) as exc:\n key = str(root)\n if key not in _warned_roots:\n _warned_roots.add(key)\n print(f\"[graphify] warning: ignoring {RC_FILENAME}: {exc}\", file=sys.stderr)\n cfg = {}\n set_language_overrides(cfg.get(\"languages\"))\n return get_language_overrides()\n\n\ndef effective_suffix(path: Path | str) -> str:\n \"\"\"The suffix graphify should treat ``path`` as having.\n\n Returns the override target when the file's extension is remapped, else\n the real suffix untouched (case preserved, so callers that distinguish\n ``.F90`` from ``.f90`` keep doing so).\n \"\"\"\n suffix = Path(path).suffix\n active = _active()\n if not active:\n return suffix\n return active.get(suffix.lower(), suffix)\n\n\ndef cache_salt(path: Path | str) -> str | None:\n \"\"\"Extra cache-key material for a remapped file, else ``None``.\n\n An AST cache entry is keyed by content, and the same bytes parse to a\n different graph under a different extractor — so a ``.inc`` cached as\n Pascal must not be served once the project declares it PHP.\n \"\"\"\n active = _active()\n if not active:\n return None\n target = active.get(Path(path).suffix.lower())\n return f\"language={target}\" if target else None\n","content_type":"text/x-python; charset=utf-8","language":"python","size":9628,"content_sha256":"d82e19f70a4a449d37822322f2a31b4d5832d9403a6a9e0911163ea390b3b6f2"},{"filename":"resolver_registry.py","content":"\"\"\"Registry for cross-file, language-specific resolution passes.\n\nSome call/reference edges can only be resolved with language-specific knowledge\n(receiver typing, qualified member calls, framework conventions). Historically\nthese ran as a hand-wired sequence of suffix-gated\n``if \u003clang>_paths: try: _resolve_\u003clang>(...)`` blocks at the tail of\n``extract.extract()``. That pattern is the de-facto extension point for\nper-language resolution; this module formalizes it so a new language plugs in by\nregistering one ``LanguageResolver`` instead of editing ``extract()``'s body.\n\nThis module deliberately knows nothing about any specific language — languages\nregister themselves (see ``extract.py``), keeping the dependency direction one\nway (``extract`` → ``resolver_registry``) and this seam small and reviewable on\nits own, separate from the multi-thousand-line ``extract`` module.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport logging\nfrom collections.abc import Sequence\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom typing import Callable\n\n_LOG = logging.getLogger(__name__)\n\n\n@dataclass(frozen=True)\nclass LanguageResolver:\n \"\"\"One cross-file, language-specific resolution pass.\n\n ``resolve`` has the signature ``(per_file, all_nodes, all_edges) -> None`` and\n mutates ``all_nodes`` / ``all_edges`` in place, matching the existing\n member-call resolvers. ``suffixes`` gates activation: the pass runs only when\n the corpus contains at least one file with one of these extensions.\n \"\"\"\n\n name: str\n suffixes: frozenset\n resolve: Callable\n\n\n# Module-level registry, populated by callers via register(). Ordered: resolvers\n# run in registration order, preserving any required sequencing between passes.\n_REGISTRY: list[LanguageResolver] = []\n\n\ndef register(resolver: LanguageResolver) -> LanguageResolver:\n \"\"\"Append a resolver to the global registry and return it (for inline use).\"\"\"\n _REGISTRY.append(resolver)\n return resolver\n\n\ndef registered_resolvers() -> list[LanguageResolver]:\n \"\"\"Return a copy of the registered resolvers, in registration order.\"\"\"\n return list(_REGISTRY)\n\n\ndef run_language_resolvers(\n paths: Sequence[Path],\n per_file: list[dict],\n all_nodes: list[dict],\n all_edges: list[dict],\n *,\n resolvers: Sequence[LanguageResolver] | None = None,\n) -> None:\n \"\"\"Run every resolver whose suffix appears in ``paths``.\n\n Behaviorally identical to the prior hand-wired sequence of suffix-gated,\n try/except-wrapped passes: same activation rule (suffix present), same\n failure handling (log a warning and continue to the next pass), same\n execution order (registration order).\n\n ``resolvers`` defaults to the global registry; tests pass an explicit list to\n exercise the driver in isolation.\n \"\"\"\n active = _REGISTRY if resolvers is None else resolvers\n # Honour project-level remaps (#2961): a `.inc` declared PHP must wake\n # the PHP resolvers, not Pascal's.\n from graphify.rcfile import effective_suffix\n suffixes_present = {effective_suffix(p) for p in paths}\n for resolver in active:\n if not (resolver.suffixes & suffixes_present):\n continue\n try:\n resolver.resolve(per_file, all_nodes, all_edges)\n except Exception as exc:\n _LOG.warning(\"%s resolution failed, skipping: %s\", resolver.name, exc)\n","content_type":"text/x-python; charset=utf-8","language":"python","size":3396,"content_sha256":"23a79bf81b0f66e3999dfd392678650e702abffaa5688ca5dce28c0d32738402"},{"filename":"swift_dispatch.py","content":"\"\"\"Member-level protocol dispatch for Swift.\n\n`store.fetch()` on an injected `Store` resolves to the protocol's own `.fetch()`\nrequirement (#3673), because that is what the call site names. The conforming\n`RemoteStore.fetch()` is a separate node, and nothing joins the two, so a\ndirected walk stops at the protocol and every chain through an injected\ndependency is cut at that point — the Swift twin of the C# gap #3003 closed.\n\nThe mechanism — single-owner, per-method `dispatches_to` edges at INFERRED,\nfanning out to every conformer — lives in `graphify.interface_dispatch` and\nis shared with the C# pass; this module fixes the Swift gate. Both ends of a pair must be\ndeclared in a `.swift` file: `implements` is resolved by name, so a Kotlin or\nJava class declaring the same supertype name binds to the Swift protocol when\nthat is the only node with the name, and linking a requirement to its method\nwould be a wrong edge rather than a missing one. An Objective-C class adopting\na Swift protocol across the bridge is a real thing, but claiming it needs the\nObjC extractor's own evidence.\n\nWhat the Swift extractor hands this pass:\n\n- Conformance arrives as `implements` from struct/enum/actor declarations, from\n a class whose protocol is declared in the same file, and from\n `extension T: P`, which `_merge_swift_extensions` folds into T before the\n registry runs. A `class C: P` whose protocol lives in another file is\n classified `inherits` — Swift spells a superclass and a protocol the same way\n and the per-file classifier cannot see P — so it is not dispatched here.\n- A default implementation in a protocol extension is folded into the\n protocol's own node, so a conformer that does not override it owns no\n candidate and gets no edge: the call really does land on the default.\n- Builtin protocols (`View`, `Codable`, …) are sourceless stubs and never pass\n the declared-in guard.\n\"\"\"\nfrom __future__ import annotations\n\nfrom graphify.interface_dispatch import resolve_interface_dispatch\n\n_SWIFT_SUFFIXES = (\".swift\",)\n\n\ndef resolve_swift_protocol_dispatch(\n per_file: list[dict],\n all_nodes: list[dict],\n all_edges: list[dict],\n) -> None:\n \"\"\"Link each Swift protocol requirement to every conformer that owns it unambiguously.\"\"\"\n resolve_interface_dispatch(per_file, all_nodes, all_edges, suffixes=_SWIFT_SUFFIXES)\n","content_type":"text/x-python; charset=utf-8","language":"python","size":2364,"content_sha256":"a8c96db90744d4b071531587a448ea2c6d2eb574a3278cb753ab626effd5c116"},{"filename":"transcribe.py","content":"# Video transcription using faster-whisper\n# Converts video/audio files to text transcripts for graph extraction\nfrom __future__ import annotations\n\nimport os\nfrom pathlib import Path\n\nfrom graphify.paths import out_path as _out_path\n\n\nVIDEO_EXTENSIONS = {'.mp4', '.mov', '.webm', '.mkv', '.avi', '.m4v', '.mp3', '.wav', '.m4a', '.ogg'}\nURL_PREFIXES = ('http://', 'https://', 'www.')\n\n_DEFAULT_MODEL = \"base\"\n_TRANSCRIPTS_DIR = str(_out_path(\"transcripts\"))\n_FALLBACK_PROMPT = \"Use proper punctuation and paragraph breaks.\"\n\n\ndef _model_name() -> str:\n return os.environ.get(\"GRAPHIFY_WHISPER_MODEL\", _DEFAULT_MODEL)\n\n\ndef _get_whisper():\n try:\n from faster_whisper import WhisperModel\n return WhisperModel\n except ImportError as exc:\n raise ImportError(\n \"Video transcription requires faster-whisper. \"\n \"Run: pip install 'graphifyy[video]'\"\n ) from exc\n\n\ndef _get_yt_dlp():\n try:\n import yt_dlp\n return yt_dlp\n except ImportError as exc:\n raise ImportError(\n \"YouTube/URL download requires yt-dlp. \"\n \"Run: pip install 'graphifyy[video]'\"\n ) from exc\n\n\ndef is_url(path: str) -> bool:\n \"\"\"Return True if the string looks like a URL rather than a file path.\"\"\"\n return any(path.startswith(p) for p in URL_PREFIXES)\n\n\ndef download_audio(url: str, output_dir: Path) -> Path:\n \"\"\"Download audio-only stream from a URL using yt-dlp.\n\n Returns the path to the downloaded audio file (.m4a or .opus).\n Uses cached file if already downloaded.\n \"\"\"\n from graphify.security import validate_url\n validate_url(url) # blocks private IPs, bad schemes before yt-dlp runs\n yt_dlp = _get_yt_dlp()\n output_dir.mkdir(parents=True, exist_ok=True)\n\n # yt-dlp uses %(title)s which can be long/weird — use a stable name based on URL hash\n import hashlib\n url_hash = hashlib.sha1(url.encode(), usedforsecurity=False).hexdigest()[:12]\n out_template = str(output_dir / f\"yt_{url_hash}.%(ext)s\")\n\n # Check for already-downloaded file\n for ext in ('.m4a', '.opus', '.mp3', '.ogg', '.wav', '.webm'):\n candidate = output_dir / f\"yt_{url_hash}{ext}\"\n if candidate.exists():\n print(f\" cached audio: {candidate.name}\")\n return candidate\n\n ydl_opts = {\n 'format': 'bestaudio[ext=m4a]/bestaudio/best',\n 'outtmpl': out_template,\n 'quiet': True,\n 'no_warnings': True,\n 'noplaylist': True,\n 'postprocessors': [], # no ffmpeg needed — use native audio\n }\n\n print(f\" downloading audio: {url[:80]} ...\", flush=True)\n with yt_dlp.YoutubeDL(ydl_opts) as ydl:\n info = ydl.extract_info(url, download=True)\n ext = info.get('ext', 'm4a')\n downloaded = output_dir / f\"yt_{url_hash}.{ext}\"\n if not downloaded.exists():\n # yt-dlp may have picked a different extension\n for p in output_dir.glob(f\"yt_{url_hash}.*\"):\n downloaded = p\n break\n return downloaded\n\n\ndef build_whisper_prompt(god_nodes: list[dict]) -> str:\n \"\"\"Build a domain hint for Whisper from god nodes extracted from the corpus.\n\n Formats the top god node labels into a topic string for Whisper.\n The coding agent (Claude Code, Codex, etc.) generates the actual one-sentence\n domain hint from these labels and passes it via GRAPHIFY_WHISPER_PROMPT or\n as initial_prompt — no separate API call needed here.\n \"\"\"\n if not god_nodes:\n return _FALLBACK_PROMPT\n\n override = os.environ.get(\"GRAPHIFY_WHISPER_PROMPT\")\n if override:\n return override\n\n labels = [n.get(\"label\", \"\") for n in god_nodes[:10] if n.get(\"label\")]\n if not labels:\n return _FALLBACK_PROMPT\n\n topics = \", \".join(labels[:5])\n return f\"Technical discussion about {topics}. Use proper punctuation and paragraph breaks.\"\n\n\ndef transcribe(\n video_path: Path | str,\n output_dir: Path | None = None,\n initial_prompt: str | None = None,\n force: bool = False,\n) -> Path:\n \"\"\"Transcribe a video/audio file or URL to a .txt transcript.\n\n If video_path is a URL, audio is downloaded first via yt-dlp.\n Returns the path to the saved transcript file.\n Uses cached transcript if it exists unless force=True.\n\n initial_prompt: domain hint for Whisper (built from corpus god nodes).\n force: re-transcribe even if transcript already exists.\n \"\"\"\n out_dir = Path(output_dir) if output_dir else Path(_TRANSCRIPTS_DIR)\n out_dir.mkdir(parents=True, exist_ok=True)\n\n if is_url(str(video_path)):\n audio_path = download_audio(str(video_path), out_dir / \"downloads\")\n else:\n audio_path = Path(video_path)\n\n transcript_path = out_dir / (audio_path.stem + \".txt\")\n if transcript_path.exists() and not force:\n return transcript_path\n\n WhisperModel = _get_whisper()\n model_name = _model_name()\n prompt = initial_prompt or _FALLBACK_PROMPT\n\n print(f\" transcribing {audio_path.name} (model={model_name}) ...\", flush=True)\n model = WhisperModel(model_name, device=\"cpu\", compute_type=\"int8\")\n segments, info = model.transcribe(\n str(audio_path),\n beam_size=5,\n initial_prompt=prompt,\n )\n\n lines = [segment.text.strip() for segment in segments if segment.text.strip()]\n transcript = \"\\n\".join(lines)\n\n transcript_path.write_text(transcript, encoding=\"utf-8\")\n lang = info.language if hasattr(info, \"language\") else \"unknown\"\n print(f\" transcript saved -> {transcript_path} (lang={lang}, {len(lines)} segments)\")\n return transcript_path\n\n\ndef transcribe_all(\n video_files: list[str],\n output_dir: Path | None = None,\n initial_prompt: str | None = None,\n) -> list[str]:\n \"\"\"Transcribe a list of video/audio files or URLs, return paths to transcript .txt files.\n\n Already-transcribed files are returned from cache instantly.\n initial_prompt is shared across all files — built once from corpus god nodes.\n \"\"\"\n if not video_files:\n return []\n\n transcript_paths = []\n for vf in video_files:\n try:\n t = transcribe(vf, output_dir, initial_prompt=initial_prompt)\n transcript_paths.append(str(t))\n except Exception as exc:\n print(f\" warning: could not transcribe {vf}: {exc}\")\n return transcript_paths\n","content_type":"text/x-python; charset=utf-8","language":"python","size":6372,"content_sha256":"b01c127af5a7bf44865e517848dbb2d3dd88af0b146674c87ea56079cd926d55"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"/graphify","type":"text"}]},{"type":"paragraph","content":[{"text":"Turn any folder of files into a navigable knowledge graph with community detection, an honest audit trail, and three outputs: interactive HTML, GraphRAG-ready JSON, and a plain-language GRAPH_REPORT.md.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Usage","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault)\n/graphify \u003cpath> # full pipeline on specific path\n/graphify https://github.com/\u003cowner>/\u003crepo> # clone repo then run full pipeline on it\n/graphify https://github.com/\u003cowner>/\u003crepo> --branch \u003cbranch> # clone a specific branch\n/graphify \u003curl1> \u003curl2> ... # clone multiple repos, build each, merge into one cross-repo graph\n/graphify \u003cpath> --mode deep # thorough extraction, richer INFERRED edges\n/graphify \u003cpath> --update # incremental - re-extract only new/changed files\n/graphify \u003cpath> --directed # build directed graph (preserves edge direction: source→target)\n/graphify \u003cpath> --whisper-model medium # use a larger Whisper model for better transcription accuracy\n/graphify \u003cpath> --cluster-only # rerun clustering on existing graph\n/graphify \u003cpath> --no-viz # skip visualization, just report + JSON\n/graphify \u003cpath> --html # (HTML is generated by default - this flag is a no-op)\n/graphify \u003cpath> --svg # also export graph.svg (embeds in Notion, GitHub)\n/graphify \u003cpath> --graphml # export graph.graphml (Gephi, yEd)\n/graphify \u003cpath> --neo4j # generate graphify-out/cypher.txt for Neo4j\n/graphify \u003cpath> --neo4j-push bolt://localhost:7687 # push directly to Neo4j\n/graphify \u003cpath> --falkordb # generate graphify-out/cypher.txt for FalkorDB\n/graphify \u003cpath> --falkordb-push falkordb://localhost:6379 # push directly to FalkorDB\n/graphify \u003cpath> --mcp # start MCP stdio server for agent access\n/graphify \u003cpath> --watch # watch folder, auto-rebuild on code changes (no LLM needed)\n/graphify \u003cpath> --wiki # build agent-crawlable wiki (index.md + one article per community)\n/graphify \u003cpath> --obsidian --obsidian-dir ~/vaults/my-project # write vault to custom path (e.g. existing vault)\n/graphify add \u003curl> # fetch URL, save to ./raw, update graph\n/graphify add \u003curl> --author \"Name\" # tag who wrote it\n/graphify add \u003curl> --contributor \"Name\" # tag who added it to the corpus\n/graphify query \"\u003cquestion>\" # BFS traversal - broad context\n/graphify query \"\u003cquestion>\" --dfs # DFS - trace a specific path\n/graphify query \"\u003cquestion>\" --budget 1500 # cap answer at N tokens\n/graphify path \"AuthModule\" \"Database\" # shortest path between two concepts\n/graphify explain \"SwinTransformer\" # plain-language explanation of a node","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"What graphify is for","type":"text"}]},{"type":"paragraph","content":[{"text":"Drop any folder of code, docs, papers, images, or video into graphify and get a queryable knowledge graph. Persistent across sessions, honest audit trail (EXTRACTED/INFERRED/AMBIGUOUS), community detection surfaces cross-document connections you wouldn't think to ask about.","type":"text"}]},{"type":"paragraph","content":[{"text":"When a graph is worth building:","type":"text","marks":[{"type":"strong"}]},{"text":" this applies when you are choosing a tool yourself; if the user invoked ","type":"text"},{"text":"/graphify","type":"text","marks":[{"type":"code_inline"}]},{"text":", follow that request as asked. AST extraction of code is fast and free, but semantic extraction of docs, papers and images runs the model over every file, which can cost minutes and tens of thousands of tokens.","type":"text"}]},{"type":"paragraph","content":[{"text":"If a graph already exists, query it. With no graph, one targeted lookup (\"where is X defined\", \"what calls Y\" in an ordinary-sized codebase) is faster and free with a plain text search such as ","type":"text"},{"text":"grep","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"Read","type":"text","marks":[{"type":"code_inline"}]},{"text":": search directly instead of running the steps below, and mention that a graph can be built if the user wants one. Build one when you expect many questions against this corpus, the corpus is large or unfamiliar, the question needs links text search cannot see (doc to code, across documents), or it is structural (communities, bridges, import cycles).","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"What You Must Do When Invoked","type":"text"}]},{"type":"paragraph","content":[{"text":"If the user invoked ","type":"text"},{"text":"/graphify --help","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"/graphify -h","type":"text","marks":[{"type":"code_inline"}]},{"text":" (with no other arguments), print the contents of the ","type":"text"},{"text":"## Usage","type":"text","marks":[{"type":"code_inline"}]},{"text":" section above verbatim and stop. Do not run any commands, do not detect files, do not default the path to ","type":"text"},{"text":".","type":"text","marks":[{"type":"code_inline"}]},{"text":". Just print the Usage block and return.","type":"text"}]},{"type":"paragraph","content":[{"text":"Fast path — existing graph:","type":"text","marks":[{"type":"strong"}]},{"text":" Before doing anything else, check whether ","type":"text"},{"text":"graphify-out/graph.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" exists. The expected location is ","type":"text"},{"text":"graphify-out/graph.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" relative to the ","type":"text"},{"text":"current working directory","type":"text","marks":[{"type":"strong"}]},{"text":" (i.e. the project root where you are running commands). If it exists AND the user's request is a natural-language question about the codebase (e.g. \"How does X work?\", \"What calls Y?\", \"Trace the data flow through Z\") and NOT an explicit rebuild command (","type":"text"},{"text":"--update","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"--cluster-only","type":"text","marks":[{"type":"code_inline"}]},{"text":", or a bare path/URL that implies fresh extraction): ","type":"text"},{"text":"skip Steps 1–5 entirely and jump straight to ","type":"text","marks":[{"type":"strong"}]},{"text":"## For /graphify query","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":".","type":"text","marks":[{"type":"strong"}]},{"text":" Run ","type":"text"},{"text":"graphify query \"\u003cquestion>\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" immediately. Do not run detect. Do not check corpus size. Do not ask the user to narrow. The graph is already built — use it.","type":"text"}]},{"type":"paragraph","content":[{"text":"If no path was given, use ","type":"text"},{"text":".","type":"text","marks":[{"type":"code_inline"}]},{"text":" (current directory). Do not ask the user for a path.","type":"text"}]},{"type":"paragraph","content":[{"text":"If the path argument starts with ","type":"text"},{"text":"https://github.com/","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"http://github.com/","type":"text","marks":[{"type":"code_inline"}]},{"text":", treat it as a GitHub URL - run Step 0 before anything else, then continue with the resolved local path.","type":"text"}]},{"type":"paragraph","content":[{"text":"Follow these steps in order unless the fast path above applies.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 0 - GitHub repos and multi-path merge (only if a URL or several paths)","type":"text"}]},{"type":"paragraph","content":[{"text":"Only when the path is one or more ","type":"text"},{"text":"https://github.com/...","type":"text","marks":[{"type":"code_inline"}]},{"text":" URLs, or several local subfolders to merge. See ","type":"text"},{"text":"references/github-and-merge.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" for the clone, cross-repo merge, and monorepo flow, then continue with the resolved local path. A plain local path skips this step.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 1 - Ensure graphify is installed","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Detect the correct Python interpreter (handles uv tool, pipx, venv, system installs)\nPYTHON=\"\"\nGRAPHIFY_BIN=$(which graphify 2>/dev/null)\n# 1. uv tool installs — most reliable on modern Mac/Linux\nif [ -z \"$PYTHON\" ] && command -v uv >/dev/null 2>&1; then\n _UV_PY=$(uv tool run --from graphifyy python -c \"import sys; print(sys.executable)\" 2>/dev/null)\n if [ -n \"$_UV_PY\" ]; then PYTHON=\"$_UV_PY\"; fi\nfi\n# 2. Read shebang from graphify binary (pipx and direct pip installs)\nif [ -z \"$PYTHON\" ] && [ -n \"$GRAPHIFY_BIN\" ]; then\n _SHEBANG=$(head -1 \"$GRAPHIFY_BIN\" | tr -d '#!')\n case \"$_SHEBANG\" in\n *[!a-zA-Z0-9/_.@-]*) ;;\n *) \"$_SHEBANG\" -c \"import graphify\" 2>/dev/null && PYTHON=\"$_SHEBANG\" ;;\n esac\nfi\n# 3. Fall back to python3\nif [ -z \"$PYTHON\" ]; then PYTHON=\"python3\"; fi\nif ! \"$PYTHON\" -c \"import graphify\" 2>/dev/null; then\n if command -v uv >/dev/null 2>&1; then\n uv tool install --upgrade graphifyy -q 2>&1 | tail -3\n _UV_PY=$(uv tool run --from graphifyy python -c \"import sys; print(sys.executable)\" 2>/dev/null)\n if [ -n \"$_UV_PY\" ]; then PYTHON=\"$_UV_PY\"; fi\n else\n \"$PYTHON\" -m pip install graphifyy -q 2>/dev/null \\\n || \"$PYTHON\" -m pip install graphifyy -q --break-system-packages 2>&1 | tail -3\n fi\nfi\n# Write interpreter path for all subsequent steps (persists across invocations)\nmkdir -p graphify-out\n\"$PYTHON\" -c \"import sys; open('graphify-out/.graphify_python', 'w', encoding='utf-8').write(sys.executable)\"\n# Save scan root so `graphify update` (no args) knows where to look next time.\n# The scan path is passed through a quoted heredoc, never substituted into the\n# command line itself: a bare `cd \u003cpath>` (or an unquoted heredoc, which\n# still expands $()/backticks in its body) would let a malicious path execute\n# as shell code the moment this line runs.\n\"$PYTHON\" -c \"import os, sys; out_path = os.path.abspath('graphify-out/.graphify_root'); os.chdir(sys.stdin.readline().rstrip('\\n')); open(out_path, 'w', encoding='utf-8').write(os.getcwd())\" \u003c\u003c'GRAPHIFY_ROOT_EOF'\nINPUT_PATH\nGRAPHIFY_ROOT_EOF","type":"text"}]},{"type":"paragraph","content":[{"text":"If the import succeeds, print nothing and move straight to Step 2.","type":"text"}]},{"type":"paragraph","content":[{"text":"In every subsequent bash block, replace ","type":"text","marks":[{"type":"strong"}]},{"text":"python3","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" with ","type":"text","marks":[{"type":"strong"}]},{"text":"$(cat graphify-out/.graphify_python)","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" to use the correct interpreter.","type":"text","marks":[{"type":"strong"}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 2 - Detect files","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"$(cat graphify-out/.graphify_python) -c \"\nimport json\nfrom graphify.detect import detect\nfrom pathlib import Path\nfrom graphify.paths import out_path\nfrom graphify.watch import _read_build_excludes, _read_build_gitignore\nresult = detect(Path('INPUT_PATH'), extra_excludes=_read_build_excludes(out_path()), gitignore=_read_build_gitignore(out_path()))\n# Write the sidecar from Python, not a shell redirect, so the same block renders\n# on PowerShell hosts without console-encoding drift (#2528).\nPath('graphify-out/.graphify_detect.json').write_text(json.dumps(result, ensure_ascii=False), encoding=\\\"utf-8\\\")\nprint(f'Detected {result[\\\"total_files\\\"]} files')\n\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Replace INPUT_PATH with the actual path the user provided. Do NOT cat or print the JSON - read it silently and present a clean summary instead:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Corpus: X files · ~Y words\n code: N files (.py .ts .go ...)\n docs: N files (.md .txt ...)\n papers: N files (.pdf ...)\n images: N files\n video: N files (.mp4 .mp3 ...)","type":"text"}]},{"type":"paragraph","content":[{"text":"Omit any category with 0 files from the summary.","type":"text"}]},{"type":"paragraph","content":[{"text":"Then act on it:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"If ","type":"text"},{"text":"total_files","type":"text","marks":[{"type":"code_inline"}]},{"text":" is 0: stop with \"No supported files found in [path].\"","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"If ","type":"text"},{"text":"skipped_sensitive","type":"text","marks":[{"type":"code_inline"}]},{"text":" is non-empty: report the count and list the skipped file names, so a wrongly-flagged source or doc is visible and can be renamed or moved (#2106).","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"If ","type":"text"},{"text":"total_words","type":"text","marks":[{"type":"code_inline"}]},{"text":" > 2,000,000 OR ","type":"text"},{"text":"total_files","type":"text","marks":[{"type":"code_inline"}]},{"text":" > 500: show the warning. Then compute the top 5 first-level subdirectories by file count:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Read ","type":"text"},{"text":"scan_root","type":"text","marks":[{"type":"code_inline"}]},{"text":" from the detect JSON (always an absolute path to the resolved INPUT_PATH).","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Concatenate all file lists across all types (","type":"text"},{"text":"code","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"document","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"paper","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"image","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"video","type":"text","marks":[{"type":"code_inline"}]},{"text":").","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Filter out any path that starts with ","type":"text"},{"text":"scan_root + \"/graphify-out/\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" to exclude converted sidecars.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For each file, strip the ","type":"text"},{"text":"scan_root","type":"text","marks":[{"type":"code_inline"}]},{"text":" prefix and take the first path component. Files directly in ","type":"text"},{"text":"scan_root","type":"text","marks":[{"type":"code_inline"}]},{"text":" with no subdirectory count as ","type":"text"},{"text":"(root)","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"If all files are in ","type":"text"},{"text":"(root)","type":"text","marks":[{"type":"code_inline"}]},{"text":" with no subdirectories, do not ask to narrow — no subfolders exist. Instead suggest ","type":"text"},{"text":"--no-cluster","type":"text","marks":[{"type":"code_inline"}]},{"text":" to skip the expensive clustering step and proceed.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Otherwise rank by count, show the top 5 with file counts, then ask which subfolder to run on. Wait for the user's answer before proceeding.","type":"text"}]}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Otherwise: proceed directly to Step 2.5 if video files were detected, or Step 3 if not.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 2.5 - Video and audio (only if video files detected)","type":"text"}]},{"type":"paragraph","content":[{"text":"Skip this step entirely if ","type":"text"},{"text":"detect","type":"text","marks":[{"type":"code_inline"}]},{"text":" returned zero ","type":"text"},{"text":"video","type":"text","marks":[{"type":"code_inline"}]},{"text":" files. When the corpus has video or audio, see ","type":"text"},{"text":"references/transcribe.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" to transcribe them to text first, then treat the transcripts as doc files in Step 3.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 3 - Extract entities and relationships","type":"text"}]},{"type":"paragraph","content":[{"text":"Before starting:","type":"text","marks":[{"type":"strong"}]},{"text":" note whether ","type":"text"},{"text":"--mode deep","type":"text","marks":[{"type":"code_inline"}]},{"text":" was given. You must pass ","type":"text"},{"text":"DEEP_MODE=true","type":"text","marks":[{"type":"code_inline"}]},{"text":" to every subagent in Step B2 if it was. Track this from the original invocation - do not lose it.","type":"text"}]},{"type":"paragraph","content":[{"text":"This step has two parts: ","type":"text"},{"text":"structural extraction","type":"text","marks":[{"type":"strong"}]},{"text":" (deterministic, free) and ","type":"text"},{"text":"semantic extraction","type":"text","marks":[{"type":"strong"}]},{"text":" (LLM, costs tokens).","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"graphify needs no API key. Never ask the user for one, and never block on one.","type":"text","marks":[{"type":"strong"}]},{"text":" Code is extracted structurally (AST) with no LLM and no key at all — a code-only corpus (the common ","type":"text"},{"text":"/graphify .","type":"text","marks":[{"type":"code_inline"}]},{"text":" on a repo) skips semantic extraction entirely, so it needs nothing here: go straight to Part A and skip Part B. Semantic extraction (only for docs, papers, and images) uses Gemini ","type":"text"},{"text":"only if","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"GEMINI_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"GOOGLE_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" is already set; otherwise the host agent itself is the LLM. graphify does ","type":"text"},{"text":"not","type":"text","marks":[{"type":"strong"}]},{"text":" read ","type":"text"},{"text":"ANTHROPIC_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"OPENAI_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":", or any other provider key. If you catch yourself about to prompt for, wait on, or stop because of a missing API key, that is a misread of this skill — proceed without one.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Before semantic extraction:","type":"text","marks":[{"type":"strong"}]},{"text":" check whether ","type":"text"},{"text":"GEMINI_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"GOOGLE_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" is set. If neither is set, print this one-liner to the user:","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"Tip: set ","type":"text"},{"text":"GEMINI_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"GOOGLE_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" to use Gemini for semantic extraction (","type":"text"},{"text":"pip install 'graphifyy[gemini]'","type":"text","marks":[{"type":"code_inline"}]},{"text":").","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Print it once, then continue — do not wait for the user to supply a key. If ","type":"text"},{"text":"GEMINI_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"GOOGLE_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" IS set, use ","type":"text"},{"text":"graphify.llm.extract_corpus_parallel(files, backend=\"gemini\")","type":"text","marks":[{"type":"code_inline"}]},{"text":" for semantic extraction instead of dispatching subagents. The default Gemini model is ","type":"text"},{"text":"gemini-3-flash-preview","type":"text","marks":[{"type":"code_inline"}]},{"text":"; set ","type":"text"},{"text":"GRAPHIFY_GEMINI_MODEL","type":"text","marks":[{"type":"code_inline"}]},{"text":" or pass ","type":"text"},{"text":"--model","type":"text","marks":[{"type":"code_inline"}]},{"text":" in headless CLI flows to override it.","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"No other API keys are read.","type":"text","marks":[{"type":"strong"}]},{"text":" When ","type":"text"},{"text":"GEMINI_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"GOOGLE_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" are unset, semantic extraction falls to the host agent itself — the running session is the LLM. On a host that dispatches subagents (e.g. Claude Code), dispatch them as written in Part B. On a host that runs the CLI directly in a terminal and cannot dispatch subagents, do not stall: a code-only corpus has no semantic work, so write the empty semantic file (Part B \"Fast path\") and continue to Part C; for a corpus with docs/papers/images, either set a Gemini key or extract those inline yourself, but in no case prompt for ","type":"text"},{"text":"ANTHROPIC_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" — that prompt is a misread of this skill.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Run Part A (AST) and Part B (semantic) in parallel. Dispatch all semantic subagents AND start AST extraction in the same message. Both can run simultaneously since they operate on different file types. Merge results in Part C as before.","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Note: Parallelizing AST + semantic saves 5-15s on large corpora. AST is deterministic and fast; start it while subagents are processing docs/papers.","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Part A - Structural extraction for code files","type":"text"}]},{"type":"paragraph","content":[{"text":"For any code files detected, run AST extraction in parallel with Part B subagents:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"$(cat graphify-out/.graphify_python) -c \"\nimport sys, json\nfrom graphify.extract import collect_files, extract\nfrom pathlib import Path\nimport json\n\ncode_files = []\ndetect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encoding=\\\"utf-8\\\"))\nfor f in detect.get('files', {}).get('code', []):\n code_files.extend(collect_files(Path(f)) if Path(f).is_dir() else [Path(f)])\n\nif code_files:\n # root= anchors node ids and source_file on the scan root; cache_root only says\n # where graphify-out/cache lives. Pass both: without root= the anchor falls back\n # to cache_root, so pointing the cache elsewhere would rename every node (#3727).\n result = extract(code_files, cache_root=Path('INPUT_PATH'), root=Path('INPUT_PATH'))\n Path('graphify-out/.graphify_ast.json').write_text(json.dumps(result, indent=2, ensure_ascii=False), encoding=\\\"utf-8\\\")\n print(f'AST: {len(result[\\\"nodes\\\"])} nodes, {len(result[\\\"edges\\\"])} edges')\nelse:\n Path('graphify-out/.graphify_ast.json').write_text(json.dumps({'nodes':[],'edges':[],'input_tokens':0,'output_tokens':0}, ensure_ascii=False), encoding=\\\"utf-8\\\")\n print('No code files - skipping AST extraction')\n\"","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Part B - Semantic extraction (parallel subagents)","type":"text"}]},{"type":"paragraph","content":[{"text":"Fast path:","type":"text","marks":[{"type":"strong"}]},{"text":" If detection found zero docs, papers, and images (code-only corpus), skip Part B entirely and go straight to Part C. AST handles code - there is nothing for semantic subagents to do. ","type":"text"},{"text":"First write an empty semantic file","type":"text","marks":[{"type":"strong"}]},{"text":" so Part C's merge has its input (it reads ","type":"text"},{"text":".graphify_semantic.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" unconditionally; without this a code-only run hits ","type":"text"},{"text":"FileNotFoundError","type":"text","marks":[{"type":"code_inline"}]},{"text":"):","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"$(cat graphify-out/.graphify_python) -c \"\nimport json\nfrom pathlib import Path\nPath('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],'edges':[],'hyperedges':[],'input_tokens':0,'output_tokens':0}), encoding='utf-8')\n\"","type":"text"}]},{"type":"paragraph","content":[{"text":"MANDATORY: You MUST use the Agent tool here. Reading files yourself one-by-one is forbidden - it is 5-10x slower. If you do not use the Agent tool you are doing this wrong.","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Before dispatching subagents, print a timing estimate:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Load ","type":"text"},{"text":"total_words","type":"text","marks":[{"type":"code_inline"}]},{"text":" and file counts from ","type":"text"},{"text":"graphify-out/.graphify_detect.json","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Estimate agents needed: ","type":"text"},{"text":"ceil(uncached_non_code_files / 22)","type":"text","marks":[{"type":"code_inline"}]},{"text":" (chunk size is 20-25 files, capped at ~6000 total lines)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit))","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Print: \"Semantic extraction: ~N files → X agents, estimated ~Ys\"","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Step B0 - Check extraction cache first","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Before dispatching any subagents, check which files already have cached extraction results:","type":"text"}]},{"type":"paragraph","content":[{"text":"SPEC_PATH below is the ","type":"text"},{"text":"absolute","type":"text","marks":[{"type":"strong"}]},{"text":" path of the ","type":"text"},{"text":"references/extraction-spec.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"$(cat graphify-out/.graphify_python) -c \"\nimport json\nfrom graphify.cache import check_semantic_cache\nfrom pathlib import Path\n\ndetect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encoding=\\\"utf-8\\\"))\n# Only content files go to semantic extraction. Code is already covered structurally\n# by the AST pass (Part A); flattening every category here makes subagents re-read\n# every source file (#1392). Video is transcribed to a document in Step 2.5 first.\nall_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]\n\ncached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')\n\n# Always (re)write the cache file: write hits, else DELETE any leftover from a prior\n# run so Part C never merges a stale .graphify_cached.json (#1392).\nif cached_nodes or cached_edges or cached_hyperedges:\n Path('graphify-out/.graphify_cached.json').write_text(json.dumps({'nodes': cached_nodes, 'edges': cached_edges, 'hyperedges': cached_hyperedges}, ensure_ascii=False), encoding=\\\"utf-8\\\")\nelse:\n Path('graphify-out/.graphify_cached.json').unlink(missing_ok=True)\nPath('graphify-out/.graphify_uncached.txt').write_text('\\n'.join(uncached), encoding=\\\"utf-8\\\")\n# Nothing has been dispatched yet, so any chunk file on disk is a leftover from an\n# interrupted run, and Step B3 merges every .graphify_chunk_*.json it finds.\nfor stale in Path('graphify-out').glob('.graphify_chunk_*.json'):\n stale.unlink()\nprint(f'Cache: {len(all_files)-len(uncached)} files hit, {len(uncached)} files need extraction')\n\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Only dispatch subagents for files listed in ","type":"text"},{"text":"graphify-out/.graphify_uncached.txt","type":"text","marks":[{"type":"code_inline"}]},{"text":". If all files are cached, skip Steps B1 and B2 but still run Step B3's commands: its merge is the only Part B step that writes ","type":"text"},{"text":"graphify-out/.graphify_semantic.json","type":"text","marks":[{"type":"code_inline"}]},{"text":", and Part C reads that file unconditionally. Do not write an empty ","type":"text"},{"text":".graphify_semantic.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" instead, because that drops every cached node.","type":"text"}]},{"type":"paragraph","content":[{"text":"Step B1 - Split into chunks","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Load files from ","type":"text"},{"text":"graphify-out/.graphify_uncached.txt","type":"text","marks":[{"type":"code_inline"}]},{"text":". Split into chunks bounded by BOTH limits: at most 20-25 files AND at most ~6000 total lines per chunk. A text file longer than 2000 lines gets a chunk to itself - that is past the Read tool's single-call cap, so sharing a chunk with other files guarantees it is only partly read. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted.","type":"text"}]},{"type":"paragraph","content":[{"text":"Both bounds matter. The line bound is not only about the 2000-line Read cap: a subagent holding 25 files splits its attention across all of them and skims the dense ones, and a 590-line plan can come back with zero nodes that way. Bounding content as well as file count is what keeps a chunk's coverage honest (#2225).","type":"text"}]},{"type":"paragraph","content":[{"text":"Step B2 - Dispatch ALL subagents in a single message","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Call the Agent tool multiple times IN THE SAME RESPONSE - one call per chunk. This is the only way they run in parallel. If you make one Agent call, wait, then make another, you are doing it sequentially and defeating the purpose.","type":"text"}]},{"type":"paragraph","content":[{"text":"IMPORTANT - subagent type:","type":"text","marks":[{"type":"strong"}]},{"text":" Always use ","type":"text"},{"text":"subagent_type=\"general-purpose\"","type":"text","marks":[{"type":"code_inline"}]},{"text":". Do NOT use ","type":"text"},{"text":"Explore","type":"text","marks":[{"type":"code_inline"}]},{"text":" - it is read-only and cannot write chunk files to disk, which silently drops extraction results. General-purpose has Write and Bash access which the subagent needs.","type":"text"}]},{"type":"paragraph","content":[{"text":"Concrete example for 3 chunks:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"[Agent tool call 1: files 1-15, subagent_type=\"general-purpose\"]\n[Agent tool call 2: files 16-30, subagent_type=\"general-purpose\"]\n[Agent tool call 3: files 31-45, subagent_type=\"general-purpose\"]","type":"text"}]},{"type":"paragraph","content":[{"text":"All three in one message. Not three separate messages.","type":"text"}]},{"type":"paragraph","content":[{"text":"Each subagent receives this exact prompt (substitute FILE_LIST, CHUNK_NUM, TOTAL_CHUNKS, DEEP_MODE, and CHUNK_PATH).","type":"text"}]},{"type":"paragraph","content":[{"text":"CHUNK_PATH must be an ","type":"text"},{"text":"absolute","type":"text","marks":[{"type":"strong"}]},{"text":" path — derive it before dispatching:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"PROJECT_ROOT=$(pwd) # cwd — where Part C globs graphify-out/ (NOT .graphify_root/scan dir, #1392)\n# Then for chunk N: CHUNK_PATH=\"${PROJECT_ROOT}/graphify-out/.graphify_chunk_0N.json\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Subagent prompt template:","type":"text"}]},{"type":"paragraph","content":[{"text":"See ","type":"text"},{"text":"references/extraction-spec.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" for the exact subagent prompt (JSON schema, node-ID rules, confidence rubric, frontmatter, hyperedge, and vision rules). Load it only here, only when at least one chunk holds a doc, paper, or image; a pure-code corpus has skipped Part B and never reads it. Pass each subagent that prompt verbatim with FILE_LIST, CHUNK_NUM, TOTAL_CHUNKS, DEEP_MODE, and CHUNK_PATH substituted, and have it write the result to CHUNK_PATH.","type":"text"}]},{"type":"paragraph","content":[{"text":"Step B3 - Collect, cache, and merge","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Wait for all subagents. For each result:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Check that ","type":"text"},{"text":"graphify-out/.graphify_chunk_NN.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" exists on disk — this is the success signal","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"If the file exists and contains valid JSON with ","type":"text"},{"text":"nodes","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"edges","type":"text","marks":[{"type":"code_inline"}]},{"text":", include it and save to cache","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: \"chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent.\" Do not silently skip.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure ","type":"text"},{"text":"subagent_type=\"general-purpose\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" is used.","type":"text"}]},{"type":"paragraph","content":[{"text":"Step B3.5 - Verify each reported file was actually read to its end","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"A subagent that only read the first Read call's worth of a long file still writes a well-formed chunk, so \"chunk file exists and parses\" is not evidence of coverage. Compare each file's self-reported ","type":"text"},{"text":"coverage","type":"text","marks":[{"type":"code_inline"}]},{"text":" against its real length:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"$(cat graphify-out/.graphify_python) -c \"\nimport json, glob\nfrom pathlib import Path\n\ncovered = {}\nfor c in sorted(glob.glob('graphify-out/.graphify_chunk_*.json')):\n d = json.loads(Path(c).read_text(encoding=\\\"utf-8\\\"))\n for src, n in (d.get('coverage') or {}).items():\n covered[src] = max(covered.get(src, 0), int(n))\n\npartial = []\nfor src, n in sorted(covered.items()):\n p = Path(src)\n if not p.is_absolute():\n p = Path('INPUT_PATH') / src\n if not p.is_file():\n continue\n total = sum(1 for _ in p.open(encoding='utf-8', errors='replace'))\n if total and n \u003c total:\n partial.append((src, n, total))\n\nPath('graphify-out/.graphify_partial.txt').write_text('\\n'.join(s for s, _, _ in partial), encoding=\\\"utf-8\\\")\nif partial:\n print(f'WARNING: {len(partial)} file(s) were only partly read. They are cached as partial and retried on the next --update:')\n for src, n, total in partial:\n print(f' {src}: read {n} of {total} lines')\nelse:\n print('Coverage OK - every file a subagent reported was read to its last line')\n\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Tell the user which files came up short and what was lost (the line counts are printed above). The nodes already extracted for those files are kept - they are still true - but the file is cached as partial and left unstamped in the manifest below, so the next ","type":"text"},{"text":"--update","type":"text","marks":[{"type":"code_inline"}]},{"text":" re-dispatches it and replace-on-re-extract swaps the partial node set for a complete one. This is the same mechanism the library path uses for a truncated LLM response (#933, #1950), so both paths now recover instead of silently keeping a half-read file forever. A file a subagent did not report at all is not counted as partial - there is nothing to compare against - which is why the prompt rule in ","type":"text"},{"text":"references/extraction-spec.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" asks for the count in the first place.","type":"text"}]},{"type":"paragraph","content":[{"text":"Merge all chunk files into ","type":"text"},{"text":".graphify_semantic_new.json","type":"text","marks":[{"type":"code_inline"}]},{"text":". ","type":"text"},{"text":"After each Agent call completes, read the real token counts from the Agent tool result's ","type":"text","marks":[{"type":"strong"}]},{"text":"usage","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" field and write them back into the chunk JSON before merging","type":"text","marks":[{"type":"strong"}]},{"text":" — the chunk JSON itself always has placeholder zeros. Then run:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"$(cat graphify-out/.graphify_python) -c \"\nimport json, glob\nfrom pathlib import Path\n\nchunks = sorted(glob.glob('graphify-out/.graphify_chunk_*.json'))\nall_nodes, all_edges, all_hyperedges = [], [], []\ntotal_in, total_out = 0, 0\nfor c in chunks:\n d = json.loads(Path(c).read_text(encoding=\\\"utf-8\\\"))\n all_nodes += d.get('nodes', [])\n all_edges += d.get('edges', [])\n all_hyperedges += d.get('hyperedges', [])\n total_in += d.get('input_tokens', 0)\n total_out += d.get('output_tokens', 0)\nPath('graphify-out/.graphify_semantic_new.json').write_text(json.dumps({\n 'nodes': all_nodes, 'edges': all_edges, 'hyperedges': all_hyperedges,\n 'input_tokens': total_in, 'output_tokens': total_out,\n}, indent=2, ensure_ascii=False), encoding=\\\"utf-8\\\")\nprint(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens')\n\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"$(cat graphify-out/.graphify_python) -c \"\nimport json\nfrom graphify.cache import save_semantic_cache\nfrom pathlib import Path\n\nnew = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\\\"utf-8\\\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}\nuncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\\\"utf-8\\\").splitlines() if line]\n# A partly-read file is cached with partial: True, which load_cached treats as a MISS,\n# so the next --update re-dispatches it instead of serving a truncated extraction forever\n# (#2225). Same contract the library path uses for a truncated LLM response (#933/#1950).\npartial = [line for line in Path('graphify-out/.graphify_partial.txt').read_text(encoding=\\\"utf-8\\\").splitlines() if line] if Path('graphify-out/.graphify_partial.txt').exists() else []\nsaved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH', partial_source_files=partial)\nnote = f' ({len(partial)} left partial for retry)' if partial else ''\nprint(f'Cached {saved} files' + note)\n\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Merge cached + new results into ","type":"text"},{"text":"graphify-out/.graphify_semantic.json","type":"text","marks":[{"type":"code_inline"}]},{"text":":","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"$(cat graphify-out/.graphify_python) -c \"\nimport json\nfrom pathlib import Path\n\ncached = json.loads(Path('graphify-out/.graphify_cached.json').read_text(encoding=\\\"utf-8\\\")) if Path('graphify-out/.graphify_cached.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}\nnew = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\\\"utf-8\\\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}\n\nall_nodes = cached['nodes'] + new.get('nodes', [])\nall_edges = cached['edges'] + new.get('edges', [])\nall_hyperedges = cached.get('hyperedges', []) + new.get('hyperedges', [])\nseen = set()\ndeduped = []\nfor n in all_nodes:\n if n['id'] not in seen:\n seen.add(n['id'])\n deduped.append(n)\n\nmerged = {\n 'nodes': deduped,\n 'edges': all_edges,\n 'hyperedges': all_hyperedges,\n 'input_tokens': new.get('input_tokens', 0),\n 'output_tokens': new.get('output_tokens', 0),\n}\nPath('graphify-out/.graphify_semantic.json').write_text(json.dumps(merged, indent=2, ensure_ascii=False), encoding=\\\"utf-8\\\")\nprint(f'Extraction complete - {len(deduped)} nodes, {len(all_edges)} edges ({len(cached[\\\"nodes\\\"])} from cache, {len(new.get(\\\"nodes\\\",[]))} new)')\n\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Clean up temp files: ","type":"text"},{"text":"rm -f graphify-out/.graphify_cached.json graphify-out/.graphify_uncached.txt graphify-out/.graphify_semantic_new.json","type":"text","marks":[{"type":"code_inline"}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Part C - Merge AST + semantic into final extraction","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"$(cat graphify-out/.graphify_python) -c \"\nimport sys, json\nfrom pathlib import Path\n\nast = json.loads(Path('graphify-out/.graphify_ast.json').read_text(encoding=\\\"utf-8\\\"))\nsem = json.loads(Path('graphify-out/.graphify_semantic.json').read_text(encoding=\\\"utf-8\\\"))\n\n# Stamp tier provenance on the semantic side. build._is_ast_tier reads\n# _origin when present and otherwise guesses from the SHAPE of\n# source_location ('L\u003cline>' means AST). A subagent that writes 'L12' for a\n# doc section therefore made that node read as AST, and the next\n# graphify update deleted it along with the re-extracted code file it\n# seemed to belong to (#2843). extract() stamps its own items 'ast'; the\n# semantic side must say so explicitly.\nfor n in sem['nodes']:\n n.setdefault('_origin', 'semantic')\nfor e in sem['edges']:\n e.setdefault('_origin', 'semantic')\n\n# Merge: AST nodes first, semantic nodes deduplicated by id\nseen = {n['id'] for n in ast['nodes']}\nmerged_nodes = list(ast['nodes'])\nfor n in sem['nodes']:\n if n['id'] not in seen:\n merged_nodes.append(n)\n seen.add(n['id'])\n\nmerged_edges = ast['edges'] + sem['edges']\nmerged_hyperedges = sem.get('hyperedges', [])\nmerged = {\n 'nodes': merged_nodes,\n 'edges': merged_edges,\n 'hyperedges': merged_hyperedges,\n 'input_tokens': sem.get('input_tokens', 0),\n 'output_tokens': sem.get('output_tokens', 0),\n}\nPath('graphify-out/.graphify_extract.json').write_text(json.dumps(merged, indent=2, ensure_ascii=False), encoding=\\\"utf-8\\\")\ntotal = len(merged_nodes)\nedges = len(merged_edges)\nprint(f'Merged: {total} nodes, {edges} edges ({len(ast[\\\"nodes\\\"])} AST + {len(sem[\\\"nodes\\\"])} semantic)')\n\"","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 4 - Build graph, cluster, analyze, generate outputs","type":"text"}]},{"type":"paragraph","content":[{"text":"Before starting:","type":"text","marks":[{"type":"strong"}]},{"text":" the code blocks below pass ","type":"text"},{"text":"directed=IS_DIRECTED","type":"text","marks":[{"type":"code_inline"}]},{"text":" to ","type":"text"},{"text":"build_from_json()","type":"text","marks":[{"type":"code_inline"}]},{"text":". Replace ","type":"text"},{"text":"IS_DIRECTED","type":"text","marks":[{"type":"code_inline"}]},{"text":" with ","type":"text"},{"text":"True","type":"text","marks":[{"type":"code_inline"}]},{"text":" if ","type":"text"},{"text":"--directed","type":"text","marks":[{"type":"code_inline"}]},{"text":" was given (builds a ","type":"text"},{"text":"DiGraph","type":"text","marks":[{"type":"code_inline"}]},{"text":" preserving edge direction source→target), otherwise ","type":"text"},{"text":"False","type":"text","marks":[{"type":"code_inline"}]},{"text":" (the default undirected ","type":"text"},{"text":"Graph","type":"text","marks":[{"type":"code_inline"}]},{"text":"). Substitute it the same way you substitute ","type":"text"},{"text":"INPUT_PATH","type":"text","marks":[{"type":"code_inline"}]},{"text":" — do not leave the literal ","type":"text"},{"text":"IS_DIRECTED","type":"text","marks":[{"type":"code_inline"}]},{"text":" in the code.","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"mkdir -p graphify-out\n$(cat graphify-out/.graphify_python) -c \"\nimport sys, json\nfrom graphify.build import build_from_json\nfrom graphify.cluster import cluster, score_all\nfrom graphify.analyze import god_nodes, surprising_connections, suggest_questions\nfrom graphify.report import generate\nfrom graphify.export import to_json\nfrom pathlib import Path\n\nextraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text(encoding=\\\"utf-8\\\"))\ndetection = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encoding=\\\"utf-8\\\"))\n\n# root= mirrors the --update runbook (#1361): relativize source_file to the same\n# base so the full build and incremental --update never drift apart on re-extract.\nG = build_from_json(extraction, root='INPUT_PATH', directed=IS_DIRECTED)\n# Guard BEFORE any write: an empty extraction must not clobber a good graph.json /\n# GRAPH_REPORT.md / analysis sidecar. Check immediately after build (#1392).\nif G.number_of_nodes() == 0:\n print('ERROR: Graph is empty - extraction produced no nodes.')\n print('Possible causes: all files were skipped, binary-only corpus, or extraction failed.')\n raise SystemExit(1)\ncommunities = cluster(G)\ncohesion = score_all(G, communities)\ntokens = {'input': extraction.get('input_tokens', 0), 'output': extraction.get('output_tokens', 0)}\ngods = god_nodes(G)\nsurprises = surprising_connections(G, communities)\nlabels = {cid: 'Community ' + str(cid) for cid in communities}\n# Placeholder questions - regenerated with real labels in Step 5\nquestions = suggest_questions(G, communities, labels)\n\n# Export FIRST and honor the #479 shrink-guard: to_json returns False (writing\n# nothing) when the new graph is smaller than the existing graph.json. Only write\n# GRAPH_REPORT.md + the analysis sidecar when the graph was actually written, so\n# they never describe a graph that graph.json doesn't contain (#1392).\nwrote = to_json(G, communities, 'graphify-out/graph.json')\nif not wrote:\n print('ERROR: refused to shrink graphify-out/graph.json (existing graph has more nodes; #479).')\n print('If this shrink is intentional (you deleted files), re-run a full build with --force.')\n raise SystemExit(1)\nreport = generate(G, communities, cohesion, labels, gods, surprises, detection, tokens, 'INPUT_PATH', suggested_questions=questions)\nPath('graphify-out/GRAPH_REPORT.md').write_text(report, encoding=\\\"utf-8\\\")\nanalysis = {\n 'communities': {str(k): v for k, v in communities.items()},\n 'cohesion': {str(k): v for k, v in cohesion.items()},\n 'gods': gods,\n 'surprises': surprises,\n 'questions': questions,\n}\nPath('graphify-out/.graphify_analysis.json').write_text(json.dumps(analysis, indent=2, ensure_ascii=False), encoding=\\\"utf-8\\\")\nprint(f'Graph: {G.number_of_nodes()} nodes, {G.number_of_edges()} edges, {len(communities)} communities')\n\"","type":"text"}]},{"type":"paragraph","content":[{"text":"If this step prints ","type":"text"},{"text":"ERROR: Graph is empty","type":"text","marks":[{"type":"code_inline"}]},{"text":", stop and tell the user what happened - do not proceed to labeling or visualization.","type":"text"}]},{"type":"paragraph","content":[{"text":"Replace INPUT_PATH with the actual path.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 4.5 - Graph health check (read-only integrity gate)","type":"text"}]},{"type":"paragraph","content":[{"text":"A non-destructive diagnostic on the extraction, before labeling. It surfaces edge collapse, dangling/missing endpoints, and self-loops — the silent-corruption modes of incremental updates and AST/LLM id mismatches. Read-only; never aborts.","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"$(cat graphify-out/.graphify_python) -c \"\nimport json\nfrom pathlib import Path\nfrom graphify.diagnostics import diagnose_extraction, format_diagnostic_report\n\nextraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text(encoding=\\\"utf-8\\\"))\nsummary = diagnose_extraction(extraction, directed=IS_DIRECTED, root='INPUT_PATH')\nprint(format_diagnostic_report(summary))\nflags = [f'{summary[k]} {label}' for k, label in (\n ('dangling_endpoint_edges', 'dangling-endpoint edges'),\n ('missing_endpoint_edges', 'missing-endpoint edges'),\n ('self_loop_edges', 'self-loop edges'),\n ('directed_same_endpoint_collapsed_edges', 'collapsed (directed) edges'),\n ('undirected_same_endpoint_collapsed_edges', 'collapsed (undirected) edges'),\n) if summary.get(k, 0)]\nprint('GRAPH HEALTH WARNING: ' + '; '.join(flags) + ' - graph may be incomplete/corrupt.' if flags else 'Graph health: OK (no dangling/missing/collapsed edges).')\n\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Substitute ","type":"text"},{"text":"IS_DIRECTED","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"INPUT_PATH","type":"text","marks":[{"type":"code_inline"}]},{"text":" as in Step 4. If a ","type":"text"},{"text":"GRAPH HEALTH WARNING","type":"text","marks":[{"type":"code_inline"}]},{"text":" prints, surface it in the final summary (do not abort — the graph is still usable, but the integrity issue must be visible, per the Honesty Rules).","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 5 - Label communities","type":"text"}]},{"type":"paragraph","content":[{"text":"Read ","type":"text"},{"text":"graphify-out/.graphify_analysis.json","type":"text","marks":[{"type":"code_inline"}]},{"text":". For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. \"Attention Mechanism\", \"Training Pipeline\", \"Data Loading\").","type":"text"}]},{"type":"paragraph","content":[{"text":"Then regenerate the report and save the labels for the visualizer:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"$(cat graphify-out/.graphify_python) -c \"\nimport sys, json\nfrom graphify.build import build_from_json\nfrom graphify.cluster import score_all\nfrom graphify.analyze import god_nodes, surprising_connections, suggest_questions\nfrom graphify.report import generate\nfrom graphify.export import to_json\nfrom pathlib import Path\n\nextraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text(encoding=\\\"utf-8\\\"))\ndetection = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encoding=\\\"utf-8\\\"))\nanalysis = json.loads(Path('graphify-out/.graphify_analysis.json').read_text(encoding=\\\"utf-8\\\"))\n\n# root= as in Step 4 / the --update runbook (#1361) — same base for node-key parity.\nG = build_from_json(extraction, root='INPUT_PATH', directed=IS_DIRECTED)\ncommunities = {int(k): v for k, v in analysis['communities'].items()}\ncohesion = {int(k): v for k, v in analysis['cohesion'].items()}\ntokens = {'input': extraction.get('input_tokens', 0), 'output': extraction.get('output_tokens', 0)}\n\n# LABELS - replace these with the names you chose above\nlabels = LABELS_DICT\n\n# Regenerate questions with real community labels (labels affect question phrasing)\nquestions = suggest_questions(G, communities, labels)\n\nreport = generate(G, communities, cohesion, labels, analysis['gods'], analysis['surprises'], detection, tokens, 'INPUT_PATH', suggested_questions=questions)\nPath('graphify-out/GRAPH_REPORT.md').write_text(report, encoding=\\\"utf-8\\\")\nPath('graphify-out/.graphify_labels.json').write_text(json.dumps({str(k): v for k, v in labels.items()}, ensure_ascii=False), encoding=\\\"utf-8\\\")\n# Re-export so graph.json nodes carry the curated community_name (#2490).\n# Same extraction as Step 4, so the #479 shrink-guard passes on node count;\n# if it still refuses, surface the guard message - do not force past it.\nwrote = to_json(G, communities, 'graphify-out/graph.json', community_labels=labels)\nif not wrote:\n print('ERROR: refused to shrink graphify-out/graph.json (existing graph has more nodes; #479).')\n print('If this shrink is intentional (you deleted files), re-run a full build with --force.')\nprint('Report updated with community labels')\n\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Replace ","type":"text"},{"text":"LABELS_DICT","type":"text","marks":[{"type":"code_inline"}]},{"text":" with the actual dict you constructed (e.g. ","type":"text"},{"text":"{0: \"Attention Mechanism\", 1: \"Training Pipeline\"}","type":"text","marks":[{"type":"code_inline"}]},{"text":"). Replace INPUT_PATH with the actual path.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 6 - Generate Obsidian vault (opt-in) + HTML","type":"text"}]},{"type":"paragraph","content":[{"text":"Generate HTML always","type":"text","marks":[{"type":"strong"}]},{"text":" (unless ","type":"text"},{"text":"--no-viz","type":"text","marks":[{"type":"code_inline"}]},{"text":"). ","type":"text"},{"text":"Obsidian vault only if ","type":"text","marks":[{"type":"strong"}]},{"text":"--obsidian","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" was explicitly given","type":"text","marks":[{"type":"strong"}]},{"text":" — skip it otherwise, it generates one file per node.","type":"text"}]},{"type":"paragraph","content":[{"text":"If ","type":"text"},{"text":"--obsidian","type":"text","marks":[{"type":"code_inline"}]},{"text":" was given:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"If ","type":"text"},{"text":"--obsidian-dir \u003cpath>","type":"text","marks":[{"type":"code_inline"}]},{"text":" was also given, pass it via ","type":"text"},{"text":"--dir","type":"text","marks":[{"type":"code_inline"}]},{"text":". Otherwise defaults to ","type":"text"},{"text":"graphify-out/obsidian","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"graphify export obsidian\n# or with custom dir: graphify export obsidian --dir ~/vaults/my-project","type":"text"}]},{"type":"paragraph","content":[{"text":"Generate the HTML graph (always, unless ","type":"text"},{"text":"--no-viz","type":"text","marks":[{"type":"code_inline"}]},{"text":"):","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"graphify export html # auto-aggregates to community view if graph > 5000 nodes\n# or: graphify export html --no-viz","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags)","type":"text"}]},{"type":"paragraph","content":[{"text":"These run only when their flag is present (","type":"text"},{"text":"--wiki","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"--neo4j","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"--neo4j-push","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"--falkordb","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"--falkordb-push","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"--svg","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"--graphml","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"--mcp","type":"text","marks":[{"type":"code_inline"}]},{"text":") or, for the token-reduction benchmark, when ","type":"text"},{"text":"total_words","type":"text","marks":[{"type":"code_inline"}]},{"text":" exceeds 5,000. A default run with no export flags skips all of them. See ","type":"text"},{"text":"references/exports.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" for each one. Run any ","type":"text"},{"text":"--wiki","type":"text","marks":[{"type":"code_inline"}]},{"text":" export before Step 9 cleanup so ","type":"text"},{"text":".graphify_labels.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" is still available.","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 9 - Save manifest, update cost tracker, clean up, and report","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"$(cat graphify-out/.graphify_python) -c \"\nimport json\nfrom pathlib import Path\nfrom datetime import datetime, timezone\nfrom graphify.detect import save_manifest\n\n# Save manifest for --update\ndetect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encoding=\\\"utf-8\\\"))\nextract = json.loads(Path('graphify-out/.graphify_extract.json').read_text(encoding=\\\"utf-8\\\"))\n# In --update mode, 'all_files' carries the full corpus; 'files' is the changed\n# subset. Full-rebuild mode populates only 'files', so the fallback handles that.\n# root= relativizes the manifest keys to the scan root (same base as the build),\n# so the on-disk manifest is portable across clones/machines and a later --update\n# matches cached files instead of missing every one (#1417).\n#\n# Only stamp semantic files (docs/papers/images) that ACTUALLY produced output:\n# a detected file whose chunk failed or was omitted must stay unstamped so the\n# next --update re-queues it, otherwise it is marked done and its content is lost\n# forever (#2015). This mirrors the library extract path exactly\n# (cli._stamped_manifest_files + clear_semantic + scan_corpus); do not stamp the\n# raw corpus. Code files are always stamped (AST is deterministic); only semantic\n# types are gated on output.\nfrom graphify.cli import _stamped_manifest_files\n_corpus = detect.get('all_files') or detect['files']\n# A file a subagent only partly read DID produce output, so the stamp filter alone would\n# keep it - but its node set covers just the head of the file, and stamping it means\n# detect_incremental reads it as done and never re-queues it, leaving the truncated graph\n# live forever. Name those files so they stay unstamped and are retried next run (#2225).\n_pf = Path('graphify-out/.graphify_partial.txt')\n_partial = {l.strip() for l in _pf.read_text(encoding=\\\"utf-8\\\").splitlines() if l.strip()} if _pf.exists() else set()\n_manifest_files = _stamped_manifest_files(_corpus, extract, Path('INPUT_PATH'), partial_source_files=_partial)\n# Files dispatched this run (the changed subset) but NOT stamped above still carry\n# a stale semantic_hash from a prior run; clear it so detect_incremental re-queues\n# them instead of reading them as unchanged (#1948).\n_sem_types = ('document', 'paper', 'image')\n_dispatched = {f for t, fl in detect['files'].items() if t in _sem_types for f in fl}\n_stamped = {f for fl in _manifest_files.values() for f in fl}\n_cleared = _dispatched - _stamped\n# scan_corpus = the RAW full corpus (not the stamp-filtered subset) so in-root\n# files newly excluded since last run are dropped rather than masquerading as\n# deletions; untouched files' prior rows are still preserved (#1908).\n_scan = {f for fl in _corpus.values() for f in fl}\nsave_manifest(_manifest_files, root='INPUT_PATH', scan_corpus=_scan, clear_semantic=_cleared or None)\n\n# Update cumulative cost tracker\ninput_tok = extract.get('input_tokens', 0)\noutput_tok = extract.get('output_tokens', 0)\n\ncost_path = Path('graphify-out/cost.json')\nif cost_path.exists():\n cost = json.loads(cost_path.read_text(encoding=\\\"utf-8\\\"))\nelse:\n cost = {'runs': [], 'total_input_tokens': 0, 'total_output_tokens': 0}\n\ncost['runs'].append({\n 'date': datetime.now(timezone.utc).isoformat(),\n 'input_tokens': input_tok,\n 'output_tokens': output_tok,\n 'files': detect.get('total_files', 0),\n})\ncost['total_input_tokens'] += input_tok\ncost['total_output_tokens'] += output_tok\ncost_path.write_text(json.dumps(cost, indent=2, ensure_ascii=False), encoding=\\\"utf-8\\\")\n\nprint(f'This run: {input_tok:,} input tokens, {output_tok:,} output tokens')\nprint(f'All time: {cost[\\\"total_input_tokens\\\"]:,} input, {cost[\\\"total_output_tokens\\\"]:,} output ({len(cost[\\\"runs\\\"])} runs)')\n\"\nrm -f graphify-out/.graphify_detect.json graphify-out/.graphify_extract.json graphify-out/.graphify_ast.json graphify-out/.graphify_semantic.json graphify-out/.graphify_analysis.json graphify-out/.graphify_partial.txt\nfind graphify-out -maxdepth 1 -name '.graphify_chunk_*.json' -delete 2>/dev/null\nrm -f graphify-out/.needs_update 2>/dev/null || true","type":"text"}]},{"type":"paragraph","content":[{"text":"Replace INPUT_PATH with the actual path (same value used in Steps 4-5) so the manifest is relativized to the scan root.","type":"text"}]},{"type":"paragraph","content":[{"text":"Tell the user (omit the obsidian line unless --obsidian was given):","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Graph complete. Outputs in PATH_TO_DIR/graphify-out/\n\n graph.html - interactive graph, open in browser\n GRAPH_REPORT.md - audit report\n graph.json - raw graph data\n obsidian/ - Obsidian vault (only if --obsidian was given)","type":"text"}]},{"type":"paragraph","content":[{"text":"If graphify saved you time, consider supporting it: https://github.com/sponsors/safishamsi","type":"text"}]},{"type":"paragraph","content":[{"text":"Replace PATH_TO_DIR with the actual absolute path of the directory that was processed.","type":"text"}]},{"type":"paragraph","content":[{"text":"If PATH_TO_DIR is inside a git repository, add one line to the report: ","type":"text"},{"text":"graphify-out/","type":"text","marks":[{"type":"code_inline"}]},{"text":" was written into the working tree, so unless the repository already ignores it, it will show in ","type":"text"},{"text":"git status","type":"text","marks":[{"type":"code_inline"}]},{"text":"; the user can add ","type":"text"},{"text":"graphify-out/","type":"text","marks":[{"type":"code_inline"}]},{"text":" to ","type":"text"},{"text":".gitignore","type":"text","marks":[{"type":"code_inline"}]},{"text":", or commit only the outputs their team should share (see https://docs.graphify.com/guides/team-workflows).","type":"text"}]},{"type":"paragraph","content":[{"text":"Then paste these sections from GRAPH_REPORT.md directly into the chat:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"God Nodes","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Surprising Connections","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Suggested Questions","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Do NOT paste the full report - just those three sections. Keep it concise.","type":"text"}]},{"type":"paragraph","content":[{"text":"Then immediately offer to explore. Pick the single most interesting suggested question from the report - the one that crosses the most community boundaries or has the most surprising bridge node - and ask:","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"\"The most interesting question this graph can answer: ","type":"text"},{"text":"[question]","type":"text","marks":[{"type":"strong"}]},{"text":". Want me to trace it?\"","type":"text"}]}]},{"type":"paragraph","content":[{"text":"If the user says yes, run ","type":"text"},{"text":"/graphify query \"[question]\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" on the graph and walk them through the answer using the graph structure - which nodes connect, which community boundaries get crossed, what the path reveals. Keep going as long as they want to explore. Each answer should end with a natural follow-up (\"this connects to X - want to go deeper?\") so the session feels like navigation, not a one-shot report.","type":"text"}]},{"type":"paragraph","content":[{"text":"The graph is the map. Your job after the pipeline is to be the guide.","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Interpreter guard for subcommands","type":"text"}]},{"type":"paragraph","content":[{"text":"Before running any subcommand below (","type":"text"},{"text":"--update","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"--cluster-only","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"query","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"path","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"explain","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"add","type":"text","marks":[{"type":"code_inline"}]},{"text":"), check that ","type":"text"},{"text":".graphify_python","type":"text","marks":[{"type":"code_inline"}]},{"text":" exists. If it's missing (e.g. user deleted ","type":"text"},{"text":"graphify-out/","type":"text","marks":[{"type":"code_inline"}]},{"text":"), re-resolve the interpreter first:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"if [ ! -f graphify-out/.graphify_python ]; then\n GRAPHIFY_BIN=$(which graphify 2>/dev/null)\n if [ -n \"$GRAPHIFY_BIN\" ]; then\n PYTHON=$(head -1 \"$GRAPHIFY_BIN\" | tr -d '#!')\n case \"$PYTHON\" in *[!a-zA-Z0-9/_.@-]*) PYTHON=\"python3\" ;; esac\n else\n PYTHON=\"python3\"\n fi\n mkdir -p graphify-out\n \"$PYTHON\" -c \"import sys; open('graphify-out/.graphify_python', 'w', encoding='utf-8').write(sys.executable)\"\nfi","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"For --update and --cluster-only","type":"text"}]},{"type":"paragraph","content":[{"text":"Both are non-default subcommands. ","type":"text"},{"text":"--update","type":"text","marks":[{"type":"code_inline"}]},{"text":" re-extracts only new or changed files; ","type":"text"},{"text":"--cluster-only","type":"text","marks":[{"type":"code_inline"}]},{"text":" reruns clustering on the existing graph. See ","type":"text"},{"text":"references/update.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" for both flows.","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"For /graphify query","type":"text"}]},{"type":"paragraph","content":[{"text":"When ","type":"text"},{"text":"graphify-out/graph.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" already exists and the user asks a question about the corpus, answer from the graph rather than rebuilding it:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"graphify query \"\u003cquestion>\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Before traversal, expand the question against the graph's own vocabulary so a wording mismatch does not collapse the answer to noise. If the ","type":"text"},{"text":"graphify query","type":"text","marks":[{"type":"code_inline"}]},{"text":" CLI is unavailable, fall back to an inline NetworkX traversal of ","type":"text"},{"text":"graphify-out/graph.json","type":"text","marks":[{"type":"code_inline"}]},{"text":". Answer using only what the graph output contains, and quote ","type":"text"},{"text":"source_location","type":"text","marks":[{"type":"code_inline"}]},{"text":" when citing a specific fact. For that vocab-expansion step, the BFS/DFS traversal modes, the ","type":"text"},{"text":"--budget","type":"text","marks":[{"type":"code_inline"}]},{"text":" cap, the NetworkX fallback, ","type":"text"},{"text":"save-result","type":"text","marks":[{"type":"code_inline"}]},{"text":" feedback, and the ","type":"text"},{"text":"/graphify path","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"/graphify explain","type":"text","marks":[{"type":"code_inline"}]},{"text":" flows, see ","type":"text"},{"text":"references/query.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"For /graphify add and --watch","type":"text"}]},{"type":"paragraph","content":[{"text":"Neither is part of the default build. When the user runs ","type":"text"},{"text":"/graphify add \u003curl>","type":"text","marks":[{"type":"code_inline"}]},{"text":" to fetch a URL into the corpus, or passes ","type":"text"},{"text":"--watch","type":"text","marks":[{"type":"code_inline"}]},{"text":" to auto-rebuild on file changes, see ","type":"text"},{"text":"references/add-watch.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"For the commit hook and native CLAUDE.md integration","type":"text"}]},{"type":"paragraph","content":[{"text":"When the user asks to install the post-commit auto-rebuild hook or wire graphify into a project's CLAUDE.md, see ","type":"text"},{"text":"references/hooks.md","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Honesty Rules","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Never invent an edge. If unsure, use AMBIGUOUS.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Never skip the corpus check warning.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Always show token cost in the report.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Never hide cohesion scores behind symbols - show the raw number.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Never run HTML viz on a graph with more than 5,000 nodes without warning the user.","type":"text"}]}]}]}]},"metadata":{"date":"2026-11-02","name":"graphify","author":"@skillopedia","source":{"stars":141014,"repo_name":"graphify","origin_url":"https://github.com/Graphify-Labs/graphify/blob/HEAD/graphify/skill.md","repo_owner":"Graphify-Labs","body_sha256":"322397dde49d4e0d3c06fc6f480754df2e5c602ff3b830e2b0534af10e1d2968","cluster_key":"0ff03b8f1d28e2ddee92922f1b51f09e579fd330a0790cb0a981df4522e73380","clean_bundle":{"format":"clean-skill-bundle-v1","source":"Graphify-Labs/graphify/graphify/skill.md","text_bytes":1048555,"source_commit":"32e5efe935e6844ef39a5e33409e4efe848f2172","skills_sh_rank":2080,"attachment_count":42,"text_attachments":42,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"graphify/skill.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"v1","category":"software-engineering","category_label":"Engineering"},"exact_dupes_collapsed_into_this":0},"version":"v1","category":"software-engineering","import_tag":"skills-sh-top20k-2026-11","description":"Use for any question about a codebase, its architecture, file relationships, or project content — especially when graphify-out/ exists, where the question should be treated as a graphify query first. Turns any input (code, docs, papers, images, videos) into a persistent knowledge graph with god nodes, community detection, and query/path/explain tools."}},"renderedAt":1793676211080}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.