web-search-plus — Skillopedia

Web Search Plus Stop choosing search providers. Let the skill do it for you. One command searches across 11 search providers , and a companion command extracts page text from URLs. Since 5.0 the skill runs the same engine as Web Search Plus 5.0.1: automatic routing picks the first provider by query type and moves on quickly when a provider fails, is slow or returns nothing. This skill is source-only : ranked links and extracted page text, not model-written answers. The native OpenClaw plugin is . --- 🔐 Data Handling & Privacy Read this before searching with sensitive queries. - Search querie…

, '', last)\n if last:\n return f\"{domain} — {last[:80]}\"\n return domain\n except Exception:\n return url[:60]\n\n\ndef normalize_result_url(url: str) -> str:\n \"\"\"Dedup key for a result URL (see urls.url_key).\"\"\"\n return url_key(url)\n\n\ndef deduplicate_results_across_providers(results_by_provider: List[Tuple[str, Dict[str, Any]]], max_results: int) -> Tuple[List[Dict[str, Any]], int]:\n deduped = []\n seen = set()\n dedup_count = 0\n for provider_name, data in results_by_provider:\n for item in data.get(\"results\", []):\n norm = normalize_result_url(item.get(\"url\", \"\"))\n if norm and norm in seen:\n dedup_count += 1\n continue\n if norm:\n seen.add(norm)\n item = item.copy()\n item.setdefault(\"provider\", provider_name)\n deduped.append(item)\n if len(deduped) >= max_results:\n return deduped, dedup_count\n return deduped, dedup_count\n\ndef _result_domain(url: str) -> str:\n try:\n netloc = urlparse(url or \"\").netloc.lower()\n return netloc[4:] if netloc.startswith(\"www.\") else netloc\n except Exception:\n return \"\"\n\n\n# Authority rules per query intent (wsp_core/intents.py): results from these\n# domains move up or down in the intent reranker and the quality report.\nCANONICAL_DOMAIN_RULES: Dict[str, Dict[str, List[str]]] = {\n \"docs\": {\n \"boost\": [\"docs.\", \"developer.\", \"github.com\", \"readthedocs.io\", \"modelcontextprotocol.io\"],\n \"demote\": [\"medium.com\", \"dev.to\", \"reddit.com\", \"stackoverflow.com\", \"youtube.com\"],\n },\n \"security\": {\n \"boost\": [\n \"nvd.nist.gov\", \"cve.org\", \"github.com\", \"github.com/advisories\", \"security.\",\n \"cert.europa.eu\", \"kb.cert.org\", \"cisa.gov\", \"bsi.bund.de\", \"cert.ssi.gouv.fr\",\n \"ncsc.gov.uk\", \"msrc.microsoft.com\", \"owasp.org\", \"first.org\",\n ],\n \"demote\": [\"youtube.com\", \"medium.com\", \"reddit.com\", \"stackexchange.com\"],\n },\n}\n\n\ndef _domain_matches_rule(domain: str, rule: str) -> bool:\n if rule.endswith(\".\"):\n # Label-prefix rules such as \"docs.\" / \"investor.\" / \"ir.\" match a\n # leading host label (docs.python.org), never a bare domain (notdocs.com).\n return domain.startswith(rule)\n # Exact domain or true subdomain only. A bare startswith would let\n # look-alike registrations such as openai.com.evil.example inherit\n # authority boosts (same reasoning as _blocked_domain_matches below).\n return domain == rule or domain.endswith(f\".{rule}\")\n\n\n# Known content mirrors and SEO scraper sites that republish Stack Overflow,\n# GitHub, and documentation content. These add no information over the\n# canonical source and frequently outrank it; they are removed from results\n# rather than merely demoted. Operators can extend via config\n# quality.blocked_domains or rescue a domain via quality.allowed_domains.\nSPAM_MIRROR_DOMAINS: List[str] = [\n # Stack Overflow / Q&A scrapers\n \"newbedev.com\",\n \"stackoom.com\",\n \"stackovergo.com\",\n \"syntaxfix.com\",\n \"copyprogramming.com\",\n \"devcodef1.com\",\n \"exceptionshub.com\",\n \"code-examples.net\",\n \"i-harness.com\",\n \"fixmycodeerror.com\",\n \"stacklesson.com\",\n # Python documentation look-alikes\n \"domainunion.de\",\n \"pythonlang.net\",\n \"pythonlang.de\",\n # GitHub issue/readme mirrors\n \"githubmemory.com\",\n \"gitmemory.com\",\n \"issueexplorer.com\",\n \"bleepcoder.com\",\n \"gitanswer.com\",\n # Documentation mirrors\n \"w3cub.com\",\n # Generic AI/SEO content farms already demoted by the intent reranker\n \"aizolo.com\",\n]\n\n\ndef _blocked_domain_matches(domain: str, rule: str) -> bool:\n \"\"\"Strict matcher for domain block/allow lists.\n\n Only the exact domain or true subdomains match (``newbedev.com``,\n ``de.newbedev.com``). Unlike ``_domain_matches_rule`` there is no\n ``startswith`` clause, so look-alike registrations such as\n ``newbedev.com.evil.example`` do NOT match.\n \"\"\"\n return domain == rule or domain.endswith(f\".{rule}\")\n\n\n_SITE_OPERATOR_RE = re.compile(r\"\\bsite:([a-z0-9][a-z0-9.-]*)\", re.IGNORECASE)\n\n\ndef extract_domain_constraints(query: str, include_domains: Optional[List[str]] = None) -> List[str]:\n \"\"\"Domains the user explicitly constrained the search to.\n\n Collects ``site:`` operators from the query plus ``include_domains``.\n Explicit constraints express intent: constrained domains are exempt from\n spam filtering, and domain-diversity reranking is skipped entirely.\n \"\"\"\n domains = [d.lower().rstrip(\".\") for d in _SITE_OPERATOR_RE.findall(query or \"\")]\n for entry in include_domains or []:\n if entry and entry.strip():\n domains.append(entry.lower().strip())\n return sorted(set(domains))\n\n\ndef filter_spam_results(\n results: List[Dict[str, Any]],\n extra_blocked: Optional[List[str]] = None,\n allowed: Optional[List[str]] = None,\n) -> Tuple[List[Dict[str, Any]], List[str]]:\n \"\"\"Drop results from known mirror/SEO-spam domains.\n\n Returns the kept results and the sorted unique domains that were removed.\n ``allowed`` rescues a domain from both the builtin and extra blocklists.\n \"\"\"\n blocked_rules = SPAM_MIRROR_DOMAINS + [d.lower().strip() for d in (extra_blocked or []) if d and d.strip()]\n allowed_rules = [d.lower().strip() for d in (allowed or []) if d and d.strip()]\n kept: List[Dict[str, Any]] = []\n removed_domains: List[str] = []\n for item in results:\n domain = _result_domain(item.get(\"url\", \"\"))\n if (\n domain\n and not any(_blocked_domain_matches(domain, rule) for rule in allowed_rules)\n and any(_blocked_domain_matches(domain, rule) for rule in blocked_rules)\n ):\n removed_domains.append(domain)\n continue\n kept.append(item)\n return kept, sorted(set(removed_domains))\n\n\ndef rerank_domain_diversity(\n results: List[Dict[str, Any]],\n max_per_domain: int = 2,\n query: str = \"\",\n) -> Tuple[List[Dict[str, Any]], int]:\n \"\"\"Stable rerank that stops one domain from crowding out the result list.\n\n The first ``max_per_domain`` results per domain keep their original order;\n overflow results are moved behind the diverse head (also in original\n order) instead of being dropped. Returns the reranked list and how many\n results were demoted.\n \"\"\"\n if max_per_domain \u003c 1 or len(results) \u003c 3:\n return results, 0\n # Official docs of the product the query names are not crowding: five\n # react.dev pages answer \"React useEffect cleanup\" better than a mix.\n query_terms = _query_terms(query)\n head: List[Dict[str, Any]] = []\n overflow: List[Dict[str, Any]] = []\n per_domain: Dict[str, int] = {}\n for item in results:\n domain = _result_domain(item.get(\"url\", \"\"))\n count = per_domain.get(domain, 0)\n if domain and count >= max_per_domain and not _is_named_vendor_domain(item.get(\"url\", \"\"), query_terms):\n overflow.append(item)\n continue\n per_domain[domain] = count + 1\n head.append(item)\n return head + overflow, len(overflow)\n\n\ndef _url_matches_rule(url: str, rule: str) -> bool:\n domain = _result_domain(url)\n if \"/\" not in rule:\n return _domain_matches_rule(domain, rule)\n normalized = host_and_path(url)\n normalized_rule = rule.lower().strip().rstrip(\"/\")\n return normalized == normalized_rule or normalized.startswith(f\"{normalized_rule}/\")\n\n\n# Host labels that never name a vendor on their own.\n_GENERIC_HOST_LABELS = frozenset({\n \"www\", \"docs\", \"doc\", \"developer\", \"developers\", \"dev\", \"api\", \"help\", \"support\", \"wiki\",\n \"blog\", \"news\", \"learn\", \"security\", \"com\", \"org\", \"net\", \"io\", \"dev\", \"gov\", \"edu\",\n \"co\", \"uk\", \"de\", \"app\", \"cloud\", \"github\", \"medium\", \"reddit\", \"youtube\", \"stackoverflow\",\n})\n\n\n# Social profiles answer no docs, security, academic or news question; in those\n# classes a spare result from the overfetch replaces them in the top N.\n_SOCIAL_HOSTS = (\"facebook.com\", \"instagram.com\", \"linkedin.com\", \"tiktok.com\", \"pinterest.com\", \"threads.net\")\n_SOCIAL_SINK_CLASSES = frozenset({\"docs\", \"security\", \"academic\", \"news\"})\n# Classes where the source the query names (\"Bundestag\" -> bundestag.de) is the\n# primary answer; one such spare may take the last top-N slot.\n_VENDOR_PROMOTE_CLASSES = frozenset({\"docs\", \"security\", \"news\"})\n\n\ndef _is_social(url: str) -> bool:\n return any(_domain_matches_rule(_result_domain(url), host) for host in _SOCIAL_HOSTS)\n\n\ndef _query_terms(query: str) -> set:\n return {term for term in re.findall(r\"[a-z0-9]+\", (query or \"\").lower()) if len(term) >= 3}\n\n\n# Public suffixes a vendor's own site plausibly uses. Novelty TLDs\n# (.website, .wiki, ...) host look-alikes, so they never count as the vendor.\n_VENDOR_TLDS = frozenset({\n \"com\", \"org\", \"net\", \"io\", \"dev\", \"rs\", \"sh\", \"ai\", \"app\", \"gov\", \"eu\",\n \"de\", \"at\", \"ch\", \"fr\", \"uk\", \"jp\", \"us\", \"info\",\n})\n\n\ndef _is_named_vendor_domain(url: str, query_terms: set) -> bool:\n \"\"\"True when the registrable domain of the result is named by the query.\n\n Only the label left of the public suffix counts (``fastapi`` in\n fastapi.tiangolo.com is a subdomain of tiangolo and does not; neither does\n ``injection`` in injection.readthedocs.io). ``typescriptlang.org`` matches\n \"typescript\" because the label starts with the query word.\n \"\"\"\n domain = _result_domain(url)\n if not domain or not query_terms:\n return False\n labels = [label for label in domain.split(\".\") if label]\n if len(labels) \u003c 2 or labels[-1] not in _VENDOR_TLDS:\n return False\n core = labels[-3] if len(labels) >= 3 and labels[-2] in {\"co\", \"com\", \"gov\", \"org\"} else labels[-2]\n if core in _GENERIC_HOST_LABELS:\n return False\n return any(core == term or (len(term) >= 4 and core.startswith(term)) for term in query_terms)\n\n\ndef rerank_results_for_intent(\n query: str,\n routing_class: str,\n results: List[Dict[str, Any]],\n window: Optional[int] = None,\n) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:\n \"\"\"Small authority reranker for classes where source authority beats snippet luck.\n\n ``window`` limits the rerank to the first N results; the rest keep their\n order behind it. Spare results fetched only to refill filtered slots must\n not jump ahead of the provider's own top N.\n \"\"\"\n if window is not None and 0 \u003c window \u003c len(results):\n head, rest = results[:window], results[window:]\n if routing_class in _SOCIAL_SINK_CLASSES:\n social = [item for item in head if _is_social(item.get(\"url\", \"\"))]\n spares = [item for item in rest if not _is_social(item.get(\"url\", \"\"))]\n if social and spares:\n kept = [item for item in head if not _is_social(item.get(\"url\", \"\"))]\n refill = spares[: len(social)]\n head = kept + refill\n rest = [item for item in rest if item not in refill] + social\n if routing_class in _VENDOR_PROMOTE_CLASSES:\n terms = _query_terms(query)\n if not any(_is_named_vendor_domain(item.get(\"url\", \"\"), terms) for item in head):\n vendor = next((item for item in rest if _is_named_vendor_domain(item.get(\"url\", \"\"), terms)), None)\n if vendor is not None:\n rest = [head[-1]] + [item for item in rest if item is not vendor]\n head = head[:-1] + [vendor]\n head, meta = rerank_results_for_intent(query, routing_class, head)\n return head + [item.copy() for item in rest], meta\n rules = CANONICAL_DOMAIN_RULES.get(routing_class, {})\n if not results or not rules:\n return results, {\"reranked\": False, \"routing_class\": routing_class}\n\n q = query.lower()\n query_terms = _query_terms(query)\n scored: List[Tuple[float, int, Dict[str, Any]]] = []\n for idx, item in enumerate(results):\n url = item.get(\"url\", \"\")\n title = (item.get(\"title\") or \"\").lower()\n snippet = (item.get(\"snippet\") or item.get(\"description\") or \"\").lower()\n score = float(len(results) - idx) * 0.01\n if _is_named_vendor_domain(url, query_terms):\n # The project or vendor the query names is the primary source and\n # outranks generic hosts such as github.com or readthedocs.io:\n # \"nginx proxy_pass\" -> nginx.org, \"Ivanti CVE\" -> ivanti.com.\n score += 12.0\n elif any(_url_matches_rule(url, rule) for rule in rules.get(\"boost\", [])):\n score += 10.0\n if any(_url_matches_rule(url, rule) for rule in rules.get(\"demote\", [])):\n score -= 16.0 if score >= 10.0 else 6.0\n if \"official\" in q and (\"official\" in title or \"official\" in snippet):\n score += 1.0\n scored.append((score, idx, item))\n\n reranked = [item.copy() for _, _, item in sorted(scored, key=lambda row: (-row[0], row[1]))]\n before_urls = [item.get(\"url\", \"\") for item in results]\n after_urls = [item.get(\"url\", \"\") for item in reranked]\n changed = before_urls != after_urls\n return reranked, {\n \"reranked\": changed,\n \"routing_class\": routing_class,\n \"top_domain_before\": _result_domain(results[0].get(\"url\", \"\")) if results else None,\n \"top_domain_after\": _result_domain(reranked[0].get(\"url\", \"\")) if reranked else None,\n }\n\n\ndef build_authority_signals(routing_class: str, results: List[Dict[str, Any]]) -> Dict[str, Any]:\n \"\"\"Summarize primary-source authority signals for quality reports.\"\"\"\n rules = CANONICAL_DOMAIN_RULES.get(routing_class, {})\n urls = [item.get(\"url\", \"\") for item in results if item.get(\"url\")]\n domains = [_result_domain(url) for url in urls]\n boosted_domains = []\n demoted_domains = []\n boosted_flags = []\n for url, domain in zip(urls, domains):\n boosted = any(_url_matches_rule(url, rule) for rule in rules.get(\"boost\", []))\n demoted = any(_url_matches_rule(url, rule) for rule in rules.get(\"demote\", []))\n boosted_flags.append(boosted)\n if boosted:\n boosted_domains.append(domain)\n if demoted:\n demoted_domains.append(domain)\n\n return {\n \"routing_class\": routing_class,\n \"rules_applied\": bool(rules),\n \"top_domain\": domains[0] if domains else None,\n \"canonical_domain_hits\": sorted(set(boosted_domains)),\n \"demoted_domain_hits\": sorted(set(demoted_domains)),\n \"canonical_top_result\": bool(boosted_flags and boosted_flags[0]),\n }\n\n\ndef _snippet_text(item: Dict[str, Any]) -> str:\n return \" \".join(\n str(item.get(k) or \"\")\n for k in (\"description\", \"snippet\", \"content\", \"raw_content\", \"summary\")\n ).strip()\n\n\ndef build_quality_report(\n query: str,\n result: Dict[str, Any],\n routing_info: Dict[str, Any],\n providers_considered: List[str],\n eligible_providers: List[str],\n cooldown_skips: List[Dict[str, Any]],\n errors: List[Dict[str, Any]],\n near_duplicate_threshold: float = DEFAULT_NEAR_DUPLICATE_THRESHOLD,\n) -> Dict[str, Any]:\n \"\"\"Build transparent search-quality diagnostics without changing results.\"\"\"\n results = result.get(\"results\", []) or []\n domains = [_result_domain(r.get(\"url\", \"\")) for r in results]\n domains = [d for d in domains if d]\n unique_domains = sorted(set(domains))\n duplicate_count = int(result.get(\"metadata\", {}).get(\"dedup_count\", 0) or 0)\n\n short_snippets = 0\n for item in results:\n if len(_snippet_text(item)) \u003c 40:\n short_snippets += 1\n\n extract_reasons: List[str] = []\n confidence_level = routing_info.get(\"confidence_level\") or \"unknown\"\n confidence_score = routing_info.get(\"confidence\")\n if confidence_level == \"low\" or (confidence_score is not None and float(confidence_score or 0) \u003c 0.4):\n extract_reasons.append(\"low routing confidence\")\n if len(results) \u003c 3:\n extract_reasons.append(\"few search results\")\n if results and len(unique_domains) \u003c= 1:\n extract_reasons.append(\"low domain diversity\")\n if duplicate_count:\n extract_reasons.append(\"duplicate results detected\")\n if results and short_snippets / max(len(results), 1) >= 0.5:\n extract_reasons.append(\"thin snippets\")\n\n skipped = []\n for item in cooldown_skips:\n skipped.append({\n \"provider\": item.get(\"provider\"),\n \"reason\": \"cooldown\",\n \"cooldown_remaining_seconds\": item.get(\"cooldown_remaining_seconds\"),\n })\n for err in errors:\n skipped.append({\n \"provider\": err.get(\"provider\"),\n \"reason\": \"error\",\n \"error\": err.get(\"error\"),\n })\n\n routing_class = routing_info.get(\"analysis_summary\", {}).get(\"routing_class\")\n authority_signals = build_authority_signals(routing_class, results) if routing_class else None\n\n report = {\n \"query\": query,\n \"selected_provider\": routing_info.get(\"provider\") or result.get(\"provider\"),\n \"routing_reason\": routing_info.get(\"reason\"),\n \"routing_policy\": routing_info.get(\"routing_policy\", ROUTING_POLICY),\n \"routing_class\": routing_class,\n \"language_hint\": routing_info.get(\"analysis_summary\", {}).get(\"language_hint\"),\n\n \"confidence\": confidence_level,\n \"confidence_score\": routing_info.get(\"confidence\"),\n \"providers_considered\": providers_considered,\n \"eligible_providers\": eligible_providers,\n \"skipped_providers\": skipped,\n \"result_count\": len(results),\n \"domain_count\": len(unique_domains),\n \"domains\": unique_domains,\n \"domain_diversity\": (len(unique_domains) / len(results)) if results else 0.0,\n \"duplicate_count\": duplicate_count,\n \"thin_snippet_count\": short_snippets,\n \"extract_recommended\": bool(extract_reasons),\n \"extract_reasons\": extract_reasons,\n \"scores\": routing_info.get(\"scores\", {}),\n \"authority_signals\": authority_signals,\n \"diversity\": score_diversity(\n results, near_duplicate_threshold=near_duplicate_threshold\n ),\n }\n truncated = (result.get(\"metadata\") or {}).get(\"query_truncated\")\n if truncated:\n report[\"query_truncated\"] = truncated\n return report\n\n\ndef select_research_providers(\n primary_provider: str,\n provider_priority: List[str],\n available_providers: set,\n max_providers: int = 3,\n) -> List[str]:\n \"\"\"Pick a compact provider set for research mode.\"\"\"\n preferred = [primary_provider, \"linkup\", \"tavily\", \"exa\", \"firecrawl\", \"brave\", \"serper\", \"you\", \"querit\"]\n ordered: List[str] = []\n for provider in preferred + provider_priority:\n if provider and provider in available_providers and provider not in ordered:\n ordered.append(provider)\n if len(ordered) >= max_providers:\n break\n return ordered\n","content_type":"text/x-python; charset=utf-8","language":"python","size":19810,"content_sha256":"712fc0398d68c53f7c3b2d8aeea447ba9da7d13914e7955cd442dadbd778f32d"},{"filename":"scripts/wsp_core/query_limits.py","content":"\"\"\"Per-provider query length limits.\n\nA provider that documents a hard limit rejects a longer query with a 4xx. The\nsite operators WSP appends for domain filters count towards that limit, so the\nlimit is applied to the final query: the free text is shortened at a word\nboundary, the operators stay.\n\"\"\"\n\nfrom __future__ import annotations\n\nfrom typing import Any, Dict, List, NamedTuple, Optional, Sequence, Tuple\n\nMAX_QUERY_CHARS = 2000 # generic cap in the engine entry, before any provider\n\n\nclass QueryLimit(NamedTuple):\n chars: int\n words: int\n\n\n# Only limits the provider documents. Do not add a guess.\nPROVIDER_QUERY_LIMITS: Dict[str, QueryLimit] = {\n \"brave\": QueryLimit(chars=600, words=75), # Brave Web Search API: q (max 600 chars, 75 words)\n}\n\n\ndef _fits(text: str, limit: QueryLimit) -> bool:\n return len(text) \u003c= limit.chars and len(text.split()) \u003c= limit.words\n\n\ndef fit_query(\n provider: str, text: str, operators: Sequence[str] = ()\n) -> Tuple[str, Optional[Dict[str, Any]]]:\n \"\"\"Return ``(query, info)`` for ``text`` plus appended ``operators``.\n\n ``info`` is None when the query is sent unchanged. Otherwise it describes\n the change (sizes only, never query text) for result metadata. Trailing\n ``-site:`` operators are dropped only when the operators alone leave no room\n for any free text; ``ValueError`` when the include filter alone is too long.\n \"\"\"\n text = text or \"\"\n ops: List[str] = [op for op in operators if op]\n full = \" \".join(p for p in [text, *ops] if p).strip()\n limit = PROVIDER_QUERY_LIMITS.get(provider)\n if limit is None or _fits(full, limit):\n return full, None\n\n words = text.split()\n dropped = 0\n while True:\n suffix = \" \".join(ops)\n sep = 1 if (suffix and words) else 0\n room_chars = limit.chars - len(suffix) - sep\n room_words = limit.words - len(suffix.split())\n floor = 1 if words else 0\n if room_chars >= floor and room_words >= floor:\n break\n if not ops or not ops[-1].startswith(\"-site:\"):\n raise ValueError(\n f\"Too many domain filters for {provider}: the query limit is \"\n f\"{limit.chars} characters / {limit.words} words\"\n )\n ops.pop()\n dropped += 1\n\n kept: List[str] = []\n used = 0\n for word in words:\n add = len(word) + (1 if kept else 0)\n if used + add > room_chars or len(kept) + 1 > room_words:\n break\n kept.append(word)\n used += add\n if words and not kept:\n kept = [words[0][:room_chars]] # one token longer than the limit\n query = \" \".join([*kept, *ops]).strip()\n\n info: Dict[str, Any] = {\n \"provider\": provider,\n \"limit_chars\": limit.chars,\n \"limit_words\": limit.words,\n \"original_chars\": len(full),\n \"original_words\": len(full.split()),\n \"sent_chars\": len(query),\n \"sent_words\": len(query.split()),\n }\n if dropped:\n info[\"operators_dropped\"] = dropped\n return query, info\n","content_type":"text/x-python; charset=utf-8","language":"python","size":3054,"content_sha256":"84b8d28bd6cc6378bcf916f054fe28fb0bc766b397e1572caf196035fbb7d0c5"},{"filename":"scripts/wsp_core/request_gate_v3.py","content":"\"\"\"Release-blocking source-only gates for provider registration and requests.\"\"\"\n\nfrom __future__ import annotations\n\nfrom collections.abc import Mapping, Sequence\nfrom typing import Any\n\nfrom .provider_registry import PROVIDER_SPECS\n\nSOURCE_ONLY_SEMANTICS = frozenset({\"source_results\", \"source_text\"})\n_BANNED_BODY_KEYS = frozenset(\n {\n \"messages\",\n \"system\",\n \"system_prompt\",\n \"answer\",\n \"include_answer\",\n \"synthesis\",\n \"reasoning\",\n \"claim\",\n \"verification\",\n }\n)\n_BANNED_INSTRUCTION_FRAGMENTS = (\n \"answer the user\",\n \"provide an answer\",\n \"synthesize\",\n \"reason step by step\",\n \"verify the claim\",\n)\n# Fields that carry the user's own text (search query, URL, domain filters).\n# They are data, not instructions WSP sends, so the fragment scan skips them:\n# \"best synthesizer keyboard\" is a legitimate query.\n_USER_TEXT_KEYS = frozenset(\n {\"query\", \"q\", \"url\", \"includedomains\", \"excludedomains\", \"include_domains\", \"exclude_domains\"}\n)\n\n\ndef validate_provider_mode(provider: str, capability: str) -> str:\n \"\"\"Return declared semantics or fail before provider selection/network I/O.\"\"\"\n spec = PROVIDER_SPECS.get(provider)\n if spec is None:\n raise ValueError(f\"unknown provider mode: {provider}\")\n if spec.rejected_reason:\n raise ValueError(f\"provider {provider} rejected: {spec.rejected_reason}\")\n if capability == \"search\":\n supported = spec.supports_search\n semantics = spec.search_output_semantics\n elif capability == \"extract\":\n supported = spec.supports_extract\n semantics = spec.extract_output_semantics\n else:\n raise ValueError(f\"unsupported capability: {capability}\")\n if not supported or semantics not in SOURCE_ONLY_SEMANTICS:\n raise ValueError(f\"provider {provider} has no source-only {capability} mode\")\n return semantics\n\n\ndef _walk(value: Any):\n if isinstance(value, Mapping):\n for key, child in value.items():\n yield str(key), child\n yield from _walk(child)\n elif isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)):\n for child in value:\n yield from _walk(child)\n\n\ndef validate_outbound_body(provider: str, body: Mapping[str, Any]) -> None:\n \"\"\"Reject answer-shaped outbound payloads before transport.\"\"\"\n for key, value in _walk(body):\n lowered = key.lower()\n if lowered in _BANNED_BODY_KEYS:\n if lowered == \"include_answer\" and value is False:\n continue\n raise ValueError(f\"source-only request gate rejected field: {key}\")\n if isinstance(value, str) and lowered not in _USER_TEXT_KEYS:\n text = value.lower()\n if any(fragment in text for fragment in _BANNED_INSTRUCTION_FRAGMENTS):\n raise ValueError(\"source-only request gate rejected answer instruction\")\n\n if provider == \"tavily\" and body.get(\"include_answer\") is not False:\n raise ValueError(\"tavily source-only mode requires include_answer=false\")\n if provider == \"linkup\" and body.get(\"outputType\") != \"searchResults\":\n raise ValueError(\"linkup source-only mode requires outputType=searchResults\")\n if provider == \"exa\" and body.get(\"type\") in {\"deep\", \"deep-reasoning\"}:\n raise ValueError(\"exa deep modes are not source-only\")\n","content_type":"text/x-python; charset=utf-8","language":"python","size":3388,"content_sha256":"40e42bb3031b712235178134bc05fbd6a667598c82b509666ad2b0efb73d306d"},{"filename":"scripts/wsp_core/research.py","content":"\"\"\"Research mode orchestration helpers.\"\"\"\n\nfrom __future__ import annotations\n\nimport queue\nimport threading\nimport time\nfrom concurrent.futures import TimeoutError as FuturesTimeoutError\nfrom typing import Any, Dict, List, Optional, Tuple\nfrom urllib.parse import urlparse\n\nfrom .daemon_tasks import DaemonTask\nfrom .diversity_v3 import DEFAULT_NEAR_DUPLICATE_THRESHOLD, rerank_duplicate_candidates\nfrom .quality import normalize_result_url\nfrom .span_extraction_v3 import select_spans\nfrom .urls import url_key\n\n\n# Small real-time grace given to already-submitted provider calls once the\n# (possibly fake-clock) budget reads as exhausted, so completed futures can\n# still be harvested without blocking on slow ones.\n_RESULT_GRACE_SECONDS = 0.25\n_DEFAULT_QUORUM_RESULT_TARGET_CAP = 5\n_DEFAULT_QUORUM_MIN_DOMAINS = 3\n# Reciprocal Rank Fusion constant (Cormack et al. 2009) and the extra weight\n# for each additional provider that returned the same page.\nRRF_K = 60\nAGREEMENT_BONUS = 0.25\n# Passages per extracted source. Two of up to 300 characters beat one of 500\n# in a blind usefulness judgment on the evaluation set.\nSOURCE_PASSAGES = 2\nSOURCE_PASSAGE_CHARS = 300\n\n\ndef source_passages(text: str, query: str) -> List[Dict[str, Any]]:\n \"\"\"Query-ranked passages of an extracted page, with character offsets.\"\"\"\n if not text or not query:\n return []\n try:\n spans = select_spans(text, query, max_spans=SOURCE_PASSAGES, max_span_chars=SOURCE_PASSAGE_CHARS)\n except Exception:\n return []\n return [\n {\"text\": span[\"text\"], \"start\": span[\"start\"], \"end\": span[\"end\"]}\n for span in spans\n if isinstance(span, dict) and str(span.get(\"text\") or \"\").strip()\n ]\n\n\ndef fuse_results(\n provider_results: List[Tuple[str, Dict[str, Any]]],\n max_results: int,\n *,\n k: int = RRF_K,\n agreement_bonus: float = AGREEMENT_BONUS,\n) -> Tuple[List[Dict[str, Any]], int]:\n \"\"\"Merge provider rankings with Reciprocal Rank Fusion.\n\n Each page scores sum(1 / (k + rank)) over the providers that returned it,\n times (1 + agreement_bonus * (providers - 1)). Pages are identified by\n urls.url_key, so URL variants count once. The kept copy is the one from\n the provider that ranked it highest; ``providers`` lists every provider\n that returned it. Returns the fused page list and how many duplicate\n results were merged away.\n \"\"\"\n scores: Dict[str, float] = {}\n best: Dict[str, Tuple[int, int, Dict[str, Any]]] = {}\n found_by: Dict[str, List[str]] = {}\n duplicates = 0\n for provider_index, (provider_name, payload) in enumerate(provider_results):\n seen_here = set()\n for rank, item in enumerate(payload.get(\"results\", []) or []):\n if not isinstance(item, dict):\n continue\n key = url_key(item.get(\"url\", \"\")) or f\"#{provider_index}:{rank}\"\n if key in seen_here:\n duplicates += 1\n continue\n seen_here.add(key)\n if key in scores:\n duplicates += 1\n scores[key] = scores.get(key, 0.0) + 1.0 / (k + rank + 1)\n found_by.setdefault(key, []).append(provider_name)\n if key not in best or (rank, provider_index) \u003c best[key][:2]:\n best[key] = (rank, provider_index, item)\n fused = []\n order = sorted(\n scores,\n key=lambda key: (\n -scores[key] * (1 + agreement_bonus * (len(found_by[key]) - 1)),\n best[key][0],\n best[key][1],\n ),\n )\n for key in order[: max(0, max_results)]:\n _rank, provider_index, item = best[key]\n merged = item.copy()\n merged[\"provider\"] = provider_results[provider_index][0]\n merged[\"providers\"] = list(found_by[key])\n fused.append(merged)\n return fused, duplicates\n\n\ndef _positive_int(value: Any, default: int, minimum: int = 1) -> int:\n \"\"\"Return a bounded integer without letting direct callers disable safety.\"\"\"\n if isinstance(value, bool):\n return default\n try:\n return max(minimum, int(value))\n except (TypeError, ValueError):\n return default\n\n\ndef _research_quorum_snapshot(\n provider_results: List[Tuple[str, Dict[str, Any]]],\n *,\n result_target: int,\n) -> Tuple[int, int, List[str]]:\n \"\"\"Return unique candidate/domain/provider counts for an early-return check.\n\n This intentionally mirrors stable provider order rather than arrival order.\n ``result_target`` is a minimum threshold, not a scan cap: all completed\n providers must remain able to contribute evidence even when the first one\n alone fills the public result page. Duplicate/empty responses still cannot\n manufacture a quorum.\n \"\"\"\n seen_urls = set()\n domains = set()\n contributors: List[str] = []\n candidate_count = 0\n for provider, payload in provider_results:\n contributed = False\n for item in payload.get(\"results\", []) or []:\n if not isinstance(item, dict):\n continue\n url = item.get(\"url\")\n if not isinstance(url, str) or not url.strip():\n continue\n normalized = normalize_result_url(url)\n if not normalized or normalized in seen_urls:\n continue\n seen_urls.add(normalized)\n candidate_count += 1\n contributed = True\n domain = urlparse(url).netloc.lower()\n if domain.startswith(\"www.\"):\n domain = domain[4:]\n if domain:\n domains.add(domain)\n if contributed:\n contributors.append(provider)\n return candidate_count, len(domains), contributors\n\n\ndef _quorum_metadata(\n provider_results: List[Tuple[str, Dict[str, Any]]],\n *,\n enabled: bool,\n min_contributing_providers: int,\n result_target: int,\n min_unique_domains: int,\n triggered: bool,\n) -> Dict[str, Any]:\n candidate_count, domain_count, contributors = _research_quorum_snapshot(\n provider_results, result_target=result_target\n )\n return {\n \"enabled\": enabled,\n \"triggered\": triggered,\n \"min_contributing_providers\": min_contributing_providers,\n \"result_target\": result_target,\n \"min_unique_domains\": min_unique_domains,\n \"contributing_providers\": contributors,\n \"deduplicated_result_count\": candidate_count,\n \"unique_domain_count\": domain_count,\n }\n\n\ndef run_research_mode(\n query: str,\n research_providers: List[str],\n execute_search,\n extract_urls,\n max_results: int,\n max_extract_urls: int = 3,\n time_budget_seconds: float | None = None,\n now_fn=None,\n max_workers: int | None = None,\n on_provider_timeout=None,\n diversity_rerank: bool = False,\n near_duplicate_threshold: float = DEFAULT_NEAR_DUPLICATE_THRESHOLD,\n quorum_enabled: bool = True,\n quorum_min_contributing_providers: int = 2,\n quorum_result_target_cap: int = _DEFAULT_QUORUM_RESULT_TARGET_CAP,\n quorum_min_unique_domains: int = _DEFAULT_QUORUM_MIN_DOMAINS,\n) -> Dict[str, Any]:\n \"\"\"Run broad search, deduplicate, then extract top sources for grounding.\n\n Research mode is intentionally best-effort: provider/extraction failures should\n produce diagnostics and partial search results instead of throwing away the\n whole response. Provider searches run concurrently to keep the wall-clock cost\n close to the slowest single provider rather than the sum of all of them. The\n optional time budget gates which providers are launched (checked before each\n submission, so a tight budget still skips later providers deterministically),\n bounds how long already-launched providers may run, and gates whether\n extraction runs at all — so the budget caps total wall-clock time instead of\n only limiting how many providers start.\n\n Results are harvested in completion order so a blocked early submission\n cannot hide later providers. Final provider/result ordering nevertheless\n stays in submission order for deterministic deduplication. Once at least\n two providers have contributed unique URL results, an opt-in-by-default\n conservative quorum may return early only after its capped result target\n and domain-diversity target are both met; otherwise every provider remains\n eligible and small/poor result sets retain their full recall.\n\n The completion-order/quorum design adapts ideas from the independent\n MIT-licensed Hound/Master-Fetch project by Bishesh Bhandari (dondai1234),\n reworked here around WSP's own provider, budget, and receipt contracts.\n \"\"\"\n provider_errors: List[Dict[str, Any]] = []\n now = now_fn or time.monotonic\n start = now()\n\n def remaining_budget() -> Optional[float]:\n if time_budget_seconds is None:\n return None\n return time_budget_seconds - (now() - start)\n\n # Submit providers (budget gate is sequential/deterministic); the actual\n # provider HTTP calls run concurrently on daemon threads. Daemon threads —\n # unlike ThreadPoolExecutor workers — are not joined at interpreter exit,\n # so an overdue provider cannot stall CLI/subprocess shutdown either.\n pending: List[Tuple[int, str]] = []\n tasks: Dict[int, DaemonTask] = {}\n completed = queue.Queue()\n workers = max_workers or max(1, len(research_providers))\n gate = threading.Semaphore(workers)\n\n # At most the requested result count is needed, but asking for a huge result\n # page must not turn the early-return quality bar into an arbitrary latency\n # deadline. The cap is an explicit operator setting and is deliberately\n # conservative at five by default.\n quorum_enabled = quorum_enabled is True\n quorum_min_contributing_providers = max(\n 2, _positive_int(quorum_min_contributing_providers, 2)\n )\n quorum_result_target_cap = _positive_int(\n quorum_result_target_cap, _DEFAULT_QUORUM_RESULT_TARGET_CAP\n )\n quorum_result_target = min(\n max(1, _positive_int(max_results, 1)), quorum_result_target_cap\n )\n quorum_min_unique_domains = min(\n quorum_result_target,\n _positive_int(quorum_min_unique_domains, _DEFAULT_QUORUM_MIN_DOMAINS),\n )\n\n def run_gated(provider_name: str) -> Dict[str, Any]:\n with gate:\n return execute_search(provider_name)\n\n for index, provider in enumerate(research_providers):\n remaining = remaining_budget()\n if remaining is not None and remaining \u003c= 0:\n provider_errors.append({\"provider\": provider, \"error\": \"skipped: research time budget exhausted\"})\n continue\n task = DaemonTask(run_gated, provider)\n tasks[index] = task\n pending.append((index, provider))\n task.add_done_callback(lambda _task, completed_index=index: completed.put(completed_index))\n\n results_by_index: Dict[int, Tuple[str, Dict[str, Any]]] = {}\n pending_by_index = {index: provider for index, provider in pending}\n quorum_triggered = False\n\n def provider_results_in_submission_order() -> List[Tuple[str, Dict[str, Any]]]:\n return [results_by_index[index] for index in sorted(results_by_index)]\n\n def harvest(index: int) -> None:\n \"\"\"Collect an already-completed provider without introducing a wait.\"\"\"\n provider = pending_by_index.pop(index, None)\n if provider is None:\n return\n try:\n payload = tasks[index].result(timeout=0)\n if not isinstance(payload, dict):\n raise TypeError(\"provider returned a non-object result\")\n results_by_index[index] = (provider, payload)\n except FuturesTimeoutError:\n # A completion callback and Event publication are ordered, but keep\n # this defensive branch non-blocking if a custom task ever differs.\n pending_by_index[index] = provider\n except Exception as e:\n provider_errors.append({\"provider\": provider, \"error\": str(e)})\n\n def drain_completed() -> None:\n \"\"\"Harvest all completions currently observable in arrival order.\"\"\"\n while True:\n try:\n index = completed.get_nowait()\n except queue.Empty:\n break\n harvest(index)\n # A callback is advisory. The done() fallback closes a tiny race between\n # worker completion and queue notification without waiting on any task.\n for index in sorted(pending_by_index):\n if tasks[index].done():\n harvest(index)\n\n def quorum_reached() -> bool:\n if not quorum_enabled:\n return False\n candidate_count, domain_count, contributors = _research_quorum_snapshot(\n provider_results_in_submission_order(), result_target=quorum_result_target\n )\n return (\n len(contributors) >= quorum_min_contributing_providers\n and candidate_count >= quorum_result_target\n and domain_count >= quorum_min_unique_domains\n )\n\n def mark_budget_timeouts() -> None:\n # A final non-blocking drain avoids classifying a just-finished daemon as\n # timed out. Remaining daemon tasks keep running but never block exit.\n drain_completed()\n for index in sorted(pending_by_index):\n provider = pending_by_index.pop(index)\n if on_provider_timeout is not None:\n on_provider_timeout(provider)\n provider_errors.append({\"provider\": provider, \"error\": \"timed out: research time budget exhausted\"})\n\n while pending_by_index:\n drain_completed()\n if not pending_by_index:\n break\n if quorum_reached():\n # Only incomplete tasks are preempted. Completed work was drained\n # above, preserving recall whenever it is already available.\n for index in sorted(pending_by_index):\n provider = pending_by_index.pop(index)\n provider_errors.append({\"provider\": provider, \"error\": \"preempted_after_quorum\"})\n quorum_triggered = True\n break\n\n remaining = remaining_budget()\n timeout = remaining\n if timeout is not None and timeout \u003c= 0:\n # Preserve the existing real-time grace for fake-clock callers and\n # races at the deadline; never wait beyond this short fixed grace.\n timeout = _RESULT_GRACE_SECONDS\n try:\n index = completed.get(timeout=timeout)\n except queue.Empty:\n drain_completed()\n if quorum_reached():\n continue\n current_remaining = remaining_budget()\n if current_remaining is not None and current_remaining \u003c= 0:\n mark_budget_timeouts()\n break\n # A spurious/expired wait with an injected clock: loop and derive a\n # fresh deadline instead of translating it into a provider timeout.\n continue\n harvest(index)\n\n # Completion timing must not leak into the public diagnostic order.\n provider_order = {\n provider: index for index, provider in enumerate(research_providers)\n }\n provider_errors.sort(\n key=lambda item: (\n provider_order.get(\n str(item.get(\"provider\") or \"\"), len(provider_order)\n ),\n str(item.get(\"error\") or \"\"),\n )\n )\n\n provider_results = provider_results_in_submission_order()\n\n diversity_duplicates = []\n if diversity_rerank:\n # The normal merge drops cross-provider URL duplicates. The explicit\n # diversity mode instead retains every candidate long enough to move\n # duplicate URLs and snippets behind the diverse head, then applies\n # the caller's ordinary result-count cap.\n merged_candidates: List[Dict[str, Any]] = []\n for provider_name, payload in provider_results:\n for item in payload.get(\"results\", []):\n if not isinstance(item, dict):\n continue\n candidate = item.copy()\n candidate.setdefault(\"provider\", provider_name)\n merged_candidates.append(candidate)\n reranked_candidates, diversity_duplicates = rerank_duplicate_candidates(\n merged_candidates, near_duplicate_threshold=near_duplicate_threshold\n )\n deduped = reranked_candidates[:max_results]\n dedup_count = 0\n else:\n deduped, dedup_count = fuse_results(provider_results, max_results)\n urls = [r.get(\"url\") for r in deduped if r.get(\"url\")][:max(0, max_extract_urls)]\n extracted = {\"provider\": None, \"results\": []}\n extraction_error = None\n if urls:\n remaining = remaining_budget()\n if remaining is not None and remaining \u003c= 0:\n extraction_error = \"skipped: research time budget exhausted\"\n elif remaining is None:\n try:\n extracted = extract_urls(urls) or {\"provider\": None, \"results\": []}\n except Exception as e:\n extraction_error = str(e)\n extracted = {\"provider\": None, \"results\": []}\n else:\n # Run extraction on a daemon thread so the remaining budget bounds it too.\n try:\n extracted = DaemonTask(extract_urls, urls).result(timeout=remaining) or {\"provider\": None, \"results\": []}\n except FuturesTimeoutError:\n extraction_error = \"timed out: research time budget exhausted\"\n extracted = {\"provider\": None, \"results\": []}\n except Exception as e:\n extraction_error = str(e)\n extracted = {\"provider\": None, \"results\": []}\n\n routing = {\n \"providers_queried\": [p for p, _ in provider_results],\n \"provider_errors\": provider_errors,\n \"extraction_provider\": extracted.get(\"provider\"),\n }\n if extraction_error:\n routing[\"extraction_error\"] = extraction_error\n\n source_summaries = []\n for summary in extracted.get(\"results\", []) or []:\n if isinstance(summary, dict):\n text = str(summary.get(\"content\") or summary.get(\"raw_content\") or \"\")\n summary = {**summary, \"passages\": source_passages(text, query)}\n source_summaries.append(summary)\n\n metadata = {\n \"dedup_count\": dedup_count,\n \"providers_merged\": [p for p, _ in provider_results],\n \"extracted_url_count\": len(source_summaries),\n \"research_quorum\": _quorum_metadata(\n provider_results,\n enabled=quorum_enabled,\n min_contributing_providers=quorum_min_contributing_providers,\n result_target=quorum_result_target,\n min_unique_domains=quorum_min_unique_domains,\n triggered=quorum_triggered,\n ),\n }\n if diversity_rerank:\n metadata[\"diversity_reranked\"] = len(diversity_duplicates)\n\n return {\n \"mode\": \"research\",\n \"provider\": \"research\",\n \"query\": query,\n \"results\": deduped,\n \"source_summaries\": source_summaries,\n \"routing\": routing,\n \"metadata\": metadata,\n }\n","content_type":"text/x-python; charset=utf-8","language":"python","size":19143,"content_sha256":"908ffcc412f9965e5efce445ca8606c1978256bce1602144d3752850b4663367"},{"filename":"scripts/wsp_core/routing.py","content":"\"\"\"Automatic provider routing for Web Search Plus.\n\nThe first provider for a query comes from its intent (``intents.classify_intent``)\nand a small measured table; the fallback chain after it follows\n``auto_routing.provider_priority``.\n\"\"\"\n\nimport re\nimport time\nfrom typing import Any, Dict, List, NamedTuple, Optional, Tuple\n\nfrom .config import (\n DEFAULT_CONFIG,\n apply_profile_effects,\n get_api_key,\n keyless_public_allowed,\n self_hosted_profile_error,\n)\nfrom .intents import classify_intent\nfrom .provider_registry import DEFAULT_AUTO_ALLOW, DEFAULT_PROVIDER_PRIORITY, PROVIDER_SPECS\n\n\ndef provider_configured(provider: str, config: Dict[str, Any] | None = None) -> bool:\n \"\"\"Whether a provider can run: keyed via this module's (sync-patchable)\n ``get_api_key`` binding, or keyless with its public endpoint opted in.\"\"\"\n try:\n if get_api_key(provider, config):\n return True\n except ValueError:\n # A SearXNG URL that fails validation (private address without\n # SEARXNG_ALLOW_PRIVATE=1, bad scheme) means \"not configured\" here, so\n # one bad URL cannot break eligibility for every other provider. An\n # explicit provider=\"searxng\" call still raises the blocked-URL error.\n return False\n return keyless_public_allowed(provider, config)\n\n\nROUTING_POLICY = \"routing-v3\"\n\n\ndef iter_all_selectable_provider_modes() -> Tuple[str, ...]:\n \"\"\"Enumerate every search mode that routing/fallback is allowed to select.\"\"\"\n return tuple(\n provider\n for provider, spec in PROVIDER_SPECS.items()\n if spec.supports_search\n and not spec.rejected_reason\n and spec.search_output_semantics == \"source_results\"\n )\n\n\n# ---------------------------------------------------------------------------\n# Query-language detection (one detector for routing and locale resolution)\n# ---------------------------------------------------------------------------\n# detect_query_language serves two callers from the same evidence. The hint is\n# reported in routing metadata (\"en\" when nothing matches). The inference used\n# by search_locale.resolve_locale counts distinct stopword/character signals per\n# language and only reports a language when the evidence is unambiguous. Short\n# keyword or technical queries (for example \"PostgreSQL 17 release notes\")\n# produce no inference on purpose.\n\n# Minimum number of distinct signals before a language inference is trusted.\nLANGUAGE_INFERENCE_MIN_MATCHES = 2\n\n# Common function/search words per supported language. Words shared between\n# languages (e.g. \"que\" in es/fr/pt) may appear in several sets; the strict\n# single-winner rule in detect_query_language keeps those from mis-firing.\nLANGUAGE_INFERENCE_STOPWORDS: Dict[str, frozenset] = {\n \"en\": frozenset({\n \"the\", \"and\", \"what\", \"how\", \"where\", \"when\", \"which\", \"who\",\n \"best\", \"near\", \"hours\", \"open\", \"with\", \"from\", \"for\", \"are\",\n \"is\", \"was\", \"does\", \"latest\", \"today\", \"new\",\n }),\n \"de\": frozenset({\n \"der\", \"die\", \"das\", \"und\", \"oder\", \"nicht\", \"ist\", \"sind\",\n \"ein\", \"eine\", \"einen\", \"mit\", \"für\", \"von\", \"wie\", \"wo\", \"was\",\n \"warum\", \"welche\", \"beste\", \"besten\", \"gibt\", \"öffnungszeiten\",\n \"heute\", \"morgen\", \"preis\", \"kaufen\", \"günstig\", \"nähe\",\n }),\n \"es\": frozenset({\n \"el\", \"los\", \"las\", \"una\", \"unos\", \"que\", \"qué\", \"cómo\", \"dónde\",\n \"cuál\", \"por\", \"para\", \"con\", \"mejores\", \"mejor\", \"cerca\", \"hoy\",\n \"horario\", \"horarios\", \"abierto\", \"abiertos\", \"tiendas\",\n \"restaurantes\", \"precio\", \"precios\", \"donde\", \"como\",\n }),\n \"fr\": frozenset({\n \"le\", \"les\", \"des\", \"une\", \"du\", \"où\", \"quel\", \"quelle\", \"quels\",\n \"quelles\", \"meilleur\", \"meilleure\", \"meilleurs\", \"meilleures\",\n \"horaires\", \"ouvert\", \"ouverts\", \"ouverture\", \"aujourd\", \"hui\",\n \"près\", \"proche\", \"avec\", \"pour\", \"prix\", \"cher\", \"que\",\n }),\n \"it\": frozenset({\n \"il\", \"lo\", \"gli\", \"che\", \"come\", \"dove\", \"quale\", \"quali\",\n \"migliori\", \"migliore\", \"orari\", \"orario\", \"aperto\", \"aperti\",\n \"vicino\", \"con\", \"oggi\", \"prezzo\", \"prezzi\", \"negozi\",\n \"ristoranti\", \"della\", \"delle\",\n }),\n \"pt\": frozenset({\n \"os\", \"do\", \"dos\", \"das\", \"um\", \"uma\", \"que\", \"como\", \"onde\",\n \"qual\", \"quais\", \"melhores\", \"melhor\", \"horários\", \"aberto\",\n \"perto\", \"hoje\", \"preço\", \"lojas\", \"com\", \"você\", \"para\",\n \"restaurantes\",\n }),\n \"nl\": frozenset({\n \"het\", \"een\", \"waar\", \"hoe\", \"welke\", \"beste\", \"goedkoop\",\n \"goedkoopste\", \"vandaag\", \"morgen\", \"openingstijden\", \"winkel\",\n \"winkels\", \"dichtbij\", \"buurt\", \"naar\", \"zijn\", \"niet\", \"voor\",\n }),\n}\n\n# Distinctive characters that count as one additional signal per language.\nLANGUAGE_INFERENCE_CHAR_HINTS: Dict[str, str] = {\n \"de\": \"äöüß\",\n \"es\": \"ñ¿¡\",\n \"pt\": \"ãõ\",\n \"fr\": \"œ\",\n}\n\n\n# Scripts behind the routing hint, checked in this order before any Latin words.\n_ARABIC_SCRIPT = re.compile(r'[\\u0600-\\u06ff]')\n_CYRILLIC_SCRIPT = re.compile(r'[\\u0400-\\u04ff]')\n_KANA_SCRIPT = re.compile(r'[\\u3040-\\u30ff]')\n_JAPANESE_WORDS = re.compile(r'(東京|ニュース|今日|企業|発表)')\n_HAN_SCRIPT = re.compile(r'[\\u4e00-\\u9fff]')\n# Letters outside the base Russian / Arabic alphabets (Ukrainian, Persian,\n# Urdu, ...): the script alone then no longer identifies \"ru\" or \"ar\".\n_NON_RUSSIAN_CYRILLIC = re.compile(r'[\\u0400\\u0402-\\u040f\\u0450\\u0452-\\u04ff]')\n_NON_BASE_ARABIC = re.compile(r'[\\u0671-\\u06d3\\u06f0-\\u06f9]')\n\n# Single trigger words for the routing hint, first match wins (not used for\n# inference: \"france\" or \"die\" appear in English queries too).\n_ROUTING_HINT_TRIGGERS: Tuple[Tuple[str, Any], ...] = (\n (\"es\", re.compile(r'\\b(noticias|españa|hoy|regulación|inteligencia artificial)\\b')),\n (\"fr\", re.compile(r'\\b(actualités|france|aujourd|ouverts?|dimanche|récents?|avis)\\b')),\n (\"de\", re.compile(r'\\b(der|die|das|und|oder|nicht|ist|sind|aktuelle?n?|preis|kaufen|öffnungszeiten|österreich)\\b')),\n)\n\n\nclass QueryLanguage(NamedTuple):\n \"\"\"Result of detect_query_language.\n\n ``hint`` always carries a code (\"en\" when nothing matches) and feeds\n routing. ``inferred`` is None unless the evidence is unambiguous and feeds\n locale resolution.\n \"\"\"\n\n hint: str\n inferred: Optional[str]\n\n\n# Words that alone identify a language: none is an English word, and none is\n# shared by two of the listed languages. They only count when the query has\n# no English stopword and no other language matched, so \"Bundestag\n# Abstimmung heute Ergebnis\" is German while \"Avis car rental\" stays unknown.\nLANGUAGE_STRONG_WORDS: Dict[str, frozenset] = {\n \"de\": frozenset({\n \"heute\", \"öffnungszeiten\", \"warum\", \"welche\", \"günstig\", \"nähe\",\n \"erfahrungen\", \"aktuelle\", \"aktuell\", \"neueste\", \"unterschied\", \"wie\",\n \"wetter\", \"rezept\", \"vergleich\", \"empfehlung\", \"meinungen\", \"warnung\",\n \"schwachstelle\", \"sicherheitslücke\",\n }),\n \"fr\": frozenset({\n \"aujourd\", \"horaires\", \"près\", \"dernières\", \"météo\", \"demain\",\n \"vulnérabilité\", \"meilleur\", \"étude\",\n }),\n \"es\": frozenset({\"dónde\", \"cómo\", \"qué\", \"horarios\", \"últimas\", \"noticias\", \"opiniones\", \"revisión\"}),\n \"it\": frozenset({\"oggi\", \"orari\", \"notizie\", \"migliore\", \"opinioni\"}),\n \"pt\": frozenset({\"notícias\", \"hoje\", \"você\"}),\n \"nl\": frozenset({\"vandaag\", \"openingstijden\", \"werkt\"}),\n}\n\n\ndef _strong_word_language(words: set) -> Optional[str]:\n if words & LANGUAGE_INFERENCE_STOPWORDS[\"en\"]:\n return None\n matches = {language for language, strong in LANGUAGE_STRONG_WORDS.items() if words & strong}\n return matches.pop() if len(matches) == 1 else None\n\n\ndef _stopword_language(lowered: str) -> Optional[str]:\n \"\"\"Latin-script inference: stopword and character signals, single winner.\"\"\"\n words = set(re.findall(r\"\\w+\", lowered))\n counts: Dict[str, int] = {}\n for language, stopwords in LANGUAGE_INFERENCE_STOPWORDS.items():\n count = len(words & stopwords)\n count += sum(1 for char in LANGUAGE_INFERENCE_CHAR_HINTS.get(language, \"\") if char in lowered)\n if count:\n counts[language] = count\n if not counts:\n return _strong_word_language(words)\n ranked = sorted(counts.items(), key=lambda item: (-item[1], item[0]))\n best_language, best_count = ranked[0]\n if len(ranked) > 1 and ranked[1][1] == best_count:\n return None\n if best_count \u003c LANGUAGE_INFERENCE_MIN_MATCHES:\n strong = _strong_word_language(words)\n return strong if strong == best_language else None\n return best_language\n\n\ndef _script_language(text: str) -> Tuple[Optional[str], bool]:\n \"\"\"(routing hint, confident) for non-Latin scripts; (None, False) otherwise.\n\n Kana only occurs in Japanese. Han-only text (zh or ja) and the Japanese\n word list stay hint-only, as do Cyrillic/Arabic text with letters outside\n the Russian/Arabic alphabets.\n \"\"\"\n if _ARABIC_SCRIPT.search(text):\n return \"ar\", not _NON_BASE_ARABIC.search(text)\n if _CYRILLIC_SCRIPT.search(text):\n return \"ru\", not _NON_RUSSIAN_CYRILLIC.search(text)\n if _KANA_SCRIPT.search(text):\n return \"ja\", True\n if _JAPANESE_WORDS.search(text):\n return \"ja\", False\n if _HAN_SCRIPT.search(text):\n return \"zh\", False\n return None, False\n\n\ndef detect_query_language(query: Optional[str]) -> QueryLanguage:\n \"\"\"Detect the query language for routing (``hint``) and locale (``inferred``).\n\n ``inferred`` is an ISO 639-1 code only when the evidence is unambiguous:\n either a script that identifies the language, or at least\n LANGUAGE_INFERENCE_MIN_MATCHES distinct stopword/character signals for one\n language that strictly beats every other candidate. Conflicting evidence\n (for example kana inside an English sentence) infers nothing. For example\n \"Wiener Kaffeehaus Öffnungszeiten\" infers \"de\", while a terse technical\n query such as \"DAC R2R NOS\" infers nothing and callers fall back to their\n configured default.\n \"\"\"\n text = query or \"\"\n lowered = text.lower()\n script_hint, script_confident = _script_language(text)\n candidates = {_stopword_language(lowered)}\n if script_confident:\n candidates.add(script_hint)\n candidates.discard(None)\n inferred = candidates.pop() if len(candidates) == 1 else None\n if script_hint:\n return QueryLanguage(script_hint, inferred)\n hint = next((lang for lang, pattern in _ROUTING_HINT_TRIGGERS if pattern.search(lowered)), \"en\")\n return QueryLanguage(hint, inferred)\n\n\ndef infer_query_language(query: Optional[str]) -> Optional[str]:\n \"\"\"The unambiguous language of ``detect_query_language``, else None.\"\"\"\n return detect_query_language(query).inferred\n\n\n# ---------------------------------------------------------------------------\n# Routing: query intent -> first provider\n# ---------------------------------------------------------------------------\n# Measured on the recorded evaluation set: Brave gave\n# the best results overall (nDCG@5 0.696; Serper 0.642, Exa 0.638, Tavily\n# 0.552), Exa led on academic and documentation queries and Serper on shopping,\n# in every leave-one-out fold. Security goes to Serper after a live A/B of the\n# 26 security queries with authority domains: authority hit@5 22/26 on v4.3.5,\n# 16/26 with Brave first, 23/26 with Serper first, at fewer output tokens.\n# Community stays on Brave: on all 36 community queries Brave, Serper and\n# Firecrawl each hit the authority domain 15/15, with Brave at p50 640 ms /\n# p90 733 ms against Serper's 1067 / 1686 ms.\nINTENT_FIRST_PROVIDER: Dict[str, str] = {\n \"academic\": \"exa\",\n \"docs\": \"exa\",\n \"security\": \"serper\",\n \"shopping\": \"serper\",\n}\nMEASURED_PROVIDER_ORDER: Tuple[str, ...] = (\"brave\", \"serper\", \"exa\", \"tavily\")\n\n_RECENCY_PATTERNS: Tuple[Tuple[Any, float], ...] = tuple(\n (re.compile(pattern, re.IGNORECASE), weight)\n for pattern, weight in (\n (r\"\\b(latest|newest|recent|current)\\b\", 2.5),\n (r\"\\b(today|yesterday|this week|this month)\\b\", 3.0),\n (r\"\\b(breaking|live|just|now)\\b\", 3.0),\n (r\"\\blast (hour|day|week|month)\\b\", 2.5),\n (r\"\\b(hoy|aujourd|heute|aktuell)\\b\", 2.5),\n (r\"[今日最新]\", 2.5),\n (r\"(сегодня|новости)\", 2.5),\n (r\"(اليوم|أخبار)\", 2.5),\n (r\"(最新|今天)\", 2.5),\n )\n)\n_URL_IN_QUERY = re.compile(r\"https?://\\S+|\\b[\\w-]+\\.(?:com|org|net|io|dev|ai|de|at)/\\S*\", re.IGNORECASE)\n\n\ndef detect_recency(query: str, *, year: Optional[int] = None) -> Tuple[bool, float]:\n \"\"\"(wants recent information, score). The current year counts as a signal.\"\"\"\n text = query or \"\"\n total = sum(weight for pattern, weight in _RECENCY_PATTERNS if pattern.search(text))\n current = year if year is not None else time.gmtime().tm_year\n if re.search(rf\"\\b({current}|{current + 1})\\b\", text):\n total += 2.0\n return total > 2.0, total\n\n\ndef _auto_eligible(provider: str, config: Dict[str, Any], auto_config: Dict[str, Any], disabled: set) -> bool:\n return (\n provider not in disabled\n and _provider_auto_allowed(provider, auto_config)\n and provider_configured(provider, config)\n )\n\n\ndef route_query(query: str, config: Dict[str, Any]) -> Dict[str, Any]:\n \"\"\"First provider for ``query`` among the configured, auto-allowed ones.\"\"\"\n auto_config = config.get(\"auto_routing\", DEFAULT_CONFIG[\"auto_routing\"])\n disabled = set(auto_config.get(\"disabled_providers\", []))\n intent = classify_intent(query)\n priority = list(auto_config.get(\"provider_priority\", list(DEFAULT_PROVIDER_PRIORITY)))\n custom_order = auto_config.get(\"order\") == \"custom\"\n if custom_order:\n # The user's own order: provider_priority decides, no per-intent rule.\n preferred = list(priority)\n else:\n preferred = [INTENT_FIRST_PROVIDER.get(intent.intent), *MEASURED_PROVIDER_ORDER, *priority]\n first = next(\n (p for p in preferred if p and _auto_eligible(p, config, auto_config, disabled)), None\n )\n # The order an automatic search tries providers in: the first provider, then\n # the fallback chain in provider_priority order (search._plan_search_v3).\n order: List[str] = [first] if first else []\n for provider in priority:\n if provider not in order and _auto_eligible(provider, config, auto_config, disabled):\n order.append(provider)\n auto_excluded = [\n provider for provider in PROVIDER_SPECS\n if provider not in disabled\n and not _provider_auto_allowed(provider, auto_config)\n and provider_configured(provider, config)\n ]\n recency_focused, _ = detect_recency(query)\n analysis_summary = {\n \"query_length\": len(query.split()),\n \"has_url\": bool(_URL_IN_QUERY.search(query)),\n \"recency_focused\": recency_focused,\n \"language_hint\": detect_query_language(query).hint,\n \"routing_class\": intent.intent,\n \"intent_signals\": list(intent.signals),\n }\n confidence = round(intent.confidence, 3)\n routing = {\n \"confidence\": confidence,\n \"confidence_level\": \"high\" if confidence >= 0.7 else \"medium\" if confidence >= 0.4 else \"low\",\n \"reason\": \"custom_order\" if custom_order else (f\"intent_{intent.intent}\" if intent.signals else \"no_signals_matched\"),\n \"routing_policy\": ROUTING_POLICY,\n \"exa_depth\": \"normal\",\n \"scores\": {},\n \"top_signals\": [{\"matched\": signal, \"weight\": 1.0} for signal in intent.signals[:5]],\n \"candidate_order\": order,\n \"auto_allow_excluded\": auto_excluded,\n \"analysis_summary\": analysis_summary,\n }\n if not order:\n return {\n **routing,\n \"provider\": auto_config.get(\"fallback_provider\", \"serper\"),\n \"confidence\": 0.0,\n \"confidence_level\": \"low\",\n \"reason\": \"no_available_providers\",\n }\n return {**routing, \"provider\": order[0]}\n\n\ndef auto_route_provider(query: str, config: Dict[str, Any]) -> Dict[str, Any]:\n \"\"\"Route a query to its first provider; returns the routing metadata.\"\"\"\n config = apply_profile_effects(config)\n profile_error = self_hosted_profile_error(config)\n if profile_error is not None:\n return {\n \"provider\": None,\n \"confidence\": 0.0,\n \"confidence_level\": \"low\",\n \"reason\": profile_error.error_type,\n \"error\": str(profile_error),\n \"error_type\": profile_error.error_type,\n \"scores\": {},\n \"top_signals\": [],\n \"auto_routed\": True,\n }\n auto_config = config.get(\"auto_routing\", DEFAULT_CONFIG[\"auto_routing\"])\n if auto_config.get(\"enabled\", True) is False:\n default_provider = config.get(\"default_provider\")\n if default_provider:\n return {\n \"provider\": default_provider,\n \"confidence\": 1.0,\n \"confidence_level\": \"high\",\n \"reason\": \"auto_routing_disabled_default_provider\",\n \"scores\": {default_provider: 1.0},\n \"top_signals\": [],\n \"auto_routed\": False,\n }\n # Auto routing is off but nothing is pinned (the Desktop toggle cannot set\n # default_provider): use the configured priority instead of routing to\n # no provider. Set default_provider to pin one explicitly.\n decision = route_query(query, config)\n if decision.get(\"candidate_order\"):\n decision = {**decision, \"reason\": \"auto_routing_disabled_no_default_provider\"}\n return {**decision, \"auto_routed\": False}\n return route_query(query, config)\n\n\ndef explain_routing(query: str, config: Dict[str, Any]) -> Dict[str, Any]:\n \"\"\"Routing decision with the intent evidence behind it, for --explain-routing.\"\"\"\n config = apply_profile_effects(config)\n routing = auto_route_provider(query, config)\n intent = classify_intent(query)\n auto_config = config.get(\"auto_routing\", {})\n return {\n \"query\": query,\n \"routing_decision\": {\n \"provider\": routing.get(\"provider\"),\n \"confidence\": routing.get(\"confidence\"),\n \"confidence_level\": routing.get(\"confidence_level\"),\n \"reason\": routing.get(\"reason\"),\n \"routing_policy\": routing.get(\"routing_policy\", ROUTING_POLICY),\n \"candidate_order\": routing.get(\"candidate_order\", []),\n \"auto_allow_excluded\": routing.get(\"auto_allow_excluded\", []),\n },\n \"intent\": {\n \"intent\": intent.intent,\n \"confidence\": round(intent.confidence, 3),\n \"signals\": list(intent.signals),\n },\n \"first_provider_rules\": dict(INTENT_FIRST_PROVIDER),\n \"measured_order\": list(MEASURED_PROVIDER_ORDER),\n \"query_analysis\": routing.get(\"analysis_summary\", {}),\n \"available_providers\": [\n provider\n for provider, spec in PROVIDER_SPECS.items()\n if spec.supports_search\n and provider_configured(provider, config)\n and provider not in auto_config.get(\"disabled_providers\", [])\n and _provider_auto_allowed(provider, auto_config)\n ],\n }\n\n\ndef _provider_auto_allowed(provider: str, auto_config: Dict[str, Any]) -> bool:\n \"\"\"Return whether a configured provider may be selected by auto-routing/fallback.\n\n Explicit provider calls still work; this gate only prevents low-trust or\n experimental providers from receiving user queries automatically.\n \"\"\"\n auto_allow = auto_config.get(\"auto_allow\", {}) if isinstance(auto_config, dict) else {}\n default_allowed = bool(DEFAULT_AUTO_ALLOW.get(provider, True))\n if not isinstance(auto_allow, dict):\n return default_allowed\n return bool(auto_allow.get(provider, default_allowed))\n","content_type":"text/x-python; charset=utf-8","language":"python","size":19937,"content_sha256":"ab655d092cbf6fa147ed2d9251042182d094bb1687e78e96fca31df025636e01"},{"filename":"scripts/wsp_core/runtime_v3.py","content":"\"\"\"Normalize legacy provider-core payloads into the frozen ResponseV3 DTO.\"\"\"\n\nfrom __future__ import annotations\n\nimport hashlib\nimport unicodedata\nfrom copy import deepcopy\nfrom datetime import datetime\nfrom typing import Any, Dict, List, Mapping\nfrom urllib.parse import urlsplit\n\nfrom .contract_v3 import (\n AttemptOutcome,\n Capability,\n CircuitState,\n DegradedReason,\n ErrorClass,\n ErrorV3,\n FallbackReason,\n ProviderAttemptV3,\n RequestV3,\n ResponseStatus,\n ResponseV3,\n SkipReason,\n)\nfrom .orchestrator_v3 import ProviderPlan\nfrom .urls import canonical_url\n\n\ndef _stable_id(prefix: str, *parts: object) -> str:\n raw = \"\\x1f\".join(str(part) for part in parts)\n return f\"{prefix}_{hashlib.sha256(raw.encode()).hexdigest()[:16]}\"\n\n\ndef _valid_rfc3339(value: object) -> str | None:\n if not isinstance(value, str) or not value:\n return None\n try:\n parsed = datetime.fromisoformat(value.replace(\"Z\", \"+00:00\"))\n except ValueError:\n return None\n if parsed.tzinfo is None:\n return None\n return value\n\n\ndef _error(message: str, provider: str | None = None) -> ErrorV3:\n lowered = message.lower()\n if \"self_hosted profile\" in lowered:\n error_class = ErrorClass.CONFIG\n code = \"wsp.config.self_hosted_profile_unavailable\"\n elif \"missing\" in lowered and (\"key\" in lowered or \"credential\" in lowered):\n error_class = ErrorClass.CONFIG\n code = \"wsp.config.missing_credentials\"\n elif \"timeout\" in lowered or \"timed out\" in lowered:\n error_class = ErrorClass.TIMEOUT\n code = \"wsp.provider.timeout\"\n elif \"rate\" in lowered or \"429\" in lowered:\n error_class = ErrorClass.RATE_LIMIT\n code = \"wsp.provider.rate_limit\"\n elif \"security\" in lowered or \"blocked\" in lowered:\n error_class = ErrorClass.SECURITY\n code = \"wsp.security.request_blocked\"\n else:\n error_class = ErrorClass.TRANSIENT\n code = \"wsp.provider.failed\"\n return ErrorV3(\n error_class=error_class,\n code=code,\n message=message or \"Provider execution failed\",\n retryable=error_class\n in {ErrorClass.TIMEOUT, ErrorClass.RATE_LIMIT, ErrorClass.TRANSIENT},\n provider=provider,\n )\n\n\ndef _error_items(payload: Mapping[str, Any]) -> List[Dict[str, Any]]:\n values: List[Dict[str, Any]] = []\n for candidate in (\n payload.get(\"provider_errors\"),\n payload.get(\"fallback_errors\"),\n (payload.get(\"routing\") or {}).get(\"fallback_errors\"),\n ):\n if isinstance(candidate, list):\n for item in candidate:\n if isinstance(item, dict) and item not in values:\n values.append(dict(item))\n return values\n\n\ndef _attempts(\n request: RequestV3,\n plan: ProviderPlan,\n payload: Mapping[str, Any],\n selected: str | None,\n result_count: int,\n) -> List[ProviderAttemptV3]:\n attempts: List[ProviderAttemptV3] = []\n for index, item in enumerate(_error_items(payload), 1):\n provider = str(\n item.get(\"provider\")\n or plan.candidate_order[min(index - 1, len(plan.candidate_order) - 1)]\n )\n error = _error(str(item.get(\"error\") or \"Provider execution failed\"), provider)\n attempts.append(\n ProviderAttemptV3(\n attempt_id=f\"attempt-{index}\",\n provider=provider,\n capability=request.capability,\n outcome=AttemptOutcome.FAILED,\n result_count=0,\n error=error,\n circuit_state_before=CircuitState.UNKNOWN,\n circuit_state_after=CircuitState.UNKNOWN,\n )\n )\n if selected and (result_count or not payload.get(\"error\")):\n attempts.append(\n ProviderAttemptV3(\n attempt_id=f\"attempt-{len(attempts) + 1}\",\n provider=selected,\n capability=request.capability,\n outcome=AttemptOutcome.SUCCESS,\n result_count=result_count,\n circuit_state_before=CircuitState.UNKNOWN,\n circuit_state_after=CircuitState.CLOSED,\n )\n )\n return attempts\n\n\n_SNIPPET_SEPARATOR = \"\\n\\n\"\n_MAX_AGGREGATED_SNIPPET_CHARS = 600\n_AUTHORITATIVE_SOURCE_TYPES = frozenset({\"docs\", \"paper\", \"repo\", \"reference\"})\n\n\ndef segment_canonical_text(text: str) -> List[Dict[str, Any]]:\n \"\"\"Return contiguous NFC unicode-codepoint segments without rewriting text.\"\"\"\n text = unicodedata.normalize(\"NFC\", text)\n if not text:\n return []\n return [{\"start\": 0, \"end\": len(text), \"text\": text}]\n\n\ndef _projected_text(observation: Mapping[str, Any], source_field: str) -> Dict[str, Any] | None:\n value = observation.get(source_field)\n if not isinstance(value, str):\n return None\n value = unicodedata.normalize(\"NFC\", value)\n return {\n \"text\": value,\n \"text_sha256\": hashlib.sha256(value.encode(\"utf-8\")).hexdigest(),\n \"origin\": \"provider\",\n \"provenance\": {\n \"observation_id\": observation[\"observation_id\"],\n \"source_field\": source_field,\n \"transformations\": [\"mechanical_segmentation\"],\n },\n \"segments\": segment_canonical_text(value),\n }\n\n\ndef _observation_order(observation: Mapping[str, Any]) -> tuple[str, int, str]:\n \"\"\"Stable ordering never depends on arrival order or random attempt IDs.\"\"\"\n return (\n str(observation.get(\"provider\") or \"\"),\n int(observation.get(\"provider_result_index\") or 0),\n str(observation.get(\"observation_id\") or \"\"),\n )\n\n\ndef _source_type(url: str, provider_hint: object = None) -> Dict[str, str]:\n \"\"\"Return a transparent, deliberately heuristic source classification.\"\"\"\n parsed = urlsplit(url)\n host = (parsed.hostname or \"\").casefold()\n path = (parsed.path or \"\").casefold()\n if host in {\"github.com\", \"gitlab.com\", \"bitbucket.org\"}:\n return {\"value\": \"repo\", \"method\": \"url_heuristic\", \"method_version\": \"1\", \"confidence\": \"high\"}\n if host in {\"arxiv.org\", \"doi.org\", \"semanticscholar.org\", \"pubmed.ncbi.nlm.nih.gov\"} or path.endswith(\".pdf\"):\n return {\"value\": \"paper\", \"method\": \"url_heuristic\", \"method_version\": \"1\", \"confidence\": \"high\"}\n if host.startswith(\"docs.\") or \"/docs\" in path or \"/documentation\" in path:\n return {\"value\": \"docs\", \"method\": \"url_heuristic\", \"method_version\": \"1\", \"confidence\": \"high\"}\n if host in {\"wikipedia.org\", \"en.wikipedia.org\", \"developer.mozilla.org\"}:\n return {\"value\": \"reference\", \"method\": \"url_heuristic\", \"method_version\": \"1\", \"confidence\": \"high\"}\n if any(token in host for token in (\"reddit.\", \"stackoverflow.\", \"discourse.\", \"forum.\", \"community.\")):\n return {\"value\": \"forum\", \"method\": \"url_heuristic\", \"method_version\": \"1\", \"confidence\": \"high\"}\n if host.startswith(\"news.\") or \"/news\" in path:\n return {\"value\": \"news\", \"method\": \"url_heuristic\", \"method_version\": \"1\", \"confidence\": \"medium\"}\n if host.startswith(\"blog.\") or \"/blog\" in path:\n return {\"value\": \"blog\", \"method\": \"url_heuristic\", \"method_version\": \"1\", \"confidence\": \"medium\"}\n\n hint = str(provider_hint or \"\").casefold().replace(\"_\", \"-\")\n hint_map = {\n \"official-docs\": \"docs\", \"docs\": \"docs\", \"documentation\": \"docs\",\n \"paper\": \"paper\", \"repository\": \"repo\", \"repo\": \"repo\", \"blog\": \"blog\",\n \"forum\": \"forum\", \"reference\": \"reference\", \"news\": \"news\",\n }\n if hint in hint_map:\n return {\n \"value\": hint_map[hint],\n \"method\": \"provider_hint_normalized\",\n \"method_version\": \"1\",\n \"confidence\": \"medium\",\n }\n return {\"value\": \"other\", \"method\": \"url_heuristic\", \"method_version\": \"1\", \"confidence\": \"low\"}\n\n\ndef _aggregate_snippet(members: List[Dict[str, Any]]) -> Dict[str, Any] | None:\n \"\"\"Merge provider snippets mechanically, retaining all selected source links.\"\"\"\n candidates = [\n observation for observation in members\n if isinstance(observation.get(\"snippet\"), str) and observation[\"snippet\"]\n ]\n if not candidates:\n return None\n\n def normalized(observation: Mapping[str, Any]) -> str:\n return \" \".join(str(observation[\"snippet\"]).casefold().split())\n\n # Prefer containing fragments over shorter duplicates, then restore source\n # order for display. Both phases use only stable observation evidence.\n retained: List[Dict[str, Any]] = []\n retained_normalized: List[str] = []\n for candidate in sorted(candidates, key=lambda item: (-len(normalized(item)), _observation_order(item))):\n candidate_normalized = normalized(candidate)\n if any(\n candidate_normalized in existing or existing in candidate_normalized\n for existing in retained_normalized\n ):\n continue\n retained.append(candidate)\n retained_normalized.append(candidate_normalized)\n retained.sort(key=_observation_order)\n\n fragments: List[Dict[str, Any]] = []\n used_chars = 0\n for observation in retained:\n remaining = _MAX_AGGREGATED_SNIPPET_CHARS - used_chars - (\n len(_SNIPPET_SEPARATOR) if fragments else 0\n )\n if remaining \u003c= 0:\n break\n source_text = str(observation[\"snippet\"])\n text = source_text\n transformations = [\"mechanical_segmentation\"]\n if len(text) > remaining:\n text = text[:remaining]\n transformations.append(\"deterministic_truncation\")\n fragments.append(\n {\n \"observation_id\": observation[\"observation_id\"],\n \"source_field\": \"snippet\",\n \"text\": text,\n \"transformations\": transformations,\n }\n )\n used_chars += len(text) + (len(_SNIPPET_SEPARATOR) if len(fragments) > 1 else 0)\n if len(text) \u003c len(source_text):\n break\n if not fragments:\n return None\n text = _SNIPPET_SEPARATOR.join(fragment[\"text\"] for fragment in fragments)\n return {\n \"text\": text,\n \"text_sha256\": hashlib.sha256(text.encode(\"utf-8\")).hexdigest(),\n \"origin\": \"engine\",\n \"provenance\": {\"aggregation\": \"concat\", \"separator\": _SNIPPET_SEPARATOR, \"fragments\": fragments},\n \"segments\": segment_canonical_text(text),\n }\n\n\ndef _fetch_priority(engine_rank: int, members: List[Dict[str, Any]], source_type: Mapping[str, Any]) -> Dict[str, Any]:\n consensus = len({str(member.get(\"provider\")) for member in members}) >= 2\n authoritative = source_type.get(\"value\") in _AUTHORITATIVE_SOURCE_TYPES\n reason_codes = [\n \"cluster_consensus\" if consensus else \"cluster_single_observation\",\n \"rank_top_3\" if engine_rank \u003c= 3 else \"rank_beyond_top_3\",\n \"source_type_authoritative\" if authoritative else \"source_type_general\",\n ]\n score = (2 if consensus else 0) + (1 if engine_rank \u003c= 3 else 0) + (1 if authoritative else 0)\n return {\"tier\": \"high\" if score >= 3 else \"medium\" if score >= 1 else \"low\", \"reason_codes\": reason_codes}\n\n\ndef _observation(\n item: Mapping[str, Any],\n *,\n provider: str,\n endpoint_id: str,\n attempt_id: str,\n index: int,\n capability: Capability,\n) -> Dict[str, Any] | None:\n observed_url = str(item.get(\"url\") or \"\")\n if not observed_url:\n return None\n kind = \"search_result\" if capability is Capability.SEARCH else \"extracted_document\"\n snippet = (\n unicodedata.normalize(\"NFC\", str(item.get(\"snippet\") or \"\"))\n if capability is Capability.SEARCH\n else None\n )\n text = None\n if capability is Capability.EXTRACT and not item.get(\"error\"):\n text = unicodedata.normalize(\n \"NFC\", str(item.get(\"content\") or item.get(\"raw_content\") or \"\")\n )\n score = item.get(\"score\")\n provider_score = None\n if isinstance(score, (int, float)) and not isinstance(score, bool):\n provider_score = {\"value\": float(score), \"semantics\": \"unknown\"}\n raw_date = (\n item.get(\"published_at\")\n or item.get(\"published_date\")\n or item.get(\"date\")\n or item.get(\"publish_date\")\n or item.get(\"publishedDate\")\n or item.get(\"page_age\")\n or item.get(\"age\")\n )\n published_at = None\n if isinstance(raw_date, str):\n published_at = {\"raw\": raw_date, \"normalized\": _valid_rfc3339(raw_date)}\n return {\n \"observation_id\": _stable_id(\"obs\", attempt_id, index),\n \"provider_attempt_id\": attempt_id,\n \"provider_result_index\": index,\n \"provider\": provider,\n \"endpoint_id\": endpoint_id,\n \"kind\": kind,\n \"url\": {\"observed\": observed_url, \"canonical\": canonical_url(observed_url)},\n \"title\": (\n unicodedata.normalize(\"NFC\", str(item.get(\"title\")))\n if item.get(\"title\") is not None\n else None\n ),\n \"snippet\": snippet,\n \"text\": text,\n \"provider_rank\": index + 1,\n \"provider_score\": provider_score,\n \"published_at\": published_at,\n \"provider_fields\": {},\n \"source_type\": _source_type(observed_url, item.get(\"source_type\")),\n }\n\n\ndef observations_from_legacy(\n payload: Mapping[str, Any],\n provider: str,\n capability: Capability,\n attempt_id: str,\n) -> List[Dict[str, Any]]:\n endpoint_id = f\"{provider}:{capability.value}\"\n observations = []\n for index, item in enumerate(payload.get(\"results\") or []):\n observation = _observation(\n item,\n provider=provider,\n endpoint_id=endpoint_id,\n attempt_id=attempt_id,\n index=index,\n capability=capability,\n )\n if observation is not None:\n observations.append(observation)\n return observations\n\n\ndef project_results_from_observations(\n observations: List[Dict[str, Any]],\n selected_items: List[Mapping[str, Any]] | None = None,\n) -> List[Dict[str, Any]]:\n \"\"\"Project selected source clusters without deleting non-selected observations.\"\"\"\n representatives = []\n if selected_items is None:\n representatives = [\n (observation, [observation])\n for observation in sorted(observations, key=_observation_order)\n ]\n else:\n clusters: Dict[str, List[Dict[str, Any]]] = {}\n for observation in observations:\n clusters.setdefault(observation[\"url\"][\"canonical\"], []).append(observation)\n emitted = set()\n for item in selected_items:\n canonical = canonical_url(str(item.get(\"url\") or \"\"))\n members = sorted(clusters.get(canonical) or [], key=_observation_order)\n if not members or canonical in emitted:\n continue\n observed = str(item.get(\"url\") or \"\")\n representative = next(\n (\n observation\n for observation in members\n if observation[\"url\"][\"observed\"] == observed\n ),\n members[0],\n )\n representatives.append((representative, members))\n emitted.add(canonical)\n\n results = []\n for rank, (observation, members) in enumerate(representatives, 1):\n observation_id = observation[\"observation_id\"]\n cluster_id = _stable_id(\"cluster\", observation[\"url\"][\"canonical\"])\n source_type = dict(\n observation.get(\"source_type\")\n or _source_type(observation[\"url\"][\"canonical\"])\n )\n snippet = (\n _aggregate_snippet(members)\n if len(members) > 1\n else _projected_text(observation, \"snippet\")\n )\n results.append(\n {\n \"result_id\": _stable_id(\"result\", observation_id),\n \"kind\": observation[\"kind\"],\n \"engine_rank\": rank,\n \"representative_observation_id\": observation_id,\n \"observation_ids\": [member[\"observation_id\"] for member in members],\n \"dedup_cluster_id\": cluster_id,\n \"url\": dict(observation[\"url\"]),\n \"title\": _projected_text(observation, \"title\"),\n \"snippet\": snippet,\n \"text\": _projected_text(observation, \"text\"),\n \"source_type\": source_type,\n \"fetch_priority\": _fetch_priority(rank, members, source_type),\n }\n )\n return results\n\n\ndef _source_diversity(observations: List[Dict[str, Any]], results: List[Dict[str, Any]]) -> Dict[str, Any]:\n providers = {item[\"provider\"] for item in observations}\n hosts = {\n urlsplit(item[\"url\"][\"canonical\"]).hostname\n for item in observations\n if urlsplit(item[\"url\"][\"canonical\"]).hostname\n }\n clusters = {item[\"dedup_cluster_id\"] for item in results}\n return {\n \"method\": \"component_count\",\n \"method_version\": \"1\",\n \"method_degraded\": False,\n \"provider_count\": len(providers),\n \"host_count\": len(hosts),\n \"source_family_count\": len(providers),\n \"unique_cluster_count\": len(clusters),\n }\n\n\ndef response_from_legacy(\n request: RequestV3,\n plan: ProviderPlan,\n payload: Dict[str, Any],\n) -> ResponseV3:\n \"\"\"Build a contract-valid ResponseV3 without modifying the legacy payload.\"\"\"\n request_id = request.request_id or plan.execution_id\n routing = payload.get(\"routing\") or {}\n aggregate_research = payload.get(\"provider\") == \"research\"\n selected = (\n None\n if aggregate_research\n else routing.get(\"provider\") or payload.get(\"provider\") or plan.selected_provider\n )\n raw_items = [item for item in (payload.get(\"results\") or []) if item.get(\"url\")]\n observation_items = [\n item\n for item in (payload.get(\"_v3_raw_results\") or raw_items)\n if isinstance(item, Mapping) and item.get(\"url\")\n ]\n authoritative_attempts = payload.get(\"_v3_provider_attempts\")\n provider_attempts = (\n list(authoritative_attempts)\n if isinstance(authoritative_attempts, (list, tuple))\n else _attempts(request, plan, payload, selected, len(raw_items))\n )\n successful_attempts = [\n attempt for attempt in provider_attempts if attempt.outcome is AttemptOutcome.SUCCESS\n ]\n # Prefer the selected provider's attempt: per-URL extract fallback can\n # leave later successful attempts that contributed nothing.\n successful_attempt = next(\n (attempt for attempt in reversed(successful_attempts) if attempt.provider == selected),\n successful_attempts[-1] if successful_attempts else None,\n )\n # Per-URL extract fallback can merge items from several providers; each\n # item is then tied to its own provider's successful attempt.\n multi_source_extract = request.capability is Capability.EXTRACT and len(\n {attempt.provider for attempt in successful_attempts}\n & {str(item.get(\"provider\")) for item in raw_items if not item.get(\"error\")}\n ) > 1\n if (aggregate_research or multi_source_extract) and authoritative_attempts:\n observations = []\n for attempt in provider_attempts:\n if attempt.outcome is not AttemptOutcome.SUCCESS:\n continue\n provider_items = [\n item\n for item in observation_items\n if str(item.get(\"provider\") or \"\") == attempt.provider\n ]\n if provider_items:\n observations.extend(\n observations_from_legacy(\n {**payload, \"results\": provider_items},\n attempt.provider,\n request.capability,\n attempt.attempt_id,\n )\n )\n else:\n observations = (\n observations_from_legacy(\n {**payload, \"results\": observation_items},\n str(selected),\n request.capability,\n successful_attempt.attempt_id,\n )\n if selected and successful_attempt\n else []\n )\n results = project_results_from_observations(observations, raw_items)\n failed_items = [item for item in raw_items if item.get(\"error\")]\n top_error = payload.get(\"error\")\n warnings: List[Dict[str, Any]] = []\n error = None\n if top_error and not results:\n status = ResponseStatus.FAILED\n error = next(\n (\n attempt.error\n for attempt in provider_attempts\n if attempt.provider == selected and attempt.error is not None\n ),\n None,\n ) or next(\n (\n attempt.error\n for attempt in reversed(provider_attempts)\n if attempt.error is not None\n ),\n None,\n ) or _error(str(top_error), str(selected) if selected else None)\n elif failed_items:\n status = ResponseStatus.DEGRADED\n warnings.append(\n {\n \"code\": DegradedReason.PARTIAL_EXTRACTION.value,\n \"message\": \"One or more extraction results failed.\",\n \"details\": {\"failed_count\": len(failed_items)},\n }\n )\n else:\n status = ResponseStatus.OK\n\n # A hedge loser is cancelled because another attempt won, not by a budget.\n superseded = set(payload.get(\"_v3_superseded_providers\") or ())\n budget_limited = payload.get(\"_v3_budget_limited\") is True or any(\n (\n attempt.outcome is AttemptOutcome.CANCELLED\n and attempt.provider not in superseded\n )\n or attempt.skip_reason\n in {SkipReason.BUDGET_BLOCKED, SkipReason.DEADLINE_EXCEEDED}\n for attempt in provider_attempts\n )\n if status is not ResponseStatus.FAILED and budget_limited:\n status = ResponseStatus.DEGRADED\n warnings.append(\n {\n \"code\": DegradedReason.BUDGET_LIMITED.value,\n \"message\": \"Execution was limited by its attempt or time budget.\",\n }\n )\n\n if any(\n attempt.budget_decision == \"store_unavailable\"\n for attempt in provider_attempts\n ):\n warnings.append(\n {\n \"code\": \"wsp.state.store_unavailable\",\n \"message\": (\n \"Operational state unavailable; provider execution continued \"\n \"without persistent circuit or budget state.\"\n ),\n }\n )\n\n fallback_used = bool(routing.get(\"fallback_used\") or _error_items(payload))\n reported_reason = routing.get(\"fallback_reason\")\n fallback_reason = (\n reported_reason\n if fallback_used and reported_reason in {reason.value for reason in FallbackReason}\n else FallbackReason.SELECTED_FAILED.value\n if fallback_used\n else FallbackReason.NONE.value\n )\n if request.cache.get(\"mode\") == \"bypass\":\n cache_status = {\"disposition\": \"bypassed\"}\n else:\n cache_status = {\"disposition\": \"miss\"}\n\n clusters = [\n {\n \"dedup_cluster_id\": result[\"dedup_cluster_id\"],\n \"observation_ids\": list(result[\"observation_ids\"]),\n \"representative_observation_id\": result[\"representative_observation_id\"],\n }\n for result in results\n ]\n policy_actions = [\n {\n \"action\": \"selected_as_representative\",\n \"observation_id\": result[\"representative_observation_id\"],\n \"reason\": \"dedup_representative\",\n }\n for result in results\n ]\n selected_observation_ids = {\n observation_id\n for result in results\n for observation_id in result[\"observation_ids\"]\n }\n spam_domains = set(\n ((payload.get(\"metadata\") or {}).get(\"spam_filtered\") or {}).get(\"domains\")\n or []\n )\n for observation in observations:\n if observation[\"observation_id\"] in selected_observation_ids:\n continue\n host = urlsplit(observation[\"url\"][\"canonical\"]).hostname or \"\"\n if host in spam_domains:\n policy_actions.append(\n {\n \"action\": \"excluded\",\n \"observation_id\": observation[\"observation_id\"],\n \"reason\": \"spam_domain\",\n }\n )\n freshness = (payload.get(\"metadata\") or {}).get(\"freshness\")\n if isinstance(freshness, dict):\n warnings.append(\n {\n \"code\": \"wsp.freshness.applied\",\n \"message\": \"Provider recency application receipt (see details for applied status).\",\n \"details\": {\"freshness\": deepcopy(freshness)},\n }\n )\n return ResponseV3(\n request_id=request_id,\n execution_id=plan.execution_id,\n capability=request.capability,\n status=status,\n results=results,\n observations=observations,\n policy_actions=policy_actions,\n source_diversity=_source_diversity(observations, results),\n provider_attempts=provider_attempts,\n routing_receipt={\n \"policy_id\": \"classic\",\n \"policy_revision\": str(routing.get(\"routing_policy\") or \"v2.9.1\"),\n \"mode\": plan.mode,\n \"candidate_order\": list(plan.candidate_order),\n \"selected_provider\": selected if results else None,\n \"fallback_reason\": fallback_reason,\n },\n cache_status=cache_status,\n limits_applied={\"max_results\": request.options.get(\"max_results\")}\n if request.capability is Capability.SEARCH\n else {},\n dedup_clusters=clusters,\n warnings=warnings,\n error=error,\n )\n","content_type":"text/x-python; charset=utf-8","language":"python","size":25757,"content_sha256":"2287abd375f8268d23d7aebdae3dfa73c8dfcc9c52bf24e0b3c8eb0880b8108e"},{"filename":"scripts/wsp_core/sdk/__init__.py","content":"\"\"\"Public Provider SDK for Web Search Plus 3.x.\n\nThis API is additive-only throughout the 3.x series. Provider modules should\ndepend on this package rather than private registry or dispatch modules.\n\"\"\"\n\nimport importlib\nimport sys\n\nfrom .api import (\n ExtractExecute,\n ProviderSpec,\n SearchExecute,\n extract_result,\n make_extract_result,\n make_search_result,\n register_provider,\n search_result,\n source_result,\n)\nfrom .errors import (\n DuplicateProviderError,\n ProviderConfigError,\n ProviderContractFailure,\n ProviderDiscoveryError,\n ProviderRegistrationError,\n ProviderSDKError,\n ProviderStartupDiagnostic,\n)\nfrom ..http_client import ProviderRequestError\n\n__all__ = [\n \"DuplicateProviderError\",\n \"ExtractExecute\",\n \"ProviderConfigError\",\n \"ProviderContractFailure\",\n \"ProviderDiscoveryError\",\n \"ProviderRegistrationError\",\n \"ProviderRequestError\",\n \"ProviderSDKError\",\n \"ProviderSpec\",\n \"ProviderStartupDiagnostic\",\n \"SearchExecute\",\n \"extract_result\",\n \"make_extract_result\",\n \"make_search_result\",\n \"register_provider\",\n \"search_result\",\n \"source_result\",\n]\n\n# Submodules reachable under the public name: ``wsp_sdk.\u003cname>``.\n_PUBLIC_SUBMODULES = (\"api\", \"conformance\", \"errors\")\n\n\ndef _bind_public_name(name: str = \"wsp_sdk\") -> None:\n \"\"\"Make ``name`` and ``name.\u003csubmodule>`` resolve to this package's own modules.\n\n Left alone, ``import wsp_sdk.errors`` would find the file through the\n package ``__path__`` and execute it a second time under the public name:\n the provider would raise a distinct ``ProviderConfigError`` that the\n engine's ``isinstance`` checks miss, and ``conformance`` would lose the\n package its relative imports need. Registering the modules themselves keeps\n exactly one object per name in the process. Not part of the SDK surface.\n \"\"\"\n sys.modules[name] = sys.modules[__name__]\n for submodule in _PUBLIC_SUBMODULES:\n sys.modules[f\"{name}.{submodule}\"] = importlib.import_module(f\"{__name__}.{submodule}\")\n","content_type":"text/x-python; charset=utf-8","language":"python","size":2074,"content_sha256":"cbae551acefd6130f5ba4d598f8adb50f78ba06bfb8ac7cef49ec61689092bf8"},{"filename":"scripts/wsp_core/sdk/api.py","content":"\"\"\"Stable provider-spec and source-envelope helpers for WSP 3.x.\"\"\"\n\nfrom __future__ import annotations\n\nfrom dataclasses import dataclass\nfrom typing import Any, Callable, Mapping, Sequence\n\nfrom .errors import ProviderRegistrationError\n\n\nSearchExecute = Callable[[Any, str, Any, str | None, Mapping[str, Any], Mapping[str, Any]], dict[str, Any]]\nExtractExecute = Callable[\n [Any, str, list[str], str | None, str, bool, bool, bool, Mapping[str, Any], bool],\n dict[str, Any],\n]\n\n\n@dataclass(frozen=True, init=False)\nclass ProviderSpec:\n \"\"\"A stable description of one provider module.\n\n ``id``/``kind`` are the SDK spelling. ``provider`` and the capability\n booleans remain accepted for built-in compatibility during the 3.x line.\n New provider modules should use ``id`` and ``kind`` plus the matching\n ``execute_search`` and/or ``execute_extract`` callable.\n \"\"\"\n\n provider: str\n env_var: str\n display_name: str\n description: str\n config_section: str\n supports_search: bool\n supports_extract: bool\n capability_labels: tuple[str, ...]\n auto_allowed_by_default: bool\n recommended: bool\n free_tier: str\n signup_url: str\n upstream_capabilities: tuple[str, ...]\n keyless: bool\n search_output_semantics: str | None\n extract_output_semantics: str | None\n provider_fields_allowlist: tuple[str, ...]\n rejected_reason: str | None\n execute_search: SearchExecute | None\n execute_extract: ExtractExecute | None\n supports_freshness: bool\n production: bool\n kind: str\n\n def __init__(\n self,\n provider: str | None = None,\n env_var: str = \"\",\n display_name: str = \"\",\n description: str = \"\",\n config_section: str = \"\",\n supports_search: bool | None = None,\n supports_extract: bool | None = None,\n capability_labels: Sequence[str] = (),\n auto_allowed_by_default: bool = False,\n recommended: bool = False,\n free_tier: str = \"API key required\",\n signup_url: str = \"\",\n upstream_capabilities: Sequence[str] = (),\n keyless: bool = False,\n search_output_semantics: str | None = \"source_results\",\n extract_output_semantics: str | None = \"source_text\",\n provider_fields_allowlist: Sequence[str] = (),\n rejected_reason: str | None = None,\n *,\n id: str | None = None,\n kind: str | None = None,\n execute_search: SearchExecute | None = None,\n execute_extract: ExtractExecute | None = None,\n supports_freshness: bool = False,\n freshness_supported: bool | None = None,\n production: bool = True,\n ) -> None:\n identifier = id if id is not None else provider\n if id is not None and provider is not None and id != provider:\n raise ProviderRegistrationError(\"provider id and provider must match\")\n if not isinstance(identifier, str) or not identifier.strip():\n raise ProviderRegistrationError(\"provider id is required\")\n identifier = identifier.strip().lower()\n if kind is not None:\n if kind not in {\"search\", \"extract\", \"both\", \"disabled\"}:\n raise ProviderRegistrationError(\"kind must be search, extract, both, or disabled\")\n kind_search = kind in {\"search\", \"both\"}\n kind_extract = kind in {\"extract\", \"both\"}\n if supports_search is not None and bool(supports_search) != kind_search:\n raise ProviderRegistrationError(\"kind conflicts with supports_search\")\n if supports_extract is not None and bool(supports_extract) != kind_extract:\n raise ProviderRegistrationError(\"kind conflicts with supports_extract\")\n supports_search = kind_search\n supports_extract = kind_extract\n search_enabled = bool(supports_search)\n extract_enabled = bool(supports_extract)\n resolved_kind = (\n \"both\" if search_enabled and extract_enabled else \"search\" if search_enabled\n else \"extract\" if extract_enabled else \"disabled\"\n )\n if freshness_supported is not None:\n supports_freshness = bool(freshness_supported)\n\n values = {\n \"provider\": identifier,\n \"env_var\": str(env_var),\n \"display_name\": str(display_name),\n \"description\": str(description),\n \"config_section\": str(config_section),\n \"supports_search\": search_enabled,\n \"supports_extract\": extract_enabled,\n \"capability_labels\": tuple(str(label) for label in capability_labels),\n \"auto_allowed_by_default\": bool(auto_allowed_by_default),\n \"recommended\": bool(recommended),\n \"free_tier\": str(free_tier),\n \"signup_url\": str(signup_url),\n \"upstream_capabilities\": tuple(str(label) for label in upstream_capabilities),\n \"keyless\": bool(keyless),\n \"search_output_semantics\": search_output_semantics,\n \"extract_output_semantics\": extract_output_semantics,\n \"provider_fields_allowlist\": tuple(str(item) for item in provider_fields_allowlist),\n \"rejected_reason\": rejected_reason,\n \"execute_search\": execute_search,\n \"execute_extract\": execute_extract,\n \"supports_freshness\": bool(supports_freshness),\n \"production\": bool(production),\n \"kind\": resolved_kind,\n }\n for field_name, value in values.items():\n object.__setattr__(self, field_name, value)\n self._validate_source_only_semantics()\n\n @property\n def id(self) -> str:\n \"\"\"Stable SDK alias for the historical ``provider`` field.\"\"\"\n return self.provider\n\n @property\n def freshness_supported(self) -> bool:\n \"\"\"Alias kept for readable capability metadata.\"\"\"\n return self.supports_freshness\n\n def _validate_source_only_semantics(self) -> None:\n allowed = {\"source_results\", \"source_text\"}\n if self.supports_search and self.search_output_semantics not in allowed:\n raise ProviderRegistrationError(f\"{self.provider}: search mode is not source-only\")\n if self.supports_extract and self.extract_output_semantics not in allowed:\n raise ProviderRegistrationError(f\"{self.provider}: extract mode is not source-only\")\n\n\ndef register_provider(spec: ProviderSpec) -> ProviderSpec:\n \"\"\"Validate and return a module-level provider declaration.\n\n Discovery registers the returned ``PROVIDER`` declaration atomically after\n its module loads. Keeping this helper side-effect-free lets the same\n self-contained provider module be imported safely by documentation and\n test tooling before the plugin registry is initialized.\n \"\"\"\n if not isinstance(spec, ProviderSpec):\n raise ProviderRegistrationError(\"register_provider requires ProviderSpec\")\n return spec\n\n\ndef source_result(url: str, **fields: Any) -> dict[str, Any]:\n \"\"\"Build one source result item for a search or extraction envelope.\"\"\"\n result = {\"url\": url}\n result.update(fields)\n return result\n\n\ndef search_result(\n provider: str,\n query: str,\n results: Sequence[Mapping[str, Any]],\n *,\n images: Sequence[Mapping[str, Any]] | None = None,\n metadata: Mapping[str, Any] | None = None,\n) -> dict[str, Any]:\n \"\"\"Build a source-only search envelope accepted by WSP's protocol gate.\"\"\"\n return {\n \"provider\": provider,\n \"query\": query,\n \"results\": [dict(item) for item in results],\n \"images\": [dict(item) for item in images] if images is not None else [],\n \"metadata\": dict(metadata) if metadata is not None else {},\n }\n\n\ndef extract_result(\n provider: str,\n results: Sequence[Mapping[str, Any]],\n) -> dict[str, Any]:\n \"\"\"Build a source-only extraction envelope accepted by WSP's protocol gate.\"\"\"\n return {\"provider\": provider, \"results\": [dict(item) for item in results]}\n\n\n# Explicit aliases make the constructors easy to discover without breaking the\n# concise names in the 3.1 SDK guide.\nmake_search_result = search_result\nmake_extract_result = extract_result\n","content_type":"text/x-python; charset=utf-8","language":"python","size":8178,"content_sha256":"803cf23ffad8f932a42092f8106f4f9f6009cd8e824667a9ec5a35069a3eca94"},{"filename":"scripts/wsp_core/sdk/conformance.py","content":"\"\"\"Network-free conformance checks shared by built-in and SDK providers.\"\"\"\n\nfrom __future__ import annotations\n\nimport json\nimport os\n\nfrom .api import extract_result, search_result, source_result\n\n\ndef provider_conformance_errors() -> tuple[str, ...]:\n \"\"\"Return deterministic conformance failures for every registered spec.\n\n This deliberately exercises metadata, formal dispatch membership, envelope\n validation, keyless/env consistency, and missing-key classification without\n making a provider request. It is equally strict for built-ins and modules\n discovered from ``providers.d``.\n \"\"\"\n # Imported here: the SDK registers this module at engine start-up, and\n # unittest.mock is not worth loading before a conformance run.\n from unittest.mock import patch\n\n from ..config import ProviderConfigError, validate_api_key\n from ..provider_adapter_protocol import dispatch_conformance_errors, validate_adapter_result\n from ..provider_dispatch import EXTRACT_DISPATCH, SEARCH_DISPATCH\n from ..provider_registry import PROVIDER_SPECS\n\n errors = list(dispatch_conformance_errors(SEARCH_DISPATCH, EXTRACT_DISPATCH, PROVIDER_SPECS))\n for provider, spec in PROVIDER_SPECS.items():\n if not all((spec.provider, spec.env_var, spec.display_name, spec.description, spec.config_section)):\n errors.append(f\"{provider}:incomplete_spec\")\n if spec.provider != provider or spec.id != provider:\n errors.append(f\"{provider}:id_mismatch\")\n if spec.kind not in {\"search\", \"extract\", \"both\", \"disabled\"}:\n errors.append(f\"{provider}:invalid_kind\")\n if spec.supports_search:\n try:\n validate_adapter_result(\n provider,\n \"search\",\n search_result(provider, \"conformance\", [source_result(\"https://example.invalid/source\")]),\n )\n except Exception:\n errors.append(f\"{provider}:search_envelope\")\n if spec.supports_extract:\n try:\n validate_adapter_result(\n provider,\n \"extract\",\n extract_result(provider, [source_result(\"https://example.invalid/source\", content=\"evidence\")]),\n )\n except Exception:\n errors.append(f\"{provider}:extract_envelope\")\n if spec.keyless and not spec.env_var:\n errors.append(f\"{provider}:keyless_missing_env_var\")\n with patch.dict(os.environ, {}, clear=True):\n try:\n validate_api_key(provider, {spec.config_section: {}})\n except ProviderConfigError as exc:\n try:\n payload = json.loads(str(exc))\n except json.JSONDecodeError:\n errors.append(f\"{provider}:missing_key_non_deterministic\")\n else:\n if payload.get(\"provider\") != provider or payload.get(\"env_var\") != spec.env_var:\n errors.append(f\"{provider}:missing_key_mapping\")\n else:\n errors.append(f\"{provider}:missing_key_not_rejected\")\n return tuple(errors)\n\n\ndef assert_provider_conformance() -> None:\n \"\"\"Raise one compact assertion if any registered provider drifts.\"\"\"\n errors = provider_conformance_errors()\n if errors:\n raise AssertionError(\"provider conformance failed: \" + \";\".join(errors))\n","content_type":"text/x-python; charset=utf-8","language":"python","size":3440,"content_sha256":"4fef8957b331df5540f3781cda5f1123c7c9925a355969e9ec00613e83a306d6"},{"filename":"scripts/wsp_core/sdk/errors.py","content":"\"\"\"Public, typed errors for provider modules.\n\nThe names in this module are part of the additive-only WSP 3.x provider SDK.\nProviders may raise them without importing private plugin modules.\n\"\"\"\n\nfrom __future__ import annotations\n\n\nclass ProviderSDKError(Exception):\n \"\"\"Base class for errors raised at the public provider boundary.\"\"\"\n\n\nclass ProviderConfigError(ProviderSDKError):\n \"\"\"A provider cannot run because its local configuration is invalid.\"\"\"\n\n\nclass ProviderContractFailure(ProviderSDKError):\n \"\"\"A provider returned a structurally unusable source-only envelope.\"\"\"\n\n\nclass ProviderRegistrationError(ProviderSDKError):\n \"\"\"A discovered provider specification is incomplete or inconsistent.\"\"\"\n\n\nclass DuplicateProviderError(ProviderRegistrationError):\n \"\"\"Two provider specifications claimed the same stable provider id.\"\"\"\n\n\nclass ProviderDiscoveryError(ProviderSDKError):\n \"\"\"A provider module could not be loaded during startup discovery.\"\"\"\n\n\nclass ProviderStartupDiagnostic(ProviderDiscoveryError):\n \"\"\"A safe startup diagnostic for one excluded provider module.\n\n Diagnostics intentionally retain only the module stem and stable code, not\n source paths or exception text, because startup reports are often shared.\n \"\"\"\n\n def __init__(self, module: str, code: str) -> None:\n self.module = module\n self.code = code\n super().__init__(f\"provider startup diagnostic: {module}:{code}\")\n","content_type":"text/x-python; charset=utf-8","language":"python","size":1455,"content_sha256":"54bafd9f1b8efe344ba39ac5dd5fd78120e48da4ab8d9c3d43b3df7663beee89"},{"filename":"scripts/wsp_core/search_locale.py","content":"\"\"\"Config-first search locale resolution with query-aware language inference.\n\nProviders with country/language request parameters used to receive hardcoded\nus/en defaults from DEFAULT_CONFIG. Resolution is now centralized here:\n\n- Country is config-first. Precedence: CLI flag > explicit provider config in\n config.json > explicit location hint in the query (curated city/country\n table) > ``defaults.locale.country`` > \"us\".\n- Language is query-aware. Precedence: CLI flag > explicit provider config >\n ``defaults.locale.language`` > \"en\". The value \"auto\" (as a flag or as\n ``defaults.locale.language``) enables conservative query language detection\n (see routing.detect_query_language); when the detector is not confident no\n language is sent and the provider default applies.\n\nQuery language never implies a country: a German query may come from Austria\nor Switzerland just as well as Germany, so only explicit location hints or\nconfiguration move the region.\n\"\"\"\n\nimport re\nfrom typing import Any, Dict, Optional, Tuple\n\nfrom .routing import detect_query_language\n\nFALLBACK_COUNTRY = \"us\"\nFALLBACK_LANGUAGE = \"en\"\n# Search country for a detected query language when nothing else set one.\nLANGUAGE_HOME_COUNTRY: Dict[str, str] = {\n \"de\": \"de\", \"fr\": \"fr\", \"es\": \"es\", \"it\": \"it\", \"nl\": \"nl\", \"ja\": \"jp\",\n}\n\n# Language value (flag or defaults.locale.language) that enables query language\n# detection. Not a language code: it is never sent to a provider.\nAUTO_LANGUAGE = \"auto\"\n\n# Merged-config keys that carry an explicit per-provider locale override.\n# DEFAULT_CONFIG no longer ships these keys, so their presence in the merged\n# config means the user set them in config.json — that explicit choice wins\n# over query hints and global defaults. Providers without locale parameters\n# (tavily, exa, linkup, parallel, keenable, ...) are absent.\nPROVIDER_LOCALE_CONFIG_KEYS: Dict[str, Tuple[Optional[str], Optional[str]]] = {\n \"serper\": (\"country\", \"language\"),\n \"serpbase\": (\"country\", \"language\"),\n \"brave\": (\"country\", \"search_lang\"),\n \"querit\": (\"country\", \"language\"),\n \"firecrawl\": (\"country\", None),\n \"you\": (\"country\", \"language\"),\n \"searxng\": (None, \"language\"),\n}\n\n# Small curated table of unambiguous location hints. Only well-known city and\n# country names are listed; generic example queries such as\n# \"mejores restaurantes Madrid\" or \"boulangerie Paris horaires\" resolve to the\n# matching country. Deliberately small: unknown places simply do not hint.\nLOCATION_COUNTRY_HINTS: Dict[str, str] = {\n # Austria\n \"wien\": \"at\", \"vienna\": \"at\", \"graz\": \"at\", \"salzburg\": \"at\",\n \"innsbruck\": \"at\", \"österreich\": \"at\", \"austria\": \"at\",\n # Germany\n \"berlin\": \"de\", \"münchen\": \"de\", \"munich\": \"de\", \"hamburg\": \"de\",\n \"frankfurt\": \"de\", \"deutschland\": \"de\", \"germany\": \"de\",\n # Switzerland\n \"zürich\": \"ch\", \"zurich\": \"ch\", \"schweiz\": \"ch\", \"switzerland\": \"ch\",\n # France\n \"paris\": \"fr\", \"lyon\": \"fr\", \"marseille\": \"fr\", \"france\": \"fr\",\n # Spain\n \"madrid\": \"es\", \"barcelona\": \"es\", \"españa\": \"es\", \"spain\": \"es\",\n # Italy\n \"rome\": \"it\", \"roma\": \"it\", \"milano\": \"it\", \"milan\": \"it\", \"italia\": \"it\", \"italy\": \"it\",\n # Portugal\n \"lisbon\": \"pt\", \"lisboa\": \"pt\", \"portugal\": \"pt\",\n # Netherlands\n \"amsterdam\": \"nl\", \"rotterdam\": \"nl\", \"netherlands\": \"nl\",\n # United Kingdom\n \"london\": \"gb\", \"manchester\": \"gb\", \"united kingdom\": \"gb\",\n # United States\n \"new york\": \"us\", \"chicago\": \"us\", \"san francisco\": \"us\", \"usa\": \"us\",\n}\n\n_LOCATION_HINT_PATTERNS: Tuple[Tuple[Any, str], ...] = tuple(\n (re.compile(r\"\\b\" + re.escape(place) + r\"\\b\"), country)\n for place, country in LOCATION_COUNTRY_HINTS.items()\n)\n\n\ndef provider_supports_locale(provider: str) -> bool:\n \"\"\"Whether a provider's request carries country and/or language parameters.\"\"\"\n return provider in PROVIDER_LOCALE_CONFIG_KEYS\n\n\ndef detect_location_country(query: Optional[str]) -> Optional[str]:\n \"\"\"Return the ISO 3166-1 alpha-2 country for an explicit location hint.\n\n Only returns a country when every hint in the query agrees on a single\n country; conflicting hints (e.g. a \"Paris vs Madrid\" comparison) resolve\n to None so configuration keeps deciding.\n \"\"\"\n if not query:\n return None\n lowered = query.lower()\n countries = {country for pattern, country in _LOCATION_HINT_PATTERNS if pattern.search(lowered)}\n if len(countries) == 1:\n return next(iter(countries))\n return None\n\n\ndef _normalize(value: Any) -> str:\n return str(value).strip().lower()\n\n\ndef is_auto_language(value: Any) -> bool:\n return isinstance(value, str) and _normalize(value) == AUTO_LANGUAGE\n\n\ndef apply_auto_language(\n language: Optional[str], config: Dict[str, Any]\n) -> Tuple[Optional[str], Dict[str, Any]]:\n \"\"\"Carry a per-call language \"auto\" as ``defaults.locale.language``.\n\n The v3 request contract has no value for \"auto\" (``locale.language`` is a\n two-letter code), so entry points hand it to the resolver through a copy of\n the config. Any other language passes through unchanged.\n \"\"\"\n if not is_auto_language(language):\n return language, config\n defaults = config.get(\"defaults\")\n defaults = defaults if isinstance(defaults, dict) else {}\n locale = defaults.get(\"locale\")\n locale = locale if isinstance(locale, dict) else {}\n return None, {**config, \"defaults\": {**defaults, \"locale\": {**locale, \"language\": AUTO_LANGUAGE}}}\n\n\ndef resolve_locale(\n provider: str,\n config: Optional[Dict[str, Any]],\n query: Optional[str],\n cli_country: Optional[str] = None,\n cli_language: Optional[str] = None,\n) -> Tuple[str, Optional[str], Dict[str, Any]]:\n \"\"\"Resolve ``(country, language, metadata)`` for a provider request.\n\n Precedence:\n country: CLI flag > explicit provider config > location hint in query >\n ``defaults.locale.country`` > \"us\"\n language: CLI flag > explicit provider config >\n ``defaults.locale.language`` > \"en\"\n With \"auto\" (flag or ``defaults.locale.language``) the\n detected query language replaces the configured default; an\n explicit provider config still wins. Without a confident\n detection the language is None and providers omit the\n parameter.\n\n The metadata dict follows the freshness/search_type reporting pattern:\n ``{\"country\": ..., \"language\": ..., \"source\": {\"country\": \"config|hint|cli|fallback\",\n \"language\": \"config|inferred|jev|cli|provider_default|fallback\"}}``. Country\n codes are normalized to lowercase; providers that need uppercase (brave,\n firecrawl, querit, you) upper-case them in their own request builders.\n \"\"\"\n config = config if isinstance(config, dict) else {}\n country_key, language_key = PROVIDER_LOCALE_CONFIG_KEYS.get(provider, (None, None))\n section = config.get(provider)\n if not isinstance(section, dict):\n section = {}\n defaults = config.get(\"defaults\")\n locale_defaults = defaults.get(\"locale\") if isinstance(defaults, dict) else None\n if not isinstance(locale_defaults, dict):\n locale_defaults = {}\n\n if cli_country:\n country, country_source = _normalize(cli_country), \"cli\"\n elif country_key and section.get(country_key):\n country, country_source = _normalize(section[country_key]), \"config\"\n else:\n hinted = detect_location_country(query)\n default_country = locale_defaults.get(\"country\")\n if hinted:\n country, country_source = hinted, \"hint\"\n elif default_country:\n country, country_source = _normalize(default_country), \"config\"\n else:\n country, country_source = FALLBACK_COUNTRY, \"fallback\"\n\n default_language = locale_defaults.get(\"language\")\n cli_auto = is_auto_language(cli_language)\n auto_language = cli_auto or is_auto_language(default_language)\n language: Optional[str]\n jev_lang_meta = None\n if cli_language and not cli_auto:\n language, language_source = _normalize(cli_language), \"cli\"\n elif language_key and section.get(language_key):\n language, language_source = _normalize(section[language_key]), \"config\"\n elif default_language and not auto_language:\n language, language_source = _normalize(default_language), \"config\"\n elif not auto_language:\n # Nothing configured: a confidently detected query language beats the\n # English fallback, so a French query is not searched as English.\n inferred = detect_query_language(query or \"\").inferred\n if inferred:\n language, language_source = inferred, \"inferred\"\n if country_source == \"fallback\" and inferred in LANGUAGE_HOME_COUNTRY:\n country, country_source = LANGUAGE_HOME_COUNTRY[inferred], \"hint\"\n else:\n language, language_source = FALLBACK_LANGUAGE, \"fallback\"\n else:\n language, language_source = detect_query_language(query or \"\").inferred, \"inferred\"\n if not language:\n from .jev_optional import maybe_fill_language\n\n language, jev_lang_meta = maybe_fill_language(query or \"\", language, config=config)\n language_source = \"jev\" if language else \"provider_default\"\n\n metadata = {\n \"country\": country,\n \"language\": language,\n \"source\": {\"country\": country_source, \"language\": language_source},\n }\n if jev_lang_meta:\n metadata[\"jev_language\"] = jev_lang_meta\n return country, language, metadata\n","content_type":"text/x-python; charset=utf-8","language":"python","size":9562,"content_sha256":"2c839046380050b1984cf4f7fb6a8f094500c3ca847fcb7368242c01eaa03999"},{"filename":"scripts/wsp_core/search.py","content":"#!/usr/bin/env python3\n\"\"\"\nWeb Search Plus — Unified Multi-Provider Search and Extraction with Intelligent Auto-Routing\nVersion: 5.0.1\nSupports search providers: You.com, Serper, Exa, Firecrawl, Tavily, Linkup,\nBrave Search, SerpBase, Querit, Parallel, SearXNG, Keenable.\nSupports extract providers: Firecrawl, Linkup, Parallel, Tavily, Exa, You.com, Keenable, Serper.\n\nAutomatic routing (provider \"auto\") labels the query with an intent and picks\nthe first provider from a measured table: Exa for academic and documentation\nqueries, Serper for shopping, Brave otherwise; the fallback chain follows\nauto_routing.provider_priority. See docs/ROUTING.md.\n\nUsage:\n python3 search.py --query \"...\" # Auto-route based on query\n python3 search.py --provider [you|serper|exa|firecrawl|tavily|linkup|brave|serpbase|querit|searxng|auto] --query \"...\" [options]\n\nExamples:\n python3 search.py -q \"arXiv paper LLM scaling laws\" # → Exa (academic)\n python3 search.py -q \"iPhone 16 Pro Max price\" # → Serper (shopping)\n python3 search.py -q \"latest OpenSSH CVE mitigation\" # → Brave (security)\n\"\"\"\n\nfrom __future__ import annotations\n\n\nimport argparse\nimport copy\nimport json\nimport os\nimport queue\nimport sys\nimport time\nfrom dataclasses import dataclass, field\nfrom pathlib import Path\nfrom types import SimpleNamespace\nfrom typing import List, Dict, Any, Optional, Tuple\nfrom .daemon_tasks import DaemonTask\nfrom .http_client import ProviderRequestError, request_timeout_cap\nfrom .cache import (\n CACHE_DIR,\n DEFAULT_CACHE_TTL,\n cache_clear,\n cache_get,\n cache_put,\n cache_stats,\n effective_search_cache_ttl,\n routing_class_of,\n)\nfrom .budget_preflight_v3 import daily_preflight_budget as _daily_preflight_budget\n\nfrom .config import (\n add_provider_setup_guidance,\n ProviderConfigError,\n SELF_HOSTED_SEARCH_PROVIDER_IDS,\n apply_profile_effects,\n get_api_key,\n is_self_hosted_profile,\n keyless_public_allowed,\n load_config,\n provider_configured,\n validate_api_key,\n)\nfrom .provider_health import (\n execute_provider_with_retry,\n mark_provider_failure,\n provider_in_cooldown,\n reset_provider_health,\n)\nfrom .provider_stats import latency_quantile, record_provider_outcome\nfrom .quality import (\n build_quality_report,\n deduplicate_results_across_providers,\n extract_domain_constraints,\n filter_spam_results,\n rerank_domain_diversity,\n rerank_results_for_intent,\n select_research_providers,\n)\nfrom .diversity_v3 import DEFAULT_NEAR_DUPLICATE_THRESHOLD\nfrom .provider_adapter_protocol import validate_adapter_result\nfrom .provider_dispatch import SEARCH_DISPATCH\nfrom .provider_registry import (\n EXTRACT_PROVIDER_IDS,\n PROVIDER_STARTUP_DIAGNOSTICS,\n PROVIDER_SPECS,\n SEARCH_PROVIDER_IDS,\n doctor_catalog,\n)\nfrom .query_limits import MAX_QUERY_CHARS\nfrom .request_gate_v3 import validate_provider_mode\nfrom .search_locale import (\n AUTO_LANGUAGE,\n apply_auto_language,\n is_auto_language,\n provider_supports_locale,\n resolve_locale,\n)\nfrom .env_loader import load_env_files\nfrom .research import run_research_mode\nfrom .attempt_engine_v3 import AttemptEngine, provider_attempt_context, request_deadline\nfrom .cache_v3 import ResponseCacheV3\nfrom .compat_v3 import legacy_request_to_v3, v3_response_to_legacy_search\nfrom .contract_v3 import Capability, ErrorClass, RequestV3, ResponseV3, SkipReason\nfrom .orchestrator_v3 import (\n CapabilityAdapter,\n CapabilityExecution,\n ProviderPlan,\n execute_v3_request,\n)\nfrom .runtime_v3 import response_from_legacy\nfrom .state_store_v3 import SQLiteStateStore\nfrom .urls import domain_filter_tokens, domain_filters\nfrom . import providers as _providers\nfrom . import extract as _extract\nfrom .routing import (\n ROUTING_POLICY,\n _provider_auto_allowed,\n auto_route_provider,\n explain_routing,\n)\nfrom .extract import extract_plus\n\n\ndef _load_env_file():\n \"\"\"Load plugin-local, legacy parent, and Hermes profile .env files.\"\"\"\n load_env_files(__file__)\n\n\nEXTRACT_PROVIDER_PRIORITY = _extract.EXTRACT_PROVIDER_PRIORITY\nresolve_extract_provider_priority = _extract.resolve_extract_provider_priority\n\n\nPROVIDER_DOCTOR_CATALOG = doctor_catalog()\n\n\ndef _doctor_error(error_type: str, message: str) -> Dict[str, str]:\n \"\"\"Return a deliberately sanitized doctor error.\n\n Doctor output is often pasted into issues/support chats. Keep it useful without\n echoing private URLs, filesystem paths, tokens, or corrupt raw cache values.\n \"\"\"\n safe_messages = {\n \"config\": \"Provider configuration is invalid; inspect local config/env for this provider.\",\n \"cooldown\": \"Provider health cache has an invalid cooldown entry; reset provider health cache if it persists.\",\n }\n return {\"type\": error_type, \"message\": safe_messages.get(error_type, \"Provider diagnostic failed.\")}\n\n\ndef _doctor_provider_has_error_type(provider_report: Dict[str, Any], error_type: str) -> bool:\n error = provider_report.get(\"error\")\n if isinstance(error, dict):\n return error.get(\"type\") == error_type\n if isinstance(error, list):\n return any(isinstance(item, dict) and item.get(\"type\") == error_type for item in error)\n return False\n\n\ndef _build_doctor_report(config: Dict[str, Any], *, live: bool = False) -> Dict[str, Any]:\n auto_config = config.get(\"auto_routing\", {})\n disabled = set(auto_config.get(\"disabled_providers\", []) or [])\n providers = []\n for provider, spec in PROVIDER_DOCTOR_CATALOG.items():\n errors = []\n try:\n in_cooldown, remaining = provider_in_cooldown(provider)\n cooldown = {\"active\": bool(in_cooldown), \"remaining_seconds\": int(remaining)}\n except (TypeError, ValueError):\n cooldown = {\"active\": False, \"remaining_seconds\": 0}\n errors.append(_doctor_error(\"cooldown\", \"invalid provider health cache value\"))\n\n try:\n key_present = bool(get_api_key(provider, config))\n except (ProviderConfigError, ValueError):\n key_present = False\n errors.append(_doctor_error(\"config\", \"invalid provider configuration\"))\n\n spec_obj = PROVIDER_SPECS.get(provider)\n keyless = bool(spec_obj and spec_obj.keyless)\n keyless_enabled = keyless and keyless_public_allowed(provider, config)\n\n provider_report = {\n \"provider\": provider,\n \"env_var\": spec[\"env_var\"],\n \"search_capable\": spec[\"search_capable\"],\n \"extract_capable\": spec[\"extract_capable\"],\n \"key_present\": key_present,\n \"keyless\": keyless,\n \"keyless_public_enabled\": keyless_enabled,\n \"auto_allowed\": _provider_auto_allowed(provider, auto_config),\n \"disabled\": provider in disabled,\n \"cooldown\": cooldown,\n }\n if errors:\n provider_report[\"error\"] = errors[0] if len(errors) == 1 else errors\n providers.append(provider_report)\n\n usable = [p for p in providers if (p[\"key_present\"] or p[\"keyless_public_enabled\"]) and not p[\"disabled\"]]\n config_errors = [p for p in providers if _doctor_provider_has_error_type(p, \"config\")]\n return {\n \"ok\": bool(usable) and not config_errors,\n \"mode\": \"live\" if live else \"offline\",\n \"config\": {\n \"auto_routing_enabled\": auto_config.get(\"enabled\", True),\n \"default_provider\": config.get(\"default_provider\"),\n \"fallback_provider\": auto_config.get(\"fallback_provider\"),\n \"disabled_providers\": sorted(disabled),\n },\n \"cache\": {\n \"dir\": str(CACHE_DIR),\n \"exists\": CACHE_DIR.exists(),\n \"writable\": os.access(CACHE_DIR if CACHE_DIR.exists() else CACHE_DIR.parent, os.W_OK),\n \"provider_health_file\": str(CACHE_DIR / \"provider_health.json\"),\n },\n \"providers\": providers,\n \"startup_diagnostics\": [\n {\"module\": diagnostic.module, \"code\": diagnostic.code}\n for diagnostic in PROVIDER_STARTUP_DIAGNOSTICS\n ],\n }\n\n\ndef run_provider_bench(config: Dict[str, Any], **kwargs) -> Dict[str, Any]:\n \"\"\"Run the in-process provider bakeoff (see bench.py) and return its report.\n\n Uses provider implementations directly and never records provider health\n cooldowns or provider stats.\n \"\"\"\n from . import bench\n\n return bench.run_bench(config, search_module=_providers, **kwargs)\n\n\ndef format_bench_text(report: Dict[str, Any]) -> str:\n from . import bench\n\n return bench.format_bench_text(report)\n\n\ndef _format_doctor_text(report: Dict[str, Any]) -> str:\n lines = [\"Web Search Plus Doctor\", f\"Mode: {report['mode']}\", f\"OK: {report['ok']}\", \"\", \"Providers:\"]\n for provider in report[\"providers\"]:\n capabilities = []\n if provider[\"search_capable\"]:\n capabilities.append(\"search\")\n if provider[\"extract_capable\"]:\n capabilities.append(\"extract\")\n cooldown = provider[\"cooldown\"]\n cooldown_text = f\"cooldown {cooldown['remaining_seconds']}s\" if cooldown[\"active\"] else \"no cooldown\"\n keyless_text = \"\"\n if provider.get(\"keyless\"):\n keyless_text = f\"keyless={'on' if provider.get('keyless_public_enabled') else 'off'} \"\n lines.append(\n f\"- {provider['provider']}: env={provider['env_var']} \"\n f\"key={'yes' if provider['key_present'] else 'no'} \"\n f\"{keyless_text}\"\n f\"capabilities={','.join(capabilities)} \"\n f\"auto_allowed={'yes' if provider['auto_allowed'] else 'no'} \"\n f\"disabled={'yes' if provider['disabled'] else 'no'} \"\n f\"{cooldown_text}\"\n )\n lines.extend([\"\", f\"Cache: {report['cache']['dir']} (writable={report['cache']['writable']})\"])\n return \"\\n\".join(lines)\n\n\ndef build_parser_for_tests() -> argparse.ArgumentParser:\n \"\"\"Return a minimal parser exposing registry-backed provider choices for drift tests.\"\"\"\n parser = argparse.ArgumentParser(add_help=False)\n parser.add_argument(\n \"--provider\",\n \"-p\",\n choices=[*SEARCH_PROVIDER_IDS, \"auto\"],\n help=\"Search provider (auto=intelligent routing)\",\n )\n return parser\n\n\nclass _StoreQueryText(argparse.Action):\n \"\"\"Preserve literal -- on Python 3.10 as well as newer argparse versions.\"\"\"\n\n def __call__(self, parser, namespace, values, option_string=None):\n # Python 3.10 strips '--' even from a single attached option value.\n # With nargs=None this empty list cannot represent omitted input.\n setattr(namespace, self.dest, \"--\" if values == [] else values)\n\n\ndef build_parser(config: Dict[str, Any]) -> argparse.ArgumentParser:\n \"\"\"Build the search.py CLI argument parser.\n\n Shared by the CLI ``main()`` entry and the in-process ``run_search_request``\n helper so the Hermes plugin can route argv through the exact same parsing and\n defaults without spawning a subprocess.\n \"\"\"\n from . import extract_bench_v3 as _extract_bench # CLI-only; not loaded by searches\n\n parser = argparse.ArgumentParser(\n description=\"Web Search Plus — Intelligent multi-provider search with smart auto-routing\",\n formatter_class=argparse.RawDescriptionHelpFormatter,\n epilog=\"\"\"\nAutomatic routing (provider \"auto\"):\n The query gets one of eight intents; the intent picks the first provider.\n\n academic, docs → Exa papers, studies, DOIs; API docs, error messages, code\n shopping → Serper buy, price, deals, \"best ... under 300 euros\"\n everything else → Brave community, general, local, news, security\n\n A missing provider is skipped (Brave, Serper, Exa, Tavily, then\n auto_routing.provider_priority). Fallback follows provider_priority.\n\nExamples:\n python3 search.py -q \"arXiv paper LLM scaling laws\" # → Exa (academic)\n python3 search.py -q \"python asyncio TaskGroup documentation\" # → Exa (docs)\n python3 search.py -q \"iPhone 16 Pro Max price\" # → Serper (shopping)\n python3 search.py -q \"reddit best budget mechanical keyboard\" # → Brave (community)\n python3 search.py --explain-routing -q \"your query\" # Show the decision\n\nFull docs: See README.md and SKILL.md\n \"\"\",\n )\n \n # Command arguments\n parser.add_argument(\n \"command\",\n nargs=\"?\",\n choices=[\"doctor\", \"bench\", \"extract-bench\", \"state-migrate\"],\n help=\"Run doctor, benchmark, or reversible state-migration maintenance\",\n )\n parser.add_argument(\"--json\", action=\"store_true\", help=\"Emit JSON for maintenance commands\")\n parser.add_argument(\"--live\", action=\"store_true\", help=\"Allow doctor to run live provider smokes (reserved; offline by default)\")\n parser.add_argument(\n \"--bench\",\n action=\"store_true\",\n help=\"Benchmark configured search providers against a fixed live query suite and recommend an auto_routing.provider_priority (alias for the 'bench' command; spends real provider quota)\",\n )\n parser.add_argument(\n \"--bench-providers\",\n nargs=\"+\",\n choices=list(EXTRACT_PROVIDER_IDS),\n help=\"Limit extract-bench to these direct extraction providers\",\n )\n parser.add_argument(\n \"--bench-timeout-budget\",\n type=float,\n default=_extract_bench.DEFAULT_TIMEOUT_BUDGET_SECONDS,\n help=\"Best-effort whole-run wall-clock budget for extract-bench\",\n )\n parser.add_argument(\n \"--no-history\",\n action=\"store_true\",\n help=\"Do not persist the completed extract-bench summary for the Operator Console\",\n )\n migration_action = parser.add_mutually_exclusive_group()\n migration_action.add_argument(\n \"--apply\",\n action=\"store_true\",\n help=\"Apply state-migrate after creating a verified backup (dry-run is default)\",\n )\n migration_action.add_argument(\n \"--rollback\",\n metavar=\"BACKUP_ID\",\n help=\"Rollback state-migrate using a verified backup ID\",\n )\n parser.add_argument(\n \"--migration-backup-root\",\n type=Path,\n help=\"Override the default v3/migration-backups directory\",\n )\n\n # Common arguments\n parser.add_argument(\n \"--provider\", \"-p\", \n choices=[*SEARCH_PROVIDER_IDS, \"auto\"],\n help=\"Search provider (auto=intelligent routing)\"\n )\n parser.add_argument(\n \"--query\", \"-q\", \n action=_StoreQueryText, help=\"Search query\"\n )\n parser.add_argument(\n \"--extract-urls\",\n nargs=\"*\",\n help=\"Extract content from one or more URLs instead of running a search\"\n )\n parser.add_argument(\n \"--format\",\n dest=\"output_format\",\n default=\"markdown\",\n choices=[\"markdown\", \"html\"],\n help=\"Extraction output format\"\n )\n parser.add_argument(\"--extract-images\", action=\"store_true\", help=\"Extract image metadata when supported\")\n parser.add_argument(\"--include-raw-html\", action=\"store_true\", help=\"Include raw HTML when supported\")\n parser.add_argument(\"--render-js\", action=\"store_true\", help=\"Render JavaScript before extraction when supported\")\n parser.add_argument(\"--spans\", action=\"store_true\", help=\"Select deterministic semantic spans from extracted text\")\n parser.add_argument(\"--spans-query\", action=_StoreQueryText, help=\"Query used to rank extracted semantic spans\")\n parser.add_argument(\n \"--max-results\", \"-n\", \n type=int, \n default=config.get(\"defaults\", {}).get(\"max_results\", 5),\n help=\"Maximum results (default: 5)\"\n )\n parser.add_argument(\n \"--images\", \n action=\"store_true\",\n help=\"Include images (Serper/Tavily)\"\n )\n \n # Auto-routing options\n parser.add_argument(\n \"--auto\", \"-a\",\n action=\"store_true\",\n help=\"Use intelligent auto-routing (default when no provider specified)\"\n )\n parser.add_argument(\n \"--allow-fallback\",\n action=\"store_true\",\n help=\"Allow explicit --provider calls to fall back to other configured providers on failure. Auto-routing already uses fallback.\"\n )\n parser.add_argument(\n \"--explain-routing\",\n action=\"store_true\",\n help=\"Show detailed routing analysis (debug mode)\"\n )\n \n # Locale (providers with country/language request parameters). Left unset,\n # search_locale.resolve_locale applies: explicit provider config > query\n # location hint > defaults.locale > us/en fallback. Explicit flags win.\n serper_config = config.get(\"serper\", {})\n parser.add_argument(\n \"--country\",\n default=None,\n help=\"ISO 3166-1 alpha-2 country override (e.g. at, fr); beats config defaults and query location hints\"\n )\n parser.add_argument(\n \"--language\",\n default=None,\n help=\"ISO 639-1 language override (e.g. de), or 'auto' to detect it from the query; beats config defaults\"\n )\n parser.add_argument(\n \"--type\", \n dest=\"search_type\", \n default=serper_config.get(\"type\", \"search\"),\n choices=[\"search\", \"news\", \"images\", \"videos\", \"places\", \"shopping\"]\n )\n parser.add_argument(\n \"--search-type\",\n dest=\"search_type\",\n type=str.lower,\n choices=list(_providers.SEARCH_TYPE_VALUES),\n # Shares dest with the legacy serper-only --type flag above; SUPPRESS\n # keeps --type's config-driven default when neither flag is passed.\n default=argparse.SUPPRESS,\n help=(\n \"Unified result vertical (search or news; case-insensitive). Providers with a \"\n \"native news vertical (currently serper and tinyfish) serve it directly; all other providers \"\n \"run the normal search and result metadata reports search_type.applied=false\"\n )\n )\n parser.add_argument(\n \"--time-range\",\n choices=[\"hour\", \"day\", \"week\", \"month\", \"year\"]\n )\n parser.add_argument(\n \"--freshness\",\n type=str.lower,\n choices=list(_providers.FRESHNESS_VALUES),\n help=(\n \"Unified recency filter (day, week, month, year; case-insensitive). \"\n \"Applied natively where the provider supports it (serper, brave, querit, firecrawl, \"\n \"keenable, you, searxng, exa, tavily, and tinyfish); otherwise the search runs \"\n \"unfiltered and result metadata reports freshness.applied=false\"\n )\n )\n \n # Tavily-specific\n tavily_config = config.get(\"tavily\", {})\n parser.add_argument(\n \"--depth\", \n default=tavily_config.get(\"depth\", \"basic\"), \n choices=[\"basic\", \"advanced\"]\n )\n parser.add_argument(\n \"--topic\", \n default=tavily_config.get(\"topic\", \"general\"), \n choices=[\"general\", \"news\"]\n )\n parser.add_argument(\"--raw-content\", action=\"store_true\")\n \n # Querit-specific\n querit_config = config.get(\"querit\", {})\n parser.add_argument(\n \"--querit-base-url\",\n default=querit_config.get(\"base_url\", \"https://api.querit.ai\"),\n help=\"Querit API base URL\"\n )\n parser.add_argument(\n \"--querit-base-path\",\n default=querit_config.get(\"base_path\", \"/v1/search\"),\n help=\"Querit API path\"\n )\n\n # Linkup-specific\n linkup_config = config.get(\"linkup\", {})\n parser.add_argument(\n \"--linkup-depth\",\n default=linkup_config.get(\"depth\", \"standard\"),\n choices=[\"fast\", \"standard\", \"deep\"],\n help=\"Linkup search depth: fast, standard, or deep\"\n )\n parser.add_argument(\n \"--linkup-output-type\",\n default=linkup_config.get(\"output_type\", \"searchResults\"),\n choices=[\"searchResults\"],\n help=\"Linkup source-results output (fixed by the source-only charter)\"\n )\n\n # Exa-specific\n exa_config = config.get(\"exa\", {})\n parser.add_argument(\n \"--exa-type\",\n default=exa_config.get(\"type\", \"neural\"),\n choices=[\"neural\", \"fast\", \"auto\", \"keyword\", \"instant\"],\n help=\"Exa search type (for standard search, ignored when --exa-depth is set)\"\n )\n parser.add_argument(\n \"--exa-depth\",\n default=exa_config.get(\"depth\", \"normal\"),\n choices=[\"normal\"],\n help=\"Exa source-result depth (fixed to normal by the source-only charter)\"\n )\n parser.add_argument(\n \"--exa-verbosity\",\n default=exa_config.get(\"verbosity\", \"standard\"),\n choices=[\"compact\", \"standard\", \"full\"],\n help=\"Exa text verbosity for content extraction\"\n )\n parser.add_argument(\n \"--category\",\n choices=[\n \"company\", \"research paper\", \"news\", \"pdf\", \"github\", \n \"tweet\", \"personal site\", \"linkedin profile\"\n ]\n )\n parser.add_argument(\"--start-date\")\n parser.add_argument(\"--end-date\")\n parser.add_argument(\"--similar-url\")\n\n # Firecrawl-specific\n firecrawl_config = config.get(\"firecrawl\", {})\n parser.add_argument(\n \"--firecrawl-scrape\",\n action=\"store_true\",\n help=\"Firecrawl: scrape result pages and include markdown as raw_content\"\n )\n parser.add_argument(\n \"--firecrawl-sources\",\n nargs=\"+\",\n default=firecrawl_config.get(\"sources\", [\"web\"]),\n choices=[\"web\", \"news\", \"images\"],\n help=\"Firecrawl result sources\"\n )\n \n # You.com-specific\n you_config = config.get(\"you\", {})\n parser.add_argument(\n \"--you-safesearch\",\n default=you_config.get(\"safesearch\", \"moderate\"),\n choices=[\"off\", \"moderate\", \"strict\"],\n help=\"You.com SafeSearch filter\"\n )\n parser.add_argument(\n \"--livecrawl\",\n choices=[\"web\", \"news\", \"all\"],\n help=\"You.com: fetch full page content\"\n )\n parser.add_argument(\n \"--no-news\",\n action=\"store_true\",\n help=\"You.com: exclude news results (included by default)\"\n )\n \n # SearXNG-specific\n searxng_config = config.get(\"searxng\", {})\n parser.add_argument(\n \"--searxng-url\",\n default=searxng_config.get(\"base_url\") or searxng_config.get(\"instance_url\"),\n help=\"SearXNG instance URL (e.g., https://searx.example.com)\"\n )\n parser.add_argument(\n \"--searxng-safesearch\",\n type=int,\n default=searxng_config.get(\"safesearch\", 0),\n choices=[0, 1, 2],\n help=\"SearXNG SafeSearch: 0=off, 1=moderate, 2=strict\"\n )\n parser.add_argument(\n \"--engines\",\n nargs=\"+\",\n default=searxng_config.get(\"engines\"),\n help=\"SearXNG: specific engines to use (e.g., google bing duckduckgo)\"\n )\n parser.add_argument(\n \"--categories\",\n nargs=\"+\",\n help=\"SearXNG: search categories (general, images, news, videos, etc.)\"\n )\n \n # Domain filters\n parser.add_argument(\"--include-domains\", nargs=\"+\")\n parser.add_argument(\"--exclude-domains\", nargs=\"+\")\n \n # Output\n parser.add_argument(\"--compact\", action=\"store_true\")\n parser.add_argument(\n \"--quality-report\",\n action=\"store_true\",\n help=\"Attach transparent routing/result diagnostics to the JSON output\"\n )\n parser.add_argument(\n \"--mode\",\n default=\"normal\",\n choices=[\"normal\", \"research\"],\n help=\"Search mode: normal single-provider route or research multi-provider + extraction\"\n )\n parser.add_argument(\n \"--research-providers\",\n nargs=\"+\",\n help=\"Explicit provider list for --mode research\"\n )\n parser.add_argument(\n \"--research-extract-count\",\n type=int,\n default=3,\n help=\"Number of top research-mode URLs to extract for grounding\"\n )\n parser.add_argument(\n \"--research-time-budget\",\n type=float,\n default=55.0,\n help=\"Best-effort wall-clock budget for research mode; skips remaining providers/extraction between calls when exhausted\"\n )\n \n # Caching options\n parser.add_argument(\n \"--cache-ttl\",\n type=int,\n default=DEFAULT_CACHE_TTL,\n help=f\"Cache TTL in seconds (default: {DEFAULT_CACHE_TTL} = 1 hour)\"\n )\n parser.add_argument(\n \"--no-cache\",\n action=\"store_true\",\n help=\"Bypass cache (always fetch fresh results)\"\n )\n parser.add_argument(\n \"--clear-cache\",\n action=\"store_true\",\n help=\"Clear all cached results and exit\"\n )\n parser.add_argument(\n \"--cache-stats\",\n action=\"store_true\",\n help=\"Show cache statistics and exit\"\n )\n\n return parser\n\n\ndef default_search_args(config: Dict[str, Any]) -> SimpleNamespace:\n \"\"\"Search options with the CLI defaults, built without argparse.\n\n The plugin calls the pipeline directly and never parses argv. These are\n exactly the defaults of ``build_parser(config).parse_args([])`` minus the\n maintenance-only CLI options (pinned by tests/test_direct_call_path.py).\n \"\"\"\n defaults = config.get(\"defaults\", {})\n tavily = config.get(\"tavily\", {})\n querit = config.get(\"querit\", {})\n linkup = config.get(\"linkup\", {})\n exa = config.get(\"exa\", {})\n firecrawl = config.get(\"firecrawl\", {})\n you = config.get(\"you\", {})\n searxng = config.get(\"searxng\", {})\n return SimpleNamespace(\n provider=None,\n query=None,\n output_format=\"markdown\",\n extract_images=False,\n include_raw_html=False,\n render_js=False,\n spans=False,\n spans_query=None,\n max_results=defaults.get(\"max_results\", 5),\n images=False,\n allow_fallback=False,\n country=None,\n language=None,\n search_type=config.get(\"serper\", {}).get(\"type\", \"search\"),\n time_range=None,\n freshness=None,\n depth=tavily.get(\"depth\", \"basic\"),\n topic=tavily.get(\"topic\", \"general\"),\n raw_content=False,\n querit_base_url=querit.get(\"base_url\", \"https://api.querit.ai\"),\n querit_base_path=querit.get(\"base_path\", \"/v1/search\"),\n linkup_depth=linkup.get(\"depth\", \"standard\"),\n linkup_output_type=linkup.get(\"output_type\", \"searchResults\"),\n exa_type=exa.get(\"type\", \"neural\"),\n exa_depth=exa.get(\"depth\", \"normal\"),\n exa_verbosity=exa.get(\"verbosity\", \"standard\"),\n category=None,\n start_date=None,\n end_date=None,\n similar_url=None,\n firecrawl_scrape=False,\n firecrawl_sources=list(firecrawl.get(\"sources\", [\"web\"])),\n you_safesearch=you.get(\"safesearch\", \"moderate\"),\n livecrawl=None,\n no_news=False,\n searxng_url=searxng.get(\"base_url\") or searxng.get(\"instance_url\"),\n searxng_safesearch=searxng.get(\"safesearch\", 0),\n engines=searxng.get(\"engines\"),\n categories=None,\n include_domains=None,\n exclude_domains=None,\n quality_report=False,\n mode=\"normal\",\n research_providers=None,\n research_extract_count=3,\n research_time_budget=55.0,\n cache_ttl=DEFAULT_CACHE_TTL,\n no_cache=False,\n )\n\n\ndef main():\n # Maintenance commands live in modules a search never needs; load them here.\n from . import extract_bench_v3 as _extract_bench\n from . import state_migration_v3 as _state_migration\n\n config = load_config()\n parser = build_parser(config)\n args = parser.parse_args()\n args.language, config = apply_auto_language(args.language, config)\n\n migration_options_used = bool(\n args.apply or args.rollback or args.migration_backup_root is not None\n )\n if args.command != \"state-migrate\" and migration_options_used:\n parser.error(\"--apply, --rollback, and --migration-backup-root require state-migrate\")\n\n if args.command == \"state-migrate\":\n state_path = CACHE_DIR / \"v3\" / \"state.sqlite3\"\n backup_root = args.migration_backup_root or state_path.parent / \"migration-backups\"\n if args.rollback:\n report = _state_migration.rollback_legacy_state(\n state_path=state_path,\n backup_root=backup_root,\n backup_id=args.rollback,\n )\n else:\n report = _state_migration.migrate_legacy_state(\n cache_root=CACHE_DIR,\n state_path=state_path,\n backup_root=backup_root,\n dry_run=not args.apply,\n )\n print(_state_migration.render_migration_report(report))\n if report.status not in {\"ready\", \"applied\", \"unchanged\", \"rolled_back\"}:\n raise SystemExit(2)\n return\n\n if args.command == \"doctor\":\n report = _build_doctor_report(config, live=args.live)\n if args.json or args.compact:\n indent = None if args.compact else 2\n print(json.dumps(report, indent=indent, ensure_ascii=False))\n else:\n print(_format_doctor_text(report))\n return\n\n if args.command == \"bench\" or args.bench:\n report = run_provider_bench(config, max_results=args.max_results)\n if args.json or args.compact:\n indent = None if args.compact else 2\n print(json.dumps(report, indent=indent, ensure_ascii=False))\n else:\n print(format_bench_text(report))\n return\n\n if args.command == \"extract-bench\":\n if not args.extract_urls:\n parser.error(\"extract-bench requires one or more --extract-urls\")\n report = _extract_bench.run_extract_bench(\n config,\n urls=args.extract_urls,\n providers=args.bench_providers,\n timeout_budget=args.bench_timeout_budget,\n )\n if not args.no_history:\n report[\"history_written\"] = _extract_bench.BenchmarkHistoryJournal(\n CACHE_DIR\n ).append(_extract_bench.benchmark_history_record(report))\n else:\n report[\"history_written\"] = False\n if args.json or args.compact:\n indent = None if args.compact else 2\n print(json.dumps(report, indent=indent, ensure_ascii=False))\n else:\n print(_extract_bench.format_extract_bench_text(report))\n print(\"History written: {}\".format(report[\"history_written\"]))\n return\n\n # Handle cache management commands first (before query validation). The\n # tools answer repeats from the v3 response cache, so both commands cover it.\n response_cache = ResponseCacheV3((config.get(\"v3\") or {}).get(\"cache_dir\") or CACHE_DIR)\n if args.clear_cache:\n result = cache_clear()\n result[\"v3_response_cleared\"] = response_cache.clear()\n indent = None if args.compact else 2\n print(json.dumps(result, indent=indent, ensure_ascii=False))\n return\n\n if args.cache_stats:\n result = cache_stats()\n result[\"v3_response\"] = response_cache.stats()\n indent = None if args.compact else 2\n print(json.dumps(result, indent=indent, ensure_ascii=False))\n return\n\n if args.extract_urls is not None:\n result = extract_plus(\n urls=args.extract_urls,\n provider=args.provider or \"auto\",\n output_format=args.output_format,\n include_images=args.extract_images,\n include_raw_html=args.include_raw_html,\n render_js=args.render_js,\n spans=args.spans,\n spans_query=args.spans_query,\n config=config,\n )\n indent = None if args.compact else 2\n succeeded = not result.get(\"error\") and any(\n isinstance(item, dict) and not item.get(\"error\")\n and any(item.get(field) for field in (\"content\", \"text\", \"raw_content\", \"markdown\", \"html\"))\n for item in result.get(\"results\", [])\n )\n print(json.dumps(result, indent=indent, ensure_ascii=False),\n file=sys.stdout if succeeded else sys.stderr)\n if not succeeded:\n sys.exit(1)\n return\n \n if not args.query and not args.similar_url:\n parser.error(\"--query is required (unless using --similar-url with Exa)\")\n try:\n domain_filters(args.include_domains, args.exclude_domains)\n except ValueError as exc:\n parser.error(str(exc))\n if (\n str(args.provider or \"\").lower() in _providers.DOMAIN_SUFFIX_FILTER_UNSUPPORTED\n and _providers.public_suffix_entries(args.include_domains, args.exclude_domains)\n ):\n parser.error(_providers.domain_suffix_unsupported_message(str(args.provider).lower()))\n \n # Handle --explain-routing\n if args.explain_routing:\n if not args.query:\n parser.error(\"--query is required for --explain-routing\")\n explanation = explain_routing(args.query, config)\n indent = None if args.compact else 2\n print(json.dumps(explanation, indent=indent, ensure_ascii=False))\n return\n\n payload, exit_code = _execute_search_request_core(args, config)\n if exit_code == 0:\n indent = None if args.compact else 2\n print(json.dumps(payload, indent=indent, ensure_ascii=False))\n else:\n print(json.dumps(payload, indent=2), file=sys.stderr)\n sys.exit(1)\n\n\ndef _apply_result_quality_pipeline(\n result: Dict[str, Any],\n config: Dict[str, Any],\n query: str = \"\",\n include_domains: Optional[List[str]] = None,\n) -> None:\n \"\"\"Filter mirror/SEO-spam domains and cap per-domain dominance, in place.\n\n Explicit domain constraints (``site:`` operators, ``include_domains``)\n express user intent and win: constrained domains are exempt from the spam\n filter, and the diversity rerank is skipped entirely — a deliberately\n one-domain query must not have its order shuffled.\n\n Both steps are truthful: removals and demotions are reported in\n ``result[\"metadata\"]`` so quality reports and callers can see what changed.\n \"\"\"\n results = result.get(\"results\")\n if not isinstance(results, list) or not results:\n return\n quality_config = config.get(\"quality\") if isinstance(config.get(\"quality\"), dict) else {}\n constrained_domains = extract_domain_constraints(query, include_domains)\n if quality_config.get(\"filter_spam\", True):\n allowed = list(quality_config.get(\"allowed_domains\") or []) + constrained_domains\n kept, removed_domains = filter_spam_results(\n results,\n extra_blocked=quality_config.get(\"blocked_domains\"),\n allowed=allowed,\n )\n if removed_domains:\n result[\"results\"] = kept\n result.setdefault(\"metadata\", {})[\"spam_filtered\"] = {\n \"removed_count\": len(results) - len(kept),\n \"domains\": removed_domains,\n }\n results = kept\n if constrained_domains:\n return\n try:\n max_per_domain = int(quality_config.get(\"max_results_per_domain\", 2))\n except (TypeError, ValueError):\n max_per_domain = 2\n if max_per_domain > 0:\n reranked, demoted = rerank_domain_diversity(results, max_per_domain=max_per_domain, query=query)\n if demoted:\n result[\"results\"] = reranked\n result.setdefault(\"metadata\", {})[\"domain_diversity_demoted\"] = demoted\n\n\ndef _legacy_search_cache_context(\n args, provider: str, config: Dict[str, Any]\n) -> Dict[str, Any]:\n locale_country, locale_language, _locale_meta = resolve_locale(\n provider,\n config,\n args.query,\n cli_country=args.country,\n cli_language=args.language,\n )\n return {\n \"locale\": f\"{locale_country}:{locale_language or ''}\",\n \"freshness\": args.freshness,\n \"time_range\": getattr(args, \"time_range\", None),\n \"include_domains\": sorted(args.include_domains)\n if args.include_domains\n else None,\n \"exclude_domains\": sorted(args.exclude_domains)\n if args.exclude_domains\n else None,\n \"topic\": args.topic,\n \"search_engines\": sorted(args.engines) if args.engines else None,\n \"include_news\": not args.no_news,\n \"search_type\": args.search_type,\n \"exa_type\": args.exa_type,\n \"exa_depth\": args.exa_depth,\n \"exa_verbosity\": args.exa_verbosity,\n \"category\": args.category,\n \"similar_url\": args.similar_url,\n \"mode\": args.mode,\n \"quality_report\": args.quality_report,\n }\n\n\ndef _diversity_settings(config: Dict[str, Any]) -> Tuple[bool, float]:\n \"\"\"Read the validated diversity settings with safe direct-call fallbacks.\"\"\"\n quality_config = config.get(\"quality\") if isinstance(config.get(\"quality\"), dict) else {}\n diversity_config = (\n quality_config.get(\"diversity\")\n if isinstance(quality_config.get(\"diversity\"), dict)\n else {}\n )\n rerank = diversity_config.get(\"rerank\") is True\n threshold = diversity_config.get(\n \"near_duplicate_threshold\", DEFAULT_NEAR_DUPLICATE_THRESHOLD\n )\n if isinstance(threshold, bool):\n threshold = DEFAULT_NEAR_DUPLICATE_THRESHOLD\n try:\n threshold = float(threshold)\n except (TypeError, ValueError):\n threshold = DEFAULT_NEAR_DUPLICATE_THRESHOLD\n if threshold \u003c 0.0 or threshold > 1.0:\n threshold = DEFAULT_NEAR_DUPLICATE_THRESHOLD\n return rerank, threshold\n\n\ndef _research_quorum_settings(config: Dict[str, Any]) -> Dict[str, Any]:\n \"\"\"Read validated Research early-return controls for direct callers too.\"\"\"\n raw_quality_config = config.get(\"quality\")\n quality_config = raw_quality_config if isinstance(raw_quality_config, dict) else {}\n configured = quality_config.get(\"research_quorum\")\n configured = configured if isinstance(configured, dict) else {}\n defaults = {\n \"enabled\": True,\n \"min_contributing_providers\": 2,\n \"result_target_cap\": 5,\n \"min_unique_domains\": 3,\n }\n settings = {**defaults, **configured}\n # Validation owns persisted config; conservative fallbacks keep callers that\n # invoke the public search helpers with hand-built config dicts safe.\n if settings[\"enabled\"] is not True:\n settings[\"enabled\"] = False\n for name, minimum in (\n (\"min_contributing_providers\", 2),\n (\"result_target_cap\", 1),\n (\"min_unique_domains\", 1),\n ):\n value = settings[name]\n if isinstance(value, bool):\n settings[name] = defaults[name]\n continue\n try:\n settings[name] = max(minimum, int(value))\n except (TypeError, ValueError):\n settings[name] = defaults[name]\n return settings\n\n\ndef _result_freshness(provider, requested, result):\n \"\"\"Preserve execution evidence, including cached and per-provider receipts.\"\"\"\n metadata = result.setdefault(\"metadata\", {})\n existing = metadata.get(\"freshness\")\n if isinstance(existing, dict) and existing.get(\"requested\") == requested:\n return existing\n return _providers.freshness_metadata(\n provider, requested,\n applied_published_dates=metadata.pop(\"applied_published_dates\", None) or {},\n )\n\n\ndef _finalize_research_result(\n result: Dict[str, Any],\n *,\n args,\n config: Dict[str, Any],\n routing_info: Dict[str, Any],\n providers_considered: List[str],\n research_providers: List[str],\n cooldown_skips: List[Dict[str, Any]],\n provider_payloads: Optional[Dict[str, Dict[str, Any]]] = None,\n) -> Dict[str, Any]:\n \"\"\"Apply the shared public Research Mode metadata and quality envelope.\"\"\"\n final_routing = dict(routing_info)\n final_routing[\"mode\"] = \"research\"\n final_routing[\"provider\"] = \"research\"\n result.setdefault(\"routing\", {}).update(final_routing)\n if cooldown_skips:\n result[\"routing\"][\"providers_skipped\"] = [\n {\n \"provider\": item.get(\"provider\"),\n \"reason\": \"cooldown\",\n \"cooldown_remaining_seconds\": item.get(\n \"cooldown_remaining_seconds\"\n ),\n }\n for item in cooldown_skips\n ]\n requested_freshness = _providers.effective_recency(\n getattr(args, \"time_range\", None), args.freshness\n )\n if requested_freshness:\n result.setdefault(\"metadata\", {})[\"freshness\"] = {\n \"requested\": requested_freshness,\n \"providers\": [\n _result_freshness(\n provider, requested_freshness,\n (provider_payloads or {}).get(provider) or {},\n )\n for provider in research_providers\n ],\n }\n research_search_type = getattr(args, \"search_type\", None)\n if (\n research_search_type in _providers.SEARCH_TYPE_VALUES\n and research_search_type != \"search\"\n ):\n result.setdefault(\"metadata\", {})[\"search_type\"] = {\n \"requested\": research_search_type,\n \"providers\": [\n _providers.search_type_metadata(provider, research_search_type)\n for provider in research_providers\n ],\n }\n _apply_result_quality_pipeline(\n result,\n config,\n query=args.query or \"\",\n include_domains=args.include_domains,\n )\n _diversity_rerank, near_duplicate_threshold = _diversity_settings(config)\n result[\"quality_report\"] = build_quality_report(\n query=args.query,\n result=result,\n routing_info=final_routing,\n providers_considered=providers_considered,\n eligible_providers=research_providers,\n cooldown_skips=cooldown_skips,\n errors=result.get(\"routing\", {}).get(\"provider_errors\", []),\n near_duplicate_threshold=near_duplicate_threshold,\n )\n return result\n\n\ndef _execute_search_request_core(args, config: Dict[str, Any]) -> Tuple[Dict[str, Any], int]:\n \"\"\"Run the search/research pipeline for parsed args.\n\n Returns ``(payload, exit_code)`` where exit_code 0 means a success payload bound\n for stdout and 1 means an error payload bound for stderr. This function never\n prints or calls ``sys.exit`` so the Hermes plugin can invoke it in-process\n instead of spawning a subprocess.\n \"\"\"\n config = apply_profile_effects(config)\n if getattr(args, \"query\", None):\n from .jev_optional import maybe_search_type\n\n resolved, _jev_meta = maybe_search_type(\n args.query, getattr(args, \"search_type\", None), config=config\n )\n args.search_type = resolved\n\n # Determine provider\n if args.provider == \"auto\" or (args.provider is None and not args.similar_url):\n if args.query:\n routing = auto_route_provider(args.query, config)\n if routing.get(\"error_type\"):\n return {\n \"error\": routing[\"error\"],\n \"error_type\": routing[\"error_type\"],\n \"provider\": \"auto\",\n \"query\": args.query,\n \"results\": [],\n \"metadata\": {\"profile\": config.get(\"profile\")},\n }, 1\n provider = routing[\"provider\"]\n routing_info = {\n \"auto_routed\": True,\n \"provider\": provider,\n \"confidence\": routing[\"confidence\"],\n \"confidence_level\": routing[\"confidence_level\"],\n \"reason\": routing[\"reason\"],\n \"routing_policy\": routing.get(\"routing_policy\", ROUTING_POLICY),\n \"top_signals\": routing[\"top_signals\"],\n \"scores\": routing[\"scores\"],\n \"auto_allow_excluded\": routing.get(\"auto_allow_excluded\", []),\n \"analysis_summary\": routing.get(\"analysis_summary\", {}),\n }\n else:\n provider = \"exa\"\n routing_info = {\n \"auto_routed\": True,\n \"provider\": \"exa\",\n \"confidence\": 1.0,\n \"confidence_level\": \"high\",\n \"reason\": \"similar_url_specified\",\n \"routing_policy\": ROUTING_POLICY,\n }\n else:\n provider = args.provider or \"serper\"\n routing_info = {\"auto_routed\": False, \"provider\": provider, \"routing_policy\": ROUTING_POLICY}\n profile_deviation = (\n is_self_hosted_profile(config)\n and args.provider not in (None, \"auto\")\n and provider not in SELF_HOSTED_SEARCH_PROVIDER_IDS\n )\n \n # Build provider fallback list\n auto_config = config.get(\"auto_routing\", {})\n provider_priority = auto_config.get(\"provider_priority\", list(SEARCH_PROVIDER_IDS))\n disabled_providers = auto_config.get(\"disabled_providers\", [])\n\n # Start with the selected provider, then try others in priority order.\n # Explicit provider calls are strict by default: requested provider must be\n # the actual provider. Use --allow-fallback to opt into legacy fallback.\n # Fixed-provider mode is intentionally strict: if auto-routing is disabled\n # and the saved default was selected via provider=auto, do not silently\n # fall back to other providers. Users who want fallback should keep\n # auto-routing enabled and tune priority/fallback instead.\n explicit_provider_mode = args.provider not in (None, \"auto\")\n fixed_provider_mode = bool(\n auto_config.get(\"enabled\", True) is False\n and config.get(\"default_provider\")\n and provider == config.get(\"default_provider\")\n and (args.provider == \"auto\" or (args.provider is None and not args.similar_url))\n )\n strict_provider_mode = fixed_provider_mode or (explicit_provider_mode and not args.allow_fallback)\n providers_to_try = [provider] if provider else []\n if not strict_provider_mode:\n for p in provider_priority:\n if p not in providers_to_try and p not in disabled_providers and _provider_auto_allowed(p, auto_config) and provider_configured(p, config):\n providers_to_try.append(p)\n\n # The v3 AttemptEngine owns admission/circuit state. Its single-provider\n # adapter must not consult or mutate the legacy provider-health system.\n engine_owned_attempt = bool(getattr(args, \"_v3_engine_owned_attempt\", False))\n eligible_providers = []\n cooldown_skips = []\n if engine_owned_attempt:\n eligible_providers = list(providers_to_try)\n else:\n for p in providers_to_try:\n in_cd, remaining = provider_in_cooldown(p)\n if in_cd:\n cooldown_skips.append({\"provider\": p, \"cooldown_remaining_seconds\": remaining})\n else:\n eligible_providers.append(p)\n\n if not eligible_providers:\n eligible_providers = providers_to_try[:1]\n\n # Helper function to execute search for a provider. Provider-specific\n # kwargs-building lives in provider_dispatch.SEARCH_DISPATCH; the caller\n # providers module is passed so adapters resolve implementations late.\n provider_payloads = {}\n\n def execute_search(prov: str) -> Dict[str, Any]:\n validate_provider_mode(prov, \"search\")\n key = validate_api_key(prov, config)\n adapter = SEARCH_DISPATCH.get(prov)\n if adapter is None:\n raise ValueError(f\"Unknown provider: {prov}\")\n provider_result = validate_adapter_result(\n prov,\n \"search\",\n adapter(_providers, prov, args, key, config, routing_info),\n )\n if engine_owned_attempt:\n provider_result[\"_v3_raw_results\"] = [\n dict(item)\n for item in (provider_result.get(\"results\") or [])\n if isinstance(item, dict)\n ]\n provider_payloads[prov] = provider_result\n return provider_result\n\n def execute_with_retry(prov: str) -> Dict[str, Any]:\n # The v3 AttemptEngine owns retries and circuit state, so an\n # engine-owned call runs once here and the engine retries around it.\n # Latency samples (provider_stats) are a separate signal and are\n # recorded on both paths; the hedged fallback reads them to time the\n # next attempt.\n started = time.monotonic()\n try:\n if engine_owned_attempt:\n provider_result = execute_search(prov)\n else:\n provider_result = execute_provider_with_retry(prov, lambda: execute_search(prov))\n except ProviderRequestError:\n # Only real provider interactions count as performance signal;\n # config/validation errors (e.g. missing keys) are not recorded.\n record_provider_outcome(prov, latency_seconds=time.monotonic() - started, result_count=0, error=True)\n raise\n record_provider_outcome(\n prov,\n latency_seconds=time.monotonic() - started,\n result_count=len(provider_result.get(\"results\", []) or []),\n error=False,\n )\n return provider_result\n\n cache_context = _legacy_search_cache_context(args, provider or \"\", config)\n\n providers_considered = providers_to_try.copy()\n\n if args.mode == \"research\":\n available_research_providers = {\n p for p in providers_to_try\n if p not in disabled_providers and _provider_auto_allowed(p, auto_config) and provider_configured(p, config) and not provider_in_cooldown(p)[0]\n }\n if provider and provider_configured(provider, config) and not provider_in_cooldown(provider)[0]:\n available_research_providers.add(provider)\n if args.research_providers:\n # Explicit research selection follows the same contract as an\n # explicit single-provider call: auto-routing allowlists do not\n # veto the caller's named provider. Disabled/unconfigured providers\n # remain unavailable, and cooldown skips stay visible in receipts.\n research_providers = []\n for p in args.research_providers:\n if p in disabled_providers or not provider_configured(p, config):\n continue\n in_cooldown, remaining = provider_in_cooldown(p)\n if in_cooldown:\n if not any(item.get(\"provider\") == p for item in cooldown_skips):\n cooldown_skips.append(\n {\n \"provider\": p,\n \"cooldown_remaining_seconds\": remaining,\n }\n )\n continue\n if p not in research_providers:\n research_providers.append(p)\n else:\n research_providers = select_research_providers(\n primary_provider=provider,\n provider_priority=provider_priority,\n available_providers=available_research_providers,\n max_providers=3,\n )\n\n if not research_providers:\n error_result = {\n \"error\": \"No configured providers available for research mode\",\n \"provider\": provider,\n \"query\": args.query,\n \"routing\": routing_info,\n \"cooldown_skips\": cooldown_skips,\n }\n return error_result, 1\n\n diversity_rerank, near_duplicate_threshold = _diversity_settings(config)\n quorum_settings = _research_quorum_settings(config)\n result = run_research_mode(\n query=args.query,\n research_providers=research_providers,\n execute_search=execute_with_retry,\n extract_urls=lambda urls: extract_plus(\n urls=urls,\n provider=\"auto\",\n output_format=\"markdown\",\n config=config,\n ),\n max_results=args.max_results,\n max_extract_urls=args.research_extract_count,\n time_budget_seconds=args.research_time_budget,\n diversity_rerank=diversity_rerank,\n near_duplicate_threshold=near_duplicate_threshold,\n quorum_enabled=quorum_settings[\"enabled\"],\n quorum_min_contributing_providers=quorum_settings[\n \"min_contributing_providers\"\n ],\n quorum_result_target_cap=quorum_settings[\"result_target_cap\"],\n quorum_min_unique_domains=quorum_settings[\"min_unique_domains\"],\n )\n result = _finalize_research_result(\n result,\n args=args,\n config=config,\n routing_info=routing_info,\n providers_considered=providers_considered,\n research_providers=research_providers,\n cooldown_skips=cooldown_skips,\n provider_payloads=provider_payloads,\n )\n return result, 0\n\n # Check cache first (unless --no-cache is set)\n cached_result = None\n cache_hit = False\n search_cache_ttl = effective_search_cache_ttl(\n args.query or \"\",\n freshness=getattr(args, \"time_range\", None) or getattr(args, \"freshness\", None),\n requested_ttl=args.cache_ttl,\n routing_class=routing_class_of(\n getattr(args, \"_v3_planned_routing\", None) or routing_info\n ),\n )\n if not args.no_cache and args.query:\n cached_result = cache_get(\n query=args.query,\n provider=provider,\n max_results=args.max_results,\n ttl=search_cache_ttl,\n params=cache_context,\n )\n if cached_result:\n cache_hit = True\n result = {k: v for k, v in cached_result.items() if not k.startswith(\"_cache_\")}\n if \"query\" in result:\n # The key ignores case and spacing; show this caller's own query.\n result[\"query\"] = args.query\n result[\"cached\"] = True\n result[\"cache_age_seconds\"] = int(time.time() - cached_result.get(\"_cache_timestamp\", 0))\n\n errors = []\n successful_provider = None\n successful_results: List[Tuple[str, Dict[str, Any]]] = []\n result = None if not cache_hit else result\n\n for idx, current_provider in enumerate(eligible_providers):\n if cache_hit:\n successful_provider = provider\n break\n try:\n provider_result = execute_with_retry(current_provider)\n if not engine_owned_attempt:\n reset_provider_health(current_provider)\n successful_results.append((current_provider, provider_result))\n successful_provider = current_provider\n\n # If we have enough results, stop.\n if len(provider_result.get(\"results\", [])) >= args.max_results:\n break\n\n # Only continue collecting from lower-priority providers when fallback was needed.\n if not errors:\n break\n except ProviderConfigError as e:\n if engine_owned_attempt:\n raise\n # Missing/invalid local credentials are configuration errors, not\n # provider health failures. Do not poison shared cooldown state for\n # a provider the runtime never actually contacted.\n try:\n detail = json.loads(str(e))\n except (ValueError, TypeError):\n detail = {}\n if not isinstance(detail, dict):\n detail = {}\n errors.append({\n \"provider\": current_provider,\n \"error\": detail.get(\"error\") or str(e),\n **{key: detail[key] for key in (\"env_var\", \"how_to_fix\") if key in detail},\n })\n continue\n except Exception as e:\n if engine_owned_attempt:\n raise\n error_msg = str(e)\n cooldown_info = mark_provider_failure(current_provider, error_msg, retry_after=getattr(e, \"retry_after\", None))\n errors.append({\n \"provider\": current_provider,\n \"error\": error_msg,\n \"cooldown_seconds\": cooldown_info.get(\"cooldown_seconds\"),\n })\n if len(eligible_providers) > 1:\n remaining = eligible_providers[idx + 1:]\n if remaining:\n print(json.dumps({\n \"fallback\": True,\n \"failed_provider\": current_provider,\n \"error\": error_msg,\n \"trying_next\": remaining[0],\n }), file=sys.stderr)\n continue\n\n if successful_results:\n if len(successful_results) == 1:\n result = successful_results[0][1]\n else:\n primary = successful_results[0][1].copy()\n deduped_results, dedup_count = deduplicate_results_across_providers(successful_results, args.max_results)\n primary[\"results\"] = deduped_results\n primary[\"deduplicated\"] = dedup_count > 0\n primary.setdefault(\"metadata\", {})\n primary[\"metadata\"][\"dedup_count\"] = dedup_count\n primary[\"metadata\"][\"providers_merged\"] = [p for p, _ in successful_results]\n result = primary\n\n if result is not None:\n if engine_owned_attempt and getattr(args, \"_v3_research_member\", False):\n return result, 0\n if successful_provider != provider:\n routing_info[\"fallback_used\"] = True\n routing_info[\"original_provider\"] = provider\n routing_info[\"provider\"] = successful_provider\n routing_info[\"fallback_errors\"] = errors\n\n if cooldown_skips:\n routing_info[\"cooldown_skips\"] = cooldown_skips\n\n # A v3 attempt is a fixed-provider search; rank and report with the\n # routing the engine planned for the request.\n planned_routing = getattr(args, \"_v3_planned_routing\", None)\n ranking_routing = (\n {**planned_routing, \"provider\": routing_info.get(\"provider\")}\n if planned_routing\n else routing_info\n )\n routing_class = ranking_routing.get(\"analysis_summary\", {}).get(\"routing_class\", \"general\")\n if not cache_hit and isinstance(result.get(\"results\"), list):\n reranked, rerank_metadata = rerank_results_for_intent(\n args.query or \"\", routing_class, result.get(\"results\", []),\n window=getattr(args, \"_v3_requested_results\", None),\n )\n result[\"results\"] = reranked\n if rerank_metadata.get(\"reranked\"):\n result.setdefault(\"metadata\", {})[\"intent_rerank\"] = rerank_metadata\n _apply_result_quality_pipeline(result, config, query=args.query or \"\", include_domains=args.include_domains)\n requested_results = getattr(args, \"_v3_requested_results\", None)\n if requested_results and isinstance(result.get(\"results\"), list):\n result[\"results\"] = result[\"results\"][:requested_results]\n\n result[\"routing\"] = routing_info\n\n requested_freshness = _providers.effective_recency(\n getattr(args, \"time_range\", None), args.freshness\n )\n if requested_freshness:\n result.setdefault(\"metadata\", {})[\"freshness\"] = _result_freshness(\n successful_provider or provider, requested_freshness, result,\n )\n\n requested_search_type = getattr(args, \"search_type\", None)\n if requested_search_type in _providers.SEARCH_TYPE_VALUES and requested_search_type != \"search\":\n result.setdefault(\"metadata\", {})[\"search_type\"] = _providers.search_type_metadata(\n successful_provider or provider, requested_search_type\n )\n\n # Locale transparency (freshness-metadata pattern): report the\n # resolved country/language and where each came from, but only for\n # providers whose request actually carries locale parameters.\n final_provider = successful_provider or provider\n if provider_supports_locale(final_provider):\n _, _, locale_meta = resolve_locale(\n final_provider, config, args.query,\n cli_country=args.country, cli_language=args.language,\n )\n result.setdefault(\"metadata\", {})[\"locale\"] = locale_meta\n\n if profile_deviation:\n result.setdefault(\"metadata\", {})[\"profile_deviation\"] = True\n\n if (\n not cache_hit\n and not args.no_cache\n and args.query\n ):\n cache_put(\n query=args.query,\n provider=successful_provider or provider,\n max_results=args.max_results,\n result=result,\n params=cache_context,\n )\n\n result[\"cached\"] = bool(cache_hit)\n if \"deduplicated\" not in result:\n result[\"deduplicated\"] = False\n result.setdefault(\"metadata\", {})\n result[\"metadata\"].setdefault(\"dedup_count\", 0)\n\n if args.quality_report:\n _diversity_rerank, near_duplicate_threshold = _diversity_settings(config)\n result[\"quality_report\"] = build_quality_report(\n query=args.query,\n result=result,\n routing_info=ranking_routing,\n providers_considered=providers_considered,\n eligible_providers=eligible_providers,\n cooldown_skips=cooldown_skips,\n errors=errors,\n near_duplicate_threshold=near_duplicate_threshold,\n )\n\n return result, 0\n else:\n error_result = {\n \"error\": \"All providers failed\",\n \"provider\": provider,\n \"query\": args.query,\n \"routing\": routing_info,\n \"provider_errors\": errors,\n \"cooldown_skips\": cooldown_skips,\n }\n add_provider_setup_guidance(error_result, \"search\", eligible_providers, config,\n requested_provider=getattr(args, \"provider\", None) or \"auto\")\n return error_result, 1\n\n\ndef execute_search_request(args, config: Dict[str, Any]) -> Tuple[Dict[str, Any], int]:\n \"\"\"Backward-compatible standalone CLI seam over the unchanged provider core.\"\"\"\n return _execute_search_request_core(args, config)\n\n\ndef _plan_search_v3(request: RequestV3, config: Dict[str, Any]) -> ProviderPlan:\n config = apply_profile_effects(config)\n routing_request = request.routing\n requested = str(routing_request.get(\"provider\") or \"auto\")\n auto_config = config.get(\"auto_routing\", {})\n if requested == \"auto\":\n routed = auto_route_provider(request.input[\"query\"], config)\n if routed.get(\"error_type\"):\n # ProviderPlan is intentionally non-empty even for a local\n # preflight error; execution returns before this placeholder can\n # be contacted.\n return ProviderPlan((\"keenable\",), \"keenable\", routing_metadata=dict(routed))\n selected = str(routed[\"provider\"])\n else:\n selected = requested\n routed = {}\n candidates = [selected]\n disabled = set(auto_config.get(\"disabled_providers\", []))\n research_mode = str(request.options.get(\"mode\") or \"normal\") == \"research\"\n fixed_provider_mode = (\n requested == \"auto\"\n and auto_config.get(\"enabled\", True) is False\n and bool(config.get(\"default_provider\"))\n )\n expand_candidates = routing_request.get(\"allow_fallback\", requested == \"auto\")\n if not fixed_provider_mode and (\n (research_mode and requested == \"auto\") or expand_candidates\n ):\n for provider in auto_config.get(\"provider_priority\", list(SEARCH_PROVIDER_IDS)):\n if (\n provider not in candidates\n and provider not in disabled\n and _provider_auto_allowed(provider, auto_config)\n and provider_configured(provider, config)\n ):\n candidates.append(provider)\n if research_mode:\n # Research is a bounded fan-out, not a fallback chain. Use the same\n # diversity-biased provider selection as the standalone research path,\n # constrained to the v3 planner's eligible candidates.\n candidates = select_research_providers(\n primary_provider=selected,\n provider_priority=list(\n auto_config.get(\"provider_priority\", list(SEARCH_PROVIDER_IDS))\n ),\n available_providers=set(candidates),\n max_providers=3,\n )\n return ProviderPlan(tuple(candidates), selected, routing_metadata=dict(routed))\n\n\n# Spare results requested per search attempt so filtering can refill the top N.\nSEARCH_OVERFETCH_EXTRA = 5\nSEARCH_OVERFETCH_CAP = 20\n\n\ndef _search_args_from_v3(request: RequestV3, config: Dict[str, Any]):\n # CLI defaults without argparse; request data is never parsed as argv.\n args = default_search_args(config)\n options = request.options\n routing_request = request.routing\n args.query = request.input[\"query\"]\n args.provider = routing_request.get(\"provider\") or \"auto\"\n args.max_results = options.get(\"max_results\", 5)\n args.exa_depth = options.get(\"depth\", \"normal\")\n for name in (\"time_range\", \"freshness\", \"search_type\", \"quality_report\"):\n if options.get(name):\n setattr(args, name, options[name])\n for name in (\"include_domains\", \"exclude_domains\"):\n if options.get(name):\n setattr(args, name, list(options[name]))\n args.mode = options.get(\"mode\", \"normal\")\n if args.mode != \"normal\":\n args.research_time_budget = options.get(\"research_time_budget\", 55.0)\n locale = options.get(\"locale\") or {}\n for name in (\"language\", \"country\"):\n if locale.get(name):\n setattr(args, name, locale[name])\n if routing_request.get(\"allow_fallback\") and routing_request.get(\"mode\") == \"fixed\":\n args.allow_fallback = True\n if request.cache.get(\"mode\") == \"bypass\":\n args.no_cache = True\n if \"ttl_seconds\" in request.cache:\n args.cache_ttl = request.cache[\"ttl_seconds\"]\n return args\n\n\ndef _execute_research_v3(\n request: RequestV3, plan: ProviderPlan, config: Dict[str, Any]\n) -> CapabilityExecution:\n \"\"\"Execute the planned research fan-out through authoritative v3 attempts.\"\"\"\n v3_config = config.get(\"v3\") or {}\n state_path = v3_config.get(\"state_path\") or os.path.join(\n str(CACHE_DIR), \"v3\", \"state.sqlite3\"\n )\n store = SQLiteStateStore(state_path)\n budget_limit = int(\n request.budget.get(\n \"max_provider_attempts\",\n v3_config.get(\"default_max_provider_attempts\", 3),\n )\n )\n # Research already gets resilience from independent provider fan-out. A\n # single try per member prevents an overdue daemon task from starting a new\n # billable retry after the caller's research deadline has elapsed.\n engine = AttemptEngine(store, max_attempts=1)\n providers = list(plan.candidate_order)\n scope = request.request_id or plan.execution_id\n contexts = {}\n receipts_by_provider = {}\n payloads_by_provider: Dict[str, Dict[str, Any]] = {}\n operation_started: Dict[str, Tuple[float, float]] = {}\n timed_out_providers: set[str] = set()\n daily_budget = _daily_preflight_budget(config)\n\n for provider in providers:\n contexts[provider] = provider_attempt_context(\n store, provider, Capability.SEARCH, config.get(provider) or {}, get_api_key(provider, config),\n budget_scope=scope, budget_limit_units=budget_limit, **daily_budget,\n )\n\n def execute_provider(provider: str) -> Dict[str, Any]:\n def operation() -> Dict[str, Any]:\n operation_started[provider] = (time.time(), time.monotonic())\n args = _search_args_from_v3(request, config)\n args.provider = provider\n args.mode = \"normal\"\n args.research_providers = \"\"\n args.allow_fallback = False\n args.no_cache = True\n args._v3_engine_owned_attempt = True\n args._v3_research_member = True\n provider_payload, exit_code = _execute_search_request_core(args, config)\n if exit_code:\n raise ProviderRequestError(\n str(provider_payload.get(\"error\") or \"provider failed\"),\n transient=False,\n )\n return provider_payload\n\n attempted = engine.execute(contexts[provider], operation)\n receipts_by_provider[provider] = attempted.receipt\n if attempted.payload is None:\n message = (\n attempted.receipt.error.message\n if attempted.receipt.error is not None\n else \"Provider attempt did not return a payload.\"\n )\n raise ProviderRequestError(message)\n payloads_by_provider[provider] = attempted.payload\n return attempted.payload\n\n args = _search_args_from_v3(request, config)\n time_budget_seconds = float(\n request.options.get(\"research_time_budget\")\n or getattr(args, \"research_time_budget\", 55.0)\n )\n max_wall_time_ms = request.budget.get(\"max_wall_time_ms\")\n if (\n isinstance(max_wall_time_ms, int)\n and not isinstance(max_wall_time_ms, bool)\n and max_wall_time_ms > 0\n ):\n time_budget_seconds = min(time_budget_seconds, max_wall_time_ms / 1000)\n diversity_rerank, near_duplicate_threshold = _diversity_settings(config)\n quorum_settings = _research_quorum_settings(config)\n payload = run_research_mode(\n query=str(request.input.get(\"query\") or \"\"),\n research_providers=providers,\n execute_search=execute_provider,\n extract_urls=lambda urls: extract_plus(\n urls=urls,\n provider=\"auto\",\n config=config,\n ),\n max_results=int(request.options.get(\"max_results\") or 5),\n max_extract_urls=int(getattr(args, \"research_extract_count\", 3) or 3),\n time_budget_seconds=time_budget_seconds,\n on_provider_timeout=timed_out_providers.add,\n diversity_rerank=diversity_rerank,\n near_duplicate_threshold=near_duplicate_threshold,\n quorum_enabled=quorum_settings[\"enabled\"],\n quorum_min_contributing_providers=quorum_settings[\n \"min_contributing_providers\"\n ],\n quorum_result_target_cap=quorum_settings[\"result_target_cap\"],\n quorum_min_unique_domains=quorum_settings[\"min_unique_domains\"],\n )\n extraction_error = str(\n (payload.get(\"routing\") or {}).get(\"extraction_error\") or \"\"\n ).lower()\n if \"research time budget exhausted\" in extraction_error:\n payload[\"_v3_budget_limited\"] = True\n\n completed_providers = set(\n (payload.get(\"routing\") or {}).get(\"providers_queried\") or []\n )\n receipts = []\n for provider in providers:\n receipt = (\n None\n if provider in timed_out_providers\n else receipts_by_provider.get(provider)\n )\n if receipt is None:\n started = operation_started.get(provider)\n if started is not None:\n wall_started, monotonic_started = started\n receipt = engine.cancel_started(\n contexts[provider],\n started_at=wall_started,\n duration_ms=int(\n max(0.0, time.monotonic() - monotonic_started) * 1000\n ),\n ).receipt\n else:\n receipt = engine.skip(\n contexts[provider], SkipReason.DEADLINE_EXCEEDED\n ).receipt\n receipts.append(receipt)\n\n if not completed_providers:\n payload[\"error\"] = \"All research providers failed\"\n\n routing_info = dict(plan.routing_metadata)\n routing_info.setdefault(\n \"auto_routed\", str(request.routing.get(\"provider\") or \"auto\") == \"auto\"\n )\n routing_info.setdefault(\"routing_policy\", ROUTING_POLICY)\n payload = _finalize_research_result(\n payload,\n args=args,\n config=config,\n routing_info=routing_info,\n providers_considered=providers,\n research_providers=providers,\n cooldown_skips=[],\n provider_payloads=payloads_by_provider,\n )\n\n raw_results = []\n for provider in providers:\n if provider not in completed_providers:\n continue\n provider_payload = payloads_by_provider.get(provider) or {}\n provider_items = provider_payload.get(\"_v3_raw_results\") or provider_payload.get(\n \"results\"\n ) or []\n for item in provider_items:\n if not isinstance(item, dict):\n continue\n observation = dict(item)\n observation.setdefault(\"provider\", provider)\n raw_results.append(observation)\n payload[\"_v3_raw_results\"] = raw_results\n\n stages = [\"admission\", \"provider_attempt\"]\n if any(receipt.error is not None for receipt in receipts):\n stages.append(\"error_classification\")\n stages.extend([\"retry_circuit_update\", \"dedup_fingerprint\"])\n return CapabilityExecution(\n payload=payload,\n provider_attempts=tuple(receipts),\n stages=tuple(stages),\n )\n\n\ndef _execute_search_v3(\n request: RequestV3, plan: ProviderPlan, config: Dict[str, Any]\n) -> CapabilityExecution:\n profile_error = plan.routing_metadata.get(\"error_type\")\n if profile_error:\n return CapabilityExecution(\n payload={\n \"error\": plan.routing_metadata.get(\"error\"),\n \"error_type\": profile_error,\n \"provider\": \"auto\",\n \"query\": request.input[\"query\"],\n \"results\": [],\n \"metadata\": {\"profile\": config.get(\"profile\")},\n },\n stages=(\"admission\",),\n )\n if str(request.options.get(\"mode\") or \"normal\") == \"research\":\n return _execute_research_v3(request, plan, config)\n v3_config = config.get(\"v3\") or {}\n state_path = v3_config.get(\"state_path\") or os.path.join(\n str(CACHE_DIR), \"v3\", \"state.sqlite3\"\n )\n store = SQLiteStateStore(state_path)\n budget_limit = int(\n request.budget.get(\n \"max_provider_attempts\",\n v3_config.get(\"default_max_provider_attempts\", 3),\n )\n )\n candidates = list(plan.candidate_order)\n has_fallback = len(candidates) > 1\n # With a fallback waiting, one quick try beats two slow ones: a provider\n # gets a single attempt and a shorter socket timeout. Without one (an\n # explicit provider), the retry and the provider's own timeout stay.\n engine = AttemptEngine(\n store,\n max_attempts=1 if has_fallback else int(v3_config.get(\"max_attempts_per_provider\", 2)),\n )\n attempt_timeout = (\n _positive_float(v3_config.get(\"attempt_timeout_seconds\"), 10.0) if has_fallback else None\n )\n hedge_floor = _positive_float(v3_config.get(\"hedge_min_delay_seconds\"), 2.5)\n scope = request.request_id or plan.execution_id\n daily_budget = _daily_preflight_budget(config)\n deadline = request_deadline(request.budget)\n contexts = {\n provider: provider_attempt_context(\n store, provider, Capability.SEARCH, config.get(provider) or {}, get_api_key(provider, config),\n budget_scope=scope, budget_limit_units=budget_limit, deadline_monotonic=deadline, **daily_budget,\n )\n for provider in candidates\n }\n\n def operation_for(current_provider: str):\n def operation():\n args = _search_args_from_v3(request, config)\n args.provider = current_provider\n args.allow_fallback = False\n args.no_cache = True\n args._v3_engine_owned_attempt = True\n # Ask for a few spare results: spam removal and the per-domain cap\n # then refill the list instead of leaving duplicates in the top N.\n requested = int(args.max_results or 5)\n args._v3_requested_results = requested\n args.max_results = min(SEARCH_OVERFETCH_CAP, requested + SEARCH_OVERFETCH_EXTRA)\n if str(request.routing.get(\"provider\") or \"auto\") == \"auto\":\n # Rank and report with the auto-routing decision, not the\n # fixed-provider routing of this attempt.\n args._v3_planned_routing = plan.routing_metadata\n with request_timeout_cap(attempt_timeout):\n provider_payload, exit_code = _execute_search_request_core(args, config)\n if exit_code:\n raise ProviderRequestError(\n str(provider_payload.get(\"error\") or \"provider failed\"),\n transient=False,\n )\n return provider_payload\n return operation\n\n def hedge_delay(provider: str) -> float:\n usual = latency_quantile(provider, 0.75) or 0.0\n delay = max(hedge_floor, usual)\n return min(delay, attempt_timeout) if attempt_timeout else delay\n\n race = _race_providers(engine, candidates, contexts, operation_for, hedge_delay, deadline)\n if race.provider is None and candidates and all(\n receipt.skip_reason in _OUTAGE_SKIPS for receipt in race.receipts\n ):\n race = _last_resort_probe(engine, store, candidates[0], contexts, operation_for, race)\n receipts = race.receipts\n payload = race.payload\n successful_provider = race.provider\n\n if payload is None:\n payload = {\n \"error\": \"All providers failed\",\n \"provider\": plan.selected_provider,\n \"query\": request.input[\"query\"],\n \"results\": [],\n \"routing\": {\n \"provider\": plan.selected_provider,\n \"fallback_used\": False,\n },\n \"provider_errors\": [\n {\n \"provider\": receipt.provider,\n \"error\": (\n receipt.error.message\n if receipt.error is not None\n else _SKIP_REASON_TEXT.get(receipt.skip_reason, receipt.skip_reason.value)\n if receipt.skip_reason is not None\n else \"provider attempt failed\"\n ),\n }\n for receipt in receipts\n ],\n }\n add_provider_setup_guidance(payload, \"search\", list(plan.candidate_order), config,\n requested_provider=str(request.routing.get(\"provider\") or \"auto\"))\n else:\n if race.superseded:\n payload[\"_v3_superseded_providers\"] = list(race.superseded)\n routing = payload.setdefault(\"routing\", {})\n requested = str(request.routing.get(\"provider\") or \"auto\")\n if requested == \"auto\":\n routing.update(plan.routing_metadata)\n routing[\"auto_routed\"] = True\n routing[\"provider\"] = successful_provider\n payload[\"provider\"] = successful_provider\n if successful_provider != plan.selected_provider:\n routing[\"fallback_used\"] = True\n routing[\"original_provider\"] = plan.selected_provider\n routing[\"provider\"] = successful_provider\n # No value says \"slower than usual and lost the race\". For that\n # case selected_failed is the closest: the receipt rules tie it to\n # a cancelled or failed prior attempt, which is what is recorded.\n routing[\"fallback_reason\"] = (\n \"insufficient_results\"\n if plan.selected_provider in race.empty_providers\n else \"selected_failed\"\n )\n\n stages = [\"admission\", \"provider_attempt\"]\n if any(receipt.error is not None for receipt in receipts):\n stages.append(\"error_classification\")\n stages.append(\"retry_circuit_update\")\n if sum(receipt.decision == \"attempted\" for receipt in receipts) > 1:\n stages.append(\"fallback\")\n stages.append(\"dedup_fingerprint\")\n return CapabilityExecution(\n payload=payload,\n provider_attempts=tuple(receipts),\n stages=tuple(stages),\n )\n\n\n_MAX_IN_FLIGHT = 2\n\n# Skips that mean \"recently down\", not \"not allowed\". When they exclude every\n# candidate, the top one still gets a probe, as 4.3.5 did: an answer from a\n# provider that may have recovered beats a guaranteed \"All providers failed\".\n_OUTAGE_SKIPS = frozenset({SkipReason.CIRCUIT_OPEN})\n\n\ndef _last_resort_probe(engine, store, provider, contexts, operation_for, race):\n context = contexts[provider]\n now = int(time.time())\n for error_class in (ErrorClass.TRANSIENT, ErrorClass.TIMEOUT):\n store.record_success(context.circuit_key, error_class, now=now)\n execution = engine.execute(context, operation_for(provider))\n receipts = [execution.receipt if r.provider == provider else r for r in race.receipts]\n payload = execution.payload\n if payload is None:\n return _Race(None, None, receipts, race.empty_providers)\n empty = [] if payload.get(\"results\") else [provider]\n return _Race(provider, payload, receipts, empty)\n\n\n# Why a provider was not called, as the search error lists it. The first\n# failure names its cause (\"Out of credits: ...\"); while the resulting block\n# lasts, these keep it readable instead of a bare \"quota_blocked\". WSP's own\n# fixed text, never the provider's.\n_SKIP_REASON_TEXT = {\n SkipReason.QUOTA_BLOCKED: (\n \"Out of credits or quota at its last call; skipped for up to an hour. \"\n \"Top up the account or remove its key\"\n ),\n SkipReason.AUTH_BLOCKED: \"Authentication failed at its last call; skipped for a few minutes. Check its API key\",\n SkipReason.RATE_LIMITED: \"Rate limit reached at its last call; skipped until it resets\",\n SkipReason.CIRCUIT_OPEN: \"Failed several times in a row; skipped for about a minute\",\n SkipReason.BUDGET_BLOCKED: \"Call budget for this request or day is used up\",\n SkipReason.DEADLINE_EXCEEDED: \"Not tried: the request ran out of time\",\n}\n\n\n@dataclass\nclass _Race:\n provider: Optional[str]\n payload: Optional[Dict[str, Any]]\n receipts: List[Any]\n empty_providers: List[str]\n # Still running when another provider's non-empty answer won; cancelled by\n # that win, not by the request's time budget.\n superseded: List[str] = field(default_factory=list)\n\n\ndef _positive_float(value: Any, default: float) -> float:\n try:\n number = float(value)\n except (TypeError, ValueError):\n return default\n return number if number > 0 else default\n\n\ndef _race_providers(engine, candidates, contexts, operation_for, hedge_delay, deadline) -> _Race:\n \"\"\"Run candidates in order, hedged: the first non-empty answer wins.\n\n The next candidate starts as soon as the running one fails or answers\n empty, or when it is slower than its hedge delay (its usual p75 latency,\n at least the configured floor). At most two run at once. A provider that\n is still running when another wins is reported as cancelled; it finishes\n in the background without affecting the answer.\n \"\"\"\n done: \"queue.Queue[str]\" = queue.Queue()\n tasks: Dict[str, Tuple[DaemonTask, float, float]] = {}\n finished: Dict[str, Any] = {}\n pending = list(candidates)\n winner = None\n hedge_at = None\n deadline_hit = False\n\n def launch() -> None:\n nonlocal hedge_at\n provider = pending.pop(0)\n if deadline is not None and time.monotonic() >= deadline:\n finished[provider] = engine.skip(contexts[provider], SkipReason.DEADLINE_EXCEEDED)\n return\n task = DaemonTask(engine.execute, contexts[provider], operation_for(provider))\n tasks[provider] = (task, time.time(), time.monotonic())\n task.add_done_callback(lambda _task, name=provider: done.put(name))\n hedge_at = time.monotonic() + hedge_delay(provider)\n\n def in_flight() -> List[str]:\n return [provider for provider in tasks if provider not in finished]\n\n launch()\n while True:\n running = in_flight()\n if not running:\n if winner is not None or not pending:\n break\n launch()\n continue\n can_hedge = bool(pending) and len(running) \u003c _MAX_IN_FLIGHT\n wait = max(0.0, hedge_at - time.monotonic()) if can_hedge and hedge_at is not None else None\n if deadline is not None:\n remaining = max(0.0, deadline - time.monotonic())\n wait = remaining if wait is None else min(wait, remaining)\n try:\n provider = done.get(timeout=wait)\n except queue.Empty:\n if deadline is not None and time.monotonic() >= deadline:\n deadline_hit = True\n break\n launch() # hedge: the running provider is slower than usual\n continue\n try:\n execution = tasks[provider][0].result(timeout=0)\n except Exception: # pragma: no cover - engine.execute converts provider errors\n execution = None\n finished[provider] = execution\n payload = execution.payload if execution is not None else None\n if payload is not None and (payload.get(\"results\") or []):\n winner = provider\n break\n if pending and len(in_flight()) \u003c _MAX_IN_FLIGHT:\n launch()\n\n empty_providers = [\n provider for provider in candidates\n if finished.get(provider) is not None and finished[provider].payload is not None\n and not (finished[provider].payload.get(\"results\") or [])\n ]\n if winner is None and empty_providers:\n winner = empty_providers[0] # every answer was empty: report the truthful empty result\n receipts = []\n superseded = []\n for provider in candidates:\n execution = finished.get(provider)\n if execution is not None:\n receipts.append(execution.receipt)\n elif provider in tasks:\n if not deadline_hit:\n superseded.append(provider)\n _task, started_wall, started_monotonic = tasks[provider]\n receipts.append(engine.cancel_started(\n contexts[provider],\n started_at=started_wall,\n duration_ms=int(max(0.0, time.monotonic() - started_monotonic) * 1000),\n ).receipt)\n else:\n reason = SkipReason.DEADLINE_EXCEEDED if deadline_hit else SkipReason.POLICY_EXCLUDED\n receipts.append(engine.skip(contexts[provider], reason).receipt)\n payload = finished[winner].payload if winner is not None else None\n return _Race(winner, payload, receipts, empty_providers, superseded)\n\n\ndef _search_cache_vary(\n request: RequestV3, plan: ProviderPlan, config: Dict[str, Any]\n) -> Dict[str, Any]:\n \"\"\"Keep language \"auto\" apart from fixed-language entries in the v3 cache.\n\n A per-call \"auto\" never appears in the request (the contract has no value\n for it), so the cache key would equal that of a plain request while the\n provider call differs.\n \"\"\"\n locale = (config.get(\"defaults\") or {}).get(\"locale\") or {}\n return {\"language\": AUTO_LANGUAGE} if is_auto_language(locale.get(\"language\")) else {}\n\n\ndef _search_cache_write_eligible(\n _request: RequestV3,\n _plan: ProviderPlan,\n response: ResponseV3,\n _legacy_payload: Dict[str, Any],\n _config: Dict[str, Any],\n) -> bool:\n \"\"\"Cache only answers that have results.\n\n When every candidate answers with nothing, the cause is often transient. A\n cached empty answer would keep repeats from reaching a provider for the\n whole TTL, which defeats the empty-answer fallback.\n \"\"\"\n return bool(response.results)\n\n\ndef _search_adapter() -> CapabilityAdapter:\n return CapabilityAdapter(\n capability=Capability.SEARCH,\n plan=_plan_search_v3,\n execute=_execute_search_v3,\n normalize=response_from_legacy,\n cache_vary=_search_cache_vary,\n cache_write_eligible=_search_cache_write_eligible,\n )\n\n\ndef run_search_request_v3(\n request: RequestV3,\n *,\n config: Optional[Dict[str, Any]] = None,\n) -> ResponseV3:\n \"\"\"Execute a native search RequestV3 through the canonical orchestrator.\"\"\"\n runtime_config = apply_profile_effects(config) if config is not None else load_config()\n return execute_v3_request(request, _search_adapter(), runtime_config).response\n\n\ndef run_search_request(\n *,\n query: str,\n provider: str = \"auto\",\n count: int = 5,\n exa_depth: str = \"normal\",\n time_range: Optional[str] = None,\n freshness: Optional[str] = None,\n search_type: Optional[str] = None,\n include_domains: Optional[List[str]] = None,\n exclude_domains: Optional[List[str]] = None,\n mode: str = \"normal\",\n quality_report: bool = False,\n research_time_budget: float = 55.0,\n language: Optional[str] = None,\n country: Optional[str] = None,\n no_cache: bool = False,\n cache_ttl: Optional[int] = None,\n max_wall_time_ms: Optional[int] = None,\n config: Optional[Dict[str, Any]] = None,\n) -> Dict[str, Any]:\n \"\"\"Run a search in-process and return the result dict the CLI would emit.\n\n Mirrors the argv the Hermes plugin used to pass to the subprocess, then routes\n it through the same parser and pipeline so behaviour is identical without the\n interpreter-startup and JSON round-trip cost of spawning ``search.py``.\n \"\"\"\n if not query and not (include_domains or exclude_domains):\n return {\"error\": \"query is required\", \"provider\": provider, \"query\": query, \"results\": []}\n # Search engines ignore or reject longer queries; a 50k-char query only\n # bloats the echoed answer. Per-provider limits are applied further down.\n if isinstance(query, str) and len(query) > MAX_QUERY_CHARS:\n query = query[:MAX_QUERY_CHARS]\n requested = str(provider or \"auto\").strip().lower()\n suffix_filters = _providers.public_suffix_entries(include_domains, exclude_domains)\n try:\n freshness = _providers.normalize_freshness(freshness)\n search_type = _providers.normalize_search_type(search_type)\n domain_filters(include_domains, exclude_domains) # raises when include_domains has no usable domain\n if suffix_filters and requested in _providers.DOMAIN_SUFFIX_FILTER_UNSUPPORTED:\n raise ValueError(_providers.domain_suffix_unsupported_message(requested))\n except ValueError as exc:\n return {\"error\": str(exc), \"provider\": provider, \"query\": query, \"results\": []}\n # One list of entries whatever the caller sent (\"a.com, b.com\" or a bare string).\n include_domains = domain_filter_tokens(include_domains) or None\n exclude_domains = domain_filter_tokens(exclude_domains) or None\n config = apply_profile_effects(config) if config is not None else load_config()\n if suffix_filters:\n # A suffix filter (\".gov\") must not route to a provider that cannot apply\n # it, neither first nor as fallback or research member.\n config = copy.deepcopy(config)\n auto = config.setdefault(\"auto_routing\", {})\n auto[\"disabled_providers\"] = sorted(\n set(auto.get(\"disabled_providers\") or []) | _providers.DOMAIN_SUFFIX_FILTER_UNSUPPORTED\n )\n language, config = apply_auto_language(language, config)\n policy_mode = str((config.get(\"routing\") or {}).get(\"policy_mode\", \"classic\"))\n request = legacy_request_to_v3(\n Capability.SEARCH,\n policy_mode=policy_mode,\n payload={\n \"query\": query,\n \"provider\": provider or \"auto\",\n \"count\": count,\n \"depth\": exa_depth,\n \"time_range\": time_range,\n \"freshness\": freshness,\n \"search_type\": search_type,\n \"include_domains\": include_domains,\n \"exclude_domains\": exclude_domains,\n \"mode\": mode,\n \"quality_report\": quality_report,\n \"research_time_budget\": research_time_budget,\n \"language\": language,\n \"country\": country,\n \"no_cache\": bool(no_cache),\n \"cache_ttl\": int(cache_ttl) if cache_ttl is not None else 3600,\n \"max_wall_time_ms\": max_wall_time_ms,\n },\n )\n execution = execute_v3_request(request, _search_adapter(), config)\n return v3_response_to_legacy_search(execution)\n\n\ndef run_extract_request_v3(\n request: RequestV3,\n *,\n config: Optional[Dict[str, Any]] = None,\n) -> ResponseV3:\n \"\"\"Execute a native extract RequestV3 through the canonical orchestrator.\"\"\"\n return _extract.run_extract_request_v3(request, config=config or load_config())\n\n\ndef run_extract_request(\n urls: List[str],\n *,\n provider: str = \"auto\",\n output_format: str = \"markdown\",\n include_images: bool = False,\n include_raw_html: bool = False,\n render_js: bool = False,\n spans: bool = False,\n spans_query: Optional[str] = None,\n max_wall_time_ms: Optional[int] = None,\n config: Optional[Dict[str, Any]] = None,\n) -> Dict[str, Any]:\n \"\"\"Run URL extraction in-process and return the result dict.\"\"\"\n config = config or load_config()\n return extract_plus(\n urls=urls,\n provider=provider or \"auto\",\n output_format=output_format,\n include_images=include_images,\n include_raw_html=include_raw_html,\n render_js=render_js,\n spans=spans,\n spans_query=spans_query,\n max_wall_time_ms=max_wall_time_ms,\n config=config,\n )\n\n\nif __name__ == \"__main__\":\n main()\n","content_type":"text/x-python; charset=utf-8","language":"python","size":93235,"content_sha256":"d32e43f2ce2b1a9e0d779c6ae0f28291f3754aad9a3ce918078c742f4cee0a2a"},{"filename":"scripts/wsp_core/span_extraction_v3.py","content":"\"\"\"Deterministic, query-conditioned passage selection for WSP 3.1.\"\"\"\n\nfrom __future__ import annotations\n\nimport math\nimport re\nimport unicodedata\nfrom dataclasses import dataclass\nfrom typing import Callable, Dict, List, Union\n\n\nSpan = Dict[str, Union[int, float, str]]\nRanker = Callable[[str, str], float]\n\n_TOKEN_RE = re.compile(r\"[^\\W_]+(?:['\\N{RIGHT SINGLE QUOTATION MARK}][^\\W_]+)?\", re.UNICODE)\n_PARAGRAPH_BREAK_RE = re.compile(r\"(?:\\r?\\n[\\t \\f\\v]*){2,}\")\n_SENTENCE_END_RE = re.compile(r\"(?\u003c=[.!?])(?:[\\\"'\\N{RIGHT SINGLE QUOTATION MARK}\\)\\]]*)\\s+\")\n_MARKDOWN_HEADING_RE = re.compile(r\"(?m)^[\\t ]{0,3}(#{1,6})(?:[\\t ]+|$)\")\n\n# Heading matches add bounded, whole-section candidates rather than extending\n# ordinary passages. This independently implemented policy is inspired by the\n# Hound/Master-Fetch v11.2.0 interaction model (MIT; Bishesh Bhandari,\n# https://github.com/dondai1234/master-fetch) through respectful upstream\n# collaboration; WSP neither imports nor copies upstream code.\n_MAX_HEADING_SECTION_CANDIDATES = 2\n_MAX_HEADING_SECTION_CHARS = 1_200\n\n\n@dataclass(frozen=True)\nclass _Candidate:\n start: int\n end: int\n text: str\n\n\n@dataclass(frozen=True)\nclass _MarkdownHeading:\n start: int\n level: int\n title_start: int\n title_end: int\n\n\ndef nfc_text(text: str) -> str:\n \"\"\"Return the canonical NFC string used for all span offsets.\"\"\"\n if not isinstance(text, str):\n raise TypeError(\"text must be a string\")\n return unicodedata.normalize(\"NFC\", text)\n\n\ndef _tokens(text: str) -> List[str]:\n return [match.group(0).casefold() for match in _TOKEN_RE.finditer(text)]\n\n\ndef _trimmed_candidate(text: str, start: int, end: int) -> _Candidate | None:\n while start \u003c end and text[start].isspace():\n start += 1\n while end > start and text[end - 1].isspace():\n end -= 1\n if start >= end:\n return None\n return _Candidate(start, end, text[start:end])\n\n\ndef _split_long_segment(text: str, start: int, end: int, limit: int) -> List[_Candidate]:\n pieces: List[_Candidate] = []\n cursor = start\n while cursor \u003c end:\n boundary = min(end, cursor + limit)\n if boundary \u003c end:\n whitespace = text.rfind(\" \", cursor + max(1, limit // 2), boundary + 1)\n newline = text.rfind(\"\\n\", cursor + max(1, limit // 2), boundary + 1)\n boundary = max(whitespace, newline)\n if boundary \u003c= cursor:\n boundary = min(end, cursor + limit)\n candidate = _trimmed_candidate(text, cursor, boundary)\n if candidate is not None:\n pieces.append(candidate)\n cursor = boundary\n while cursor \u003c end and text[cursor].isspace():\n cursor += 1\n return pieces\n\n\ndef _candidates(text: str, max_span_chars: int) -> List[_Candidate]:\n candidates: List[_Candidate] = []\n paragraph_start = 0\n paragraph_ranges = []\n for match in _PARAGRAPH_BREAK_RE.finditer(text):\n paragraph_ranges.append((paragraph_start, match.start()))\n paragraph_start = match.end()\n paragraph_ranges.append((paragraph_start, len(text)))\n\n for para_start, para_end in paragraph_ranges:\n sentence_start = para_start\n sentence_ranges = []\n for match in _SENTENCE_END_RE.finditer(text, para_start, para_end):\n sentence_ranges.append((sentence_start, match.start()))\n sentence_start = match.end()\n sentence_ranges.append((sentence_start, para_end))\n\n sentence_candidates: List[_Candidate] = []\n for start, end in sentence_ranges:\n candidate = _trimmed_candidate(text, start, end)\n if candidate is None:\n continue\n if candidate.end - candidate.start \u003c= max_span_chars:\n sentence_candidates.append(candidate)\n else:\n sentence_candidates.extend(\n _split_long_segment(text, candidate.start, candidate.end, max_span_chars)\n )\n candidates.extend(sentence_candidates)\n\n # Adjacent sentences make more useful passages when they fit. They are\n # deliberately candidates, not automatic output, so the ranker remains\n # the single selection seam.\n for index in range(len(sentence_candidates) - 1):\n start = sentence_candidates[index].start\n end = sentence_candidates[index + 1].end\n if end - start \u003c= max_span_chars:\n candidates.append(_Candidate(start, end, text[start:end]))\n\n unique = {(candidate.start, candidate.end): candidate for candidate in candidates}\n return [unique[key] for key in sorted(unique)]\n\n\ndef _heading_section_candidates(\n text: str,\n query: str,\n *,\n max_span_chars: int,\n max_sections: int,\n) -> List[_Candidate]:\n \"\"\"Return the most query-relevant bounded Markdown heading sections.\n\n A Markdown section begins at its ATX heading and ends at the next heading\n of the same or shallower level. Nested headings remain in their parent's\n section. The independent section-count and character caps prevent a\n matching title from turning a long page into unbounded context.\n \"\"\"\n query_terms = set(_tokens(query))\n if not query_terms or max_sections \u003c= 0:\n return []\n\n headings: List[_MarkdownHeading] = []\n for match in _MARKDOWN_HEADING_RE.finditer(text):\n title_start = match.end()\n title_end = text.find(\"\\n\", title_start)\n if title_end \u003c 0:\n title_end = len(text)\n headings.append(\n _MarkdownHeading(\n start=match.start(),\n level=len(match.group(1)),\n title_start=title_start,\n title_end=title_end,\n )\n )\n\n section_char_budget = min(max_span_chars, _MAX_HEADING_SECTION_CHARS)\n ranked: List[tuple[int, _Candidate]] = []\n for index, heading in enumerate(headings):\n matched_terms = query_terms & set(_tokens(text[heading.title_start:heading.title_end]))\n if not matched_terms:\n continue\n\n section_end = len(text)\n for later_heading in headings[index + 1 :]:\n if later_heading.level \u003c= heading.level:\n section_end = later_heading.start\n break\n candidate = _trimmed_candidate(\n text,\n heading.start,\n min(section_end, heading.start + section_char_budget),\n )\n # Do not return a partial heading title merely because a caller chose\n # an exceptionally small generic candidate limit.\n if candidate is not None and candidate.end >= heading.title_end:\n ranked.append((len(matched_terms), candidate))\n\n ranked.sort(key=lambda item: (-item[0], item[1].start, item[1].end))\n return [candidate for _, candidate in ranked[:max_sections]]\n\n\ndef _lexical_score(candidate_text: str, query: str, *, start: int, total: int) -> float:\n candidate_tokens = _tokens(candidate_text)\n if not candidate_tokens:\n return 0.0\n\n unique_tokens = set(candidate_tokens)\n lexical_density = min(len(candidate_tokens), 80) / max(1.0, len(candidate_text) / 8.0)\n diversity = len(unique_tokens) / len(candidate_tokens)\n density_score = min(1.0, lexical_density / 4.0) + (0.2 * diversity)\n position_prior = 0.08 * (1.0 - (start / max(1, total)))\n\n query_tokens = _tokens(query)\n if not query_tokens:\n return density_score + position_prior\n\n query_unique = set(query_tokens)\n term_overlap = len(query_unique & unique_tokens) / len(query_unique)\n query_shingles = set(zip(query_tokens, query_tokens[1:]))\n candidate_shingles = set(zip(candidate_tokens, candidate_tokens[1:]))\n shingle_overlap = (\n len(query_shingles & candidate_shingles) / len(query_shingles)\n if query_shingles\n else 0.0\n )\n occurrence_bonus = sum(candidate_tokens.count(term) for term in query_unique)\n occurrence_bonus = min(1.0, occurrence_bonus / max(1, len(query_tokens)))\n return (\n (4.0 * term_overlap)\n + (2.0 * shingle_overlap)\n + (0.5 * occurrence_bonus)\n + (0.2 * density_score)\n + position_prior\n )\n\n\ndef select_spans(\n text: str,\n query: str | None,\n *,\n max_spans: int = 3,\n max_span_chars: int = 600,\n ranker: Ranker | None = None,\n) -> List[Span]:\n \"\"\"Select deterministic, non-overlapping spans from NFC-normalized ``text``.\n\n ``ranker`` is the pluggable semantic-backend seam. When supplied, it is\n called as ``ranker(candidate_text, normalized_query)`` and must return a\n finite numeric score. The default ranker uses only lexical overlap,\n lexical density, and a mild position prior.\n \"\"\"\n if isinstance(max_spans, bool) or not isinstance(max_spans, int):\n raise TypeError(\"max_spans must be an integer\")\n if isinstance(max_span_chars, bool) or not isinstance(max_span_chars, int):\n raise TypeError(\"max_span_chars must be an integer\")\n if query is not None and not isinstance(query, str):\n raise TypeError(\"query must be a string or None\")\n if ranker is not None and not callable(ranker):\n raise TypeError(\"ranker must be callable or None\")\n if max_spans \u003c= 0 or max_span_chars \u003c= 0:\n return []\n\n normalized = nfc_text(text)\n normalized_query = nfc_text(query or \"\").strip()\n heading_candidates = _heading_section_candidates(\n normalized,\n normalized_query,\n max_span_chars=max_span_chars,\n max_sections=min(max_spans, _MAX_HEADING_SECTION_CANDIDATES),\n )\n all_candidates = {\n (candidate.start, candidate.end): candidate\n for candidate in [*_candidates(normalized, max_span_chars), *heading_candidates]\n }\n ranked = []\n score_by_range = {}\n for candidate in all_candidates.values():\n if ranker is None:\n score = _lexical_score(\n candidate.text,\n normalized_query,\n start=candidate.start,\n total=len(normalized),\n )\n else:\n score = ranker(candidate.text, normalized_query)\n if isinstance(score, bool) or not isinstance(score, (int, float)):\n raise TypeError(\"ranker scores must be numeric\")\n score = float(score)\n if not math.isfinite(score):\n raise ValueError(\"ranker scores must be finite\")\n ranked.append((score, candidate))\n score_by_range[(candidate.start, candidate.end)] = score\n\n ranked.sort(key=lambda item: (-item[0], item[1].start, item[1].end))\n selected: List[tuple[float, _Candidate]] = []\n for candidate in heading_candidates:\n if any(\n candidate.start \u003c existing.end and existing.start \u003c candidate.end\n for _, existing in selected\n ):\n continue\n selected.append((score_by_range[(candidate.start, candidate.end)], candidate))\n if len(selected) >= max_spans:\n break\n for score, candidate in ranked:\n if len(selected) >= max_spans:\n break\n if any(\n candidate.start \u003c existing.end and existing.start \u003c candidate.end\n for _, existing in selected\n ):\n continue\n selected.append((score, candidate))\n\n selected.sort(key=lambda item: item[1].start)\n return [\n {\n \"start\": candidate.start,\n \"end\": candidate.end,\n \"text\": normalized[candidate.start:candidate.end],\n \"score\": score,\n }\n for score, candidate in selected\n ]\n","content_type":"text/x-python; charset=utf-8","language":"python","size":11558,"content_sha256":"3aeade3651667b5adcb5c03c64ba8cde47e389898cae7e89299801e6637ac711"},{"filename":"scripts/wsp_core/state_migration_v3.py","content":"#!/usr/bin/env python3\n\"\"\"Dry-run-first migration of legacy WSP JSON state into SQLite v3.\n\nThe migration is deliberately reversible and compatibility-safe: both legacy\nJSON files are imported in one SQLite transaction, but neither source file is\nmodified or deleted. Raw legacy error strings are never persisted in SQLite.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport argparse\nimport hashlib\nimport json\nimport math\nimport os\nimport re\nimport shutil\nimport sqlite3\nimport tempfile\nimport time\nfrom dataclasses import asdict, dataclass\nfrom datetime import datetime, timezone\nfrom pathlib import Path\nfrom typing import Any\n\nfrom .state_store_v3 import SQLiteStateStore\n\n\nBACKUP_OWNER = \"web-search-plus-state-migration-v3\"\nBACKUP_SCHEMA_VERSION = 1\nMIGRATION_ID = \"legacy-json-v1\"\nMAX_SOURCE_BYTES = 8 * 1024 * 1024\nMAX_PROVIDERS = 256\nMAX_SAMPLES_PER_PROVIDER = 1000\n_PROVIDER_RE = re.compile(r\"^[a-z0-9][a-z0-9_-]{0,63}$\")\n_BACKUP_ID_RE = re.compile(r\"^[0-9]{8}T[0-9]{6}Z-[a-f0-9]{12}(?:-[0-9]+)?$\")\n\n\n@dataclass(frozen=True)\nclass MigrationReport:\n action: str\n status: str\n dry_run: bool\n sqlite_available: bool\n health_providers: int = 0\n adaptive_providers: int = 0\n adaptive_samples: int = 0\n source_digest: str | None = None\n backup_id: str | None = None\n error_code: str | None = None\n\n def to_dict(self) -> dict[str, Any]:\n return {key: value for key, value in asdict(self).items() if value is not None}\n\n\ndef render_migration_report(report: MigrationReport) -> str:\n \"\"\"Render a deterministic, support-safe report with no local paths.\"\"\"\n return json.dumps(report.to_dict(), sort_keys=True, separators=(\",\", \":\"))\n\n\ndef _safe_regular_file(path: Path) -> bool:\n if path.is_symlink():\n raise ValueError(\"symlink_source\")\n if path.exists() and not path.is_file():\n raise ValueError(\"non_regular_source\")\n return path.exists()\n\n\ndef _read_json_mapping(path: Path) -> dict[str, Any]:\n if not _safe_regular_file(path):\n return {}\n if path.stat().st_size > MAX_SOURCE_BYTES:\n raise ValueError(\"source_too_large\")\n try:\n value = json.loads(path.read_text(encoding=\"utf-8\"))\n except (OSError, UnicodeError, json.JSONDecodeError) as exc:\n raise ValueError(\"invalid_json\") from exc\n if not isinstance(value, dict) or len(value) > MAX_PROVIDERS:\n raise ValueError(\"invalid_mapping\")\n return value\n\n\ndef _provider_name(value: Any) -> str:\n if not isinstance(value, str):\n raise ValueError(\"invalid_provider\")\n normalized = value.strip().lower()\n if not _PROVIDER_RE.fullmatch(normalized):\n raise ValueError(\"invalid_provider\")\n return normalized\n\n\ndef _nonnegative_int(value: Any, *, default: int = 0) -> int:\n if value is None:\n return default\n if isinstance(value, bool) or not isinstance(value, (int, float)):\n raise ValueError(\"invalid_number\")\n numeric = float(value)\n if not math.isfinite(numeric) or numeric \u003c 0:\n raise ValueError(\"invalid_number\")\n return int(numeric)\n\n\ndef _normalize_health(raw: dict[str, Any]) -> list[tuple[str, int, int, int, int]]:\n rows: list[tuple[str, int, int, int, int]] = []\n for provider_value, entry in raw.items():\n provider = _provider_name(provider_value)\n if not isinstance(entry, dict):\n raise ValueError(\"invalid_health_entry\")\n rows.append(\n (\n provider,\n _nonnegative_int(entry.get(\"failure_count\")),\n _nonnegative_int(entry.get(\"cooldown_until\")),\n _nonnegative_int(entry.get(\"cooldown_seconds\")),\n _nonnegative_int(entry.get(\"last_failure_at\")),\n )\n )\n return sorted(rows)\n\n\ndef _normalize_samples(\n raw: dict[str, Any],\n) -> list[tuple[str, int, int, int, int, int]]:\n rows: list[tuple[str, int, int, int, int, int]] = []\n for provider_value, samples in raw.items():\n provider = _provider_name(provider_value)\n if not isinstance(samples, list) or len(samples) > MAX_SAMPLES_PER_PROVIDER:\n raise ValueError(\"invalid_sample_list\")\n for source_index, sample in enumerate(samples):\n if not isinstance(sample, dict):\n raise ValueError(\"invalid_sample\")\n latency = sample.get(\"lat\", 0)\n if isinstance(latency, bool) or not isinstance(latency, (int, float)):\n raise ValueError(\"invalid_latency\")\n latency_value = float(latency)\n if not math.isfinite(latency_value) or latency_value \u003c 0:\n raise ValueError(\"invalid_latency\")\n error = sample.get(\"err\", False)\n if not isinstance(error, bool):\n raise ValueError(\"invalid_error_flag\")\n rows.append(\n (\n provider,\n source_index,\n _nonnegative_int(sample.get(\"t\")),\n int(round(latency_value * 1000)),\n _nonnegative_int(sample.get(\"n\")),\n int(error),\n )\n )\n return sorted(rows)\n\n\ndef _source_digest(\n health_rows: list[tuple[str, int, int, int, int]],\n sample_rows: list[tuple[str, int, int, int, int, int]],\n) -> str:\n canonical = json.dumps(\n {\"health\": health_rows, \"adaptive_samples\": sample_rows},\n ensure_ascii=True,\n separators=(\",\", \":\"),\n sort_keys=True,\n ).encode(\"utf-8\")\n return hashlib.sha256(canonical).hexdigest()\n\n\ndef _nearest_existing_parent(path: Path) -> Path:\n current = path\n while not current.exists() and current != current.parent:\n current = current.parent\n return current\n\n\ndef _sqlite_preflight(path: Path) -> tuple[bool, str | None]:\n try:\n if path.is_symlink():\n return False, None\n if not path.exists():\n parent = _nearest_existing_parent(path.parent)\n return parent.is_dir() and os.access(parent, os.W_OK), None\n if not path.is_file():\n return False, None\n uri = f\"file:{path.resolve()}?mode=ro&immutable=1\"\n connection = sqlite3.connect(uri, uri=True)\n try:\n quick_check = connection.execute(\"PRAGMA quick_check\").fetchone()\n if not quick_check or quick_check[0] != \"ok\":\n return False, None\n exists = connection.execute(\n \"SELECT 1 FROM sqlite_master WHERE type='table' AND name=?\",\n (\"legacy_state_migrations\",),\n ).fetchone()\n if not exists:\n return True, None\n row = connection.execute(\n \"SELECT source_digest FROM legacy_state_migrations WHERE migration_id=?\",\n (MIGRATION_ID,),\n ).fetchone()\n return True, str(row[0]) if row else None\n finally:\n connection.close()\n except (OSError, sqlite3.Error):\n return False, None\n\n\ndef _sha256_file(path: Path) -> str:\n digest = hashlib.sha256()\n with path.open(\"rb\") as handle:\n for chunk in iter(lambda: handle.read(1024 * 1024), b\"\"):\n digest.update(chunk)\n return digest.hexdigest()\n\n\ndef _source_file_hashes(cache_root: Path) -> dict[str, str | None]:\n hashes: dict[str, str | None] = {}\n for name in (\"provider_health.json\", \"provider_stats.json\"):\n source = cache_root / name\n hashes[name] = _sha256_file(source) if _safe_regular_file(source) else None\n return hashes\n\n\ndef _source_files_match(\n cache_root: Path, expected: dict[str, str | None]\n) -> bool:\n try:\n return _source_file_hashes(cache_root) == expected\n except (OSError, ValueError):\n return False\n\n\ndef _copy_secure(source: Path, destination: Path) -> None:\n _safe_regular_file(source)\n shutil.copyfile(source, destination)\n os.chmod(destination, 0o600)\n\n\ndef _atomic_json(path: Path, payload: dict[str, Any]) -> None:\n descriptor, temporary = tempfile.mkstemp(\n prefix=f\".{path.name}.\", suffix=\".tmp\", dir=str(path.parent)\n )\n try:\n os.fchmod(descriptor, 0o600)\n with os.fdopen(descriptor, \"w\", encoding=\"utf-8\") as handle:\n json.dump(payload, handle, sort_keys=True, separators=(\",\", \":\"))\n handle.write(\"\\n\")\n handle.flush()\n os.fsync(handle.fileno())\n os.replace(temporary, path)\n except Exception:\n try:\n os.unlink(temporary)\n except OSError:\n pass\n raise\n\n\ndef _next_backup_id(backup_root: Path, now: int, digest: str) -> str:\n timestamp = datetime.fromtimestamp(now, timezone.utc).strftime(\"%Y%m%dT%H%M%SZ\")\n base = f\"{timestamp}-{digest[:12]}\"\n candidate = base\n counter = 1\n while (backup_root / candidate).exists():\n candidate = f\"{base}-{counter}\"\n counter += 1\n return candidate\n\n\ndef _create_backup(\n *,\n cache_root: Path,\n state_path: Path,\n backup_root: Path,\n now: int,\n source_digest: str,\n health_providers: int,\n adaptive_providers: int,\n adaptive_samples: int,\n expected_source_hashes: dict[str, str | None],\n) -> str:\n secret_path = state_path.with_name(f\"{state_path.name}.secret\")\n if state_path.is_symlink() or secret_path.is_symlink():\n raise OSError(\"unsafe_state_path\")\n if not _source_files_match(cache_root, expected_source_hashes):\n raise OSError(\"legacy_source_changed\")\n if backup_root.is_symlink():\n raise OSError(\"unsafe_backup_root\")\n backup_root.mkdir(parents=True, exist_ok=True, mode=0o700)\n os.chmod(backup_root, 0o700)\n backup_id = _next_backup_id(backup_root, now, source_digest)\n backup_dir = backup_root / backup_id\n backup_dir.mkdir(mode=0o700)\n files: dict[str, str] = {}\n source_paths = {\n \"provider_health.json\": cache_root / \"provider_health.json\",\n \"provider_stats.json\": cache_root / \"provider_stats.json\",\n \"state.sqlite3.secret\": secret_path,\n }\n manifest = {\n \"owner\": BACKUP_OWNER,\n \"schema_version\": BACKUP_SCHEMA_VERSION,\n \"backup_id\": backup_id,\n \"created_at\": now,\n \"source_digest\": source_digest,\n \"database_existed\": state_path.exists(),\n \"secret_existed\": secret_path.exists(),\n \"health_existed\": source_paths[\"provider_health.json\"].exists(),\n \"stats_existed\": source_paths[\"provider_stats.json\"].exists(),\n \"health_providers\": health_providers,\n \"adaptive_providers\": adaptive_providers,\n \"adaptive_samples\": adaptive_samples,\n \"files\": files,\n }\n try:\n if state_path.exists():\n _safe_regular_file(state_path)\n target = backup_dir / \"state.sqlite3\"\n source_connection = sqlite3.connect(\n f\"file:{state_path.resolve()}?mode=ro\", uri=True\n )\n destination_connection = sqlite3.connect(target)\n try:\n source_connection.backup(destination_connection)\n finally:\n destination_connection.close()\n source_connection.close()\n os.chmod(target, 0o600)\n files[target.name] = _sha256_file(target)\n for name, source in source_paths.items():\n if source.exists():\n target = backup_dir / name\n _copy_secure(source, target)\n files[name] = _sha256_file(target)\n for name, expected_digest in expected_source_hashes.items():\n if files.get(name) != expected_digest:\n raise OSError(\"legacy_source_changed\")\n if not _source_files_match(cache_root, expected_source_hashes):\n raise OSError(\"legacy_source_changed\")\n _atomic_json(backup_dir / \"manifest.json\", manifest)\n except Exception:\n shutil.rmtree(backup_dir, ignore_errors=True)\n if backup_root.exists() and not any(backup_root.iterdir()):\n backup_root.rmdir()\n raise\n return backup_id\n\n\ndef _report(\n *,\n action: str,\n status: str,\n dry_run: bool,\n sqlite_available: bool,\n health_rows: list[tuple[str, int, int, int, int]] | None = None,\n sample_rows: list[tuple[str, int, int, int, int, int]] | None = None,\n source_digest: str | None = None,\n backup_id: str | None = None,\n error_code: str | None = None,\n) -> MigrationReport:\n health_rows = health_rows or []\n sample_rows = sample_rows or []\n return MigrationReport(\n action=action,\n status=status,\n dry_run=dry_run,\n sqlite_available=sqlite_available,\n health_providers=len(health_rows),\n adaptive_providers=len({row[0] for row in sample_rows}),\n adaptive_samples=len(sample_rows),\n source_digest=source_digest,\n backup_id=backup_id,\n error_code=error_code,\n )\n\n\ndef migrate_legacy_state(\n *,\n cache_root: str | Path,\n state_path: str | Path,\n backup_root: str | Path | None = None,\n dry_run: bool = True,\n now: int | None = None,\n) -> MigrationReport:\n \"\"\"Plan or atomically apply the two-file legacy-state migration.\"\"\"\n cache_root = Path(cache_root)\n state_path = Path(state_path)\n backup_root = (\n Path(backup_root)\n if backup_root is not None\n else state_path.parent / \"migration-backups\"\n )\n action = \"dry_run\" if dry_run else \"apply\"\n try:\n source_file_hashes = _source_file_hashes(cache_root)\n health_raw = _read_json_mapping(cache_root / \"provider_health.json\")\n stats_raw = _read_json_mapping(cache_root / \"provider_stats.json\")\n if _source_file_hashes(cache_root) != source_file_hashes:\n raise ValueError(\"legacy_source_changed\")\n health_rows = _normalize_health(health_raw)\n sample_rows = _normalize_samples(stats_raw)\n except (OSError, ValueError):\n return _report(\n action=action,\n status=\"blocked\",\n dry_run=dry_run,\n sqlite_available=False,\n error_code=\"invalid_legacy_state\",\n )\n digest = _source_digest(health_rows, sample_rows)\n sqlite_available, current_digest = _sqlite_preflight(state_path)\n if not sqlite_available:\n return _report(\n action=action,\n status=\"degraded\",\n dry_run=dry_run,\n sqlite_available=False,\n health_rows=health_rows,\n sample_rows=sample_rows,\n source_digest=digest,\n error_code=\"sqlite_unavailable\",\n )\n if not health_raw and not stats_raw:\n return _report(\n action=action,\n status=\"unchanged\",\n dry_run=dry_run,\n sqlite_available=True,\n source_digest=digest,\n )\n if current_digest == digest:\n return _report(\n action=action,\n status=\"unchanged\",\n dry_run=dry_run,\n sqlite_available=True,\n health_rows=health_rows,\n sample_rows=sample_rows,\n source_digest=digest,\n )\n if dry_run:\n return _report(\n action=action,\n status=\"ready\",\n dry_run=True,\n sqlite_available=True,\n health_rows=health_rows,\n sample_rows=sample_rows,\n source_digest=digest,\n )\n\n applied_at = int(now if now is not None else time.time())\n try:\n backup_id = _create_backup(\n cache_root=cache_root,\n state_path=state_path,\n backup_root=backup_root,\n now=applied_at,\n source_digest=digest,\n health_providers=len(health_rows),\n adaptive_providers=len({row[0] for row in sample_rows}),\n adaptive_samples=len(sample_rows),\n expected_source_hashes=source_file_hashes,\n )\n except (OSError, sqlite3.Error, ValueError):\n return _report(\n action=action,\n status=\"blocked\",\n dry_run=False,\n sqlite_available=True,\n health_rows=health_rows,\n sample_rows=sample_rows,\n source_digest=digest,\n error_code=\"backup_failed\",\n )\n\n if not _source_files_match(cache_root, source_file_hashes):\n return _report(\n action=action,\n status=\"blocked\",\n dry_run=False,\n sqlite_available=True,\n health_rows=health_rows,\n sample_rows=sample_rows,\n source_digest=digest,\n backup_id=backup_id,\n error_code=\"legacy_source_changed\",\n )\n\n try:\n store = SQLiteStateStore(state_path)\n if not store.available:\n raise sqlite3.OperationalError(\"state_store_unavailable\")\n connection = sqlite3.connect(state_path, timeout=5, isolation_level=None)\n try:\n connection.execute(\"PRAGMA busy_timeout=5000\")\n connection.execute(\"PRAGMA foreign_keys=ON\")\n connection.execute(\"BEGIN IMMEDIATE\")\n connection.execute(\"DELETE FROM legacy_provider_health\")\n connection.execute(\"DELETE FROM adaptive_samples_v3\")\n connection.executemany(\n \"\"\"\n INSERT INTO legacy_provider_health (\n provider, failure_count, cooldown_until, cooldown_seconds,\n last_failure_at, source_digest, migrated_at\n ) VALUES (?, ?, ?, ?, ?, ?, ?)\n \"\"\",\n [(*row, digest, applied_at) for row in health_rows],\n )\n connection.executemany(\n \"\"\"\n INSERT INTO adaptive_samples_v3 (\n provider, source_index, sample_time, latency_ms,\n result_count, error, source_digest, migrated_at\n ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)\n \"\"\",\n [(*row, digest, applied_at) for row in sample_rows],\n )\n connection.execute(\n \"\"\"\n INSERT INTO legacy_state_migrations (\n migration_id, source_digest, applied_at, health_providers,\n adaptive_providers, adaptive_samples\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT(migration_id) DO UPDATE SET\n source_digest=excluded.source_digest,\n applied_at=excluded.applied_at,\n health_providers=excluded.health_providers,\n adaptive_providers=excluded.adaptive_providers,\n adaptive_samples=excluded.adaptive_samples\n \"\"\",\n (\n MIGRATION_ID,\n digest,\n applied_at,\n len(health_rows),\n len({row[0] for row in sample_rows}),\n len(sample_rows),\n ),\n )\n connection.execute(\"COMMIT\")\n except Exception:\n if connection.in_transaction:\n connection.execute(\"ROLLBACK\")\n raise\n finally:\n connection.close()\n os.chmod(state_path, 0o600)\n secret_path = state_path.with_name(f\"{state_path.name}.secret\")\n if secret_path.exists():\n os.chmod(secret_path, 0o600)\n except (OSError, sqlite3.Error):\n rollback = rollback_legacy_state(\n state_path=state_path,\n backup_root=backup_root,\n backup_id=backup_id,\n )\n return _report(\n action=action,\n status=\"degraded\" if rollback.status == \"rolled_back\" else \"blocked\",\n dry_run=False,\n sqlite_available=False,\n health_rows=health_rows,\n sample_rows=sample_rows,\n source_digest=digest,\n backup_id=backup_id,\n error_code=(\n \"sqlite_write_failed\" if rollback.status == \"rolled_back\" else \"rollback_failed\"\n ),\n )\n\n return _report(\n action=action,\n status=\"applied\",\n dry_run=False,\n sqlite_available=True,\n health_rows=health_rows,\n sample_rows=sample_rows,\n source_digest=digest,\n backup_id=backup_id,\n )\n\n\ndef _validated_manifest(backup_dir: Path, backup_id: str) -> dict[str, Any]:\n if backup_dir.is_symlink() or not backup_dir.is_dir():\n raise ValueError(\"invalid_backup\")\n manifest_path = backup_dir / \"manifest.json\"\n if manifest_path.is_symlink() or not manifest_path.is_file():\n raise ValueError(\"invalid_backup\")\n manifest = json.loads(manifest_path.read_text(encoding=\"utf-8\"))\n if (\n not isinstance(manifest, dict)\n or manifest.get(\"owner\") != BACKUP_OWNER\n or manifest.get(\"schema_version\") != BACKUP_SCHEMA_VERSION\n or manifest.get(\"backup_id\") != backup_id\n or not isinstance(manifest.get(\"files\"), dict)\n ):\n raise ValueError(\"invalid_backup\")\n boolean_fields = (\n \"database_existed\",\n \"secret_existed\",\n \"health_existed\",\n \"stats_existed\",\n )\n count_fields = (\"health_providers\", \"adaptive_providers\", \"adaptive_samples\")\n if any(not isinstance(manifest.get(field), bool) for field in boolean_fields):\n raise ValueError(\"invalid_backup\")\n if any(\n isinstance(manifest.get(field), bool)\n or not isinstance(manifest.get(field), int)\n or manifest[field] \u003c 0\n for field in count_fields\n ):\n raise ValueError(\"invalid_backup\")\n if not isinstance(manifest.get(\"created_at\"), int) or not re.fullmatch(\n r\"[a-f0-9]{64}\", str(manifest.get(\"source_digest\", \"\"))\n ):\n raise ValueError(\"invalid_backup\")\n expected_files = {\n name\n for name, flag in (\n (\"state.sqlite3\", manifest[\"database_existed\"]),\n (\"state.sqlite3.secret\", manifest[\"secret_existed\"]),\n (\"provider_health.json\", manifest[\"health_existed\"]),\n (\"provider_stats.json\", manifest[\"stats_existed\"]),\n )\n if flag\n }\n if set(manifest[\"files\"]) != expected_files:\n raise ValueError(\"invalid_backup\")\n for name, expected_digest in manifest[\"files\"].items():\n if name not in {\n \"state.sqlite3\",\n \"state.sqlite3.secret\",\n \"provider_health.json\",\n \"provider_stats.json\",\n }:\n raise ValueError(\"invalid_backup\")\n candidate = backup_dir / name\n if candidate.is_symlink() or not candidate.is_file():\n raise ValueError(\"invalid_backup\")\n if not isinstance(expected_digest, str) or _sha256_file(candidate) != expected_digest:\n raise ValueError(\"invalid_backup\")\n return manifest\n\n\ndef _atomic_restore(source: Path, destination: Path) -> None:\n destination.parent.mkdir(parents=True, exist_ok=True, mode=0o700)\n descriptor, temporary = tempfile.mkstemp(\n prefix=f\".{destination.name}.\", suffix=\".restore\", dir=str(destination.parent)\n )\n os.close(descriptor)\n try:\n shutil.copyfile(source, temporary)\n os.chmod(temporary, 0o600)\n os.replace(temporary, destination)\n except Exception:\n try:\n os.unlink(temporary)\n except OSError:\n pass\n raise\n\n\ndef rollback_legacy_state(\n *,\n state_path: str | Path,\n backup_root: str | Path,\n backup_id: str,\n) -> MigrationReport:\n \"\"\"Restore the exact pre-migration database/secret from a verified backup.\"\"\"\n state_path = Path(state_path)\n backup_root = Path(backup_root)\n secret_path = state_path.with_name(f\"{state_path.name}.secret\")\n live_paths = (\n state_path,\n secret_path,\n Path(f\"{state_path}-wal\"),\n Path(f\"{state_path}-shm\"),\n )\n if (\n not _BACKUP_ID_RE.fullmatch(backup_id)\n or backup_root.is_symlink()\n or any(path.is_symlink() for path in live_paths)\n ):\n return _report(\n action=\"rollback\",\n status=\"blocked\",\n dry_run=False,\n sqlite_available=False,\n backup_id=backup_id or None,\n error_code=\"backup_invalid\",\n )\n backup_dir = backup_root / backup_id\n try:\n manifest = _validated_manifest(backup_dir, backup_id)\n for suffix in (\"-wal\", \"-shm\"):\n sidecar = Path(f\"{state_path}{suffix}\")\n if sidecar.exists() and not sidecar.is_symlink():\n sidecar.unlink()\n if manifest.get(\"database_existed\"):\n _atomic_restore(backup_dir / \"state.sqlite3\", state_path)\n elif state_path.exists() and not state_path.is_symlink():\n state_path.unlink()\n if manifest.get(\"secret_existed\"):\n _atomic_restore(backup_dir / \"state.sqlite3.secret\", secret_path)\n elif secret_path.exists() and not secret_path.is_symlink():\n secret_path.unlink()\n except (OSError, ValueError, json.JSONDecodeError):\n return _report(\n action=\"rollback\",\n status=\"blocked\",\n dry_run=False,\n sqlite_available=False,\n backup_id=backup_id,\n error_code=\"backup_invalid\",\n )\n return MigrationReport(\n action=\"rollback\",\n status=\"rolled_back\",\n dry_run=False,\n sqlite_available=bool(manifest.get(\"database_existed\")),\n health_providers=int(manifest.get(\"health_providers\", 0)),\n adaptive_providers=int(manifest.get(\"adaptive_providers\", 0)),\n adaptive_samples=int(manifest.get(\"adaptive_samples\", 0)),\n source_digest=str(manifest.get(\"source_digest\", \"\")),\n backup_id=backup_id,\n )\n\n\ndef _default_cache_root() -> Path:\n configured = os.environ.get(\"WSP_CACHE_DIR\")\n if configured:\n return Path(configured)\n return Path(__file__).resolve().parent.parent.parent / \".cache\"\n\n\ndef build_parser() -> argparse.ArgumentParser:\n parser = argparse.ArgumentParser(\n description=\"Dry-run-first migration of WSP legacy JSON state into SQLite v3\"\n )\n parser.add_argument(\"--cache-root\", type=Path, default=_default_cache_root())\n parser.add_argument(\"--state-path\", type=Path)\n parser.add_argument(\"--backup-root\", type=Path)\n action = parser.add_mutually_exclusive_group()\n action.add_argument(\"--apply\", action=\"store_true\", help=\"Apply after backup\")\n action.add_argument(\"--rollback\", metavar=\"BACKUP_ID\", help=\"Restore a verified backup\")\n return parser\n\n\ndef main(argv: list[str] | None = None) -> int:\n args = build_parser().parse_args(argv)\n cache_root = args.cache_root\n state_path = args.state_path or cache_root / \"v3\" / \"state.sqlite3\"\n backup_root = args.backup_root or state_path.parent / \"migration-backups\"\n if args.rollback:\n report = rollback_legacy_state(\n state_path=state_path,\n backup_root=backup_root,\n backup_id=args.rollback,\n )\n else:\n report = migrate_legacy_state(\n cache_root=cache_root,\n state_path=state_path,\n backup_root=backup_root,\n dry_run=not args.apply,\n )\n print(render_migration_report(report))\n return 0 if report.status in {\"ready\", \"applied\", \"unchanged\", \"rolled_back\"} else 2\n\n\nif __name__ == \"__main__\":\n raise SystemExit(main())\n","content_type":"text/x-python; charset=utf-8","language":"python","size":27360,"content_sha256":"14669f213bf3f194eb9f301d3b3620adb54000669230c7a99beb014b91be8e78"},{"filename":"scripts/wsp_core/state_store_v3.py","content":"\"\"\"Fail-closed SQLite operational state for the Web Search Plus v3 engine.\"\"\"\n\nfrom __future__ import annotations\n\nimport hashlib\nimport hmac\nimport math\nimport os\nimport secrets\nimport sqlite3\nimport stat\nimport threading\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom typing import Optional\n\nfrom .contract_v3 import Capability, CircuitState, ErrorClass, SkipReason\n\n\nSCHEMA_VERSION = 3\nDEFAULT_OPEN_SECONDS = {\n ErrorClass.AUTH: 300,\n ErrorClass.QUOTA: 3600,\n ErrorClass.RATE_LIMIT: 60,\n ErrorClass.TRANSIENT: 60,\n ErrorClass.TIMEOUT: 60,\n}\n# Longest circuit-open window taken from a provider's Retry-After; matches the\n# legacy cooldown ceiling so a hostile or buggy header cannot park a provider.\nMAX_OPEN_SECONDS = 3600\n# A single 5xx or timeout is usually a blip. Transient and timeout buckets\n# block admission only after this many consecutive failures; a success\n# deletes the bucket and resets the count. Auth, quota and rate-limit\n# buckets still block on the first failure: they are not blips.\nCONSECUTIVE_FAILURES_TO_OPEN = {\n ErrorClass.TRANSIENT: 3,\n ErrorClass.TIMEOUT: 3,\n}\n\n# Database files whose schema this process has initialized, keyed by path and\n# inode: a deleted or replaced file (restore, another process's migration) is\n# initialized again, as every store construction did before.\n_INITIALIZED_STATE_FILES: set[tuple[str, int, int]] = set()\n_INITIALIZED_STATE_FILES_LOCK = threading.Lock()\n\n\ndef _state_file_identity(path: Path) -> Optional[tuple[str, int, int]]:\n try:\n metadata = os.stat(path)\n except OSError:\n return None\n return (os.path.realpath(path), metadata.st_dev, metadata.st_ino)\n\n\ndef credential_fingerprint(\n secret: str | None, *, local_secret: bytes\n) -> str:\n \"\"\"Return an HMAC identity that is useless without the local state secret.\"\"\"\n material = secret if secret else \"\u003canonymous>\"\n return hmac.new(\n local_secret, material.encode(\"utf-8\"), hashlib.sha256\n ).hexdigest()[:24]\n\n\n@dataclass(frozen=True)\nclass CircuitKey:\n provider: str\n capability: Capability\n endpoint: str\n credential_fingerprint: str\n\n def values(self) -> tuple[str, str, str, str]:\n return (\n self.provider,\n self.capability.value,\n self.endpoint,\n self.credential_fingerprint,\n )\n\n\n@dataclass(frozen=True)\nclass CircuitRecord:\n state: CircuitState = CircuitState.CLOSED\n failure_count: int = 0\n open_until: Optional[int] = None\n updated_at: int = 0\n\n\n@dataclass(frozen=True)\nclass AdmissionDecision:\n allowed: bool\n circuit_state: CircuitState\n skip_reason: Optional[SkipReason] = None\n store_available: bool = True\n blocking_error_class: Optional[ErrorClass] = None\n # Transient and timeout buckets that hold failures below the open\n # threshold; a success must clear them or blips add up across requests.\n failing_error_classes: tuple[ErrorClass, ...] = ()\n\n\n@dataclass(frozen=True)\nclass BudgetRecord:\n scope: str\n window_key: str\n limit_units: int\n used_units: int\n reserved_units: int\n\n\ndef initialize_state_schema(connection: sqlite3.Connection) -> None:\n \"\"\"Create or upgrade the additive v3 operational-state schema.\"\"\"\n connection.executescript(\n f\"\"\"\n CREATE TABLE IF NOT EXISTS circuit_state (\n provider TEXT NOT NULL,\n capability TEXT NOT NULL,\n endpoint TEXT NOT NULL,\n credential_fingerprint TEXT NOT NULL,\n error_class TEXT NOT NULL,\n state TEXT NOT NULL,\n failure_count INTEGER NOT NULL DEFAULT 0,\n open_until INTEGER,\n updated_at INTEGER NOT NULL,\n PRIMARY KEY (\n provider, capability, endpoint,\n credential_fingerprint, error_class\n )\n );\n CREATE TABLE IF NOT EXISTS budget_ledger (\n scope TEXT NOT NULL,\n window_key TEXT NOT NULL,\n limit_units INTEGER NOT NULL,\n used_units INTEGER NOT NULL DEFAULT 0,\n reserved_units INTEGER NOT NULL DEFAULT 0,\n PRIMARY KEY (scope, window_key)\n );\n CREATE TABLE IF NOT EXISTS legacy_provider_health (\n provider TEXT PRIMARY KEY,\n failure_count INTEGER NOT NULL CHECK (failure_count >= 0),\n cooldown_until INTEGER NOT NULL CHECK (cooldown_until >= 0),\n cooldown_seconds INTEGER NOT NULL CHECK (cooldown_seconds >= 0),\n last_failure_at INTEGER NOT NULL CHECK (last_failure_at >= 0),\n source_digest TEXT NOT NULL,\n migrated_at INTEGER NOT NULL\n );\n CREATE TABLE IF NOT EXISTS adaptive_samples_v3 (\n provider TEXT NOT NULL,\n source_index INTEGER NOT NULL CHECK (source_index >= 0),\n sample_time INTEGER NOT NULL CHECK (sample_time >= 0),\n latency_ms INTEGER NOT NULL CHECK (latency_ms >= 0),\n result_count INTEGER NOT NULL CHECK (result_count >= 0),\n error INTEGER NOT NULL CHECK (error IN (0, 1)),\n source_digest TEXT NOT NULL,\n migrated_at INTEGER NOT NULL,\n PRIMARY KEY (provider, source_index)\n );\n CREATE INDEX IF NOT EXISTS adaptive_samples_v3_time\n ON adaptive_samples_v3(provider, sample_time);\n CREATE TABLE IF NOT EXISTS legacy_state_migrations (\n migration_id TEXT PRIMARY KEY,\n source_digest TEXT NOT NULL,\n applied_at INTEGER NOT NULL,\n health_providers INTEGER NOT NULL,\n adaptive_providers INTEGER NOT NULL,\n adaptive_samples INTEGER NOT NULL\n );\n PRAGMA user_version={SCHEMA_VERSION};\n \"\"\"\n )\n\n\nclass SQLiteStateStore:\n \"\"\"Durable policy state; persisted blocks fail closed, I/O loss degrades safely.\"\"\"\n\n def __init__(self, path: str | Path):\n self.path = Path(path)\n self.secret_path = Path(f\"{self.path}.secret\")\n self._read_only = False\n self._local_secret = secrets.token_bytes(32)\n self._secret_available = False\n self._available = False\n self._request_budgets: dict[tuple[str, str], list[int]] = {}\n self._request_budgets_lock = threading.Lock()\n self._initialize_local_secret()\n self._initialize()\n if not self._secret_available:\n self._available = False\n\n @classmethod\n def open_readonly(cls, path: str | Path) -> \"SQLiteStateStore\":\n \"\"\"Open an existing owned database for aggregate reads without writes.\"\"\"\n store = cls.__new__(cls)\n store.path = Path(path)\n store.secret_path = Path(f\"{store.path}.secret\")\n store._read_only = True\n store._local_secret = b\"\"\n store._secret_available = False\n store._available = False\n store._request_budgets = {}\n store._request_budgets_lock = threading.Lock()\n try:\n absolute = Path(os.path.abspath(os.fspath(store.path)))\n current = Path(os.path.sep)\n for component in absolute.parts[1:]:\n current /= component\n if stat.S_ISLNK(os.lstat(current).st_mode):\n return store\n metadata = os.lstat(absolute)\n if not stat.S_ISREG(metadata.st_mode) or metadata.st_uid != os.geteuid():\n return store\n connection = store._connect()\n connection.close()\n store._available = True\n except (OSError, sqlite3.Error):\n store._available = False\n return store\n\n def _initialize_local_secret(self) -> None:\n \"\"\"Load or atomically create the DB-local HMAC key with mode 0600.\"\"\"\n try:\n self.secret_path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)\n try:\n fd = os.open(\n self.secret_path,\n os.O_WRONLY | os.O_CREAT | os.O_EXCL,\n 0o600,\n )\n except FileExistsError:\n fd = None\n if fd is not None:\n with os.fdopen(fd, \"wb\") as handle:\n handle.write(self._local_secret)\n persisted = self.secret_path.read_bytes()\n if len(persisted) \u003c 32:\n raise OSError(\"state secret is truncated\")\n os.chmod(self.secret_path, 0o600)\n self._local_secret = persisted\n self._secret_available = True\n except OSError:\n # Ephemeral HMAC identity is safe; durable circuit state is unavailable.\n self._available = False\n\n def fingerprint_credential(self, credential: str | None) -> str:\n return credential_fingerprint(\n credential, local_secret=self._local_secret\n )\n\n @property\n def available(self) -> bool:\n return self._available\n\n def _connect(self) -> sqlite3.Connection:\n if self._read_only:\n connection = sqlite3.connect(\n f\"{self.path.resolve().as_uri()}?mode=ro\",\n uri=True,\n timeout=5,\n isolation_level=None,\n )\n else:\n connection = sqlite3.connect(self.path, timeout=5, isolation_level=None)\n # This store intentionally uses short-lived WAL connections from\n # concurrent research workers. SQLite normally checkpoints WAL when\n # the apparent last connection closes, which requires an exclusive\n # lock and can deadlock with another thread opening a connection.\n # Normal WAL auto-checkpointing still bounds the journal without doing\n # blocking persistence work during request teardown.\n no_checkpoint_on_close = getattr(\n sqlite3, \"SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE\", None\n )\n setconfig = getattr(connection, \"setconfig\", None)\n if no_checkpoint_on_close is not None and setconfig is not None:\n try:\n setconfig(no_checkpoint_on_close, True)\n except sqlite3.Error:\n # Older SQLite builds may expose the Python API without this\n # option; state access must retain its existing degradation.\n pass\n connection.row_factory = sqlite3.Row\n connection.execute(\"PRAGMA busy_timeout=5000\")\n return connection\n\n def _initialize(self) -> None:\n try:\n self.path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)\n with _INITIALIZED_STATE_FILES_LOCK:\n if _state_file_identity(self.path) not in _INITIALIZED_STATE_FILES:\n connection = self._connect()\n try:\n connection.execute(\"PRAGMA journal_mode=WAL\")\n connection.execute(\"PRAGMA foreign_keys=ON\")\n initialize_state_schema(connection)\n finally:\n connection.close()\n identity = _state_file_identity(self.path)\n if identity is not None:\n _INITIALIZED_STATE_FILES.add(identity)\n self._available = True\n except (OSError, sqlite3.Error):\n self._available = False\n\n @staticmethod\n def _state_for(error_class: ErrorClass) -> CircuitState:\n if error_class is ErrorClass.AUTH:\n return CircuitState.BLOCKED_AUTH\n if error_class is ErrorClass.QUOTA:\n return CircuitState.BLOCKED_QUOTA\n return CircuitState.OPEN\n\n def record_failure(\n self,\n key: CircuitKey,\n error_class: ErrorClass,\n *,\n now: int,\n retry_after_seconds: float | None = None,\n ) -> CircuitRecord:\n if not self._available:\n return CircuitRecord(CircuitState.UNKNOWN)\n state = self._state_for(error_class)\n if retry_after_seconds is not None and math.isfinite(retry_after_seconds):\n seconds = int(min(retry_after_seconds, MAX_OPEN_SECONDS))\n else:\n seconds = DEFAULT_OPEN_SECONDS.get(error_class, 60)\n open_until = now + min(max(1, seconds), MAX_OPEN_SECONDS)\n try:\n connection = self._connect()\n try:\n connection.execute(\"BEGIN IMMEDIATE\")\n connection.execute(\n \"\"\"\n INSERT INTO circuit_state (\n provider, capability, endpoint, credential_fingerprint,\n error_class, state, failure_count, open_until, updated_at\n ) VALUES (?, ?, ?, ?, ?, ?, 1, ?, ?)\n ON CONFLICT (\n provider, capability, endpoint,\n credential_fingerprint, error_class\n ) DO UPDATE SET\n state=excluded.state,\n failure_count=circuit_state.failure_count + 1,\n open_until=excluded.open_until,\n updated_at=excluded.updated_at\n \"\"\",\n (*key.values(), error_class.value, state.value, open_until, now),\n )\n connection.commit()\n finally:\n connection.close()\n except sqlite3.Error:\n self._available = False\n return CircuitRecord(CircuitState.UNKNOWN)\n return self.get_circuit(key, error_class)\n\n def record_success(\n self, key: CircuitKey, error_class: ErrorClass, *, now: int\n ) -> None:\n del now\n if not self._available:\n return\n try:\n connection = self._connect()\n try:\n connection.execute(\n \"\"\"\n DELETE FROM circuit_state\n WHERE provider=? AND capability=? AND endpoint=?\n AND credential_fingerprint=? AND error_class=?\n \"\"\",\n (*key.values(), error_class.value),\n )\n finally:\n connection.close()\n except sqlite3.Error:\n self._available = False\n\n def get_circuit(\n self, key: CircuitKey, error_class: ErrorClass\n ) -> CircuitRecord:\n if not self._available:\n return CircuitRecord(CircuitState.UNKNOWN)\n try:\n connection = self._connect()\n try:\n row = connection.execute(\n \"\"\"\n SELECT state, failure_count, open_until, updated_at\n FROM circuit_state\n WHERE provider=? AND capability=? AND endpoint=?\n AND credential_fingerprint=? AND error_class=?\n \"\"\",\n (*key.values(), error_class.value),\n ).fetchone()\n finally:\n connection.close()\n except sqlite3.Error:\n self._available = False\n return CircuitRecord(CircuitState.UNKNOWN)\n return self._circuit_record(row)\n\n @staticmethod\n def _circuit_record(row: Optional[sqlite3.Row]) -> CircuitRecord:\n if row is None:\n return CircuitRecord()\n return CircuitRecord(\n state=CircuitState(row[\"state\"]),\n failure_count=int(row[\"failure_count\"]),\n open_until=(int(row[\"open_until\"]) if row[\"open_until\"] is not None else None),\n updated_at=int(row[\"updated_at\"]),\n )\n\n def _claim_half_open(\n self, key: CircuitKey, error_class: ErrorClass, *, now: int\n ) -> bool:\n \"\"\"Atomically lease one probe for an expired circuit bucket.\"\"\"\n if not self._available:\n return False\n try:\n connection = self._connect()\n try:\n connection.execute(\"BEGIN IMMEDIATE\")\n cursor = connection.execute(\n \"\"\"\n UPDATE circuit_state\n SET state=?, open_until=?, updated_at=?\n WHERE provider=? AND capability=? AND endpoint=?\n AND credential_fingerprint=? AND error_class=?\n AND state IN (?, ?, ?, ?)\n AND open_until IS NOT NULL AND open_until \u003c= ?\n \"\"\",\n (\n CircuitState.HALF_OPEN.value,\n now + 60,\n now,\n *key.values(),\n error_class.value,\n CircuitState.BLOCKED_AUTH.value,\n CircuitState.BLOCKED_QUOTA.value,\n CircuitState.OPEN.value,\n CircuitState.HALF_OPEN.value,\n now,\n ),\n )\n connection.commit()\n return cursor.rowcount == 1\n finally:\n connection.close()\n except sqlite3.Error:\n self._available = False\n return False\n\n def admit(self, key: CircuitKey, *, now: int) -> AdmissionDecision:\n if not self._available:\n return AdmissionDecision(\n True,\n CircuitState.UNKNOWN,\n None,\n store_available=False,\n )\n checks = (\n (ErrorClass.AUTH, SkipReason.AUTH_BLOCKED),\n (ErrorClass.QUOTA, SkipReason.QUOTA_BLOCKED),\n (ErrorClass.RATE_LIMIT, SkipReason.RATE_LIMITED),\n (ErrorClass.TRANSIENT, SkipReason.CIRCUIT_OPEN),\n (ErrorClass.TIMEOUT, SkipReason.CIRCUIT_OPEN),\n )\n records = self._get_circuits(key, tuple(error for error, _ in checks))\n if not self._available:\n return AdmissionDecision(\n True, CircuitState.UNKNOWN, None, store_available=False\n )\n failing = tuple(\n error\n for error in CONSECUTIVE_FAILURES_TO_OPEN\n if records[error].failure_count > 0\n )\n expired = []\n for error_class, skip_reason in checks:\n record = records[error_class]\n if record.state is CircuitState.CLOSED:\n continue\n threshold = CONSECUTIVE_FAILURES_TO_OPEN.get(error_class, 1)\n if (\n record.state is CircuitState.OPEN\n and record.failure_count \u003c threshold\n ):\n continue\n active = record.open_until is None or record.open_until > now\n if active:\n return AdmissionDecision(\n False,\n record.state,\n skip_reason,\n blocking_error_class=error_class,\n )\n expired.append((error_class, skip_reason))\n\n for error_class, skip_reason in expired:\n if self._claim_half_open(key, error_class, now=now):\n return AdmissionDecision(\n True,\n CircuitState.HALF_OPEN,\n blocking_error_class=error_class,\n failing_error_classes=failing,\n )\n if not self._available:\n return AdmissionDecision(\n True,\n CircuitState.UNKNOWN,\n None,\n store_available=False,\n )\n current = self.get_circuit(key, error_class)\n return AdmissionDecision(\n False,\n current.state,\n skip_reason,\n blocking_error_class=error_class,\n )\n return AdmissionDecision(\n True, CircuitState.CLOSED, failing_error_classes=failing\n )\n\n def _get_circuits(\n self, key: CircuitKey, error_classes: tuple[ErrorClass, ...]\n ) -> dict[ErrorClass, CircuitRecord]:\n \"\"\"Fetch the circuit buckets for one admission with one SQLite query.\"\"\"\n if not self._available:\n return {error: CircuitRecord(CircuitState.UNKNOWN) for error in error_classes}\n try:\n connection = self._connect()\n try:\n rows = connection.execute(\n \"\"\"\n SELECT error_class, state, failure_count, open_until, updated_at\n FROM circuit_state\n WHERE provider=? AND capability=? AND endpoint=?\n AND credential_fingerprint=?\n \"\"\",\n key.values(),\n ).fetchall()\n finally:\n connection.close()\n except sqlite3.Error:\n self._available = False\n return {error: CircuitRecord(CircuitState.UNKNOWN) for error in error_classes}\n by_error = {row[\"error_class\"]: row for row in rows}\n return {error: self._circuit_record(by_error.get(error.value)) for error in error_classes}\n\n def configure_budget(\n self, scope: str, window_key: str, *, limit_units: int\n ) -> None:\n if limit_units \u003c 0:\n raise ValueError(\"budget limit_units must be non-negative\")\n if window_key == \"request\":\n with self._request_budgets_lock:\n budget = self._request_budgets.get((scope, window_key))\n if budget is None:\n self._request_budgets[(scope, window_key)] = [limit_units, 0, 0]\n else:\n budget[0] = limit_units\n return\n if not self._available:\n return\n try:\n connection = self._connect()\n try:\n connection.execute(\n \"\"\"\n INSERT INTO budget_ledger (\n scope, window_key, limit_units, used_units, reserved_units\n ) VALUES (?, ?, ?, 0, 0)\n ON CONFLICT(scope, window_key)\n DO UPDATE SET limit_units=excluded.limit_units\n \"\"\",\n (scope, window_key, limit_units),\n )\n finally:\n connection.close()\n except sqlite3.Error:\n self._available = False\n\n def reserve_budget(self, scope: str, window_key: str, *, units: int) -> bool:\n if units \u003c 0:\n raise ValueError(\"budget units must be non-negative\")\n if window_key == \"request\":\n with self._request_budgets_lock:\n budget = self._request_budgets.get((scope, window_key))\n if budget is None or sum(budget[1:]) + units > budget[0]:\n return False\n budget[2] += units\n return True\n if not self._available:\n return False\n try:\n connection = self._connect()\n try:\n connection.execute(\"BEGIN IMMEDIATE\")\n cursor = connection.execute(\n \"\"\"\n UPDATE budget_ledger\n SET reserved_units=reserved_units + ?\n WHERE scope=? AND window_key=?\n AND used_units + reserved_units + ? \u003c= limit_units\n \"\"\",\n (units, scope, window_key, units),\n )\n allowed = cursor.rowcount == 1\n connection.commit()\n return allowed\n finally:\n connection.close()\n except sqlite3.Error:\n self._available = False\n return False\n\n def release_budget(self, scope: str, window_key: str, *, units: int) -> bool:\n return self.reconcile_budget(\n scope, window_key, reserved_units=units, actual_units=0\n )\n\n def commit_budget(self, scope: str, window_key: str, *, units: int) -> bool:\n return self.reconcile_budget(\n scope, window_key, reserved_units=units, actual_units=units\n )\n\n def reconcile_budget(\n self,\n scope: str,\n window_key: str,\n *,\n reserved_units: int,\n actual_units: int,\n ) -> bool:\n \"\"\"Atomically commit actual cost and release the unused reservation.\"\"\"\n if reserved_units \u003c 0 or actual_units \u003c 0:\n raise ValueError(\"budget units must be non-negative\")\n if actual_units > reserved_units:\n raise ValueError(\"actual budget units cannot exceed reserved units\")\n if window_key == \"request\":\n with self._request_budgets_lock:\n budget = self._request_budgets.get((scope, window_key))\n if budget is None or budget[2] \u003c reserved_units:\n raise RuntimeError(\"budget reservation missing during reconciliation\")\n budget[1] += actual_units\n budget[2] -= reserved_units\n return True\n if not self._available:\n return False\n try:\n connection = self._connect()\n try:\n connection.execute(\"BEGIN IMMEDIATE\")\n cursor = connection.execute(\n \"\"\"\n UPDATE budget_ledger\n SET used_units=used_units + ?,\n reserved_units=reserved_units - ?\n WHERE scope=? AND window_key=?\n AND reserved_units >= ?\n \"\"\",\n (\n actual_units,\n reserved_units,\n scope,\n window_key,\n reserved_units,\n ),\n )\n if cursor.rowcount != 1:\n connection.rollback()\n raise RuntimeError(\"budget reservation missing during reconciliation\")\n connection.commit()\n return True\n finally:\n connection.close()\n except sqlite3.Error:\n self._available = False\n return False\n\n def get_budget(self, scope: str, window_key: str) -> BudgetRecord:\n if window_key == \"request\":\n with self._request_budgets_lock:\n budget = self._request_budgets.get((scope, window_key))\n if budget is None:\n raise KeyError((scope, window_key))\n return BudgetRecord(scope, window_key, *budget)\n if not self._available:\n raise RuntimeError(\"state store unavailable\")\n try:\n connection = self._connect()\n try:\n row = connection.execute(\n \"\"\"\n SELECT limit_units, used_units, reserved_units\n FROM budget_ledger WHERE scope=? AND window_key=?\n \"\"\",\n (scope, window_key),\n ).fetchone()\n finally:\n connection.close()\n except sqlite3.Error as exc:\n self._available = False\n raise RuntimeError(\"state store unavailable\") from exc\n if row is None:\n raise KeyError((scope, window_key))\n return BudgetRecord(\n scope,\n window_key,\n int(row[\"limit_units\"]),\n int(row[\"used_units\"]),\n int(row[\"reserved_units\"]),\n )\n\n def read_budget_snapshot(\n self, scope: str, window_key: str\n ) -> BudgetRecord | None:\n \"\"\"Read an optional ledger row without creating or updating state.\n\n Callers that require a zero-write read must obtain the store through\n :meth:`open_readonly`; a missing row is distinct from an unavailable\n database so policy can fail closed only when necessary.\n \"\"\"\n if window_key == \"request\":\n with self._request_budgets_lock:\n budget = self._request_budgets.get((scope, window_key))\n return BudgetRecord(scope, window_key, *budget) if budget else None\n if not self._available:\n return None\n try:\n connection = self._connect()\n try:\n row = connection.execute(\n \"\"\"\n SELECT limit_units, used_units, reserved_units\n FROM budget_ledger WHERE scope=? AND window_key=?\n \"\"\",\n (scope, window_key),\n ).fetchone()\n finally:\n connection.close()\n except sqlite3.Error:\n self._available = False\n return None\n if row is None:\n return None\n return BudgetRecord(\n scope,\n window_key,\n int(row[\"limit_units\"]),\n int(row[\"used_units\"]),\n int(row[\"reserved_units\"]),\n )\n\n def adaptive_sample_rows(self) -> list[tuple[str, int, int, int, int]]:\n \"\"\"Return (provider, sample_time, latency_ms, result_count, error) rows.\"\"\"\n if not self._available:\n return []\n try:\n connection = self._connect()\n try:\n rows = connection.execute(\n \"\"\"\n SELECT provider, sample_time, latency_ms, result_count, error\n FROM adaptive_samples_v3\n ORDER BY provider, sample_time\n \"\"\"\n ).fetchall()\n finally:\n connection.close()\n return [\n (\n str(row[\"provider\"]),\n int(row[\"sample_time\"]),\n int(row[\"latency_ms\"]),\n int(row[\"result_count\"]),\n int(row[\"error\"]),\n )\n for row in rows\n ]\n except sqlite3.Error:\n self._available = False\n return []\n","content_type":"text/x-python; charset=utf-8","language":"python","size":29782,"content_sha256":"76978beae82183c38b7e869104d3cba3d070861d590e82c994b6fd01840247bb"},{"filename":"scripts/wsp_core/urls.py","content":"\"\"\"The one URL normalisation used for every comparison WSP makes.\n\nResult dedup, diversity scoring, research fusion and the v3 observation\n``url.canonical`` value all compare URLs. Before 5.0 each had its own rules\n(scheme, ``www.``, trailing slash, tracking parameters, ports were handled\ndifferently in six places). They now share these functions:\n\n- ``canonical_url`` - a normalised URL with scheme, for display and the v3\n ``canonical`` value;\n- ``url_key`` - the same without scheme, the identity used for dedup and\n fusion (``http`` and ``https`` copies of a page are one page);\n- ``strip_tracking_params`` - remove tracking parameters but keep the URL as\n the user should see and open it;\n- ``host_and_path`` - for path-prefix rules such as ``github.com/org``;\n- ``domain_filters`` - the ``include_domains`` / ``exclude_domains`` of a\n search as bare hostnames that are safe to put after ``site:``.\n\nNo network access: redirects and content-level canonical links are out of\nscope.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport ipaddress\nimport re\nfrom typing import Any, List, Optional, Tuple\nfrom urllib.parse import SplitResult, parse_qsl, urlencode, urlsplit, urlunsplit\n\nTRACKING_PARAMETER_NAMES = frozenset(\n {\n \"dclid\",\n \"fbclid\",\n \"gclid\",\n \"igshid\",\n \"mc_cid\",\n \"mc_eid\",\n \"mkt_tok\",\n \"msclkid\",\n \"oly_anon_id\",\n \"oly_enc_id\",\n \"ref\",\n \"srsltid\",\n \"vero_id\",\n \"yclid\",\n \"_ga\",\n }\n)\n\n# Host prefixes that serve the same page as the bare host. Only stripped when\n# at least two labels remain, so \"m.com\" stays \"m.com\".\n_ALIAS_HOST_PREFIXES = (\"www.\", \"m.\", \"amp.\")\n# AMP renderings of a page: a trailing \"/amp\" path segment, a leading \"/amp/\"\n# segment, or an amp query marker.\n_AMP_PATH_SUFFIX = re.compile(r\"/amp/?$\", re.IGNORECASE)\n_AMP_PATH_PREFIX = re.compile(r\"^/amp(?=/)\", re.IGNORECASE)\n\n\ndef url_parts(value: object) -> Optional[SplitResult]:\n \"\"\"Parse a URL or host-like value without letting parser errors escape.\"\"\"\n if not isinstance(value, str) or not value.strip():\n return None\n candidate = value.strip()\n if any(character.isspace() for character in candidate):\n return None\n try:\n parsed = urlsplit(candidate)\n except (TypeError, ValueError):\n return None\n if not parsed.netloc and not parsed.scheme:\n try:\n parsed = urlsplit(\"//\" + candidate)\n except (TypeError, ValueError):\n return None\n return parsed\n\n\ndef normalized_host(parsed: Any) -> str:\n \"\"\"Lower-cased, IDNA-encoded host (IP addresses unchanged).\"\"\"\n try:\n host = parsed.hostname or \"\"\n except ValueError:\n return \"\"\n host = host.rstrip(\".\").casefold()\n if not host:\n return \"\"\n try:\n ipaddress.ip_address(host)\n return host\n except ValueError:\n pass\n try:\n return host.encode(\"idna\").decode(\"ascii\").lower()\n except UnicodeError:\n return host\n\n\ndef is_tracking_parameter(name: str) -> bool:\n normalized = name.casefold()\n return normalized.startswith(\"utm_\") or normalized in TRACKING_PARAMETER_NAMES\n\n\ndef _is_amp_marker(name: str, value: str) -> bool:\n lowered = name.casefold()\n return lowered == \"amp\" or (lowered == \"outputtype\" and value.casefold() == \"amp\")\n\n\ndef _identity_host(host: str) -> str:\n try:\n ipaddress.ip_address(host)\n return host\n except ValueError:\n pass\n stripped = True\n while stripped: # \"www.m.example.com\" -> \"example.com\"\n stripped = False\n for prefix in _ALIAS_HOST_PREFIXES:\n if host.startswith(prefix) and host.count(\".\") >= 2:\n host, stripped = host[len(prefix):], True\n return host\n\n\ndef canonical_url(url: object) -> str:\n \"\"\"Normalised URL for comparison: one spelling per page, scheme kept.\n\n Lower-cases scheme and host, IDNA-encodes the host, drops ``www.``, ``m.``\n and ``amp.`` host aliases, default ports, the fragment, a trailing slash,\n AMP path segments and query markers, and tracking parameters (``utm_*``,\n ``gclid``, ``srsltid``, ...). Remaining query pairs, which often identify\n the page (``watch?v=``), are kept and sorted. Returns \"\" for invalid input.\n \"\"\"\n parsed = url_parts(url)\n if parsed is None:\n return \"\"\n host = normalized_host(parsed)\n if not host:\n return \"\"\n host = _identity_host(host)\n try:\n port = parsed.port\n except ValueError:\n return \"\"\n scheme = parsed.scheme.casefold()\n display = f\"[{host}]\" if \":\" in host else host\n if port is not None and not ((scheme == \"http\" and port == 80) or (scheme == \"https\" and port == 443)):\n netloc = f\"{display}:{port}\"\n else:\n netloc = display\n path = _AMP_PATH_PREFIX.sub(\"\", _AMP_PATH_SUFFIX.sub(\"\", parsed.path)).rstrip(\"/\")\n try:\n pairs = parse_qsl(parsed.query, keep_blank_values=True)\n except ValueError:\n return \"\"\n kept = sorted(\n (name, value) for name, value in pairs\n if not is_tracking_parameter(name) and not _is_amp_marker(name, value)\n )\n return urlunsplit((scheme, netloc, path, urlencode(kept, doseq=True), \"\"))\n\n\ndef url_key(url: object) -> str:\n \"\"\"Identity of a page across scheme and presentation variants (\"\" if invalid).\"\"\"\n canonical = canonical_url(url)\n return canonical.split(\"://\", 1)[1] if \"://\" in canonical else canonical.lstrip(\"/\")\n\n\ndef strip_tracking_params(url: str) -> str:\n \"\"\"Remove tracking parameters only; the URL otherwise stays as given.\"\"\"\n if not url:\n return \"\"\n try:\n parsed = urlsplit(url)\n pairs = parse_qsl(parsed.query, keep_blank_values=True)\n except ValueError:\n return url\n kept = [(name, value) for name, value in pairs if not is_tracking_parameter(name)]\n return urlunsplit(parsed._replace(query=urlencode(kept, doseq=True)))\n\n\ndef host_and_path(url: str) -> str:\n \"\"\"Identity host plus path without trailing slash, for path-prefix rules.\"\"\"\n parsed = url_parts(url)\n if parsed is None:\n return \"\"\n host = normalized_host(parsed)\n if not host:\n return \"\"\n return f\"{_identity_host(host)}{parsed.path.rstrip('/')}\"\n\n\n# Domain filters. Brave, Serper, SerpBase, You.com and Firecrawl take them as\n# ``site:`` / ``-site:`` operators inside the query text, so an entry must be a\n# bare hostname and nothing else: anything that could carry a second operator\n# (\"site:evil.com\", \"--help\") is rejected here, not escaped later.\nSITE_OPERATOR_LIMIT = 10 # operators per list\n_FILTER_HOST = re.compile(\n r\"(?=.{1,253}\\Z)(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\\.)+(?:[a-z]{2,63}|xn--[a-z0-9-]{1,59})\"\n)\n# A suffix filter such as ``.gov`` or ``*.ac.uk``: the labels after the dot.\n_FILTER_SUFFIX = re.compile(\n r\"(?=.{1,253}\\Z)(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\\.)*(?:[a-z]{2,63}|xn--[a-z0-9-]{1,59})\"\n)\n_FILTER_SEPARATORS = re.compile(r\"[\\s,]+\")\n\n\ndef domain_filter_tokens(value: Any) -> List[str]:\n \"\"\"The entries of a domain filter: a string or a list, split on commas and spaces.\n\n An LLM passes ``\"docs.rs\"``, ``\"docs.rs, tokio.rs\"`` or ``[\"docs.rs\"]`` for\n the same thing. Blank entries are dropped; nothing else is judged here.\n \"\"\"\n if value is None:\n return []\n items = value if isinstance(value, (list, tuple)) else [value]\n tokens: List[str] = []\n for item in items:\n if item is not None:\n tokens.extend(token for token in _FILTER_SEPARATORS.split(str(item)) if token)\n return tokens\n\n\ndef domain_filter_host(entry: str) -> Optional[str]:\n \"\"\"One domain-filter entry as a bare ASCII hostname, or None if it is not a domain.\n\n A URL is reduced to its host (scheme, userinfo, port and path dropped),\n ``www.`` and a trailing dot are removed and an internationalised name is\n converted to punycode. Only ``label.label.tld`` made of ``[a-z0-9-]`` is\n accepted, so the result is always safe to put after ``site:``. An entry\n written as a suffix (``.gov``, ``*.ac.uk``) stays a suffix: ``gov``,\n ``ac.uk``, which ``site:`` also takes.\n \"\"\"\n candidate = entry.strip()\n if candidate.startswith((\".\", \"*.\")):\n return _filter_suffix(candidate.removeprefix(\"*\").removeprefix(\".\"))\n try:\n host = urlsplit(candidate if \"://\" in candidate else \"//\" + candidate).hostname\n except ValueError:\n return None\n if not host:\n return None\n host = host.removesuffix(\".\")\n if host.startswith(\"www.\") and host.count(\".\") >= 2:\n host = host[4:]\n try:\n host = host.encode(\"idna\").decode(\"ascii\")\n except UnicodeError:\n return None\n return host if _FILTER_HOST.fullmatch(host) else None\n\n\ndef _filter_suffix(suffix: str) -> Optional[str]:\n try:\n suffix = suffix.lower().encode(\"idna\").decode(\"ascii\")\n except UnicodeError:\n return None\n return suffix if _FILTER_SUFFIX.fullmatch(suffix) else None\n\n\ndef wildcard_domain_entries(value: Any) -> List[str]:\n \"\"\"Domain-filter entries for an API field that takes ``*.example.com`` (Tavily).\n\n Hosts come out as from :func:`domain_filter_host`; a wildcard entry\n (``.example.com``, ``*.example.com``) becomes ``*.example.com``, the form such\n a field documents. Whether the field takes a bare public suffix such as\n ``*.gov`` is the caller's concern. Unusable entries are skipped.\n \"\"\"\n entries: List[str] = []\n for token in domain_filter_tokens(value):\n host = domain_filter_host(token)\n if host is None:\n continue\n entry = f\"*.{host}\" if token.strip().startswith((\".\", \"*.\")) else host\n if entry not in entries:\n entries.append(entry)\n return entries\n\n\ndef _hosts(tokens: List[str]) -> List[str]:\n hosts: List[str] = []\n for token in tokens:\n host = domain_filter_host(token)\n if host is not None and host not in hosts:\n hosts.append(host)\n return hosts\n\n\ndef domain_filters(include: Any, exclude: Any) -> Tuple[List[str], List[str]]:\n \"\"\"Validated ``(include, exclude)`` hostnames for one search.\n\n Each argument may be a string (comma or space separated) or a list. An\n unusable exclude entry is skipped. A host named in both lists is excluded.\n Raises ``ValueError`` (the message is meant for the caller) when include\n entries were given but none is left: searching unrestricted would answer a\n different question than the one asked.\n \"\"\"\n include_tokens = domain_filter_tokens(include)\n include_hosts = _hosts(include_tokens)\n exclude_hosts = _hosts(domain_filter_tokens(exclude))\n if include_tokens and not include_hosts:\n shown = \", \".join(repr(token[:60]) for token in include_tokens[:3])\n raise ValueError(\n \"Invalid include_domains value: {}{}. Entries must be hostnames such as docs.rs \"\n \"(a list, or one string separated by commas or spaces).\".format(\n shown, \", ...\" if len(include_tokens) > 3 else \"\"\n )\n )\n allowed = [host for host in include_hosts if host not in exclude_hosts]\n if include_hosts and not allowed:\n raise ValueError(\n \"Invalid include_domains value: every domain is also in exclude_domains, \"\n \"which takes precedence, so no domain is left to search.\"\n )\n return allowed, exclude_hosts\n","content_type":"text/x-python; charset=utf-8","language":"python","size":11445,"content_sha256":"7b56d5b78e80f90f1d5399a1d2e88cf4583da3d7b3c62f861d87d1fbcfe478bb"},{"filename":"scripts/wsp_skill.py","content":"\"\"\"OpenClaw skill adapter over the vendored Web Search Plus engine.\n\nThe engine in ``scripts/wsp_core/`` is a copy of the ``wsp_core`` package of\nhermes-web-search-plus v5.0.1 (commit 82cceb3) plus the follow-up Brave query\nlimit fix (600 characters / 75 words, commit 3115de6). Not copied: the Operator\nConsole server (``ui.py``, ``web/``) and the TypeSafe setup helper\n(``jev_setup.py``); ``providers.d`` is not shipped. No engine file is edited.\nThis module keeps the skill's own interface on top of the engine:\n\n- environment variable aliases from the 4.x skill (``WSP_KEENABLE_ALLOW_PUBLIC``,\n ``WSP_LOCALE_COUNTRY``/``WSP_LOCALE_LANGUAGE``, ``WSP_DISABLE_CACHE``,\n ``WSP_ALLOW_PRIVATE_URLS``, ``WSP_EXTRACT_CHAR_LIMIT``);\n- 4.x ``config.json`` layouts (top-level ``locale``, provider country/language\n keys, the old default provider order, the old SerpBase endpoint);\n- the skill's provider set: Parallel ships in the engine but is not part of this\n skill, so it is removed from every automatic chain and refused when named;\n- the skill's file and network footprint: ``.env`` files are read from the skill\n directory only (not from a Hermes profile), an invalid ``config.json`` is\n reported instead of renamed, and the optional TypeSafe (JEV) helper stays off;\n- the CLI flags of ``scripts/search.py`` and ``scripts/extract.py``.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport argparse\nimport json\nimport os\nimport re\nimport sys\nfrom pathlib import Path\nfrom typing import Any, Dict, List, Optional\n\nSCRIPTS_DIR = Path(__file__).resolve().parent\nSKILL_DIR = SCRIPTS_DIR.parent\nif str(SCRIPTS_DIR) not in sys.path:\n sys.path.insert(0, str(SCRIPTS_DIR))\n\n# --- Provider plugins are not part of the skill -----------------------------\n# The engine imports every module in scripts/providers.d when its provider\n# registry loads. The skill ships no such directory and runs no provider code\n# beyond the vendored engine, so refuse to start while one exists instead of\n# executing it.\nPROVIDERS_D = SCRIPTS_DIR / \"providers.d\"\nif PROVIDERS_D.exists():\n sys.stderr.write(\n f\"Web Search Plus skill: {PROVIDERS_D} is not supported in the skill and would \"\n \"load extra provider code. Remove it, or use the Hermes plugin for \"\n \"providers.d providers.\\n\"\n )\n raise SystemExit(2)\n\n# --- Bootstrap: must run before wsp_core.config is imported -----------------\n# wsp_core.config loads .env files when it is imported: scripts/.env, the skill\n# directory's .env and the Hermes profile .env. The skill reads only its own\n# directory, so the Hermes profile path is pointed at the skill .env (deduped).\nfrom wsp_core import env_loader as _env_loader # noqa: E402\n\n_env_loader.get_hermes_env_path = lambda: SKILL_DIR / \".env\"\n\nfrom wsp_core import config as _config # noqa: E402\nfrom wsp_core.provider_registry import DEFAULT_PROVIDER_PRIORITY # noqa: E402\n\nfrom url_security import BLOCKED_METADATA_HOSTS, validate_outbound_url # noqa: E402\n\n# Providers the engine ships but this skill does not offer.\nEXCLUDED_PROVIDERS = (\"parallel\",)\nEXCLUDED_ENV_VARS = (\"PARALLEL_API_KEY\",)\n\n# 4.x skill name -> engine name. The old name is copied over only when the\n# engine name is unset, so an explicit engine setting always wins.\nENV_ALIASES = {\n \"WSP_KEENABLE_ALLOW_PUBLIC\": \"KEENABLE_ALLOW_PUBLIC\",\n}\n\nDISABLE_CACHE_ENV = \"WSP_DISABLE_CACHE\"\nALLOW_PRIVATE_ENV = \"WSP_ALLOW_PRIVATE_URLS\"\nLOCALE_COUNTRY_ENV = \"WSP_LOCALE_COUNTRY\"\nLOCALE_LANGUAGE_ENV = \"WSP_LOCALE_LANGUAGE\"\nEXTRACT_CHAR_LIMIT_ENV = \"WSP_EXTRACT_CHAR_LIMIT\"\nDEFAULT_EXTRACT_CHAR_LIMIT = 15000\n\n# Default provider order written by every 4.x skill setup (config.example.json).\nSKILL_4X_DEFAULT_PRIORITY = [\n \"tavily\", \"linkup\", \"querit\", \"exa\", \"firecrawl\", \"brave\", \"serper\", \"you\", \"searxng\", \"keenable\",\n]\n# 4.x skill SerpBase endpoint; the engine uses SerpBase's current API.\nSKILL_4X_SERPBASE_URLS = {\"https://api.serpbase.com/search\"}\n# Provider sections whose country/language keys the 4.x skill ignored (only\n# serper.country/language seeded the global locale). In the engine these keys\n# pin the provider's locale, so they are moved out of the way.\n_PROVIDER_LOCALE_KEYS = {\n \"serper\": (\"country\", \"language\"),\n \"serpbase\": (\"country\", \"language\"),\n \"brave\": (\"country\", \"search_lang\"),\n \"querit\": (\"country\", \"language\"),\n \"firecrawl\": (\"country\",),\n \"you\": (\"country\", \"language\"),\n \"searxng\": (\"language\",),\n}\n\n# 4.x synthesis options that the source-only engine no longer runs.\nLEGACY_EXA_DEPTHS = (\"deep\", \"deep-reasoning\")\nLEGACY_LINKUP_OUTPUT_TYPES = (\"sourcedAnswer\",)\n\n\ndef _env_on(name: str) -> bool:\n return str(os.environ.get(name, \"\")).strip().lower() in {\"1\", \"true\", \"yes\", \"on\"}\n\n\ndef _notice(message: str) -> None:\n print(json.dumps({\"warning\": message}, ensure_ascii=False), file=sys.stderr)\n\n\ndef apply_env_aliases() -> None:\n \"\"\"Map 4.x env names to the engine and drop credentials of excluded providers.\"\"\"\n for old, new in ENV_ALIASES.items():\n value = os.environ.get(old)\n if value is not None and not os.environ.get(new):\n os.environ[new] = value\n for name in EXCLUDED_ENV_VARS:\n os.environ.pop(name, None)\n\n\ndef _warn_invalid_config(config_path: Path, reason: str) -> None:\n \"\"\"Report an invalid config.json without renaming it (the engine moves it aside).\"\"\"\n _notice(f\"Invalid config ignored ({config_path.name}): {reason}\")\n\n\n_config._quarantine_runtime_config = _warn_invalid_config\napply_env_aliases()\n\n# --- Setup guidance: point at this skill instead of the Hermes plugin --------\n_HERMES_SETUP_RE = re.compile(r\"python3 ~/\\.hermes/plugins/web-search-plus/setup\\.py setup(?: --preset [\\w-]+| [\\w-]+)?\")\n_GUIDANCE_REPLACEMENTS = (\n (\"to your Hermes profile .env\", \"to the skill's .env file\"),\n (\"Store API keys in your Hermes profile .env, not inline in config.json.\",\n \"Store API keys in the skill's .env file or the environment, not inline in config.json.\"),\n)\nSKILL_SETUP_COMMAND = \"python3 scripts/setup.py\"\n\n\ndef skill_guidance(value: Any) -> Any:\n \"\"\"Rewrite the engine's Hermes setup hints for this skill, recursively.\"\"\"\n if isinstance(value, str):\n text = _HERMES_SETUP_RE.sub(SKILL_SETUP_COMMAND, value)\n for old, new in _GUIDANCE_REPLACEMENTS:\n text = text.replace(old, new)\n return text\n if isinstance(value, list):\n return [skill_guidance(item) for item in value]\n if isinstance(value, dict):\n return {key: skill_guidance(item) for key, item in value.items()}\n return value\n\n\ndef _install_guidance_patches() -> None:\n from wsp_core import errors_v3, extract as extract_module, search as search_module\n\n original_missing = errors_v3.missing_key_guidance\n errors_v3.missing_key_guidance = lambda provider: skill_guidance(original_missing(provider))\n\n original_add = _config.add_provider_setup_guidance\n\n def add_provider_setup_guidance(payload, *args, **kwargs):\n original_add(payload, *args, **kwargs)\n payload.update(skill_guidance(dict(payload)))\n\n for module in (_config, search_module, extract_module):\n module.add_provider_setup_guidance = add_provider_setup_guidance\n\n\n_install_guidance_patches()\n\n\ndef _section(config: Dict[str, Any], *path: str) -> Dict[str, Any]:\n node = config\n for key in path:\n child = node.get(key)\n if not isinstance(child, dict):\n child = {}\n node[key] = child\n node = child\n return node\n\n\ndef _adapt_locale(config: Dict[str, Any]) -> None:\n locale = _section(config, \"defaults\", \"locale\")\n legacy = config.get(\"locale\")\n legacy = legacy if isinstance(legacy, dict) else {}\n serper = config.get(\"serper\") if isinstance(config.get(\"serper\"), dict) else {}\n seeds = {\n \"country\": [legacy.get(\"country\"), os.environ.get(LOCALE_COUNTRY_ENV),\n serper.get(\"country\") if str(serper.get(\"country\") or \"\").lower() != \"us\" else None],\n \"language\": [legacy.get(\"language\"), os.environ.get(LOCALE_LANGUAGE_ENV),\n serper.get(\"language\") if str(serper.get(\"language\") or \"\").lower() != \"en\" else None],\n }\n for key, candidates in seeds.items():\n if locale.get(key):\n continue\n for value in candidates:\n text = str(value or \"\").strip().lower()\n if text:\n locale[key] = text\n break\n for provider, keys in _PROVIDER_LOCALE_KEYS.items():\n section = config.get(provider)\n if isinstance(section, dict):\n for key in keys:\n section.pop(key, None)\n\n\ndef _adapt_routing(config: Dict[str, Any]) -> None:\n auto = _section(config, \"auto_routing\")\n priority = [str(p) for p in auto.get(\"provider_priority\") or []]\n if auto.get(\"order\") != \"custom\" and priority[: len(SKILL_4X_DEFAULT_PRIORITY)] == SKILL_4X_DEFAULT_PRIORITY:\n priority = list(DEFAULT_PROVIDER_PRIORITY)\n auto[\"provider_priority\"] = [p for p in priority if p not in EXCLUDED_PROVIDERS]\n auto[\"extract_provider_priority\"] = [\n p for p in auto.get(\"extract_provider_priority\") or [] if p not in EXCLUDED_PROVIDERS\n ]\n disabled = [p for p in auto.get(\"disabled_providers\") or [] if p not in EXCLUDED_PROVIDERS]\n auto[\"disabled_providers\"] = disabled + list(EXCLUDED_PROVIDERS)\n allow = auto.get(\"auto_allow\") if isinstance(auto.get(\"auto_allow\"), dict) else {}\n allow.update({p: False for p in EXCLUDED_PROVIDERS})\n auto[\"auto_allow\"] = allow\n if auto.get(\"fallback_provider\") in EXCLUDED_PROVIDERS:\n auto[\"fallback_provider\"] = \"serper\"\n if config.get(\"default_provider\") in EXCLUDED_PROVIDERS:\n config[\"default_provider\"] = None\n # 4.x: with auto-routing off every query went to defaults.provider.\n if auto.get(\"enabled\", True) is False and not config.get(\"default_provider\"):\n legacy_default = (config.get(\"defaults\") or {}).get(\"provider\")\n if legacy_default and legacy_default not in EXCLUDED_PROVIDERS and legacy_default not in disabled:\n config[\"default_provider\"] = legacy_default\n for provider in EXCLUDED_PROVIDERS:\n section = config.get(provider)\n if isinstance(section, dict):\n section.pop(\"api_key\", None)\n section.pop(\"apiKey\", None)\n\n\ndef adapt_config(config: Dict[str, Any], *, allow_private_urls: bool = False) -> Dict[str, Any]:\n \"\"\"Apply the skill's compatibility rules to an engine config in place.\"\"\"\n _adapt_locale(config)\n _adapt_routing(config)\n serpbase = config.get(\"serpbase\")\n if isinstance(serpbase, dict) and serpbase.get(\"api_url\") in SKILL_4X_SERPBASE_URLS:\n serpbase[\"api_url\"] = _config.DEFAULT_CONFIG[\"serpbase\"][\"api_url\"]\n if allow_private_urls or _env_on(ALLOW_PRIVATE_ENV):\n _section(config, \"extract\")[\"allow_private_urls\"] = True\n raw_limit = str(os.environ.get(EXTRACT_CHAR_LIMIT_ENV, \"\")).strip()\n if raw_limit:\n try:\n limit = int(raw_limit)\n except ValueError:\n limit = 0\n if limit > 0:\n _section(config, \"web\")[\"extract_char_limit\"] = limit\n # The optional TypeSafe (JEV) helper calls a service outside the declared\n # host list; the skill keeps it off.\n _section(config, \"jev\")[\"enabled\"] = False\n return config\n\n\ndef load_config() -> Dict[str, Any]:\n \"\"\"Engine config from the skill's config.json, adapted for the skill.\"\"\"\n return adapt_config(_config.load_config())\n\n\ndef prepare_cache_dir() -> None:\n \"\"\"Keep everything the engine writes owner-only (dir 0700, files 0600).\n\n The engine creates some state files (``v3/state.sqlite3``) with the process\n umask; the CLI entry points tighten it instead of editing engine code.\n \"\"\"\n from wsp_core.cache import CACHE_DIR\n\n os.umask(0o077)\n try:\n CACHE_DIR.mkdir(parents=True, exist_ok=True, mode=0o700)\n except OSError:\n pass\n\n\ndef cache_disabled() -> bool:\n return str(os.environ.get(DISABLE_CACHE_ENV, \"\")).strip() == \"1\"\n\n\ndef private_urls_allowed() -> bool:\n return _env_on(ALLOW_PRIVATE_ENV)\n\n\n# --- search.py ---------------------------------------------------------------\n\n\ndef _patch_search_parser(search_module) -> None:\n original_build_parser = getattr(search_module, \"_skill_original_build_parser\", search_module.build_parser)\n search_module._skill_original_build_parser = original_build_parser\n\n def build_parser(config: Dict[str, Any]) -> argparse.ArgumentParser:\n parser = original_build_parser(config)\n for action in parser._actions:\n if action.dest == \"provider\" and action.choices:\n action.choices = [c for c in action.choices if c not in EXCLUDED_PROVIDERS]\n elif action.dest == \"bench_providers\" and action.choices:\n action.choices = [c for c in action.choices if c not in EXCLUDED_PROVIDERS]\n elif action.dest == \"exa_depth\":\n action.choices = list(action.choices or [\"normal\"]) + list(LEGACY_EXA_DEPTHS)\n elif action.dest == \"linkup_output_type\":\n action.choices = list(action.choices or [\"searchResults\"]) + list(LEGACY_LINKUP_OUTPUT_TYPES)\n parser.add_argument(\n \"--allow-private-urls\",\n action=\"store_true\",\n help=f\"Allow URLs (--similar-url, research extraction) on private/internal networks; \"\n f\"cloud metadata endpoints stay blocked (see {ALLOW_PRIVATE_ENV})\",\n )\n original_parse_args = parser.parse_args\n\n def parse_args(args=None, namespace=None):\n ns = original_parse_args(args, namespace)\n return _finish_search_args(parser, ns, config)\n\n parser.parse_args = parse_args\n return parser\n\n search_module.build_parser = build_parser\n\n\ndef _finish_search_args(parser, ns, config: Dict[str, Any]):\n for provider in [ns.provider, *(ns.research_providers or [])]:\n if provider and str(provider).strip().lower() in EXCLUDED_PROVIDERS:\n parser.error(f\"provider '{provider}' is not part of this skill\")\n if ns.exa_depth in LEGACY_EXA_DEPTHS:\n _notice(f\"--exa-depth {ns.exa_depth} is no longer supported (source-only results); using normal\")\n ns.exa_depth = \"normal\"\n if ns.linkup_output_type in LEGACY_LINKUP_OUTPUT_TYPES:\n _notice(\"--linkup-output-type sourcedAnswer is no longer supported (source-only results); using searchResults\")\n ns.linkup_output_type = \"searchResults\"\n if isinstance(ns.max_results, int):\n ns.max_results = max(1, min(20, ns.max_results))\n allow_private = bool(ns.allow_private_urls) or private_urls_allowed()\n if allow_private:\n _section(config, \"extract\")[\"allow_private_urls\"] = True\n if ns.similar_url:\n try:\n ns.similar_url = validate_outbound_url(ns.similar_url, allow_private=allow_private, label=\"--similar-url\")\n except ValueError as exc:\n print(json.dumps({\"error\": str(exc)}, ensure_ascii=False), file=sys.stderr)\n sys.exit(1)\n if cache_disabled():\n ns.no_cache = True\n return ns\n\n\n# Search options the v3 request carries itself. Every other parsed CLI option\n# is copied onto the per-attempt arguments from the CLI invocation.\n_V3_REQUEST_OPTIONS = frozenset({\n \"query\", \"provider\", \"max_results\", \"exa_depth\", \"time_range\", \"freshness\", \"search_type\",\n \"quality_report\", \"include_domains\", \"exclude_domains\", \"mode\", \"research_time_budget\",\n \"language\", \"country\", \"allow_fallback\", \"no_cache\", \"cache_ttl\",\n})\n_cli_args_template = None\n\n\ndef _run_cli_search_v3(search_module, args, config: Dict[str, Any]):\n \"\"\"Run a normal CLI search through the engine's v3 path (hedged fallback).\n\n Upstream ``python search.py`` runs providers one after another; the Hermes\n tool uses the v3 path, which starts the next provider when one fails, is\n slow or answers empty. The skill CLI uses the v3 path too and carries the\n remaining CLI options (provider-specific flags) into each attempt.\n \"\"\"\n global _cli_args_template\n from wsp_core import providers as providers_module\n from wsp_core.compat_v3 import legacy_request_to_v3, v3_response_to_legacy_search\n from wsp_core.contract_v3 import Capability\n from wsp_core.orchestrator_v3 import execute_v3_request\n from wsp_core.query_limits import MAX_QUERY_CHARS\n from wsp_core.urls import domain_filter_tokens\n\n defaults = vars(search_module.default_search_args(config))\n custom_options = any(\n getattr(args, name, None) != value for name, value in defaults.items() if name not in _V3_REQUEST_OPTIONS\n )\n runtime_config = config\n requested = str(args.provider or \"auto\").strip().lower()\n if requested == \"auto\" and providers_module.public_suffix_entries(args.include_domains, args.exclude_domains):\n runtime_config = json.loads(json.dumps(config))\n auto = runtime_config.setdefault(\"auto_routing\", {})\n auto[\"disabled_providers\"] = sorted(\n set(auto.get(\"disabled_providers\") or []) | set(providers_module.DOMAIN_SUFFIX_FILTER_UNSUPPORTED)\n )\n request = legacy_request_to_v3(\n Capability.SEARCH,\n policy_mode=str((runtime_config.get(\"routing\") or {}).get(\"policy_mode\", \"classic\")),\n payload={\n \"query\": args.query[:MAX_QUERY_CHARS],\n \"provider\": requested,\n \"count\": args.max_results,\n \"depth\": args.exa_depth,\n \"time_range\": args.time_range,\n \"freshness\": args.freshness,\n \"search_type\": args.search_type,\n \"include_domains\": domain_filter_tokens(args.include_domains) or None,\n \"exclude_domains\": domain_filter_tokens(args.exclude_domains) or None,\n \"mode\": \"normal\",\n \"quality_report\": bool(args.quality_report),\n \"language\": args.language,\n \"country\": args.country,\n \"allow_fallback\": bool(args.allow_fallback) if requested != \"auto\" else True,\n # The response cache keys on the request options above. A search\n # with other provider flags is not served from or written to it.\n \"no_cache\": bool(args.no_cache or custom_options),\n \"cache_ttl\": int(args.cache_ttl),\n },\n )\n _cli_args_template = args\n execution = execute_v3_request(request, search_module._search_adapter(), runtime_config)\n payload = v3_response_to_legacy_search(execution)\n return payload, (1 if payload.get(\"error\") else 0)\n\n\ndef _patch_search_execution(search_module) -> None:\n if getattr(search_module, \"_skill_execution_patched\", False):\n return\n original_core = search_module._execute_search_request_core\n original_args_from_v3 = search_module._search_args_from_v3\n\n def search_args_from_v3(request, config):\n args = original_args_from_v3(request, config)\n template = _cli_args_template\n if template is not None:\n for name, value in vars(template).items():\n if name not in _V3_REQUEST_OPTIONS and not name.startswith(\"_\"):\n setattr(args, name, value)\n return args\n\n def execute_search_request_core(args, config):\n if (\n getattr(args, \"_v3_engine_owned_attempt\", False)\n or getattr(args, \"mode\", \"normal\") != \"normal\"\n or not getattr(args, \"query\", None)\n ):\n return original_core(args, config)\n return _run_cli_search_v3(search_module, args, config)\n\n def extract_plus(urls, provider=\"auto\", output_format=\"markdown\", include_images=False,\n include_raw_html=False, render_js=False, spans=False, spans_query=None,\n config=None, max_wall_time_ms=None):\n # search.py --extract-urls and research-mode extraction get the same\n # guards and per-page budget as scripts/extract.py.\n return extract_urls(\n list(urls or []), provider=provider, output_format=output_format, include_images=include_images,\n include_raw_html=include_raw_html, render_js=render_js, spans=spans, spans_query=spans_query,\n config=config, max_wall_time_ms=max_wall_time_ms,\n )\n\n search_module._search_args_from_v3 = search_args_from_v3\n search_module._execute_search_request_core = execute_search_request_core\n search_module.extract_plus = extract_plus\n search_module._skill_execution_patched = True\n\n\ndef run_search() -> None:\n \"\"\"``python3 scripts/search.py`` entry point.\"\"\"\n from wsp_core import search as search_module\n\n prepare_cache_dir()\n search_module.load_config = load_config\n _patch_search_parser(search_module)\n _patch_search_execution(search_module)\n search_module.main()\n\n\n# --- extract.py --------------------------------------------------------------\n\n\ndef extract_char_limit(override: Optional[int] = None, config: Optional[Dict[str, Any]] = None) -> int:\n if override and override > 0:\n return override\n try:\n value = int(((config or {}).get(\"web\") or {}).get(\"extract_char_limit\") or 0)\n except (TypeError, ValueError):\n value = 0\n return value if value > 0 else DEFAULT_EXTRACT_CHAR_LIMIT\n\n\ndef _split_content(content: str, limit: int):\n head_chars = min(max(1, (limit * 2) // 3), max(1, limit - 1))\n tail_chars = min(max(1, limit // 5), max(1, limit - head_chars))\n if head_chars + tail_chars >= len(content):\n return content, \"\", 0\n head = content[:head_chars].rstrip()\n tail = content[-tail_chars:].lstrip()\n return head, tail, max(0, len(content) - len(head) - len(tail))\n\n\n_BASE64_MARKDOWN_IMAGE_RE = re.compile(r\"!\\[([^\\]]*)\\]\$\\s*data:image/[^)]+\$\", re.IGNORECASE)\n_BASE64_HTML_IMAGE_RE = re.compile(r\"\u003cimg\\b(?=[^>]*\\bsrc=[\\\"']data:image/)[^>]*>\", re.IGNORECASE)\n_HTML_ALT_RE = re.compile(r\"\\balt=[\\\"']([^\\\"']*)[\\\"']\", re.IGNORECASE)\n\n\ndef sanitize_extract_content(content: str) -> str:\n \"\"\"Replace inline base64 images with ``[IMAGE: alt]`` placeholders.\"\"\"\n\n def markdown_image(match):\n return f\"[IMAGE: {(match.group(1) or 'image').strip() or 'image'}]\"\n\n def html_image(match):\n alt = _HTML_ALT_RE.search(match.group(0))\n return f\"[IMAGE: {((alt.group(1) if alt else '') or 'image').strip() or 'image'}]\"\n\n return _BASE64_HTML_IMAGE_RE.sub(html_image, _BASE64_MARKDOWN_IMAGE_RE.sub(markdown_image, content))\n\n\ndef apply_char_limit(item: Dict[str, Any], limit: int) -> None:\n \"\"\"Sanitize one extracted page and cut it to a head/tail window when over ``limit``.\n\n The engine caps the whole batch to its context budget; this adds the\n skill's per-URL inline budget. The footer reports the page's real length\n and the stored full-text file when the engine kept one.\n \"\"\"\n content = item.get(\"content\")\n if not isinstance(content, str):\n return\n raw_same = item.get(\"raw_content\") == content\n content = sanitize_extract_content(content)\n item[\"content\"] = content\n if raw_same:\n item[\"raw_content\"] = content\n elif isinstance(item.get(\"raw_content\"), str):\n item[\"raw_content\"] = sanitize_extract_content(item[\"raw_content\"])\n full_text = item.get(\"full_text\") if isinstance(item.get(\"full_text\"), dict) else {}\n original = full_text.get(\"original_chars\")\n if not isinstance(original, int):\n original = item.get(\"original_content_length\") if isinstance(item.get(\"original_content_length\"), int) else len(content)\n if len(content) \u003c= limit:\n if original > len(content):\n item[\"truncated\"] = True\n item[\"original_chars\"] = original\n return\n head, tail, _omitted = _split_content(content, limit)\n shown = len(head) + len(tail)\n footer = [\n \"\",\n \"---\",\n f\"[Content truncated: original {original} chars; {max(0, original - shown)} chars omitted; showing head and tail.]\",\n ]\n if full_text.get(\"stored\") and full_text.get(\"path\"):\n footer.append(f\"Full cleaned text stored at: {full_text['path']}\")\n else:\n footer.append(f\"Raise {EXTRACT_CHAR_LIMIT_ENV} (or --extract-char-limit) for a larger inline budget, \"\n \"or extract a more specific URL for the omitted section.\")\n new_content = f\"{head}\\n\\n[... omitted middle ...]\\n\\n{tail}\\n\" + \"\\n\".join(footer)\n if item.get(\"raw_content\") == content:\n item[\"raw_content\"] = new_content\n item[\"content\"] = new_content\n item[\"truncated\"] = True\n item[\"original_chars\"] = original\n\n\ndef _metadata_host(url: str) -> bool:\n from urllib.parse import urlparse\n\n try:\n host = (urlparse(url.strip()).hostname or \"\").lower().rstrip(\".\")\n except ValueError:\n return False\n return host in BLOCKED_METADATA_HOSTS\n\n\ndef extract_urls(\n urls: List[str],\n provider: str = \"auto\",\n output_format: str = \"markdown\",\n include_images: bool = False,\n include_raw_html: bool = False,\n render_js: bool = False,\n spans: bool = False,\n spans_query: Optional[str] = None,\n allow_private: bool = False,\n char_limit: Optional[int] = None,\n no_cache: bool = False,\n config: Optional[Dict[str, Any]] = None,\n max_wall_time_ms: Optional[int] = None,\n) -> Dict[str, Any]:\n \"\"\"Extract URLs through the engine with the skill's guards and inline budget.\"\"\"\n from wsp_core import extract as extract_module\n from wsp_core.compat_v3 import legacy_request_to_v3, v3_response_to_legacy_extract\n from wsp_core.contract_v3 import Capability\n from wsp_core.orchestrator_v3 import execute_v3_request\n\n selected = provider or \"auto\"\n if str(selected).lower() in EXCLUDED_PROVIDERS:\n return {\"provider\": selected, \"results\": [], \"error\": f\"Provider {selected} is not part of this skill\",\n \"requested_provider\": selected}\n urls = [u.strip() for u in urls if isinstance(u, str) and u.strip()]\n if not urls:\n return {\"provider\": selected, \"results\": [], \"error\": \"No URLs provided\", \"requested_provider\": selected}\n runtime_config = adapt_config(config if config is not None else _config.load_config(),\n allow_private_urls=allow_private)\n no_cache = no_cache or cache_disabled()\n\n # With private targets allowed the engine skips its network checks; cloud\n # metadata endpoints stay blocked here, per URL.\n blocked = {}\n if runtime_config.get(\"extract\", {}).get(\"allow_private_urls\") is True:\n blocked = {u: f\"Extraction URL blocked: {u} is a cloud metadata endpoint\" for u in urls if _metadata_host(u)}\n to_fetch = [u for u in urls if u not in blocked]\n\n if to_fetch:\n request = legacy_request_to_v3(\n Capability.EXTRACT,\n {\n \"urls\": to_fetch,\n \"provider\": selected,\n \"output_format\": output_format,\n \"include_images\": include_images,\n \"include_raw_html\": include_raw_html,\n \"render_js\": render_js,\n \"spans\": spans,\n \"spans_query\": spans_query,\n \"no_cache\": no_cache,\n \"max_wall_time_ms\": max_wall_time_ms,\n },\n )\n execution = execute_v3_request(request, extract_module._extract_adapter(), runtime_config)\n result = v3_response_to_legacy_extract(execution)\n extract_module._resolve_full_text_paths(result, runtime_config)\n else:\n result = {\"provider\": selected, \"results\": [], \"error\": \"; \".join(blocked.values())}\n\n if blocked and to_fetch:\n by_url = {item.get(\"url\"): item for item in result.get(\"results\") or [] if isinstance(item, dict)}\n merged = []\n for url in urls:\n if url in blocked:\n merged.append({\"url\": url, \"error\": blocked[url]})\n elif url in by_url:\n merged.append(by_url.pop(url))\n merged.extend(by_url.values())\n result[\"results\"] = merged\n\n limit = extract_char_limit(char_limit, runtime_config)\n for item in result.get(\"results\") or []:\n if isinstance(item, dict) and not item.get(\"error\"):\n apply_char_limit(item, limit)\n return result\n\n\ndef extract_succeeded(result: Dict[str, Any]) -> bool:\n return not result.get(\"error\") and any(\n isinstance(item, dict) and not item.get(\"error\")\n and any(item.get(field) for field in (\"content\", \"text\", \"raw_content\", \"markdown\", \"html\"))\n for item in result.get(\"results\", [])\n )\n\n\ndef build_extract_parser() -> argparse.ArgumentParser:\n from wsp_core.provider_registry import EXTRACT_PROVIDER_IDS\n\n providers = [p for p in EXTRACT_PROVIDER_IDS if p not in EXCLUDED_PROVIDERS]\n parser = argparse.ArgumentParser(description=\"Extract URL content with automatic provider fallback\")\n parser.add_argument(\"--url\", dest=\"urls\", action=\"append\", help=\"URL to extract (repeatable)\")\n parser.add_argument(\"--provider\", default=\"auto\", type=str.lower, choices=[\"auto\"] + providers)\n parser.add_argument(\"--format\", default=\"markdown\", choices=[\"markdown\", \"html\"])\n parser.add_argument(\"--include-images\", action=\"store_true\")\n parser.add_argument(\"--include-raw-html\", action=\"store_true\")\n parser.add_argument(\"--render-js\", action=\"store_true\")\n parser.add_argument(\"--spans\", action=\"store_true\", help=\"Select query-ranked passages from the extracted text\")\n parser.add_argument(\"--spans-query\", help=\"Query used to rank --spans passages\")\n parser.add_argument(\"--allow-private-urls\", action=\"store_true\",\n help=f\"Allow URLs that resolve to private/internal networks (off by default; see {ALLOW_PRIVATE_ENV}); \"\n \"cloud metadata endpoints stay blocked\")\n parser.add_argument(\"--extract-char-limit\", type=int,\n help=f\"Inline content budget per URL in characters (default {DEFAULT_EXTRACT_CHAR_LIMIT}; \"\n f\"longer pages return a head/tail window). Set {EXTRACT_CHAR_LIMIT_ENV} to change the default.\")\n parser.add_argument(\"--no-cache\", action=\"store_true\",\n help=f\"Do not read or write the extraction cache (set {DISABLE_CACHE_ENV}=1 to disable it globally)\")\n parser.add_argument(\"--compact\", action=\"store_true\", help=\"Print compact JSON\")\n return parser\n\n\ndef run_extract(argv: Optional[List[str]] = None) -> None:\n \"\"\"``python3 scripts/extract.py`` entry point.\"\"\"\n args = build_extract_parser().parse_args(argv)\n prepare_cache_dir()\n result = extract_urls(\n args.urls or [],\n provider=args.provider,\n output_format=args.format,\n include_images=args.include_images,\n include_raw_html=args.include_raw_html,\n render_js=args.render_js,\n spans=args.spans,\n spans_query=args.spans_query,\n allow_private=args.allow_private_urls,\n char_limit=args.extract_char_limit,\n no_cache=args.no_cache,\n )\n print(json.dumps(result, ensure_ascii=False, indent=None if args.compact else 2))\n if not extract_succeeded(result):\n sys.exit(1)\n","content_type":"text/x-python; charset=utf-8","language":"python","size":31131,"content_sha256":"e80cb2bc75487ef46aeea03019db10b7fd5e8fb02c40ccec6c360dffb4a177f3"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"Web Search Plus","type":"text"}]},{"type":"paragraph","content":[{"text":"Stop choosing search providers. Let the skill do it for you.","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"One command searches across ","type":"text"},{"text":"11 search providers","type":"text","marks":[{"type":"strong"}]},{"text":", and a companion command extracts page text from URLs. Since 5.0 the skill runs the same engine as Web Search Plus 5.0.1: automatic routing picks the first provider by query type and moves on quickly when a provider fails, is slow or returns nothing.","type":"text"}]},{"type":"paragraph","content":[{"text":"This skill is ","type":"text"},{"text":"source-only","type":"text","marks":[{"type":"strong"}]},{"text":": ranked links and extracted page text, not model-written answers. The native OpenClaw plugin is ","type":"text"},{"text":"web-search-plus-plugin-v2","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"🔐 Data Handling & Privacy","type":"text"}]},{"type":"paragraph","content":[{"text":"Read this before searching with sensitive queries.","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Search queries and extraction URLs are sent to third-party providers.","type":"text","marks":[{"type":"strong"}]},{"text":" Every search sends your query text to the provider that serves it (Serper, Brave, Tavily, Linkup, Querit, Exa, Firecrawl, SerpBase, Keenable, You.com, or your SearXNG instance). When a provider fails, is slow or returns nothing, the query also goes to the next provider in line. Every extraction sends the target URL to the chosen extraction provider (Tavily, Exa, Linkup, Firecrawl, You.com, Keenable, Serper), whose servers fetch the page. Each provider's own privacy policy and retention rules apply.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For sensitive work, select the provider explicitly","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"--provider \u003cname>","type":"text","marks":[{"type":"code_inline"}]},{"text":") instead of relying on automatic routing, so you control exactly which third party receives the query. An explicit provider does not fall back unless you add ","type":"text"},{"text":"--allow-fallback","type":"text","marks":[{"type":"code_inline"}]},{"text":". Self-hosted SearXNG keeps queries on infrastructure you control.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Do not submit internal or private URLs for extraction.","type":"text","marks":[{"type":"strong"}]},{"text":" URLs you extract are forwarded to external services. The skill blocks private, loopback and link-local targets and cloud metadata endpoints by default (see Security below).","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Local caching is on by default.","type":"text","marks":[{"type":"strong"}]},{"text":" Search and extraction answers (including the query text), the full text of long extracted pages (kept up to 7 days), provider state (","type":"text"},{"text":"v3/state.sqlite3","type":"text","marks":[{"type":"code_inline"}]},{"text":": circuit breakers, budgets, latency samples; ","type":"text"},{"text":"provider_stats.json","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"provider_health.json","type":"text","marks":[{"type":"code_inline"}]},{"text":") and routing receipts without query text are stored under the cache directory (","type":"text"},{"text":".cache","type":"text","marks":[{"type":"code_inline"}]},{"text":" by default, ","type":"text"},{"text":"WSP_CACHE_DIR","type":"text","marks":[{"type":"code_inline"}]},{"text":" to relocate). Directories are created ","type":"text"},{"text":"0700","type":"text","marks":[{"type":"code_inline"}]},{"text":", files ","type":"text"},{"text":"0600","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Bypass the answer cache for one call: ","type":"text"},{"text":"--no-cache","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Bypass it for every call: ","type":"text"},{"text":"WSP_DISABLE_CACHE=1","type":"text","marks":[{"type":"code_inline"}]},{"text":" (provider state is still kept)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Wipe cached answers: ","type":"text"},{"text":"python3 scripts/search.py --clear-cache","type":"text","marks":[{"type":"code_inline"}]},{"text":" (inspect with ","type":"text"},{"text":"--cache-stats","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"API keys are never logged or cached","type":"text","marks":[{"type":"strong"}]},{"text":"; error text is sanitized before it reaches the cache or stderr.","type":"text"}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"🎯 Triggers","type":"text"}]},{"type":"paragraph","content":[{"text":"To avoid auto-activating on everyday requests, the manifest registers only narrowly scoped trigger phrases:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"web search plus","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"wsp search","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"search the web for","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"multi-provider web search","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"extract url content","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"extract content from url","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"paragraph","content":[{"text":"Generic words like \"search\", \"find\", \"look up\", or \"research\" intentionally do ","type":"text"},{"text":"not","type":"text","marks":[{"type":"strong"}]},{"text":" trigger this skill.","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"✨ What Makes This Different?","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Just search","type":"text","marks":[{"type":"strong"}]},{"text":": no need to think about which provider to use","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Routing by query type","type":"text","marks":[{"type":"strong"}]},{"text":": Brave for general, news, local and community queries, Exa for docs and academic queries, Serper for security and shopping queries","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Fast fallback","type":"text","marks":[{"type":"strong"}]},{"text":": if the first provider fails, answers with nothing, or is slower than usual (at least 2.5 s), the next one starts and the first useful answer wins","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"11 providers, 1 interface","type":"text","marks":[{"type":"strong"}]},{"text":": general web, research, semantic discovery, privacy-first, prepaid-credits, and extraction-capable providers together","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"URL extraction included","type":"text","marks":[{"type":"strong"}]},{"text":": markdown/HTML page text with fallback across seven providers; one bad URL no longer sinks the batch","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Research mode","type":"text","marks":[{"type":"strong"}]},{"text":": several providers at once, results merged by rank, top sources extracted with short query-ranked passages","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Works with just 1 credential","type":"text","marks":[{"type":"strong"}]},{"text":": start with any single provider, add more later","type":"text"}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"🚀 Quick Start","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Interactive setup (recommended for first run)\npython3 scripts/setup.py\n\n# Or manually\ncp .env.example .env\npython3 scripts/search.py -q \"latest OpenClaw release\"\npython3 scripts/extract.py --url https://example.com","type":"text"}]},{"type":"paragraph","content":[{"text":"With only one key, Brave is the best single choice for automatic routing; Exa and Serper cover the other query types.","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"🔑 Providers","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Search providers","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Brave","type":"text","marks":[{"type":"strong"}]},{"text":": independent web index; first choice for general, news, local and community queries","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Serper","type":"text","marks":[{"type":"strong"}]},{"text":": Google-style results; first choice for security and shopping queries","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Exa","type":"text","marks":[{"type":"strong"}]},{"text":": semantic search with source URLs; first choice for docs and academic queries, also ","type":"text"},{"text":"--similar-url","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Tavily","type":"text","marks":[{"type":"strong"}]},{"text":": research-oriented search; fourth in the fallback chain","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Querit","type":"text","marks":[{"type":"strong"}]},{"text":": multilingual and international search; explicit-only unless allowed in ","type":"text"},{"text":"auto_routing.auto_allow","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Linkup","type":"text","marks":[{"type":"strong"}]},{"text":": source-grounded search","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Firecrawl","type":"text","marks":[{"type":"strong"}]},{"text":": search with scrape-ready metadata","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"You.com","type":"text","marks":[{"type":"strong"}]},{"text":": current-web snippets","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"SearXNG","type":"text","marks":[{"type":"strong"}]},{"text":": private/self-hosted search; no API key, just an instance URL","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"SerpBase","type":"text","marks":[{"type":"strong"}]},{"text":": low-cost Google SERP with prepaid credits; explicit/fallback-only (","type":"text"},{"text":"--provider serpbase","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Keenable","type":"text","marks":[{"type":"strong"}]},{"text":": independent web index; keyed via ","type":"text"},{"text":"KEENABLE_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" or keyless against the ","type":"text"},{"text":"opt-in","type":"text","marks":[{"type":"strong"}]},{"text":" shared public tier (","type":"text"},{"text":"WSP_KEENABLE_ALLOW_PUBLIC=1","type":"text","marks":[{"type":"code_inline"}]},{"text":", no SLA, warning in result metadata). Last in the fallback chain","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Extraction providers","type":"text"}]},{"type":"paragraph","content":[{"text":"scripts/extract.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" falls back in this order:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Tavily","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Exa","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Linkup","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Firecrawl","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"You.com","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Keenable","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Serper","type":"text","marks":[{"type":"strong"}]},{"text":" (webpage scraper via ","type":"text"},{"text":"scrape.serper.dev","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"🧠 Routing at a Glance","type":"text"}]},{"type":"paragraph","content":[{"text":"Automatic routing gives each query one of eight types and picks the first provider from that:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"text"},"content":[{"text":"academic, docs → Exa\nsecurity, shopping → Serper\neverything else → Brave (general, news, local, community)","type":"text"}]},{"type":"paragraph","content":[{"text":"A provider without a key is skipped. Fallback then follows ","type":"text"},{"text":"auto_routing.provider_priority","type":"text","marks":[{"type":"code_inline"}]},{"text":", by default:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"text"},"content":[{"text":"brave → serper → exa → tavily → you → firecrawl → linkup → searxng → keenable","type":"text"}]},{"type":"paragraph","content":[{"text":"SerpBase and Querit are in the list too but are only used when named with ","type":"text"},{"text":"--provider","type":"text","marks":[{"type":"code_inline"}]},{"text":" or allowed with ","type":"text"},{"text":"\"auto_routing\": {\"auto_allow\": {\"querit\": true}}","type":"text","marks":[{"type":"code_inline"}]},{"text":". To use your own order for every query instead, set ","type":"text"},{"text":"\"auto_routing\": {\"order\": \"custom\", \"provider_priority\": [...]}","type":"text","marks":[{"type":"code_inline"}]},{"text":" in ","type":"text"},{"text":"config.json","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"Examples:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/search.py -q \"weather in Vienna today\" # local → Brave\npython3 scripts/search.py -q \"python asyncio TaskGroup documentation\" # docs → Exa\npython3 scripts/search.py -q \"iPhone 16 Pro Max price\" # shopping → Serper\npython3 scripts/search.py -p firecrawl -q \"YC startups web scraping\" # explicit provider\npython3 scripts/search.py -p serpbase -q \"best laptop 2026\" # explicit SerpBase","type":"text"}]},{"type":"paragraph","content":[{"text":"Show the decision without searching:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/search.py --explain-routing -q \"your query\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Queries starting with a dash can use ","type":"text"},{"text":"--query=-foo","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"-q -- \"-foo\"","type":"text","marks":[{"type":"code_inline"}]},{"text":". When ","type":"text"},{"text":"-n","type":"text","marks":[{"type":"code_inline"}]},{"text":" is omitted, ","type":"text"},{"text":"defaults.max_results","type":"text","marks":[{"type":"code_inline"}]},{"text":" supplies the count (fallback 5); counts are clamped to 1–20. Long queries are shortened at a word boundary when a provider has a length limit (Brave: 600 characters / 75 words), and the output says so.","type":"text"}]},{"type":"paragraph","content":[{"text":"--cache-ttl SECONDS","type":"text","marks":[{"type":"code_inline"}]},{"text":" sets the answer-cache lifetime; recency, date filters and news/security queries (300 s) can only lower it. ","type":"text"},{"text":"--no-cache","type":"text","marks":[{"type":"code_inline"}]},{"text":" bypasses the cache. Set ","type":"text"},{"text":"WSP_HTTP_KEEPALIVE=0","type":"text","marks":[{"type":"code_inline"}]},{"text":" to disable connection reuse; requests through proxies use the standard transport.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Freshness, news vertical & locale","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/search.py -q \"AI regulation\" --freshness week\n# providers with native date filters receive the mapped value; others run the\n# normal search and report freshness.applied=false in metadata\n\npython3 scripts/search.py -p serper --type news -q \"quantum computing\"\n# Serper serves the Google news vertical natively; other providers report\n# search_type.applied=false\n\npython3 scripts/search.py -q \"beste Kaffeehäuser Wien\"\n# location hints in the query set the country (at); a clearly German query is\n# searched in German. Resolved values appear in metadata.locale.","type":"text"}]},{"type":"paragraph","content":[{"text":"Without a configured language, a clearly detected query language is sent to the provider; unclear queries go out as English. Set a fixed language with ","type":"text"},{"text":"\"defaults\": {\"locale\": {\"language\": \"en\"}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" in ","type":"text"},{"text":"config.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"WSP_LOCALE_LANGUAGE=en","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Domain filters and result quality","type":"text"}]},{"type":"paragraph","content":[{"text":"--include-domains","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"--exclude-domains","type":"text","marks":[{"type":"code_inline"}]},{"text":" work on every search provider (Brave, Serper, SerpBase and You.com get ","type":"text"},{"text":"site:","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"-site:","type":"text","marks":[{"type":"code_inline"}]},{"text":" operators). Known mirror and spam domains are removed, and one domain fills at most two of the top results; explicit domain filters bypass both. For docs, security and news queries the site of the named vendor or project ranks first.","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"📖 Extraction Examples","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/extract.py --url https://example.com\npython3 scripts/extract.py --url https://docs.linkup.so --provider linkup\npython3 scripts/extract.py --url https://example.com --url https://example.org --include-images\npython3 scripts/extract.py --url https://example.com --format html --include-raw-html\npython3 scripts/extract.py --url https://example.com --provider serper\npython3 scripts/extract.py --url https://example.com --extract-char-limit 30000\npython3 scripts/extract.py --url https://example.com --spans --spans-query \"pricing\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Each URL gets its own result: a blocked or failing URL gets an ","type":"text"},{"text":"error","type":"text","marks":[{"type":"code_inline"}]},{"text":" entry and the others are still extracted, retried on the next provider if needed, in the order you gave them. Long pages return a head/tail window plus a footer with the real page length (default 15,000 characters per page; ","type":"text"},{"text":"WSP_EXTRACT_CHAR_LIMIT","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"--extract-char-limit","type":"text","marks":[{"type":"code_inline"}]},{"text":"). When the engine kept the full text, the footer names the file. Inline base64 images become ","type":"text"},{"text":"[IMAGE: alt]","type":"text","marks":[{"type":"code_inline"}]},{"text":" placeholders. Extraction answers are cached for an hour (","type":"text"},{"text":"--no-cache","type":"text","marks":[{"type":"code_inline"}]},{"text":" to bypass).","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"🔬 Research Mode & Quality Reports","type":"text"}]},{"type":"paragraph","content":[{"text":"Research mode queries up to three providers at once, merges their results by rank (a page found by several providers ranks higher and lists them in ","type":"text"},{"text":"providers","type":"text","marks":[{"type":"code_inline"}]},{"text":"), then extracts the top sources and adds short query-ranked passages:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/search.py --mode research -q \"EU AI Act obligations for foundation models\"\npython3 scripts/search.py --mode research -q \"...\" --research-providers tavily linkup exa\npython3 scripts/search.py --mode research -q \"...\" --research-extract-count 2 --research-time-budget 30","type":"text"}]},{"type":"paragraph","content":[{"text":"Quality reports add routing and result diagnostics, including authority signals:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/search.py -q \"official anthropic claude release notes\" --quality-report","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"⚙️ Configuration Notes","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":".env.example","type":"text","marks":[{"type":"code_inline"}]},{"text":" documents supported env vars; ","type":"text"},{"text":".env","type":"text","marks":[{"type":"code_inline"}]},{"text":" is read from the skill directory only","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"config.example.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" shows the 5.0 layout; ","type":"text"},{"text":"config.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" is your local config","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"4.x ","type":"text"},{"text":"config.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" files keep working: a top-level ","type":"text"},{"text":"locale","type":"text","marks":[{"type":"code_inline"}]},{"text":" block, provider ","type":"text"},{"text":"country","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"language","type":"text","marks":[{"type":"code_inline"}]},{"text":" keys, the old default provider order and the old SerpBase endpoint are translated when the config is loaded","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"SerpBase is ","type":"text"},{"text":"explicit/fallback-only","type":"text","marks":[{"type":"strong"}]},{"text":" by default","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Keenable is last in search and extraction fallback; enable keyless use with ","type":"text"},{"text":"WSP_KEENABLE_ALLOW_PUBLIC=1","type":"text","marks":[{"type":"code_inline"}]},{"text":" (or ","type":"text"},{"text":"KEENABLE_ALLOW_PUBLIC=1","type":"text","marks":[{"type":"code_inline"}]},{"text":", or ","type":"text"},{"text":"\"keenable\": {\"allow_public\": true}","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Locale defaults: ","type":"text"},{"text":"\"defaults\": {\"locale\": {\"country\": \"...\", \"language\": \"...\"}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" in ","type":"text"},{"text":"config.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" (or ","type":"text"},{"text":"WSP_LOCALE_COUNTRY","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"WSP_LOCALE_LANGUAGE","type":"text","marks":[{"type":"code_inline"}]},{"text":"); ","type":"text"},{"text":"--country","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"--language","type":"text","marks":[{"type":"code_inline"}]},{"text":" always win","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Rate limits: ","type":"text"},{"text":"Retry-After","type":"text","marks":[{"type":"code_inline"}]},{"text":" waits are capped at 30 s; a longer one skips the retry and leaves the provider blocked (at most one hour). Missing keys never block a provider","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Extra commands: ","type":"text"},{"text":"python3 scripts/search.py doctor","type":"text","marks":[{"type":"code_inline"}]},{"text":" (offline configuration check), ","type":"text"},{"text":"python3 scripts/search.py --extract-urls URL ...","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"🔒 Security","type":"text"}]},{"type":"paragraph","content":[{"text":"URL SSRF protection (extraction, ","type":"text","marks":[{"type":"strong"}]},{"text":"--similar-url","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":"):","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Only ","type":"text"},{"text":"http","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"https","type":"text","marks":[{"type":"code_inline"}]},{"text":" URLs are accepted; URLs that different URL parsers read differently are rejected","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Hostnames are resolved and blocked if they point at private, loopback, link-local or reserved ranges; error messages do not reveal the resolved address","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Cloud metadata endpoints (","type":"text"},{"text":"169.254.169.254","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"metadata.google.internal","type":"text","marks":[{"type":"code_inline"}]},{"text":") are always blocked","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Opt out for trusted private networks with ","type":"text"},{"text":"--allow-private-urls","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"WSP_ALLOW_PRIVATE_URLS=1","type":"text","marks":[{"type":"code_inline"}]},{"text":" (off by default; metadata endpoints stay blocked)","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"SearXNG SSRF protection:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Enforces ","type":"text"},{"text":"http","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"https","type":"text","marks":[{"type":"code_inline"}]},{"text":" only","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A SearXNG URL on a private or LAN address needs ","type":"text"},{"text":"SEARXNG_ALLOW_PRIVATE=1","type":"text","marks":[{"type":"code_inline"}]},{"text":"; without it SearXNG is skipped and other providers keep working","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Local data:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Cache directory ","type":"text"},{"text":"0700","type":"text","marks":[{"type":"code_inline"}]},{"text":", files ","type":"text"},{"text":"0600","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"API keys are never written to the cache or logs","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"The skill starts no subprocesses and opens no listening sockets","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Declared permissions","type":"text","marks":[{"type":"strong"}]},{"text":" (see ","type":"text"},{"text":"package.json → clawhub.permissions","type":"text","marks":[{"type":"code_inline"}]},{"text":"):","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Outbound network access limited to the listed provider API hosts (plus any user-configured SearXNG instance)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Environment reads limited to provider ","type":"text"},{"text":"*_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" vars, ","type":"text"},{"text":"SEARXNG_*","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"KEENABLE_ALLOW_PUBLIC","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"WEB_SEARCH_PLUS_CONFIG","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"WSP_*","type":"text","marks":[{"type":"code_inline"}]},{"text":" settings","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Filesystem writes limited to the cache directory (and ","type":"text"},{"text":"config.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" from the setup wizard)","type":"text"}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"✅ Verification","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 -m unittest discover -s tests -p 'test_*.py'\npython3 scripts/search.py --explain-routing -q \"find credible sources for climate change impacts\"\npython3 scripts/search.py doctor\npython3 scripts/extract.py --url https://example.com --provider auto --compact","type":"text"}]}]},"metadata":{"date":"2026-11-02","name":"web-search-plus","tags":["search","keenable","news","locale","web-search","web-extract","serper","brave","tavily","querit","linkup","exa","firecrawl","you","searxng","serpbase","google","multilingual-search","research","semantic-search","auto-routing","multi-provider","shopping","rag","free-tier","privacy","self-hosted"],"author":"@skillopedia","source":{"stars":25,"repo_name":"web-search-plus","origin_url":"https://github.com/robbyczgw-cla/web-search-plus/blob/HEAD/SKILL.md","repo_owner":"robbyczgw-cla","body_sha256":"91af8f15fb56c64b7537f1e1ea97d5211b84086dbbbefb63bc45466fab36fc28","cluster_key":"55c031ad36287be3b49c2daf50a57beffc20bb7a254fdd52bc531e6e10f914be","clean_bundle":{"format":"clean-skill-bundle-v1","source":"robbyczgw-cla/web-search-plus/SKILL.md","text_bytes":956923,"source_commit":"cfaaea434d77ceed9216cc02ef2b477a9e46705a","skills_sh_rank":11171,"attachment_count":52,"text_attachments":52,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"5.0.1","category":"browser-automation-scraping","category_label":"Browser"},"exact_dupes_collapsed_into_this":0},"version":"5.0.1","category":"browser-automation-scraping","metadata":{"openclaw":{"requires":{"env":{"EXA_API_KEY":"optional","YOU_API_KEY":"optional","BRAVE_API_KEY":"optional","LINKUP_API_KEY":"optional","QUERIT_API_KEY":"optional","SERPER_API_KEY":"optional","TAVILY_API_KEY":"optional","KEENABLE_API_KEY":"optional — Keenable independent web index (search + extraction); keyless public tier opt-in via WSP_KEENABLE_ALLOW_PUBLIC=1","SERPBASE_API_KEY":"optional — explicit/fallback-only Google SERP provider with prepaid credits","FIRECRAWL_API_KEY":"optional","SEARXNG_INSTANCE_URL":"optional"},"bins":["python3","bash"],"note":"Only ONE provider key or SEARXNG_INSTANCE_URL is needed for search. Brave, Serper and Exa get the most use from automatic routing. Extraction requires one of Tavily, Exa, Linkup, Firecrawl, You.com, Keenable, or Serper.","permissions":{"env":"reads provider *_API_KEY vars, SEARXNG_INSTANCE_URL, SEARXNG_ALLOW_PRIVATE, KEENABLE_ALLOW_PUBLIC, WEB_SEARCH_PLUS_CONFIG, WSP_BUDGET_PREFLIGHT_OFF, and WSP_* settings (WSP_CACHE_DIR, WSP_DISABLE_CACHE, WSP_ALLOW_PRIVATE_URLS, WSP_KEENABLE_ALLOW_PUBLIC, WSP_EXTRACT_CHAR_LIMIT, WSP_LOCALE_COUNTRY, WSP_LOCALE_LANGUAGE, WSP_HTTP_KEEPALIVE)","network":"outbound HTTPS to the configured provider API hosts only (google.serper.dev, api.search.brave.com, api.tavily.com, api.querit.ai, api.linkup.so, api.exa.ai, api.firecrawl.dev, ydc-index.io, api.serpbase.dev, api.keenable.ai, scrape.serper.dev, plus any user-configured SearXNG instance)","filesystem":"writes to the cache directory (.cache by default, WSP_CACHE_DIR override): search and extraction response cache, full text of long extracted pages, provider state (state.sqlite3, provider_stats.json, provider_health.json) and routing receipts without query text; the setup wizard writes config.json in the skill directory"}}}},"import_tag":"skills-sh-top20k-2026-11","description":"Search the web and extract URLs with routing across Serper, Brave, Tavily, Querit, Linkup, Exa, Firecrawl, You.com, SearXNG, SerpBase, and Keenable. Returns ranked sources and page text; supports freshness filters and research across providers."}},"renderedAt":1793666133362}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.