Tech News Digest Evidence-led tech digest with unified collection and one editorial selection shared across formats. Follow as the authoritative mode policy. Editorial Contract - Daily preserves original sections: 2–4 sentence overview; configured topics (normally LLM, AI Agent, Crypto, Frontier Tech; aim for 3–5 qualified items each); KOL viewpoints ≤3; releases ≤3 repositories; project discovery ≤3; blog picks ≤3; ≤30 unique items overall. News/blog explanations normally contain 2–3 concise sentences of change, evidence and implication. Retain empty headings with a coverage note, never fill…

, '', title) # Remove \" | Site Name\" endings\n \n # Normalize whitespace and punctuation\n title = re.sub(r'\\s+', ' ', title).strip()\n title = re.sub(r'[^\\w\\s]', '', title.lower())\n \n return title\n\n\ndef calculate_title_similarity(title1: str, title2: str) -> float:\n \"\"\"Calculate similarity between two titles.\"\"\"\n norm1 = normalize_title(title1)\n norm2 = normalize_title(title2)\n \n if not norm1 or not norm2:\n return 0.0\n \n return SequenceMatcher(None, norm1, norm2).ratio()\n\n\ndef get_domain(url: str) -> str:\n \"\"\"Extract domain from URL.\"\"\"\n try:\n return urlparse(url).netloc.lower().replace('www.', '')\n except Exception:\n return ''\n\n\n# Query parameters that carry no article identity — safe to drop when comparing URLs.\nTRACKING_PARAMS = {\n \"utm_source\", \"utm_medium\", \"utm_campaign\", \"utm_term\", \"utm_content\", \"utm_id\",\n \"ref\", \"referrer\", \"source\", \"src\", \"fbclid\", \"gclid\", \"mc_cid\", \"mc_eid\",\n \"spm\", \"share_id\", \"_hsenc\", \"_hsmi\", \"igshid\", \"si\", \"feature\", \"at_medium\",\n}\n\n\ndef normalize_url(url: str) -> str:\n \"\"\"Normalize URL for dedup comparison.\n\n Strips fragment, trailing slash, `www.` and known tracking parameters, but\n KEEPS the remaining query string: for sites like YouTube (`/watch?v=ID`) the\n query *is* the article identity, and dropping it collapses every video from\n every channel into a single entry.\n \"\"\"\n try:\n parsed = urlparse(url)\n domain = parsed.netloc.lower().replace('www.', '')\n path = parsed.path.rstrip('/')\n kept = [\n (k, v) for k, v in parse_qsl(parsed.query, keep_blank_values=True)\n if k.lower() not in TRACKING_PARAMS\n ]\n query = urlencode(sorted(kept)) if kept else \"\"\n return f\"{domain}{path}?{query}\" if query else f\"{domain}{path}\"\n except Exception:\n return url\n\n\ndef calculate_base_score(article: Dict[str, Any], source: Dict[str, Any]) -> float:\n \"\"\"Calculate base quality score for an article.\"\"\"\n score = 0.0\n \n # Priority source bonus\n if source.get(\"priority\", False):\n score += SCORE_PRIORITY_SOURCE\n \n # Recency bonus (\u003c 24 hours)\n try:\n article_date = datetime.fromisoformat(article[\"date\"].replace('Z', '+00:00'))\n hours_old = (datetime.now(timezone.utc) - article_date).total_seconds() / 3600\n if hours_old \u003c 24:\n score += SCORE_RECENT\n except Exception:\n pass\n \n # Twitter engagement bonus (tiered)\n if source.get(\"source_type\") == \"twitter\" and \"metrics\" in article:\n metrics = article[\"metrics\"]\n likes = metrics.get(\"like_count\", 0)\n retweets = metrics.get(\"retweet_count\", 0)\n \n if likes >= 1000 or retweets >= 500:\n score += SCORE_ENGAGEMENT_VIRAL\n elif likes >= 500 or retweets >= 200:\n score += SCORE_ENGAGEMENT_HIGH\n elif likes >= 100 or retweets >= 50:\n score += SCORE_ENGAGEMENT_MED\n elif likes >= 50 or retweets >= 20:\n score += SCORE_ENGAGEMENT_LOW\n\n # RSS from priority sources get extra weight (official blogs, research papers)\n if source.get(\"source_type\") == \"rss\" and source.get(\"priority\", False):\n score += 2 # Extra priority RSS bonus\n\n return score\n\n\ndef _extract_tokens(title: str) -> Set[str]:\n \"\"\"Extract significant tokens from a normalized title for bucketing.\"\"\"\n norm = normalize_title(title)\n # Split into tokens, filter short/common words\n stopwords = {'the', 'a', 'an', 'is', 'are', 'was', 'were', 'in', 'on', 'at',\n 'to', 'for', 'of', 'and', 'or', 'with', 'by', 'from', 'as', 'it',\n 'its', 'that', 'this', 'be', 'has', 'had', 'have', 'not', 'but',\n 'what', 'how', 'new', 'will', 'can', 'do', 'does', 'did'}\n tokens = set()\n for word in norm.split():\n if len(word) >= 3 and word not in stopwords:\n tokens.add(word)\n return tokens\n\n\ndef _build_token_buckets(articles: List[Dict[str, Any]]) -> Dict[int, Set[int]]:\n \"\"\"Build token-based buckets mapping each article index to candidate duplicate indices.\n\n Two articles are candidates if they share 2+ significant tokens, or if their\n normalized titles are identical (which catches short titles like \"GPT-5\" that\n carry fewer than 2 significant tokens).\n Returns dict: article_index -> set of candidate article indices to compare against.\n \"\"\"\n from collections import defaultdict\n\n # token -> list of article indices\n token_to_indices: Dict[str, List[int]] = defaultdict(list)\n title_to_indices: Dict[str, List[int]] = defaultdict(list)\n article_tokens: List[Set[str]] = []\n\n for i, article in enumerate(articles):\n title = article.get(\"title\", \"\")\n tokens = _extract_tokens(title)\n article_tokens.append(tokens)\n for token in tokens:\n token_to_indices[token].append(i)\n norm = normalize_title(title)\n if norm:\n title_to_indices[norm].append(i)\n\n # For each article, find candidates sharing 2+ tokens\n candidates: Dict[int, Set[int]] = defaultdict(set)\n for i, tokens in enumerate(article_tokens):\n # Count how many tokens each other article shares with this one\n overlap_count: Dict[int, int] = defaultdict(int)\n for token in tokens:\n for j in token_to_indices[token]:\n if j != i:\n overlap_count[j] += 1\n for j, count in overlap_count.items():\n if count >= 2:\n candidates[i].add(j)\n\n # Identical normalized titles are always candidates\n for indices in title_to_indices.values():\n if len(indices) > 1:\n for i in indices:\n candidates[i].update(j for j in indices if j != i)\n\n return candidates\n\n\ndef _absorb_duplicate(primary: Dict[str, Any], duplicate: Dict[str, Any]) -> None:\n \"\"\"Fold a duplicate's provenance into the surviving article.\n\n Cross-source corroboration is tracked here rather than in a separate\n exact-title pass, so that near-duplicate headlines (which the similarity\n dedup catches but exact matching does not) still earn the multi-source bonus\n instead of being silently discarded.\n \"\"\"\n types: Set[str] = primary.setdefault(\"_source_types\", {primary.get(\"source_type\", \"\")})\n types.add(duplicate.get(\"source_type\", \"\"))\n\n names: List[str] = primary.setdefault(\"_source_names\", [primary.get(\"source_name\", \"\")])\n dup_name = duplicate.get(\"source_name\", \"\")\n if dup_name and dup_name not in names:\n names.append(dup_name)\n\n\ndef apply_multi_source_bonus(articles: List[Dict[str, Any]]) -> List[Dict[str, Any]]:\n \"\"\"Reward articles corroborated by more than one source type.\n\n Bonus is SCORE_MULTI_SOURCE per *additional* source type, so a story seen in\n two source types gets +5 (matching the documented weighting).\n \"\"\"\n boosted = 0\n for article in articles:\n types = {t for t in article.pop(\"_source_types\", set()) if t}\n names = [n for n in article.pop(\"_source_names\", []) if n]\n if len(types) \u003c 2:\n continue\n article[\"quality_score\"] = article.get(\"quality_score\", 0) + SCORE_MULTI_SOURCE * (len(types) - 1)\n article[\"multi_source\"] = True\n article[\"source_count\"] = len(names) or len(types)\n article[\"all_sources\"] = names[:3]\n boosted += 1\n\n logging.info(f\"Multi-source bonus applied to {boosted} articles\")\n return articles\n\n\ndef deduplicate_articles(articles: List[Dict[str, Any]]) -> List[Dict[str, Any]]:\n \"\"\"Remove duplicate articles based on title similarity.\n \n Uses token-based bucketing to avoid O(n²) SequenceMatcher comparisons.\n Only articles sharing 2+ significant title tokens are compared.\n Domain saturation is handled separately per-topic after grouping.\n \"\"\"\n if not articles:\n return articles\n \n # Sort by quality score (highest first) to keep best versions\n articles.sort(key=lambda x: x.get(\"quality_score\", 0), reverse=True)\n\n # Phase 1: URL dedup (exact URL match after normalization)\n url_seen: Dict[str, int] = {} # normalized_url -> index in articles\n url_duplicates: Set[int] = set()\n for i, article in enumerate(articles):\n url = article.get(\"link\", \"\")\n if not url:\n continue\n norm_url = normalize_url(url)\n if norm_url in url_seen:\n # Keep the one with higher quality_score (articles already sorted by score)\n url_duplicates.add(i)\n _absorb_duplicate(articles[url_seen[norm_url]], article)\n logging.debug(f\"URL duplicate: {url} ~= {articles[url_seen[norm_url]].get('link','')}\")\n else:\n url_seen[norm_url] = i\n\n if url_duplicates:\n articles = [a for i, a in enumerate(articles) if i not in url_duplicates]\n logging.info(f\"URL dedup: removed {len(url_duplicates)} duplicates\")\n\n # Phase 2: Title similarity dedup\n deduplicated = []\n\n # Build token buckets for candidate pairs\n candidates = _build_token_buckets(articles)\n \n # Track which indices have been marked as duplicates\n duplicate_indices: Set[int] = set()\n \n for i, article in enumerate(articles):\n if i in duplicate_indices:\n continue\n \n title = article.get(\"title\", \"\")\n \n # Mark future candidates as duplicates using SequenceMatcher (only within bucket)\n for j in candidates.get(i, set()):\n if j > i and j not in duplicate_indices:\n other_title = articles[j].get(\"title\", \"\")\n # Quick length check — titles with >30% length difference are unlikely duplicates\n norm_i = normalize_title(title)\n norm_j = normalize_title(other_title)\n if abs(len(norm_i) - len(norm_j)) > 0.3 * max(len(norm_i), len(norm_j), 1):\n continue\n similarity = calculate_title_similarity(title, other_title)\n if similarity >= TITLE_SIMILARITY_THRESHOLD:\n logging.debug(f\"Title duplicate: '{other_title}' ~= '{title}' ({similarity:.2f})\")\n duplicate_indices.add(j)\n _absorb_duplicate(article, articles[j])\n \n deduplicated.append(article)\n \n logging.info(f\"Deduplication: {len(articles)} → {len(deduplicated)} articles\")\n return deduplicated\n\n\n# Domains exempt from per-topic limits (multi-author platforms)\nDOMAIN_LIMIT_EXEMPT = {\"x.com\", \"twitter.com\", \"github.com\", \"reddit.com\",\n \"youtube.com\", \"youtu.be\", \"arxiv.org\"}\n\ndef apply_domain_limits(articles: List[Dict[str, Any]], max_per_domain: int = 3) -> List[Dict[str, Any]]:\n \"\"\"Limit articles per domain within a single topic group.\n \n Should be called per-topic after group_by_topics() to ensure\n each topic gets its own domain budget.\n \"\"\"\n domain_counts: Dict[str, int] = {}\n result = []\n for article in articles:\n domain = get_domain(article.get(\"link\", \"\"))\n if domain and domain not in DOMAIN_LIMIT_EXEMPT:\n count = domain_counts.get(domain, 0)\n if count >= max_per_domain:\n logging.debug(f\"Domain limit ({max_per_domain}): skipping {domain} article in topic\")\n continue\n domain_counts[domain] = count + 1\n result.append(article)\n return result\n\n\ndef load_previous_digests(archive_dir: Path, days: int = 14) -> Set[str]:\n \"\"\"Load titles from previous digests to avoid repeats.\n \n Args:\n archive_dir: Path to digest archive directory\n days: Number of days to look back (default: 14, increased from 7)\n \"\"\"\n if not archive_dir.exists():\n return set()\n \n seen_titles = set()\n cutoff = datetime.now() - timedelta(days=days)\n \n try:\n for file_path in archive_dir.glob(\"*.md\"):\n # Extract date from filename\n match = re.search(r'(\\d{4}-\\d{2}-\\d{2})', file_path.name)\n if match:\n try:\n file_date = datetime.strptime(match.group(1), \"%Y-%m-%d\")\n if file_date \u003c cutoff:\n continue\n except ValueError:\n continue\n \n # Extract titles from markdown\n with open(file_path, 'r', encoding='utf-8') as f:\n content = f.read()\n \n # Simple title extraction (assumes format like \"- [Title](link)\")\n for match in re.finditer(r'-\\s*\\[([^\\]]+)\\]', content):\n title = normalize_title(match.group(1))\n if title:\n seen_titles.add(title)\n \n except Exception as e:\n logging.debug(f\"Failed to load previous digests: {e}\")\n \n logging.info(f\"Loaded {len(seen_titles)} titles from previous {days} days\")\n return seen_titles\n\n\ndef apply_previous_digest_penalty(articles: List[Dict[str, Any]], \n previous_titles: Set[str]) -> List[Dict[str, Any]]:\n \"\"\"Apply penalty to articles that appeared in previous digests.\"\"\"\n if not previous_titles:\n return articles\n \n penalized_count = 0\n for article in articles:\n norm_title = normalize_title(article.get(\"title\", \"\"))\n if norm_title in previous_titles:\n article[\"quality_score\"] = article.get(\"quality_score\", 0) + PENALTY_OLD_REPORT\n article[\"in_previous_digest\"] = True\n penalized_count += 1\n \n logging.info(f\"Applied previous digest penalty to {penalized_count} articles\")\n return articles\n\n\ndef load_topic_order(defaults_dir: Optional[Path], config_dir: Optional[Path] = None) -> Optional[List[str]]:\n \"\"\"Read configured topic ids, in file order, from defaults + user overlay.\"\"\"\n if not defaults_dir:\n return None\n try:\n try:\n from config_loader import load_merged_topics\n except ImportError:\n sys.path.append(str(Path(__file__).resolve().parent))\n from config_loader import load_merged_topics\n topics = load_merged_topics(Path(defaults_dir), config_dir)\n ids = [t[\"id\"] for t in topics if t.get(\"id\")]\n return ids or None\n except Exception as e:\n logging.warning(f\"Could not read topic order ({e}); using built-in ordering\")\n return None\n\n\ndef build_topic_priority(topic_ids: Optional[List[str]] = None) -> Dict[str, int]:\n \"\"\"Build the topic priority map used to assign multi-topic articles.\n\n Derived from the order topics appear in topics.json so the map cannot drift\n out of sync with the configured topic ids. `uncategorized` always sorts last.\n \"\"\"\n ids = list(topic_ids) if topic_ids else list(DEFAULT_TOPIC_ORDER)\n priority = {topic_id: i for i, topic_id in enumerate(ids)}\n priority.setdefault(\"uncategorized\", len(priority))\n return priority\n\n\ndef group_by_topics(articles: List[Dict[str, Any]], dedup_across_topics: bool = True,\n topic_priority: Optional[Dict[str, int]] = None) -> Dict[str, List[Dict[str, Any]]]:\n \"\"\"Group articles by their topics.\n\n Args:\n articles: List of articles to group\n dedup_across_topics: If True, ensure each article appears in only one topic\n (first topic by priority order)\n topic_priority: topic_id -> rank; lower ranks win when an article matches\n several topics. Defaults to the built-in topic order.\n \"\"\"\n topic_groups = {}\n seen_article_ids: Set[str] = set() # Track which articles have been placed\n\n if topic_priority is None:\n topic_priority = build_topic_priority()\n\n # Unknown topics sort after every configured one\n unknown_rank = len(topic_priority) + 1\n\n # Sort topics by priority for deterministic assignment\n def get_topic_priority(topic: str) -> int:\n return topic_priority.get(topic, unknown_rank)\n\n for article in articles:\n topics = article.get(\"topics\", [])\n if not topics:\n topics = [\"uncategorized\"]\n \n # Sort topics by priority to pick the best one\n sorted_topics = sorted(topics, key=get_topic_priority)\n \n # Create unique article ID for tracking\n article_id = normalize_title(article.get(\"title\", \"\"))\n \n if dedup_across_topics:\n # Check if this article has already been assigned to a topic\n if article_id in seen_article_ids:\n logging.debug(f\"Skip duplicate across topics: '{article.get('title', '')[:50]}...'\")\n continue\n seen_article_ids.add(article_id)\n \n # Assign to first (highest priority) topic\n primary_topic = sorted_topics[0]\n \n if primary_topic not in topic_groups:\n topic_groups[primary_topic] = []\n \n # Add copy with single topic for cleaner grouping\n article_copy = article.copy()\n article_copy[\"primary_topic\"] = primary_topic\n article_copy[\"all_topics\"] = topics # Keep original topics for reference\n topic_groups[primary_topic].append(article_copy)\n \n # Sort articles within each topic by quality score\n for topic in topic_groups:\n topic_groups[topic].sort(key=lambda x: x.get(\"quality_score\", 0), reverse=True)\n \n return topic_groups\n\n\ndef main():\n \"\"\"Main merge and scoring function.\"\"\"\n parser = argparse.ArgumentParser(\n description=\"Merge articles from all sources with quality scoring and deduplication.\",\n formatter_class=argparse.RawDescriptionHelpFormatter,\n epilog=\"\"\"\nExamples:\n python3 merge-sources.py --rss rss.json --twitter twitter.json --web web.json\n python3 merge-sources.py --rss rss.json --output merged.json --verbose\n python3 merge-sources.py --archive-dir workspace/archive/tech-digest\n \"\"\"\n )\n \n parser.add_argument(\n \"--rss\",\n type=Path,\n help=\"RSS fetch results JSON file\"\n )\n \n parser.add_argument(\n \"--twitter\",\n type=Path,\n help=\"Twitter fetch results JSON file\"\n )\n \n parser.add_argument(\n \"--web\",\n type=Path,\n help=\"Web search results JSON file\"\n )\n \n parser.add_argument(\n \"--github\",\n type=Path,\n help=\"GitHub releases results JSON file\"\n )\n \n parser.add_argument(\n \"--trending\",\n type=Path,\n help=\"GitHub trending repos JSON file\"\n )\n \n parser.add_argument(\n \"--reddit\",\n type=Path,\n help=\"Reddit posts results JSON file\"\n )\n \n parser.add_argument(\n \"--output\", \"-o\",\n type=Path,\n help=\"Output JSON path (default: auto-generated temp file)\"\n )\n \n parser.add_argument(\n \"--archive-dir\",\n type=Path,\n help=\"Archive directory for previous digest penalty\"\n )\n\n parser.add_argument(\n \"--defaults\",\n type=Path,\n default=Path(__file__).resolve().parent.parent / \"config\" / \"defaults\",\n help=\"Skill defaults config dir, used to derive topic priority order\"\n )\n\n parser.add_argument(\n \"--config\",\n type=Path,\n help=\"User configuration directory for overlays (optional)\"\n )\n\n parser.add_argument(\n \"--verbose\", \"-v\",\n action=\"store_true\",\n help=\"Enable verbose logging\"\n )\n \n args = parser.parse_args()\n logger = setup_logging(args.verbose)\n \n # Auto-generate unique output path if not specified\n if not args.output:\n fd, temp_path = tempfile.mkstemp(prefix=\"tech-news-digest-merged-\", suffix=\".json\")\n os.close(fd)\n args.output = Path(temp_path)\n \n try:\n # Load source data\n rss_data = load_source_data(args.rss)\n twitter_data = load_source_data(args.twitter)\n web_data = load_source_data(args.web)\n github_data = load_source_data(args.github)\n trending_data = load_source_data(args.trending) if hasattr(args, \"trending\") else None\n reddit_data = load_source_data(args.reddit)\n \n logger.info(f\"Loaded sources - RSS: {rss_data.get('total_articles', 0)}, \"\n f\"Twitter: {twitter_data.get('total_articles', 0)}, \"\n f\"Web: {web_data.get('total_articles', 0)}, \"\n f\"GitHub: {github_data.get('total_articles', 0)} releases + {trending_data.get('total', 0) if trending_data else 0} trending, \"\n f\"Reddit: {reddit_data.get('total_posts', 0)}\")\n \n # Collect all articles with source context\n all_articles = []\n \n # Process RSS articles\n for source in rss_data.get(\"sources\", []):\n for article in source.get(\"articles\", []):\n article[\"source_type\"] = \"rss\"\n article[\"source_name\"] = source.get(\"name\", \"\")\n article[\"source_id\"] = source.get(\"source_id\", \"\")\n article[\"quality_score\"] = calculate_base_score(article, source)\n all_articles.append(article)\n \n # Process Twitter articles\n for source in twitter_data.get(\"sources\", []):\n for article in source.get(\"articles\", []):\n article[\"source_type\"] = \"twitter\"\n article[\"source_name\"] = f\"@{source.get('handle', '')}\"\n article[\"display_name\"] = source.get(\"name\", \"\")\n article[\"source_id\"] = source.get(\"source_id\", \"\")\n article[\"quality_score\"] = calculate_base_score(article, source)\n all_articles.append(article)\n \n # Process Web articles\n for topic_result in web_data.get(\"topics\", []):\n for article in topic_result.get(\"articles\", []):\n article[\"source_type\"] = \"web\"\n article[\"source_name\"] = \"Web Search\"\n article[\"source_id\"] = f\"web-{topic_result.get('topic_id', '')}\"\n # Build a minimal source dict so web articles go through the same scoring\n web_source = {\n \"source_type\": \"web\",\n \"priority\": False,\n }\n article[\"quality_score\"] = calculate_base_score(article, web_source)\n all_articles.append(article)\n \n # Process GitHub articles\n for source in github_data.get(\"sources\", []):\n for article in source.get(\"articles\", []):\n article[\"source_type\"] = \"github\"\n article[\"source_name\"] = source.get(\"name\", \"\")\n article[\"source_id\"] = source.get(\"source_id\", \"\")\n article[\"quality_score\"] = calculate_base_score(article, source)\n all_articles.append(article)\n \n # Process Reddit articles\n for source in reddit_data.get(\"subreddits\", []):\n for article in source.get(\"articles\", []):\n article[\"source_type\"] = \"reddit\"\n article[\"source_name\"] = f\"r/{source.get('subreddit', '')}\"\n article[\"source_id\"] = source.get(\"source_id\", \"\")\n reddit_source = {\n \"source_type\": \"reddit\",\n \"priority\": source.get(\"priority\", False),\n }\n article[\"quality_score\"] = calculate_base_score(article, reddit_source)\n # Reddit score bonus\n score = article.get(\"score\", 0)\n if score > 500:\n article[\"quality_score\"] += 5\n elif score > 200:\n article[\"quality_score\"] += 3\n elif score > 100:\n article[\"quality_score\"] += 1\n all_articles.append(article)\n \n\n # Search discoveries keep the legacy source type, not a growth-based score.\n if trending_data:\n for repo in trending_data.get(\"repos\", []):\n article = {\n \"title\": f\"{repo['repo']}: {repo['description']}\" if repo.get('description') else repo['repo'],\n \"link\": repo.get(\"url\", f\"https://github.com/{repo['repo']}\"),\n \"snippet\": repo.get(\"description\", \"\"),\n \"date\": repo.get(\"pushed_at\", \"\"),\n \"source\": \"github-trending\",\n \"source_type\": \"github_trending\",\n \"topics\": repo.get(\"topics\", []),\n \"stars\": repo.get(\"stars\", 0),\n \"discovery_method\": \"github_search\",\n \"forks\": repo.get(\"forks\", 0),\n \"language\": repo.get(\"language\", \"\"),\n # Lifetime popularity and legacy estimates are not daily growth.\n \"quality_score\": 5,\n }\n all_articles.append(article)\n total_collected = len(all_articles)\n logger.info(f\"Total articles collected: {total_collected}\")\n \n # Load previous digest titles for penalty\n previous_titles = set()\n if args.archive_dir:\n previous_titles = load_previous_digests(args.archive_dir)\n \n # Apply previous digest penalty\n all_articles = apply_previous_digest_penalty(all_articles, previous_titles)\n\n # Deduplicate articles, folding each duplicate's provenance into the survivor\n all_articles = deduplicate_articles(all_articles)\n\n # Reward stories corroborated across source types\n all_articles = apply_multi_source_bonus(all_articles)\n\n # Group by topics (with cross-topic deduplication)\n topic_priority = build_topic_priority(load_topic_order(args.defaults, args.config))\n logger.debug(f\"Topic priority: {topic_priority}\")\n topic_groups = group_by_topics(all_articles, dedup_across_topics=True,\n topic_priority=topic_priority)\n \n # Apply per-topic domain limits (max 3 articles per domain per topic)\n for topic in topic_groups:\n before = len(topic_groups[topic])\n topic_groups[topic] = apply_domain_limits(topic_groups[topic])\n after = len(topic_groups[topic])\n if before != after:\n logger.info(f\"Domain limits ({topic}): {before} → {after}\")\n \n # Recalculate total after domain limits\n total_after_domain_limits = sum(len(articles) for articles in topic_groups.values())\n\n\n topic_counts = {topic: len(articles) for topic, articles in topic_groups.items()}\n \n output = {\n \"generated\": datetime.now(timezone.utc).isoformat(),\n # Mirrored at the top level so run-pipeline.py can report the count\n \"total_articles\": total_after_domain_limits,\n \"input_sources\": {\n \"rss_articles\": rss_data.get(\"total_articles\", 0),\n \"twitter_articles\": twitter_data.get(\"total_articles\", 0),\n \"web_articles\": web_data.get(\"total_articles\", 0),\n \"github_articles\": github_data.get(\"total_articles\", 0),\n \"github_trending\": trending_data.get(\"total\", 0) if trending_data else 0,\n \"reddit_posts\": reddit_data.get(\"total_posts\", 0),\n \"total_input\": total_collected\n },\n \"processing\": {\n \"deduplication_applied\": True,\n \"multi_source_merging\": True,\n \"previous_digest_penalty\": len(previous_titles) > 0,\n \"quality_scoring\": True\n },\n \"output_stats\": {\n \"total_articles\": total_after_domain_limits,\n \"topics_count\": len(topic_groups),\n \"topic_distribution\": topic_counts\n },\n \"topics\": {\n topic: {\n \"count\": len(articles),\n \"articles\": articles\n } for topic, articles in topic_groups.items()\n }\n }\n \n # Write output\n json_str = json.dumps(output, ensure_ascii=False, indent=2)\n with open(args.output, \"w\", encoding='utf-8') as f:\n f.write(json_str)\n \n logger.info(\"✅ Merged and scored articles:\")\n logger.info(f\" Input: {total_collected} articles\")\n logger.info(f\" Output: {total_after_domain_limits} articles across {len(topic_groups)} topics\")\n logger.info(f\" File: {args.output}\")\n \n return 0\n \n except Exception as e:\n logger.error(f\"💥 Merge failed: {e}\")\n return 1\n\n\nif __name__ == \"__main__\":\n sys.exit(main())","content_type":"text/x-python; charset=utf-8","language":"python","size":31406,"content_sha256":"d4c81746a6504b874f3cfb018544838e893a0554e4835d8651033cb3f611b6d1"},{"filename":"scripts/run-pipeline.py","content":"#!/usr/bin/env python3\n\"\"\"\nUnified data collection pipeline for tech-news-digest.\n\nRuns all 6 fetch steps (RSS, Twitter, GitHub, GitHub Trending, Reddit, Web) in parallel,\nthen merges + deduplicates + scores into a single output JSON.\n\nReplaces the agent's sequential 6-step tool-call loop with one command,\neliminating ~60-120s of LLM round-trip overhead.\n\nUsage:\n python3 run-pipeline.py \\\n --defaults \u003cSKILL_DIR>/config/defaults \\\n --config \u003cWORKSPACE>/config \\\n --hours 48 --freshness pd \\\n --archive-dir \u003cWORKSPACE>/archive/tech-news-digest/ \\\n --output /tmp/td-merged.json \\\n --verbose\n\"\"\"\n\nimport json\nimport sys\nimport os\nimport subprocess\nimport time\nimport argparse\nimport logging\nfrom concurrent.futures import ThreadPoolExecutor, as_completed\nfrom pathlib import Path\nfrom typing import Dict, Any\n\nSCRIPTS_DIR = Path(__file__).parent\nDEFAULT_TIMEOUT = 180 # per-step timeout in seconds\n\n\ndef setup_logging(verbose: bool) -> logging.Logger:\n level = logging.DEBUG if verbose else logging.INFO\n logging.basicConfig(\n level=level,\n format=\"%(asctime)s [%(levelname)s] %(message)s\",\n datefmt=\"%H:%M:%S\",\n )\n return logging.getLogger(__name__)\n\n\ndef run_step(\n name: str,\n script: str,\n args_list: list,\n output_path: Path,\n timeout: int = DEFAULT_TIMEOUT,\n force: bool = False,\n) -> Dict[str, Any]:\n \"\"\"Run a fetch script as a subprocess, return result metadata.\"\"\"\n t0 = time.time()\n cmd = [sys.executable, str(SCRIPTS_DIR / script)] + args_list + [\n \"--output\", str(output_path),\n ]\n if force:\n cmd.append(\"--force\")\n\n try:\n result = subprocess.run(\n cmd,\n capture_output=True,\n text=True,\n timeout=timeout,\n env=os.environ,\n )\n elapsed = time.time() - t0\n ok = result.returncode == 0\n\n # Try to read output stats\n count = 0\n if ok and output_path.exists():\n try:\n with open(output_path) as f:\n data = json.load(f)\n count = (\n data.get(\"total_articles\")\n or data.get(\"total_posts\")\n or data.get(\"total_releases\")\n or data.get(\"total_results\")\n or data.get(\"total\")\n or 0\n )\n except (json.JSONDecodeError, OSError):\n pass\n\n return {\n \"name\": name,\n \"status\": \"ok\" if ok else \"error\",\n \"elapsed_s\": round(elapsed, 1),\n \"count\": count,\n \"stderr_tail\": (result.stderr or \"\").strip().split(\"\\n\")[-3:] if not ok else [],\n }\n\n except subprocess.TimeoutExpired:\n elapsed = time.time() - t0\n return {\n \"name\": name,\n \"status\": \"timeout\",\n \"elapsed_s\": round(elapsed, 1),\n \"count\": 0,\n \"stderr_tail\": [f\"Killed after {timeout}s\"],\n }\n except Exception as e:\n elapsed = time.time() - t0\n return {\n \"name\": name,\n \"status\": \"error\",\n \"elapsed_s\": round(elapsed, 1),\n \"count\": 0,\n \"stderr_tail\": [str(e)],\n }\n\n\ndef main() -> int:\n parser = argparse.ArgumentParser(\n description=\"Run the full tech-news-digest data pipeline in one shot.\",\n formatter_class=argparse.RawDescriptionHelpFormatter,\n )\n _script_dir = Path(__file__).resolve().parent\n _default_defaults = _script_dir.parent / \"config\" / \"defaults\"\n parser.add_argument(\"--defaults\", type=Path, default=_default_defaults, help=\"Skill defaults config dir (default: \u003cskill>/config/defaults)\")\n parser.add_argument(\"--config\", type=Path, default=None, help=\"User config overlay dir\")\n parser.add_argument(\"--hours\", type=int, default=48, help=\"Time window in hours\")\n parser.add_argument(\"--freshness\", type=str, default=\"pd\", help=\"Web search freshness (pd/pw/pm)\")\n parser.add_argument(\"--archive-dir\", type=Path, default=None, help=\"Archive dir for dedup penalty\")\n parser.add_argument(\"--output\", \"-o\", type=Path, default=Path(\"/tmp/td-merged.json\"), help=\"Final merged output\")\n parser.add_argument(\"--step-timeout\", type=int, default=DEFAULT_TIMEOUT, help=\"Per-step timeout (seconds)\")\n parser.add_argument(\"--twitter-backend\", choices=[\"official\", \"twitterapiio\", \"getxapi\", \"auto\"], default=None, help=\"Twitter API backend to use\")\n parser.add_argument(\"--verbose\", \"-v\", action=\"store_true\")\n parser.add_argument(\"--force\", action=\"store_true\", help=\"Force re-fetch ignoring caches\")\n parser.add_argument(\"--enrich\", action=\"store_true\", help=\"Enable full-text enrichment for top articles\")\n parser.add_argument(\"--skip\", type=str, default=\"\", help=\"Comma-separated list of steps to skip (rss,twitter,github,trending,reddit,web)\")\n parser.add_argument(\"--only\", type=str, default=\"\", help=\"Comma-separated list of steps to run (rss,twitter,github,trending,reddit,web). Others are skipped.\")\n parser.add_argument(\"--reuse-dir\", type=Path, default=None, help=\"Reuse existing intermediate directory instead of creating new one\")\n parser.add_argument(\"--debug\", action=\"store_true\", help=\"Keep intermediate fetch outputs (rss.json, twitter.json, etc.) alongside final output\")\n\n args = parser.parse_args()\n logger = setup_logging(args.verbose)\n\n # Parse --skip and --only into sets of canonical step keys.\n # Keys are matched exactly, so \"trending\" must be its own key rather than a\n # substring of the display name \"GitHub Trending\".\n all_step_keys = {\"rss\", \"twitter\", \"github\", \"trending\", \"reddit\", \"web\"}\n\n def parse_steps(value: str, flag: str) -> set:\n requested = set(s.strip().lower() for s in value.split(',') if s.strip())\n unknown = requested - all_step_keys - {\"enrich\"}\n if unknown:\n logger.warning(f\"⚠️ {flag}: unknown step(s) {sorted(unknown)}; valid: {sorted(all_step_keys)}\")\n return requested\n\n skip_steps = parse_steps(args.skip, \"--skip\")\n only_steps = parse_steps(args.only, \"--only\")\n\n # --only takes precedence: skip everything not in the list\n if only_steps:\n skip_steps = all_step_keys - only_steps\n logger.info(f\"🎯 --only {args.only} → running: {sorted(all_step_keys - skip_steps)}\")\n\n # Intermediate output paths\n import tempfile\n if args.reuse_dir:\n _run_dir = str(args.reuse_dir)\n os.makedirs(_run_dir, exist_ok=True)\n else:\n _run_dir = tempfile.mkdtemp(prefix=\"td-pipeline-\")\n tmp_rss = Path(_run_dir) / \"rss.json\"\n tmp_twitter = Path(_run_dir) / \"twitter.json\"\n tmp_github = Path(_run_dir) / \"github.json\"\n tmp_trending = Path(_run_dir) / \"trending.json\"\n tmp_reddit = Path(_run_dir) / \"reddit.json\"\n tmp_web = Path(_run_dir) / \"web.json\"\n logger.info(f\"📁 Run directory: {_run_dir}\")\n\n # Common args for all fetch scripts\n common = [\"--defaults\", str(args.defaults)]\n if args.config:\n common += [\"--config\", str(args.config)]\n common += [\"--hours\", str(args.hours)]\n verbose_flag = [\"--verbose\"] if args.verbose else []\n\n # Define the 6 parallel fetch steps as (key, display name, script, args, output)\n steps = [\n (\"rss\", \"RSS\", \"fetch-rss.py\", common + verbose_flag, tmp_rss),\n (\"twitter\", \"Twitter\", \"fetch-twitter.py\", common + verbose_flag + ([\"--backend\", args.twitter_backend] if args.twitter_backend else []), tmp_twitter),\n (\"github\", \"GitHub\", \"fetch-github.py\", common + verbose_flag, tmp_github),\n (\"trending\", \"GitHub Trending\", \"fetch-github.py\", [\"--trending\", \"--hours\", str(args.hours)] + verbose_flag, tmp_trending),\n (\"reddit\", \"Reddit\", \"fetch-reddit.py\", common + verbose_flag, tmp_reddit),\n (\"web\", \"Web\", \"fetch-web.py\",\n [\"--defaults\", str(args.defaults)]\n + ([\"--config\", str(args.config)] if args.config else [])\n + [\"--freshness\", args.freshness]\n + verbose_flag,\n tmp_web),\n ]\n\n # Filter steps by --skip and --reuse-dir\n active_steps = []\n for step_key, name, script, step_args, out_path in steps:\n if step_key in skip_steps:\n logger.info(f\" ⏭️ {name}: skipped (--skip)\")\n continue\n if args.reuse_dir and out_path.exists() and not args.force:\n logger.info(f\" ♻️ {name}: reusing existing {out_path}\")\n continue\n active_steps.append((name, script, step_args, out_path))\n\n logger.info(f\"🚀 Starting pipeline: {len(active_steps)}/{len(steps)} sources, {args.hours}h window, freshness={args.freshness}\")\n t_start = time.time()\n\n # Phase 1: Parallel fetch\n step_results = []\n if active_steps:\n with ThreadPoolExecutor(max_workers=len(active_steps)) as pool:\n futures = {}\n for name, script, step_args, out_path in active_steps:\n f = pool.submit(run_step, name, script, step_args, out_path, args.step_timeout, args.force)\n futures[f] = name\n\n for future in as_completed(futures):\n res = future.result()\n step_results.append(res)\n status_icon = {\"ok\": \"✅\", \"error\": \"❌\", \"timeout\": \"⏰\"}.get(res[\"status\"], \"?\")\n logger.info(f\" {status_icon} {res['name']}: {res['count']} items ({res['elapsed_s']}s)\")\n if res[\"status\"] != \"ok\" and res[\"stderr_tail\"]:\n for line in res[\"stderr_tail\"]:\n logger.debug(f\" {line}\")\n\n fetch_elapsed = time.time() - t_start\n logger.info(f\"📡 Fetch phase done in {fetch_elapsed:.1f}s\")\n\n # Phase 2: Merge\n logger.info(\"🔀 Merging & scoring...\")\n merge_args = [\"--verbose\"] if args.verbose else []\n for flag, path in [(\"--rss\", tmp_rss), (\"--twitter\", tmp_twitter),\n (\"--github\", tmp_github), (\"--trending\", tmp_trending), (\"--reddit\", tmp_reddit),\n (\"--web\", tmp_web)]:\n if path.exists():\n merge_args += [flag, str(path)]\n if args.archive_dir:\n merge_args += [\"--archive-dir\", str(args.archive_dir)]\n # Topic priority order is derived from topics.json, so merge needs the config dirs\n merge_args += [\"--defaults\", str(args.defaults)]\n if args.config:\n merge_args += [\"--config\", str(args.config)]\n\n merge_result = run_step(\"Merge\", \"merge-sources.py\", merge_args, args.output, timeout=60, force=False)\n\n # Phase 3: Enrich high-scoring articles with full text\n if merge_result[\"status\"] == \"ok\" and args.enrich and \"enrich\" not in skip_steps:\n logger.info(\"📰 Enriching top articles with full text...\")\n enrich_args = [\"--input\", str(args.output), \"--output\", str(args.output)]\n enrich_args += [\"--verbose\"] if args.verbose else []\n enrich_result = run_step(\"Enrich\", \"enrich-articles.py\", enrich_args, args.output, timeout=120, force=False)\n else:\n enrich_result = {\"name\": \"Enrich\", \"status\": \"skipped\", \"elapsed_s\": 0, \"count\": 0, \"stderr_tail\": []}\n\n # Phase 4: Record source health from the fetch outputs\n health_output = Path(_run_dir) / \"health.json\"\n health_args = []\n for flag, path in [(\"--rss\", tmp_rss), (\"--twitter\", tmp_twitter), (\"--github\", tmp_github), (\"--reddit\", tmp_reddit), (\"--web\", tmp_web)]:\n if path.exists():\n health_args += [flag, str(path)]\n health_args += [\"--output\", str(health_output)]\n health_args += [\"--verbose\"] if args.verbose else []\n health_result = run_step(\"Health\", \"source-health.py\", health_args, health_output, timeout=30, force=False)\n\n health_summary = None\n if health_result[\"status\"] == \"ok\" and health_output.exists():\n try:\n with open(health_output) as f:\n health_summary = json.load(f)\n except (json.JSONDecodeError, OSError):\n health_summary = None\n\n total_elapsed = time.time() - t_start\n\n # Summary\n logger.info(f\"{'=' * 50}\")\n logger.info(f\"📊 Pipeline Summary ({total_elapsed:.1f}s total)\")\n for r in step_results:\n logger.info(f\" {r['name']:10s} {r['status']:7s} {r['count']:4d} items {r['elapsed_s']:5.1f}s\")\n logger.info(f\" {'Merge':10s} {merge_result['status']:7s} {merge_result.get('count',0):4d} items {merge_result['elapsed_s']:5.1f}s\")\n logger.info(f\" {'Health':10s} {health_result['status']:7s} {health_summary.get('tracked_sources', 0) if health_summary else 0:4d} tracked {health_result['elapsed_s']:5.1f}s\")\n logger.info(f\" Output: {args.output}\")\n\n if merge_result[\"status\"] != \"ok\":\n logger.error(f\"❌ Merge failed: {merge_result['stderr_tail']}\")\n\n # Write pipeline metadata alongside output for agent consumption\n meta = {\n \"pipeline_version\": \"1.1.0\",\n \"total_elapsed_s\": round(total_elapsed, 1),\n \"fetch_elapsed_s\": round(fetch_elapsed, 1),\n \"steps\": step_results,\n \"merge\": merge_result,\n \"enrich\": enrich_result,\n \"health\": health_result,\n \"health_summary\": health_summary,\n \"output\": str(args.output),\n }\n meta_path = args.output.with_suffix(\".meta.json\")\n with open(meta_path, \"w\") as f:\n json.dump(meta, f, indent=2)\n\n if args.debug:\n # Copy intermediate files next to final output for inspection\n import shutil\n debug_dir = args.output.parent / f\"{args.output.stem}-debug\"\n debug_dir.mkdir(parents=True, exist_ok=True)\n for fname in os.listdir(_run_dir):\n src = Path(_run_dir) / fname\n if src.is_file():\n shutil.copy2(str(src), str(debug_dir / fname))\n logger.info(f\"🔍 Debug: intermediate files saved to {debug_dir}\")\n meta[\"debug_dir\"] = str(debug_dir)\n # Rewrite meta with debug_dir\n with open(meta_path, \"w\") as f:\n json.dump(meta, f, indent=2)\n\n if not args.reuse_dir:\n import shutil\n try:\n shutil.rmtree(_run_dir)\n logger.debug(f\"Cleaned up {_run_dir}\")\n except Exception:\n pass\n\n logger.info(f\"✅ Done → {args.output}\")\n\n failed_steps = [r for r in step_results if r[\"status\"] != \"ok\"]\n if merge_result[\"status\"] != \"ok\":\n return 1\n if failed_steps:\n logger.error(\"❌ Fetch failed for: %s\", \", \".join(r[\"name\"] for r in failed_steps))\n return 1\n return 0\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n","content_type":"text/x-python; charset=utf-8","language":"python","size":14534,"content_sha256":"28ca57915cfad4bdc77a1ecf584cfe1eb14564bcd1c3c197635af860aa44d44d"},{"filename":"scripts/sanitize-html.py","content":"#!/usr/bin/env python3\n\"\"\"\nSanitize media digest markdown report into safe HTML email.\n\nReads a markdown report file, escapes all text content to prevent XSS,\nand outputs a styled HTML email body safe for injection into email clients.\n\nUsage:\n python3 sanitize-html.py --input /tmp/md-report.md --output /tmp/md-email.html [--verbose]\n\nSecurity:\n - All text content is HTML-escaped (prevents XSS from malicious RSS/Twitter/web content)\n - Only whitelisted tags/attributes are allowed\n - URLs are validated (must be http/https)\n - No JavaScript, event handlers, or data: URIs allowed\n\"\"\"\n\nimport argparse\nimport html\nimport re\nimport sys\nimport logging\nfrom urllib.parse import urlparse\n\n\ndef escape(text: str) -> str:\n \"\"\"HTML-escape text content.\"\"\"\n return html.escape(text, quote=True)\n\n\ndef is_safe_url(url: str) -> bool:\n \"\"\"Validate URL is http(s) only — no javascript:, data:, etc.\"\"\"\n try:\n parsed = urlparse(url.strip())\n return parsed.scheme in ('http', 'https')\n except Exception:\n return False\n\n\ndef safe_link(url: str, label: str = None, style: str = \"color:#0969da;font-size:13px\") -> str:\n \"\"\"Generate a safe HTML link with escaped content.\"\"\"\n url = url.strip()\n if not is_safe_url(url):\n return escape(label or url)\n escaped_url = escape(url)\n escaped_label = escape(label or url)\n return f'\u003ca href=\"{escaped_url}\" style=\"{style}\">{escaped_label}\u003c/a>'\n\n\ndef markdown_to_safe_html(md_content: str) -> str:\n \"\"\"Convert markdown digest report to sanitized HTML email.\"\"\"\n lines = md_content.strip().split('\\n')\n html_parts = []\n \n # Email wrapper open\n html_parts.append(\n '\u003cdiv style=\"max-width:640px;margin:0 auto;font-family:'\n '-apple-system,BlinkMacSystemFont,\\'Segoe UI\\',Roboto,sans-serif;'\n 'color:#1a1a1a;line-height:1.6\">'\n )\n \n in_list = False\n \n for line in lines:\n stripped = line.strip()\n \n # Skip empty lines\n if not stripped:\n if in_list:\n html_parts.append('\u003c/ul>')\n in_list = False\n continue\n \n # H1: # Title\n if stripped.startswith('# '):\n title = escape(stripped[2:])\n html_parts.append(\n f'\u003ch1 style=\"font-size:22px;border-bottom:2px solid #e5e5e5;'\n f'padding-bottom:8px\">{title}\u003c/h1>'\n )\n continue\n \n # H2: ## Section\n if stripped.startswith('## '):\n if in_list:\n html_parts.append('\u003c/ul>')\n in_list = False\n section = escape(stripped[3:])\n html_parts.append(\n f'\u003ch2 style=\"font-size:17px;margin-top:24px;color:#333\">{section}\u003c/h2>'\n )\n continue\n \n # Blockquote: > executive summary\n if stripped.startswith('> '):\n text = escape(stripped[2:])\n html_parts.append(\n f'\u003cp style=\"color:#555;font-size:14px;background:#f8f9fa;'\n f'padding:12px;border-radius:6px\">{text}\u003c/p>'\n )\n continue\n \n # Horizontal rule\n if stripped == '---':\n html_parts.append('\u003chr style=\"border:none;border-top:1px solid #e5e5e5;margin:24px 0\">')\n continue\n \n # Bullet items: • or -\n if stripped.startswith('• ') or stripped.startswith('- '):\n if not in_list:\n html_parts.append('\u003cul style=\"padding-left:20px\">')\n in_list = True\n item_text = stripped[2:]\n safe_item = _process_inline(item_text)\n html_parts.append(f'\u003cli style=\"margin-bottom:10px\">{safe_item}\u003c/li>')\n continue\n \n # Continuation of bullet (indented line with link)\n if stripped.startswith('\u003chttp') and in_list:\n url = stripped.strip('\u003c> ')\n link = safe_link(url)\n html_parts.append(f'\u003cli style=\"margin-bottom:2px;list-style:none\">{link}\u003c/li>')\n continue\n \n # Stats/footer line\n if stripped.startswith('📊') or stripped.startswith('🤖'):\n text = _process_inline(stripped)\n html_parts.append(f'\u003cp style=\"font-size:12px;color:#888\">{text}\u003c/p>')\n continue\n \n # Regular paragraph\n text = _process_inline(stripped)\n html_parts.append(f'\u003cp>{text}\u003c/p>')\n \n if in_list:\n html_parts.append('\u003c/ul>')\n \n html_parts.append('\u003c/div>')\n return '\\n'.join(html_parts)\n\n\ndef _process_inline(text: str) -> str:\n \"\"\"Render raw markdown tokens once, escaping every text/attribute value.\n\n Consume labelled links before angle autolinks so Discord's [label](\u003curl>)\n cannot be rewritten into HTML and then mistaken for an unsafe URL. Never\n run markdown substitutions over generated HTML (or over code/URL content).\n \"\"\"\n tokens = re.compile(\n r'`(?P\u003ccode>[^`]+)`'\n r'|\\[(?P\u003clabel>[^\\]\\n]+)\\]\\('\n r'(?:\u003c(?P\u003csuppressed>[^\u003c>\\n]+)>|(?P\u003curl>(?:[^()\\n]|\\([^()\\n]*\\))+))\\)'\n r'|\u003c(?P\u003cangle>https?://[^\u003c>\\n]+)>'\n r'|\\*\\*(?P\u003cbold>.+?)\\*\\*'\n )\n parts = []\n position = 0\n for match in tokens.finditer(text):\n parts.append(escape(text[position:match.start()]))\n if match.group('code') is not None:\n parts.append('\u003ccode style=\"font-size:12px;color:#888;background:#f4f4f4;padding:2px 6px;border-radius:3px\">' + escape(match.group('code')) + '\u003c/code>')\n elif match.group('bold') is not None:\n parts.append('\u003cstrong>' + _process_inline(match.group('bold')) + '\u003c/strong>')\n else:\n url = (match.group('suppressed') or match.group('url') or match.group('angle')).strip()\n label = match.group('label')\n if is_safe_url(url):\n if label is None:\n label = urlparse(url).netloc or url\n parts.append(f'\u003ca href=\"{escape(url)}\" style=\"color:#0969da;font-size:13px\">{escape(label)}\u003c/a>')\n else:\n parts.append(escape(label if label is not None else url))\n position = match.end()\n parts.append(escape(text[position:]))\n result = ''.join(parts)\n return result\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"Convert markdown digest to sanitized HTML email\"\n )\n parser.add_argument(\"--input\", \"-i\", required=True, help=\"Input markdown file\")\n parser.add_argument(\"--output\", \"-o\", required=True, help=\"Output HTML file\")\n parser.add_argument(\"--verbose\", \"-v\", action=\"store_true\")\n args = parser.parse_args()\n \n logging.basicConfig(\n level=logging.DEBUG if args.verbose else logging.INFO,\n format=\"%(levelname)s: %(message)s\"\n )\n \n try:\n with open(args.input, 'r') as f:\n md_content = f.read()\n except FileNotFoundError:\n logging.error(f\"Input file not found: {args.input}\")\n sys.exit(1)\n \n logging.info(f\"Converting {args.input} ({len(md_content)} chars)\")\n \n html_output = markdown_to_safe_html(md_content)\n \n with open(args.output, 'w') as f:\n f.write(html_output)\n \n logging.info(f\"Wrote sanitized HTML to {args.output} ({len(html_output)} chars)\")\n\n\nif __name__ == \"__main__\":\n main()\n","content_type":"text/x-python; charset=utf-8","language":"python","size":7332,"content_sha256":"dc11098870c03cc547938871fd891911dd04f347f1eeeb62ce1c34c93960ca23"},{"filename":"scripts/send-email.py","content":"#!/usr/bin/env python3\n\"\"\"\nSend HTML email with optional PDF attachment via msmtp or sendmail.\n\nProperly constructs MIME multipart message so HTML body renders correctly\neven when attachments are included.\n\nUsage:\n python3 send-email.py --to [email protected] --subject \"Daily Digest\" \\\n --html /tmp/td-email.html [--attach /tmp/td-digest.pdf] [--from \"Bot \[email protected]>\"]\n\"\"\"\n\nimport argparse\nimport subprocess\nimport sys\nimport logging\nfrom email.mime.multipart import MIMEMultipart\nfrom email.mime.text import MIMEText\nimport re\nimport html as html_lib\nfrom email.mime.application import MIMEApplication\nfrom email.utils import formatdate\nfrom pathlib import Path\n\n\ndef _html_to_plain_text(html_content: str) -> str:\n \"\"\"Best-effort plain-text fallback for clients that do not render HTML.\"\"\"\n text = re.sub(r\"(?i)\u003cbr\\s*/?>\", \"\\n\", html_content)\n text = re.sub(r\"(?i)\u003c/(p|div|h[1-6]|li|ul|ol)>\", \"\\n\", text)\n text = re.sub(r\"\u003c[^>]+>\", \"\", text)\n text = html_lib.unescape(text)\n text = re.sub(r\"\\n{3,}\", \"\\n\\n\", text)\n return text.strip() or \"HTML email body attached.\"\n\n\ndef build_message(subject: str, from_addr: str, to_addrs: list,\n html_path: Path, attach_path: Path = None) -> str:\n \"\"\"Build a standards-compliant MIME message.\n\n Structure:\n multipart/mixed\n multipart/alternative\n text/plain\n text/html\n application/pdf (optional)\n\n Nesting text/html inside multipart/alternative avoids clients treating the\n HTML body as a raw attachment or displaying markup as plain text when a PDF\n attachment is also present.\n \"\"\"\n html_content = html_path.read_text(encoding='utf-8')\n plain_content = _html_to_plain_text(html_content)\n\n msg = MIMEMultipart('mixed')\n alt = MIMEMultipart('alternative')\n alt.attach(MIMEText(plain_content, 'plain', 'utf-8'))\n alt.attach(MIMEText(html_content, 'html', 'utf-8'))\n msg.attach(alt)\n\n if attach_path and attach_path.exists():\n pdf_data = attach_path.read_bytes()\n pdf_part = MIMEApplication(pdf_data, _subtype='pdf')\n pdf_part.add_header('Content-Disposition', 'attachment',\n filename=attach_path.name)\n msg.attach(pdf_part)\n\n msg['Subject'] = subject\n msg['From'] = from_addr\n msg['To'] = ', '.join(to_addrs)\n msg['Date'] = formatdate(localtime=True)\n msg['X-Mailer'] = 'Hermes Agent digest send-email.py'\n\n return msg.as_string()\n\n\ndef send_via_msmtp(message: str, to_addrs: list) -> bool:\n \"\"\"Send via msmtp (preferred).\"\"\"\n try:\n result = subprocess.run(\n ['msmtp', '--read-envelope-from'] + to_addrs,\n input=message.encode('utf-8'),\n capture_output=True,\n timeout=30\n )\n if result.returncode == 0:\n return True\n logging.error(f\"msmtp failed: {result.stderr.decode()}\")\n return False\n except FileNotFoundError:\n logging.debug(\"msmtp not found\")\n return False\n except Exception as e:\n logging.error(f\"msmtp error: {e}\")\n return False\n\n\ndef send_via_sendmail(message: str, to_addrs: list) -> bool:\n \"\"\"Send via sendmail (fallback).\"\"\"\n for cmd in ['sendmail', '/usr/sbin/sendmail']:\n try:\n result = subprocess.run(\n [cmd, '-t'] + to_addrs,\n input=message.encode('utf-8'),\n capture_output=True,\n timeout=30\n )\n if result.returncode == 0:\n return True\n logging.error(f\"{cmd} failed: {result.stderr.decode()}\")\n except FileNotFoundError:\n continue\n except Exception as e:\n logging.error(f\"{cmd} error: {e}\")\n return False\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"Send HTML email with optional PDF attachment\",\n formatter_class=argparse.RawDescriptionHelpFormatter,\n epilog=\"\"\"\\\nExamples:\n python3 send-email.py --to [email protected] --subject \"Daily Digest\" --html /tmp/td-email.html\n python3 send-email.py --to [email protected] --to [email protected] --subject \"Weekly\" --html body.html --attach digest.pdf\n python3 send-email.py --to [email protected] --subject \"Test\" --html body.html --from \"Bot \[email protected]>\"\n\"\"\"\n )\n parser.add_argument('--to', action='append', required=True, help='Recipient email (repeatable)')\n parser.add_argument('--subject', '-s', required=True, help='Email subject')\n parser.add_argument('--html', required=True, type=Path, help='HTML body file')\n parser.add_argument('--attach', type=Path, default=None, help='PDF attachment file')\n parser.add_argument('--from', dest='from_addr', default=None, help='From address')\n parser.add_argument('--verbose', '-v', action='store_true')\n args = parser.parse_args()\n \n logging.basicConfig(\n level=logging.DEBUG if args.verbose else logging.INFO,\n format=\"%(levelname)s: %(message)s\"\n )\n \n if not args.html.exists():\n logging.error(f\"HTML file not found: {args.html}\")\n sys.exit(1)\n \n # Expand comma-separated addresses\n to_addrs = []\n for addr in args.to:\n to_addrs.extend([a.strip() for a in addr.split(',') if a.strip()])\n \n from_addr = args.from_addr or 'noreply@localhost'\n \n logging.info(f\"Building email: {args.subject} → {', '.join(to_addrs)}\")\n if args.attach:\n logging.info(f\"Attachment: {args.attach} ({'exists' if args.attach.exists() else 'MISSING'})\")\n \n message = build_message(args.subject, from_addr, to_addrs, args.html, args.attach)\n \n # Try msmtp first, then sendmail\n if send_via_msmtp(message, to_addrs):\n logging.info(\"✅ Sent via msmtp\")\n return 0\n \n if send_via_sendmail(message, to_addrs):\n logging.info(\"✅ Sent via sendmail\")\n return 0\n \n logging.error(\"❌ All send methods failed\")\n return 1\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n","content_type":"text/x-python; charset=utf-8","language":"python","size":5982,"content_sha256":"9d499d522eadd3c6ddb6b97afb11a8c42176b0d38dd45b004850b9814d4f84e8"},{"filename":"scripts/source-health.py","content":"#!/usr/bin/env python3\n\"\"\"\nSource health monitoring for tech-news-digest pipeline.\n\nTracks per-source success/failure history and reports unhealthy sources.\n\nUsage:\n python3 source-health.py --rss rss.json --twitter twitter.json --github github.json [--output health.json]\n\"\"\"\n\nimport json\nimport sys\nimport argparse\nimport logging\nimport time\nfrom pathlib import Path\nfrom typing import Dict, Any, Optional\nfrom datetime import datetime, timezone\n\ntry:\n from config_loader import get_state_path\nexcept ImportError:\n sys.path.append(str(Path(__file__).resolve().parent))\n from config_loader import get_state_path\n\n# Health history spans several days and needs at least 2 samples before it can\n# flag anything, so it must outlive /tmp.\nHEALTH_FILE = str(get_state_path(\"source-health.json\"))\nDEFAULT_HEALTH_FILE = HEALTH_FILE\nLEGACY_HEALTH_FILE = \"/tmp/tech-news-digest-source-health.json\"\nHISTORY_DAYS = 7\nFAILURE_THRESHOLD = 0.5 # >50% failure rate triggers warning\nSKIPPED_STATUSES = {\"interface\", \"skipped\", \"no_credentials\"}\n\n\ndef setup_logging(verbose: bool) -> logging.Logger:\n level = logging.DEBUG if verbose else logging.INFO\n logging.basicConfig(level=level, format='%(asctime)s - %(levelname)s - %(message)s')\n return logging.getLogger(__name__)\n\n\ndef load_health_data() -> Dict[str, Any]:\n # The legacy /tmp store is only consulted when HEALTH_FILE is the default —\n # a caller that redirects HEALTH_FILE (tests, sandboxes) must stay isolated.\n candidates = [HEALTH_FILE]\n if HEALTH_FILE == DEFAULT_HEALTH_FILE:\n candidates.append(LEGACY_HEALTH_FILE)\n for path in candidates:\n try:\n with open(path, 'r') as f:\n return json.load(f)\n except (FileNotFoundError, json.JSONDecodeError):\n continue\n return {}\n\n\ndef save_health_data(data: Dict[str, Any]) -> None:\n with open(HEALTH_FILE, 'w') as f:\n json.dump(data, f, indent=2)\n\n\ndef load_source_file(path: Optional[Path]) -> list:\n if not path or not path.exists():\n return []\n try:\n with open(path, 'r') as f:\n data = json.load(f)\n return data.get(\"sources\", [])\n except (json.JSONDecodeError, OSError):\n return []\n\n\ndef load_source_file_flexible(path: Optional[Path]) -> list:\n \"\"\"Load sources from a JSON file, trying 'sources', 'subreddits', and 'topics' keys.\"\"\"\n if not path or not path.exists():\n return []\n try:\n with open(path, 'r') as f:\n data = json.load(f)\n # Try standard keys\n if \"sources\" in data:\n return data[\"sources\"]\n if \"subreddits\" in data:\n return data[\"subreddits\"]\n if \"topics\" in data:\n # Create synthetic sources from topic results\n synthetic = []\n for topic in data[\"topics\"]:\n synthetic.append({\n \"source_id\": f\"web-{topic.get('topic_id', 'unknown')}\",\n \"name\": f\"Web: {topic.get('topic_id', 'unknown')}\",\n \"status\": topic.get(\"status\", \"ok\"),\n \"articles\": topic.get(\"articles\", []),\n })\n return synthetic\n return []\n except (json.JSONDecodeError, OSError):\n return []\n\n\ndef update_health(health: Dict[str, Any], sources: list, now: float) -> None:\n cutoff = now - HISTORY_DAYS * 86400\n for source in sources:\n sid = source.get(\"source_id\", source.get(\"id\", \"unknown\"))\n if sid not in health:\n health[sid] = {\"name\": source.get(\"name\", sid), \"checks\": []}\n # Prune old entries\n health[sid][\"checks\"] = [c for c in health[sid][\"checks\"] if c[\"ts\"] > cutoff]\n status = source.get(\"status\")\n ok = None if status in SKIPPED_STATUSES else status == \"ok\"\n health[sid][\"checks\"].append({\n \"ts\": now,\n \"ok\": ok,\n })\n\n\ndef report_unhealthy(health: Dict[str, Any], logger: logging.Logger) -> int:\n unhealthy = 0\n for sid, info in health.items():\n checks = info.get(\"checks\", [])\n checked = [c for c in checks if c.get(\"ok\") is not None]\n if len(checked) \u003c 2:\n continue\n failures = sum(1 for c in checked if not c[\"ok\"])\n rate = failures / len(checked)\n if rate > FAILURE_THRESHOLD:\n logger.warning(f\"⚠️ Unhealthy source: {info.get('name', sid)} \"\n f\"({failures}/{len(checked)} failures, {rate:.0%} failure rate)\")\n unhealthy += 1\n return unhealthy\n\n\ndef main():\n parser = argparse.ArgumentParser(description=\"Track source health for tech-news-digest pipeline.\")\n parser.add_argument(\"--rss\", type=Path, help=\"RSS output JSON\")\n parser.add_argument(\"--twitter\", type=Path, help=\"Twitter output JSON\")\n parser.add_argument(\"--github\", type=Path, help=\"GitHub output JSON\")\n parser.add_argument(\"--reddit\", type=Path, help=\"Reddit output JSON\")\n parser.add_argument(\"--web\", type=Path, help=\"Web search output JSON\")\n parser.add_argument(\"--output\", type=Path, help=\"Optional JSON summary output path\")\n parser.add_argument(\"--verbose\", \"-v\", action=\"store_true\")\n args = parser.parse_args()\n\n logger = setup_logging(args.verbose)\n health = load_health_data()\n now = time.time()\n processed_inputs = []\n\n # Standard sources (use 'sources' key)\n for label, path in [(\"rss\", args.rss), (\"twitter\", args.twitter), (\"github\", args.github)]:\n sources = load_source_file(path)\n if sources:\n update_health(health, sources, now)\n processed_inputs.append({\"name\": label, \"path\": str(path), \"count\": len(sources)})\n\n # Reddit and Web use flexible loading (subreddits/topics keys)\n for label, path in [(\"reddit\", args.reddit), (\"web\", args.web)]:\n sources = load_source_file_flexible(path)\n if sources:\n update_health(health, sources, now)\n processed_inputs.append({\"name\": label, \"path\": str(path), \"count\": len(sources)})\n\n save_health_data(health)\n unhealthy = report_unhealthy(health, logger)\n\n total = len(health)\n summary = {\n \"status\": \"ok\",\n \"tracked_sources\": total,\n \"unhealthy_sources\": unhealthy,\n \"inputs\": processed_inputs,\n \"health_file\": HEALTH_FILE,\n \"checked_at\": datetime.fromtimestamp(now, timezone.utc).isoformat(),\n }\n\n if args.output:\n with open(args.output, 'w') as f:\n json.dump(summary, f, indent=2)\n\n logger.info(f\"📊 Health check: {total} sources tracked, {unhealthy} unhealthy\")\n return 0\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n","content_type":"text/x-python; charset=utf-8","language":"python","size":6655,"content_sha256":"0cdaf49bd3821760a35a307c73ecaa5c2bd2c164d32042733649e17b0669f085"},{"filename":"scripts/summarize-merged.py","content":"#!/usr/bin/env python3\n\"\"\"\nPrint a human-readable summary of merged JSON data for LLM consumption.\n\nUsage:\n python3 summarize-merged.py [--input /tmp/td-merged.json] [--top N] [--topic TOPIC]\n\"\"\"\n\nimport json\nimport argparse\nfrom pathlib import Path\n\n\ndef summarize(data: dict, top_n: int = 10, topic_filter: str = None):\n \"\"\"Print structured summary of merged data.\"\"\"\n \n # Metadata\n meta = data.get(\"output_stats\", {})\n print(\"=== Merged Data Summary ===\")\n print(f\"Total articles: {meta.get('total_articles', '?')}\")\n print(f\"Topics: {', '.join(data.get('topics', {}).keys())}\")\n print()\n \n topics = data.get(\"topics\", {})\n \n for topic_id, topic_data in topics.items():\n if topic_filter and topic_id != topic_filter:\n continue\n \n articles = topic_data.get(\"articles\", [])\n if not isinstance(articles, list):\n continue\n \n print(f\"=== {topic_id} ({len(articles)} articles) ===\")\n \n # Sort by quality_score descending\n sorted_articles = sorted(\n [a for a in articles if isinstance(a, dict)],\n key=lambda a: a.get(\"quality_score\", 0),\n reverse=True\n )\n \n for i, a in enumerate(sorted_articles[:top_n]):\n title = a.get(\"title\", \"?\")[:100]\n source = a.get(\"source_name\", \"?\")\n source_type = a.get(\"source_type\", \"?\")\n qs = a.get(\"quality_score\", 0)\n link = a.get(\"link\") or a.get(\"reddit_url\") or a.get(\"external_url\", \"\")\n snippet = (a.get(\"snippet\") or a.get(\"summary\") or \"\")[:150]\n \n # Metrics for Twitter\n metrics = a.get(\"metrics\", {})\n display_name = a.get(\"display_name\", \"\")\n \n print(f\"\\n [{i+1}] ({qs:.0f}pts) [{source_type}] {title}\")\n print(f\" Source: {source}\", end=\"\")\n if display_name:\n print(f\" ({display_name})\", end=\"\")\n print()\n if link:\n print(f\" Link: {link}\")\n if snippet:\n print(f\" Snippet: {snippet}\")\n if metrics:\n parts = []\n for k, v in metrics.items():\n if v and v > 0:\n parts.append(f\"{k}={v}\")\n if parts:\n print(f\" Metrics: {', '.join(parts)}\")\n \n # Reddit-specific\n reddit_score = a.get(\"score\")\n num_comments = a.get(\"num_comments\")\n if reddit_score is not None:\n print(f\" Reddit: {reddit_score}↑\", end=\"\")\n if num_comments:\n print(f\" · {num_comments} comments\", end=\"\")\n print()\n \n print()\n\n\ndef main():\n parser = argparse.ArgumentParser(description=\"Summarize merged JSON for LLM consumption\")\n parser.add_argument(\"--input\", \"-i\", type=Path, default=Path(\"/tmp/td-merged.json\"))\n parser.add_argument(\"--top\", \"-n\", type=int, default=10, help=\"Top N articles per topic\")\n parser.add_argument(\"--topic\", \"-t\", type=str, default=None, help=\"Filter to specific topic\")\n args = parser.parse_args()\n \n if not args.input.exists():\n print(f\"Error: {args.input} not found. Run the pipeline first.\")\n return\n \n with open(args.input) as f:\n data = json.load(f)\n \n summarize(data, top_n=args.top, topic_filter=args.topic)\n\n\nif __name__ == \"__main__\":\n main()\n","content_type":"text/x-python; charset=utf-8","language":"python","size":3525,"content_sha256":"89a02358806efff340f5e5c6ced11cde75651450fda8d7ae1516f27bea6d962d"},{"filename":"scripts/test-pipeline.sh","content":"#!/bin/bash\n# Pipeline smoke test — runs fetch steps with filtering, validates outputs\n# Usage:\n# ./test-pipeline.sh # run all sources\n# ./test-pipeline.sh --only twitter,rss # only these source types\n# ./test-pipeline.sh --skip web # skip web search\n# ./test-pipeline.sh --topics crypto # only sources with these topics\n# ./test-pipeline.sh --ids sama-twitter,openai-rss # specific source IDs\n# ./test-pipeline.sh --hours 12 # custom time window\n# ./test-pipeline.sh --keep # keep output dir after test\n# ./test-pipeline.sh --twitter-backend twitterapiio # force twitter backend\nset -e\n\nSCRIPT_DIR=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\nDEFAULTS=\"$SCRIPT_DIR/../config/defaults\"\nOUTDIR=$(mktemp -d /tmp/tech-digest-test-XXXXXX)\nPASSED=0\nSKIPPED=0\nFAILED=0\nHOURS=24\nKEEP=false\nONLY=\"\"\nSKIP=\"\"\nTOPICS=\"\"\nIDS=\"\"\nTWITTER_BACKEND=\"\"\nVERBOSE=\"\"\nCONFIG=\"\"\n\n# ── Parse args ──\nwhile [[ $# -gt 0 ]]; do\n case \"$1\" in\n --only) ONLY=\"$2\"; shift 2 ;;\n --skip) SKIP=\"$2\"; shift 2 ;;\n --topics) TOPICS=\"$2\"; shift 2 ;;\n --ids) IDS=\"$2\"; shift 2 ;;\n --hours) HOURS=\"$2\"; shift 2 ;;\n --keep) KEEP=true; shift ;;\n --twitter-backend|--backend) TWITTER_BACKEND=\"$2\"; shift 2 ;;\n --config) CONFIG=\"$2\"; shift 2 ;;\n --verbose|-v) VERBOSE=\"--verbose\"; shift ;;\n --help|-h)\n cat \u003c\u003c'HELP'\nPipeline smoke test — runs fetch steps with filtering, merges, and validates outputs.\n\nUSAGE:\n ./test-pipeline.sh [OPTIONS]\n\nOPTIONS:\n --only TYPES Only run these source types (comma-separated)\n Values: rss, twitter, github, reddit, web\n Example: --only twitter,rss\n\n --skip TYPES Skip these source types (comma-separated)\n Values: rss, twitter, github, reddit, web\n Example: --skip web,reddit\n\n --topics TOPICS Only include sources matching these topics (comma-separated)\n Values: llm, ai-agent, frontier-tech, crypto\n Example: --topics crypto,llm\n\n --ids IDS Only include specific source IDs (comma-separated)\n IDs are defined in config/defaults/sources.json\n Example: --ids sama-twitter,openai-rss,vitalik-twitter\n\n --hours N Time window for fetching articles (default: 24)\n Example: --hours 48\n\n --twitter-backend NAME\n Force a specific Twitter API backend\n Values: official, twitterapiio, auto\n official = X API v2 (needs X_BEARER_TOKEN)\n twitterapiio = twitterapi.io (needs TWITTERAPI_IO_KEY)\n auto = try twitterapiio first, fallback to official\n\n --config DIR User config overlay directory (optional)\n Example: --config workspace/config\n\n --verbose, -v Enable verbose logging for fetch scripts\n\n --keep Keep output directory after test (default: clean up on success)\n\n --help, -h Show this help message\n\nEXAMPLES:\n ./test-pipeline.sh # full pipeline, all sources\n ./test-pipeline.sh --only twitter --twitter-backend twitterapiio # twitter only via twitterapi.io\n ./test-pipeline.sh --topics crypto --hours 48 --keep # crypto sources, 48h window\n ./test-pipeline.sh --skip web,reddit -v # skip web+reddit, verbose\n ./test-pipeline.sh --ids sama-twitter,karpathy-twitter --only twitter\n\nENVIRONMENT:\n X_BEARER_TOKEN Official X API v2 bearer token (for --backend official)\n TWITTERAPI_IO_KEY twitterapi.io API key (for --backend twitterapiio)\n TWITTER_API_BACKEND Default twitter backend if --backend not given (official|twitterapiio|auto)\n BRAVE_API_KEY Brave Search API key (for web fetch)\n GITHUB_TOKEN GitHub token (optional, increases GitHub API rate limits)\nHELP\n exit 0\n ;;\n *) echo \"Unknown option: $1\"; exit 1 ;;\n esac\ndone\n\n# ── Helpers ──\nshould_run() {\n local type=\"$1\"\n # Check --only filter\n if [ -n \"$ONLY\" ]; then\n echo \",$ONLY,\" | grep -qi \",$type,\" || return 1\n fi\n # Check --skip filter\n if [ -n \"$SKIP\" ]; then\n echo \",$SKIP,\" | grep -qi \",$type,\" && return 1\n fi\n return 0\n}\n\nrun_step() {\n local name=\"$1\"; shift\n local start=$(date +%s)\n if \"$@\" 2>&1; then\n local elapsed=$(( $(date +%s) - start ))\n echo \"✅ $name (${elapsed}s)\"\n PASSED=$((PASSED + 1))\n else\n local code=$?\n local elapsed=$(( $(date +%s) - start ))\n echo \"❌ $name (exit $code, ${elapsed}s)\"\n FAILED=$((FAILED + 1))\n fi\n}\n\nvalidate_json() {\n local file=\"$1\" name=\"$2\"\n if [ -f \"$file\" ] && python3 -c \"\nimport json, sys\nd = json.load(open(sys.argv[1]))\n# Print summary stats\nif 'sources' in d and isinstance(d['sources'], list):\n ok = sum(1 for s in d['sources'] if s.get('status') == 'ok')\n total = len(d['sources'])\n articles = sum(s.get('count', len(s.get('articles', []))) for s in d['sources'])\n print(f' 📊 {ok}/{total} sources ok, {articles} articles')\nelif 'topics' in d:\n topics = d['topics']\n if isinstance(topics, dict):\n total = sum(len(t.get('articles', [])) for t in topics.values())\n print(f' 📊 {len(topics)} topics, {total} articles')\n elif isinstance(topics, list):\n total = sum(len(t.get('articles', [])) for t in topics)\n print(f' 📊 {len(topics)} topics, {total} articles')\n\" \"$file\" 2>/dev/null; then\n echo \"✅ $name JSON valid\"\n PASSED=$((PASSED + 1))\n else\n echo \"❌ $name JSON invalid or missing\"\n FAILED=$((FAILED + 1))\n fi\n}\n\n# ── Generate filtered sources if --topics or --ids specified ──\nEXTRA_ARGS=()\nif [ -n \"$TOPICS\" ] || [ -n \"$IDS\" ]; then\n FILTER_CONFIG=\"$OUTDIR/filter-config\"\n mkdir -p \"$FILTER_CONFIG\"\n python3 -c \"\nimport json, sys\ntopics_filter = '${TOPICS}'.split(',') if '${TOPICS}' else []\nids_filter = '${IDS}'.split(',') if '${IDS}' else []\n\nd = json.load(open('${DEFAULTS}/sources.json'))\nfiltered = []\nfor s in d['sources']:\n if ids_filter and s['id'] not in ids_filter:\n continue\n if topics_filter and not any(t in s.get('topics', []) for t in topics_filter):\n continue\n filtered.append(s)\n\nd['sources'] = filtered\nprint(f'Filtered: {len(filtered)} sources', file=sys.stderr)\njson.dump(d, open('${FILTER_CONFIG}/sources.json', 'w'), indent=2)\n\" 2>&1\n DEFAULTS=\"$FILTER_CONFIG\"\nfi\n\nif [ -n \"$CONFIG\" ]; then\n EXTRA_ARGS+=(\"--config\" \"$CONFIG\")\nfi\nif [ -n \"$VERBOSE\" ]; then\n EXTRA_ARGS+=(\"$VERBOSE\")\nfi\n\necho \"🧪 Pipeline Test (hours=$HOURS, outdir=$OUTDIR)\"\necho \" Sources: $(python3 -c \"import json; d=json.load(open('${DEFAULTS}/sources.json')); types={}\nfor s in d['sources']: t=s['type']; types[t]=types.get(t,0)+1\nprint(' | '.join(f'{t}:{n}' for t,n in sorted(types.items())))\" 2>/dev/null)\"\necho \"\"\n\n# ── Fetch steps ──\n\n# RSS\nif should_run \"rss\"; then\n run_step \"fetch-rss\" python3 \"$SCRIPT_DIR/fetch-rss.py\" --defaults \"$DEFAULTS\" --hours \"$HOURS\" --output \"$OUTDIR/rss.json\" --force \"${EXTRA_ARGS[@]}\"\n validate_json \"$OUTDIR/rss.json\" \"rss\"\nelse\n echo \"⏭ fetch-rss (skipped)\"\n SKIPPED=$((SKIPPED + 1))\nfi\n\n# GitHub\nif should_run \"github\"; then\n run_step \"fetch-github\" python3 \"$SCRIPT_DIR/fetch-github.py\" --defaults \"$DEFAULTS\" --hours \"$HOURS\" --output \"$OUTDIR/github.json\" --force \"${EXTRA_ARGS[@]}\"\n validate_json \"$OUTDIR/github.json\" \"github\"\nelse\n echo \"⏭ fetch-github (skipped)\"\n SKIPPED=$((SKIPPED + 1))\nfi\n\n# Twitter\nif should_run \"twitter\"; then\n TWITTER_ARGS=(\"--defaults\" \"$DEFAULTS\" \"--hours\" \"$HOURS\" \"--output\" \"$OUTDIR/twitter.json\" \"--force\" \"${EXTRA_ARGS[@]}\")\n [ -n \"$TWITTER_BACKEND\" ] && TWITTER_ARGS+=(\"--backend\" \"$TWITTER_BACKEND\")\n\n if [ -n \"$X_BEARER_TOKEN\" ] || [ -n \"$TWITTERAPI_IO_KEY\" ]; then\n run_step \"fetch-twitter\" python3 \"$SCRIPT_DIR/fetch-twitter.py\" \"${TWITTER_ARGS[@]}\"\n validate_json \"$OUTDIR/twitter.json\" \"twitter\"\n else\n echo \"⏭ fetch-twitter (no X_BEARER_TOKEN or TWITTERAPI_IO_KEY)\"\n SKIPPED=$((SKIPPED + 1))\n fi\nelse\n echo \"⏭ fetch-twitter (skipped)\"\n SKIPPED=$((SKIPPED + 1))\nfi\n\n# Reddit\nif should_run \"reddit\"; then\n if [ -f \"$SCRIPT_DIR/fetch-reddit.py\" ]; then\n run_step \"fetch-reddit\" python3 \"$SCRIPT_DIR/fetch-reddit.py\" --defaults \"$DEFAULTS\" --hours \"$HOURS\" --output \"$OUTDIR/reddit.json\" --force \"${EXTRA_ARGS[@]}\"\n validate_json \"$OUTDIR/reddit.json\" \"reddit\"\n else\n echo \"⏭ fetch-reddit (script not found)\"\n SKIPPED=$((SKIPPED + 1))\n fi\nelse\n echo \"⏭ fetch-reddit (skipped)\"\n SKIPPED=$((SKIPPED + 1))\nfi\n\n# Web search\nif should_run \"web\"; then\n if [ -n \"$BRAVE_API_KEY\" ]; then\n run_step \"fetch-web\" python3 \"$SCRIPT_DIR/fetch-web.py\" --defaults \"$DEFAULTS\" --freshness pd --output \"$OUTDIR/web.json\" --force \"${EXTRA_ARGS[@]}\"\n validate_json \"$OUTDIR/web.json\" \"web\"\n else\n echo \"⏭ fetch-web (no BRAVE_API_KEY)\"\n SKIPPED=$((SKIPPED + 1))\n fi\nelse\n echo \"⏭ fetch-web (skipped)\"\n SKIPPED=$((SKIPPED + 1))\nfi\n\n# ── Merge ──\nMERGE_ARGS=(\"--output\" \"$OUTDIR/merged.json\")\n[ -f \"$OUTDIR/rss.json\" ] && MERGE_ARGS+=(\"--rss\" \"$OUTDIR/rss.json\")\n[ -f \"$OUTDIR/twitter.json\" ] && MERGE_ARGS+=(\"--twitter\" \"$OUTDIR/twitter.json\")\n[ -f \"$OUTDIR/web.json\" ] && MERGE_ARGS+=(\"--web\" \"$OUTDIR/web.json\")\n[ -f \"$OUTDIR/github.json\" ] && MERGE_ARGS+=(\"--github\" \"$OUTDIR/github.json\")\n[ -f \"$OUTDIR/reddit.json\" ] && MERGE_ARGS+=(\"--reddit\" \"$OUTDIR/reddit.json\")\n\nif [ ${#MERGE_ARGS[@]} -gt 2 ]; then\n run_step \"merge-sources\" python3 \"$SCRIPT_DIR/merge-sources.py\" \"${MERGE_ARGS[@]}\"\n validate_json \"$OUTDIR/merged.json\" \"merged\"\n\n # Validate merged structure\n if python3 -c \"\nimport json, sys\nd = json.load(open(sys.argv[1]))\nassert 'topics' in d and 'output_stats' in d\nstats = d['output_stats']\nprint(f' 📊 Merged: {stats.get(\\\"total_articles\\\", \\\"?\\\")} articles across {len(d[\\\"topics\\\"])} topics')\n\" \"$OUTDIR/merged.json\" 2>/dev/null; then\n echo \"✅ merged structure valid\"\n PASSED=$((PASSED + 1))\n else\n echo \"❌ merged structure invalid\"\n FAILED=$((FAILED + 1))\n fi\nelse\n echo \"⏭ merge (no source files to merge)\"\n SKIPPED=$((SKIPPED + 1))\nfi\n\n# ── Summary ──\necho \"\"\necho \"━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\"\necho \"📊 Results: $PASSED passed, $FAILED failed, $SKIPPED skipped\"\necho \" Output: $OUTDIR\"\nif [ \"$KEEP\" = false ] && [ \"$FAILED\" -eq 0 ]; then\n rm -rf \"$OUTDIR\"\n echo \" (cleaned up — use --keep to preserve)\"\nfi\n[ \"$FAILED\" -eq 0 ] && exit 0 || exit 1\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":10975,"content_sha256":"3c155dde10b6ac033d39cc96cba20b46c1c49402d2a4160acff2bcd4ebb9c308"},{"filename":"scripts/validate-config.py","content":"#!/usr/bin/env python3\n\"\"\"\nConfiguration validation script for tech-news-digest.\n\nValidates sources.json and topics.json against JSON Schema and performs\nadditional consistency checks.\n\nUsage:\n python3 validate-config.py [--defaults DEFAULTS_DIR] [--config CONFIG_DIR] [--verbose]\n\"\"\"\n\nimport json\nimport argparse\nimport logging\nimport sys\nfrom pathlib import Path\nfrom typing import Dict, Any\n\ntry:\n from jsonschema import validate, ValidationError\n HAS_JSONSCHEMA = True\nexcept ImportError:\n HAS_JSONSCHEMA = False\n\n\ndef setup_logging(verbose: bool) -> logging.Logger:\n \"\"\"Setup logging configuration.\"\"\"\n level = logging.DEBUG if verbose else logging.INFO\n logging.basicConfig(\n level=level,\n format='%(asctime)s - %(levelname)s - %(message)s',\n datefmt='%Y-%m-%d %H:%M:%S'\n )\n return logging.getLogger(__name__)\n\n\ndef load_json_file(file_path: Path) -> Dict[str, Any]:\n \"\"\"Load and parse JSON file.\"\"\"\n try:\n with open(file_path, 'r', encoding='utf-8') as f:\n return json.load(f)\n except FileNotFoundError:\n raise FileNotFoundError(f\"Config file not found: {file_path}\")\n except json.JSONDecodeError as e:\n raise ValueError(f\"Invalid JSON in {file_path}: {e}\")\n\n\ndef validate_against_schema(data: Dict[str, Any], schema: Dict[str, Any], \n config_type: str) -> bool:\n \"\"\"Validate data against JSON schema.\"\"\"\n if not HAS_JSONSCHEMA:\n logging.warning(\"jsonschema not available, skipping schema validation\")\n return True\n \n try:\n # Extract the relevant schema definition\n if config_type == \"sources\":\n schema_def = {\n \"type\": \"object\",\n \"required\": [\"sources\"],\n \"properties\": {\n \"sources\": {\n \"type\": \"array\", \n \"items\": schema[\"definitions\"][\"source\"]\n }\n }\n }\n elif config_type == \"topics\":\n schema_def = {\n \"type\": \"object\",\n \"required\": [\"topics\"],\n \"properties\": {\n \"topics\": {\n \"type\": \"array\",\n \"items\": schema[\"definitions\"][\"topic\"]\n }\n }\n }\n else:\n raise ValueError(f\"Unknown config type: {config_type}\")\n \n validate(instance=data, schema=schema_def)\n logging.info(f\"✅ {config_type}.json passed schema validation\")\n return True\n \n except ValidationError as e:\n logging.error(f\"❌ Schema validation failed for {config_type}.json:\")\n logging.error(f\" Path: {' -> '.join(str(p) for p in e.absolute_path)}\")\n logging.error(f\" Error: {e.message}\")\n return False\n\n\ndef validate_sources_consistency(sources_data: Dict[str, Any], \n topics_data: Dict[str, Any]) -> bool:\n \"\"\"Validate consistency between sources and topics.\"\"\"\n errors = []\n \n # Get valid topic IDs\n valid_topics = {topic[\"id\"] for topic in topics_data[\"topics\"]}\n logging.debug(f\"Valid topic IDs: {valid_topics}\")\n \n # Check source topic references\n for source in sources_data[\"sources\"]:\n source_id = source.get(\"id\", \"unknown\")\n source_topics = set(source.get(\"topics\", []))\n \n # Check for invalid topic references\n invalid_topics = source_topics - valid_topics\n if invalid_topics:\n errors.append(f\"Source '{source_id}' references invalid topics: {invalid_topics}\")\n \n # Check for empty topic lists\n if not source_topics:\n errors.append(f\"Source '{source_id}' has no topics assigned\")\n \n # Check for duplicate source IDs\n source_ids = [source.get(\"id\") for source in sources_data[\"sources\"]]\n duplicates = {id for id in source_ids if source_ids.count(id) > 1}\n if duplicates:\n errors.append(f\"Duplicate source IDs found: {duplicates}\")\n \n # Check for duplicate topic IDs\n topic_ids = [topic.get(\"id\") for topic in topics_data[\"topics\"]]\n duplicates = {id for id in topic_ids if topic_ids.count(id) > 1}\n if duplicates:\n errors.append(f\"Duplicate topic IDs found: {duplicates}\")\n \n if errors:\n logging.error(\"❌ Consistency validation failed:\")\n for error in errors:\n logging.error(f\" {error}\")\n return False\n else:\n logging.info(\"✅ Consistency validation passed\")\n return True\n\n\ndef validate_source_types(sources_data: Dict[str, Any]) -> bool:\n \"\"\"Validate source-type specific requirements.\"\"\"\n errors = []\n \n for source in sources_data[\"sources\"]:\n source_id = source.get(\"id\", \"unknown\")\n source_type = source.get(\"type\")\n \n if source_type == \"rss\":\n if not source.get(\"url\"):\n errors.append(f\"RSS source '{source_id}' missing required 'url' field\")\n elif source_type == \"twitter\":\n if not source.get(\"handle\"):\n errors.append(f\"Twitter source '{source_id}' missing required 'handle' field\")\n elif source_type == \"github\":\n if not source.get(\"repo\"):\n errors.append(f\"GitHub source '{source_id}' missing required 'repo' field\")\n elif source_type == \"reddit\":\n if not source.get(\"subreddit\"):\n errors.append(f\"Reddit source '{source_id}' missing required 'subreddit' field\")\n elif source_type == \"web\":\n # Web sources are handled by topics, no specific validation needed\n pass\n else:\n errors.append(f\"Source '{source_id}' has invalid type: {source_type}\")\n \n if errors:\n logging.error(\"❌ Source type validation failed:\")\n for error in errors:\n logging.error(f\" {error}\")\n return False\n else:\n logging.info(\"✅ Source type validation passed\")\n return True\n\n\ndef main():\n \"\"\"Main validation function.\"\"\"\n parser = argparse.ArgumentParser(\n description=\"Validate tech-news-digest configuration files\",\n formatter_class=argparse.RawDescriptionHelpFormatter,\n epilog=\"\"\"\nExamples:\n python3 validate-config.py\n python3 validate-config.py --defaults config/defaults --config workspace/config --verbose\n python3 validate-config.py --config workspace/config --verbose # backward compatibility\n \"\"\"\n )\n \n parser.add_argument(\n \"--defaults\",\n type=Path,\n default=Path(\"config/defaults\"),\n help=\"Default configuration directory with skill defaults (default: config/defaults)\"\n )\n \n parser.add_argument(\n \"--config\",\n type=Path,\n help=\"User configuration directory for overlays (optional)\"\n )\n \n parser.add_argument(\n \"--verbose\", \"-v\",\n action=\"store_true\", \n help=\"Enable verbose logging\"\n )\n \n args = parser.parse_args()\n logger = setup_logging(args.verbose)\n \n # Load config_loader for merged configurations\n try:\n from config_loader import load_merged_sources, load_merged_topics\n except ImportError:\n # Fallback for relative import\n import sys\n sys.path.append(str(Path(__file__).parent))\n from config_loader import load_merged_sources, load_merged_topics\n \n # File paths\n schema_path = Path(\"config/schema.json\")\n \n if args.config:\n logger.info(f\"Validating merged configuration: defaults={args.defaults}, config={args.config}\")\n else:\n logger.info(f\"Validating default configuration: {args.defaults}\")\n \n try:\n # Backward compatibility: if only --config provided, use old behavior\n if args.config and args.defaults == Path(\"config/defaults\") and not args.defaults.exists():\n logger.debug(\"Backward compatibility mode: using --config as sole source\")\n defaults_dir = args.config\n config_dir = None\n else:\n defaults_dir = args.defaults\n config_dir = args.config\n \n # Load schema\n schema = load_json_file(schema_path)\n logger.debug(\"Loaded schema.json\")\n \n # Load merged configuration data\n merged_sources = load_merged_sources(defaults_dir, config_dir)\n merged_topics = load_merged_topics(defaults_dir, config_dir)\n \n # Convert to the format expected by validation functions\n sources_data = {\"sources\": merged_sources}\n topics_data = {\"topics\": merged_topics}\n \n logger.debug(f\"Loaded {len(merged_sources)} merged sources, {len(merged_topics)} merged topics\")\n \n # Perform validations\n all_valid = True\n \n # Schema validation\n all_valid &= validate_against_schema(sources_data, schema, \"sources\")\n all_valid &= validate_against_schema(topics_data, schema, \"topics\")\n \n # Consistency validation\n all_valid &= validate_sources_consistency(sources_data, topics_data)\n \n # Source type validation \n all_valid &= validate_source_types(sources_data)\n \n # Summary\n if all_valid:\n logger.info(\"🎉 All validations passed!\")\n return 0\n else:\n logger.error(\"💥 Validation failed!\")\n return 1\n \n except Exception as e:\n logger.error(f\"💥 Validation error: {e}\")\n return 1\n\n\nif __name__ == \"__main__\":\n sys.exit(main())","content_type":"text/x-python; charset=utf-8","language":"python","size":9622,"content_sha256":"59359324cabe2bca0ac8192b537146b255e72ccf79aaf6e439525bfddc9a16ee"},{"filename":"scripts/validate-digest.py","content":"#!/usr/bin/env python3\n\"\"\"Offline, deterministic quality gate for Markdown/Discord digests.\n\nExit codes: 0 pass, 1 content violations, 2 CLI/input errors. No network or\nsemantic fact checking. All top-level bullets are content unless they are an\nexplicit attribution line. Only clearly proposed next-week experiments may\nomit citations. URL identity normalizes host/scheme case and removes fragments and\ncommon tracking parameters; redirects and event identity are not checked.\n\"\"\"\nimport argparse\nfrom html import unescape\nfrom pathlib import Path\nimport re\nimport sys\nfrom urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit\n\nBULLET = re.compile(r\"^( *)([-•])\\s+(.+)$\")\nMARKDOWN_START = re.compile(r\"\\[([^\\]\\n]*)\\]\\(\")\nANGLE = re.compile(r\"\u003c([^\u003c>\\s]+)>\")\nRAW_URL = re.compile(r\"https?://[^\\s\u003c>\\[\\]，。；！？]+\", re.I)\nREPO = re.compile(r\"(?\u003c![\\w/])([A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+)(?![\\w/])\")\nVERSION = re.compile(r\"(?\u003c!\\w)v?\\d+\\.\\d+(?:\\.[\\w.+-]+)*(?:-[\\w.-]+)?\", re.I)\nCJK = re.compile(r\"[\\u3400-\\u9fff\\U00020000-\\U0002ffff]\")\nGROWTH = re.compile(r\"\\+\\s*\\d[\\d,]*(?:\\.\\d+)?\\s*[kmb万亿]?\\s*/\\s*(?:day\\b|日)\", re.I)\nSCORE = re.compile(r\"^(?:🔥\\s*\\d|\\[?\\s*(?:quality[ _-]?score|quality score|score|质量分|质量评分)\\s*[:：=]\\s*\\d)\", re.I)\nLEGACY = re.compile(r\"(?\u003c![A-Za-z])KOL(?![A-Za-z])|关键人物动态|GitHub\\s*(?:Trending|趋势|热门)|^Trending\\b\", re.I)\nRELEASE_HEADING = re.compile(r\"\\breleases?\\b|发布精选|版本发布|开源发布|GitHub\\s*发布\", re.I)\nCOMBINED_HEADING = re.compile(r\"\\btools?\\b|工具\", re.I)\nEXPERIMENT_HEADING = re.compile(r\"下周.*(?:实验|尝试|验证)|next[ -]week.*(?:experiment|try|validation)\", re.I)\nPROPOSAL = re.compile(r\"^(?:建议|尝试|计划|拟|试用|验证|对比|测试|评估|复现|可以|不妨|try\\b|test\\b|evaluate\\b|compare\\b|experiment\\b|consider\\b|plan\\b)\", re.I)\n\n\ndef plain(text):\n return re.sub(r\"[*_`]\", \"\", text).strip()\n\n\ndef attribution(line):\n return bool(re.match(r\"^(?:🤖\\s*)?(?:Generated by\\b|Powered by\\b|由\\s*tech-news-digest.*生成)\", plain(line), re.I))\n\n\ndef heading(line):\n stripped = line.strip()\n match = re.match(r\"^#{1,6}\\s+(.+?)(?:\\s+#+)?$\", stripped)\n if match:\n return plain(match.group(1))\n if re.fullmatch(r\"\\*\\*.+\\*\\*\", stripped):\n return plain(stripped)\n return None\n\n\ndef valid_url(url):\n \"\"\"Validate HTTP(S) syntax, not network reachability.\"\"\"\n if re.search(r\"[\\s\u003c>\\x00-\\x1f]\", url):\n return False\n try:\n parsed = urlsplit(url)\n # Accessing port also validates nonnumeric/out-of-range ports.\n _ = parsed.port\n host = (parsed.hostname or \"\").encode(\"idna\").decode(\"ascii\")\n if \":\" not in host and not all(\n re.fullmatch(r\"[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?\", label)\n for label in host.rstrip(\".\").split(\".\")\n ):\n return False\n return (parsed.scheme.lower() in (\"http\", \"https\") and bool(parsed.hostname)\n and not parsed.username and not parsed.password\n and not re.search(r\"[\\\\{}|^`]\", parsed.netloc))\n except (ValueError, UnicodeError):\n return False\n\n\ndef links(text):\n \"\"\"Return (URL, start, end) for inline Markdown and angle autolinks.\n\n Balanced parentheses in destinations and optional quoted titles work;\n reference-style links and HTML anchors are deliberately outside this gate.\n \"\"\"\n result = []\n for match in MARKDOWN_START.finditer(text):\n pos = match.end()\n depth = 1\n quote = None\n while pos \u003c len(text) and depth:\n char = text[pos]\n if char == \"\\\\\":\n pos += 2\n continue\n if quote:\n if char == quote:\n quote = None\n elif char in ('\"', \"'\") and pos > match.end() and text[pos - 1].isspace():\n quote = char\n elif char == \"(\":\n depth += 1\n elif char == \")\":\n depth -= 1\n pos += 1\n if depth:\n continue\n target = text[match.end():pos - 1].strip()\n # Only split off a well-formed optional title, never whitespace in a URL.\n target = re.sub(r'\\s+(?:\"[^\"\\n]*\"|\\'[^\\'\\n]*\\')

Tech News Digest Evidence-led tech digest with unified collection and one editorial selection shared across formats. Follow as the authoritative mode policy. Editorial Contract - Daily preserves original sections: 2–4 sentence overview; configured topics (normally LLM, AI Agent, Crypto, Frontier Tech; aim for 3–5 qualified items each); KOL viewpoints ≤3; releases ≤3 repositories; project discovery ≤3; blog picks ≤3; ≤30 unique items overall. News/blog explanations normally contain 2–3 concise sentences of change, evidence and implication. Retain empty headings with a coverage note, never fill…

, '', target)\n if target.startswith(\"\u003c\") and target.endswith(\">\"):\n target = target[1:-1]\n result.append((unescape(target), match.start(), pos))\n for match in ANGLE.finditer(text):\n if not any(start \u003c= match.start() \u003c end for _, start, end in result):\n result.append((unescape(match.group(1)), match.start(), match.end()))\n return sorted(result, key=lambda value: value[1])\n\n\ndef source_urls(text):\n \"\"\"Include bare URLs in deduplication, but not as qualifying citations.\"\"\"\n found = links(text)\n result = [url for url, _, _ in found if valid_url(url)]\n for match in RAW_URL.finditer(text):\n if any(start \u003c= match.start() \u003c end for _, start, end in found):\n continue\n url = match.group().rstrip(\".,;:!?)\")\n if valid_url(url):\n result.append(url)\n return result\n\n\ndef url_key(url):\n parts = urlsplit(url)\n query = [(key, value) for key, value in parse_qsl(parts.query, keep_blank_values=True)\n if not key.lower().startswith(\"utm_\") and key.lower() not in (\"fbclid\", \"gclid\")]\n return urlunsplit((parts.scheme.lower(), parts.netloc.lower(), parts.path, urlencode(query), \"\"))\n\n\ndef section_gate(title, mode):\n \"\"\"Exact editorial headings only; don't infer a section from prose.\"\"\"\n title = re.sub(r\"^[^\\w\\u3400-\\u9fff]+\", \"\", title).strip().lower()\n title = re.sub(r\"\\s*[（(].*$\", \"\", title).strip()\n if title in (\"值得读\", \"延伸阅读\", \"阅读\", \"reading\", \"博客精选\", \"blog picks\"):\n return \"reading\", 3 if mode == \"daily\" else 4\n if title in (\"项目发现\", \"github 项目发现\", \"值得试\", \"project discovery\", \"worth trying\"):\n return \"discovery\", 3\n if mode == \"daily\" and title in (\"kol 动态\", \"kol动态\", \"kol updates\", \"关键人物动态\"):\n return \"kol\", 3\n if mode == \"daily\" and title in (\"可行动\", \"action\", \"actions\"):\n return \"action\", 2\n if mode == \"weekly\" and title in (\"观点与深度阅读\", \"观点 / 深度阅读\", \"viewpoints / deep reading\", \"viewpoints and deep reading\", \"deep reading\", \"viewpoints\"):\n return \"reading\", 4\n if mode == \"weekly\" and (title in (\"下周关注\", \"下周观察\", \"next-week watch\", \"next week watch\", \"下周验证清单\") or EXPERIMENT_HEADING.search(title)):\n return \"watch\", 3\n return None\n\n\ndef repository(text, urls):\n for url in urls:\n parts = urlsplit(url)\n path = parts.path.strip(\"/\").split(\"/\")\n if parts.hostname and parts.hostname.lower() in (\"github.com\", \"www.github.com\") and len(path) >= 2:\n return \"/\".join(path[:2]).removesuffix(\".git\").lower()\n # Only inspect the label before explanatory prose, not arbitrary slashes.\n label = re.split(r\"\\s[—–]|\\s-\\s|：\", text, maxsplit=1)[0]\n match = REPO.search(plain(label))\n return match.group(1).lower() if match else None\n\n\ndef explanation(text, repo):\n # Keep non-source Markdown labels (often the actual repository title).\n value = text\n for url, start, end in reversed(links(text)):\n label = re.match(r\"\\[([^\\]]*)\\]\", text[start:end])\n replacement = label.group(1) if label else \"\"\n # Citation labels carry no explanation; substantive linked text does.\n if re.fullmatch(r\"(?:来源|源|发布|原文|链接|详情|source|release(?: notes)?|link|read more)\", plain(replacement), re.I):\n replacement = \"\"\n value = value[:start] + replacement + value[end:]\n value = RAW_URL.sub(\"\", value)\n if repo:\n value = re.sub(re.escape(repo), \"\", value, flags=re.I)\n value = VERSION.sub(\"\", value)\n value = plain(value)\n # Formatting delimiters around the repo/version are not explanation text.\n return value.strip(\" \\t\\r\\n—–-:：|()（）\")\n\n\ndef body_length(text):\n \"\"\"Exclude only URL targets; retain labels, markup, titles and whitespace.\"\"\"\n found = links(text)\n size = len(text)\n for _, start, end in found:\n fragment = text[start:end]\n prefix = MARKDOWN_START.match(fragment)\n target = fragment[prefix.end():-1].strip() if prefix else fragment[1:-1]\n target = re.sub(r'\\s+(?:\"[^\"\\n]*\"|\\'[^\\'\\n]*\\')

Tech News Digest Evidence-led tech digest with unified collection and one editorial selection shared across formats. Follow as the authoritative mode policy. Editorial Contract - Daily preserves original sections: 2–4 sentence overview; configured topics (normally LLM, AI Agent, Crypto, Frontier Tech; aim for 3–5 qualified items each); KOL viewpoints ≤3; releases ≤3 repositories; project discovery ≤3; blog picks ≤3; ≤30 unique items overall. News/blog explanations normally contain 2–3 concise sentences of change, evidence and implication. Retain empty headings with a coverage note, never fill…

, '', target)\n if target.startswith(\"\u003c\") and target.endswith(\">\"):\n target = target[1:-1]\n size -= len(target)\n for match in RAW_URL.finditer(text):\n if not any(start \u003c= match.start() \u003c end for _, start, end in found):\n size -= len(match.group().rstrip(\".,;:!?)\"))\n return size\n\n\ndef check_topic_headings(lines, expected):\n \"\"\"Use caller-supplied localized labels, never hard-code configured topics.\"\"\"\n errors = []\n sections = [(i, heading(line)) for i, line in enumerate(lines) if heading(line) is not None]\n positions = []\n for title in expected:\n matches = [(i, pos) for i, (pos, found) in enumerate(sections) if found == plain(title)]\n if len(matches) != 1:\n errors.append(f\"[topic-heading] expected exactly one configured heading: {title}\")\n continue\n index, pos = matches[0]\n positions.append(pos)\n end = sections[index + 1][0] if index + 1 \u003c len(sections) else len(lines)\n content = [line.strip() for line in lines[pos + 1:end] if line.strip() and line.strip() != \"---\"]\n if not content:\n errors.append(f\"[coverage-note] empty topic needs an honest coverage note: {title}\")\n if positions != sorted(positions):\n errors.append(\"[topic-order] configured topics must retain their order\")\n return errors\n\n\ndef validate(text, mode=\"daily\", topic_headings=None):\n \"\"\"Return human-readable violations, each with a stable bracketed code.\"\"\"\n if mode not in (\"daily\", \"weekly\"):\n raise ValueError(\"mode must be daily or weekly\")\n errors = []\n lines = text.splitlines()\n if mode == \"weekly\":\n size = body_length(text)\n if size > 10000:\n errors.append(f\"[body-limit] {size} body characters exceeds weekly cap 10000\")\n if topic_headings:\n errors.extend(check_topic_headings(lines, topic_headings))\n matches = [BULLET.match(line) for line in lines]\n # A uniformly indented Markdown list is valid; deeper bullets are children.\n indents = [len(match.group(1)) for match in matches if match and len(match.group(1)) \u003c= 3]\n base = min(indents, default=0)\n blocks = []\n current = None\n section = \"\"\n content_lines = []\n for number, line in enumerate(lines, 1):\n stripped = line.strip()\n bullet = BULLET.match(line)\n candidate = bullet.group(3) if bullet else stripped\n if attribution(candidate):\n current = None\n continue\n content_lines.append((number, line))\n if GROWTH.search(line):\n errors.append(f\"line {number}: [daily-growth] unsupported +N/day or +N/日 metric\")\n if re.search(r\"(?:Data\\s+Sources|数据源(?:统计)?|数据来源(?:统计)?)\\s*[:：]\", plain(line), re.I):\n errors.append(f\"line {number}: [operational-footer] remove data-source statistics\")\n title = heading(line)\n if title is not None:\n section = title\n current = None\n if LEGACY.search(title) and not (mode == \"daily\" and re.search(r\"KOL|关键人物动态\", title, re.I)):\n errors.append(f\"line {number}: [legacy-heading] remove fixed KOL/Trending section\")\n continue\n if re.fullmatch(r\"\\s*(?:-{3,}|\\*{3,}|_{3,})\\s*\", line):\n current = None\n continue\n if bullet and len(bullet.group(1)) == base:\n current = {\"line\": number, \"section\": section, \"text\": bullet.group(3)}\n blocks.append(current)\n elif current is not None and stripped:\n current[\"text\"] += \"\\n\" + stripped\n\n limit, release_limit = (30, 3) if mode == \"daily\" else (35, 5)\n if not blocks:\n errors.append(\"[empty-digest] no top-level content bullets\")\n if len(blocks) > limit:\n errors.append(f\"[item-limit] {len(blocks)} content bullets exceeds {mode} cap {limit}\")\n\n seen_urls = {}\n for number, line in content_lines:\n for url in source_urls(line):\n key = url_key(url)\n if key in seen_urls:\n errors.append(f\"line {number}: [duplicate-url] {url} (first on line {seen_urls[key]})\")\n else:\n seen_urls[key] = number\n\n release_count = 0\n seen_repos = {}\n section_counts = {}\n for block in blocks:\n number, body, title = block[\"line\"], block[\"text\"], block[\"section\"]\n gate = section_gate(title, mode)\n if gate:\n name, cap = gate\n section_counts[name] = section_counts.get(name, 0) + 1\n if section_counts[name] == cap + 1:\n errors.append(f\"line {number}: [section-limit] {name} exceeds {mode} cap {cap}\")\n citations = links(body)\n invalid = any(not valid_url(url) for url, _, _ in citations)\n proposed = EXPERIMENT_HEADING.search(title) and (\n PROPOSAL.search(plain(body)) or re.search(r\"拟议|尚未执行|proposed|not performed\", title, re.I))\n if mode == \"weekly\" and gate and gate[0] == \"watch\":\n proposed = proposed or (PROPOSAL.search(plain(body)) and re.search(\n r\"拟议实验|建议实验|proposed experiment|尚未执行|not performed\", body, re.I))\n if invalid or (not any(valid_url(url) for url, _, _ in citations) and not proposed):\n errors.append(f\"line {number}: [source-link] expected a valid HTTP(S) Markdown or angle source link\")\n if SCORE.search(plain(body)):\n errors.append(f\"line {number}: [quality-score] remove public quality-score prefix\")\n urls = source_urls(body)\n repo = repository(body, urls)\n dedicated = RELEASE_HEADING.search(title) and not COMBINED_HEADING.search(title)\n if mode == \"weekly\" and re.search(r\"关键发布\", title):\n dedicated = True\n explicit = any(re.search(r\"/releases(?:/|$)\", urlsplit(url).path) for url in urls)\n labeled_release = bool(repo and VERSION.search(body) and RELEASE_HEADING.search(title))\n if not (dedicated or explicit or labeled_release):\n continue\n release_count += 1\n if repo:\n if repo in seen_repos:\n errors.append(f\"line {number}: [duplicate-repo] {repo} (first on line {seen_repos[repo]})\")\n seen_repos[repo] = number\n summary = explanation(body, repo)\n if CJK.search(summary) and len(summary) > 80:\n errors.append(f\"line {number}: [release-length] explanation has {len(summary)} characters; cap is 80\")\n if release_count > release_limit:\n errors.append(f\"[release-limit] {release_count} releases exceeds {mode} cap {release_limit}\")\n return errors\n\n\ndef main(argv=None):\n parser = argparse.ArgumentParser(description=__doc__)\n parser.add_argument(\"--input\", required=True, type=Path, help=\"UTF-8 Markdown/Discord digest\")\n parser.add_argument(\"--mode\", required=True, choices=(\"daily\", \"weekly\"))\n parser.add_argument(\"--topic-heading\", action=\"append\", default=[],\n help=\"Weekly: repeat configured localized headings in order for heading/coverage checks\")\n args = parser.parse_args(argv)\n try:\n text = args.input.read_text(encoding=\"utf-8\")\n except (OSError, UnicodeError) as exc:\n print(f\"ERROR: cannot read {args.input}: {exc}\", file=sys.stderr)\n return 2\n errors = validate(text, args.mode, topic_headings=args.topic_heading)\n if errors:\n print(f\"FAIL: {len(errors)} quality-gate violation(s)\", file=sys.stderr)\n for error in errors:\n print(f\"- {error}\", file=sys.stderr)\n return 1\n print(f\"PASS: {args.mode} digest satisfies deterministic quality gates\")\n return 0\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n","content_type":"text/x-python; charset=utf-8","language":"python","size":16235,"content_sha256":"f5b7c09560527480416952e302443de556291ab3a84f9d98f2986ca8c93c5c51"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"Tech News Digest","type":"text"}]},{"type":"paragraph","content":[{"text":"Evidence-led tech digest with unified collection and one editorial selection shared across formats. Follow ","type":"text"},{"text":"references/digest-prompt.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" as the authoritative mode policy.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Editorial Contract","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Daily preserves original sections: 2–4 sentence overview; configured topics (normally LLM, AI Agent, Crypto, Frontier Tech; aim for 3–5 qualified items each); KOL viewpoints ≤3; releases ≤3 repositories; project discovery ≤3; blog picks ≤3; ≤30 unique items overall. News/blog explanations normally contain 2–3 concise sentences of change, evidence and implication. Retain empty headings with a coverage note, never filler. Optimize content within sections: a request to shorten GitHub releases must not become a whole-report redesign or one-message digest.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Weekly ≤35 unique items: overview; configured topic sections in configured order (normally LLM, AI Agent, Crypto, Frontier Tech), with 2–4 substantial syntheses per topic as an advisory aim. Each synthesis uses 2–4 concise sentences joining week progress, evidence and implications, not concatenated headlines. Retain empty topic, key-release, project and viewpoint/reading headings with honest non-bullet coverage notes. Key releases ≤5 repositories (Chinese explanation ≤80 characters); projects ≤3; viewpoints/deep reading ≤4; optional next-week watch ≤3 pending releases, fixes or uncertainties. Do not force experiments; factual watch claims need citations, clearly proposed/not-performed experiments retain the narrow citation exemption.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Caps are not quotas. Count evidence events inside syntheses and standalone watch items. Preserve topic sections in both modes; no global three-theme constraint. Body limits: daily ≤6500 characters, weekly ≤10000, excluding URL targets; never fill to the limit.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Mandatory security coverage: follow the primary-alert/search and pre-delivery checks in ","type":"text"},{"text":"references/digest-prompt.md","type":"text","marks":[{"type":"code_inline"}]},{"text":". Preserve existing topics; use optional ","type":"text"},{"text":"🛡️ 安全警报","type":"text","marks":[{"type":"code_inline"}]},{"text":" after the overview only for credible urgent incidents in either mode; weekly may also use an optional dedicated security/risk section for consequential nonurgent risks. Omit when empty and count/deduplicate within existing caps. Log checks and decisions; failed collection is not evidence of no incidents.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Relevance, impact and evidence before internal scores. Exceptional claims need original technical evidence and independent assessment where possible, otherwise exclude/downgrade. No public scores, social metrics or operational statistics. Daily keeps substantive, nonduplicative KOL viewpoints. A material coverage gap may receive one concise caveat without raw counts; details stay in the final operational log.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Deduplicate events and canonical URLs across the whole report. Daily repeats require incremental developments; weekly may synthesize daily coverage. Daily keeps blog picks; weekly viewpoints/deep reading may include original opinions, papers, docs, postmortems or essays.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Project discovery is not verified trending; no lifetime-derived growth or repeated mature repos without meaningful changes. Preserve the prompt's consequential-release rules, exact versions and official links.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Plain bullets with bold titles; concise inline Discord links ","type":"text"},{"text":"[来源](\u003cURL>)","type":"text","marks":[{"type":"code_inline"}]},{"text":". Section-aware chunks ≤1700 characters including numbering. Apply the requested language, including Simplified Chinese explanations when configured.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Validate the canonical archive with ","type":"text"},{"text":"python3 scripts/validate-digest.py --input FILE --mode daily","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"--mode weekly","type":"text","marks":[{"type":"code_inline"}]},{"text":" before delivery; fix all errors. Weekly production runs append one ","type":"text"},{"text":"--topic-heading 'EXACT LOCALIZED HEADING'","type":"text","marks":[{"type":"code_inline"}]},{"text":" per configured topic in order; without these flags the gate cannot infer custom headings. Missing/failing validation blocks delivery. Manually verify unique-event counts inside syntheses, required non-topic headings/overview, security placement, evidence/coverage honesty and cross-format parity too.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Discord, Markdown, email and PDF use identical selections/judgments. Operational stats/runtime attribution belong in the final log, not content.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Legacy ","type":"text"},{"text":"ITEMS_PER_SECTION","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"BLOG_PICKS_COUNT","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"EXTRA_SECTIONS","type":"text","marks":[{"type":"code_inline"}]},{"text":" are advisory parameters superseded by mode policy: ignore their counts/extra sections, especially duplicate weekly trends.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Quick Start","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Configuration Setup","type":"text","marks":[{"type":"strong"}]},{"text":": Default configs are in ","type":"text"},{"text":"config/defaults/","type":"text","marks":[{"type":"code_inline"}]},{"text":". Copy to workspace for customization:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"mkdir -p workspace/config\ncp config/defaults/sources.json workspace/config/tech-news-digest-sources.json\ncp config/defaults/topics.json workspace/config/tech-news-digest-topics.json","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Environment Variables","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"TWITTERAPI_IO_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" - twitterapi.io API key (optional, preferred)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"X_BEARER_TOKEN","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Twitter/X official API bearer token (optional, fallback)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"TAVILY_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Tavily Search API key, alternative to Brave (optional)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"WEB_SEARCH_BACKEND","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Web search backend: auto|brave|tavily (optional, default: auto)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"BRAVE_API_KEYS","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Brave Search API keys, comma-separated for rotation (optional)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"BRAVE_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Single Brave key fallback (optional)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"GITHUB_TOKEN","type":"text","marks":[{"type":"code_inline"}]},{"text":" - GitHub personal access token (optional, improves rate limits)","type":"text"}]}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Generate Digest","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Unified pipeline (recommended) — runs all 6 sources in parallel + merge\npython3 scripts/run-pipeline.py \\\n --defaults config/defaults \\\n --config workspace/config \\\n --hours 48 --freshness pd \\\n --archive-dir workspace/archive/tech-news-digest/ \\\n --output /tmp/td-merged.json --verbose --force","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Select and Validate","type":"text","marks":[{"type":"strong"}]},{"text":": Follow ","type":"text"},{"text":"references/digest-prompt.md","type":"text","marks":[{"type":"code_inline"}]},{"text":"; validate the canonical Markdown before rendering the same selections to Discord, email and PDF.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Configuration Files","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"sources.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Unified Data Sources","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"json"},"content":[{"text":"{\n \"sources\": [\n {\n \"id\": \"openai-rss\",\n \"type\": \"rss\",\n \"name\": \"OpenAI Blog\",\n \"url\": \"https://openai.com/blog/rss.xml\",\n \"enabled\": true,\n \"priority\": true,\n \"topics\": [\"llm\", \"ai-agent\"],\n \"note\": \"Official OpenAI updates\"\n },\n {\n \"id\": \"sama-twitter\",\n \"type\": \"twitter\", \n \"name\": \"Sam Altman\",\n \"handle\": \"sama\",\n \"enabled\": true,\n \"priority\": true,\n \"topics\": [\"llm\", \"frontier-tech\"],\n \"note\": \"OpenAI CEO\"\n }\n ]\n}","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"topics.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Enhanced Topic Definitions","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"json"},"content":[{"text":"{\n \"topics\": [\n {\n \"id\": \"llm\",\n \"emoji\": \"🧠\",\n \"label\": \"LLM / Large Models\",\n \"description\": \"Large Language Models, foundation models, breakthroughs\",\n \"search\": {\n \"queries\": [\"LLM latest news\", \"large language model breakthroughs\"],\n \"must_include\": [\"LLM\", \"large language model\", \"foundation model\"],\n \"exclude\": [\"tutorial\", \"beginner guide\"]\n },\n \"display\": {\n \"max_items\": 8,\n \"style\": \"detailed\"\n }\n }\n ]\n}","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Scripts Pipeline","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"run-pipeline.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Unified Pipeline (Recommended)","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/run-pipeline.py \\\n --defaults config/defaults [--config CONFIG_DIR] \\\n --hours 48 --freshness pd \\\n --archive-dir workspace/archive/tech-news-digest/ \\\n --output /tmp/td-merged.json --verbose --force","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Features","type":"text","marks":[{"type":"strong"}]},{"text":": Runs all 6 fetch steps in parallel, then merges + deduplicates + scores","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Output","type":"text","marks":[{"type":"strong"}]},{"text":": Final merged JSON ready for report generation (~30s total)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Metadata","type":"text","marks":[{"type":"strong"}]},{"text":": Saves per-step timing and counts to ","type":"text"},{"text":"*.meta.json","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"GitHub Auth","type":"text","marks":[{"type":"strong"}]},{"text":": Auto-generates GitHub App token if ","type":"text"},{"text":"$GITHUB_TOKEN","type":"text","marks":[{"type":"code_inline"}]},{"text":" not set","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Fallback","type":"text","marks":[{"type":"strong"}]},{"text":": If this fails, run individual scripts below","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Individual Scripts (Fallback)","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"fetch-rss.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" - RSS Feed Fetcher","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/fetch-rss.py [--defaults DIR] [--config DIR] [--hours 48] [--output FILE] [--verbose]","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Parallel fetching (10 workers, max 2 concurrent per host), retry with jittered backoff, feedparser + regex fallback","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Timeout: 30s per feed, ETag/Last-Modified caching in ","type":"text"},{"text":"$TECH_NEWS_DIGEST_STATE_DIR","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"fetch-twitter.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Twitter/X KOL Monitor","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/fetch-twitter.py [--defaults DIR] [--config DIR] [--hours 48] [--output FILE] [--backend auto|official|twitterapiio]","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Backend auto-detection: uses twitterapi.io if ","type":"text"},{"text":"TWITTERAPI_IO_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" set, else official X API v2 if ","type":"text"},{"text":"X_BEARER_TOKEN","type":"text","marks":[{"type":"code_inline"}]},{"text":" set","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Rate limit handling, engagement metrics, retry with backoff","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"fetch-web.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Web Search Engine","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/fetch-web.py [--defaults DIR] [--config DIR] [--freshness pd] [--output FILE]","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Auto-detects Brave API rate limit: paid plans → parallel queries, free → sequential","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Without API: generates search interface for agents","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"fetch-github.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" - GitHub Releases Monitor","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/fetch-github.py [--defaults DIR] [--config DIR] [--hours 168] [--output FILE]","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Parallel fetching (10 workers), 30s timeout","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Auth priority: ","type":"text"},{"text":"$GITHUB_TOKEN","type":"text","marks":[{"type":"code_inline"}]},{"text":" → GitHub App auto-generate → ","type":"text"},{"text":"gh","type":"text","marks":[{"type":"code_inline"}]},{"text":" CLI → ","type":"text"},{"text":"releases.atom","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Without a token (or on a 403/429 rate limit) it reads ","type":"text"},{"text":"github.com/\u003crepo>/releases.atom","type":"text","marks":[{"type":"code_inline"}]},{"text":", which needs no auth, consumes no REST quota, and also surfaces tag-only repos such as ","type":"text"},{"text":"torvalds/linux","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"The Atom feed lists ","type":"text"},{"text":"every","type":"text","marks":[{"type":"em"}]},{"text":" tag, so entries are kept only when the tag carries a ","type":"text"},{"text":"\u003cmajor>.\u003cminor>","type":"text","marks":[{"type":"code_inline"}]},{"text":" version (","type":"text"},{"text":"v2.13.0","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"v6.19-rc4","type":"text","marks":[{"type":"code_inline"}]},{"text":"). This drops CI tags such as PyTorch's ","type":"text"},{"text":"trunk/\u003csha>","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"viable/strict/\u003cepoch>","type":"text","marks":[{"type":"code_inline"}]},{"text":", which would otherwise crowd out real releases","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"fetch-github.py --trending","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Project Discovery (legacy flag)","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/fetch-github.py --trending [--hours 48] [--output FILE] [--verbose]","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Searches GitHub API for discovery candidates; search results do not establish genuine trends.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Neutral internal base score 5, not a growth signal. No ","type":"text"},{"text":"daily_stars_est","type":"text","marks":[{"type":"code_inline"}]},{"text":" or lifetime-derived growth. Daily discovery ≤3, weekly ≤3; require a current reason to try the project.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"fetch-reddit.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Reddit Posts Fetcher","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/fetch-reddit.py [--defaults DIR] [--config DIR] [--hours 48] [--output FILE]","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Parallel fetching (4 workers), public JSON API (no auth required)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"13 subreddits with score filtering","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"enrich-articles.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Article Full-Text Enrichment","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/enrich-articles.py --input merged.json --output enriched.json [--min-score 10] [--max-articles 15] [--verbose]","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Fetches full article text for high-scoring articles","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Cloudflare Markdown for Agents (preferred) → HTML extraction (fallback) → Skip (paywalled/social)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Blog domain whitelist with lower score threshold (≥3)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Parallel fetching (5 workers, 10s timeout)","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"merge-sources.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Quality Scoring & Deduplication","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/merge-sources.py --rss FILE --twitter FILE --web FILE --github FILE --reddit FILE","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Quality scoring, URL dedup (tracking params stripped, query preserved), title similarity dedup (75%), previous digest penalty","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Multi-source bonus: +5 per additional source type corroborating the same story","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Topic priority is derived from ","type":"text"},{"text":"topics.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" order; pass ","type":"text"},{"text":"--defaults","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"--config","type":"text","marks":[{"type":"code_inline"}]},{"text":" to keep it in sync","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Output: topic-grouped articles sorted by score","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"validate-config.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Configuration Validator","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/validate-config.py [--defaults DIR] [--config DIR] [--verbose]","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"JSON schema validation, topic reference checks, duplicate ID detection","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"generate-pdf.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" - PDF Report Generator","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/generate-pdf.py --input report.md --output digest.pdf [--verbose]","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Converts markdown digest to styled A4 PDF with Chinese typography (Noto Sans CJK SC)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Emoji icons, page headers/footers, blue accent theme. Requires ","type":"text"},{"text":"weasyprint","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"sanitize-html.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Safe HTML Email Converter","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/sanitize-html.py --input report.md --output email.html [--verbose]","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Converts markdown to XSS-safe HTML email with inline CSS","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"URL whitelist (http/https only), HTML-escaped text content","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"source-health.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Source Health Monitor","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/source-health.py --rss FILE --twitter FILE --github FILE --reddit FILE --web FILE [--verbose]","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Tracks per-source success/failure history over 7 days in ","type":"text"},{"text":"$TECH_NEWS_DIGEST_STATE_DIR/source-health.json","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Reports unhealthy sources (>50% failure rate, minimum 2 samples)","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"summarize-merged.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Merged Data Summary","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 scripts/summarize-merged.py --input merged.json [--top N] [--topic TOPIC]","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Human-readable summary of merged data for LLM consumption","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Shows top articles per topic with scores and metrics","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"User Customization","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Workspace Configuration Override","type":"text"}]},{"type":"paragraph","content":[{"text":"Place custom configs in ","type":"text"},{"text":"workspace/config/","type":"text","marks":[{"type":"code_inline"}]},{"text":" to override defaults:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Sources","type":"text","marks":[{"type":"strong"}]},{"text":": Append new sources, disable defaults with ","type":"text"},{"text":"\"enabled\": false","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Topics","type":"text","marks":[{"type":"strong"}]},{"text":": Override topic definitions, search queries, display settings","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Merge Logic","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Sources with same ","type":"text"},{"text":"id","type":"text","marks":[{"type":"code_inline"}]},{"text":" → user version takes precedence","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Sources with new ","type":"text"},{"text":"id","type":"text","marks":[{"type":"code_inline"}]},{"text":" → appended to defaults","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Topics with same ","type":"text"},{"text":"id","type":"text","marks":[{"type":"code_inline"}]},{"text":" → user version completely replaces default","type":"text"}]}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Example Workspace Override","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"json"},"content":[{"text":"// workspace/config/tech-news-digest-sources.json\n{\n \"sources\": [\n {\n \"id\": \"simonwillison-rss\",\n \"enabled\": false,\n \"note\": \"Disabled: too noisy for my use case\"\n },\n {\n \"id\": \"my-custom-blog\", \n \"type\": \"rss\",\n \"name\": \"My Custom Tech Blog\",\n \"url\": \"https://myblog.com/rss\",\n \"enabled\": true,\n \"priority\": true,\n \"topics\": [\"frontier-tech\"]\n }\n ]\n}","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Templates & Output","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Canonical Markdown (","type":"text"},{"text":"references/templates/markdown.md","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Daily/weekly structure governed by ","type":"text"},{"text":"references/digest-prompt.md","type":"text","marks":[{"type":"code_inline"}]},{"text":"; one validated selection for all formats.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Discord Template (","type":"text"},{"text":"references/templates/discord.md","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Plain bullets, bold titles, inline ","type":"text"},{"text":"[来源](\u003cURL>)","type":"text","marks":[{"type":"code_inline"}]},{"text":" links; section-aware chunks ≤1700 characters including numbering.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Email Template (","type":"text"},{"text":"references/templates/email.md","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Convert validated Markdown with ","type":"text"},{"text":"sanitize-html.py","type":"text","marks":[{"type":"code_inline"}]},{"text":", send the HTML file with ","type":"text"},{"text":"send-email.py","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Same selections, no extra metadata sections or operational statistics.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"PDF Template (","type":"text"},{"text":"references/templates/pdf.md","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Same canonical report, A4 Chinese typography and page numbers.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Generated via ","type":"text"},{"text":"scripts/generate-pdf.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" (requires ","type":"text"},{"text":"weasyprint","type":"text","marks":[{"type":"code_inline"}]},{"text":"); no PDF-only content.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Default Sources (216 enabled)","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"RSS Feeds (93)","type":"text","marks":[{"type":"strong"}]},{"text":": AI labs, arXiv, research blogs, tech news, crypto news, Chinese tech media, YouTube channels","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Twitter/X KOLs (76)","type":"text","marks":[{"type":"strong"}]},{"text":": AI researchers, model labs, crypto leaders, tech executives","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"GitHub Repos (47)","type":"text","marks":[{"type":"strong"}]},{"text":": Major open-source projects (llama.cpp, vLLM, SGLang, LangGraph, MCP, etc.)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Reddit (13, disabled by default)","type":"text","marks":[{"type":"strong"}]},{"text":": Reddit blocks datacenter IPs; set ","type":"text"},{"text":"REDDIT_CLIENT_ID","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"REDDIT_CLIENT_SECRET","type":"text","marks":[{"type":"code_inline"}]},{"text":" and re-enable in your overlay","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Web Search (4 topics)","type":"text","marks":[{"type":"strong"}]},{"text":": LLM, AI Agent, Crypto, Frontier Tech","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"A further 7 RSS/GitHub defaults ship disabled because the upstream feed is dead; each carries a ","type":"text"},{"text":"note","type":"text","marks":[{"type":"code_inline"}]},{"text":" saying why. All sources pre-configured with appropriate topic tags and priority levels.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Dependencies","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"pip install -r requirements.txt","type":"text"}]},{"type":"paragraph","content":[{"text":"Optional but Recommended","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"feedparser>=6.0.0","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Better RSS parsing (fallback to regex if unavailable)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"jsonschema>=4.0.0","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Configuration validation","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"All scripts work with Python 3.8+ standard library only.","type":"text","marks":[{"type":"strong"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Monitoring & Operations","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Health Checks","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Validate configuration\npython3 scripts/validate-config.py --verbose\n\n# Test RSS feeds\npython3 scripts/fetch-rss.py --hours 1 --verbose\n\n# Check Twitter API\npython3 scripts/fetch-twitter.py --hours 1 --verbose","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Archive Management","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Digests automatically archived to ","type":"text"},{"text":"\u003cworkspace>/archive/tech-news-digest/","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Previous digest titles used for duplicate detection","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Old archives cleaned automatically (90+ days)","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Error Handling","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Network Failures","type":"text","marks":[{"type":"strong"}]},{"text":": Retry with exponential backoff","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Rate Limits","type":"text","marks":[{"type":"strong"}]},{"text":": Automatic retry with appropriate delays","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Invalid Content","type":"text","marks":[{"type":"strong"}]},{"text":": Graceful degradation, detailed logging","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Configuration Errors","type":"text","marks":[{"type":"strong"}]},{"text":": Schema validation with helpful messages","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"API Keys & Environment","type":"text"}]},{"type":"paragraph","content":[{"text":"Set in ","type":"text"},{"text":"~/.zshenv","type":"text","marks":[{"type":"code_inline"}]},{"text":" or similar:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Twitter (at least one required for Twitter source)\nexport TWITTERAPI_IO_KEY=\"your_key\" # twitterapi.io key (preferred)\nexport X_BEARER_TOKEN=\"your_bearer_token\" # Official X API v2 (fallback)\nexport TWITTER_API_BACKEND=\"auto\" # auto|twitterapiio|official (default: auto)\n\n# Web Search (optional, enables web search layer)\nexport WEB_SEARCH_BACKEND=\"auto\" # auto|brave|tavily (default: auto)\nexport TAVILY_API_KEY=\"tvly-xxx\" # Tavily Search API (free 1000/mo)\n\n# Brave Search (alternative)\nexport BRAVE_API_KEYS=\"key1,key2,key3\" # Multiple keys, comma-separated rotation\nexport BRAVE_API_KEY=\"key1\" # Single key fallback\nexport BRAVE_PLAN=\"free\" # Override rate limit detection: free|pro\n\n# Reddit (required to enable the Reddit layer — it ships disabled)\nexport REDDIT_CLIENT_ID=\"xxx\" # Script-type app from reddit.com/prefs/apps\nexport REDDIT_CLIENT_SECRET=\"xxx\"\nexport REDDIT_USERNAME=\"xxx\" # Optional, User-Agent string only\n\n# GitHub (optional, unlocks richer release notes)\nexport GITHUB_TOKEN=\"ghp_xxx\" # PAT (simplest)\nexport GH_APP_ID=\"12345\" # Or use GitHub App for auto-token\nexport GH_APP_INSTALL_ID=\"67890\"\nexport GH_APP_KEY_FILE=\"/path/to/key.pem\"","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Twitter","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"TWITTERAPI_IO_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":" preferred ($3-5/mo); ","type":"text"},{"text":"X_BEARER_TOKEN","type":"text","marks":[{"type":"code_inline"}]},{"text":" as fallback; ","type":"text"},{"text":"auto","type":"text","marks":[{"type":"code_inline"}]},{"text":" mode tries twitterapiio first","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Web Search","type":"text","marks":[{"type":"strong"}]},{"text":": Tavily (preferred in auto mode) or Brave; optional, fallback to agent web_search if unavailable","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"GitHub","type":"text","marks":[{"type":"strong"}]},{"text":": Auto-generates token from GitHub App if PAT not set; unauthenticated fallback (60 req/hr)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Reddit","type":"text","marks":[{"type":"strong"}]},{"text":": Disabled by default — Reddit returns 403 to datacenter IPs. Set ","type":"text"},{"text":"REDDIT_CLIENT_ID","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"REDDIT_CLIENT_SECRET","type":"text","marks":[{"type":"code_inline"}]},{"text":" and re-enable the sources to use the OAuth API","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Cron / Scheduled Task Integration","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"OpenClaw Cron (Recommended)","type":"text"}]},{"type":"paragraph","content":[{"text":"The cron prompt should ","type":"text"},{"text":"NOT","type":"text","marks":[{"type":"strong"}]},{"text":" hardcode the pipeline steps. Instead, reference ","type":"text"},{"text":"references/digest-prompt.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" and only pass configuration parameters. This ensures the pipeline logic stays in the skill repo and is consistent across all installations.","type":"text"}]},{"type":"paragraph","content":[{"text":"Existing cron jobs may still pass ","type":"text"},{"text":"ITEMS_PER_SECTION","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"BLOG_PICKS_COUNT","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"EXTRA_SECTIONS","type":"text","marks":[{"type":"code_inline"}]},{"text":". Accept them for compatibility but ignore their quotas/extra sections; mode policy supersedes them. New examples omit them.","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Daily Digest Cron Prompt","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Read \u003cSKILL_DIR>/references/digest-prompt.md and follow the complete workflow to generate a daily digest.\n\nReplace placeholders with:\n- MODE = daily\n- TIME_WINDOW = past 1-2 days\n- FRESHNESS = pd\n- RSS_HOURS = 48\n- ENRICH = true\n- SUBJECT = Daily Tech Digest - YYYY-MM-DD\n- WORKSPACE = \u003cyour workspace path>\n- SKILL_DIR = \u003cyour skill install path>\n- DISCORD_CHANNEL_ID = \u003cyour channel id>\n- EMAIL = (optional)\n- LANGUAGE = English\n- TEMPLATE = discord\n\nFollow every step in the prompt template strictly. Do not skip any steps.","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Weekly Digest Cron Prompt","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Read \u003cSKILL_DIR>/references/digest-prompt.md and follow the complete workflow to generate a weekly digest.\n\nReplace placeholders with:\n- MODE = weekly\n- TIME_WINDOW = past 7 days\n- FRESHNESS = pw\n- RSS_HOURS = 168\n- ENRICH = true\n- SUBJECT = Weekly Tech Digest - YYYY-MM-DD\n- WORKSPACE = \u003cyour workspace path>\n- SKILL_DIR = \u003cyour skill install path>\n- DISCORD_CHANNEL_ID = \u003cyour channel id>\n- EMAIL = (optional)\n- LANGUAGE = English\n- TEMPLATE = discord\n\nFollow every step in the prompt template strictly. Do not skip any steps.","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Why This Pattern?","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Single source of truth","type":"text","marks":[{"type":"strong"}]},{"text":": Pipeline logic lives in ","type":"text"},{"text":"digest-prompt.md","type":"text","marks":[{"type":"code_inline"}]},{"text":", not scattered across cron configs","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Portable","type":"text","marks":[{"type":"strong"}]},{"text":": Same skill on different OpenClaw instances, just change paths and channel IDs","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Maintainable","type":"text","marks":[{"type":"strong"}]},{"text":": Update the skill → all cron jobs pick up changes automatically","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Anti-pattern","type":"text","marks":[{"type":"strong"}]},{"text":": Do NOT copy pipeline steps into the cron prompt — it will drift out of sync","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Multi-Channel Delivery Limitation","type":"text"}]},{"type":"paragraph","content":[{"text":"OpenClaw enforces ","type":"text"},{"text":"cross-provider isolation","type":"text","marks":[{"type":"strong"}]},{"text":": a single session can only send messages to one provider (e.g., Discord OR Telegram, not both). If you need to deliver digests to multiple platforms, create ","type":"text"},{"text":"separate cron jobs","type":"text","marks":[{"type":"strong"}]},{"text":" for each provider:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"# Job 1: Discord + Email\n- DISCORD_CHANNEL_ID = \u003cyour-discord-channel-id>\n- EMAIL = [email protected]\n- TEMPLATE = discord\n\n# Job 2: Telegram DM\n- DISCORD_CHANNEL_ID = (none)\n- EMAIL = (none)\n- TEMPLATE = telegram","type":"text"}]},{"type":"paragraph","content":[{"text":"Replace ","type":"text"},{"text":"DISCORD_CHANNEL_ID","type":"text","marks":[{"type":"code_inline"}]},{"text":" delivery with the target platform's delivery in the second job's prompt.","type":"text"}]},{"type":"paragraph","content":[{"text":"This is a security feature, not a bug — it prevents accidental cross-context data leakage.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Security Notes","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Execution Model","type":"text"}]},{"type":"paragraph","content":[{"text":"This skill uses a ","type":"text"},{"text":"prompt template pattern","type":"text","marks":[{"type":"strong"}]},{"text":": the agent reads ","type":"text"},{"text":"digest-prompt.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" and follows its instructions. This is the standard OpenClaw skill execution model — the agent interprets structured instructions from skill-provided files. All instructions are shipped with the skill bundle and can be audited before installation.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Network Access","type":"text"}]},{"type":"paragraph","content":[{"text":"The Python scripts make outbound requests to:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"RSS feed URLs (configured in ","type":"text"},{"text":"tech-news-digest-sources.json","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Twitter/X API (","type":"text"},{"text":"api.x.com","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"api.twitterapi.io","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Brave Search API (","type":"text"},{"text":"api.search.brave.com","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Tavily Search API (","type":"text"},{"text":"api.tavily.com","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"GitHub API (","type":"text"},{"text":"api.github.com","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Reddit JSON API (","type":"text"},{"text":"reddit.com","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"No data is sent to any other endpoints. All API keys are read from environment variables declared in the skill metadata.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Shell Safety","type":"text"}]},{"type":"paragraph","content":[{"text":"Email delivery uses ","type":"text"},{"text":"send-email.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" which constructs proper MIME multipart messages with HTML body + optional PDF attachment. Subject formats are hardcoded (","type":"text"},{"text":"Daily Tech Digest - YYYY-MM-DD","type":"text","marks":[{"type":"code_inline"}]},{"text":"). PDF generation uses ","type":"text"},{"text":"generate-pdf.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" via ","type":"text"},{"text":"weasyprint","type":"text","marks":[{"type":"code_inline"}]},{"text":". The prompt template explicitly prohibits interpolating untrusted content (article titles, tweet text, etc.) into shell arguments. Email addresses and subjects must be static placeholder values only.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"File Access","type":"text"}]},{"type":"paragraph","content":[{"text":"Scripts read from ","type":"text"},{"text":"config/","type":"text","marks":[{"type":"code_inline"}]},{"text":" and write to ","type":"text"},{"text":"workspace/archive/","type":"text","marks":[{"type":"code_inline"}]},{"text":". No files outside the workspace are accessed.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Support & Troubleshooting","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Common Issues","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"RSS feeds failing","type":"text","marks":[{"type":"strong"}]},{"text":": Check network connectivity, use ","type":"text"},{"text":"--verbose","type":"text","marks":[{"type":"code_inline"}]},{"text":" for details","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Twitter rate limits","type":"text","marks":[{"type":"strong"}]},{"text":": Reduce sources or increase interval","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Configuration errors","type":"text","marks":[{"type":"strong"}]},{"text":": Run ","type":"text"},{"text":"validate-config.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" for specific issues","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"No articles found","type":"text","marks":[{"type":"strong"}]},{"text":": Check time window (","type":"text"},{"text":"--hours","type":"text","marks":[{"type":"code_inline"}]},{"text":") and source enablement","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Debug Mode","type":"text"}]},{"type":"paragraph","content":[{"text":"All scripts support ","type":"text"},{"text":"--verbose","type":"text","marks":[{"type":"code_inline"}]},{"text":" flag for detailed logging and troubleshooting.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Performance Tuning","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Parallel Workers","type":"text","marks":[{"type":"strong"}]},{"text":": Adjust ","type":"text"},{"text":"MAX_WORKERS","type":"text","marks":[{"type":"code_inline"}]},{"text":" in scripts for your system","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Timeout Settings","type":"text","marks":[{"type":"strong"}]},{"text":": Increase ","type":"text"},{"text":"TIMEOUT","type":"text","marks":[{"type":"code_inline"}]},{"text":" for slow networks","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Article Limits","type":"text","marks":[{"type":"strong"}]},{"text":": Adjust ","type":"text"},{"text":"MAX_ARTICLES_PER_FEED","type":"text","marks":[{"type":"code_inline"}]},{"text":" based on needs","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Security Considerations","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Shell Execution","type":"text"}]},{"type":"paragraph","content":[{"text":"The digest prompt instructs agents to run Python scripts via shell commands. All script paths and arguments are skill-defined constants — no user input is interpolated into commands. Two scripts use ","type":"text"},{"text":"subprocess","type":"text","marks":[{"type":"code_inline"}]},{"text":":","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"run-pipeline.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" orchestrates child fetch scripts (all within ","type":"text"},{"text":"scripts/","type":"text","marks":[{"type":"code_inline"}]},{"text":" directory)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"fetch-github.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" has two subprocess calls:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"openssl dgst -sha256 -sign","type":"text","marks":[{"type":"code_inline"}]},{"text":" for JWT signing (only if ","type":"text"},{"text":"GH_APP_*","type":"text","marks":[{"type":"code_inline"}]},{"text":" env vars are set — signs a self-constructed JWT payload, no user content involved)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"gh auth token","type":"text","marks":[{"type":"code_inline"}]},{"text":" CLI fallback (only if ","type":"text"},{"text":"gh","type":"text","marks":[{"type":"code_inline"}]},{"text":" is installed — reads from gh's own credential store)","type":"text"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"No user-supplied or fetched content is ever interpolated into subprocess arguments. Email delivery uses ","type":"text"},{"text":"send-email.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" which builds MIME messages programmatically — no shell interpolation. PDF generation uses ","type":"text"},{"text":"generate-pdf.py","type":"text","marks":[{"type":"code_inline"}]},{"text":" via ","type":"text"},{"text":"weasyprint","type":"text","marks":[{"type":"code_inline"}]},{"text":". Email subjects are static format strings only — never constructed from fetched data.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Credential & File Access","type":"text"}]},{"type":"paragraph","content":[{"text":"Scripts do ","type":"text"},{"text":"not","type":"text","marks":[{"type":"strong"}]},{"text":" directly read ","type":"text"},{"text":"~/.config/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"~/.ssh/","type":"text","marks":[{"type":"code_inline"}]},{"text":", or any credential files. All API tokens are read from environment variables declared in the skill metadata. The GitHub auth cascade is:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"$GITHUB_TOKEN","type":"text","marks":[{"type":"code_inline"}]},{"text":" env var (you control what to provide)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"GitHub App token generation (only if you set ","type":"text"},{"text":"GH_APP_ID","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"GH_APP_INSTALL_ID","type":"text","marks":[{"type":"code_inline"}]},{"text":", and ","type":"text"},{"text":"GH_APP_KEY_FILE","type":"text","marks":[{"type":"code_inline"}]},{"text":" — uses inline JWT signing via ","type":"text"},{"text":"openssl","type":"text","marks":[{"type":"code_inline"}]},{"text":" CLI, no external scripts involved)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"gh auth token","type":"text","marks":[{"type":"code_inline"}]},{"text":" CLI (delegates to gh's own secure credential store)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Unauthenticated (60 req/hr, safe fallback)","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"If you prefer no automatic credential discovery, simply set ","type":"text"},{"text":"$GITHUB_TOKEN","type":"text","marks":[{"type":"code_inline"}]},{"text":" and the script will use it directly without attempting steps 2-3.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Dependency Installation","type":"text"}]},{"type":"paragraph","content":[{"text":"This skill does ","type":"text"},{"text":"not","type":"text","marks":[{"type":"strong"}]},{"text":" install any packages. ","type":"text"},{"text":"requirements.txt","type":"text","marks":[{"type":"code_inline"}]},{"text":" lists optional dependencies (","type":"text"},{"text":"feedparser","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"jsonschema","type":"text","marks":[{"type":"code_inline"}]},{"text":") for reference only. All scripts work with Python 3.8+ standard library. Users should install optional deps in a virtualenv if desired — the skill never runs ","type":"text"},{"text":"pip install","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Input Sanitization","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"URL resolution rejects non-HTTP(S) schemes (javascript:, data:, etc.)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"RSS fallback parsing uses simple, non-backtracking regex patterns (no ReDoS risk)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"All fetched content is treated as untrusted data for display only","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Network Access","type":"text"}]},{"type":"paragraph","content":[{"text":"Scripts make outbound HTTP requests to configured RSS feeds, Twitter API, GitHub API, Reddit JSON API, Brave Search API, and Tavily Search API. No inbound connections or listeners are created.","type":"text"}]}]},"metadata":{"env":[{"name":"TWITTER_API_BACKEND","required":false,"description":"Twitter API backend: 'official', 'twitterapiio', or 'auto' (default: auto)"},{"name":"X_BEARER_TOKEN","required":false,"description":"Twitter/X API bearer token for KOL monitoring (official backend)"},{"name":"TWITTERAPI_IO_KEY","required":false,"description":"twitterapi.io API key for KOL monitoring (twitterapiio backend)"},{"name":"TAVILY_API_KEY","required":false,"description":"Tavily Search API key (alternative to Brave)"},{"name":"WEB_SEARCH_BACKEND","required":false,"description":"Web search backend: auto (default), brave, or tavily"},{"name":"BRAVE_API_KEYS","required":false,"description":"Brave Search API keys (comma-separated for rotation)"},{"name":"BRAVE_API_KEY","required":false,"description":"Brave Search API key (single key fallback)"},{"name":"REDDIT_CLIENT_ID","required":false,"description":"Reddit API app client ID for OAuth fallback when public JSON is blocked"},{"name":"REDDIT_CLIENT_SECRET","required":false,"description":"Reddit API app client secret for OAuth fallback when public JSON is blocked"},{"name":"REDDIT_USERNAME","required":false,"description":"Optional Reddit username used only in the OAuth User-Agent string"},{"name":"GITHUB_TOKEN","required":false,"description":"GitHub token for higher API rate limits (auto-generated from GitHub App if not set)"},{"name":"GH_APP_ID","required":false,"description":"GitHub App ID for automatic installation token generation"},{"name":"GH_APP_INSTALL_ID","required":false,"description":"GitHub App Installation ID for automatic token generation"},{"name":"GH_APP_KEY_FILE","required":false,"description":"Path to GitHub App private key PEM file"},{"name":"TECH_NEWS_DIGEST_STATE_DIR","required":false,"description":"Directory for cross-run state (RSS/GitHub caches, source health). Default: $XDG_STATE_HOME/tech-news-digest or ~/.local/state/tech-news-digest"}],"date":"2026-11-02","name":"tech-news-digest","files":{"read":[{"config/defaults/":"Default source and topic configurations"},{"references/":"Prompt templates and output templates"},{"scripts/":"Python pipeline scripts"},{"\u003cworkspace>/archive/tech-news-digest/":"Previous digests for dedup"}],"write":[{"$TECH_NEWS_DIGEST_STATE_DIR/":"RSS/GitHub caches and source health history"},{"/tmp/td-*.json":"Temporary pipeline intermediate outputs"},{"/tmp/td-email.html":"Temporary email HTML body"},{"/tmp/td-digest.pdf":"Generated PDF digest"},{"\u003cworkspace>/archive/tech-news-digest/":"Saved digest archives"}]},"tools":[{"python3":"Required. Runs data collection and merge scripts."},{"mail":"Optional. msmtp-based mail command for email delivery (preferred)."},{"gog":"Optional. Gmail CLI for email delivery (fallback if mail not available)."}],"author":"@skillopedia","source":{"stars":77,"repo_name":"tech-news-digest","origin_url":"https://github.com/draco-agent/tech-news-digest/blob/HEAD/SKILL.md","repo_owner":"draco-agent","body_sha256":"f84a8e03e560be72c13e6bbef480a2520463b542c385ade64453edd651b912fd","cluster_key":"d3f76bf3ae8d859ae4c3d0944a909529dc221e1e95332893ebfe92b266e1c955","clean_bundle":{"format":"clean-skill-bundle-v1","source":"draco-agent/tech-news-digest/SKILL.md","text_bytes":312799,"source_commit":"c3aeafb940221debd597ed2963dd783755757f4c","skills_sh_rank":15163,"attachment_count":22,"text_attachments":22,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"3.17.1","category":"media-content","category_label":"Media"},"exact_dupes_collapsed_into_this":0},"version":"3.17.1","category":"media-content","homepage":"https://github.com/draco-agent/tech-news-digest","metadata":{"openclaw":{"requires":{"bins":["python3"]},"optionalBins":["mail","msmtp","gog","gh","openssl","weasyprint"]}},"import_tag":"skills-sh-top20k-2026-11","description":"Generate tech news digests with unified source model, quality scoring, and multi-format output. Six-source data collection from RSS feeds, Twitter/X, GitHub releases, project discovery, Reddit, and web search. Pipeline-based scripts with retry mechanisms and deduplication. Supports Discord, email, and markdown templates."}},"renderedAt":1793661734119}

Tech News Digest Evidence-led tech digest with unified collection and one editorial selection shared across formats. Follow as the authoritative mode policy. Editorial Contract - Daily preserves original sections: 2–4 sentence overview; configured topics (normally LLM, AI Agent, Crypto, Frontier Tech; aim for 3–5 qualified items each); KOL viewpoints ≤3; releases ≤3 repositories; project discovery ≤3; blog picks ≤3; ≤30 unique items overall. News/blog explanations normally contain 2–3 concise sentences of change, evidence and implication. Retain empty headings with a coverage note, never fill…