Markdown Proxy - URL to Markdown 将任意 URL 转为干净的 Markdown。支持需要登录的页面、PDF、专有平台。 Trigger Priority（先触发，再分流） - 用户给出 URL 并要求“读取、抓取、提取、解析、转 Markdown”时，优先使用本 Skill。 - 用户要求“读取这篇链接后再写稿、总结、翻译、做播客或分析”时，先用本 Skill 取回原文，再把 Markdown 交给下游 Skill；不要因为最终任务是写作而跳过抓取。 - 和飞书文档属于专用路由。不要先试普通网页打开或通用内容解析器。 - 用户已经贴出全文时不触发。YouTube 交给 ，普通搜索问题交给搜索工具。 URL Routing (先判断再执行) 收到 URL 后，先判断类型，不同类型走不同通道： | URL Pattern | Route To | Reason | |-------------|----------|--------| | | | 先代理，验证码页自动回退 Playwright | | | | 需飞书 API 认证 | | | skill | YouTube 有专用工具链 | | | 提取 arXiv ID → | HuggingFace 论文页实际是 arXiv 镜像，先找到 arXiv 链接再走 LaTeX 提取 | | | | 从…

,\n ]\n for pat in patterns:\n m = re.search(pat, input_str)\n if m:\n return m.group(1)\n raise ValueError(f\"Cannot extract arXiv ID from: {input_str}\")\n\n\ndef download_source(arxiv_id, tmpdir):\n \"\"\"Download arXiv LaTeX source tarball.\"\"\"\n url = f\"https://arxiv.org/e-print/{arxiv_id}\"\n req = Request(url, headers={\"User-Agent\": \"arxiv-tex-extractor/1.0\"})\n\n # Try tarball first, fall back to PDF-only\n try:\n resp = urlopen(req, timeout=30)\n data = resp.read()\n except URLError as e:\n # Maybe the paper only has PDF source\n raise RuntimeError(f\"Failed to download source for {arxiv_id}: {e}\")\n\n # Check if we got a gzipped tarball or PDF\n if data[:4] == b'\\x1f\\x8b': # gzip magic\n tarpath = Path(tmpdir) / \"source.tar.gz\"\n tarpath.write_bytes(data)\n with tarfile.open(tarpath, \"r:gz\") as tar:\n tar.extractall(tmpdir, filter='data')\n tarpath.unlink()\n return True\n elif data[:4] == b'%PDF': # PDF - no TeX source available\n return False\n else:\n # Try as uncompressed tar\n try:\n tarpath = Path(tmpdir) / \"source.tar\"\n tarpath.write_bytes(data)\n with tarfile.open(tarpath, \"r\") as tar:\n tar.extractall(tmpdir, filter='data')\n tarpath.unlink()\n return True\n except tarfile.ReadError:\n return False\n\n\ndef find_main_tex(tmpdir):\n \"\"\"Find the main .tex file in extracted source.\"\"\"\n tex_files = list(Path(tmpdir).glob(\"**/*.tex\"))\n if not tex_files:\n return None\n\n # Priority: filename starts with \"main\", then largest file, then any\n main_names = [f for f in tex_files if f.stem.lower() in (\"main\", \"paper\", \"article\", \"root\")]\n if main_names:\n return main_names[0]\n\n # Look for \\documentclass in files\n for f in tex_files:\n try:\n content = f.read_text(errors=\"replace\")\n if r\"\\begin{document}\" in content:\n return f\n except Exception:\n continue\n\n # Fallback: largest .tex file\n return max(tex_files, key=lambda f: f.stat().st_size)\n\n\ndef strip_latex_comments(content):\n \"\"\"Remove LaTeX comments (lines starting with %, ignoring escaped %).\"\"\"\n lines = []\n for line in content.split(\"\\n\"):\n # Remove comments but not \\%\n stripped = re.sub(r'(?\u003c!\\\\)%.*

Markdown Proxy - URL to Markdown 将任意 URL 转为干净的 Markdown。支持需要登录的页面、PDF、专有平台。 Trigger Priority（先触发，再分流） - 用户给出 URL 并要求“读取、抓取、提取、解析、转 Markdown”时，优先使用本 Skill。 - 用户要求“读取这篇链接后再写稿、总结、翻译、做播客或分析”时，先用本 Skill 取回原文，再把 Markdown 交给下游 Skill；不要因为最终任务是写作而跳过抓取。 - 和飞书文档属于专用路由。不要先试普通网页打开或通用内容解析器。 - 用户已经贴出全文时不触发。YouTube 交给 ，普通搜索问题交给搜索工具。 URL Routing (先判断再执行) 收到 URL 后，先判断类型，不同类型走不同通道： | URL Pattern | Route To | Reason | |-------------|----------|--------| | | | 先代理，验证码页自动回退 Playwright | | | | 需飞书 API 认证 | | | skill | YouTube 有专用工具链 | | | 提取 arXiv ID → | HuggingFace 论文页实际是 arXiv 镜像，先找到 arXiv 链接再走 LaTeX 提取 | | | | 从…

, '', line)\n if stripped.strip() or line.strip() == \"\":\n lines.append(stripped)\n return \"\\n\".join(lines)\n\n\ndef resolve_inputs(content, base_dir):\n r\"\"\"Resolve \\input{xxx} and \\include{xxx} recursively.\"\"\"\n base = Path(base_dir)\n\n def _resolve(match):\n fname = match.group(1).strip()\n # Try with and without .tex extension\n for suffix in (\".tex\", \"\"):\n path = base / (fname + suffix) if suffix else base / fname\n if not path.exists():\n # Try relative to base_dir subdirs\n for subdir in base.glob(\"*/\"):\n path = subdir / (fname + (suffix or \"\"))\n if not path.exists():\n continue\n if path.exists():\n try:\n sub = path.read_text(errors=\"replace\")\n sub = strip_latex_comments(sub)\n # Strip any \\documentclass / preamble from sub-file\n sub = re.sub(r'^.*?\\\\begin\\{document\\}', '', sub, flags=re.DOTALL)\n sub = re.sub(r'\\\\end\\{document\\}.*

Markdown Proxy - URL to Markdown 将任意 URL 转为干净的 Markdown。支持需要登录的页面、PDF、专有平台。 Trigger Priority（先触发，再分流） - 用户给出 URL 并要求“读取、抓取、提取、解析、转 Markdown”时，优先使用本 Skill。 - 用户要求“读取这篇链接后再写稿、总结、翻译、做播客或分析”时，先用本 Skill 取回原文，再把 Markdown 交给下游 Skill；不要因为最终任务是写作而跳过抓取。 - 和飞书文档属于专用路由。不要先试普通网页打开或通用内容解析器。 - 用户已经贴出全文时不触发。YouTube 交给 ，普通搜索问题交给搜索工具。 URL Routing (先判断再执行) 收到 URL 后，先判断类型，不同类型走不同通道： | URL Pattern | Route To | Reason | |-------------|----------|--------| | | | 先代理，验证码页自动回退 Playwright | | | | 需飞书 API 认证 | | | skill | YouTube 有专用工具链 | | | 提取 arXiv ID → | HuggingFace 论文页实际是 arXiv 镜像，先找到 arXiv 链接再走 LaTeX 提取 | | | | 从…

, '', sub, flags=re.DOTALL)\n # Remove preamble commands\n sub = re.sub(r'\\\\usepackage(\\[.*?\\])?\\{.*?\\}', '', sub)\n return resolve_inputs(sub, base_dir)\n except Exception:\n pass\n # File not found in any subdir of base_dir\n return f\"[引用文件: {fname}]\"\n\n # Handle \\include and \\input\n content = re.sub(r'\\\\(?:input|include)\\{([^}]+)\\}', _resolve, content)\n return content\n\n\ndef strip_body_formatting(text):\n \"\"\"Remove LaTeX formatting commands from body text, keeping content.\"\"\"\n # Order matters\n\n # Remove \\begin{xxx}...\\end{xxx} for layout-only envs (center, flushleft, etc)\n # Keep equation, figure, table, align, gather for structural parsing\n text = re.sub(r'\\\\begin\\{(?:center|flushleft|flushright|itemize|enumerate)\\}(.*?)\\\\end\\{(?:center|flushleft|flushright|itemize|enumerate)\\}',\n lambda m: m.group(1), text, flags=re.DOTALL)\n\n # \\item → bullet\n text = re.sub(r'\\\\item\\s+', '\\n- ', text)\n text = re.sub(r'\\\\item\\[(.*?)\\]', r'\\n- **\\1**: ', text)\n\n # \\paragraph{...} → **...**\\n\n text = re.sub(r'\\\\paragraph\\{([^}]+)\\}', r'\\n**\\1** ', text)\n # \\textbf{...} → **...**\n text = re.sub(r'\\\\textbf\\{([^}]+)\\}', r'**\\1**', text)\n # \\emph{...} → *...*\n text = re.sub(r'\\\\emph\\{([^}]+)\\}', r'*\\1*', text)\n # \\textit{...} → *...*\n text = re.sub(r'\\\\textit\\{([^}]+)\\}', r'*\\1*', text)\n\n # \\url{...} → keep URL\n text = re.sub(r'\\\\url\\{([^}]+)\\}', r'\\1', text)\n # \\href{url}{text} → [text](url)\n text = re.sub(r'\\\\href\\{([^}]+)\\}\\{([^}]+)\\}', r'[\\2](\\1)', text)\n\n # \\cite{...} → [文献引用]\n text = re.sub(r'\\\\cite(?:\\[.*?\\])?\\{([^}]+)\\}', '[文献]', text)\n text = re.sub(r'\\\\citep(?:\\[.*?\\])?\\{([^}]+)\\}', '[文献]', text)\n text = re.sub(r'\\\\citet(?:\\[.*?\\])?\\{([^}]+)\\}', '[文献]', text)\n\n # \\ref{...} → [引用]\n text = re.sub(r'\\\\ref\\{([^}]+)\\}', r'[\\1]', text)\n text = re.sub(r'\\\\label\\{[^}]*\\}', '', text)\n\n # \\footnote{...} → (注: ...)\n text = re.sub(r'\\\\footnote\\{([^}]*(?:\\{[^}]*\\}[^}]*)*)\\}',\n r'(注: \\1)', text)\n\n # Remove common formatting commands (keep their content)\n simple_cmds = [\n 'texttt', 'textrm', 'textsf', 'textsc', 'underline',\n 'text', 'mbox', 'small', 'large', 'Large', 'LARGE',\n 'tiny', 'footnotesize', 'normalsize', 'sc', 'bf',\n 'it', 'tt', 'rm', 'sf',\n ]\n for cmd in simple_cmds:\n text = re.sub(rf'\\\\{cmd}\\{{([^}}]*(?:\\{{[^}}]*\\}}[^}}]*)*)\\}}',\n r'\\1', text)\n\n # Remove simple commands with no args\n text = re.sub(r'\\\\(?:noindent|newpage|clearpage|pagebreak|linebreak)\\b', '', text)\n text = re.sub(r'\\\\vspace\\{[^}]*\\}', '', text)\n text = re.sub(r'\\\\hspace\\{[^}]*\\}', '', text)\n\n # Clean up LaTeX special chars\n text = text.replace(r'\\&', '&')\n text = text.replace(r'\\%', '%')\n text = text.replace(r'\\#', '#')\n text = text.replace(r'\\_', '_')\n text = text.replace(r'\\~', '~')\n text = text.replace('~', ' ')\n text = text.replace(r'\\ ', ' ')\n text = text.replace(r'``', '\"')\n text = text.replace(\"''\", '\"')\n\n # Remove remaining simple \\begin/\\end for table-aligned envs (tabularx, etc.)\n text = re.sub(r'\\\\begin\\{(?:tabular|tabularx|tabu|longtable|wraptable)\\*?\\}.*?\\\\end\\{(?:tabular|tabularx|tabu|longtable|wraptable)\\*?\\}',\n '[表格数据]', text, flags=re.DOTALL)\n\n # Clean up whitespace\n text = re.sub(r'\\n{3,}', '\\n\\n', text)\n text = re.sub(r' +', ' ', text)\n\n return text\n\n\ndef _find_matching_brace(text, start):\n \"\"\"Find the matching closing brace for an opening brace at start-1.\n Returns the index of the closing brace, or -1 if not found.\"\"\"\n depth = 0\n i = start\n while i \u003c len(text):\n if text[i] == '{':\n depth += 1\n elif text[i] == '}':\n depth -= 1\n if depth == 0:\n return i\n i += 1\n return -1\n\ndef extract_title_author(content):\n \"\"\"Extract title and author from LaTeX preamble.\"\"\"\n title_match = re.search(r'\\\\title\\{((?:[^{}]|\\{[^{}]*\\})*)\\}', content, re.DOTALL)\n title = title_match.group(1).strip() if title_match else \"\"\n\n # Clean title\n title = strip_body_formatting(title)\n title = re.sub(r'\\s+', ' ', title).strip()\n\n # Extract authors using brace counting (handles nested braces in \\thanks etc.)\n authors = []\n author_cmd = r'\\author{'\n author_start = content.find(author_cmd)\n if author_start >= 0:\n brace_start = author_start + len(author_cmd) - 1 # position OF '{'\n brace_end = _find_matching_brace(content, brace_start)\n if brace_end >= 0:\n author_text = content[brace_start + 1:brace_end]\n\n # Remove \\thanks{...} blocks (content may span multiple lines)\n author_text = re.sub(r'\\\\thanks\\{(?:[^{}]|\\{[^{}]*\\})*\\}', '', author_text, flags=re.DOTALL)\n # Remove \\footnotemark[...]\n author_text = re.sub(r'\\\\footnotemark\\[.*?\\]', '', author_text)\n # Normalize \\And/\\AND/\\and to separator\n author_text = re.sub(r'\\\\AND\\b', '|||', author_text)\n author_text = re.sub(r'\\\\And\\b', '|||', author_text)\n author_text = re.sub(r'\\\\and\\b', '|||', author_text)\n\n for part in author_text.split('|||'):\n # Strip LaTeX line breaks\n part = re.sub(r'\\\\\\\\[ \\t]*', ' ', part)\n # Remove brace groups (affiliations, formatting)\n part = re.sub(r'\\{[^}]*\\}', '', part)\n # Remove remaining LaTeX commands\n part = re.sub(r'\\\\[a-zA-Z]+\\b', '', part)\n part = strip_body_formatting(part.strip())\n part = re.sub(r'\\s+', ' ', part).strip()\n # Remove emails, URLs, stray braces\n part = re.sub(r'\\S+@\\S+', '', part)\n part = re.sub(r'http\\S+', '', part)\n part = re.sub(r'[{}]', '', part)\n part = part.strip(',. ')\n if part and len(part) > 3:\n authors.append(part)\n\n return title, authors\n\n\ndef extract_abstract(content):\n \"\"\"Extract abstract from document body.\"\"\"\n m = re.search(r'\\\\begin\\{abstract\\}(.*?)\\\\end\\{abstract\\}', content, re.DOTALL)\n if m:\n return strip_body_formatting(m.group(1).strip())\n return \"\"\n\n\ndef extract_table_data(table_body):\n \"\"\"Parse LaTeX tabular data into markdown table.\"\"\"\n lines = []\n inside = False\n for line in table_body.split(\"\\n\"):\n line = line.strip()\n if line.startswith(r\"\\begin{tabular\"):\n inside = True\n continue\n if line.startswith(r\"\\end{tabular\"):\n break\n if inside and \"&\" in line and not line.startswith(\"%\"):\n # Skip \\hline, \\toprule etc\n if re.match(r'^\\\\(?:hline|toprule|midrule|bottomrule|cline)', line):\n continue\n # Skip column spec lines\n if re.match(r'^[lcr|]+\\s*

Markdown Proxy - URL to Markdown 将任意 URL 转为干净的 Markdown。支持需要登录的页面、PDF、专有平台。 Trigger Priority（先触发，再分流） - 用户给出 URL 并要求“读取、抓取、提取、解析、转 Markdown”时，优先使用本 Skill。 - 用户要求“读取这篇链接后再写稿、总结、翻译、做播客或分析”时，先用本 Skill 取回原文，再把 Markdown 交给下游 Skill；不要因为最终任务是写作而跳过抓取。 - 和飞书文档属于专用路由。不要先试普通网页打开或通用内容解析器。 - 用户已经贴出全文时不触发。YouTube 交给 ，普通搜索问题交给搜索工具。 URL Routing (先判断再执行) 收到 URL 后，先判断类型，不同类型走不同通道： | URL Pattern | Route To | Reason | |-------------|----------|--------| | | | 先代理，验证码页自动回退 Playwright | | | | 需飞书 API 认证 | | | skill | YouTube 有专用工具链 | | | 提取 arXiv ID → | HuggingFace 论文页实际是 arXiv 镜像，先找到 arXiv 链接再走 LaTeX 提取 | | | | 从…

, line):\n continue\n # Split by &, clean each cell\n cells = []\n for cell in re.split(r'(?\u003c!\\\\)&', line):\n cell = cell.strip()\n cell = re.sub(r'\\\\\\\\

Markdown Proxy - URL to Markdown 将任意 URL 转为干净的 Markdown。支持需要登录的页面、PDF、专有平台。 Trigger Priority（先触发，再分流） - 用户给出 URL 并要求“读取、抓取、提取、解析、转 Markdown”时，优先使用本 Skill。 - 用户要求“读取这篇链接后再写稿、总结、翻译、做播客或分析”时，先用本 Skill 取回原文，再把 Markdown 交给下游 Skill；不要因为最终任务是写作而跳过抓取。 - 和飞书文档属于专用路由。不要先试普通网页打开或通用内容解析器。 - 用户已经贴出全文时不触发。YouTube 交给 ，普通搜索问题交给搜索工具。 URL Routing (先判断再执行) 收到 URL 后，先判断类型，不同类型走不同通道： | URL Pattern | Route To | Reason | |-------------|----------|--------| | | | 先代理，验证码页自动回退 Playwright | | | | 需飞书 API 认证 | | | skill | YouTube 有专用工具链 | | | 提取 arXiv ID → | HuggingFace 论文页实际是 arXiv 镜像，先找到 arXiv 链接再走 LaTeX 提取 | | | | 从…

, '', cell)\n cell = strip_body_formatting(cell)\n cell = re.sub(r'\\s+', ' ', cell).strip()\n cells.append(cell)\n if any(cells):\n lines.append(cells)\n\n if not lines:\n return \"[表格数据]\"\n\n # Build markdown table\n result = []\n # Header row\n result.append(\"| \" + \" | \".join(lines[0]) + \" |\")\n result.append(\"|\" + \"|\".join([\"---\"] * len(lines[0])) + \"|\")\n for row in lines[1:]:\n # Pad to match header length\n while len(row) \u003c len(lines[0]):\n row.append(\"\")\n result.append(\"| \" + \" | \".join(row[:len(lines[0])]) + \" |\")\n\n return \"\\n\".join(result)\n\n\ndef parse_document(content, base_dir, abstract):\n \"\"\"Parse LaTeX document body into structured sections.\"\"\"\n # Remove \\maketitle early, before any position-based work\n content = re.sub(r'\\\\maketitle', '', content)\n\n # Verify \\begin{document} exists\n doc_start_marker = r\"\\begin{document}\"\n if doc_start_marker not in content:\n return abstract or \"\"\n\n pos = content.index(doc_start_marker) + len(doc_start_marker)\n\n # Extract abstract first (use content from \\begin{document} onward)\n doc_body = content[pos:]\n abs_match = re.search(r'\\\\begin\\{abstract\\}(.*?)\\\\end\\{abstract\\}', doc_body, re.DOTALL)\n\n result_parts = []\n if abs_match:\n pos = pos + abs_match.end() # pos now points past \\end{abstract}\n result_parts.append((\"\\n## Abstract\\n\\n\" + extract_abstract(content) + \"\\n\", pos))\n\n last_pos = pos\n\n # Find all structural markers in the remaining content\n sec_pattern = r'\\\\(section|subsection|subsubsection)\\*?\\{(.*?)\\}'\n figure_pattern = r'\\\\begin\\{figure\\*?\\}(.*?)\\\\end\\{figure\\*?\\}'\n table_pattern = r'\\\\begin\\{table\\*?\\}(.*?)\\\\end\\{table\\*?\\}'\n equation_pattern = r'\\\\begin\\{(equation|align|gather|multline)\\*?\\}(.*?)\\\\end\\{\\1\\*?\\}'\n\n elements = []\n for m in re.finditer(sec_pattern, content):\n elements.append((\"section\", m.start(), m.end(), m))\n for m in re.finditer(figure_pattern, content, re.DOTALL):\n elements.append((\"figure\", m.start(), m.end(), m))\n for m in re.finditer(table_pattern, content, re.DOTALL):\n elements.append((\"table\", m.start(), m.end(), m))\n for m in re.finditer(equation_pattern, content, re.DOTALL):\n elements.append((\"equation\", m.start(), m.end(), m))\n\n elements.sort(key=lambda x: x[1])\n\n figure_count = 0\n table_count = 0\n\n for elem_type, start, end, match in elements:\n if start \u003c pos:\n continue\n\n # Body text before this element\n body_text = content[last_pos:start]\n body_text = strip_body_formatting(body_text).strip()\n if body_text and len(body_text) > 20:\n result_parts.append((body_text + \"\\n\", start))\n\n if elem_type == \"section\":\n level_name = {\"section\": \"#\", \"subsection\": \"##\", \"subsubsection\": \"###\"}\n level = match.group(1)\n title = strip_body_formatting(match.group(2)).strip()\n prefix = level_name.get(level, \"####\")\n result_parts.append((f\"\\n{prefix} {title}\\n\\n\", end))\n\n elif elem_type == \"figure\":\n figure_count += 1\n fig_content = match.group(1)\n cap_match = re.search(r'\\\\caption\\{((?:[^{}]|\\{[^{}]*\\})*)\\}', fig_content, re.DOTALL)\n caption = \"\"\n if cap_match:\n caption = strip_body_formatting(cap_match.group(1).strip())\n caption = re.sub(r'\\s+', ' ', caption)\n img_paths = re.findall(r'\\\\includegraphics(?:\\[.*?\\])?\\{([^}]+)\\}', fig_content)\n img_refs = \", \".join(img_paths) if img_paths else \"\"\n result_parts.append((\n f\"\\n**Figure {figure_count}**\" +\n (f\": {caption}\\n\" if caption else \"\\n\") +\n (f\"[图片: {img_refs}]\\n\\n\" if img_refs else \"\\n\"),\n end\n ))\n\n elif elem_type == \"table\":\n table_count += 1\n tab_content = match.group(1)\n cap_match = re.search(r'\\\\caption\\{((?:[^{}]|\\{[^{}]*\\})*)\\}', tab_content, re.DOTALL)\n caption = \"\"\n if cap_match:\n caption = strip_body_formatting(cap_match.group(1).strip())\n caption = re.sub(r'\\s+', ' ', caption)\n tab_match = re.search(r'\\\\begin\\{tabular\\*?\\}(.*?)\\\\end\\{tabular\\*?\\}', tab_content, re.DOTALL)\n if tab_match:\n table_md = extract_table_data(tab_match.group(0))\n else:\n table_md = \"[表格 - 无 tabular 数据]\"\n result_parts.append((\n f\"\\n**Table {table_count}**\" +\n (f\": {caption}\\n\" if caption else \"\\n\") +\n f\"{table_md}\\n\\n\",\n end\n ))\n\n elif elem_type == \"equation\":\n eq_content = match.group(2).strip()\n result_parts.append((f\"\\n$\\n{eq_content}\\n$\\n\\n\", end))\n\n last_pos = end\n\n # Remaining body text after last element\n rest = content[last_pos:]\n rest = re.sub(r'\\\\end\\{document\\}.*', '', rest, flags=re.DOTALL)\n rest = strip_body_formatting(rest).strip()\n if rest and len(rest) > 20:\n result_parts.append((rest + \"\\n\", last_pos))\n\n # Sort by position and join\n result_parts.sort(key=lambda x: x[1])\n return \"\".join(p[0] for p in result_parts)\n\n\ndef find_media_files(source_dir, base_name=\"\"):\n \"\"\"Scan extracted source for image files and map them to figure references.\n\n Returns:\n dict: {filename_stem: [relative_paths]} for all image files found.\n Also returns a flat list of all image paths keyed by 'all'.\n \"\"\"\n img_exts = {'.pdf', '.png', '.jpg', '.jpeg', '.eps', '.svg'}\n media = {\"_all\": []}\n source = Path(source_dir)\n\n for f in sorted(source.rglob(\"*\")):\n if f.suffix.lower() in img_exts and f.is_file():\n rel = str(f.relative_to(source))\n media[\"_all\"].append(rel)\n # Index by stem (fig1, figure1, etc.) for easy lookup\n stem = f.stem.lower().replace(\"_\", \"\").replace(\"-\", \"\")\n if stem not in media:\n media[stem] = []\n media[stem].append(rel)\n\n return media\n\n\ndef _resolve_figure_files(img_refs, media_map, source_dir):\n r\"\"\"Given \\includegraphics references, find matching local files.\n\n Args:\n img_refs: list of paths from \\includegraphics{xxx}\n media_map: result from find_media_files()\n source_dir: Path to extracted source\n\n Returns:\n list of relative paths to matching files (may be empty)\n \"\"\"\n found = []\n source = Path(source_dir)\n for ref in img_refs:\n ref_path = Path(ref)\n ref_stem = ref_path.stem.lower().replace(\"_\", \"\").replace(\"-\", \"\")\n\n # Try exact match first\n exact = source / ref\n if exact.exists():\n found.append(str(ref))\n continue\n\n # Try with different extensions\n for ext in ('.pdf', '.png', '.jpg', '.jpeg', '.eps'):\n candidate = source / (ref_path.parent / (ref_path.stem + ext))\n if candidate.exists():\n found.append(str(candidate.relative_to(source)))\n break\n else:\n # Try fuzzy match via media_map\n if ref_stem in media_map:\n found.extend(media_map[ref_stem])\n\n return found\n\n\ndef extract_tex(arxiv_input, output_json=False, output_dir=None):\n \"\"\"Main extraction pipeline.\n\n Args:\n arxiv_input: arXiv URL or ID\n output_json: if True, return dict instead of markdown string\n output_dir: if set, persist extracted files here (no auto-cleanup)\n \"\"\"\n arxiv_id = extract_arxiv_id(arxiv_input)\n\n # Use persistent dir if specified, otherwise temp\n if output_dir:\n tmpdir = output_dir\n Path(tmpdir).mkdir(parents=True, exist_ok=True)\n # Clean any previous extraction\n for item in Path(tmpdir).iterdir():\n if item.is_dir():\n shutil.rmtree(item)\n else:\n item.unlink()\n else:\n tmpdir = tempfile.mkdtemp()\n\n try:\n success = download_source(arxiv_id, tmpdir)\n if not success:\n return {\"error\": \"No LaTeX source available. This paper may only have PDF.\", \"arxiv_id\": arxiv_id}\n\n main_tex = find_main_tex(tmpdir)\n if not main_tex:\n return {\"error\": \"No .tex file found in source archive.\", \"arxiv_id\": arxiv_id}\n\n content = main_tex.read_text(errors=\"replace\")\n content = strip_latex_comments(content)\n\n # Resolve \\input/\\include\n content = resolve_inputs(content, main_tex.parent)\n content = strip_latex_comments(content) # re-strip after input resolution\n\n # Extract metadata\n title, authors = extract_title_author(content)\n abstract = extract_abstract(content)\n\n # Scan for media files (before parsing document for figure-file mapping)\n media_map = find_media_files(tmpdir)\n\n # Parse document body\n doc_match = re.search(r'\\\\begin\\{document\\}(.*?)\\\\end\\{document\\}', content, re.DOTALL)\n if not doc_match:\n return {\"error\": \"No \\\\begin{document} found.\", \"arxiv_id\": arxiv_id}\n\n doc_body = doc_match.group(0)\n doc_content = parse_document(doc_body, main_tex.parent, abstract)\n\n # Build figure media inventory from the parsed content\n figures = []\n figure_pattern = r'\\*\\*Figure (\\d+)\\*\\*:?\\s*(.*?)(?:\\n\\[图片:\\s*(.*?)\\])?\\n'\n for m in re.finditer(figure_pattern, doc_content):\n fig_num = int(m.group(1))\n caption = m.group(2).strip() if m.group(2) else \"\"\n img_refs_str = m.group(3) or \"\"\n img_refs = [r.strip() for r in img_refs_str.split(\",\") if r.strip()]\n local_files = _resolve_figure_files(img_refs, media_map, tmpdir) if img_refs else []\n figures.append({\n \"index\": fig_num,\n \"caption\": caption,\n \"tex_refs\": img_refs,\n \"local_files\": local_files,\n })\n\n # Build media inventory\n media = {\n \"source_dir\": str(Path(tmpdir).resolve()),\n \"all_images\": media_map.get(\"_all\", []),\n \"figures\": figures,\n }\n\n # Build final markdown\n md = f\"\"\"---\ntitle: \"{title}\"\nauthors: {json.dumps(authors, ensure_ascii=False)}\narxiv: https://arxiv.org/abs/{arxiv_id}\nsource: LaTeX source\n---\n\n# {title}\n\n**Authors**: {\", \".join(authors)}\n\n**arXiv**: [{arxiv_id}](https://arxiv.org/abs/{arxiv_id})\n\n{doc_content}\n\"\"\"\n\n if output_json:\n return {\n \"arxiv_id\": arxiv_id,\n \"title\": title,\n \"authors\": authors,\n \"abstract\": abstract,\n \"markdown\": md,\n \"tex_file\": str(main_tex.name),\n \"media\": media,\n }\n\n return md\n\n finally:\n # Only cleanup if using temp dir (not user-specified output_dir)\n if not output_dir:\n shutil.rmtree(tmpdir, ignore_errors=True)\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Extract structured Markdown from arXiv LaTeX source.\")\n parser.add_argument(\"arxiv_input\", help=\"arXiv URL or identifier, for example 1706.03762\")\n parser.add_argument(\"--json\", action=\"store_true\", dest=\"as_json\", help=\"Return structured JSON\")\n parser.add_argument(\"--output-dir\", help=\"Persist extracted source and media files to this directory\")\n args = parser.parse_args()\n\n try:\n result = extract_tex(args.arxiv_input, output_json=args.as_json, output_dir=args.output_dir)\n if args.as_json:\n print(json.dumps(result, ensure_ascii=False))\n else:\n if isinstance(result, dict) and \"error\" in result:\n parser.error(result[\"error\"])\n print(result)\n except Exception as e:\n parser.error(str(e))\n","content_type":"text/x-python; charset=utf-8","language":"python","size":24000,"content_sha256":"75089ad2baacf5c6e1a66cb120423c9f708801c7c8c1067066c0079ef644bec4"},{"filename":"scripts/fetch_feishu.py","content":"#!/usr/bin/env python3\n\"\"\"Fetch Feishu/Lark document as Markdown. Standalone script using Feishu Open API.\"\"\"\n\nimport argparse\nimport json\nimport os\nimport re\nimport requests\n\nFEISHU_API_BASE = \"https://open.feishu.cn/open-apis\"\n\n\ndef get_tenant_access_token():\n \"\"\"获取 tenant_access_token\"\"\"\n app_id = os.environ.get(\"FEISHU_APP_ID\")\n app_secret = os.environ.get(\"FEISHU_APP_SECRET\")\n if not app_id or not app_secret:\n return None, \"环境变量 FEISHU_APP_ID 或 FEISHU_APP_SECRET 未设置\"\n\n url = f\"{FEISHU_API_BASE}/auth/v3/tenant_access_token/internal\"\n resp = requests.post(url, json={\"app_id\": app_id, \"app_secret\": app_secret})\n data = resp.json()\n if data.get(\"code\") != 0:\n return None, f\"获取 token 失败: {data.get('msg', resp.text)}\"\n return data[\"tenant_access_token\"], None\n\n\ndef parse_feishu_url(url):\n \"\"\"从飞书 URL 解析 document_id 和文档类型\"\"\"\n patterns = [\n (r\"feishu\\.cn/docx/([A-Za-z0-9]+)\", \"docx\"),\n (r\"feishu\\.cn/docs/([A-Za-z0-9]+)\", \"doc\"),\n (r\"feishu\\.cn/wiki/([A-Za-z0-9]+)\", \"wiki\"),\n (r\"larksuite\\.com/docx/([A-Za-z0-9]+)\", \"docx\"),\n (r\"larksuite\\.com/docs/([A-Za-z0-9]+)\", \"doc\"),\n (r\"larksuite\\.com/wiki/([A-Za-z0-9]+)\", \"wiki\"),\n ]\n for pattern, doc_type in patterns:\n m = re.search(pattern, url)\n if m:\n return m.group(1), doc_type\n return None, None\n\n\ndef get_document_info(token, doc_id):\n \"\"\"获取文档元信息（标题等）\"\"\"\n url = f\"{FEISHU_API_BASE}/docx/v1/documents/{doc_id}\"\n headers = {\"Authorization\": f\"Bearer {token}\"}\n resp = requests.get(url, headers=headers)\n data = resp.json()\n if data.get(\"code\") == 0:\n return data.get(\"data\", {}).get(\"document\", {})\n return {}\n\n\ndef get_document_blocks(token, doc_id):\n \"\"\"获取文档所有 blocks\"\"\"\n url = f\"{FEISHU_API_BASE}/docx/v1/documents/{doc_id}/blocks\"\n headers = {\"Authorization\": f\"Bearer {token}\"}\n all_blocks = []\n page_token = None\n\n while True:\n params = {\"page_size\": 500}\n if page_token:\n params[\"page_token\"] = page_token\n resp = requests.get(url, headers=headers, params=params)\n data = resp.json()\n if data.get(\"code\") != 0:\n return None, f\"获取 blocks 失败: {data.get('msg', resp.text)}\"\n\n items = data.get(\"data\", {}).get(\"items\", [])\n all_blocks.extend(items)\n\n if not data.get(\"data\", {}).get(\"has_more\", False):\n break\n page_token = data[\"data\"].get(\"page_token\")\n\n return all_blocks, None\n\n\ndef get_wiki_node(token, wiki_token):\n \"\"\"获取知识库节点信息，返回实际的 obj_token 和 obj_type\"\"\"\n url = f\"{FEISHU_API_BASE}/wiki/v2/spaces/get_node\"\n headers = {\"Authorization\": f\"Bearer {token}\"}\n resp = requests.get(url, headers=headers, params={\"token\": wiki_token})\n data = resp.json()\n if data.get(\"code\") == 0:\n node = data.get(\"data\", {}).get(\"node\", {})\n return node.get(\"obj_token\"), node.get(\"obj_type\")\n return None, None\n\n\ndef extract_text_from_elements(elements):\n \"\"\"从 text_run / mention_user 等元素中提取文本\"\"\"\n if not elements:\n return \"\"\n parts = []\n for el in elements:\n if \"text_run\" in el:\n tr = el[\"text_run\"]\n text = tr.get(\"content\", \"\")\n style = tr.get(\"text_element_style\", {})\n if style.get(\"bold\"):\n text = f\"**{text}**\"\n if style.get(\"italic\"):\n text = f\"*{text}*\"\n if style.get(\"strikethrough\"):\n text = f\"~~{text}~~\"\n if style.get(\"inline_code\"):\n text = f\"`{text}`\"\n if style.get(\"link\", {}).get(\"url\"):\n import urllib.parse\n link_url = urllib.parse.unquote(style[\"link\"][\"url\"])\n text = f\"[{text}]({link_url})\"\n parts.append(text)\n elif \"mention_user\" in el:\n parts.append(f\"@{el['mention_user'].get('user_id', 'user')}\")\n elif \"equation\" in el:\n parts.append(f\"${el['equation'].get('content', '')}$\")\n return \"\".join(parts)\n\n\ndef blocks_to_markdown(blocks):\n \"\"\"将飞书 blocks 转为 Markdown\"\"\"\n lines = []\n ordered_list_counter = {} # parent_id -> counter\n\n for block in blocks:\n block_type = block.get(\"block_type\")\n parent_id = block.get(\"parent_id\", \"\")\n\n # 1 = Page, 2 = Text, 3 = Heading1, 4 = Heading2, ..., 9 = Heading7+\n # 10 = BulletList, 11 = OrderedList, 12 = Code, 13 = Quote\n # 14 = Equation, 15 = Todo, 16 = Divider\n # 17 = Image, 18 = TableCell, 19 = Table\n # 22 = Callout, 23 = ChatCard, 27 = Grid, 28 = GridColumn\n\n if block_type == 2: # Text\n text_data = block.get(\"text\", {})\n text = extract_text_from_elements(text_data.get(\"elements\", []))\n if text.strip():\n lines.append(text)\n else:\n lines.append(\"\")\n\n elif block_type in (3, 4, 5, 6, 7, 8, 9): # Heading 1-7\n level = block_type - 2\n heading_data = block.get(\"heading\" + str(level), {}) or block.get(\"heading\", {})\n # Try multiple key formats\n for key in [f\"heading{level}\", \"heading\"]:\n if key in block:\n heading_data = block[key]\n break\n text = extract_text_from_elements(heading_data.get(\"elements\", []))\n lines.append(f\"{'#' * level} {text}\")\n\n elif block_type == 10: # Bullet list\n text_data = block.get(\"bullet\", {})\n text = extract_text_from_elements(text_data.get(\"elements\", []))\n lines.append(f\"- {text}\")\n\n elif block_type == 11: # Ordered list\n text_data = block.get(\"ordered\", {})\n text = extract_text_from_elements(text_data.get(\"elements\", []))\n counter = ordered_list_counter.get(parent_id, 0) + 1\n ordered_list_counter[parent_id] = counter\n lines.append(f\"{counter}. {text}\")\n\n elif block_type == 12: # Code block\n code_data = block.get(\"code\", {})\n text = extract_text_from_elements(code_data.get(\"elements\", []))\n lang = code_data.get(\"style\", {}).get(\"language\", \"\")\n # Map language codes\n lang_map = {1: \"plaintext\", 2: \"abap\", 3: \"ada\", 4: \"apache\", 5: \"apex\",\n 6: \"assembly\", 7: \"bash\", 8: \"c\", 9: \"csharp\", 10: \"cpp\",\n 11: \"clojure\", 12: \"cmake\", 13: \"coffeescript\", 14: \"css\",\n 15: \"d\", 16: \"dart\", 17: \"delphi\", 18: \"django\", 19: \"dockerfile\",\n 20: \"elixir\", 21: \"elm\", 22: \"erlang\", 23: \"fortran\",\n 24: \"fsharp\", 25: \"go\", 26: \"graphql\", 27: \"groovy\", 28: \"haskell\",\n 29: \"html\", 30: \"http\", 31: \"java\", 32: \"javascript\",\n 33: \"json\", 34: \"julia\", 35: \"kotlin\", 36: \"latex\", 37: \"lisp\",\n 38: \"lua\", 39: \"makefile\", 40: \"markdown\", 41: \"matlab\",\n 42: \"nginx\", 43: \"objectivec\", 44: \"ocaml\", 45: \"perl\",\n 46: \"php\", 47: \"powershell\", 48: \"properties\", 49: \"protobuf\",\n 50: \"python\", 51: \"r\", 52: \"ruby\", 53: \"rust\", 54: \"scala\",\n 55: \"scheme\", 56: \"scss\", 57: \"shell\", 58: \"sql\", 59: \"swift\",\n 60: \"thrift\", 61: \"toml\", 62: \"typescript\", 63: \"vbnet\",\n 64: \"verilog\", 65: \"vhdl\", 66: \"visual_basic\", 67: \"vue\",\n 68: \"xml\", 69: \"yaml\"}\n lang_str = lang_map.get(lang, \"\") if isinstance(lang, int) else str(lang)\n lines.append(f\"```{lang_str}\")\n lines.append(text)\n lines.append(\"```\")\n\n elif block_type == 13: # Quote\n text_data = block.get(\"quote\", {})\n text = extract_text_from_elements(text_data.get(\"elements\", []))\n lines.append(f\"> {text}\")\n\n elif block_type == 14: # Equation block\n eq_data = block.get(\"equation\", {})\n text = extract_text_from_elements(eq_data.get(\"elements\", []))\n lines.append(f\"$\\n{text}\\n$\")\n\n elif block_type == 15: # Todo\n todo_data = block.get(\"todo\", {})\n text = extract_text_from_elements(todo_data.get(\"elements\", []))\n done = todo_data.get(\"style\", {}).get(\"done\", False)\n checkbox = \"[x]\" if done else \"[ ]\"\n lines.append(f\"- {checkbox} {text}\")\n\n elif block_type == 16: # Divider\n lines.append(\"---\")\n\n elif block_type == 17: # Image\n image_data = block.get(\"image\", {})\n token_val = image_data.get(\"token\", \"\")\n lines.append(f\"![image](feishu-image://{token_val})\")\n\n elif block_type == 22: # Callout\n callout_data = block.get(\"callout\", {})\n # Callout is a container, children will be processed separately\n emoji = callout_data.get(\"emoji_id\", \"\")\n if emoji:\n lines.append(f\"> {emoji}\")\n\n elif block_type == 1: # Page (root), skip\n pass\n\n else:\n # Unknown block type, try to extract any text\n for key in block:\n if isinstance(block[key], dict) and \"elements\" in block[key]:\n text = extract_text_from_elements(block[key][\"elements\"])\n if text.strip():\n lines.append(text)\n break\n\n return \"\\n\\n\".join(lines)\n\n\ndef fetch_feishu_doc(url_or_id):\n \"\"\"主函数：获取飞书文档并转为 Markdown\"\"\"\n # 解析 URL\n doc_id, doc_type = parse_feishu_url(url_or_id)\n if not doc_id:\n # 可能直接传了 doc_token\n doc_id = url_or_id\n doc_type = \"docx\"\n\n # 获取 token\n token, err = get_tenant_access_token()\n if err:\n return {\"error\": err}\n\n # Wiki 需要先获取实际文档 ID\n if doc_type == \"wiki\":\n real_id, real_type = get_wiki_node(token, doc_id)\n if real_id:\n doc_id = real_id\n doc_type = real_type or \"docx\"\n else:\n return {\"error\": f\"无法获取知识库节点信息: {doc_id}\"}\n\n # 获取文档信息\n doc_info = get_document_info(token, doc_id)\n title = doc_info.get(\"title\", \"\")\n\n # 获取 blocks\n blocks, err = get_document_blocks(token, doc_id)\n if err:\n return {\"error\": err}\n\n # 转换为 Markdown\n content = blocks_to_markdown(blocks)\n\n return {\n \"title\": title,\n \"document_id\": doc_id,\n \"url\": url_or_id,\n \"content\": content,\n }\n\n\ndef format_as_markdown(result):\n \"\"\"格式化为 Markdown 文档\"\"\"\n if \"error\" in result:\n return f\"Error: {result['error']}\"\n\n parts = [\"---\"]\n if result.get(\"title\"):\n parts.append(f'title: \"{result[\"title\"]}\"')\n parts.append(f'document_id: \"{result[\"document_id\"]}\"')\n if result.get(\"url\"):\n parts.append(f'url: \"{result[\"url\"]}\"')\n parts.append(\"---\")\n parts.append(\"\")\n if result.get(\"title\"):\n parts.append(f\"# {result['title']}\")\n parts.append(\"\")\n parts.append(result.get(\"content\", \"\"))\n return \"\\n\".join(parts)\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(\n description=\"Fetch a Feishu/Lark document as Markdown. Requires FEISHU_APP_ID and FEISHU_APP_SECRET.\"\n )\n parser.add_argument(\"url_or_token\", help=\"Feishu/Lark document URL or document token\")\n parser.add_argument(\"--json\", action=\"store_true\", dest=\"use_json\", help=\"Return structured JSON\")\n args = parser.parse_args()\n\n result = fetch_feishu_doc(args.url_or_token)\n\n if args.use_json:\n print(json.dumps(result, ensure_ascii=False, indent=2))\n else:\n print(format_as_markdown(result))\n","content_type":"text/x-python; charset=utf-8","language":"python","size":11965,"content_sha256":"5fd65dbb5f79c38bac0a9883b4291c46b6d6d181c5bc58c7e8960e64a151cb5d"},{"filename":"scripts/fetch_weixin.py","content":"#!/usr/bin/env python3\n\"\"\"Fetch WeChat (公众号) article as Markdown. Standalone script using Playwright + BeautifulSoup.\"\"\"\n\nimport argparse\nimport json\nimport asyncio\nimport re\n\nasync def fetch_weixin_article(url: str) -> dict:\n \"\"\"Fetch and parse a WeChat article, return dict with title, author, publish_time, content.\"\"\"\n try:\n from playwright.async_api import async_playwright\n except ImportError:\n return {\"error\": \"playwright not installed. Run: pip install playwright && playwright install chromium\"}\n\n try:\n from bs4 import BeautifulSoup\n except ImportError:\n return {\"error\": \"beautifulsoup4 not installed. Run: pip install beautifulsoup4 lxml\"}\n\n html = None\n async with async_playwright() as p:\n browser = await p.chromium.launch(headless=True)\n page = await browser.new_page(\n user_agent=\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36\"\n )\n try:\n await page.goto(url, wait_until=\"domcontentloaded\", timeout=30000)\n await page.wait_for_selector(\"#js_content\", timeout=15000)\n html = await page.content()\n except Exception as e:\n await browser.close()\n return {\"error\": f\"Failed to load page: {e}\"}\n await browser.close()\n\n if not html:\n return {\"error\": \"No HTML content retrieved\"}\n\n soup = BeautifulSoup(html, \"lxml\")\n\n # Extract title\n title_el = soup.select_one(\"#activity-name\")\n title = title_el.get_text(strip=True) if title_el else \"\"\n\n # Extract author\n author_el = soup.select_one(\"#js_author_name\") or soup.select_one(\".rich_media_meta_text\")\n author = author_el.get_text(strip=True) if author_el else \"\"\n\n # Extract publish time\n time_el = soup.select_one(\"#publish_time\")\n publish_time = time_el.get_text(strip=True) if time_el else \"\"\n\n # Extract content\n content_el = soup.select_one(\"#js_content\")\n if not content_el:\n return {\"error\": \"Could not find article content (#js_content)\"}\n\n # Convert to markdown-like text\n # Remove scripts and styles\n for tag in content_el.find_all([\"script\", \"style\"]):\n tag.decompose()\n\n # Process images - extract src or data-src\n for img in content_el.find_all(\"img\"):\n src = img.get(\"data-src\") or img.get(\"src\") or \"\"\n if src:\n img.replace_with(f\"\\n![image]({src})\\n\")\n else:\n img.decompose()\n\n # Get text with basic formatting\n lines = []\n for element in content_el.find_all([\"p\", \"h1\", \"h2\", \"h3\", \"h4\", \"section\", \"blockquote\"]):\n text = element.get_text(strip=True)\n if not text:\n continue\n tag = element.name\n if tag in (\"h1\", \"h2\", \"h3\", \"h4\"):\n prefix = \"#\" * int(tag[1])\n lines.append(f\"{prefix} {text}\")\n elif tag == \"blockquote\":\n lines.append(f\"> {text}\")\n else:\n lines.append(text)\n\n content = \"\\n\\n\".join(lines)\n\n # If structured extraction got nothing, fall back to plain text\n if not content.strip():\n content = content_el.get_text(\"\\n\", strip=True)\n\n return {\n \"title\": title,\n \"author\": author,\n \"publish_time\": publish_time,\n \"content\": content,\n \"url\": url,\n }\n\n\ndef format_as_markdown(result: dict) -> str:\n \"\"\"Format result dict as a Markdown document.\"\"\"\n if \"error\" in result:\n return f\"Error: {result['error']}\"\n\n parts = [\"---\"]\n if result.get(\"title\"):\n parts.append(f\"title: \\\"{result['title']}\\\"\")\n if result.get(\"author\"):\n parts.append(f\"author: \\\"{result['author']}\\\"\")\n if result.get(\"publish_time\"):\n parts.append(f\"date: \\\"{result['publish_time']}\\\"\")\n parts.append(f\"url: \\\"{result['url']}\\\"\")\n parts.append(\"---\")\n parts.append(\"\")\n if result.get(\"title\"):\n parts.append(f\"# {result['title']}\")\n parts.append(\"\")\n parts.append(result.get(\"content\", \"\"))\n return \"\\n\".join(parts)\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Fetch a WeChat public-account article as Markdown.\")\n parser.add_argument(\"url\", help=\"WeChat article URL under mp.weixin.qq.com\")\n parser.add_argument(\"--json\", action=\"store_true\", dest=\"use_json\", help=\"Return structured JSON\")\n args = parser.parse_args()\n\n result = asyncio.run(fetch_weixin_article(args.url))\n\n if args.use_json:\n print(json.dumps(result, ensure_ascii=False, indent=2))\n else:\n print(format_as_markdown(result))\n","content_type":"text/x-python; charset=utf-8","language":"python","size":4616,"content_sha256":"c02319a0c5aa72b6411a86e680ff10cfe8975fa30dab223077f289459f8be464"},{"filename":"scripts/fetch_weixin.sh","content":"#!/usr/bin/env bash\n# Fetch a WeChat public-account article as Markdown.\n# Proxy extraction is attempted first. Verification/login walls automatically\n# fall back to the bundled Playwright extractor.\nset -euo pipefail\n\nURL=\"${1:-}\"\nif [[ -z \"$URL\" || \"$URL\" != https://mp.weixin.qq.com/* ]]; then\n echo \"Usage: fetch_weixin.sh https://mp.weixin.qq.com/s/... [--json]\" >&2\n exit 64\nfi\n\nOUTPUT_MODE=\"${2:-}\"\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# Proxy output cannot honor the JSON contract, so structured callers go\n# directly to the browser extractor.\nif [[ \"$OUTPUT_MODE\" != \"--json\" ]]; then\n if OUTPUT=$(bash \"$SCRIPT_DIR/fetch.sh\" \"$URL\" 2>/dev/null); then\n printf '%s\\n' \"$OUTPUT\"\n exit 0\n fi\nfi\n\nif python3 -c 'import playwright, bs4, lxml' >/dev/null 2>&1; then\n exec python3 \"$SCRIPT_DIR/fetch_weixin.py\" \"$URL\" ${OUTPUT_MODE:+\"$OUTPUT_MODE\"}\nfi\n\nif command -v uv >/dev/null 2>&1; then\n exec uv run --quiet \\\n --with playwright \\\n --with beautifulsoup4 \\\n --with lxml \\\n python \"$SCRIPT_DIR/fetch_weixin.py\" \"$URL\" ${OUTPUT_MODE:+\"$OUTPUT_MODE\"}\nfi\n\ncat >&2 \u003c\u003c'EOF'\nERROR: WeChat proxy extraction was blocked and the Playwright fallback is unavailable.\nInstall once:\n python3 -m pip install playwright beautifulsoup4 lxml\n python3 -m playwright install chromium\nOr install uv so the Python packages can run in an isolated environment:\n https://docs.astral.sh/uv/\nEOF\nexit 1\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":1432,"content_sha256":"f88e6cfcc4e0b968d6545ec653e018c895c992270c95eb9584a8c01c56db7c62"},{"filename":"scripts/fetch.sh","content":"#!/usr/bin/env bash\n# Fetch a URL as Markdown via proxy cascade with auto-fallback.\n# Usage: fetch.sh \u003curl> [proxy_url]\n# Example: fetch.sh https://example.com http://127.0.0.1:7890\nset -euo pipefail\n\nURL=\"${1:?Usage: fetch.sh \u003curl> [proxy_url]}\"\nPROXY=\"${2:-}\"\n\n_curl() {\n if [ -n \"$PROXY\" ]; then\n https_proxy=\"$PROXY\" http_proxy=\"$PROXY\" curl -sL \"$@\"\n else\n curl -sL \"$@\"\n fi\n}\n\n_has_content() {\n local content=\"$1\"\n local line_count\n line_count=$(echo \"$content\" | wc -l | tr -d ' ')\n\n # Must have more than 5 lines\n [ \"$line_count\" -gt 5 ] || return 1\n\n # Filter out common error pages\n echo \"$content\" | grep -Eiq \"Don't miss what's happening|Access Denied|404 Not Found|环境异常|完成验证后即可继续访问|访问过于频繁|verify you are human|captcha|该内容已被发布者删除|此内容因违规无法查看\" && return 1\n\n return 0\n}\n\n# 1. r.jina.ai - wide coverage, preserves image links\nOUT=$(_curl \"https://r.jina.ai/$URL\" 2>/dev/null || true)\nif _has_content \"$OUT\"; then\n echo \"$OUT\"\n exit 0\nfi\n\n# 2. defuddle.md - cleaner output with YAML frontmatter\nOUT=$(_curl \"https://defuddle.md/$URL\" 2>/dev/null || true)\nif _has_content \"$OUT\"; then\n echo \"$OUT\"\n exit 0\nfi\n\n# 3. agent-fetch - last resort local tool\nif command -v npx &>/dev/null; then\n OUT=$(npx --yes agent-fetch \"$URL\" --json 2>/dev/null || true)\n if _has_content \"$OUT\"; then\n echo \"$OUT\"\n exit 0\n fi\nfi\n\necho \"ERROR: All fetch methods failed for: $URL\" >&2\nexit 1\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":1489,"content_sha256":"41a16cc19d3da0771ed1b4966cbb4e329088ea9ee878b6ef8eac28912dac4f91"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"Markdown Proxy - URL to Markdown","type":"text"}]},{"type":"paragraph","content":[{"text":"将任意 URL 转为干净的 Markdown。支持需要登录的页面、PDF、专有平台。","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Trigger Priority（先触发，再分流）","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"用户给出 URL 并要求“读取、抓取、提取、解析、转 Markdown”时，优先使用本 Skill。","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"用户要求“读取这篇链接后再写稿、总结、翻译、做播客或分析”时，先用本 Skill 取回原文，再把 Markdown 交给下游 Skill；不要因为最终任务是写作而跳过抓取。","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"mp.weixin.qq.com","type":"text","marks":[{"type":"code_inline"}]},{"text":" 和飞书文档属于专用路由。不要先试普通网页打开或通用内容解析器。","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"用户已经贴出全文时不触发。YouTube 交给 ","type":"text"},{"text":"qiaomu-youtube-download","type":"text","marks":[{"type":"code_inline"}]},{"text":"，普通搜索问题交给搜索工具。","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"URL Routing (先判断再执行)","type":"text"}]},{"type":"paragraph","content":[{"text":"收到 URL 后，先判断类型，不同类型走不同通道：","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"URL Pattern","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Route To","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Reason","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"mp.weixin.qq.com","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"scripts/fetch_weixin.sh","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"先代理，验证码页自动回退 Playwright","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"feishu.cn/docx/","type":"text","marks":[{"type":"code_inline"}]},{"text":" ","type":"text"},{"text":"feishu.cn/wiki/","type":"text","marks":[{"type":"code_inline"}]},{"text":" ","type":"text"},{"text":"larksuite.com/docx/","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"scripts/fetch_feishu.py","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"需飞书 API 认证","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"youtube.com","type":"text","marks":[{"type":"code_inline"}]},{"text":" ","type":"text"},{"text":"youtu.be","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"qiaomu-youtube-download","type":"text","marks":[{"type":"code_inline"}]},{"text":" skill","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"YouTube 有专用工具链","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"huggingface.co/papers/","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"提取 arXiv ID → ","type":"text"},{"text":"scripts/extract_tex.py","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"HuggingFace 论文页实际是 arXiv 镜像，先找到 arXiv 链接再走 LaTeX 提取","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"arxiv.org/abs/","type":"text","marks":[{"type":"code_inline"}]},{"text":" ","type":"text"},{"text":"arxiv.org/pdf/","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"scripts/extract_tex.py","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"从 LaTeX 源码提取结构化内容 (章节/图表/公式)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":".pdf","type":"text","marks":[{"type":"code_inline"}]},{"text":" (URL or local path)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"scripts/extract_pdf.sh","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"PDF 专用提取","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"All other URLs","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"scripts/fetch.sh","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"代理级联自动 fallback","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Workflow","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 1: Route by URL Type","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"if URL contains \"mp.weixin.qq.com\":\n → bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch_weixin.sh \"URL\"\n → Done\n\nif URL contains \"feishu.cn/docx/\" or \"feishu.cn/wiki/\" or \"larksuite.com/docx/\":\n → python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch_feishu.py \"URL\"\n → Done\n\nif URL contains \"huggingface.co/papers/\":\n → First fetch the page (WebFetch) to find the arXiv URL\n → Then python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_tex.py \"{arxiv_url}\"\n → Done\n\nif URL contains \"arxiv.org/abs/\" or \"arxiv.org/pdf/\":\n → python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_tex.py \"URL\"\n → Done\n\nif URL contains \"youtube.com\" or \"youtu.be\":\n → Call qiaomu-youtube-download skill\n → Done\n\nif URL ends with \".pdf\" or is local PDF path:\n if remote URL:\n → Try: curl -sL \"https://r.jina.ai/{url}\"\n → If fails: download + extract_pdf.sh\n if local path:\n → bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_pdf.sh \"PATH\"\n → Done\n\nelse:\n → bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch.sh \"URL\"\n → Done","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 2: Display Content","type":"text"}]},{"type":"paragraph","content":[{"text":"After fetching, show to user:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Title: {title}\nAuthor: {author} (if available)\nSource: {platform} (公众号 / 飞书文档 / 网页 / PDF)\nURL: {original_url}\n\nSummary\n{3-5 sentence summary}\n\nContent\n{full Markdown, truncated at 200 lines if long}","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 3: Continue or Save","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Composite request","type":"text","marks":[{"type":"strong"}]},{"text":"（“读取后写稿/总结/分析”）：把提取结果直接交给下游任务，在同一轮继续；除非用户要求，不必额外保存源文件。","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Extraction-only request","type":"text","marks":[{"type":"strong"}]},{"text":"（“只读取/转 Markdown”）：保存到 ","type":"text"},{"text":"~/Downloads/{title}.md","type":"text","marks":[{"type":"code_inline"}]},{"text":"，使用 YAML frontmatter。","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Filename: use article title, remove special characters.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Format: YAML frontmatter (title, author, date, url, source) + Markdown body.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Tell the user the saved path.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Skip saving if the user says “just preview” or “don't save”.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Only stop after extraction when extraction was the complete request. If the user asked for a downstream deliverable, continue to that deliverable.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Examples","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"General URL","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch.sh \"https://example.com/article\"","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"X/Twitter Post","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch.sh \"https://x.com/username/status/1234567890\"","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"WeChat Article","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch_weixin.sh \"https://mp.weixin.qq.com/s/abc123\"","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Feishu Document","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch_feishu.py \"https://xxx.feishu.cn/docx/xxxxxxxx\"","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"arXiv LaTeX Source","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_tex.py \"https://arxiv.org/abs/1706.03762\"","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"PDF (Remote)","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"curl -sL \"https://r.jina.ai/https://example.com/paper.pdf\"","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"PDF (Local)","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_pdf.sh \"/path/to/paper.pdf\"","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"With Custom Proxy","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch.sh \"https://example.com\" \"http://127.0.0.1:7890\"","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Notes","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"r.jina.ai and defuddle.md require no API key","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"fetch.sh","type":"text","marks":[{"type":"code_inline"}]},{"text":" handles proxy cascade with automatic fallback","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Content validation: filters error, login-wall, and WeChat verification pages; requires >5 lines","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"WeChat wrapper prefers dependency-free proxies, then uses local Playwright; when Python packages are missing and ","type":"text"},{"text":"uv","type":"text","marks":[{"type":"code_inline"}]},{"text":" is available, it runs them in an isolated environment","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Playwright fallback requires a Chromium runtime; install once with ","type":"text"},{"text":"python3 -m playwright install chromium","type":"text","marks":[{"type":"code_inline"}]},{"text":" if absent","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Feishu script requires: ","type":"text"},{"text":"FEISHU_APP_ID","type":"text","marks":[{"type":"code_inline"}]},{"text":" + ","type":"text"},{"text":"FEISHU_APP_SECRET","type":"text","marks":[{"type":"code_inline"}]},{"text":" env vars","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"PDF extraction tries: marker-pdf → pdftotext → pypdf","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For detailed method documentation, see ","type":"text"},{"text":"references/methods.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]}]},"metadata":{"date":"2026-11-02","name":"qiaomu-markdown-proxy","author":"@skillopedia","source":{"stars":508,"repo_name":"qiaomu-markdown-proxy","origin_url":"https://github.com/joeseesun/qiaomu-markdown-proxy/blob/HEAD/SKILL.md","repo_owner":"joeseesun","body_sha256":"7c17dd4e3c432dd98e3b9bb80b6ca9ec8c81b838166d4a368dd09f5a17901070","cluster_key":"0354285bd42672593d2be9aaf6261c2cdb913370857b3bbfbb718dbe96d59e0c","clean_bundle":{"format":"clean-skill-bundle-v1","source":"joeseesun/qiaomu-markdown-proxy/SKILL.md","text_bytes":48146,"source_commit":"ab56e0aa4003221762ab89efce5e11fbbe54ece2","skills_sh_rank":16702,"attachment_count":7,"text_attachments":7,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"2.1.0","category":"browser-automation-scraping","category_label":"Browser"},"exact_dupes_collapsed_into_this":0},"version":"2.1.0","category":"browser-automation-scraping","import_tag":"skills-sh-top20k-2026-11","description":"Read, fetch, extract, parse, or convert a URL or link into clean Markdown. Use FIRST whenever a user asks to read source content from a webpage, especially WeChat/微信公众号 mp.weixin.qq.com, Feishu/Lark docs, X/Twitter, PDFs, arXiv papers, or a URL that will feed a later summary, rewrite, article, podcast, or analysis. For WeChat and Feishu, prefer this specialist extractor over generic web open or generic content parsers. Exclude YouTube, pure web search, already-pasted text, and local non-PDF files."}},"renderedAt":1793676394822}

Markdown Proxy - URL to Markdown 将任意 URL 转为干净的 Markdown。支持需要登录的页面、PDF、专有平台。 Trigger Priority（先触发，再分流） - 用户给出 URL 并要求“读取、抓取、提取、解析、转 Markdown”时，优先使用本 Skill。 - 用户要求“读取这篇链接后再写稿、总结、翻译、做播客或分析”时，先用本 Skill 取回原文，再把 Markdown 交给下游 Skill；不要因为最终任务是写作而跳过抓取。 - 和飞书文档属于专用路由。不要先试普通网页打开或通用内容解析器。 - 用户已经贴出全文时不触发。YouTube 交给 ，普通搜索问题交给搜索工具。 URL Routing (先判断再执行) 收到 URL 后，先判断类型，不同类型走不同通道： | URL Pattern | Route To | Reason | |-------------|----------|--------| | | | 先代理，验证码页自动回退 Playwright | | | | 需飞书 API 认证 | | | skill | YouTube 有专用工具链 | | | 提取 arXiv ID → | HuggingFace 论文页实际是 arXiv 镜像，先找到 arXiv 链接再走 LaTeX 提取 | | | | 从…