log-ops — Skillopedia

Log Operations Practical patterns for analyzing log files -- especially JSONL format used in agent conversation logs, benchmark outputs, and structured application logs. Log Format Decision Tree Prerequisites Required (must be installed): - (ripgrep) - text search, prefiltering. Install: / - - JSON/JSONL extraction and transformation. Install: / Optional (enhanced capabilities, gracefully degraded without): - - interactive log exploration with SQL queries. Install: / WSL: - (angle-grinder) - pipeline aggregation syntax. Install: - (Miller) - CSV/TSV log analysis. Install: / - - parallel proce…

\\t' -k3 -rn | column -t -N DIR,FILE,ERRORS\n\n# Search for a pattern and show matching lines with source file\nfd -e jsonl . logs/ -x bash -c '\n rg \"\\\"error\\\"\" \"$1\" 2>/dev/null | while read line; do\n echo \"$1: $line\"\n done\n' _ {}\n```\n\n### Build Summary Table from Multiple Log Files\n\n```bash\n# Summary statistics per log file\necho -e \"FILE\\tLINES\\tERRORS\\tWARNS\\tFIRST_TS\\tLAST_TS\"\nfd -e jsonl . logs/ | while read f; do\n lines=$(wc -l \u003c \"$f\")\n errors=$(rg -c '\"error\"' \"$f\" 2>/dev/null || echo 0)\n warns=$(rg -c '\"warn\"' \"$f\" 2>/dev/null || echo 0)\n first=$(head -1 \"$f\" | jq -r '.timestamp // \"unknown\"')\n last=$(tail -1 \"$f\" | jq -r '.timestamp // \"unknown\"')\n echo -e \"$(basename \"$f\")\\t$lines\\t$errors\\t$warns\\t$first\\t$last\"\ndone | column -t\n\n# Health check across all services\nfd -e jsonl -d 1 . /var/log/services/ -x bash -c '\n svc=$(basename \"$1\" .jsonl)\n last_error=$(tac \"$1\" | jq -r \"select(.level == \\\"error\\\") | .timestamp\" 2>/dev/null | head -1)\n error_count=$(rg -c \"\\\"error\\\"\" \"$1\" 2>/dev/null || echo 0)\n echo -e \"$svc\\t$error_count\\t${last_error:-none}\"\n' _ {} | sort | column -t -N SERVICE,ERRORS,LAST_ERROR\n```\n\n### Identify Common Failure Patterns Across Runs\n\n```bash\n# Extract all error messages across trial directories\nfd -e jsonl . trials/ -x jq -r 'select(.level == \"error\") | .message' {} |\n sort | uniq -c | sort -rn | head -20\n\n# Find which trials share the same failure\nfd -e jsonl . trials/ -x bash -c '\n trial=$(echo \"$1\" | rg -o \"trial-[^/]+\")\n jq -r \"select(.level == \\\"error\\\") | .message\" \"$1\" 2>/dev/null |\n while read msg; do echo -e \"$trial\\t$msg\"; done\n' _ {} |\n sort -t

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

\\t' -k2 |\n awk -F'\\t' '\n prev != $2 { if (NR > 1 && count > 1) print count, prev_msg, trials; count=0; trials=\"\" }\n { count++; trials = trials \" \" $1; prev = $2; prev_msg = $2 }\n END { if (count > 1) print count, prev_msg, trials }\n ' | sort -rn | head -10\n\n# Correlation: which errors appear together\nfd -e jsonl . trials/ -x bash -c '\n trial=$(echo \"$1\" | rg -o \"trial-[^/]+\")\n errors=$(jq -r \"select(.level == \\\"error\\\") | .message\" \"$1\" 2>/dev/null | sort -u | paste -sd \"|\")\n [ -n \"$errors\" ] && echo -e \"$trial\\t$errors\"\n' _ {} | sort -t

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

\\t' -k2 | uniq -f1 -c | sort -rn\n```\n\n### fd + rg + jq Composition\n\n```bash\n# The canonical three-stage pipeline for multi-directory log analysis:\n# 1. fd: find the files\n# 2. rg: prefilter for speed\n# 3. jq: structured extraction\n\n# Example: find all timeout errors across services, extract details\nfd -e jsonl . /var/log/ | # find log files\n xargs rg -l '\"timeout\"' | # filter to files with timeouts\n xargs -I{} jq -c '\n select(.message | test(\"timeout\")) |\n {file: input_filename, ts: .timestamp, svc: .service, msg: .message}\n ' {}\n\n# Example: aggregate error counts by service across all log directories\nfd -e jsonl . /var/log/ -x rg -c '\"error\"' {} | # count errors per file\n awk -F: '{\n split($1, parts, \"/\")\n svc = parts[length(parts)-1]\n gsub(/\\.jsonl$/, \"\", svc)\n sum[svc] += $2\n }\n END { for (s in sum) print sum[s], s }' |\n sort -rn\n\n# Example: find the most recent error across all services\nfd -e jsonl . /var/log/ -x tail -1 {} | # last line of each file\n jq -sc '\n map(select(.level == \"error\")) |\n sort_by(.timestamp) |\n .[-1] |\n {service: .service, timestamp: .timestamp, message: .message}\n '\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":18360,"content_sha256":"ebdb6eb9a8464153c400c7e5a93a2f3818a661faff776947bd9f8e3639227d48"},{"filename":"references/jsonl-patterns.md","content":"# JSONL Patterns Reference\n\nComprehensive patterns for working with JSONL (JSON Lines) files -- one JSON object per line, the dominant format for structured logs, agent conversation records, and streaming data.\n\n## JSONL Basics\n\n### Format Rules\n\n- One valid JSON object per line\n- No trailing commas between lines\n- No wrapping array or outer object\n- Each line is independently parseable\n- Newlines within string values must be escaped as `\\n`\n\n### Streaming vs Slurp\n\n```bash\n# STREAMING (default): processes one line at a time, constant memory\njq -c 'select(.level == \"error\")' app.jsonl\n\n# SLURP (-s): loads ALL lines into a single array, requires memory for entire file\njq -sc 'group_by(.level)' app.jsonl\n\n# Rule of thumb:\n# File \u003c 100MB --> slurp is fine\n# File 100MB-1GB --> slurp with caution, prefer streaming + sort/uniq\n# File > 1GB --> never slurp, use streaming or split+parallel\n```\n\n### Key jq Flags for JSONL\n\n| Flag | Purpose | Example |\n|------|---------|---------|\n| `-c` | Compact output (one line per object) | `jq -c '.' file.jsonl` |\n| `-r` | Raw string output (no quotes) | `jq -r '.message' file.jsonl` |\n| `-s` | Slurp all lines into array | `jq -s 'length' file.jsonl` |\n| `-e` | Exit with error if output is false/null | `jq -e '.status == 200' line.json` |\n| `-R` | Read each line as raw string | `jq -R 'fromjson? // empty' messy.jsonl` |\n| `--stream` | SAX-style path/value pairs | `jq --stream '.' huge.json` |\n| `--slurpfile` | Load a file as variable | `jq --slurpfile ids ids.json 'select(.id | IN($ids[][]))' data.jsonl` |\n| `--arg` | Pass string variable | `jq --arg name \"foo\" 'select(.name == $name)' data.jsonl` |\n| `--argjson` | Pass JSON variable | `jq --argjson min 100 'select(.count > $min)' data.jsonl` |\n| `--unbuffered` | Flush output after each line | `tail -f app.jsonl \\| jq --unbuffered -r '.message'` |\n\n---\n\n## Extraction Patterns\n\n### Select by Field Value\n\n```bash\n# Exact match\njq -c 'select(.level == \"error\")' app.jsonl\n\n# Numeric comparison\njq -c 'select(.status >= 400)' app.jsonl\n\n# String contains\njq -c 'select(.message | test(\"timeout\"))' app.jsonl\n\n# Regex match\njq -c 'select(.path | test(\"^/api/v[0-9]+/users\"))' app.jsonl\n\n# Case-insensitive match\njq -c 'select(.message | test(\"error\"; \"i\"))' app.jsonl\n\n# Null check\njq -c 'select(.error != null)' app.jsonl\n\n# Boolean field\njq -c 'select(.retry == true)' app.jsonl\n```\n\n### Select by Nested Field\n\n```bash\n# Dot notation for nesting\njq -c 'select(.request.method == \"POST\")' app.jsonl\n\n# Deep nesting\njq -c 'select(.context.user.role == \"admin\")' app.jsonl\n\n# Safe navigation (no error if path missing)\njq -c 'select(.request?.headers?[\"authorization\"] != null)' app.jsonl\n```\n\n### Select by Array Contains\n\n```bash\n# Array contains value\njq -c 'select(.tags | index(\"critical\"))' app.jsonl\n\n# Any element matches condition\njq -c 'select(.events | any(.type == \"error\"))' app.jsonl\n\n# All elements match condition\njq -c 'select(.checks | all(.passed == true))' app.jsonl\n\n# Array length\njq -c 'select((.retries | length) > 3)' app.jsonl\n```\n\n### Extract and Flatten Nested Structures\n\n```bash\n# Flatten one level of nesting\njq -c '{timestamp, level, msg: .message, user: .context.user.id}' app.jsonl\n\n# Explode array into separate lines\njq -c '.events[]' app.jsonl\n\n# Flatten array with parent context\njq -c '. as $parent | .events[] | {request_id: $parent.request_id, event: .type, ts: .timestamp}' app.jsonl\n\n# Extract from array of objects\njq -c '.results[] | select(.score \u003c 0.5) | {name, score}' results.jsonl\n\n# Recursive descent (find all values for a key at any depth)\njq -c '.. | .error_message? // empty' app.jsonl\n```\n\n### Handle Optional and Nullable Fields\n\n```bash\n# Default value for missing field\njq -r '.region // \"unknown\"' app.jsonl\n\n# Default for nested missing field\njq -r '.response.body.error // .response.status_text // \"no error info\"' app.jsonl\n\n# Skip lines where field is missing (instead of outputting null)\njq -r '.optional_field // empty' app.jsonl\n\n# Coalesce multiple possible fields\njq -r '(.error_message // .err_msg // .error // \"none\")' app.jsonl\n\n# Type check before access\njq -c 'if .data | type == \"array\" then .data | length else 0 end' app.jsonl\n```\n\n### Multi-Level Nesting (Agent Conversation Logs)\n\n```bash\n# Claude Code conversation logs have deeply nested tool calls\n# Structure: {role, content: [{type: \"tool_use\", name, input}, ...]}\n\n# Extract all tool call names\njq -c '.content[]? | select(.type == \"tool_use\") | .name' conversation.jsonl\n\n# Extract tool inputs\njq -c '.content[]? | select(.type == \"tool_use\") | {tool: .name, input: .input}' conversation.jsonl\n\n# Extract text content blocks\njq -r '.content[]? | select(.type == \"text\") | .text' conversation.jsonl\n\n# Extract tool results\njq -c '.content[]? | select(.type == \"tool_result\") | {tool_use_id, content}' conversation.jsonl\n\n# Find tool calls that contain specific patterns in their input\njq -c '.content[]? | select(.type == \"tool_use\" and (.input | tostring | test(\"SELECT\")))' conversation.jsonl\n```\n\n### De-Escape Nested JSON Strings\n\n```bash\n# When a field contains a JSON string that needs parsing\njq -c '.payload | fromjson' app.jsonl\n\n# Safe de-escape (skip if not valid JSON)\njq -c '.payload | fromjson? // {raw: .}' app.jsonl\n\n# Double-escaped JSON (escaped twice)\njq -c '.data | fromjson | fromjson' app.jsonl\n\n# Extract field from de-escaped nested JSON\njq -r '.payload | fromjson | .result.status' app.jsonl\n\n# Handle mixed escaped/unescaped\njq -c 'if (.payload | type) == \"string\" then .payload | fromjson else .payload end' app.jsonl\n```\n\n---\n\n## Aggregation Patterns\n\nAll aggregation patterns use `-s` (slurp) which loads the entire file into memory. For large files, prefilter with `rg` first.\n\n### Count by Field Value\n\n```bash\n# Count per level\njq -sc 'group_by(.level) | map({level: .[0].level, count: length})' app.jsonl\n\n# Count per status code\njq -sc 'group_by(.status) | map({status: .[0].status, count: length}) | sort_by(-.count)' app.jsonl\n\n# Count unique values\njq -sc '[.[].user_id] | unique | length' app.jsonl\n\n# Frequency distribution\njq -rsc 'group_by(.level) | map(\"\\(.[0].level)\\t\\(length)\") | .[]' app.jsonl\n```\n\n### Sum, Average, Min, Max\n\n```bash\n# Sum\njq -sc 'map(.bytes) | add' app.jsonl\n\n# Average\njq -sc 'map(.duration_ms) | add / length' app.jsonl\n\n# Min and max\njq -sc 'map(.duration_ms) | {min: min, max: max, avg: (add / length)}' app.jsonl\n\n# Percentile approximation (p50, p95, p99)\njq -sc '\n map(.duration_ms) | sort |\n length as $n |\n {\n p50: .[($n * 0.50 | floor)],\n p95: .[($n * 0.95 | floor)],\n p99: .[($n * 0.99 | floor)],\n max: .[-1]\n }\n' app.jsonl\n\n# Sum grouped by category\njq -sc '\n group_by(.service) |\n map({service: .[0].service, total_bytes: (map(.bytes) | add)})\n' app.jsonl\n```\n\n### Group By with Aggregation\n\n```bash\n# Group by service, show count and error rate\njq -sc '\n group_by(.service) |\n map({\n service: .[0].service,\n total: length,\n errors: (map(select(.level == \"error\")) | length),\n error_rate: ((map(select(.level == \"error\")) | length) / length * 100 | round)\n })\n' app.jsonl\n\n# Group by hour\njq -sc '\n group_by(.timestamp | split(\"T\")[1] | split(\":\")[0]) |\n map({\n hour: .[0].timestamp | split(\"T\")[1] | split(\":\")[0],\n count: length\n })\n' app.jsonl\n\n# Nested group by (service then level)\njq -sc '\n group_by(.service) |\n map({\n service: .[0].service,\n by_level: (group_by(.level) | map({level: .[0].level, n: length}))\n })\n' app.jsonl\n```\n\n### Top-N Queries\n\n```bash\n# Top 10 slowest requests\njq -sc 'sort_by(-.duration_ms) | .[:10] | .[] | {path: .path, ms: .duration_ms}' app.jsonl\n\n# Top 5 most frequent errors\njq -sc '\n map(select(.level == \"error\")) |\n group_by(.message) |\n map({message: .[0].message, count: length}) |\n sort_by(-.count) | .[:5]\n' app.jsonl\n\n# Top users by request count\njq -sc '\n group_by(.user_id) |\n map({user: .[0].user_id, requests: length}) |\n sort_by(-.requests) | .[:10]\n' app.jsonl\n```\n\n### Histogram and Distribution Analysis\n\n```bash\n# Response time histogram (buckets: 0-100, 100-500, 500-1000, 1000+)\njq -sc '\n map(.duration_ms) |\n {\n \"0-100ms\": (map(select(. \u003c 100)) | length),\n \"100-500ms\": (map(select(. >= 100 and . \u003c 500)) | length),\n \"500-1000ms\": (map(select(. >= 500 and . \u003c 1000)) | length),\n \"1000ms+\": (map(select(. >= 1000)) | length)\n }\n' app.jsonl\n\n# Status code distribution\njq -rsc '\n group_by(.status) |\n map(\"\\(.[0].status)\\t\\(length)\") |\n sort | .[]\n' app.jsonl\n\n# Log level distribution over time (by hour)\njq -rsc '\n group_by(.timestamp | split(\"T\")[1] | split(\":\")[0]) |\n map(\n (.[0].timestamp | split(\"T\")[1] | split(\":\")[0]) as $hour |\n {\n hour: $hour,\n info: (map(select(.level == \"info\")) | length),\n warn: (map(select(.level == \"warn\")) | length),\n error: (map(select(.level == \"error\")) | length)\n }\n ) | .[] | [.hour, .info, .warn, .error] | @tsv\n' app.jsonl | column -t -N HOUR,INFO,WARN,ERROR\n```\n\n### Running Totals and Cumulative Sums\n\n```bash\n# Cumulative error count over time\njq -sc '\n sort_by(.timestamp) |\n reduce .[] as $item (\n {total: 0, rows: []};\n .total += 1 |\n .rows += [{ts: $item.timestamp, cumulative: .total}]\n ) | .rows[] | [.ts, .cumulative] | @tsv\n' \u003c(jq -c 'select(.level == \"error\")' app.jsonl)\n\n# Running average of response times\njq -sc '\n sort_by(.timestamp) |\n foreach .[] as $item (\n {n: 0, sum: 0};\n .n += 1 | .sum += $item.duration_ms;\n {ts: $item.timestamp, running_avg: (.sum / .n | round)}\n )\n' app.jsonl\n```\n\n---\n\n## Transformation Patterns\n\n### Reshape Objects\n\n```bash\n# Flatten nested to flat\njq -c '{\n ts: .timestamp,\n level: .level,\n msg: .message,\n user: .context.user.id,\n method: .request.method,\n path: .request.path\n}' app.jsonl\n\n# Add computed fields\njq -c '. + {\n date: (.timestamp | split(\"T\")[0]),\n hour: (.timestamp | split(\"T\")[1] | split(\":\")[0] | tonumber),\n is_error: (.level == \"error\")\n}' app.jsonl\n\n# Rename fields\njq -c '{timestamp: .ts, message: .msg, severity: .lvl}' app.jsonl\n\n# Remove fields\njq -c 'del(.stack_trace, .internal_debug_info)' app.jsonl\n```\n\n### Merge Fields from Multiple Lines\n\n```bash\n# Combine start and end events by request_id\njq -sc '\n group_by(.request_id) |\n map(\n (map(select(.event == \"start\")) | .[0]) as $start |\n (map(select(.event == \"end\")) | .[0]) as $end |\n {\n request_id: .[0].request_id,\n start: $start.timestamp,\n end: $end.timestamp,\n status: $end.status,\n path: $start.path\n }\n )[]\n' events.jsonl\n\n# Merge consecutive lines (e.g., multiline log entries)\njq -sc '\n reduce .[] as $item (\n [];\n if (. | length) == 0 then [$item]\n elif $item.continuation == true then\n (.[-1].message += \"\\n\" + $item.message) | .\n else . + [$item]\n end\n )[]\n' app.jsonl\n```\n\n### Convert Between Formats\n\n```bash\n# JSONL to CSV\njq -r '[.timestamp, .level, .message] | @csv' app.jsonl > app.csv\n\n# JSONL to TSV\njq -r '[.timestamp, .level, .message] | @tsv' app.jsonl > app.tsv\n\n# JSONL to CSV with header\necho \"timestamp,level,message\" > app.csv\njq -r '[.timestamp, .level, .message] | @csv' app.jsonl >> app.csv\n\n# CSV to JSONL (using mlr)\nmlr --c2j cat app.csv > app.jsonl\n\n# JSONL to formatted table\njq -r '[.timestamp, .level, .message] | @tsv' app.jsonl | column -t -s

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

\\t'\n\n# JSONL to markdown table\necho \"| Timestamp | Level | Message |\"\necho \"|-----------|-------|---------|\"\njq -r '\"| \\(.timestamp) | \\(.level) | \\(.message) |\"' app.jsonl\n```\n\n### Annotate Lines with Computed Fields\n\n```bash\n# Add line number\njq -c --argjson n 0 '. + {line_num: (input_line_number)}' app.jsonl\n\n# Add duration since previous event (requires slurp)\njq -sc '\n sort_by(.timestamp) |\n . as $all |\n [range(length)] |\n map(\n $all[.] + (\n if . > 0 then {gap_from_prev: \"computed\"}\n else {gap_from_prev: null}\n end\n )\n )[]\n' app.jsonl\n\n# Tag lines matching criteria\njq -c '. + {\n severity_class: (\n if .level == \"error\" or .level == \"fatal\" then \"critical\"\n elif .level == \"warn\" then \"warning\"\n else \"normal\"\n end\n )\n}' app.jsonl\n\n# Enrich with filename when processing multiple files\nfd -e jsonl . logs/ -x bash -c 'jq -c --arg src \"$1\" \". + {source: \\$src}\" \"$1\"' _ {}\n```\n\n---\n\n## Comparison Patterns\n\n### Diff Two JSONL Files by Matching Key\n\n```bash\n# Find entries in A but not in B (by id)\njq -r '.id' b.jsonl | sort > /tmp/b_ids.txt\njq -c --slurpfile bids \u003c(jq -Rs 'split(\"\\n\") | map(select(. != \"\"))' /tmp/b_ids.txt) '\n select(.id | IN($bids[0][])) | not\n' a.jsonl\n\n# Simpler approach using comm\njq -r '.id' a.jsonl | sort > /tmp/a_ids.txt\njq -r '.id' b.jsonl | sort > /tmp/b_ids.txt\ncomm -23 /tmp/a_ids.txt /tmp/b_ids.txt # IDs in A but not B\ncomm -13 /tmp/a_ids.txt /tmp/b_ids.txt # IDs in B but not A\ncomm -12 /tmp/a_ids.txt /tmp/b_ids.txt # IDs in both\n\n# Find records that exist in both but have different values\njq -sc '\n [., input] |\n (.[0] | map({(.id): .}) | add) as $a |\n (.[1] | map({(.id): .}) | add) as $b |\n ($a | keys) as $keys |\n [$keys[] | select($a[.] != $b[.])] |\n map({id: ., a: $a[.], b: $b[.]})\n' \u003c(jq -sc '.' a.jsonl) \u003c(jq -sc '.' b.jsonl)\n```\n\n### Side-by-Side Field Comparison\n\n```bash\n# Compare a specific field between two runs\npaste \u003c(jq -r '[.id, .score] | @tsv' run1.jsonl | sort) \\\n \u003c(jq -r '[.id, .score] | @tsv' run2.jsonl | sort) |\n awk -F'\\t' '$2 != $4 {print $1, \"run1=\" $2, \"run2=\" $4}'\n\n# Summary comparison of two log files\necho \"=== File A ===\" && jq -sc '{\n lines: length,\n errors: (map(select(.level == \"error\")) | length),\n unique_users: ([.[].user_id] | unique | length)\n}' a.jsonl\necho \"=== File B ===\" && jq -sc '{\n lines: length,\n errors: (map(select(.level == \"error\")) | length),\n unique_users: ([.[].user_id] | unique | length)\n}' b.jsonl\n```\n\n### Find New, Missing, and Changed Records\n\n```bash\n# Comprehensive diff report\njq -r '.id' a.jsonl | sort > /tmp/a.ids\njq -r '.id' b.jsonl | sort > /tmp/b.ids\n\necho \"--- New in B (not in A) ---\"\ncomm -13 /tmp/a.ids /tmp/b.ids\n\necho \"--- Removed from A (not in B) ---\"\ncomm -23 /tmp/a.ids /tmp/b.ids\n\necho \"--- Changed (in both, different values) ---\"\ncomm -12 /tmp/a.ids /tmp/b.ids | while read id; do\n a_hash=$(rg \"\\\"id\\\":\\\"$id\\\"\" a.jsonl | md5sum | cut -d' ' -f1)\n b_hash=$(rg \"\\\"id\\\":\\\"$id\\\"\" b.jsonl | md5sum | cut -d' ' -f1)\n [ \"$a_hash\" != \"$b_hash\" ] && echo \"$id\"\ndone\n```\n\n---\n\n## Performance Patterns\n\n### Two-Stage rg + jq Pipeline\n\nThe single most important performance pattern. ripgrep is 10-100x faster than jq at scanning text.\n\n```bash\n# BAD: jq scans every line (slow on large files)\njq -c 'select(.level == \"error\" and .service == \"auth\")' huge.jsonl\n\n# GOOD: rg filters text first, jq only parses matching lines\nrg '\"error\"' huge.jsonl | rg '\"auth\"' | jq -c '.'\n\n# GOOD: for precise matching after rg prefilter\nrg '\"error\"' huge.jsonl | jq -c 'select(.level == \"error\" and .service == \"auth\")'\n\n# Benchmarks (typical 1GB JSONL file):\n# jq alone: 45 seconds\n# rg + jq: 3 seconds\n# rg alone: 0.8 seconds\n```\n\n### GNU parallel for Splitting Large Files\n\n```bash\n# Split a 10GB file and process in parallel\nsplit -l 500000 huge.jsonl /tmp/chunk_\n\n# Count errors across all chunks\nls /tmp/chunk_* | parallel \"rg -c '\\\"error\\\"' {}\" | awk -F: '{sum+=$2} END {print sum}'\n\n# Extract and merge results\nls /tmp/chunk_* | parallel \"jq -c 'select(.level == \\\"error\\\")' {}\" > all_errors.jsonl\n\n# Cleanup\nrm /tmp/chunk_*\n\n# One-liner with process substitution\nparallel --pipe -L 100000 'jq -c \"select(.level == \\\"error\\\")\"' \u003c huge.jsonl > errors.jsonl\n```\n\n### jq --stream for SAX-Style Processing\n\nFor files too large to fit in memory, even line-by-line (e.g., a single 5GB JSON array).\n\n```bash\n# Count items in a huge JSON array without loading it\njq --stream 'select(.[0] | length == 1) | .[0][0]' huge-array.json | tail -1\n\n# Extract specific field from each item in huge array\njq -cn --stream 'fromstream(1 | truncate_stream(inputs)) | .name' huge-array.json\n\n# Filter items from huge array\njq -cn --stream '\n fromstream(1 | truncate_stream(inputs)) |\n select(.status == \"failed\")\n' huge-array.json\n```\n\n### Indexing Frequently-Queried Files\n\n```bash\n# Build an index of line offsets by key value\nawk '{\n match($0, /\"request_id\":\"([^\"]+)\"/, m)\n if (m[1]) print m[1], NR\n}' app.jsonl | sort > app.idx\n\n# Look up specific request by index\nLINE=$(grep \"req-abc-123\" app.idx | awk '{print $2}')\nsed -n \"${LINE}p\" app.jsonl | jq .\n\n# Build a SQLite index for repeated queries\nsqlite3 log_index.db \"CREATE TABLE idx (request_id TEXT, line INTEGER)\"\nawk '{\n match($0, /\"request_id\":\"([^\"]+)\"/, m)\n if (m[1]) print \"INSERT INTO idx VALUES (\\047\" m[1] \"\\047, \" NR \");\"\n}' app.jsonl | sqlite3 log_index.db\n\n# Query by index\nLINE=$(sqlite3 log_index.db \"SELECT line FROM idx WHERE request_id = 'req-abc-123'\")\nsed -n \"${LINE}p\" app.jsonl | jq .\n```\n\n### Memory-Efficient Aggregation Without Slurp\n\n```bash\n# Count by level without loading entire file\njq -r '.level' app.jsonl | sort | uniq -c | sort -rn\n\n# Top error messages without slurp\njq -r 'select(.level == \"error\") | .message' app.jsonl | sort | uniq -c | sort -rn | head -20\n\n# Unique users without slurp\njq -r '.user_id' app.jsonl | sort -u | wc -l\n\n# Sum without slurp\njq -r '.bytes' app.jsonl | awk '{sum+=$1} END {print sum}'\n\n# These are all O(1) memory (streaming) vs O(n) memory (slurp)\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":17411,"content_sha256":"f6bfc35c9415d0670f07c38b4cea35b59fc5253aa01692bf4db84acfcf29c024"},{"filename":"references/tool-setup.md","content":"# Tool Setup Reference\n\nInstallation, configuration, and key commands for log analysis tools. Each tool includes install commands for all platforms, the most useful flags, and integration patterns.\n\n---\n\n## jq -- JSON/JSONL Processor\n\nThe primary tool for structured log analysis. Processes JSONL line by line (streaming) or as a batch (slurp).\n\n### Installation\n\n```bash\n# macOS\nbrew install jq\n\n# Ubuntu/Debian\nsudo apt install jq\n\n# Windows\nchoco install jq\n# or\nwinget install jqlang.jq\n\n# Verify\njq --version\n```\n\n### Key Flags\n\n| Flag | Purpose | Example |\n|------|---------|---------|\n| `-c` | Compact output (one JSON per line) | `jq -c '.' file.jsonl` |\n| `-r` | Raw string output (no quotes) | `jq -r '.message' file.jsonl` |\n| `-s` | Slurp: read all lines into array | `jq -s 'length' file.jsonl` |\n| `-S` | Sort object keys | `jq -S '.' file.json` |\n| `-e` | Exit 1 if output is false/null | `jq -e '.ok' file.json` |\n| `-R` | Read lines as raw strings | `jq -R 'fromjson?' messy.jsonl` |\n| `-n` | Null input (use with inputs) | `jq -n '[inputs]' file.jsonl` |\n| `--arg` | Pass string variable | `jq --arg id \"42\" 'select(.id == $id)'` |\n| `--argjson` | Pass JSON variable | `jq --argjson n 10 'select(.count > $n)'` |\n| `--slurpfile` | Load file as variable | `jq --slurpfile ids ids.json 'select(.id | IN($ids[][]))'` |\n| `--stream` | SAX-style path/value output | `jq --stream '.' huge.json` |\n| `--unbuffered` | Flush after each output | `tail -f f.jsonl \\| jq --unbuffered '.'` |\n| `--tab` | Use tabs for indentation | `jq --tab '.' file.json` |\n\n### Essential Commands\n\n```bash\n# Pretty print a single JSON object\njq '.' file.json\n\n# Validate JSONL (report bad lines)\njq -c '.' file.jsonl > /dev/null 2>&1 || echo \"Invalid JSON detected\"\n\n# Find and show invalid lines\nawk '{\n cmd = \"echo \" \"'\\'''\" $0 \"'\\'''\" \" | jq . 2>/dev/null\"\n if (system(cmd) != 0) print NR\": \"$0\n}' file.jsonl\n\n# Better: use jq -R to find invalid lines\njq -R 'fromjson? // error' file.jsonl 2>&1 | rg \"error\" | head\n\n# Count lines in JSONL\njq -sc 'length' file.jsonl\n\n# Get unique keys across all objects\njq -sc '[.[] | keys[]] | unique' file.jsonl\n\n# Get schema (keys and types) from first line\nhead -1 file.jsonl | jq '[to_entries[] | {key, type: (.value | type)}]'\n\n# Reformat JSONL with consistent key ordering\njq -cS '.' file.jsonl > normalized.jsonl\n```\n\n### Debugging jq Expressions\n\n```bash\n# Use debug to print intermediate values to stderr\njq '.items[] | debug | select(.active)' file.json\n\n# Use @text to see what jq thinks a value is\njq '.field | @text' file.json\n\n# Use type to check value types\njq '.field | type' file.json\n\n# Build expressions incrementally\njq '.' file.json # Start: see full structure\njq '.items' file.json # Navigate to array\njq '.items[]' file.json # Iterate array\njq '.items[] | .name' file.json # Extract field\njq '.items[] | select(.active)' file.json # Filter\n\n# Common error: \"Cannot iterate over null\"\n# Fix: use ? operator\njq '.items[]?' file.json # Won't error if items is null\n\n# Common error: \"null is not iterable\"\n# Fix: default empty array\njq '(.items // [])[]' file.json\n```\n\n### Integration with Other Tools\n\n```bash\n# rg prefilter then jq parse\nrg '\"error\"' app.jsonl | jq -r '.message'\n\n# jq output to column for alignment\njq -r '[.name, .status, .duration] | @tsv' app.jsonl | column -t\n\n# jq output to sort/uniq for frequency\njq -r '.error_type' errors.jsonl | sort | uniq -c | sort -rn\n\n# jq to CSV for spreadsheet import\njq -r '[.timestamp, .level, .message] | @csv' app.jsonl > export.csv\n\n# jq with xargs for per-line processing\njq -r '.file_path' manifest.jsonl | xargs wc -l\n```\n\n---\n\n## lnav -- Log File Navigator\n\nInteractive terminal-based log viewer with SQL support, automatic format detection, timeline view, and filtering. Ideal for exploratory analysis.\n\n### Installation\n\n```bash\n# macOS\nbrew install lnav\n\n# Ubuntu/Debian\nsudo apt install lnav\n\n# Windows (via Chocolatey)\nchoco install lnav\n\n# From source\ncurl -LO https://github.com/tstack/lnav/releases/download/v0.12.2/lnav-0.12.2-linux-musl-x86_64.zip\nunzip lnav-0.12.2-linux-musl-x86_64.zip\nsudo cp lnav-0.12.2/lnav /usr/local/bin/\n\n# Verify\nlnav -V\n```\n\n### Key Features\n\n| Feature | Access | Description |\n|---------|--------|-------------|\n| Auto-detect format | Automatic | Recognizes syslog, Apache, nginx, JSON, and many more |\n| SQL queries | `:` then SQL | Run SQL against log data |\n| Filter in/out | `i` / `o` | Interactive include/exclude filters |\n| Bookmarks | `m` | Mark lines for later reference |\n| Timeline | `t` | Show time histogram |\n| Pretty print | `p` | Toggle pretty-printing JSON |\n| Headless mode | `-n -c \"...\"` | Non-interactive command execution |\n| Compressed files | Automatic | Handles .gz, .bz2, .xz transparently |\n\n### Essential Commands\n\n```bash\n# Open log file(s)\nlnav app.log\nlnav /var/log/syslog /var/log/auth.log # multiple files, merged by timestamp\n\n# Open JSONL logs\nlnav app.jsonl\n\n# Open compressed logs\nlnav app.log.gz\n\n# Open all logs in a directory\nlnav /var/log/myapp/\n\n# Headless mode: run query and output results\nlnav -n -c \";SELECT count(*) FROM logline WHERE log_level = 'error'\" app.log\n\n# Headless mode: filter and export\nlnav -n -c \";SELECT log_time, log_body FROM logline WHERE log_level = 'error'\" \\\n -c \":write-csv-to errors.csv\" app.log\n\n# Headless mode: get stats\nlnav -n -c \";SELECT log_level, count(*) as cnt FROM logline GROUP BY log_level ORDER BY cnt DESC\" app.log\n```\n\n### SQL Mode Recipes\n\n```sql\n-- Error count by hour\nSELECT strftime('%Y-%m-%d %H', log_time) as hour, count(*) as errors\nFROM logline WHERE log_level = 'error'\nGROUP BY hour ORDER BY hour;\n\n-- Top error messages\nSELECT log_body, count(*) as cnt\nFROM logline WHERE log_level = 'error'\nGROUP BY log_body ORDER BY cnt DESC LIMIT 10;\n\n-- Time between events\nSELECT log_time, log_body,\n julianday(log_time) - julianday(lag(log_time) OVER (ORDER BY log_time)) as gap_days\nFROM logline WHERE log_level = 'error';\n\n-- Log volume over time\nSELECT strftime('%Y-%m-%d %H:%M', log_time) as minute, count(*) as lines\nFROM logline\nGROUP BY minute ORDER BY minute;\n```\n\n### Custom Log Formats\n\n```json\n// ~/.lnav/formats/installed/myapp.json\n{\n \"myapp_log\": {\n \"title\": \"My Application Log\",\n \"regex\": {\n \"std\": {\n \"pattern\": \"^(?\u003ctimestamp>\\\\d{4}-\\\\d{2}-\\\\d{2}T\\\\d{2}:\\\\d{2}:\\\\d{2})\\\\s+\\\\[(?\u003clevel>\\\\w+)\\\\]\\\\s+(?\u003cbody>.*)\"\n }\n },\n \"timestamp-format\": [\"%Y-%m-%dT%H:%M:%S\"],\n \"level\": {\n \"error\": \"ERROR\",\n \"warning\": \"WARN\",\n \"info\": \"INFO\",\n \"debug\": \"DEBUG\"\n }\n }\n}\n```\n\n### Interactive Keyboard Shortcuts\n\n| Key | Action |\n|-----|--------|\n| `/` | Search forward (regex) |\n| `n` / `N` | Next/previous search match |\n| `i` | Toggle filter: show only matching lines |\n| `o` | Toggle filter: hide matching lines |\n| `TAB` | Switch between views (log, text, help) |\n| `t` | Toggle timeline histogram |\n| `m` | Set bookmark on current line |\n| `u` / `U` | Next/previous bookmark |\n| `z` / `Z` | Zoom in/out on timeline |\n| `p` | Toggle pretty-print for JSON |\n| `e` / `E` | Next/previous error |\n| `w` / `W` | Next/previous warning |\n| `:` | Enter command mode |\n| `;` | Enter SQL query mode |\n\n---\n\n## angle-grinder (agrind) -- Log Pipeline Aggregation\n\nPipeline-based aggregation tool designed for log analysis. Think SQL-like queries in a streaming pipeline syntax.\n\n### Installation\n\n```bash\n# Via cargo (all platforms)\ncargo install ag\n\n# macOS\nbrew install angle-grinder\n\n# Verify\nagrind --version\n```\n\n### Pipeline Syntax\n\n```\n\u003cinput_pattern> | \u003coperator1> | \u003coperator2> | ...\n```\n\n### Essential Commands\n\n```bash\n# Count log levels\ncat app.log | agrind '* | parse \"* [*] *\" as ts, level, msg | count by level'\n\n# Top URLs\ncat access.log | agrind '* | parse \"* * * * * * *\" as ip, _, _, ts, method, url, status | count by url | sort by _count desc | head 10'\n\n# Average response time by endpoint\ncat access.log | agrind '* | parse \"* *ms\" as prefix, duration | avg of duration by prefix'\n\n# Error frequency over time\ncat app.log | agrind '* | parse \"*T*:*:* [ERROR]*\" as date, hour, min, sec, msg | count by hour'\n\n# Filter then aggregate\ncat app.log | agrind '* | where level == \"error\" | count by msg | sort by _count desc'\n\n# JSON log fields\ncat app.jsonl | agrind '* | json | where level == \"error\" | count by message'\n```\n\n### Operators Reference\n\n| Operator | Purpose | Example |\n|----------|---------|---------|\n| `parse` | Extract fields with pattern | `parse \"* [*] *\" as a, b, c` |\n| `json` | Parse JSON log lines | `json` |\n| `where` | Filter rows | `where level == \"error\"` |\n| `count` | Count (optionally by group) | `count by level` |\n| `sum` | Sum a field | `sum of bytes` |\n| `avg` | Average a field | `avg of duration` |\n| `min` / `max` | Min/max of field | `min of response_time` |\n| `sort` | Sort results | `sort by _count desc` |\n| `head` | Limit results | `head 10` |\n| `uniq` | Unique values | `uniq by user_id` |\n| `percentile` | Percentile calc | `p50 of duration, p99 of duration` |\n\n---\n\n## rg (ripgrep) -- Fast Pattern Search\n\nAlready covered extensively in file-search skill. Here are log-specific flags and patterns.\n\n### Log-Specific Flags\n\n| Flag | Purpose | Example |\n|------|---------|---------|\n| `-c` | Count matches per file | `rg -c \"ERROR\" /var/log/*.log` |\n| `-l` | List files with matches | `rg -l \"timeout\" /var/log/` |\n| `-L` | List files without matches | `rg -L \"healthy\" /var/log/` |\n| `--stats` | Show match statistics | `rg --stats \"error\" app.log` |\n| `-A N` | Show N lines after match | `rg -A5 \"Exception\" app.log` |\n| `-B N` | Show N lines before match | `rg -B3 \"FATAL\" app.log` |\n| `-C N` | Show N lines context | `rg -C5 \"crash\" app.log` |\n| `-U` | Multiline matching | `rg -U \"Error.*\\n.*at \" app.log` |\n| `--json` | JSON output format | `rg --json \"error\" app.log` |\n| `-a` | Search binary files | `rg -a \"pattern\" binary.log` |\n| `--line-buffered` | Flush per line (for tail) | `tail -f app.log \\| rg --line-buffered \"error\"` |\n| `-F` | Fixed string (no regex) | `rg -F \"stack[0]\" app.log` |\n| `-f FILE` | Patterns from file | `rg -f patterns.txt app.log` |\n| `-v` | Invert match | `rg -v \"DEBUG\" app.log` |\n\n### Log Search Recipes\n\n```bash\n# Find errors across all log files recursively\nrg \"ERROR|FATAL|CRITICAL\" /var/log/\n\n# Count errors per file, sorted\nrg -c \"ERROR\" /var/log/ 2>/dev/null | sort -t: -k2 -rn\n\n# Find stack traces (multiline)\nrg -U \"Exception.*\\n(\\s+at .*\\n)+\" app.log\n\n# Extract timestamps of errors\nrg \"ERROR\" app.log | rg -o \"^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\"\n\n# Search compressed log files\nrg -z \"error\" app.log.gz\n\n# Search JSONL for specific field value (text-level, fast but approximate)\nrg '\"level\":\"error\"' app.jsonl\n\n# Search JSONL for value in specific key (avoid matching wrong key)\nrg '\"user_id\":\"user-42\"' app.jsonl\n\n# Negative lookahead: errors that are NOT timeouts\nrg \"ERROR(?!.*timeout)\" app.log\n\n# Time-bounded search (extract lines between two timestamps)\nrg \"2026-03-08T1[4-5]:\" app.log\n```\n\n### rg JSON Output Mode\n\n```bash\n# Get structured output from rg (useful for programmatic processing)\nrg --json \"error\" app.log | jq -c 'select(.type == \"match\") | {file: .data.path.text, line: .data.line_number, text: .data.lines.text}'\n\n# Count matches with file info\nrg --json \"error\" app.log | jq -c 'select(.type == \"summary\") | .data.stats'\n```\n\n---\n\n## awk -- Column-Based Log Processing\n\nPre-installed on all Unix systems. Best for space/tab delimited logs with consistent column structure.\n\n### Common Recipes\n\n```bash\n# Apache/nginx combined log format columns:\n# $1=IP $2=ident $3=user $4=date $5=time $6=tz $7=method $8=path $9=proto $10=status $11=size\n\n# Status code distribution\nawk '{print $9}' access.log | sort | uniq -c | sort -rn\n\n# Requests per IP\nawk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20\n\n# 5xx errors with paths\nawk '$9 >= 500 {print $1, $9, $7}' access.log\n\n# Average response size\nawk '{sum += $10; n++} END {printf \"Avg: %.0f bytes\\n\", sum/n}' access.log\n\n# Requests per minute\nawk '{print substr($4, 2, 17)}' access.log | sort | uniq -c | tail -20\n\n# Bandwidth by path\nawk '{bytes[$7] += $10} END {for (p in bytes) printf \"%10d %s\\n\", bytes[p], p}' access.log | sort -rn | head -20\n\n# Custom delimiter (e.g., pipe-separated)\nawk -F'|' '{print $3, $5}' custom.log\n\n# Time-range filter (syslog format)\nawk '/^Mar 8 14:/ {print}' syslog\n\n# Calculate time difference between first and last line\nawk 'NR==1 {first=$1\" \"$2} END {last=$1\" \"$2; print \"From:\", first, \"To:\", last}' app.log\n```\n\n### awk for Key-Value Logs\n\n```bash\n# Parse key=value format (logfmt)\nawk '{\n for (i=1; i\u003c=NF; i++) {\n split($i, kv, \"=\")\n if (kv[1] == \"duration\") sum += kv[2]; n++\n }\n} END {print \"avg_duration=\" sum/n}' app.log\n\n# Extract specific key from logfmt\nawk '{\n for (i=1; i\u003c=NF; i++) {\n split($i, kv, \"=\")\n if (kv[1] == \"status\" && kv[2] >= 500) print $0\n }\n}' app.log\n```\n\n---\n\n## GNU parallel -- Parallel Log Processing\n\nSplits work across CPU cores for processing large log files.\n\n### Installation\n\n```bash\n# macOS\nbrew install parallel\n\n# Ubuntu/Debian\nsudo apt install parallel\n\n# Verify\nparallel --version\n```\n\n### Essential Commands\n\n```bash\n# Process multiple log files in parallel\nls /var/log/app-*.jsonl | parallel \"jq -c 'select(.level == \\\"error\\\")' {} > {.}_errors.jsonl\"\n\n# Split large file and process chunks in parallel\nsplit -l 200000 huge.jsonl /tmp/chunk_\nls /tmp/chunk_* | parallel \"jq -r '.message' {} | sort | uniq -c\" | sort -rn | head -20\n\n# Parallel grep across many files\nfd -e jsonl . /var/log/ | parallel \"rg -c '\\\"error\\\"' {} 2>/dev/null\" | sort -t: -k2 -rn\n\n# Pipe-based parallelism (no temp files)\nparallel --pipe -L 50000 \"jq -c 'select(.level == \\\"error\\\")'\" \u003c huge.jsonl > errors.jsonl\n\n# Parallel with progress bar\nls /var/log/app-*.jsonl | parallel --bar \"jq -sc 'length' {}\" | awk '{sum+=$1} END {print sum, \"total lines\"}'\n\n# Number of jobs (default: CPU cores)\nls *.jsonl | parallel -j 4 \"jq -c 'select(.status >= 500)' {}\"\n```\n\n### Combining with split\n\n```bash\n# Full workflow: split, process in parallel, merge results\nFILE=huge.jsonl\nCHUNKS=/tmp/log_chunks\nmkdir -p \"$CHUNKS\"\n\n# Split\nsplit -l 100000 \"$FILE\" \"$CHUNKS/chunk_\"\n\n# Process in parallel\nls \"$CHUNKS\"/chunk_* | parallel \"jq -r 'select(.level == \\\"error\\\") | .message' {}\" |\n sort | uniq -c | sort -rn > error_summary.txt\n\n# Cleanup\nrm -rf \"$CHUNKS\"\n```\n\n---\n\n## Miller (mlr) -- CSV/TSV Log Analysis\n\nLike awk, sed, and jq combined but specifically for structured record data (CSV, TSV, JSON).\n\n### Installation\n\n```bash\n# macOS\nbrew install miller\n\n# Ubuntu/Debian\nsudo apt install miller\n\n# Windows\nchoco install miller\n\n# Verify\nmlr --version\n```\n\n### Essential Commands\n\n```bash\n# View CSV with headers\nmlr --csv head -n 10 access_log.csv\n\n# Filter rows\nmlr --csv filter '$status >= 400' access_log.csv\n\n# Sort by column\nmlr --csv sort-by -nr duration access_log.csv\n\n# Statistics\nmlr --csv stats1 -a min,max,mean,p95 -f duration access_log.csv\n\n# Group by with stats\nmlr --csv stats1 -a count,mean -f duration -g endpoint access_log.csv\n\n# Convert formats\nmlr --c2j cat access_log.csv # CSV to JSON\nmlr --c2t cat access_log.csv # CSV to TSV (table)\nmlr --j2c cat access_log.json # JSON to CSV\nmlr --c2p cat access_log.csv # CSV to pretty-print table\n\n# Top-N by group\nmlr --csv top -n 5 -f duration -g endpoint access_log.csv\n\n# Add computed fields\nmlr --csv put '$error = ($status >= 400 ? \"yes\" : \"no\")' access_log.csv\n\n# Decimate (sample every Nth row)\nmlr --csv sample -k 100 huge_log.csv\n\n# Uniq count\nmlr --csv count-distinct -f status access_log.csv\n\n# Histogram\nmlr --csv decimate -g status -n 1 access_log.csv | mlr --csv count-distinct -f status\n\n# Join two CSV files\nmlr --csv join -j user_id -f users.csv then sort-by user_id access_log.csv\n```\n\n### TSV from jq to mlr Pipeline\n\n```bash\n# Extract JSONL to TSV, then use mlr for analysis\njq -r '[.timestamp, .level, .duration_ms, .path] | @tsv' app.jsonl > /tmp/extracted.tsv\nmlr --tsvlite --from /tmp/extracted.tsv \\\n label timestamp,level,duration_ms,path then \\\n filter '$level == \"error\"' then \\\n stats1 -a count,mean -f duration_ms -g path then \\\n sort-by -nr count\n```\n\n---\n\n## Tool Integration Cheat Sheet\n\n### Combining Tools\n\n```bash\n# fd + rg + jq: find files, prefilter, extract\nfd -e jsonl . logs/ | xargs rg -l '\"error\"' | xargs jq -c 'select(.level == \"error\") | {ts: .timestamp, msg: .message}'\n\n# rg + jq + column: search, extract, format\nrg '\"timeout\"' app.jsonl | jq -r '[.timestamp, .service, .message] | @tsv' | column -t\n\n# jq + sort + uniq: aggregate without slurp\njq -r '.error_type' errors.jsonl | sort | uniq -c | sort -rn\n\n# tail + rg + jq: live monitoring with extraction\ntail -f app.jsonl | rg --line-buffered '\"error\"' | jq --unbuffered -r '[.timestamp, .message] | @tsv'\n\n# fd + parallel + jq: parallel extraction across many files\nfd -e jsonl . logs/ | parallel \"jq -c 'select(.level == \\\"error\\\")' {}\" > all_errors.jsonl\n\n# jq + mlr: structured extraction then statistical analysis\njq -r '[.path, .duration_ms, .status] | @csv' app.jsonl | \\\n mlr --csv label path,duration,status then \\\n stats1 -a p50,p95,p99 -f duration -g path then \\\n sort-by -nr p95\n\n# lnav + headless SQL: non-interactive queries\nlnav -n -c \";SELECT log_level, count(*) FROM logline GROUP BY log_level\" app.log\n```\n\n### Decision Guide: Which Combination?\n\n```\nTask: Explore unknown log file\n --> lnav (interactive, auto-detects format)\n\nTask: Quick search for pattern\n --> rg \"pattern\" file.log\n\nTask: Extract fields from JSONL\n --> jq -r '[.field1, .field2] | @tsv' file.jsonl\n\nTask: Count/aggregate JSONL (\u003c100MB)\n --> jq -sc 'group_by(.x) | map(...)' file.jsonl\n\nTask: Count/aggregate JSONL (>100MB)\n --> jq -r '.field' file.jsonl | sort | uniq -c | sort -rn\n\nTask: Search large JSONL then extract\n --> rg \"pattern\" file.jsonl | jq -r '.field'\n\nTask: CSV/TSV log statistics\n --> mlr --csv stats1 -a mean,p95 -f duration file.csv\n\nTask: Process many log files in parallel\n --> fd -e jsonl . dir/ | parallel \"jq ...\"\n\nTask: Pipeline aggregation on text logs\n --> cat file.log | agrind '* | parse ... | count by ...'\n\nTask: Live monitoring with filtering\n --> tail -f file.jsonl | rg --line-buffered \"x\" | jq --unbuffered '.'\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":18371,"content_sha256":"70b9b1ae2402f0080363690c3f619689378e9d758ec5502b83e6701551846477"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"Log Operations","type":"text"}]},{"type":"paragraph","content":[{"text":"Practical patterns for analyzing log files -- especially JSONL format used in agent conversation logs, benchmark outputs, and structured application logs.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Log Format Decision Tree","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Unknown Log File\n│\n├─ Is it one JSON object per line?\n│ ├─ Yes ──────────────────────── JSONL\n│ │ ├─ Small file (\u003c100MB)\n│ │ │ └─ jq for extraction, jq -s for aggregation\n│ │ ├─ Large file (100MB-1GB)\n│ │ │ └─ rg prefilter then pipe to jq\n│ │ └─ Huge file (>1GB)\n│ │ └─ split + parallel jq, or jq --stream\n│ │\n│ └─ No\n│ ├─ Is it one large JSON object/array?\n│ │ └─ Yes ──────────────── Single JSON\n│ │ └─ jq --stream for SAX-style, or jq directly if fits in memory\n│ │\n│ ├─ Does it have key=value pairs?\n│ │ └─ Yes ──────────────── Structured (logfmt / key-value)\n│ │ └─ rg for search, awk/sd for extraction, angle-grinder for aggregation\n│ │\n│ ├─ Does it follow syslog format? (timestamp hostname service[pid]: message)\n│ │ └─ Yes ──────────────── Syslog\n│ │ └─ rg for search, awk for column extraction, lnav for interactive\n│ │\n│ ├─ Is it space/tab delimited with consistent columns?\n│ │ └─ Yes ──────────────── Column-based (access logs, CSV)\n│ │ └─ awk for extraction, mlr for CSV, rg for pattern search\n│ │\n│ └─ Mixed or unstructured\n│ └─ Plain text ─────────── Freeform\n│ └─ rg for search, rg -A/-B for context, lnav for exploration","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Prerequisites","type":"text"}]},{"type":"paragraph","content":[{"text":"Required","type":"text","marks":[{"type":"strong"}]},{"text":" (must be installed):","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"rg","type":"text","marks":[{"type":"code_inline"}]},{"text":" (ripgrep) - text search, prefiltering. Install: ","type":"text"},{"text":"cargo install ripgrep","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"choco install ripgrep","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"jq","type":"text","marks":[{"type":"code_inline"}]},{"text":" - JSON/JSONL extraction and transformation. Install: ","type":"text"},{"text":"brew install jq","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"choco install jq","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"paragraph","content":[{"text":"Optional","type":"text","marks":[{"type":"strong"}]},{"text":" (enhanced capabilities, gracefully degraded without):","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"lnav","type":"text","marks":[{"type":"code_inline"}]},{"text":" - interactive log exploration with SQL queries. Install: ","type":"text"},{"text":"brew install lnav","type":"text","marks":[{"type":"code_inline"}]},{"text":" / WSL: ","type":"text"},{"text":"apt install lnav","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"agrind","type":"text","marks":[{"type":"code_inline"}]},{"text":" (angle-grinder) - pipeline aggregation syntax. Install: ","type":"text"},{"text":"cargo install ag","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"mlr","type":"text","marks":[{"type":"code_inline"}]},{"text":" (Miller) - CSV/TSV log analysis. Install: ","type":"text"},{"text":"brew install miller","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"choco install miller","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"GNU parallel","type":"text","marks":[{"type":"code_inline"}]},{"text":" - parallel processing of split files. Install: ","type":"text"},{"text":"brew install parallel","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"All patterns in this skill work with just rg + jq. Optional tools add interactive exploration (lnav), pipeline aggregation (agrind), and tabular analysis (mlr).","type":"text"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Tool Selection Matrix","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Tool","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Best For","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Speed","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Required?","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"rg","type":"text","marks":[{"type":"code_inline"}]},{"text":" (ripgrep)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Raw pattern matching in any format","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Fastest","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Yes","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"jq","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"JSONL structured extraction and transformation","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Fast","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Yes","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"jq -s","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"JSONL aggregation (slurp all lines into array)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Medium (loads all into memory)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Yes (part of jq)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"lnav","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Interactive exploration, SQL over logs","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Interactive","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Optional","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"agrind","type":"text","marks":[{"type":"code_inline"}]},{"text":" (angle-grinder)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Pipeline aggregation and counting","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Fast","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Optional","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"awk","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Column-based log formats, field extraction","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Fast","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Pre-installed","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"mlr","type":"text","marks":[{"type":"code_inline"}]},{"text":" (Miller)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"CSV/TSV log analysis, statistics","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Fast","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Optional","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"fd","type":"text","marks":[{"type":"code_inline"}]},{"text":" + ","type":"text"},{"text":"rg","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Searching across many log directories","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Fast","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Pre-installed in dev-shell","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"GNU parallel","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Splitting large files for parallel processing","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"N/A (orchestrator)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Optional","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"When to Use What","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Need to...\n│\n├─ Find lines matching a pattern\n│ └─ rg (always fastest for text search)\n│\n├─ Extract specific fields from JSONL\n│ └─ jq -r '[.field1, .field2] | @tsv'\n│\n├─ Count/aggregate over JSONL\n│ └─ jq -sc 'group_by(.field) | map({key: .[0].field, n: length})'\n│\n├─ Search JSONL by value then format results\n│ └─ rg '\"error\"' file.jsonl | jq -r '.message' (two-stage)\n│\n├─ Explore interactively with filtering/SQL\n│ └─ lnav file.log\n│\n├─ Aggregate with pipeline syntax\n│ └─ agrind '* | parse \"* * *\" as ts, level, msg | count by level'\n│\n├─ Extract columns from space-delimited logs\n│ └─ awk '{print $1, $4, $7}' access.log\n│\n└─ Process CSV/TSV logs with headers\n └─ mlr --csv filter '$status >= 400' then stats1 -a count -f status","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"JSONL Quick Reference","type":"text"}]},{"type":"paragraph","content":[{"text":"The most common format for structured logs. One JSON object per line, no trailing commas, no wrapping array.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Stream Filtering (line by line, constant memory)","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Filter by field value\njq -c 'select(.level == \"error\")' app.jsonl\n\n# Filter by nested field\njq -c 'select(.request.method == \"POST\")' app.jsonl\n\n# Filter by multiple conditions\njq -c 'select(.level == \"error\" and .status >= 500)' app.jsonl\n\n# Filter by array contains\njq -c 'select(.tags | index(\"critical\"))' app.jsonl\n\n# Filter by field existence\njq -c 'select(.stack_trace != null)' app.jsonl\n\n# Negate a filter\njq -c 'select(.level != \"debug\")' app.jsonl","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Field Extraction","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Extract single field\njq -r '.message' app.jsonl\n\n# Extract multiple fields as TSV\njq -r '[.timestamp, .level, .message] | @tsv' app.jsonl\n\n# Extract with default for missing fields\njq -r '.error_code // \"none\"' app.jsonl\n\n# Extract nested field safely\njq -r '.response.headers[\"content-type\"] // \"unknown\"' app.jsonl","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Aggregation (requires slurp: loads entire file)","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Count by field value\njq -sc 'group_by(.level) | map({level: .[0].level, count: length})' app.jsonl\n\n# Top-N most common values\njq -sc '[.[].error_type] | group_by(.) | map({type: .[0], count: length}) | sort_by(-.count) | .[:10]' app.jsonl\n\n# Sum a numeric field\njq -sc 'map(.duration_ms) | add' app.jsonl\n\n# Average\njq -sc 'map(.duration_ms) | add / length' app.jsonl\n\n# Min and max\njq -sc 'map(.duration_ms) | {min: min, max: max}' app.jsonl","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Nested Extraction (agent logs, complex structures)","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Extract tool calls from conversation logs\njq -c '.content[]? | select(.type == \"tool_use\") | .name' conversation.jsonl\n\n# De-escape nested JSON strings\njq -c '.content | fromjson' app.jsonl\n\n# Flatten nested arrays\njq -c '[.events[]? | .action]' app.jsonl\n\n# Extract from arrays of objects\njq -c '.results[]? | select(.passed == false) | {test: .name, error: .message}' results.jsonl","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Two-Stage Pipeline (rg for speed, jq for structure)","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Fast prefilter then structured extraction\nrg '\"error\"' app.jsonl | jq -r '[.timestamp, .message] | @tsv'\n\n# Search for specific value then aggregate\nrg '\"timeout\"' app.jsonl | jq -sc 'length'\n\n# Pattern match then extract\nrg '\"user_id\":\"u-123\"' app.jsonl | jq -c '{ts: .timestamp, action: .action}'","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Time-Range Filtering","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Filter by timestamp range (ISO 8601 string comparison works)\njq -c 'select(.timestamp > \"2026-03-08T10:00\" and .timestamp \u003c \"2026-03-08T11:00\")' app.jsonl\n\n# Events in the last N minutes (using epoch seconds)\njq -c --arg cutoff \"$(date -d '30 minutes ago' +%s)\" 'select((.timestamp | sub(\"\\\\.[0-9]+Z$\"; \"Z\") | fromdate) > ($cutoff | tonumber))' app.jsonl\n\n# Extract hour for histogram\njq -r '.timestamp | split(\"T\")[1] | split(\":\")[0]' app.jsonl | sort | uniq -c","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Cross-File Join","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Extract IDs from one file, search in another\njq -r '.request_id' errors.jsonl | while read id; do\n rg \"\\\"$id\\\"\" responses.jsonl | jq -c '{id: .request_id, status: .status}'\ndone\n\n# Faster: build lookup, then join\njq -r '.request_id' errors.jsonl | sort -u > /tmp/error_ids.txt\nrg -Ff /tmp/error_ids.txt responses.jsonl | jq -c '{id: .request_id, status: .status}'\n\n# Join two JSONL files by key using jq --slurpfile\njq --slurpfile lookup \u003c(jq -sc 'map({(.id): .}) | add' lookup.jsonl) \\\n '. + ($lookup[0][.ref_id] // {})' main.jsonl","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Plain Text Log Patterns","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Pattern Search with Context","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Show 5 lines before and after each match\nrg -B5 -A5 \"OutOfMemoryError\" app.log\n\n# Show only matching files\nrg -l \"FATAL\" /var/log/\n\n# Count matches per file\nrg -c \"ERROR\" /var/log/*.log | sort -t: -k2 -rn\n\n# Multiline patterns (stack traces)\nrg -U \"Exception.*\\n(\\s+at .*\\n)+\" app.log","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Column Extraction with awk","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Apache/nginx access log: extract status codes\nawk '{print $9}' access.log | sort | uniq -c | sort -rn\n\n# Extract specific time range from syslog\nawk '$0 >= \"Mar 8 10:00\" && $0 \u003c= \"Mar 8 11:00\"' syslog\n\n# Calculate average response time (column 11)\nawk '{sum += $11; n++} END {print sum/n}' access.log\n\n# Filter by status code and show URL + response time\nawk '$9 >= 500 {print $7, $11\"ms\"}' access.log","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Live Monitoring","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Follow with filtering\ntail -f app.log | rg --line-buffered \"ERROR\"\n\n# Follow JSONL and extract fields\ntail -f app.jsonl | jq --unbuffered -r '[.timestamp, .level, .message] | @tsv'\n\n# Follow multiple files\ntail -f /var/log/service-*.log | rg --line-buffered \"error|warn\"","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Timeline Reconstruction","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Extracting and Sorting by Timestamp","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Merge multiple log files by timestamp\nsort -t' ' -k1,2 service-a.log service-b.log > timeline.log\n\n# JSONL: sort by timestamp field\njq -sc 'sort_by(.timestamp)[]' combined.jsonl > sorted.jsonl\n\n# Extract timestamps and calculate gaps\njq -r '.timestamp' app.jsonl | awk '\n NR > 1 {\n cmd = \"date -d \\\"\" prev \"\\\" +%s\"; cmd | getline t1; close(cmd)\n cmd = \"date -d \\\"\" $0 \"\\\" +%s\"; cmd | getline t2; close(cmd)\n gap = t2 - t1\n if (gap > 5) print gap \"s gap before \" $0\n }\n { prev = $0 }\n'\n\n# Quick duration between first and last event\njq -sc '{start: .[0].timestamp, end: .[-1].timestamp}' app.jsonl","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Calculating Durations Between Events","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Duration between paired events (start/end)\njq -sc '\n group_by(.request_id) |\n map(\n (map(select(.event == \"start\")) | .[0].timestamp) as $start |\n (map(select(.event == \"end\")) | .[0].timestamp) as $end |\n {id: .[0].request_id, start: $start, end: $end}\n )\n' events.jsonl\n\n# Identify the slowest phase\njq -sc '\n sort_by(.timestamp) |\n [range(1; length) | {\n from: .[.-1].event,\n to: .[.].event,\n gap: ((.[.].ts_epoch) - (.[.-1].ts_epoch))\n }] |\n sort_by(-.gap) | .[0]\n' events.jsonl","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Cross-Log Correlation","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"By Correlation ID","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Find a request across all service logs\nfd -e jsonl . /var/log/services/ -x rg \"\\\"req-abc-123\\\"\" {}\n\n# Build a timeline for a single request\nfd -e jsonl . /var/log/services/ -x rg \"\\\"req-abc-123\\\"\" {} \\; | jq -sc 'sort_by(.timestamp)[] | [.timestamp, .service, .event] | @tsv'","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"By Timestamp Window","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Find events within 2 seconds of a known event\n# First get the target timestamp\nTARGET=\"2026-03-08T14:23:15\"\njq -c --arg t \"$TARGET\" '\n select(\n .timestamp > ($t | sub(\"15$\"; \"13\")) and\n .timestamp \u003c ($t | sub(\"15$\"; \"17\"))\n )\n' other-service.jsonl","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"By Session/User","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Reconstruct a user session across log files\nfd -e jsonl . /var/log/ -x rg \"\\\"user-42\\\"\" {} \\; |\n jq -sc 'sort_by(.timestamp)[] | [.timestamp, .service, .action] | @tsv'","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Large File Strategies","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Search Recent Only","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Last 10,000 lines (fast for append-only logs)\ntail -n 10000 huge.log | rg \"pattern\"\n\n# Last N lines of JSONL with structured extraction\ntail -n 5000 huge.jsonl | jq -c 'select(.level == \"error\")'","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Split for Parallel Processing","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Split into 100K-line chunks\nsplit -l 100000 huge.jsonl /tmp/chunk_\n\n# Process in parallel\nfd 'chunk_' /tmp/ -x jq -c 'select(.level == \"error\")' {} > errors.jsonl\n\n# With GNU parallel\nsplit -l 100000 huge.jsonl /tmp/chunk_\nls /tmp/chunk_* | parallel 'jq -c \"select(.level == \\\"error\\\")\" {} >> /tmp/errors.jsonl'","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Streaming for Huge Single JSON","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# SAX-style processing of a huge JSON array\njq --stream 'select(.[0][0] == \"results\" and .[0][-1] == \"status\") | .[1]' huge.json\n\n# Extract items from a huge array without loading all\njq -cn --stream 'fromstream(1 | truncate_stream(inputs))' huge-array.json","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Two-Stage Always","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# ALWAYS faster: rg filters text, jq parses survivors\nrg '\"error\"' huge.jsonl | jq -r '.message'\n\n# vs. SLOW: jq reads and parses every line\njq -r 'select(.level == \"error\") | .message' huge.jsonl","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Search Across Directories","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Multi-Directory Patterns","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Find all JSONL files with errors across trial directories\nfd -e jsonl . trials/ -x rg -l '\"error\"' {}\n\n# Count errors per log file across directories\nfd -e jsonl . trials/ -x bash -c 'echo \"$(rg -c \"\\\"error\\\"\" \"$1\" 2>/dev/null || echo 0) $1\"' _ {}\n\n# Extract and aggregate across directories\nfd -e jsonl . trials/ -x jq -c 'select(.level == \"error\") | {file: input_filename, msg: .message}' {}\n\n# Build summary table from multiple runs\nfor dir in trials/*/; do\n total=$(wc -l \u003c \"$dir/results.jsonl\")\n errors=$(rg -c '\"error\"' \"$dir/results.jsonl\" 2>/dev/null || echo 0)\n echo -e \"$dir\\t$total\\t$errors\"\ndone | column -t -N DIR,TOTAL,ERRORS","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Common Gotchas","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Gotcha","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Why It Hurts","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Fix","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"jq -s","type":"text","marks":[{"type":"code_inline"}]},{"text":" on huge files loads everything into memory","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"OOM crash or swap thrashing on files over ~500MB","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use streaming: ","type":"text"},{"text":"rg","type":"text","marks":[{"type":"code_inline"}]},{"text":" prefilter, ","type":"text"},{"text":"jq --stream","type":"text","marks":[{"type":"code_inline"}]},{"text":", or ","type":"text"},{"text":"split","type":"text","marks":[{"type":"code_inline"}]},{"text":" + parallel","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"JSONL with embedded newlines in string values","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Line-by-line tools (rg, awk, head) split a single record across lines","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"jq -c","type":"text","marks":[{"type":"code_inline"}]},{"text":" to re-compact, or ","type":"text"},{"text":"jq -R 'fromjson?'","type":"text","marks":[{"type":"code_inline"}]},{"text":" to skip malformed lines","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"rg matches JSON keys, not just values","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"rg \"error\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" matches ","type":"text"},{"text":"{\"error_count\": 0}","type":"text","marks":[{"type":"code_inline"}]},{"text":" which is not an error","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"rg '\"level\":\"error\"'","type":"text","marks":[{"type":"code_inline"}]},{"text":" or pipe to ","type":"text"},{"text":"jq 'select(.level == \"error\")'","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Timezone mismatches in timestamp comparisons","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Events appear out of order or time ranges miss data","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Normalize to UTC before comparing: `jq '.timestamp","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Unicode and escape sequences in log messages","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"jq chokes on invalid UTF-8 or double-escaped strings","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Prefilter with ","type":"text"},{"text":"rg -a","type":"text","marks":[{"type":"code_inline"}]},{"text":" (binary mode), or use ","type":"text"},{"text":"jq -R","type":"text","marks":[{"type":"code_inline"}]},{"text":" for raw strings","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Inconsistent JSON schemas across log lines","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"jq","type":"text","marks":[{"type":"code_inline"}]},{"text":" errors on lines missing expected fields","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"//","type":"text","marks":[{"type":"code_inline"}]},{"text":" operator for defaults: ","type":"text"},{"text":".field // \"missing\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"?","type":"text","marks":[{"type":"code_inline"}]},{"text":" for optional: ","type":"text"},{"text":".arr[]?","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Forgetting ","type":"text"},{"text":"-c","type":"text","marks":[{"type":"code_inline"}]},{"text":" flag with jq on JSONL","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"jq pretty-prints each line, output is no longer valid JSONL","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Always use ","type":"text"},{"text":"jq -c","type":"text","marks":[{"type":"code_inline"}]},{"text":" when output feeds into another JSONL consumer","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"tail -f with jq buffering","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Output appears delayed or not at all","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"jq --unbuffered","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"stdbuf -oL jq","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Sorting JSONL by timestamp without slurp","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"sort","type":"text","marks":[{"type":"code_inline"}]},{"text":" command does lexicographic sort on whole lines, not by field","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Either ","type":"text"},{"text":"jq -sc 'sort_by(.timestamp)[]'","type":"text","marks":[{"type":"code_inline"}]},{"text":" or extract timestamp prefix first","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Assuming log files are complete","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Logs may be rotated, compressed, or still being written","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Check for ","type":"text"},{"text":".gz","type":"text","marks":[{"type":"code_inline"}]},{"text":" rotated files: ","type":"text"},{"text":"fd -e gz . /var/log/ -x zcat {} | rg pattern","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Single quotes in jq on Windows","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"PowerShell/cmd do not handle single quotes the same as bash","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use double quotes with escaped inner quotes, or write jq filter to a file","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Reference Files","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"File","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Contents","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Lines","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/jsonl-patterns.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"JSONL extraction, aggregation, transformation, comparison, and performance patterns","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"~700","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/analysis-workflows.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Agent conversation analysis, application log analysis, benchmark result parsing, cross-directory workflows","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"~600","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/tool-setup.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Installation and configuration for jq, lnav, angle-grinder, rg, awk, GNU parallel, Miller","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"~450","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"See Also","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"data-processing","type":"text","marks":[{"type":"strong"}]},{"text":" -- JSON/YAML/TOML processing with jq and yq","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"debug-ops","type":"text","marks":[{"type":"strong"}]},{"text":" -- Systematic debugging methodology, log-based debugging section","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"monitoring-ops","type":"text","marks":[{"type":"strong"}]},{"text":" -- Production observability, alerting, dashboards","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"file-search","type":"text","marks":[{"type":"strong"}]},{"text":" -- Finding files with fd, searching code with rg","type":"text"}]}]}]},{"type":"hr","attrs":{"markup":"---"}}]},"metadata":{"date":"2026-06-05","name":"log-ops","author":"@skillopedia","source":{"stars":21,"repo_name":"claude-mods","origin_url":"https://github.com/0xdarkmatter/claude-mods/blob/HEAD/skills/log-ops/SKILL.md","repo_owner":"0xdarkmatter","body_sha256":"504b5fd118bb76d6271b0e8baaa8b2bc88be91bca53d8252aa032132ebec34a7","cluster_key":"7aea0203eb6ad0afb628192679077e2ead7dfc1aebc6007a89b7b8b9c98f48f5","clean_bundle":{"format":"clean-skill-bundle-v1","source":"0xdarkmatter/claude-mods/skills/log-ops/SKILL.md","attachments":[{"id":"8a740897-e460-5550-ab1c-1daccc243e62","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/8a740897-e460-5550-ab1c-1daccc243e62/attachment","path":"assets/.gitkeep","size":0,"sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","contentType":"text/plain; charset=utf-8"},{"id":"8441ac7a-c556-5c80-8bff-125acba0d51c","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/8441ac7a-c556-5c80-8bff-125acba0d51c/attachment.md","path":"references/analysis-workflows.md","size":18360,"sha256":"ebdb6eb9a8464153c400c7e5a93a2f3818a661faff776947bd9f8e3639227d48","contentType":"text/markdown; charset=utf-8"},{"id":"881735a0-54e7-595d-9d79-8f06a8d1a83e","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/881735a0-54e7-595d-9d79-8f06a8d1a83e/attachment.md","path":"references/jsonl-patterns.md","size":17411,"sha256":"f6bfc35c9415d0670f07c38b4cea35b59fc5253aa01692bf4db84acfcf29c024","contentType":"text/markdown; charset=utf-8"},{"id":"2e2f8f63-8e61-55f0-8b04-d3aef91f8957","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/2e2f8f63-8e61-55f0-8b04-d3aef91f8957/attachment.md","path":"references/tool-setup.md","size":18371,"sha256":"70b9b1ae2402f0080363690c3f619689378e9d758ec5502b83e6701551846477","contentType":"text/markdown; charset=utf-8"},{"id":"2c75f582-5622-5d78-9dae-aa4141ae6fa5","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/2c75f582-5622-5d78-9dae-aa4141ae6fa5/attachment","path":"scripts/.gitkeep","size":0,"sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","contentType":"text/plain; charset=utf-8"}],"bundle_sha256":"2d00c20e629f496fad9c05611809cda8b1a2a92530039c66c6cc91762c4434e3","attachment_count":5,"text_attachments":3,"attachment_storage":"skillopedia-attachments-v1","binary_attachments":2,"excluded_attachments":[]},"cluster_size":1,"skill_md_path":"skills/log-ops/SKILL.md","import_metadata":{"date":"2026-06-05","author":"@skillopedia","version":"v1","category":"devops-infrastructure","category_label":"DevOps"},"exact_dupes_collapsed_into_this":0},"license":"MIT","version":"v1","category":"devops-infrastructure","metadata":{"author":"claude-mods","related-skills":"data-processing, debug-ops, monitoring-ops, file-search, introspect"},"import_tag":"clean-skills-v1","description":"Log analysis and JSONL processing - structured extraction, cross-log correlation, timeline reconstruction, pattern search","allowed-tools":"Read Edit Write Bash Glob Grep Agent"}},"renderedAt":1782979722618}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.