input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

]\n\n{{#if multiline_pattern}}\nmultiline:\n pattern: '{{multiline_pattern}}'\n negate: {{multiline_negate}}\n match: {{multiline_match}}\n max_lines: 5000\n timeout: 10s\n{{/if}}\n\ntags:\n{{#if preserve_original_event}}\n - preserve_original_event\n{{/if}}\n{{#each tags as |tag|}}\n - {{tag}}\n{{/each}}\n{{#contains \"forwarded\" tags}}\npublisher_pipeline.disable_host: true\n{{/contains}}\n{{#if processors}}\nprocessors:\n{{processors}}\n{{/if}}\n```\n\n## Syntax differences between filestream and logfile\n\n| Aspect | Filestream | Logfile (deprecated) |\n|---|---|---|\n| File exclusions | `prospector.scanner.exclude_files:` (YAML list) | `exclude_files: ['\\.gz

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

]` (inline array) |\n| Multiline | `parsers:` block with nested `- multiline:` | `multiline:` or `multiline.*` flat keys |\n| Deprecation flag | Not needed | `allow_deprecated_use: true` required |\n| State tracking | Improved file identity with fingerprinting | Registry-based |\n| Status | Recommended for all new integrations | Deprecated |\n\n## Validation rules\n\n### 1. Paths must use Handlebars variables\n\nPaths must iterate over a template variable. Hardcoded paths prevent users from customizing the file locations through the integration UI.\n\n```yaml\n# correct\npaths:\n{{#each paths as |path|}}\n - {{path}}\n{{/each}}\n\n# wrong -- hardcoded\npaths:\n - /var/log/app/*.log\n```\n\n### 2. Exclude compressed and rotated files\n\nLog directories commonly contain `.gz` or `.zip` archives from log rotation. These must be excluded to avoid re-ingesting old data or attempting to read compressed content.\n\n```yaml\n# filestream\nprospector.scanner.exclude_files:\n - '\\.gz

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

\n - '\\.zip

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

\n\n# logfile\nexclude_files: ['\\.gz

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

]\n```\n\nOmitting this exclusion is an error for any data stream where log rotation may produce compressed files.\n\n### 3. Filestream uses parsers, not multiline keys\n\nFilestream replaced the top-level `multiline.*` keys with a `parsers:` pipeline. Using the old syntax in a filestream template is invalid.\n\n```yaml\n# correct -- filestream\nparsers:\n - multiline:\n pattern: '^{'\n negate: true\n match: after\n\n# wrong -- old syntax in a filestream template\nmultiline.pattern: '^{'\nmultiline.negate: true\nmultiline.match: after\n```\n\n### 4. Filestream uses prospector.scanner for exclusions\n\nThe logfile `exclude_files` key does not work in filestream templates. Use `prospector.scanner.exclude_files` instead.\n\n```yaml\n# correct -- filestream\nprospector.scanner.exclude_files:\n{{#each exclude_files as |pattern|}}\n - {{pattern}}\n{{/each}}\n\n# wrong -- logfile syntax in a filestream template\nexclude_files:\n - '\\.gz

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

\n```\n\n### 5. Logfile must include allow_deprecated_use\n\nEvery `log.yml.hbs` template must contain `allow_deprecated_use: true`. Without this flag the agent emits deprecation warnings or refuses to start (depending on version).\n\n### 6. New integrations must use filestream\n\nIf the PR introduces a new integration or a new data stream and uses `log.yml.hbs`, flag it. The logfile input should only appear in existing integrations that have not yet migrated. New work must use `filestream.yml.hbs`.\n\n### 7. Multiline must include max_lines and timeout\n\nAny multiline configuration should set `max_lines` and `timeout` to prevent unbounded memory growth when a multiline pattern fails to match a terminating line.\n\n```yaml\n# filestream\nparsers:\n - multiline:\n pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'\n negate: true\n match: after\n max_lines: 500\n timeout: 5s\n\n# logfile\nmultiline:\n pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'\n negate: true\n match: after\n max_lines: 5000\n timeout: 10s\n```\n\n## Multiline patterns\n\n### JSON logs (pretty-printed)\n\nWhen JSON objects span multiple lines, match the opening brace and group subsequent lines.\n\n```yaml\nparsers:\n - multiline:\n pattern: '^{'\n negate: true\n match: after\n max_lines: 5000\n timeout: 10s\n```\n\n### Timestamp-based logs\n\nLines that begin with a timestamp start a new event. Lines without a leading timestamp are continuations.\n\n```yaml\n# ISO timestamp\nparsers:\n - multiline:\n pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'\n negate: true\n match: after\n\n# Syslog timestamp\nparsers:\n - multiline:\n pattern: '^[A-Z][a-z]{2} [0-9]{1,2} [0-9]{2}:'\n negate: true\n match: after\n```\n\n### Stack traces (Java / Python)\n\nStack trace continuation lines are appended to the preceding event.\n\n```yaml\n# Java stack traces\nparsers:\n - multiline:\n pattern: '^\\s+(at |\\.\\.\\.)'\n negate: false\n match: after\n\n# Python tracebacks\nparsers:\n - multiline:\n pattern: '^Traceback|^\\s+File'\n negate: false\n match: after\n```\n\n### Whitespace continuation\n\nLines starting with whitespace are treated as continuations of the previous line.\n\n```yaml\nparsers:\n - multiline:\n pattern: '^\\s'\n negate: false\n match: after\n```\n\n## Parsers pipeline (filestream only)\n\nFilestream supports a pipeline of parsers that process log lines before they become events. Parsers execute in order.\n\n| Parser | Description |\n|---|---|\n| `multiline` | Combine multiline messages into a single event |\n| `ndjson` | Parse newline-delimited JSON |\n| `container` | Parse Docker/Kubernetes container log format |\n| `syslog` | Parse RFC 3164 / RFC 5424 syslog lines |\n\nExample combining container parsing with multiline:\n\n```yaml\nparsers:\n - container: {}\n - multiline:\n pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'\n negate: true\n match: after\n```\n\n## Include and exclude filters\n\n### prospector.scanner.exclude_files (filestream)\n\nRegex patterns matched against the full file path. Matching files are skipped entirely.\n\n```yaml\nprospector.scanner.exclude_files:\n - '\\.gz

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

\n - '\\.zip

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

\n - '\\.old

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

\n```\n\n### prospector.scanner.include_files (filestream)\n\nWhen set, only files matching at least one pattern are collected. All others are ignored.\n\n```yaml\nprospector.scanner.include_files:\n - '\\.log

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

\n - '\\.json

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

\n```\n\n### exclude_files (logfile)\n\nSame purpose, different syntax. Takes an inline YAML array.\n\n```yaml\nexclude_files: ['\\.gz

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

]\n```\n\n## Close and clean options\n\nThese settings control when file handles are released and when state entries are cleaned up. Correct values depend on the log rotation strategy of the monitored application.\n\n### Filestream\n\n```yaml\n{{#if close_inactive}}\nclose.on_state_change.inactive: {{close_inactive}}\n{{/if}}\n{{#if close_removed}}\nclose.on_state_change.removed: {{close_removed}}\n{{/if}}\n{{#if clean_removed}}\nclean_removed: {{clean_removed}}\n{{/if}}\n```\n\n| Setting | Description | Typical value |\n|---|---|---|\n| `close.on_state_change.inactive` | Close the file handle after no new data for this duration | `5m` |\n| `close.on_state_change.removed` | Close when the file is removed from disk | `5s` |\n| `clean_removed` | Remove state entry for files that have been deleted | `true` |\n\n### Logfile\n\n```yaml\n{{#if close_inactive}}\nclose_inactive: {{close_inactive}}\n{{/if}}\n{{#if ignore_older}}\nignore_older: {{ignore_older}}\n{{/if}}\n{{#if scan_frequency}}\nscan_frequency: {{scan_frequency}}\n{{/if}}\n```\n\n| Setting | Description | Typical value |\n|---|---|---|\n| `close_inactive` | Close file handle after inactivity | `5m` |\n| `ignore_older` | Skip files not modified within this duration | `72h` |\n| `scan_frequency` | How often to check for new files | `10s` |\n\n## Rotation handling\n\nFilestream handles log rotation through file identity tracking. It recognizes that a rotated file (e.g., `app.log` renamed to `app.log.1`) is the same file and continues reading from where it left off, then picks up the new `app.log`.\n\nKey settings that affect rotation behavior:\n\n```yaml\nprospector.scanner.check_interval: 10s\nprospector.scanner.fingerprint.enabled: true\nprospector.scanner.symlinks: true\n```\n\n| Setting | Purpose |\n|---|---|\n| `check_interval` | How frequently the scanner looks for new or changed files |\n| `fingerprint.enabled` | Use content fingerprinting instead of inode/device for file identity |\n| `symlinks` | Follow symbolic links (disabled by default) |\n\nFor logfile input, symlink handling requires an explicit flag:\n\n```yaml\n{{#if symlinks}}\nsymlinks: {{symlinks}}\n{{/if}}\n```\n\n## Harvester configuration\n\n### Encoding\n\nWhen log files use a non-UTF-8 encoding, specify it explicitly. Mismatched encoding produces garbled text.\n\n```yaml\n{{#if encoding}}\nencoding: {{encoding}}\n{{/if}}\n```\n\nCommon values: `utf-8` (default), `latin1`, `utf-16-be`, `utf-16-le`, `big5`, `shift-jis`.\n\n### Line terminator\n\nMost files use the default (auto-detected), but some formats require explicit configuration:\n\n```yaml\n{{#if line_terminator}}\nline_terminator: {{line_terminator}}\n{{/if}}\n```\n\nValues: `auto`, `line_feed`, `vertical_tab`, `form_feed`, `carriage_return_line_feed`, `next_line`, `line_separator`, `paragraph_separator`, `null_terminator`.\n\n### Recursive glob\n\nEnable `**` expansion in path patterns to match files in subdirectories:\n\n```yaml\nprospector.scanner.recursive_glob: true\n```\n\n## Parameters reference\n\n### Filestream\n\n| Parameter | Type | Description |\n|---|---|---|\n| `paths` | array | File paths, supports glob patterns |\n| `prospector.scanner.exclude_files` | array | Regex patterns to exclude |\n| `prospector.scanner.include_files` | array | Regex patterns to include |\n| `prospector.scanner.symlinks` | bool | Follow symlinks |\n| `prospector.scanner.recursive_glob` | bool | Enable `**` in paths |\n| `prospector.scanner.check_interval` | duration | File scan interval |\n| `prospector.scanner.fingerprint.enabled` | bool | Use content fingerprinting for file identity |\n| `ignore_older` | duration | Ignore files not modified within this duration |\n| `ignore_inactive` | string | Ignore inactive files (`since_first_start`, `since_last_start`) |\n| `encoding` | string | File encoding |\n| `line_terminator` | string | Line terminator override |\n| `close.on_state_change.inactive` | duration | Close file handle after inactivity |\n| `close.on_state_change.removed` | duration | Close after file removal |\n| `clean_removed` | bool | Remove state for deleted files |\n| `parsers` | array | Parser pipeline |\n\n### Logfile\n\n| Parameter | Type | Description |\n|---|---|---|\n| `paths` | array | File paths, supports glob patterns |\n| `allow_deprecated_use` | bool | Required, must be `true` |\n| `exclude_files` | array | Regex patterns to exclude (inline format) |\n| `multiline.pattern` | string | Multiline start pattern |\n| `multiline.negate` | bool | Negate the pattern match |\n| `multiline.match` | string | `after` or `before` |\n| `multiline.max_lines` | int | Maximum lines per event |\n| `multiline.timeout` | duration | Multiline timeout |\n| `close_inactive` | duration | Close file handle after inactivity |\n| `ignore_older` | duration | Skip old files |\n| `scan_frequency` | duration | File discovery interval |\n| `symlinks` | bool | Follow symlinks |\n| `encoding` | string | File encoding |\n\n## Migration from logfile to filestream\n\nWhen evaluating whether a `log.yml.hbs` can be migrated, apply these translations:\n\n| Logfile | Filestream |\n|---|---|\n| `exclude_files: ['\\.gz

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

]` | `prospector.scanner.exclude_files:` followed by YAML list items |\n| `multiline:` block or `multiline.*` keys | `parsers:` block with `- multiline:` |\n| `allow_deprecated_use: true` | Remove entirely |\n| `close_inactive: 5m` | `close.on_state_change.inactive: 5m` |\n| `scan_frequency: 10s` | `prospector.scanner.check_interval: 10s` |\n| `symlinks: true` | `prospector.scanner.symlinks: true` |\n\n### Before (log.yml.hbs)\n\n```yaml\npaths:\n{{#each paths as |path|}}\n - {{path}}\n{{/each}}\nexclude_files: ['\\.gz

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

]\nallow_deprecated_use: true\nmultiline:\n pattern: '^{'\n negate: true\n match: after\n max_lines: 5000\n timeout: 10s\n```\n\n### After (filestream.yml.hbs)\n\n```yaml\npaths:\n{{#each paths as |path|}}\n - {{path}}\n{{/each}}\nprospector.scanner.exclude_files:\n - '\\.gz

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

\nparsers:\n - multiline:\n pattern: '^{'\n negate: true\n match: after\n max_lines: 5000\n timeout: 10s\n```\n\n## Common configuration patterns\n\n### Basic log file (filestream)\n\n```yaml\npaths:\n{{#each paths as |path|}}\n - {{path}}\n{{/each}}\n\nprospector.scanner.exclude_files:\n - '\\.gz

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

\n```\n\n### JSON log files (filestream)\n\n```yaml\npaths:\n{{#each paths as |path|}}\n - {{path}}\n{{/each}}\n\nprospector.scanner.exclude_files:\n - '\\.gz

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

\n\n{{#if multiline_json}}\nparsers:\n - multiline:\n pattern: '^{'\n negate: true\n match: after\n max_lines: 5000\n timeout: 10s\n{{/if}}\n\n{{#if custom}}\n{{custom}}\n{{/if}}\n```\n\n### Application logs with stack traces (filestream)\n\n```yaml\npaths:\n{{#each paths as |path|}}\n - {{path}}\n{{/each}}\n\nprospector.scanner.exclude_files:\n - '\\.gz

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

\n - '\\.old

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

\n\nparsers:\n - multiline:\n pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'\n negate: true\n match: after\n max_lines: 500\n timeout: 5s\n\nprocessors:\n- add_locale: ~\n{{#if processors}}\n{{processors}}\n{{/if}}\n```\n\n### Syslog files (logfile, legacy)\n\n```yaml\npaths:\n{{#each paths as |path|}}\n - {{path}}\n{{/each}}\nexclude_files: ['\\.gz

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…

]\nallow_deprecated_use: true\nmultiline:\n pattern: '^\\s'\n negate: false\n match: after\n\nprocessors:\n- add_locale: ~\n```\n\n## Error handling considerations\n\n- **Paths that may not exist yet**: The agent tolerates missing paths on startup. If the directory appears later, files are picked up on the next scan cycle.\n- **Permission issues**: The agent process must have read access to log files and their parent directories. Journal or container log paths may require elevated permissions.\n- **Disk space for state**: The filestream registry and logfile state file grow with the number of tracked files. In high-churn environments (many short-lived log files), ensure `clean_removed` is enabled to prevent unbounded state growth.\n\n## Review checklist\n\n### Path configuration\n\n- [ ] Paths use `{{#each paths}}` iteration -- **HIGH**\n- [ ] No hardcoded paths -- **HIGH**\n- [ ] Glob patterns are appropriate for the target application -- **MEDIUM**\n\n### File exclusions\n\n- [ ] Compressed files excluded (`.gz input-configurations — Skillopedia , `.zip input-configurations — Skillopedia ) -- **HIGH**\n- [ ] Old/rotated file patterns excluded if applicable -- **MEDIUM**\n- [ ] Correct syntax for input type (`prospector.scanner.exclude_files` for filestream, `exclude_files` for logfile) -- **HIGH**\n\n### Multiline\n\n- [ ] Uses `parsers:` in filestream, `multiline:` in logfile -- **HIGH**\n- [ ] Pattern appropriate for the log format -- **HIGH**\n- [ ] `max_lines` set -- **MEDIUM**\n- [ ] `timeout` set -- **MEDIUM**\n- [ ] `negate` and `match` values correct for the pattern strategy -- **HIGH**\n\n### Input type selection\n\n- [ ] New integrations use filestream, not logfile -- **HIGH**\n- [ ] Logfile templates include `allow_deprecated_use: true` -- **HIGH**\n- [ ] Migration to filestream assessed for existing logfile data streams -- **LOW**\n\n### Harvester and performance\n\n- [ ] Close timeouts appropriate for log volume and rotation strategy -- **MEDIUM**\n- [ ] Encoding specified if non-UTF-8 logs -- **MEDIUM**\n- [ ] Rotation handling considered (fingerprinting, symlinks) -- **LOW**\n- [ ] `clean_removed` enabled for high-churn environments -- **LOW**\n\n### Common patterns\n\n- [ ] `preserve_original_event` is conditional (`{{#if}}`) -- **MEDIUM**\n- [ ] `forwarded` tag and `publisher_pipeline.disable_host` are coupled -- **MEDIUM**\n- [ ] Custom processors passthrough at top level -- **LOW**\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":16718,"content_sha256":"193e68768d36075728bfaf7c73f2d40bc6902c051903fbe858ec3c7ddcdb19da"},{"filename":"references/gcp-pubsub-guide.md","content":"# GCP Pub/Sub input guide\n\nComplete reference for building and reviewing `gcp-pubsub.yml.hbs` templates in Elastic integrations. This input pulls messages from a Google Cloud Pub/Sub subscription.\n\n> **Documentation**: [GCP Pub/Sub Input Reference](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-gcp-pubsub.html)\n\n## File location\n\n```\npackages/\u003cpackage>/data_stream/\u003cdata_stream>/agent/stream/gcp-pubsub.yml.hbs\n```\n\n## Required structure\n\n```yaml\nproject_id: {{project_id}}\n\nsubscription.name: {{subscription_name}}\n\n{{#if topic}}\nsubscription.create: true\ntopic: {{topic}}\n{{/if}}\n\n{{#if credentials_file}}\ncredentials_file: {{credentials_file}}\n{{/if}}\n{{#if credentials_json}}\ncredentials_json: {{credentials_json}}\n{{/if}}\n\ntags:\n{{#if preserve_original_event}}\n - preserve_original_event\n{{/if}}\n{{#each tags as |tag|}}\n - {{tag}}\n{{/each}}\n{{#contains \"forwarded\" tags}}\npublisher_pipeline.disable_host: true\n{{/contains}}\n\n{{#if processors}}\nprocessors:\n{{processors}}\n{{/if}}\n```\n\n## Validation rules\n\n### 1. Project ID required\n\n```yaml\n# correct\nproject_id: {{project_id}}\n\n# wrong -- hardcoded\nproject_id: my-gcp-project-123\n\n# wrong -- missing\n# no project_id specified\n```\n\n### 2. Subscription required (not topic alone)\n\nThe Pub/Sub input is pull-based and requires a subscription. A topic alone is not sufficient -- you cannot pull messages directly from a topic.\n\n```yaml\n# correct -- subscription name\nsubscription.name: {{subscription_name}}\n\n# correct -- auto-create subscription on a topic\nsubscription.name: {{subscription_name}}\nsubscription.create: true\ntopic: {{topic}}\n\n# wrong -- topic without subscription\ntopic: my-topic\n# cannot pull messages without a subscription\n```\n\nWhen `subscription.create` is `true`, the input creates the subscription if it does not exist. The `topic` field is required alongside `subscription.create` so the input knows which topic to subscribe to.\n\n### 3. Authentication must be provided\n\nIntegrations should support both `credentials_file` and `credentials_json`. Workload Identity is acceptable when running on GCE/GKE.\n\n```yaml\n# correct -- both options available\n{{#if credentials_file}}\ncredentials_file: {{credentials_file}}\n{{/if}}\n{{#if credentials_json}}\ncredentials_json: {{credentials_json}}\n{{/if}}\n\n# acceptable -- workload identity (no explicit credentials)\n# when running on GCE/GKE with attached service account\n\n# wrong -- hardcoded credentials\ncredentials_json: '{\"type\": \"service_account\", \"project_id\": \"...\"}'\n```\n\nCredential fields must use `type: password` in the manifest.\n\n### 4. No hardcoded values\n\nAll user-configurable fields must reference Handlebars variables. This includes project ID, subscription name, topic, and credentials.\n\n## Authentication patterns\n\n### Service account credentials file\n\nThe user provides a file path on the agent host pointing to a service account JSON key.\n\n```yaml\n{{#if credentials_file}}\ncredentials_file: {{credentials_file}}\n{{/if}}\n```\n\n### Service account credentials JSON\n\nThe user pastes the service account JSON key content directly into the integration configuration.\n\n```yaml\n{{#if credentials_json}}\ncredentials_json: {{credentials_json}}\n{{/if}}\n```\n\n### Workload Identity\n\nWhen the agent runs on GCE or GKE with an attached service account, no explicit credentials are needed. The Pub/Sub client library uses Application Default Credentials (ADC) automatically.\n\nIf both `credentials_file` and `credentials_json` are omitted, the client falls back to ADC. This is the expected behavior for Workload Identity deployments. The integration documentation should note this as a supported option.\n\n### Credential refresh\n\nService account key files contain long-lived credentials. For production deployments, Workload Identity or ADC with automatic token refresh is preferred over static key files. Static short-lived tokens (e.g., manually generated access tokens) will expire and cause silent collection failures.\n\n## Subscription configuration\n\n### Basic subscription\n\n```yaml\nproject_id: {{project_id}}\nsubscription.name: {{subscription_name}}\n```\n\nThe subscription must already exist in the GCP project.\n\n### Auto-create subscription\n\n```yaml\nproject_id: {{project_id}}\nsubscription.name: {{subscription_name}}\nsubscription.create: true\ntopic: {{topic}}\n```\n\nThe input creates the subscription on startup if it does not exist. The service account must have `pubsub.subscriptions.create` permission on the project or topic.\n\n## Message handling\n\n### Acknowledgment deadline\n\nThe acknowledgment deadline controls how long Pub/Sub waits for an ACK before redelivering the message.\n\n```yaml\n{{#if subscription.ack_deadline}}\nsubscription.ack_deadline: {{subscription.ack_deadline}}\n{{/if}}\n```\n\n- Too short: causes redelivery of messages that are still being processed, leading to duplicates\n- Too long: delays reprocessing when a message genuinely fails\n\nThe default is typically 10 seconds. Increase for integrations where processing (including ingest pipeline execution) takes longer.\n\n### Message retention\n\nRetention is configured at the subscription level in GCP, not in the input template. If the integration documentation specifies retention requirements, verify they are addressed in the GCP setup instructions rather than the template.\n\n### Message ordering\n\nIf ordering is required, the subscription must be configured with ordering enabled (using ordering keys) in GCP. The input preserves order when the subscription provides ordered messages. Flag if ordering requirements are documented but not addressed in the setup guide.\n\n### Flow control\n\nFlow control limits the number of outstanding (unacknowledged) messages the input holds in memory.\n\n```yaml\n{{#if subscription.max_outstanding_messages}}\nsubscription.max_outstanding_messages: {{subscription.max_outstanding_messages}}\n{{/if}}\n{{#if subscription.max_outstanding_bytes}}\nsubscription.max_outstanding_bytes: {{subscription.max_outstanding_bytes}}\n{{/if}}\n```\n\n- `max_outstanding_messages`: maximum number of unacknowledged messages held in memory\n- `max_outstanding_bytes`: maximum total bytes of unacknowledged messages\n\nValues should align with the agent's processing capacity and available memory. Too high can cause backpressure and OOM. Too low throttles throughput unnecessarily.\n\n## Performance\n\n### Number of worker goroutines\n\n```yaml\n{{#if subscription.num_goroutines}}\nsubscription.num_goroutines: {{subscription.num_goroutines}}\n{{/if}}\n```\n\nControls pull concurrency. The default (1) is sufficient for low to moderate throughput. Increase for high-throughput subscriptions. Each goroutine maintains its own streaming pull connection.\n\n### Subscription type\n\nThis input is pull-based. It opens streaming pull connections to the Pub/Sub service. Push subscriptions (where Pub/Sub sends HTTP requests to an endpoint) require a different architecture and are not supported by this input.\n\n### Concurrent acknowledgment\n\nEnsure `subscription.max_outstanding_messages` is at least equal to `queue.mem.flush.min_events` (if configured) to avoid blocking the input. If the outstanding message limit is lower than the flush threshold, the input stalls waiting for the queue to flush while holding unacknowledged messages.\n\n## Common configuration patterns\n\n### Basic Pub/Sub collection\n\n```yaml\nproject_id: {{project_id}}\nsubscription.name: {{subscription_name}}\n\n{{#if credentials_file}}\ncredentials_file: {{credentials_file}}\n{{/if}}\n{{#if credentials_json}}\ncredentials_json: {{credentials_json}}\n{{/if}}\n```\n\n### Auto-create subscription with flow control\n\n```yaml\nproject_id: {{project_id}}\nsubscription.name: {{subscription_name}}\nsubscription.create: true\ntopic: {{topic}}\n\n{{#if subscription.num_goroutines}}\nsubscription.num_goroutines: {{subscription.num_goroutines}}\n{{/if}}\n{{#if subscription.max_outstanding_messages}}\nsubscription.max_outstanding_messages: {{subscription.max_outstanding_messages}}\n{{/if}}\n{{#if subscription.max_outstanding_bytes}}\nsubscription.max_outstanding_bytes: {{subscription.max_outstanding_bytes}}\n{{/if}}\n\n{{#if credentials_file}}\ncredentials_file: {{credentials_file}}\n{{/if}}\n{{#if credentials_json}}\ncredentials_json: {{credentials_json}}\n{{/if}}\n```\n\n## Error handling\n\n### Retry and NACK behavior\n\nMessages should be NACKed on transient failure so Pub/Sub redelivers them. Messages should only be ACKed after successful processing and publication to Elasticsearch. Review that the integration does not ACK messages before they are safely indexed.\n\n### Dead-letter topics\n\nFor subscriptions with a dead-letter topic configured in GCP, messages that exceed the maximum delivery attempts are forwarded to the dead-letter topic instead of being retried indefinitely. This is configured in GCP, not in the template, but the integration documentation should note it if applicable.\n\n### Credential refresh\n\nVerify that the credential method supports automatic refresh:\n- `credentials_file` pointing to a service account key: long-lived, no refresh needed\n- `credentials_json` with a service account key: long-lived, no refresh needed\n- Workload Identity / ADC: automatic token refresh\n- Manually generated access tokens: will expire, not recommended for production\n\n## Parameters reference\n\n| Parameter | Type | Description |\n|---|---|---|\n| `project_id` | string | GCP project ID |\n| `subscription.name` | string | Pub/Sub subscription name |\n| `subscription.create` | bool | Create subscription if it does not exist |\n| `topic` | string | Topic name (required when `subscription.create` is true) |\n| `subscription.num_goroutines` | int | Number of worker goroutines for pull concurrency |\n| `subscription.max_outstanding_messages` | int | Flow control: max unacknowledged messages |\n| `subscription.max_outstanding_bytes` | int | Flow control: max unacknowledged bytes |\n| `subscription.ack_deadline` | duration | Acknowledgment deadline before redelivery |\n| `credentials_file` | string | Path to service account JSON key file |\n| `credentials_json` | string | Service account JSON key content |\n\n## Review checklist\n\n### Configuration\n- [ ] `project_id` uses a variable\n- [ ] `subscription.name` uses a variable\n- [ ] `topic` specified when `subscription.create` is true\n- [ ] No topic-only configuration without a subscription\n\n### Authentication\n- [ ] No hardcoded credentials in the template\n- [ ] Both `credentials_file` and `credentials_json` supported\n- [ ] Workload Identity documented as an option for GCE/GKE\n- [ ] Credential fields use `type: password` in the manifest\n- [ ] Credentials support refresh (not static short-lived tokens)\n\n### Message handling\n- [ ] Acknowledgment deadline appropriate for expected processing time\n- [ ] Retention requirements addressed (in GCP setup, not template)\n- [ ] Flow control configured (`max_outstanding_messages`, `max_outstanding_bytes`)\n- [ ] Ordering addressed if required\n\n### Performance\n- [ ] Worker count (`num_goroutines`) suitable for expected throughput\n- [ ] `max_outstanding_messages` >= `queue.mem.flush.min_events` (if applicable)\n\n### Error handling\n- [ ] Messages NACKed on transient failure for redelivery\n- [ ] Messages ACKed only after successful processing\n- [ ] Dead-letter topic noted if applicable\n\n### Tags and processors\n- [ ] Tags block follows common patterns\n- [ ] Processors passthrough at top level\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":11353,"content_sha256":"139429063f1f878b632a627d2a1870510dc4c7cb09c38d7dc6e6ef4d892f204f"},{"filename":"references/gcs-guide.md","content":"# Google Cloud Storage input guide\n\nComplete reference for building and reviewing `gcs.yml.hbs` templates in Elastic integrations. This input polls Google Cloud Storage buckets for objects and processes them as log events.\n\n> **Documentation**: [GCS Input Reference](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-gcs.html)\n\n## File location\n\n```\npackages/\u003cpackage>/data_stream/\u003cdata_stream>/agent/stream/gcs.yml.hbs\n```\n\n## Required structure\n\n```yaml\nproject_id: {{project_id}}\n\nbuckets:\n - name: {{bucket_name}}\n {{#if file_selectors}}\n file_selectors:\n {{#each file_selectors as |selector|}}\n - regex: \"{{selector.regex}}\"\n {{/each}}\n {{/if}}\n\n{{#if credentials_file}}\ncredentials_file: {{credentials_file}}\n{{/if}}\n{{#if credentials_json}}\ncredentials_json: {{credentials_json}}\n{{/if}}\n\n{{#if poll_interval}}\npoll_interval: {{poll_interval}}\n{{/if}}\n{{#if parse_json}}\nparse_json: {{parse_json}}\n{{/if}}\n{{#if timestamp_epoch}}\ntimestamp_epoch: {{timestamp_epoch}}\n{{/if}}\n\ntags:\n{{#if preserve_original_event}}\n - preserve_original_event\n{{/if}}\n{{#each tags as |tag|}}\n - {{tag}}\n{{/each}}\n{{#contains \"forwarded\" tags}}\npublisher_pipeline.disable_host: true\n{{/contains}}\n\n{{#if processors}}\nprocessors:\n{{processors}}\n{{/if}}\n```\n\n## Validation rules\n\n### 1. Project ID required\n\nEvery GCS template must specify the GCP project ID via a variable.\n\n```yaml\n# correct\nproject_id: {{project_id}}\n\n# wrong -- hardcoded\nproject_id: my-project-12345\n\n# wrong -- missing\n# no project_id specified\n```\n\n### 2. Bucket name required and variable\n\nAt least one bucket must be specified, and the name must reference a Handlebars variable.\n\n```yaml\n# correct\nbuckets:\n - name: {{bucket_name}}\n\n# wrong -- hardcoded\nbuckets:\n - name: my-hardcoded-bucket\n```\n\n### 3. Authentication must be provided\n\nIntegrations should support both `credentials_file` and `credentials_json` to cover different deployment models. Workload Identity (no explicit credentials) is acceptable when running on GCE/GKE with an attached service account.\n\n```yaml\n# correct -- both options available\n{{#if credentials_file}}\ncredentials_file: {{credentials_file}}\n{{/if}}\n{{#if credentials_json}}\ncredentials_json: {{credentials_json}}\n{{/if}}\n\n# acceptable -- workload identity (no explicit credentials needed)\n# when running on GCE/GKE with attached service account\n\n# wrong -- hardcoded credentials\ncredentials_json: '{\"type\": \"service_account\", \"project_id\": \"...\"}'\n```\n\n### 4. File selectors must use variables\n\nWhen file filtering is needed, the regex patterns must be user-configurable, not hardcoded.\n\n```yaml\n# correct\n{{#if file_selectors}}\nfile_selectors:\n{{#each file_selectors as |selector|}}\n - regex: \"{{selector.regex}}\"\n{{/each}}\n{{/if}}\n\n# wrong -- hardcoded patterns\nfile_selectors:\n - regex: \".*\\\\.log$\"\n```\n\n## Authentication patterns\n\n### Service account credentials file\n\nThe user provides a path to a service account JSON key file on the agent host.\n\n```yaml\n{{#if credentials_file}}\ncredentials_file: {{credentials_file}}\n{{/if}}\n```\n\n### Service account credentials JSON\n\nThe user pastes the service account JSON content directly into the integration configuration. The manifest should use `type: password` for this field.\n\n```yaml\n{{#if credentials_json}}\ncredentials_json: {{credentials_json}}\n{{/if}}\n```\n\n### Workload Identity\n\nWhen the agent runs on GCE or GKE with an attached service account, no explicit credentials are needed. The GCS client library uses Application Default Credentials automatically. The integration documentation should mention this as a supported option.\n\nIf both `credentials_file` and `credentials_json` are omitted, the client falls back to ADC. This is the expected behavior for Workload Identity deployments.\n\n## Polling and content handling\n\n### Poll interval\n\n```yaml\n{{#if poll_interval}}\npoll_interval: {{poll_interval}}\n{{/if}}\n```\n\nControls how frequently the input checks the bucket for new or modified objects. The default varies by implementation. Set appropriately based on how often new objects appear in the bucket.\n\n### JSON parsing\n\n```yaml\n{{#if parse_json}}\nparse_json: {{parse_json}}\n{{/if}}\n```\n\nWhen enabled, the input parses each object as JSON. Useful for structured log files (e.g., Cloud Audit Logs exported to GCS in JSON format). When disabled, each line of the object becomes a separate event.\n\n### Timestamp epoch\n\n```yaml\n{{#if timestamp_epoch}}\ntimestamp_epoch: {{timestamp_epoch}}\n{{/if}}\n```\n\nSets a starting point for object processing. Objects with a last-modified time before this epoch are skipped. Useful to avoid reprocessing historical data on first startup.\n\n## Common configuration patterns\n\n### Basic GCS collection\n\n```yaml\nproject_id: {{project_id}}\n\nbuckets:\n - name: {{bucket_name}}\n\n{{#if credentials_file}}\ncredentials_file: {{credentials_file}}\n{{/if}}\n{{#if credentials_json}}\ncredentials_json: {{credentials_json}}\n{{/if}}\n```\n\n### With file filtering\n\n```yaml\nproject_id: {{project_id}}\n\nbuckets:\n - name: {{bucket_name}}\n {{#if file_selectors}}\n file_selectors:\n {{#each file_selectors as |selector|}}\n - regex: \"{{selector.regex}}\"\n {{/each}}\n {{/if}}\n {{#if max_workers}}\n max_workers: {{max_workers}}\n {{/if}}\n\n{{#if credentials_file}}\ncredentials_file: {{credentials_file}}\n{{/if}}\n```\n\n### With object prefix filtering\n\n```yaml\nproject_id: {{project_id}}\n\nbuckets:\n - name: {{bucket_name}}\n {{#if bucket_list_prefix}}\n bucket_list_prefix: {{bucket_list_prefix}}\n {{/if}}\n\n{{#if credentials_file}}\ncredentials_file: {{credentials_file}}\n{{/if}}\n{{#if credentials_json}}\ncredentials_json: {{credentials_json}}\n{{/if}}\n```\n\n`bucket_list_prefix` limits the bucket listing to objects whose key starts with the given prefix. This is more efficient than `file_selectors` when the filtering can be expressed as a prefix, because it reduces the number of objects the GCS API returns.\n\n### JSON log processing\n\n```yaml\nproject_id: {{project_id}}\n\nbuckets:\n - name: {{bucket_name}}\n parse_json: true\n\n{{#if credentials_file}}\ncredentials_file: {{credentials_file}}\n{{/if}}\n{{#if credentials_json}}\ncredentials_json: {{credentials_json}}\n{{/if}}\n```\n\n## Advanced patterns\n\n### Encoding and content type\n\n```yaml\n{{#if encoding}}\nencoding: {{encoding}}\n{{/if}}\n{{#if content_type}}\ncontent_type: {{content_type}}\n{{/if}}\n```\n\nSpecify encoding for objects that are not UTF-8. Content type can override automatic detection when needed.\n\n### Parallel workers per bucket\n\n```yaml\nbuckets:\n - name: {{bucket_name}}\n {{#if max_workers}}\n max_workers: {{max_workers}}\n {{/if}}\n```\n\nControls how many objects are processed in parallel within a single bucket. Higher values improve throughput for buckets with many small objects but increase memory usage.\n\n## Parameters reference\n\n| Parameter | Type | Description |\n|---|---|---|\n| `project_id` | string | GCP project ID |\n| `buckets` | array | List of bucket configurations |\n| `buckets[].name` | string | Bucket name |\n| `buckets[].file_selectors` | array | Regex patterns for filtering objects |\n| `buckets[].max_workers` | int | Parallel workers per bucket |\n| `buckets[].bucket_list_prefix` | string | Object key prefix filter |\n| `credentials_file` | string | Path to service account JSON key file |\n| `credentials_json` | string | Service account JSON key content |\n| `poll_interval` | duration | Polling interval for new objects |\n| `parse_json` | bool | Parse objects as JSON |\n| `timestamp_epoch` | int | Starting timestamp (epoch seconds) |\n| `encoding` | string | Character encoding for object content |\n| `content_type` | string | Content type override |\n\n## Review checklist\n\n### Configuration\n- [ ] `project_id` uses a variable\n- [ ] `buckets[].name` uses a variable\n- [ ] File selectors are configurable (not hardcoded regex)\n- [ ] Poll interval is configurable with a sensible default\n\n### Authentication\n- [ ] No hardcoded credentials in the template\n- [ ] Both `credentials_file` and `credentials_json` supported\n- [ ] Workload Identity documented as an option for GCE/GKE deployments\n- [ ] Credential fields use `type: password` in the manifest\n\n### File processing\n- [ ] Object prefix filtering (`bucket_list_prefix`) used when applicable\n- [ ] Encoding specified for non-UTF-8 content\n- [ ] JSON parsing configurable where applicable\n- [ ] `timestamp_epoch` available to control initial processing window\n\n### Tags and processors\n- [ ] Tags block follows common patterns\n- [ ] Processors passthrough at top level\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":8556,"content_sha256":"ed7e5b801ea5e0e02a004292a00edf62c8891cc02194526e4c90d049fb24419c"},{"filename":"references/http-endpoint-guide.md","content":"# HTTP Endpoint input guide\n\nComplete reference for building and reviewing `http_endpoint.yml.hbs` templates in Elastic integrations.\n\nDocumentation: [HTTP Endpoint Input Reference](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-http_endpoint.html)\n\n## File location\n\n```\npackages/\u003cpackage>/data_stream/\u003cdata_stream>/agent/stream/http_endpoint.yml.hbs\n```\n\n## Required structure\n\nThe HTTP Endpoint input receives data by listening for incoming HTTP requests (webhooks). Every template must configure listener settings and should include authentication and SSL/TLS.\n\n```yaml\nlisten_address: {{listen_address}}\nlisten_port: {{listen_port}}\n\n{{#if url}}\nurl: {{url}}\n{{/if}}\n\n{{#if ssl}}\nssl: {{ssl}}\n{{/if}}\n\n{{#if secret_header}}\nsecret.header: {{secret_header}}\n{{/if}}\n{{#if secret_value}}\nsecret.value: {{secret_value}}\n{{/if}}\n{{#if basic_auth}}\nbasic_auth: {{basic_auth}}\n{{/if}}\n\n{{#if content_type}}\ncontent_type: {{content_type}}\n{{/if}}\n{{#if prefix}}\nprefix: {{prefix}}\n{{/if}}\n\n{{#if response_code}}\nresponse_code: {{response_code}}\n{{/if}}\n{{#if response_body}}\nresponse_body: {{response_body}}\n{{/if}}\n\n{{#if enable_request_tracer}}\ntracer.filename: \"../../logs/http_endpoint/http-request-trace-*.ndjson\"\ntracer.maxbackups: 5\n{{/if}}\n```\n\n## Validation rules\n\n### 1. Listen address and port must use variables\n\nThe bind address and port must never be hardcoded. They must reference Handlebars variables so users can configure them through the manifest.\n\n```yaml\n# Correct\nlisten_address: {{listen_address}}\nlisten_port: {{listen_port}}\n\n# Never acceptable\nlisten_address: 0.0.0.0\nlisten_port: 8080\n```\n\n### 2. URL path should be specific\n\nThe endpoint path should identify the integration or data source. Generic paths like `/` or `/webhook` create conflicts when multiple integrations run on the same agent.\n\n```yaml\n{{#if url}}\nurl: {{url}}\n{{/if}}\n```\n\nGood defaults: `/webhooks/vendor-name`, `/api/v1/events`. Avoid: `/`, `/webhook`.\n\n### 3. SSL/TLS should be available\n\nProduction webhook endpoints should support HTTPS. The SSL configuration block must be present as a conditional.\n\n```yaml\n{{#if ssl}}\nssl: {{ssl}}\n{{/if}}\n```\n\nThe SSL object typically includes:\n\n```yaml\nssl:\n enabled: true\n certificate: /path/to/cert.pem\n key: /path/to/key.pem\n```\n\n### 4. Authentication recommended\n\nA webhook endpoint without authentication is a security risk. Templates should support at least one authentication method: secret header validation, HMAC signature verification, or basic authentication.\n\n```yaml\n{{#if secret_header}}\nsecret.header: {{secret_header}}\n{{/if}}\n{{#if secret_value}}\nsecret.value: {{secret_value}}\n{{/if}}\n```\n\n### 5. Content type handling\n\nThe expected content type should be configurable when the webhook may send different formats.\n\n```yaml\n{{#if content_type}}\ncontent_type: {{content_type}}\n{{/if}}\n```\n\nCommon values: `application/json`, `application/x-ndjson`, `application/x-www-form-urlencoded`.\n\n### 6. Credential redaction with state\n\nThe HTTP Endpoint input supports CEL programs and state management. Credentials stored in the `state` block are safe if they are listed in `redact.fields`. The redaction mechanism masks these values in logs and request traces.\n\n```yaml\nstate:\n api_key: '{{api_key}}'\n\nredact:\n fields:\n - api_key\n```\n\nCredentials in state **with** a corresponding `redact.fields` entry are not a security issue. Credentials in state **without** a corresponding `redact.fields` entry, or hardcoded credentials not using template variables, are security issues.\n\n### 7. Request tracer path must match input type name\n\nThe tracer log directory must match the input type name. A mismatch causes the agent to fail at startup with a path validation error ([elastic/integrations#17619](https://github.com/elastic/integrations/pull/17619)).\n\nFor `http_endpoint`, the directory is `http_endpoint`. Tracer support was added in 8.12 ([elastic/beats#36957](https://github.com/elastic/beats/pull/36957)).\n\nTwo configuration styles exist:\n\n| Style | Min version | Notes |\n|---|---|---|\n| `{{#if enable_request_tracer}}` conditional | 8.12.0+ | Wraps `filename` + `maxbackups` in a Handlebars guard |\n| `tracer:` block with `enabled:` field | 8.15.0+ | Allows trace log cleanup when disabled ([elastic/beats#40005](https://github.com/elastic/beats/pull/40005)) |\n\nConditional format (pre-8.15.0 compatibility):\n\n```yaml\n{{#if enable_request_tracer}}\ntracer.filename: \"../../logs/http_endpoint/http-request-trace-*.ndjson\"\ntracer.maxbackups: 5\n{{/if}}\n```\n\nBlock format (8.15.0+):\n\n```yaml\ntracer:\n enabled: {{enable_request_tracer}}\n filename: \"../../logs/http_endpoint/http-request-trace-*.ndjson\"\n maxbackups: 5\n```\n\nInvalid examples:\n\n```yaml\n# Directory does not match input type\ntracer.filename: \"../../logs/webhook/http-request-trace-*.ndjson\"\n\n# Relative path outside logs directory\ntracer.filename: \"./logs/trace.log\"\n```\n\n## Authentication patterns\n\n### Secret header validation\n\nThe webhook sender includes a secret value in a specific HTTP header. The input validates the header value before accepting the request.\n\n```yaml\n{{#if secret_header}}\nsecret.header: {{secret_header}}\n{{/if}}\n{{#if secret_value}}\nsecret.value: {{secret_value}}\n{{/if}}\n```\n\nExample: the webhook sends `X-Webhook-Secret: my-secret-token`. The template sets `secret.header` to the header name and `secret.value` to the expected value.\n\n### HMAC signature validation\n\nFor webhooks that sign the request body with HMAC (e.g., GitHub webhooks). The input computes the HMAC of the body and compares it to the signature in the header.\n\n```yaml\n{{#if hmac_header}}\nhmac.header: {{hmac_header}}\n{{/if}}\n{{#if hmac_key}}\nhmac.key: {{hmac_key}}\n{{/if}}\n{{#if hmac_type}}\nhmac.type: {{hmac_type}}\n{{/if}}\n{{#if hmac_prefix}}\nhmac.prefix: {{hmac_prefix}}\n{{/if}}\n```\n\nGitHub webhook example:\n\n```yaml\nhmac.header: X-Hub-Signature-256\nhmac.key: {{hmac_key}}\nhmac.type: sha256\nhmac.prefix: \"sha256=\"\n```\n\n### Basic authentication\n\nStandard HTTP basic authentication.\n\n```yaml\n{{#if username}}\nbasic_auth.username: {{username}}\n{{/if}}\n{{#if password}}\nbasic_auth.password: {{password}}\n{{/if}}\n```\n\n### CRC token verification\n\nSome webhook providers (e.g., Twitter/X) send a CRC challenge request that the endpoint must respond to before the provider starts sending events.\n\n```yaml\ncrc.provider: {{crc_provider}}\ncrc.secret: {{crc_secret}}\n```\n\n## Response configuration\n\nThe response sent back to the webhook caller can be customized. This is useful for webhook verification handshakes or for providers that expect a specific response format.\n\n```yaml\n{{#if response_code}}\nresponse_code: {{response_code}}\n{{/if}}\n{{#if response_body}}\nresponse_body: {{response_body}}\n{{/if}}\n```\n\nExample with a custom response:\n\n```yaml\nresponse_code: 200\nresponse_body: '{\"status\": \"received\"}'\n```\n\n## Request limits\n\nLarge payloads or slow clients can be constrained with body size and timeout limits.\n\n```yaml\n{{#if max_body_size}}\nmax_body_size: {{max_body_size}}\n{{/if}}\n{{#if timeout}}\ntimeout: {{timeout}}\n{{/if}}\n```\n\n## CEL program support\n\nThe HTTP Endpoint input supports optional CEL programs for request processing. When a `program` field is present, the input uses CEL to transform incoming requests before they become events.\n\n```yaml\n{{#if program}}\nprogram: |\n // CEL program for request processing\n{{/if}}\n```\n\nWhen CEL is used with state and credentials, the redaction rules from validation rule 6 apply.\n\n## Common configuration patterns\n\n### Basic webhook receiver\n\n```yaml\nlisten_address: {{listen_address}}\nlisten_port: {{listen_port}}\nurl: {{url}}\n\n{{#if secret_header}}\nsecret.header: {{secret_header}}\n{{/if}}\n{{#if secret_value}}\nsecret.value: {{secret_value}}\n{{/if}}\n\n{{#if ssl}}\nssl: {{ssl}}\n{{/if}}\n```\n\n### GitHub webhook with HMAC\n\n```yaml\nlisten_address: {{listen_address}}\nlisten_port: {{listen_port}}\nurl: /webhooks/github\n\ncontent_type: application/json\n\n{{#if hmac_key}}\nhmac.header: X-Hub-Signature-256\nhmac.key: {{hmac_key}}\nhmac.type: sha256\nhmac.prefix: \"sha256=\"\n{{/if}}\n\n{{#if ssl}}\nssl: {{ssl}}\n{{/if}}\n```\n\n### Streaming API with CEL and state\n\n```yaml\nlisten_address: {{listen_address}}\nlisten_port: {{listen_port}}\nurl: {{url}}\n\nstate:\n api_key: '{{api_key}}'\n\nredact:\n fields:\n - api_key\n\n{{#if ssl}}\nssl: {{ssl}}\n{{/if}}\n```\n\n## Parameters reference\n\n| Parameter | Type | Description |\n|---|---|---|\n| `listen_address` | string | Bind address for the listener |\n| `listen_port` | int | Port to listen on |\n| `url` | string | URL path to listen on |\n| `ssl` | object | SSL/TLS configuration |\n| `secret.header` | string | Header name containing the secret value |\n| `secret.value` | string | Expected secret value |\n| `basic_auth.username` | string | Basic auth username |\n| `basic_auth.password` | string | Basic auth password |\n| `hmac.header` | string | Header containing the HMAC signature |\n| `hmac.key` | string | HMAC secret key |\n| `hmac.type` | string | Hash algorithm (sha256, sha1) |\n| `hmac.prefix` | string | Signature prefix string |\n| `content_type` | string | Expected content type of incoming requests |\n| `prefix` | string | JSON key prefix for incoming data |\n| `response_code` | int | HTTP status code to return |\n| `response_body` | string | Response body to return |\n| `max_body_size` | string | Maximum allowed request body size |\n| `timeout` | duration | Request timeout for slow clients |\n| `crc.provider` | string | CRC challenge provider name |\n| `crc.secret` | string | CRC challenge secret |\n| `tracer.enabled` | bool | Enable/disable request tracing (8.15.0+) |\n| `tracer.filename` | string | Trace log file path (8.12.0+) |\n| `tracer.maxbackups` | int | Maximum number of trace log backup files |\n\n## Review checklist\n\n### Listener configuration\n\n- [ ] `listen_address` uses a Handlebars variable -- **HIGH**\n- [ ] `listen_port` uses a Handlebars variable -- **HIGH**\n- [ ] URL path is specific, not generic (`/` or `/webhook`) -- **MEDIUM**\n\n### Security\n\n- [ ] At least one authentication method configured (secret header, HMAC, or basic auth) -- **HIGH**\n- [ ] No hardcoded secrets or credentials -- **CRITICAL**\n- [ ] SSL/TLS configuration block available -- **HIGH**\n- [ ] Webhook signature validation present when the provider supports it (e.g., HMAC for GitHub) -- **MEDIUM**\n- [ ] Credentials in `state` have corresponding `redact.fields` entries -- **HIGH**\n\n### Content handling\n\n- [ ] Content type configurable when the webhook may send different formats -- **MEDIUM**\n- [ ] Response code and body customizable when the provider requires specific responses -- **LOW**\n\n### Request processing\n\n- [ ] Max body size configured when large payloads are expected -- **LOW**\n- [ ] Request timeout set -- **LOW**\n- [ ] CEL program validated if present -- **MEDIUM**\n\n### Request tracer\n\n- [ ] Tracer directory matches `http_endpoint` -- **HIGH**\n- [ ] Tracer format correct for target stack version (conditional vs block) -- **LOW**\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":10969,"content_sha256":"afb09fba225344d510920bc864f3ac5d293a359b9b986aa2df2bc76a9064cd03"},{"filename":"references/httpjson-guide.md","content":"# HTTPJSON input guide\n\nComplete reference for building and reviewing `httpjson.yml.hbs` templates in Elastic integrations.\n\n## Template syntax\n\nHTTPJSON templates use Go templates with `[[` `]]` delimiters (not `{{` `}}`, which are reserved for Handlebars). The Go template layer executes at request/response time inside the agent.\n\n### Variable access\n\n| Variable | Description |\n|---|---|\n| `.cursor` | Map of persisted cursor values from the previous poll cycle |\n| `.last_response.body` | Parsed body of the most recent HTTP response |\n| `.last_response.header.Get` | Access response headers, e.g. `[[ .last_response.header.Get \"Link\" ]]` |\n| `.last_response.url.params.Get` | Access URL query parameters from the response URL |\n| `.last_response.page` | Zero-indexed page counter incremented per pagination step |\n| `.last_event` | The last event produced in the current page |\n| `.first_event` | The first event produced in the current page |\n| `.body` | The request body being constructed (available in request transforms) |\n| `.header.Get` | Access request headers (available in request transforms) |\n\n### Date formatting\n\nGo templates use the reference time `2006-01-02T15:04:05Z07:00` (Mon Jan 2 15:04:05 MST 2006) for format strings. Common patterns:\n\n| Format string | Output example |\n|---|---|\n| `2006-01-02T15:04:05Z` | `2024-03-15T14:30:00Z` |\n| `2006-01-02T15:04:05.000Z` | `2024-03-15T14:30:00.000Z` |\n| `2006-01-02` | `2024-03-15` |\n| `01/02/2006` | `03/15/2024` |\n| `1136214245` | Unix epoch seconds |\n\n### Conditionals\n\n```yaml\nvalue: >-\n [[- if .last_response.body.nextPageToken -]]\n [[- .last_response.body.nextPageToken -]]\n [[- end -]]\n```\n\nThe conditional evaluates the value; when false or empty, it produces an empty string. This pattern is critical for pagination termination.\n\n### Math operations\n\n```yaml\nvalue: '[[ add .last_response.page 1 ]]'\nvalue: '[[ mul .last_response.body.meta.per_page .last_response.page ]]'\n```\n\n## Required structure\n\nEvery HTTPJSON template must include these top-level keys in order.\n\n### interval\n\n```yaml\ninterval: {{interval}}\n```\n\nMust reference a Handlebars variable, never hardcoded.\n\n### Request tracer\n\nTwo valid styles exist. Use the block format for packages targeting 8.15.0+.\n\nConditional format (pre-8.15.0 compatibility):\n\n```yaml\n{{#if enable_request_tracer}}\nrequest.tracer.filename: \"../../logs/httpjson/httpjson-\u003cinput-type-name>.ndjson\"\nrequest.tracer.maxbackups: 5\n{{/if}}\n```\n\nBlock format (8.15.0+):\n\n```yaml\nrequest.tracer:\n filename: \"../../logs/httpjson/httpjson-\u003cinput-type-name>.ndjson\"\n maxbackups: 5\n enabled: {{enable_request_tracer}}\n```\n\nThe `\u003cinput-type-name>` in the tracer path must match the specific input type name for the data stream (e.g., `httpjson-myapi-events`).\n\n### Request configuration\n\n```yaml\nrequest.method: GET\nrequest.url: {{url}}/api/v1/events\n{{#if proxy_url}}\nrequest.proxy_url: {{proxy_url}}\n{{/if}}\n{{#if ssl}}\nrequest.ssl: {{ssl}}\n{{/if}}\nrequest.timeout: {{request_timeout}}\n```\n\n- `request.method`: GET or POST\n- `request.url`: must use a Handlebars variable for the base URL\n- Proxy and SSL are conditional blocks\n- Timeout references a manifest variable\n\n### Authentication\n\nSee the Authentication patterns section below for all supported methods.\n\n### request.transforms\n\nTransforms modify the request before it is sent. Common uses: setting headers, adding query parameters, injecting cursor values.\n\n```yaml\nrequest.transforms:\n - set:\n target: url.params.since\n value: '[[ .cursor.since ]]'\n default: '[[ now (parseDuration \"-{{initial_interval}}\") ]]'\n - set:\n target: header.Authorization\n value: 'Bearer {{api_token}}'\n```\n\n### response.split\n\nSplits the response body into individual events. Always targets the array field containing the records.\n\n```yaml\nresponse.split:\n target: body.data\n type: array\n ignore_empty_value: true\n```\n\n### Pagination\n\nSee the Pagination patterns section below.\n\n### Cursor\n\nSee the Cursor persistence section below.\n\n### Tags and processors\n\nFollow the common patterns from `common-input-patterns.md`:\n\n```yaml\ntags:\n{{#if preserve_original_event}}\n - preserve_original_event\n{{/if}}\n{{#each tags as |tag|}}\n - {{tag}}\n{{/each}}\n{{#contains \"forwarded\" tags}}\npublisher_pipeline.disable_host: true\n{{/contains}}\n{{#if processors}}\nprocessors:\n{{processors}}\n{{/if}}\n```\n\n## Validation rules\n\nThese nine rules apply to every HTTPJSON template.\n\n### 1. Interval must use a variable\n\n`interval` must reference `{{interval}}`, not a hardcoded value like `60s`.\n\n### 2. Tracer path must match input type name\n\nThe tracer filename must contain the specific input type identifier for the data stream. Two valid styles:\n\n- Conditional: `{{#if enable_request_tracer}}request.tracer.filename: \"../../logs/httpjson/httpjson-\u003cname>.ndjson\"{{/if}}`\n- Block: `request.tracer:` with `filename:` and `enabled: {{enable_request_tracer}}`\n\n### 3. response.split on array field with ignore_empty_value\n\nThe split must target the correct array field from the API response and include `ignore_empty_value: true` to handle empty responses gracefully:\n\n```yaml\nresponse.split:\n target: body.results\n type: array\n ignore_empty_value: true\n```\n\n### 4. Pagination must terminate\n\nEvery pagination block must contain a conditional that evaluates to an empty string when there are no more pages. Without this, the agent loops indefinitely.\n\n```yaml\nresponse.pagination:\n - set:\n target: url.params.cursor\n value: >-\n [[- if .last_response.body.meta.next_cursor -]]\n [[- .last_response.body.meta.next_cursor -]]\n [[- end -]]\n fail_on_template_error: true\n```\n\nWhen the API returns no `next_cursor`, the template produces an empty string, which (combined with `fail_on_template_error: true`) stops pagination.\n\n### 5. fail_on_template_error required on pagination sets\n\nAll `set` transforms in `response.pagination` that use Go template conditionals must include `fail_on_template_error: true`. This ensures that an empty value (from a termination conditional) correctly signals the end of pagination rather than silently continuing.\n\n### 6. Cursor ignore_empty_value: true\n\nCursor entries should include `ignore_empty_value: true` so the cursor is not overwritten with empty values on pages that lack the tracked field:\n\n```yaml\ncursor:\n since:\n value: '[[ .last_event.timestamp ]]'\n ignore_empty_value: true\n```\n\n### 7. No hardcoded credentials\n\nAuthentication credentials (tokens, API keys, client secrets, passwords) must always reference Handlebars variables. Hardcoded credentials in templates are a critical security issue.\n\n### 8. Preserve time parameters during pagination\n\nWhen the request uses time-range parameters (e.g., `since`, `until`, `start_time`, `end_time`), these parameters must be preserved during pagination. If pagination modifies the URL parameters, it must not overwrite existing time bounds. Use `default` to set initial values and cursor values to maintain them:\n\n```yaml\nrequest.transforms:\n - set:\n target: url.params.since\n value: '[[ .cursor.since ]]'\n default: '[[ now (parseDuration \"-{{initial_interval}}\") ]]'\n - set:\n target: url.params.until\n value: '[[ now ]]'\n```\n\n### 9. POST must set response.request_body_on_pagination\n\nWhen `request.method` is POST, the template must include:\n\n```yaml\nresponse.request_body_on_pagination: true\n```\n\nThis ensures the original POST body is preserved during pagination requests. Without it, pagination requests revert to empty bodies.\n\n## Pagination patterns\n\n### Cursor-based pagination\n\nThe API returns a cursor token; the next request includes it to fetch the next page.\n\n```yaml\nresponse.pagination:\n - set:\n target: url.params.cursor\n value: >-\n [[- if .last_response.body.meta.next_cursor -]]\n [[- .last_response.body.meta.next_cursor -]]\n [[- end -]]\n fail_on_template_error: true\n```\n\n### Offset-based pagination\n\nThe API accepts an offset parameter. Increment by page size each iteration.\n\n```yaml\nresponse.pagination:\n - set:\n target: url.params.offset\n value: '[[ add .last_response.body.meta.offset .last_response.body.meta.limit ]]'\n - set:\n target: url.params.limit\n value: '[[ .last_response.body.meta.limit ]]'\n - set:\n target: url.params.offset\n value: >-\n [[- if gt (len .last_response.body.data) 0 -]]\n [[- add .last_response.body.meta.offset .last_response.body.meta.limit -]]\n [[- end -]]\n fail_on_template_error: true\n```\n\n### Page number pagination\n\nThe API accepts a page number parameter.\n\n```yaml\nresponse.pagination:\n - set:\n target: url.params.page\n value: >-\n [[- if .last_response.body.has_more -]]\n [[- add .last_response.page 2 -]]\n [[- end -]]\n fail_on_template_error: true\n```\n\nNote: `.last_response.page` is zero-indexed, so add 2 to get the next 1-indexed page number.\n\n### Link-based pagination (Link header)\n\nThe API returns a `Link` header with a `next` URL.\n\n```yaml\nresponse.pagination:\n - set:\n target: url.value\n value: >-\n [[- if .last_response.header.Get \"Link\" -]]\n [[- with (parseLinkHeader (.last_response.header.Get \"Link\")) -]]\n [[- .next -]]\n [[- end -]]\n [[- end -]]\n fail_on_template_error: true\n```\n\n### POST body pagination\n\nFor POST APIs where pagination tokens go in the request body. Requires `response.request_body_on_pagination: true`.\n\n```yaml\nresponse.request_body_on_pagination: true\nresponse.pagination:\n - set:\n target: body.pagination_token\n value: >-\n [[- if .last_response.body.pagination_token -]]\n [[- .last_response.body.pagination_token -]]\n [[- end -]]\n fail_on_template_error: true\n```\n\n### Sync ID pattern\n\nSome APIs use a two-phase sync model: an initial call returns a sync ID, subsequent calls use it to poll for changes.\n\n```yaml\nresponse.pagination:\n - set:\n target: url.params.sync_id\n value: '[[ .last_response.body.sync_id ]]'\n - set:\n target: url.params.cursor\n value: >-\n [[- if .last_response.body.has_more -]]\n [[- .last_response.body.cursor -]]\n [[- end -]]\n fail_on_template_error: true\n```\n\n## Advanced patterns\n\n### Complex POST body with JSON filters\n\nSome APIs require complex JSON bodies with filters, date ranges, and nested structures:\n\n```yaml\nrequest.method: POST\nrequest.body:\n query:\n filters:\n - field: \"created_at\"\n operator: \"gte\"\n value: '[[ .cursor.since ]]'\n sort:\n - field: \"created_at\"\n order: \"asc\"\n limit: {{batch_size}}\nresponse.request_body_on_pagination: true\n```\n\n### Advanced security authentication\n\nFor APIs requiring multi-step authentication (e.g., token exchange before data requests), use `request.transforms` to manage the auth flow:\n\n```yaml\nauth:\n oauth2:\n client:\n id: '{{client_id}}'\n secret: '{{client_secret}}'\n token_url: '{{token_url}}'\n scopes:\n - '{{scope}}'\n```\n\n### Nested response splits\n\nWhen the response contains nested arrays (e.g., events inside records inside pages):\n\n```yaml\nresponse.split:\n target: body.records\n type: array\n ignore_empty_value: true\n split:\n target: body.records.events\n type: array\n keep_parent: true\n ignore_empty_value: true\n```\n\nUse `keep_parent: true` on nested splits to preserve parent fields in child events.\n\n### Map type split\n\nWhen the response is a map (object) rather than an array, use `type: map`:\n\n```yaml\nresponse.split:\n target: body.data\n type: map\n ignore_empty_value: true\n```\n\nEach key-value pair becomes a separate event. The key is available as `.key` and the value as `.value` in subsequent transforms.\n\n### Handlebars array parameters\n\nWhen a manifest variable is an array (e.g., multiple endpoints or resource types), iterate with Handlebars:\n\n```yaml\n{{#each resource_types}}\n- request.url: {{../url}}/api/v1/{{this}}\n request.method: GET\n{{/each}}\n```\n\n### Rate limiting configuration\n\n```yaml\nrequest.rate_limit:\n limit: '[[ .last_response.header.Get \"X-RateLimit-Limit\" ]]'\n remaining: '[[ .last_response.header.Get \"X-RateLimit-Remaining\" ]]'\n reset: '[[ .last_response.header.Get \"X-RateLimit-Reset\" ]]'\n```\n\n### Cursor update only on first page\n\nWhen the cursor should only update from the first page of results (to avoid advancing the cursor past unpaginated data):\n\n```yaml\ncursor:\n since:\n value: >-\n [[- if eq .last_response.page 0 -]]\n [[- .first_event.timestamp -]]\n [[- end -]]\n ignore_empty_value: true\n```\n\n## Authentication patterns\n\n> **Federated Identity / Cloud Connectors:** the patterns below are the standard agent-based methods. For agentless Identity Federation (`auth.aws`, `use_cloud_connectors`, `var_groups`, `provider_permissions`), follow `references/federated-identity-aws.md`. `httpjson` is federation-eligible when used in an agentless AWS package.\n\n### Basic authentication\n\n```yaml\nauth.basic:\n user: '{{username}}'\n password: '{{password}}'\n```\n\n### API key header\n\n```yaml\nrequest.transforms:\n - set:\n target: header.Authorization\n value: '{{api_key_prefix}} {{api_key}}'\n```\n\nOr using a dedicated header name:\n\n```yaml\nrequest.transforms:\n - set:\n target: header.X-API-Key\n value: '{{api_key}}'\n```\n\n### OAuth2 client credentials\n\n```yaml\nauth.oauth2:\n client.id: '{{client_id}}'\n client.secret: '{{client_secret}}'\n token_url: '{{token_url}}'\n```\n\n### OAuth2 with custom provider\n\n```yaml\nauth.oauth2:\n client.id: '{{client_id}}'\n client.secret: '{{client_secret}}'\n token_url: '{{token_url}}'\n scopes:\n{{#each oauth_scopes}}\n - {{this}}\n{{/each}}\n endpoint_params:\n resource:\n - '{{resource}}'\n```\n\n### Mutual exclusion\n\nWhen an integration supports multiple authentication methods, use Handlebars conditionals to select one:\n\n```yaml\n{{#if auth_method_basic}}\nauth.basic:\n user: '{{username}}'\n password: '{{password}}'\n{{/if}}\n{{#if auth_method_oauth2}}\nauth.oauth2:\n client.id: '{{client_id}}'\n client.secret: '{{client_secret}}'\n token_url: '{{token_url}}'\n{{/if}}\n{{#if auth_method_api_key}}\nrequest.transforms:\n - set:\n target: header.Authorization\n value: 'Bearer {{api_key}}'\n{{/if}}\n```\n\nOnly one `auth_method_*` variable should be true at a time. The manifest should enforce this with `show_user` and conditional `required` settings, or by using a single `auth_type` select variable that controls which credential fields are shown.\n\n## Cursor persistence\n\n### How cursor persistence works\n\nThe HTTPJSON input persists cursor values between poll cycles. Cursor values are only saved when events have been successfully published to Elasticsearch. If the agent restarts, it resumes from the last persisted cursor.\n\n### ignore_empty_value semantics\n\nWhen `ignore_empty_value: true` is set on a cursor entry, the cursor retains its previous value if the current evaluation produces an empty string. This is essential for:\n- Responses that do not contain the tracked field on every page\n- Empty responses where no events are produced\n- First-page-only cursor updates\n\n### Empty response handling with pipeline drop\n\nSome APIs return cursor tokens even with empty data arrays. If the pipeline uses a `drop` processor to discard empty events, no events are published, and the cursor is not updated. This creates an infinite loop where the agent keeps requesting the same empty page.\n\nTo handle this, ensure the `response.split` has `ignore_empty_value: true` so empty arrays produce no events (and no phantom events that need dropping), and the cursor tracks a field that only appears when real data exists.\n\n## API conformance table\n\nMapping from common API pagination styles to the appropriate HTTPJSON pattern.\n\n| API pagination style | HTTPJSON pattern | Key fields |\n|---|---|---|\n| Returns `next_cursor` / `cursor` token | Cursor-based | `response.pagination` with cursor conditional |\n| Returns total count + accepts offset | Offset-based | `response.pagination` with add/limit math |\n| Returns `has_more` + page number | Page number | `response.pagination` with page increment |\n| Returns `Link` header with rel=\"next\" | Link-based | `response.pagination` with `parseLinkHeader` |\n| POST body with pagination token | POST body | `response.request_body_on_pagination: true` |\n| Sync ID + incremental cursor | Sync ID | Two-step: sync_id preserved, cursor conditional |\n| Returns `next_url` in body | Cursor-based (URL variant) | Set `url.value` from response body field |\n| Returns results until empty page | Offset or page with length check | Terminate when `len .last_response.body.data` is 0 |\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":16707,"content_sha256":"02011b8e29f6e31e17911ba49e8ab5c62ad658a65a766d518b3c8419829c86f0"},{"filename":"references/journald-guide.md","content":"# Journald input guide\n\nComplete reference for building and reviewing `journald.yml.hbs` templates in Elastic integrations.\n\nThe journald input reads log entries from systemd journal files. It supports filtering by systemd unit, syslog identifier, transport, priority, and other journal fields.\n\n## File location\n\n```\npackages/\u003cpackage>/data_stream/\u003cdata_stream>/agent/stream/journald.yml.hbs\n```\n\n## Required structure\n\n```yaml\n{{#if paths}}\npaths:\n{{#each paths as |path|}}\n - {{path}}\n{{/each}}\n{{/if}}\n\n{{#if include_matches}}\ninclude_matches:\n{{#each include_matches as |match|}}\n - {{match}}\n{{/each}}\n{{/if}}\n\n{{#if seek}}\nseek: {{seek}}\n{{/if}}\n\ntags:\n{{#if preserve_original_event}}\n - preserve_original_event\n{{/if}}\n{{#each tags as |tag|}}\n - {{tag}}\n{{/each}}\n{{#contains \"forwarded\" tags}}\npublisher_pipeline.disable_host: true\n{{/contains}}\n{{#if processors}}\nprocessors:\n{{processors}}\n{{/if}}\n```\n\n## Validation rules\n\n### 1. Include matches should be present and use variables\n\nEvery journald template should filter journal entries to the relevant systemd unit or syslog identifier. Collecting the entire journal without filtering produces excessive, unrelated data.\n\n```yaml\n# correct -- configurable via template variable\n{{#if include_matches}}\ninclude_matches:\n{{#each include_matches as |match|}}\n - {{match}}\n{{/each}}\n{{/if}}\n\n# acceptable -- fixed filter for a specific service integration\ninclude_matches:\n - _SYSTEMD_UNIT=nginx.service\n\n# wrong -- hardcoded unit name when the integration should be configurable\ninclude_matches:\n - _SYSTEMD_UNIT=myapp.service\n```\n\nWhen the integration targets a specific, well-known service (e.g., Docker, sshd), a hardcoded unit name is acceptable. When the unit name varies by deployment, it must be a template variable.\n\n### 2. Seek position should be configurable\n\nThe `seek` parameter controls where reading begins when no cursor state exists. It should be a template variable so the user can choose between `head`, `tail`, and `cursor`.\n\n```yaml\n# correct\n{{#if seek}}\nseek: {{seek}}\n{{/if}}\n\n# wrong -- hardcoded\nseek: head\n```\n\nValues:\n\n| Value | Behavior |\n|---|---|\n| `head` | Start from the oldest available entry |\n| `tail` | Start from the newest entry (skip history) |\n| `cursor` | Resume from the last recorded position (default) |\n\n### 3. Journal paths are optional but must use variables when present\n\nIf not specified, the input reads from the default system journal at `/var/log/journal`. Custom paths are needed when journal files are stored in a non-standard location (e.g., container mounts, remote journal directories).\n\n```yaml\n# correct -- optional, defaults to system journal\n{{#if paths}}\npaths:\n{{#each paths as |path|}}\n - {{path}}\n{{/each}}\n{{/if}}\n\n# wrong -- hardcoded path\npaths:\n - /var/log/journal\n```\n\n### 4. Include match fields must use valid journal field names\n\nJournal field matchers follow the format `FIELD_NAME=value`. Common valid fields are listed in the field mapping section below. Using an invalid field name silently produces no matches.\n\n### 5. No overly broad collection\n\nA journald template without any `include_matches` collects every entry from the journal. This is almost never the intended behavior for an integration data stream. Flag templates that lack filtering.\n\n## Include match fields\n\nThese are the standard systemd journal fields used in `include_matches` filters.\n\n| Field | Description | Example value |\n|---|---|---|\n| `_SYSTEMD_UNIT` | Systemd unit name | `nginx.service` |\n| `SYSLOG_IDENTIFIER` | Syslog program identifier | `myapp` |\n| `_TRANSPORT` | Journal transport mechanism | `syslog`, `journal`, `stdout`, `kernel` |\n| `PRIORITY` | Syslog priority level (0 = emergency, 7 = debug) | `0` through `7` |\n| `_UID` | User ID of the logging process | `1000` |\n| `_GID` | Group ID of the logging process | `1000` |\n| `_COMM` | Process command name | `nginx` |\n| `_EXE` | Process executable path | `/usr/sbin/nginx` |\n| `_PID` | Process ID | `12345` |\n\nMultiple `include_matches` entries are OR-combined within the same field and AND-combined across different fields.\n\n## Cursor handling\n\nThe journald input tracks its reading position using journal cursors. When the agent restarts, it resumes from the last persisted cursor position.\n\n### Seek vs cursor interaction\n\n| Scenario | Behavior |\n|---|---|\n| First start, no cursor | Uses `seek` value (`head`, `tail`, or default `cursor`) |\n| Restart with existing cursor | Resumes from cursor regardless of `seek` value |\n| Cursor points to deleted entry | Uses `cursor_seek_fallback` if set, otherwise `head` |\n\n### cursor_seek_fallback\n\nWhen the cursor references a journal entry that has been rotated out, this setting determines where to resume:\n\n```yaml\n{{#if cursor_seek_fallback}}\ncursor_seek_fallback: {{cursor_seek_fallback}}\n{{/if}}\n```\n\nValues: `head`, `tail`. If not set, defaults to `head` (re-read all available entries).\n\n## Field mapping conventions\n\nJournal entries produce fields under the `journald` namespace. The input maps standard journal fields to ECS and Elastic agent fields:\n\n| Journal field | Mapped Elastic field |\n|---|---|\n| `MESSAGE` | `message` |\n| `_HOSTNAME` | `host.name` |\n| `_SYSTEMD_UNIT` | `systemd.unit` |\n| `SYSLOG_IDENTIFIER` | `syslog.identifier` |\n| `PRIORITY` | `syslog.priority` |\n| `_TRANSPORT` | `journald.transport` |\n| `_PID` | `process.pid` |\n| `_UID` | `user.id` |\n| `_GID` | `group.id` |\n| `_COMM` | `process.name` |\n| `_EXE` | `process.executable` |\n\nCustom journal fields (application-defined) appear under `journald.custom`.\n\n## Advanced configuration\n\n### Field filtering\n\nReduce event size by including or excluding specific journal fields. Useful when the journal entry contains many fields that are not needed for the integration.\n\n```yaml\n{{#if include_fields}}\ninclude_fields:\n{{#each include_fields as |field|}}\n - {{field}}\n{{/each}}\n{{/if}}\n{{#if exclude_fields}}\nexclude_fields:\n{{#each exclude_fields as |field|}}\n - {{field}}\n{{/each}}\n{{/if}}\n```\n\n### Kernel messages\n\nControl whether kernel ring buffer messages (`_TRANSPORT=kernel`) are included. Some integrations need kernel logs (e.g., firewall, audit), while others should exclude them to reduce noise.\n\n```yaml\n{{#if include_kernel_messages}}\ninclude_kernel_messages: {{include_kernel_messages}}\n{{/if}}\n```\n\n### Since filter\n\nOnly read entries newer than a relative time. Useful for preventing historical replay on first start:\n\n```yaml\n{{#if since}}\nsince: {{since}}\n{{/if}}\n```\n\n## Common configuration patterns\n\n### System journal for a specific service\n\n```yaml\n{{#if include_matches}}\ninclude_matches:\n{{#each include_matches as |match|}}\n - {{match}}\n{{/each}}\n{{/if}}\n\n{{#if seek}}\nseek: {{seek}}\n{{/if}}\n```\n\n### Specific service with fixed unit\n\n```yaml\ninclude_matches:\n - _SYSTEMD_UNIT={{unit_name}}\n\n{{#if seek}}\nseek: {{seek}}\n{{/if}}\n```\n\n### Container runtime logs\n\n```yaml\ninclude_matches:\n - _SYSTEMD_UNIT=docker.service\n - _SYSTEMD_UNIT=containerd.service\n\n{{#if seek}}\nseek: {{seek}}\n{{/if}}\n```\n\n### Syslog identifier filtering\n\n```yaml\ninclude_matches:\n - SYSLOG_IDENTIFIER={{syslog_identifier}}\n\n{{#if seek}}\nseek: {{seek}}\n{{/if}}\n```\n\n## Parameters reference\n\n| Parameter | Type | Description |\n|---|---|---|\n| `paths` | array | Journal file paths (defaults to `/var/log/journal` if omitted) |\n| `include_matches` | array | Journal field matchers in `FIELD=value` format |\n| `seek` | string | Start position: `head`, `tail`, `cursor` |\n| `cursor_seek_fallback` | string | Fallback when cursor is invalid: `head`, `tail` |\n| `since` | duration | Only entries newer than this relative time |\n| `include_fields` | array | Journal fields to include in events |\n| `exclude_fields` | array | Journal fields to exclude from events |\n| `include_kernel_messages` | bool | Include kernel ring buffer messages |\n\n## Error handling considerations\n\n- **Missing journal directories**: The agent tolerates a missing `/var/log/journal` at startup and retries. If the journal uses volatile storage (`/run/log/journal`), entries do not persist across reboots.\n- **Permission issues**: Reading the system journal requires membership in the `systemd-journal` group or root access. The agent process must have appropriate permissions.\n- **Cursor recovery after agent restart**: If the journal has been rotated between agent stops, the cursor may point to a deleted entry. Configure `cursor_seek_fallback` to control behavior in this case.\n\n## Review checklist\n\n### Filtering\n\n- [ ] `include_matches` present to avoid collecting the entire journal -- **HIGH**\n- [ ] Match fields use valid journal field names -- **HIGH**\n- [ ] Match values use template variables when the target unit varies by deployment -- **MEDIUM**\n- [ ] Not overly broad (no unfiltered collection without justification) -- **HIGH**\n\n### Seek and cursor\n\n- [ ] `seek` position is configurable via template variable -- **MEDIUM**\n- [ ] `cursor_seek_fallback` set if journal rotation is expected -- **LOW**\n- [ ] Default seek behavior is appropriate for the data stream's use case -- **MEDIUM**\n\n### Paths\n\n- [ ] Paths use template variables when present -- **HIGH**\n- [ ] Omitted when the default system journal is the intended source -- **LOW**\n- [ ] Journal path accessible by the agent process -- **MEDIUM**\n\n### Advanced settings\n\n- [ ] Field filtering reduces event size when journal entries contain many unused fields -- **LOW**\n- [ ] Kernel messages included or excluded as appropriate for the data stream -- **LOW**\n- [ ] `since` filter considered for data streams that should not replay history -- **LOW**\n\n### Common patterns\n\n- [ ] `preserve_original_event` is conditional (`{{#if}}`) -- **MEDIUM**\n- [ ] `forwarded` tag and `publisher_pipeline.disable_host` are coupled -- **MEDIUM**\n- [ ] Custom processors passthrough at top level -- **LOW**\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":9850,"content_sha256":"3d2626fae6b86d0a7d01e302c373a5b96b18eb12bd5b03956ec82e3c1ecfebfc"},{"filename":"references/tcp-udp-guide.md","content":"# TCP and UDP input guide\n\nComplete reference for building and reviewing `tcp.yml.hbs` and `udp.yml.hbs` templates in Elastic integrations. TCP and UDP are listener-based inputs that receive data pushed to the agent over a network socket. They share most structural conventions but differ in transport-layer capabilities.\n\n> **Documentation**: [TCP Input Reference](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-tcp.html) | [UDP Input Reference](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-udp.html)\n\n## File location\n\n```\npackages/\u003cpackage>/data_stream/\u003cdata_stream>/agent/stream/tcp.yml.hbs\npackages/\u003cpackage>/data_stream/\u003cdata_stream>/agent/stream/udp.yml.hbs\n```\n\n## Required structure\n\n### TCP template\n\n```yaml\nhost: {{listen_address}}:{{listen_port}}\n\n{{#if max_message_size}}\nmax_message_size: {{max_message_size}}\n{{/if}}\n{{#if framing}}\nframing: {{framing}}\n{{/if}}\n{{#if line_delimiter}}\nline_delimiter: {{line_delimiter}}\n{{/if}}\n\n{{#if max_connections}}\nmax_connections: {{max_connections}}\n{{/if}}\n{{#if timeout}}\ntimeout: {{timeout}}\n{{/if}}\n\n{{#if ssl}}\nssl: {{ssl}}\n{{/if}}\n\ntags:\n{{#if preserve_original_event}}\n - preserve_original_event\n{{/if}}\n{{#each tags as |tag|}}\n - {{tag}}\n{{/each}}\n{{#contains \"forwarded\" tags}}\npublisher_pipeline.disable_host: true\n{{/contains}}\n\n{{#if processors}}\nprocessors:\n{{processors}}\n{{/if}}\n```\n\n### UDP template\n\n```yaml\nhost: {{listen_address}}:{{listen_port}}\n\n{{#if max_message_size}}\nmax_message_size: {{max_message_size}}\n{{/if}}\n{{#if read_buffer_size}}\nread_buffer: {{read_buffer_size}}\n{{/if}}\n{{#if timeout}}\ntimeout: {{timeout}}\n{{/if}}\n\ntags:\n{{#if preserve_original_event}}\n - preserve_original_event\n{{/if}}\n{{#each tags as |tag|}}\n - {{tag}}\n{{/each}}\n{{#contains \"forwarded\" tags}}\npublisher_pipeline.disable_host: true\n{{/contains}}\n\n{{#if processors}}\nprocessors:\n{{processors}}\n{{/if}}\n```\n\n## TCP vs UDP selection\n\n| Aspect | TCP | UDP |\n|---|---|---|\n| Reliability | Guaranteed delivery | No delivery guarantee |\n| Ordering | Ordered | No ordering guarantee |\n| Connection model | Connection-oriented | Connectionless |\n| SSL/TLS | Supported | Not supported |\n| Framing | Delimiter or RFC 6587 | Datagram boundaries |\n| Typical use | Reliable logging, sensitive data | High-volume syslog |\n\nSecurity-sensitive logs (firewall, authentication, PII) should use TCP with TLS. UDP is appropriate for high-volume, loss-tolerant syslog streams.\n\n## Validation rules\n\n### 1. Host must use variables\n\nThe `host` field must reference Handlebars variables, never hardcoded addresses or ports.\n\n```yaml\n# correct\nhost: {{listen_address}}:{{listen_port}}\n\n# wrong -- hardcoded\nhost: 0.0.0.0:514\nhost: localhost:9000\n```\n\n### 2. Framing must be valid (TCP only)\n\nTCP supports two framing modes. Syslog streams require `rfc6587`.\n\n```yaml\n# valid values\nframing: delimiter # line-based, the default\nframing: rfc6587 # syslog octet counting\n\n# invalid\nframing: json\nframing: custom\n```\n\nA TCP syslog stream without `framing: rfc6587` is a defect. Delimiter framing will silently corrupt multi-line syslog messages.\n\n### 3. SSL/TLS for sensitive data (TCP only)\n\nTCP templates that carry security-sensitive logs must include the SSL block. UDP does not support SSL.\n\n```yaml\n{{#if ssl}}\nssl: {{ssl}}\n{{/if}}\n```\n\nThe corresponding manifest should expose the full SSL configuration: `certificate_authorities`, `certificate`, `key`, and `verification_mode`.\n\n### 4. Buffer sizing for high-volume UDP\n\nUDP sockets drop packets when the kernel receive buffer overflows. High-volume syslog sources must expose a configurable read buffer.\n\n```yaml\n{{#if read_buffer_size}}\nread_buffer: {{read_buffer_size}}\n{{/if}}\n```\n\nA reasonable default for high-volume syslog is `100MiB`. Missing buffer configuration on a high-volume UDP stream is a review finding.\n\n### 5. Message size limits\n\nBoth TCP and UDP should expose `max_message_size` when messages can exceed the default.\n\n```yaml\n{{#if max_message_size}}\nmax_message_size: {{max_message_size}}\n{{/if}}\n```\n\nTypical values: `8KiB` for standard syslog, `64KiB` for extended syslog or structured log formats.\n\n### 6. Preserve original event before processing\n\nWhen the template includes processors that modify the `message` field (especially syslog parsing), the `copy_fields` processor that preserves the original event must appear before any parsing processor.\n\n```yaml\nprocessors:\n{{#if preserve_original_event}}\n- copy_fields:\n fields:\n - from: message\n to: event.original\n{{/if}}\n{{#if syslog}}\n- syslog:\n {{syslog_options}}\n{{/if}}\n{{processors}}\n```\n\nReversing the order means `event.original` captures the already-parsed message instead of the raw input.\n\n## Syslog-specific patterns\n\n### Syslog over TCP\n\n```yaml\nhost: {{listen_address}}:{{listen_port}}\nframing: rfc6587\nmax_message_size: 64KiB\n\ntags:\n{{#if preserve_original_event}}\n - preserve_original_event\n{{/if}}\n{{#each tags as |tag|}}\n - {{tag}}\n{{/each}}\n{{#contains \"forwarded\" tags}}\npublisher_pipeline.disable_host: true\n{{/contains}}\n\nprocessors:\n{{#if preserve_original_event}}\n- copy_fields:\n fields:\n - from: message\n to: event.original\n{{/if}}\n- syslog:\n field: message\n format: auto\n{{#if processors}}\n{{processors}}\n{{/if}}\n```\n\n### Syslog over UDP\n\n```yaml\nhost: {{listen_address}}:{{listen_port}}\nmax_message_size: 64KiB\nread_buffer: 100MiB\n\ntags:\n{{#if preserve_original_event}}\n - preserve_original_event\n{{/if}}\n{{#each tags as |tag|}}\n - {{tag}}\n{{/each}}\n{{#contains \"forwarded\" tags}}\npublisher_pipeline.disable_host: true\n{{/contains}}\n\nprocessors:\n{{#if preserve_original_event}}\n- copy_fields:\n fields:\n - from: message\n to: event.original\n{{/if}}\n- syslog:\n field: message\n format: auto\n{{#if processors}}\n{{processors}}\n{{/if}}\n```\n\n### Required syslog stream variables\n\nSyslog data streams must expose the following variables in the manifest:\n\n- **`ssl`** (TCP only): Full SSL/TLS configuration object. Type `yaml`. Many syslog sources (firewalls, network devices) transmit over TLS and the integration must support it.\n- **`processors`**: Custom processors passthrough. Always present per the common patterns.\n- **`syslog_options`** or equivalent: Controls the syslog processor format. At minimum, expose `format: auto` as the default.\n\n### The `tz_offset` variable convention\n\nSyslog messages frequently arrive without timezone information. Integrations should expose a `tz_offset` variable that lets users declare the source device's UTC offset so the ingest pipeline can stamp events with the correct time.\n\nConvention in the manifest:\n\n```yaml\n- name: tz_offset\n type: text\n title: Timezone Offset\n description: >-\n IANA time zone or UTC offset for timestamps without timezone information\n (e.g., America/New_York, +02:00, UTC). Defaults to UTC.\n multi: false\n required: false\n show_user: true\n default: UTC\n```\n\nThe ingest pipeline or a processor uses this value to adjust parsed timestamps. Without it, events from devices in non-UTC zones will have incorrect timestamps.\n\n## Advanced patterns\n\n### Secure TCP with TLS\n\n```yaml\nhost: {{listen_address}}:{{listen_port}}\n\n{{#if ssl}}\nssl: {{ssl}}\n{{/if}}\n```\n\nThe manifest must expose the SSL block with sub-fields or accept a YAML object. The SSL configuration supports:\n- `certificate_authorities`: CA certificates for client verification\n- `certificate`: Server certificate\n- `key`: Server private key\n- `verification_mode`: `full`, `certificate`, `strict`, or `none`\n\n### High-volume TCP listener\n\n```yaml\nhost: {{listen_address}}:{{listen_port}}\nmax_connections: {{max_connections}}\nmax_message_size: {{max_message_size}}\ntimeout: {{timeout}}\n```\n\n### Keep-alive (TCP only)\n\n```yaml\n{{#if keep_alive}}\nkeep_alive: {{keep_alive}}\n{{/if}}\n```\n\nPersistent connections benefit from keep-alive to detect dead clients without waiting for a timeout.\n\n### Client IP restrictions (TCP only)\n\n```yaml\n{{#if include_source_ips}}\ninclude_source_ips:\n{{#each include_source_ips as |ip|}}\n - {{ip}}\n{{/each}}\n{{/if}}\n```\n\n### Queue size (UDP only)\n\n```yaml\n{{#if queue_size}}\nqueue_size: {{queue_size}}\n{{/if}}\n```\n\n### Encoding\n\n```yaml\n{{#if encoding}}\nencoding: {{encoding}}\n{{/if}}\n```\n\nSpecify when the source sends non-UTF-8 encoded data.\n\n## Parameters reference\n\n### TCP parameters\n\n| Parameter | Type | Description |\n|---|---|---|\n| `host` | string | Listen address and port (`address:port`) |\n| `max_message_size` | size | Maximum message size (e.g., `64KiB`) |\n| `framing` | string | `delimiter` or `rfc6587` |\n| `line_delimiter` | string | Delimiter character(s) for delimiter framing |\n| `max_connections` | int | Maximum concurrent connections |\n| `timeout` | duration | Connection timeout |\n| `ssl` | object | SSL/TLS configuration |\n| `keep_alive` | duration | TCP keep-alive interval |\n| `keep_null` | bool | Keep null values in parsed JSON |\n| `encoding` | string | Character encoding |\n| `include_source_ips` | array | Allowed client IP addresses |\n\n### UDP parameters\n\n| Parameter | Type | Description |\n|---|---|---|\n| `host` | string | Listen address and port (`address:port`) |\n| `max_message_size` | size | Maximum message size (e.g., `64KiB`) |\n| `read_buffer` | size | Socket receive buffer size (e.g., `100MiB`) |\n| `timeout` | duration | Read timeout |\n| `queue_size` | int | Internal message queue size |\n| `keep_null` | bool | Keep null values |\n| `encoding` | string | Character encoding |\n\n## Review checklist\n\n### Listener configuration\n- [ ] `host` uses `{{listen_address}}:{{listen_port}}` variables\n- [ ] `max_message_size` exposed when messages may exceed defaults\n- [ ] `timeout` configured where appropriate\n\n### TCP-specific\n- [ ] `framing` set to `rfc6587` for syslog streams\n- [ ] `ssl` block present for security-sensitive data\n- [ ] `max_connections` exposed for high-connection-count sources\n- [ ] Keep-alive configured for persistent connections\n- [ ] Client IP restrictions considered for security-sensitive listeners\n\n### UDP-specific\n- [ ] `read_buffer` configured for high-volume sources\n- [ ] UDP is appropriate for the data sensitivity level (not used for secrets or PII)\n- [ ] Queue size appropriate for message volume\n\n### Syslog streams\n- [ ] Syslog processor present with `format: auto`\n- [ ] `preserve_original_event` copy occurs before syslog parsing\n- [ ] `tz_offset` variable exposed in the manifest for timezone handling\n- [ ] `ssl` variable exposed in the manifest (TCP syslog)\n- [ ] `processors` passthrough present\n\n### Processing\n- [ ] `preserve_original_event` copies message before any parsing\n- [ ] Tags block follows common patterns (conditional `preserve_original_event`, `each` tags, `forwarded`/`disable_host` coupling)\n- [ ] Processors passthrough at top level\n- [ ] Encoding specified for non-UTF-8 data\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":10859,"content_sha256":"1cdd4a87c0b6b4b198fcfd8425d3d6eee0f0d956893a5ff256324cb59db88e58"},{"filename":"references/websocket-guide.md","content":"# WebSocket input guide\n\nComplete reference for building and reviewing `websocket.yml.hbs` templates in Elastic integrations.\n\nDocumentation: [WebSocket Input Reference](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-websocket.html)\n\n## File location\n\n```\npackages/\u003cpackage>/data_stream/\u003cdata_stream>/agent/stream/websocket.yml.hbs\n```\n\n## Required structure\n\nThe WebSocket input maintains a persistent connection to a WebSocket server and processes incoming messages as events. Every template must configure the connection URL and typically includes authentication, SSL, and message handling.\n\n```yaml\nurl: {{url}}\n\n{{#if auth.basic}}\nauth.basic:\n user: {{username}}\n password: {{password}}\n{{/if}}\n{{#if headers}}\nheaders:\n{{headers}}\n{{/if}}\n\n{{#if ssl}}\nssl: {{ssl}}\n{{/if}}\n\n{{#if proxy_url}}\nproxy_url: {{proxy_url}}\n{{/if}}\n\n{{#if program}}\nprogram: |\n {{program}}\n{{/if}}\n```\n\n## Validation rules\n\n### 1. URL must use a variable\n\nThe WebSocket URL must reference a Handlebars variable. Hardcoded URLs prevent users from configuring the endpoint.\n\n```yaml\n# Correct\nurl: {{url}}\n\n# Never acceptable\nurl: wss://api.example.com/stream\n\n# Wrong protocol -- must be ws:// or wss://, not https://\nurl: https://api.example.com/stream\n```\n\n### 2. Authentication must use variables\n\nAll credential values must reference Handlebars variables. Hardcoded credentials are a critical security issue.\n\n```yaml\n# Correct -- basic auth\n{{#if auth.basic}}\nauth.basic:\n user: {{username}}\n password: {{password}}\n{{/if}}\n\n# Correct -- token in header\n{{#if api_token}}\nheaders:\n Authorization: Bearer {{api_token}}\n{{/if}}\n\n# Never acceptable\nauth.basic:\n password: 'my-secret-password'\n```\n\n### 3. SSL configuration for WSS URLs\n\nWebSocket connections using `wss://` require SSL/TLS. The SSL configuration block must be available.\n\n```yaml\n{{#if ssl}}\nssl: {{ssl}}\n{{/if}}\n```\n\n### 4. Credential redaction for state\n\nThe WebSocket input supports CEL programs and state management. Credentials stored in the `state` block are safe if they are listed in `redact.fields`. The redaction mechanism masks these values in logs and traces.\n\n```yaml\nstate:\n api_key: '{{api_key}}'\n\nredact:\n fields:\n - api_key\n```\n\nCredentials in state **with** a corresponding `redact.fields` entry are not a security issue. Credentials in state **without** a corresponding `redact.fields` entry, or hardcoded credentials not using template variables, are security issues.\n\n### 5. CEL program presence\n\nMany WebSocket integrations require a CEL program to parse incoming messages, filter heartbeats, handle subscription flows, or transform data before it becomes events. When a `program` field is present, the CEL code must be reviewed for correctness.\n\n```yaml\n{{#if program}}\nprogram: |\n {{program}}\n{{/if}}\n```\n\nInline CEL programs (not using a variable) are also valid:\n\n```yaml\nprogram: |\n state.response.decode_json().as(event, {\n \"events\": [{\"message\": event.encode_json()}],\n })\n```\n\nNote: older integrations may use `bytes(state.response).decode_json()` instead of `state.response.decode_json()`. The `bytes()` conversion is not required in newer beats versions (same version dependency as the CEL input). Omit it in new integrations.\n\n## URL program\n\nThe `url_program` config runs a CEL program **once** before the WebSocket connection is established. It uses the `state` object (including cursor values) to dynamically construct the connection URL. Available from beats 8.15+.\n\n```yaml\nurl: ws://api.example.com/v1/stream\nstate:\n initial_start_time: \"2022-01-01T00:00:00Z\"\nurl_program: |\n state.url + \"?since=\" + state.?cursor.since.orValue(state.initial_start_time)\nprogram: |\n state.response.decode_json().as(msg, {\n \"events\": [{\"message\": msg.encode_json()}],\n \"cursor\": {\"since\": msg.timestamp}\n })\n```\n\nUse `url_program` when:\n- The WebSocket URL needs a cursor or timestamp parameter to resume from the last position\n- The URL varies based on state values set during previous evaluations\n- The endpoint requires query parameters derived from runtime context\n\nThe program must evaluate to a valid URL string. It has access to `state.url` (the configured base URL), `state.cursor` (persisted cursor values), and any custom state fields.\n\n## Authentication patterns\n\n### Basic authentication\n\nStandard HTTP basic auth sent during the WebSocket handshake.\n\n```yaml\n{{#if auth.basic}}\nauth.basic:\n user: {{username}}\n password: {{password}}\n{{/if}}\n```\n\n### Token-based authentication via headers\n\nAPI tokens or bearer tokens sent as HTTP headers during the handshake.\n\n```yaml\n{{#if api_token}}\nheaders:\n Authorization: Bearer {{api_token}}\n{{/if}}\n```\n\nHeaders can also carry additional metadata:\n\n```yaml\n{{#if api_token}}\nheaders:\n Authorization: Bearer {{api_token}}\n X-API-Version: {{api_version}}\n{{/if}}\n```\n\n### Authentication via CEL state\n\nWhen authentication tokens need to be used within the CEL program (e.g., for subscription messages sent after connection), they are placed in the `state` block with redaction.\n\n```yaml\nstate:\n api_key: '{{api_key}}'\n\nredact:\n fields:\n - api_key\n\nprogram: |\n // CEL program that uses state.api_key for subscription\n```\n\n## Reconnection and keepalive\n\n### Retry settings\n\nWebSocket connections may drop due to network issues or server restarts. Retry parameters control automatic reconnection behavior.\n\n| Parameter | Type | Description |\n|---|---|---|\n| `retry.wait_min` | duration | Minimum wait before reconnecting |\n| `retry.wait_max` | duration | Maximum wait before reconnecting |\n| `retry.max_attempts` | int | Maximum number of reconnection attempts |\n\nThese parameters should be appropriate for the target API. Aggressive retry settings may trigger rate limiting; overly conservative settings increase data loss during outages.\n\n### Ping interval and wait timeout\n\nLong-lived WebSocket connections need keepalive mechanisms to detect dead connections.\n\n```yaml\n{{#if ping_interval}}\nping_interval: {{ping_interval}}\n{{/if}}\n{{#if wait_timeout}}\nwait_timeout: {{wait_timeout}}\n{{/if}}\n```\n\n`ping_interval` controls how often the client sends WebSocket ping frames. `wait_timeout` controls how long the client waits for a message before considering the connection dead.\n\n## CEL program patterns\n\nThe WebSocket input uses CEL programs for message processing. The program receives each message in `state.response` and must return a map containing an `events` array.\n\n### Basic message handling\n\nDecode the JSON message and emit it as a single event:\n\n```yaml\nprogram: |\n state.response.decode_json().as(msg, {\n \"events\": [{\"message\": msg.encode_json()}],\n })\n```\n\n### Conditional message filtering\n\nSkip non-data messages (heartbeats, acknowledgements) and only emit data events:\n\n```yaml\nprogram: |\n state.response.decode_json().as(msg,\n has(msg.type) && msg.type == \"data\" ?\n {\n \"events\": [{\"message\": msg.payload.encode_json()}],\n }\n :\n {\n \"events\": [],\n }\n )\n```\n\n### Heartbeat filtering\n\nMany streaming APIs send periodic heartbeat messages that should be discarded:\n\n```yaml\nprogram: |\n state.response.decode_json().as(msg,\n msg.type == \"heartbeat\" ?\n {\"events\": []}\n :\n {\"events\": [{\"message\": msg.encode_json()}]}\n )\n```\n\n### Batch message handling\n\nWhen a single WebSocket message contains an array of events:\n\n```yaml\nprogram: |\n state.response.decode_json().as(msg,\n msg.type == \"batch\" ?\n {\"events\": msg.data.map(e, {\"message\": e.encode_json()})}\n :\n {\"events\": [{\"message\": msg.encode_json()}]}\n )\n```\n\n### Subscription messages\n\nSome WebSocket APIs require sending a subscription message after the connection is established. This is typically handled within the CEL program or through initial state configuration.\n\n## CEL program detection\n\nWhen a WebSocket template contains a `program` field with CEL code, the CEL code itself may need review according to CEL-specific validation rules. Look for:\n\n- `state.response` access patterns for message parsing\n- `encode_json()` / `decode_json()` for serialization\n- Conditional logic for message type filtering\n- State mutations that carry values between messages\n- Subscription or authentication flows embedded in the program\n\nIf the CEL program is complex (pagination, state management, multi-step flows), it should be reviewed against the CEL program skill's rules in addition to the WebSocket input rules.\n\n## Common configuration patterns\n\n### Basic WebSocket connection\n\n```yaml\nurl: {{url}}\n\n{{#if api_token}}\nheaders:\n Authorization: Bearer {{api_token}}\n{{/if}}\n\n{{#if ssl}}\nssl: {{ssl}}\n{{/if}}\n```\n\n### With basic authentication\n\n```yaml\nurl: {{url}}\n\nauth.basic:\n user: {{username}}\n password: {{password}}\n\n{{#if ssl}}\nssl: {{ssl}}\n{{/if}}\n```\n\n### With CEL message processing\n\n```yaml\nurl: {{url}}\n\n{{#if api_token}}\nheaders:\n Authorization: Bearer {{api_token}}\n{{/if}}\n\nprogram: |\n state.response.decode_json().as(msg,\n has(msg.type) && msg.type == \"data\" ?\n {\n \"events\": [{\"message\": msg.payload.encode_json()}],\n }\n :\n {\n \"events\": [],\n }\n )\n\n{{#if ssl}}\nssl: {{ssl}}\n{{/if}}\n```\n\n### Streaming API with proxy\n\n```yaml\nurl: {{url}}\n\n{{#if api_token}}\nheaders:\n Authorization: Bearer {{api_token}}\n X-API-Version: {{api_version}}\n{{/if}}\n\n{{#if proxy_url}}\nproxy_url: {{proxy_url}}\n{{/if}}\n\n{{#if ssl}}\nssl: {{ssl}}\n{{/if}}\n```\n\n## Parameters reference\n\n| Parameter | Type | Description |\n|---|---|---|\n| `url` | string | WebSocket URL (`ws://` or `wss://`) |\n| `program` | string | CEL program for message handling |\n| `state` | object | Initial state passed to the CEL program |\n| `redact.fields` | array | State fields to mask in logs and traces |\n| `auth.basic.user` | string | Basic auth username |\n| `auth.basic.password` | string | Basic auth password |\n| `headers` | map | Additional HTTP headers sent during handshake |\n| `ssl` | object | SSL/TLS configuration |\n| `proxy_url` | string | HTTP proxy URL |\n| `ping_interval` | duration | WebSocket ping frame interval |\n| `wait_timeout` | duration | Maximum wait time for a message |\n| `retry.wait_min` | duration | Minimum reconnection wait |\n| `retry.wait_max` | duration | Maximum reconnection wait |\n| `retry.max_attempts` | int | Maximum reconnection attempts |\n\n## Review checklist\n\n### Connection\n\n- [ ] URL uses a Handlebars variable -- **HIGH**\n- [ ] URL protocol is correct (`ws://` or `wss://`, not `http://` or `https://`) -- **HIGH**\n- [ ] SSL configuration available for `wss://` connections -- **HIGH**\n\n### Authentication\n\n- [ ] No hardcoded credentials -- **CRITICAL**\n- [ ] Auth method appropriate for the target API (basic, header token, or CEL state) -- **HIGH**\n- [ ] Token-based auth uses headers with Handlebars variables -- **MEDIUM**\n- [ ] Credentials in `state` have corresponding `redact.fields` entries -- **HIGH**\n\n### Message handling\n\n- [ ] CEL program present when message parsing/filtering is needed -- **HIGH**\n- [ ] Heartbeat/ping messages filtered in the CEL program -- **MEDIUM**\n- [ ] Batch messages unpacked into individual events -- **MEDIUM**\n- [ ] Subscription messages handled if the API requires them -- **MEDIUM**\n\n### Reconnection\n\n- [ ] Retry settings appropriate for the target API -- **MEDIUM**\n- [ ] `ping_interval` configured for long-lived connections -- **LOW**\n- [ ] `wait_timeout` set to detect dead connections -- **LOW**\n- [ ] Proxy configuration available -- **LOW**\n\n### CEL program (when present)\n\n- [ ] `state.response` correctly accessed and decoded -- **HIGH**\n- [ ] Events array properly constructed in all code paths -- **HIGH**\n- [ ] Non-data messages (heartbeats, acks) produce empty events arrays -- **MEDIUM**\n- [ ] Complex CEL programs reviewed against CEL skill rules -- **MEDIUM**\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":11810,"content_sha256":"8c13b6e6d2c5ee34273cfc68c985424033b5d64cf8905ffddf41d817474e1b2c"},{"filename":"references/winlog-guide.md","content":"# Windows Event Log (winlog) input guide\n\nComplete reference for building and reviewing `winlog.yml.hbs` templates in Elastic integrations.\n\nThe winlog input reads from Windows Event Log channels. It supports filtering by event ID, provider, level, and advanced XML queries.\n\n## File location\n\n```\npackages/\u003cpackage>/data_stream/\u003cdata_stream>/agent/stream/winlog.yml.hbs\n```\n\n## Required structure\n\n```yaml\nname: {{channel_name}}\n\n{{#if event_id}}\nevent_id: {{event_id}}\n{{/if}}\n{{#if ignore_older}}\nignore_older: {{ignore_older}}\n{{/if}}\n{{#if level}}\nlevel: {{level}}\n{{/if}}\n{{#if provider}}\nprovider:\n{{#each provider as |p|}}\n - {{p}}\n{{/each}}\n{{/if}}\n\ntags:\n{{#if preserve_original_event}}\n - preserve_original_event\n{{/if}}\n{{#each tags as |tag|}}\n - {{tag}}\n{{/each}}\n{{#contains \"forwarded\" tags}}\npublisher_pipeline.disable_host: true\n{{/contains}}\n{{#if processors}}\nprocessors:\n{{processors}}\n{{/if}}\n```\n\n## Validation rules\n\n### 1. Channel name must be specified\n\nEvery winlog template must have a `name` field identifying the Windows Event Log channel. This is the only strictly required field.\n\n```yaml\n# correct -- configurable via variable\nname: {{channel_name}}\n\n# correct -- fixed channel for a well-known log source\nname: Security\nname: Microsoft-Windows-Sysmon/Operational\n\n# wrong -- missing\n# (no name field)\n```\n\n### 2. Event ID filtering should be configurable\n\nFor channels that produce large volumes of events (especially Security), event ID filtering prevents ingesting irrelevant data. The filter should be a template variable so users can customize it.\n\n```yaml\n# correct -- configurable\n{{#if event_id}}\nevent_id: {{event_id}}\n{{/if}}\n\n# acceptable -- fixed IDs for a specific use case\nevent_id: 4624, 4625, 4634\n\n# concern -- Security channel without any event_id filter collects all security events\nname: Security\n# (no event_id)\n```\n\nThe `event_id` value is a comma-separated string of event IDs. Ranges are not supported at the input level.\n\n### 3. Ignore older for large logs\n\nHigh-volume channels (Security, Sysmon) can accumulate millions of events. Without `ignore_older`, the agent processes the entire backlog on first start.\n\n```yaml\n# correct -- configurable\n{{#if ignore_older}}\nignore_older: {{ignore_older}}\n{{/if}}\n```\n\n### 4. Provider filtering narrows collection scope\n\nWhen a channel contains events from multiple providers but only a subset is relevant, use the `provider` filter.\n\n```yaml\n# correct -- filter to specific providers\n{{#if provider}}\nprovider:\n{{#each provider as |p|}}\n - {{p}}\n{{/each}}\n{{/if}}\n```\n\n### 5. Channel names must be exact\n\nWindows Event Log channel names are case-sensitive and must match exactly. Common errors include missing the `/Operational` suffix on Microsoft channels or using forward slashes vs backslashes.\n\n```yaml\n# correct\nname: Microsoft-Windows-Sysmon/Operational\nname: Microsoft-Windows-PowerShell/Operational\n\n# wrong -- missing suffix\nname: Microsoft-Windows-Sysmon\n```\n\n## Event log channels\n\n### Standard channels\n\n| Channel | Description | Typical volume |\n|---|---|---|\n| `Security` | Security and audit events (logon, privilege use, policy changes) | Very high |\n| `System` | System-level events (services, drivers, hardware) | Medium |\n| `Application` | Application-level events | Medium |\n\n### Common Microsoft channels\n\n| Channel | Description |\n|---|---|\n| `Microsoft-Windows-Sysmon/Operational` | Sysmon process, network, file events |\n| `Microsoft-Windows-PowerShell/Operational` | PowerShell script execution |\n| `Microsoft-Windows-Windows Defender/Operational` | Defender detections and status |\n| `Microsoft-Windows-WMI-Activity/Operational` | WMI activity events |\n| `Microsoft-Windows-TaskScheduler/Operational` | Scheduled task events |\n| `Microsoft-Windows-Windows Firewall With Advanced Security/Firewall` | Firewall rule changes |\n| `Microsoft-Windows-TerminalServices-LocalSessionManager/Operational` | RDP session events |\n\n### Forwarded events\n\nThe `ForwardedEvents` channel contains events forwarded from other machines via Windows Event Forwarding (WEF):\n\n```yaml\nname: ForwardedEvents\n\n{{#if forwarded}}\nforwarded: {{forwarded}}\n{{/if}}\n```\n\nWhen reading forwarded events, set `forwarded: true` so the input correctly handles the forwarded event format, which wraps the original event XML in an additional envelope.\n\n## Providers\n\nProviders identify the source component that generated an event within a channel. Filtering by provider is useful when a channel aggregates events from multiple sources.\n\n```yaml\nprovider:\n - Microsoft-Windows-Security-Auditing\n - Microsoft-Windows-Eventlog\n```\n\nCommon provider names by channel:\n\n| Channel | Notable providers |\n|---|---|\n| Security | `Microsoft-Windows-Security-Auditing` |\n| System | `Microsoft-Windows-Kernel-General`, `Service Control Manager`, `Microsoft-Windows-WindowsUpdateClient` |\n| Application | Varies by installed applications |\n| Sysmon/Operational | `Microsoft-Windows-Sysmon` |\n| PowerShell/Operational | `Microsoft-Windows-PowerShell` |\n\n## Event IDs\n\n### Security channel key event IDs\n\n| Event ID | Description |\n|---|---|\n| 4624 | Successful logon |\n| 4625 | Failed logon |\n| 4634 | Logoff |\n| 4648 | Logon with explicit credentials |\n| 4672 | Special privileges assigned |\n| 4688 | Process creation |\n| 4689 | Process termination |\n| 4697 | Service installation |\n| 4720 | User account created |\n| 4722 | User account enabled |\n| 4732 | Member added to security group |\n| 4768 | Kerberos TGT requested |\n| 4769 | Kerberos service ticket requested |\n| 4776 | NTLM authentication |\n\n### Sysmon key event IDs\n\n| Event ID | Description |\n|---|---|\n| 1 | Process creation |\n| 2 | File creation time change |\n| 3 | Network connection |\n| 5 | Process terminated |\n| 7 | Image loaded |\n| 8 | CreateRemoteThread |\n| 10 | Process access |\n| 11 | File create |\n| 12, 13, 14 | Registry events |\n| 15 | FileCreateStreamHash |\n| 22 | DNS query |\n\n## XML rendering\n\nWindows Event Log events have an underlying XML structure. The `include_xml` option controls whether the raw XML is included in the output event.\n\n```yaml\n{{#if include_xml}}\ninclude_xml: {{include_xml}}\n{{/if}}\n```\n\nWhen `include_xml: true`, the full event XML is stored in `winlog.xml`. This is useful for:\n- Integrations that need fields not extracted by default\n- Forensic or compliance use cases requiring the original event format\n- Debugging event parsing issues\n\nThe trade-off is increased event size. Only enable when the downstream pipeline or use case requires the raw XML.\n\n## Forwarded events\n\nWindows Event Forwarding (WEF) collects events from remote machines into a central `ForwardedEvents` channel. Templates reading forwarded events need special handling:\n\n```yaml\nname: ForwardedEvents\nforwarded: true\n```\n\nKey considerations:\n- The `forwarded` flag tells the input to unwrap the forwarding envelope and extract the original event.\n- The `Computer` field in forwarded events reflects the originating machine, not the collector.\n- Event rendering may fail if the collecting machine lacks the provider manifest for the original event source. See the language handling section.\n\n## Language handling\n\nWindows renders event messages using provider-specific message DLLs. The language of the rendered message depends on the system locale unless overridden.\n\n```yaml\n{{#if language}}\nlanguage: {{language}}\n{{/if}}\n```\n\nLanguage codes use the IETF BCP 47 format (e.g., `en-US`, `de-DE`, `ja-JP`). The default is the system locale.\n\nWhen to configure language explicitly:\n- **Forwarded events**: The collecting machine may have a different locale than the originating machine. Setting `language: 0` uses the system default, while a specific locale ensures consistent rendering.\n- **Multi-language environments**: When agents run on machines with varying locales but events should be in a consistent language for analysis.\n- **Missing message DLLs**: On forwarded event collectors that lack the provider message tables, rendered messages may be empty or contain only parameter substitution placeholders.\n\n## API type\n\nThe winlog input supports two API implementations:\n\n```yaml\n{{#if api}}\napi: {{api}}\n{{/if}}\n```\n\n| API | Description |\n|---|---|\n| `wineventlog` | Default Windows Event Log API. Stable, broadly compatible. |\n| `wineventlog-experimental` | Experimental implementation with improved performance for high-volume channels. |\n\nUse the default unless the integration documentation specifically requires the experimental API.\n\n## Advanced configuration\n\n### Batch read size\n\nControls how many events are read from the channel in a single API call. Higher values improve throughput but increase memory usage.\n\n```yaml\n{{#if batch_read_size}}\nbatch_read_size: {{batch_read_size}}\n{{/if}}\n```\n\nDefault is 100. For high-volume channels (Security on domain controllers), values of 500-1000 may improve performance.\n\n### Level filtering\n\nFilter events by severity level. Only events at or above the specified level are collected.\n\n```yaml\n{{#if level}}\nlevel: {{level}}\n{{/if}}\n```\n\n| Level | Numeric value | Description |\n|---|---|---|\n| `critical` | 1 | Critical errors |\n| `error` | 2 | Errors |\n| `warning` | 3 | Warnings |\n| `information` | 4 | Informational |\n| `verbose` | 5 | Verbose/debug |\n\nSetting `level: warning` collects warning, error, and critical events.\n\n### Multiple channels\n\nEach winlog template handles a single channel. To collect from multiple channels, create separate stream files:\n\n```\nagent/stream/winlog-security.yml.hbs -> name: Security\nagent/stream/winlog-system.yml.hbs -> name: System\nagent/stream/winlog-application.yml.hbs -> name: Application\n```\n\nEach file follows the same template structure independently.\n\n## Common configuration patterns\n\n### Security events\n\n```yaml\nname: Security\n\n{{#if event_id}}\nevent_id: {{event_id}}\n{{/if}}\n\n{{#if ignore_older}}\nignore_older: {{ignore_older}}\n{{/if}}\n\n{{#if processors}}\nprocessors:\n{{processors}}\n{{/if}}\n```\n\n### Sysmon events\n\n```yaml\nname: Microsoft-Windows-Sysmon/Operational\n\n{{#if event_id}}\nevent_id: {{event_id}}\n{{/if}}\n```\n\n### PowerShell events\n\n```yaml\nname: Microsoft-Windows-PowerShell/Operational\n\n{{#if event_id}}\nevent_id: {{event_id}}\n{{/if}}\n```\n\n### Forwarded events with language override\n\n```yaml\nname: ForwardedEvents\nforwarded: true\nlanguage: 0\n\n{{#if event_id}}\nevent_id: {{event_id}}\n{{/if}}\n\n{{#if ignore_older}}\nignore_older: {{ignore_older}}\n{{/if}}\n```\n\n## Parameters reference\n\n| Parameter | Type | Description |\n|---|---|---|\n| `name` | string | Event log channel name (required) |\n| `event_id` | string | Comma-separated event IDs to collect |\n| `ignore_older` | duration | Ignore events older than this duration |\n| `level` | string | Minimum event level (`critical`, `error`, `warning`, `information`, `verbose`) |\n| `provider` | array | Event provider names to filter |\n| `include_xml` | bool | Include raw event XML in output |\n| `forwarded` | bool | Enable forwarded event handling |\n| `language` | string/int | Language for event message rendering (BCP 47 or `0` for system default) |\n| `api` | string | API type: `wineventlog` or `wineventlog-experimental` |\n| `batch_read_size` | int | Events per batch read (default 100) |\n\n## Error handling considerations\n\n- **Missing event log channels**: Custom application channels may not exist on all machines. The agent logs a warning but continues. Templates for vendor-specific channels should document this.\n- **Permission issues**: Reading the Security event log requires `SeSecurityPrivilege` or membership in the `Event Log Readers` group. Standard Application and System channels are readable by all authenticated users.\n- **Large batch sizes and memory**: Setting `batch_read_size` too high on high-volume channels can cause memory pressure. Start with the default and increase only if throughput is insufficient.\n- **Message rendering failures**: On forwarded event collectors or minimal Windows installations, provider message DLLs may be absent. Events will have populated `winlog.event_id` and `winlog.event_data` fields but empty or placeholder `message` fields.\n\n## Review checklist\n\n### Channel configuration\n\n- [ ] `name` field present with correct channel name -- **HIGH**\n- [ ] Channel name exact and correctly cased (including `/Operational` suffix where needed) -- **HIGH**\n- [ ] Separate template files for each channel when collecting from multiple channels -- **MEDIUM**\n\n### Event filtering\n\n- [ ] Event ID filter present for high-volume channels (Security, Sysmon) -- **HIGH**\n- [ ] Event ID filter is configurable via template variable -- **MEDIUM**\n- [ ] Provider filter used when channel aggregates multiple sources -- **LOW**\n- [ ] Level filter set when only high-severity events are needed -- **LOW**\n\n### Performance\n\n- [ ] `ignore_older` configured for high-volume channels -- **MEDIUM**\n- [ ] `batch_read_size` appropriate for expected event volume -- **LOW**\n- [ ] API type documented if using experimental -- **LOW**\n\n### Forwarded events\n\n- [ ] `forwarded: true` set when reading `ForwardedEvents` channel -- **HIGH**\n- [ ] Language handling configured for multi-locale environments -- **MEDIUM**\n\n### XML rendering\n\n- [ ] `include_xml` enabled only when raw XML is needed downstream -- **LOW**\n- [ ] Event size impact considered when `include_xml: true` -- **LOW**\n\n### Common patterns\n\n- [ ] `preserve_original_event` is conditional (`{{#if}}`) -- **MEDIUM**\n- [ ] `forwarded` tag and `publisher_pipeline.disable_host` are coupled -- **MEDIUM**\n- [ ] Custom processors passthrough at top level -- **LOW**\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":13591,"content_sha256":"96acccd7e459d5a007582df663c6204450fb255a61fb40c248f203466ac57616"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"input-configurations","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"When to use","type":"text"}]},{"type":"paragraph","content":[{"text":"Load this skill whenever tasks include:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"building, modifying, or reviewing ","type":"text"},{"text":"agent/stream/*.yml.hbs","type":"text","marks":[{"type":"code_inline"}]},{"text":" templates for non-CEL input types","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"configuring file-based inputs (Filestream, Logfile, Journald, Winlog)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"enabling Federated Identity (Cloud Connectors) on an AWS integration package","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"When not to use","type":"text"}]},{"type":"paragraph","content":[{"text":"Do not use this skill as the primary guide for:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"CEL ","type":"text"},{"text":"program","type":"text","marks":[{"type":"em"}]},{"text":" development (","type":"text"},{"text":"cel-programs","type":"text","marks":[{"type":"code_inline"}]},{"text":") -- CEL program structure, state model, and mito workflow. Exception: the ","type":"text"},{"text":"auth.aws","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"use_cloud_connectors","type":"text","marks":[{"type":"code_inline"}]},{"text":" template block for Federated Identity is owned here via ","type":"text"},{"text":"references/federated-identity-aws.md","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"ingest pipeline processor design (","type":"text"},{"text":"ingest-pipelines","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"field mappings and ECS compliance (","type":"text"},{"text":"ecs-field-mappings","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"var_groups","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"provider_permissions","type":"text","marks":[{"type":"code_inline"}]},{"text":" ","type":"text"},{"text":"schema","type":"text","marks":[{"type":"em"}]},{"text":" and ","type":"text"},{"text":"format_version","type":"text","marks":[{"type":"code_inline"}]},{"text":" floors alone (","type":"text"},{"text":"package-spec","type":"text","marks":[{"type":"code_inline"}]},{"text":") -- use this skill for the end-to-end federation procedure that applies them","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Mandatory first read","type":"text"}]},{"type":"paragraph","content":[{"text":"Always load ","type":"text","marks":[{"type":"strong"}]},{"text":"references/common-input-patterns.md","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" first.","type":"text","marks":[{"type":"strong"}]},{"text":" It covers patterns that apply to every input type (tags, processors passthrough, variable conventions, ","type":"text"},{"text":"forwarded","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"publisher_pipeline.disable_host","type":"text","marks":[{"type":"code_inline"}]},{"text":" coupling). These patterns are prerequisites for all type-specific guides.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Type routing table","type":"text"}]},{"type":"paragraph","content":[{"text":"Detect the input type from the filename pattern in ","type":"text"},{"text":"agent/stream/","type":"text","marks":[{"type":"code_inline"}]},{"text":" or from the data stream manifest ","type":"text"},{"text":"input:","type":"text","marks":[{"type":"code_inline"}]},{"text":" field, then load the matching guide.","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Input type","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Filename pattern","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Guide","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"HTTPJSON","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"httpjson.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/httpjson-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"AWS S3","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"aws-s3.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/aws-s3-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"CloudWatch","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"aws-cloudwatch.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/aws-cloudwatch-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Azure Blob Storage","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"azure-blob-storage.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/azure-blob-storage-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Azure Event Hub","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"azure-eventhub.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/azure-eventhub-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"GCS","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"gcs.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/gcs-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"GCP Pub/Sub","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"gcp-pubsub.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/gcp-pubsub-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"TCP","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"tcp.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/tcp-udp-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"UDP","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"udp.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/tcp-udp-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"HTTP Endpoint","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"http_endpoint.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/http-endpoint-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Filestream","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"filestream.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/filestream-logfile-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Logfile","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"log.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/filestream-logfile-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Journald","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"journald.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/journald-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Winlog","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"winlog.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/winlog-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"WebSocket","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"websocket.yml.hbs","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/websocket-guide.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"Load ","type":"text"},{"text":"only","type":"text","marks":[{"type":"strong"}]},{"text":" the guide for the detected input type, not all guides.","type":"text"}]},{"type":"paragraph","content":[{"text":"For ","type":"text"},{"text":"Federated Identity","type":"text","marks":[{"type":"strong"}]},{"text":" tasks (any eligible input type, including ","type":"text"},{"text":"cel","type":"text","marks":[{"type":"code_inline"}]},{"text":"), load ","type":"text"},{"text":"references/federated-identity-aws.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" regardless of which input type is involved.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Handoff","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For ","type":"text"},{"text":"CEL program logic","type":"text","marks":[{"type":"strong"}]},{"text":", hand off to the ","type":"text"},{"text":"cel-programs","type":"text","marks":[{"type":"code_inline"}]},{"text":" skill. Keep this skill loaded for Federated Identity ","type":"text"},{"text":"auth.aws","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"use_cloud_connectors","type":"text","marks":[{"type":"code_inline"}]},{"text":" edits on ","type":"text"},{"text":"cel.yml.hbs","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For ","type":"text"},{"text":"manifest schema","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"var_groups","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"provider_permissions","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"format_version","type":"text","marks":[{"type":"code_inline"}]},{"text":" / conditions floors), hand off to the ","type":"text"},{"text":"package-spec","type":"text","marks":[{"type":"code_inline"}]},{"text":" skill (","type":"text"},{"text":"references/var-groups-and-provider-permissions.md","type":"text","marks":[{"type":"code_inline"}]},{"text":").","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For ","type":"text"},{"text":"pipeline issues","type":"text","marks":[{"type":"strong"}]},{"text":" discovered while reviewing input templates, hand off to the ","type":"text"},{"text":"ingest-pipelines","type":"text","marks":[{"type":"code_inline"}]},{"text":" skill.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For ","type":"text"},{"text":"field mapping issues","type":"text","marks":[{"type":"strong"}]},{"text":" found in template variable wiring, hand off to the ","type":"text"},{"text":"ecs-field-mappings","type":"text","marks":[{"type":"code_inline"}]},{"text":" skill.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"References","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/common-input-patterns.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- tags, processors passthrough, variable conventions, review flags (applies to ALL input types)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/httpjson-guide.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- HTTPJSON template syntax, structure, validation rules, pagination patterns, authentication, cursor persistence","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/aws-s3-guide.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- S3 bucket/SQS notification collection","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/aws-cloudwatch-guide.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- CloudWatch log group collection","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/azure-blob-storage-guide.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- Azure Blob Storage collection","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/azure-eventhub-guide.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- Azure Event Hub collection","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/gcs-guide.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- Google Cloud Storage collection","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/gcp-pubsub-guide.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- GCP Pub/Sub collection","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/tcp-udp-guide.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- TCP and UDP (syslog-style) listeners","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/http-endpoint-guide.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- HTTP Endpoint (webhook receiver)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/filestream-logfile-guide.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- Filestream and legacy Logfile inputs","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/journald-guide.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- Journald collection","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/winlog-guide.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- Windows Event Log collection","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/websocket-guide.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- WebSocket streaming (may embed CEL)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/federated-identity-aws.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- AWS Federated Identity procedure: input classification, federation vars, ","type":"text"},{"text":"auth.aws","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"use_cloud_connectors","type":"text","marks":[{"type":"code_inline"}]},{"text":", input gating (schema lives in ","type":"text"},{"text":"package-spec","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]}]}]},"metadata":{"date":"2026-11-02","name":"input-configurations","author":"@skillopedia","source":{"stars":16,"repo_name":"integration-skills","origin_url":"https://github.com/elastic/integration-skills/blob/HEAD/skills/input-configurations/SKILL.md","repo_owner":"elastic","body_sha256":"7ebdb16054d21fed821ab4b438db071c0a554e060d6137dea750a916ab63ce78","cluster_key":"cb262f124f08673e138b136754a6bb1c09ea745db1e0b3cfe3939c836ab0f2e8","clean_bundle":{"format":"clean-skill-bundle-v1","source":"elastic/integration-skills/skills/input-configurations/SKILL.md","text_bytes":173940,"source_commit":"39792cf81750ee54cba5b653fbfb69706871d42b","skills_sh_rank":19615,"attachment_count":15,"text_attachments":15,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"skills/input-configurations/SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"v1","category":"integrations-apis","category_label":"Integrations"},"exact_dupes_collapsed_into_this":0},"license":"Apache-2.0","version":"v1","category":"integrations-apis","metadata":{"author":"elastic","version":"1.0"},"import_tag":"skills-sh-top20k-2026-11","description":"Input template configuration for Elastic integrations. Covers agent stream templates (agent/stream/*.yml.hbs) for all non-CEL input types: HTTPJSON, AWS S3, CloudWatch, Azure Blob, Azure EventHub, GCS, GCP Pub/Sub, TCP, UDP, HTTP Endpoint, Filestream, Logfile, Journald, Winlog, and WebSocket. Also covers Federated Identity (Cloud Connectors) for agentless AWS integrations, including the auth.aws / use_cloud_connectors block on CEL stream templates. For CEL program logic, use the cel-programs skill."}},"renderedAt":1793666025672}

input-configurations When to use Load this skill whenever tasks include: - building, modifying, or reviewing templates for non-CEL input types - configuring request, response, pagination, cursor, or authentication blocks in HTTPJSON templates - wiring up cloud storage inputs (AWS S3, GCS, Azure Blob, Azure EventHub) - setting up network inputs (TCP, UDP, HTTP Endpoint, WebSocket) - configuring file-based inputs (Filestream, Logfile, Journald, Winlog) - enabling Federated Identity (Cloud Connectors) on an AWS integration package When not to use Do not use this skill as the primary guide for: -…