integration-testing When to use Load this skill whenever tasks include: - authoring or debugging pipeline test fixtures ( / inputs, output, config files) - setting up or debugging system tests ( , , mock APIs, 0-hits failures) - writing script tests for failure paths, API error handling, or package upgrades When not to use Do not use this skill as the primary guide for: - ingest pipeline processor design and architecture ( ) - CEL program development ( ) - broad elastic-package command selection and stack lifecycle ( ) Reference files — load the one that matches your test type | Test type | L…

\n\nexec echo ${CONFIG_PROFILES}\nstdout '/\\.elastic-package/profiles

integration-testing When to use Load this skill whenever tasks include: - authoring or debugging pipeline test fixtures ( / inputs, output, config files) - setting up or debugging system tests ( , , mock APIs, 0-hits failures) - writing script tests for failure paths, API error handling, or package upgrades When not to use Do not use this skill as the primary guide for: - ingest pipeline processor design and architecture ( ) - CEL program development ( ) - broad elastic-package command selection and stack lifecycle ( ) Reference files — load the one that matches your test type | Test type | L…

\n\nexec echo ${PACKAGE_NAME}\nstdout '^o365

integration-testing When to use Load this skill whenever tasks include: - authoring or debugging pipeline test fixtures ( / inputs, output, config files) - setting up or debugging system tests ( , , mock APIs, 0-hits failures) - writing script tests for failure paths, API error handling, or package upgrades When not to use Do not use this skill as the primary guide for: - ingest pipeline processor design and architecture ( ) - CEL program development ( ) - broad elastic-package command selection and stack lifecycle ( ) Reference files — load the one that matches your test type | Test type | L…

\n\nexec echo ${PACKAGE_ROOT}\nstdout '/packages/o365

integration-testing When to use Load this skill whenever tasks include: - authoring or debugging pipeline test fixtures ( / inputs, output, config files) - setting up or debugging system tests ( , , mock APIs, 0-hits failures) - writing script tests for failure paths, API error handling, or package upgrades When not to use Do not use this skill as the primary guide for: - ingest pipeline processor design and architecture ( ) - CEL program development ( ) - broad elastic-package command selection and stack lifecycle ( ) Reference files — load the one that matches your test type | Test type | L…

\n\nexec echo ${DATA_STREAM}\nstdout '^audit

integration-testing When to use Load this skill whenever tasks include: - authoring or debugging pipeline test fixtures ( / inputs, output, config files) - setting up or debugging system tests ( , , mock APIs, 0-hits failures) - writing script tests for failure paths, API error handling, or package upgrades When not to use Do not use this skill as the primary guide for: - ingest pipeline processor design and architecture ( ) - CEL program development ( ) - broad elastic-package command selection and stack lifecycle ( ) Reference files — load the one that matches your test type | Test type | L…

\n\nexec echo ${DATA_STREAM_ROOT}\nstdout '/packages/o365/data_stream/audit

integration-testing When to use Load this skill whenever tasks include: - authoring or debugging pipeline test fixtures ( / inputs, output, config files) - setting up or debugging system tests ( , , mock APIs, 0-hits failures) - writing script tests for failure paths, API error handling, or package upgrades When not to use Do not use this skill as the primary guide for: - ingest pipeline processor design and architecture ( ) - CEL program development ( ) - broad elastic-package command selection and stack lifecycle ( ) Reference files — load the one that matches your test type | Test type | L…

\n\nexec echo ${CURRENT_VERSION}\nstdout '^[0-9]+\\.[0-9]+\\.[0-9]+

integration-testing When to use Load this skill whenever tasks include: - authoring or debugging pipeline test fixtures ( / inputs, output, config files) - setting up or debugging system tests ( , , mock APIs, 0-hits failures) - writing script tests for failure paths, API error handling, or package upgrades When not to use Do not use this skill as the primary guide for: - ingest pipeline processor design and architecture ( ) - CEL program development ( ) - broad elastic-package command selection and stack lifecycle ( ) Reference files — load the one that matches your test type | Test type | L…

\n\nexec echo ${PREVIOUS_VERSION}\nstdout '^[0-9]+\\.[0-9]+\\.[0-9]+

integration-testing When to use Load this skill whenever tasks include: - authoring or debugging pipeline test fixtures ( / inputs, output, config files) - setting up or debugging system tests ( , , mock APIs, 0-hits failures) - writing script tests for failure paths, API error handling, or package upgrades When not to use Do not use this skill as the primary guide for: - ingest pipeline processor design and architecture ( ) - CEL program development ( ) - broad elastic-package command selection and stack lifecycle ( ) Reference files — load the one that matches your test type | Test type | L…

\n```\n\nUse regex for version assertions so the test doesn't break on every release.\n\n## System-level test skeleton\n\nEvery system-level script test follows this structure:\n\n```txtar\n# Description of what this test verifies.\n\n[!external_stack] skip 'Skipping external stack test.'\n[!exec:jq] skip 'Skipping test requiring absent jq command'\n\n# 1. Connect.\nuse_stack -profile ${CONFIG_PROFILES}/${PROFILE}\ninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -network_name NETWORK_NAME\n\n# 2. Start mock service.\ndocker_up -profile ${CONFIG_PROFILES}/${PROFILE} -network ${NETWORK_NAME} \u003cmock-name>\n\n# 3. Install package and create policy.\nadd_package -profile ${CONFIG_PROFILES}/${PROFILE}\nadd_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} test_config.yaml DATA_STREAM_NAME\n\n# 4. Assert.\nget_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want \u003cN> -timeout 5m ${DATA_STREAM_NAME}\ncp stdout got_docs.json\nexec jq '\u003cquery>' got_docs.json\nstdout '\u003cexpected>'\n\n# 5. Clean up.\nremove_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} ${DATA_STREAM_NAME}\nuninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -timeout 1m\ndocker_down \u003cmock-name>\n```\n\n## Embedded files in the txtar\n\nThe `-- filename --` sections at the bottom of each txtar file contain:\n\n- **`test_config.yaml`** — package policy config (input type and vars). Point the integration at the mock service and set short intervals/windows.\n- **`\u003cmock-name>/docker-compose.yml`** — Docker Compose for the mock service.\n- **`\u003cmock-name>/config.yml`** — `elastic/stream` mock rules.\n\n## test_config.yaml\n\n```yaml\ninput: cel\nvars: ~\ndata_stream:\n vars:\n url: http://\u003cmock-name>:8080\n interval: 30s\n initial_interval: 1h\n preserve_original_event: true\n # ... other required vars with test values ...\n```\n\nSet `interval` and `initial_interval` to small values. Large defaults (e.g. 168h lookback) cause many API calls against the mock, slowing the test and making assertions harder to predict.\n\n## Upgrade test pattern\n\n```txtar\n[!external_stack] skip 'Skipping external stack test.'\n[!has_previous_release] skip 'No previous release to upgrade from.'\n[breaking_change] skip 'Cannot upgrade across breaking change.'\n[!exec:jq] skip 'Skipping test requiring absent jq command'\n\n# ... setup, add_package, add_package_policy, verify initial data ...\n\nupgrade_package_latest -profile ${CONFIG_PROFILES}/${PROFILE}\nstdout 'upgraded package '${PACKAGE_NAME}\n\n# ... verify data still present after upgrade ...\n\n# ... cleanup ...\n```\n\nThe `[!has_previous_release]` and `[breaking_change]` guards skip automatically when appropriate.\n\n## Mock service setup\n\nUse the `elastic/stream` Docker image (`docker.elastic.co/observability/stream`). Match the version used in the package's existing `_dev/deploy/docker/` config.\n\n### docker-compose.yml\n\n```yaml\nversion: '2.3'\nservices:\n \u003cmock-name>:\n image: docker.elastic.co/observability/stream:v0.22.0\n hostname: \u003cmock-name>\n ports:\n - 8080\n environment:\n PORT: \"8080\"\n volumes:\n - ./config.yml:/config.yml\n command:\n - http-server\n - --addr=:8080\n - --config=/config.yml\n```\n\nAlways set `hostname: \u003cmock-name>` explicitly. Without it the container defaults to the container ID, which the agent cannot resolve on the Docker network.\n\n### config.yml structure\n\nRules are matched top-down. Each rule specifies path, methods, optional query params, optional request headers, and responses:\n\n```yaml\nrules:\n - path: /api/v1/items\n methods: [GET]\n query_params:\n api_key: test-key\n responses:\n - status_code: 200\n headers:\n Content-Type:\n - \"application/json\"\n body: |-\n {\"items\": [...]}\n```\n\nTemplate variables available in response bodies:\n- **`{{ hostname }}`** — Docker container hostname (required for self-referential URLs, e.g. next-page links)\n- **`{{ env \"PORT\" }}`** — value of the PORT environment variable\n- **`{varName:regex}`** — in `query_params` values, captures variable request parameters (e.g. `startTime: \"{startTime:.*}\"`)\n- **`{{ .request.vars.varName }}`** — references a captured query param in the response body\n\n**Variable-capture catchall rules** are essential for APIs with dynamic query parameters (timestamps, cursor tokens). Without them, CEL programs with timestamp-cursor pagination loop indefinitely. Always include `{varName:.*}` captures for all dynamic query params:\n\n```yaml\n- path: /api/v1/events\n methods: ['GET']\n query_params:\n startTime: \"{startTime:.*}\"\n endTime: \"{endTime:.*}\"\n request_headers:\n Authorization:\n - \"Bearer test-token\"\n responses:\n - status_code: 200\n headers:\n Content-Type:\n - \"application/json\"\n body: |-\n {\"events\":[...], \"nextPage\": \"http://{{ hostname }}:{{ env \"PORT\" }}/api/v1/events?cursor={{ .request.vars.endTime }}\"}\n```\n\nPut more specific rules (with `query_params` or `request_headers`) before less specific ones for the same path.\n\n## Running script tests\n\n```bash\n# Run all script tests for a data stream:\nelastic-package test script -v --data-streams \u003cdata_stream>\n\n# Run a single test:\nelastic-package test script -v --data-streams \u003cdata_stream> --run \u003ctest_name>\n\n# Keep work directory for debugging:\nelastic-package test script -v --data-streams \u003cdata_stream> --work\n\n# Verbose script output:\nelastic-package test script -v --data-streams \u003cdata_stream> --verbose-scripts\n```\n\nTests require a running Elastic stack (`--external-stack`, which is the default).\n\n## Pitfalls and hard-won lessons\n\n### Assert against indexed fields, not input fields\n\n`get_docs` returns post-pipeline documents. If the ingest pipeline renames `foobar` to `foo.bar`, assert against `foo.bar`. Always check the pipeline first.\n\n### Do NOT call `remove_package` in cleanup\n\nFor real packages, Kibana often refuses removal because Fleet hasn't finished cascading the agent policy deletion to package policies. The script test runner's automatic cleanup handles package removal. **Omit `remove_package`** from cleanup.\n\n### Read the input's error-handling logic before choosing mock responses\n\nCEL and httpjson programs often treat some non-200 HTTP codes as expected, non-error conditions (e.g. 400 = \"already subscribed\", 404 = \"no new data\"). A mock returning such a code won't trigger the error path. Read the program to find which status codes actually reach error-handling code, then mock those.\n\n### Use `-confirm` with `get_docs` for exact-count assertions\n\nWhen asserting that exactly N documents arrive and no more, add `-confirm 15s`:\n\n```txtar\nget_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 1 -confirm 15s -timeout 5m ${DATA_STREAM_NAME}\n```\n\nThis waits an extra 15 seconds after reaching N to confirm no additional documents arrive.\n\n### Minimise polling and batch windows in test configs\n\nSet short `initial_interval`, `interval`, and `batch_size` values in `test_config.yaml` so each test cycle makes a predictable, minimal number of API requests.\n\n### Set `hostname` in docker-compose for mock services\n\nWithout an explicit `hostname: \u003cmock-name>` in docker-compose, the container defaults to its container ID. Agents on the Docker network cannot resolve container IDs.\n\n## Full examples (o365 integration)\n\n### Partial API failure — subscription_permission_error.txt\n\nTests that a permission error (401 AF10001) for one content type does not prevent collection from other content types. Three content types configured; one rejected by the mock. Expects 1 error event + 2 data events.\n\nKey decisions:\n- Uses **401** (not 400) because the CEL program treats 400 as success (\"already subscribed\").\n- Asserts against `.o365.audit` (post-pipeline name), not `.o365audit`.\n- No `remove_package` in cleanup.\n\n```txtar\n# Test that a subscription permission error (AF10001) for one content type\n# does not prevent collection from other content types.\n\n[!external_stack] skip 'Skipping external stack test.'\n[!exec:jq] skip 'Skipping test requiring absent jq command'\n\nuse_stack -profile ${CONFIG_PROFILES}/${PROFILE}\ninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -network_name NETWORK_NAME\ndocker_up -profile ${CONFIG_PROFILES}/${PROFILE} -network ${NETWORK_NAME} o365-mock\nadd_package -profile ${CONFIG_PROFILES}/${PROFILE}\nadd_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} test_config.yaml DATA_STREAM_NAME\n\n# Wait for documents: 1 error event + 2 data events = 3.\nget_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 3 -timeout 5m ${DATA_STREAM_NAME}\ncp stdout got_docs.json\n\n# Verify error event mentions the rejected content type.\nexec jq -r '[.hits.hits[]._source.error.message // empty] | flatten | .[]' got_docs.json\nstdout 'Audit.TypeRequiringAdditionalPermissions'\n\n# Verify 2 normal data events from the working content types.\nexec jq '[.hits.hits[]._source | select(.o365.audit != null)] | length' got_docs.json\nstdout '^2

integration-testing When to use Load this skill whenever tasks include: - authoring or debugging pipeline test fixtures ( / inputs, output, config files) - setting up or debugging system tests ( , , mock APIs, 0-hits failures) - writing script tests for failure paths, API error handling, or package upgrades When not to use Do not use this skill as the primary guide for: - ingest pipeline processor design and architecture ( ) - CEL program development ( ) - broad elastic-package command selection and stack lifecycle ( ) Reference files — load the one that matches your test type | Test type | L…

\n\n# Clean up.\nremove_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} ${DATA_STREAM_NAME}\nuninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -timeout 1m\ndocker_down o365-mock\n```\n\n### Total API failure — invalid_content_type.txt\n\nTests that a single invalid content type results in exactly one error event and no data. Uses **403** (not 400) to trigger the error path. Uses `-confirm 15s` to verify no extra documents arrive.\n\n```txtar\n# Test that configuring a single invalid content type results in an error\n# event and no data collection.\n\n[!external_stack] skip 'Skipping external stack test.'\n[!exec:jq] skip 'Skipping test requiring absent jq command'\n\nuse_stack -profile ${CONFIG_PROFILES}/${PROFILE}\ninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -network_name NETWORK_NAME\ndocker_up -profile ${CONFIG_PROFILES}/${PROFILE} -network ${NETWORK_NAME} o365-mock\nadd_package -profile ${CONFIG_PROFILES}/${PROFILE}\nadd_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} test_config.yaml DATA_STREAM_NAME\n\nget_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 1 -confirm 15s -timeout 5m ${DATA_STREAM_NAME}\ncp stdout got_docs.json\n\nexec jq -r '[.hits.hits[]._source.error.message // empty] | flatten | .[]' got_docs.json\nstdout 'Audit.Nonexistent'\n\nexec jq '[.hits.hits[]._source | select(.o365.audit != null)] | length' got_docs.json\nstdout '^0

integration-testing When to use Load this skill whenever tasks include: - authoring or debugging pipeline test fixtures ( / inputs, output, config files) - setting up or debugging system tests ( , , mock APIs, 0-hits failures) - writing script tests for failure paths, API error handling, or package upgrades When not to use Do not use this skill as the primary guide for: - ingest pipeline processor design and architecture ( ) - CEL program development ( ) - broad elastic-package command selection and stack lifecycle ( ) Reference files — load the one that matches your test type | Test type | L…

\n\nremove_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} ${DATA_STREAM_NAME}\nuninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -timeout 1m\ndocker_down o365-mock\n```\n\n### Package upgrade — upgrade.txt\n\nTests that upgrading the package doesn't break data collection. Skips automatically if there's no previous release or if the latest change is a breaking change.\n\n```txtar\n[!external_stack] skip 'Skipping external stack test.'\n[!has_previous_release] skip 'No previous release to upgrade from.'\n[breaking_change] skip 'Cannot upgrade across breaking change.'\n[!exec:jq] skip 'Skipping test requiring absent jq command'\n\nuse_stack -profile ${CONFIG_PROFILES}/${PROFILE}\ninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -network_name NETWORK_NAME\ndocker_up -profile ${CONFIG_PROFILES}/${PROFILE} -network ${NETWORK_NAME} o365-mock\nadd_package -profile ${CONFIG_PROFILES}/${PROFILE}\nadd_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} test_config.yaml DATA_STREAM_NAME\n\n# Verify initial data collection.\nget_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 2 -timeout 5m ${DATA_STREAM_NAME}\ncp stdout got_pre_upgrade.json\nexec jq '.hits.total.value' got_pre_upgrade.json\nstdout '^2

integration-testing When to use Load this skill whenever tasks include: - authoring or debugging pipeline test fixtures ( / inputs, output, config files) - setting up or debugging system tests ( , , mock APIs, 0-hits failures) - writing script tests for failure paths, API error handling, or package upgrades When not to use Do not use this skill as the primary guide for: - ingest pipeline processor design and architecture ( ) - CEL program development ( ) - broad elastic-package command selection and stack lifecycle ( ) Reference files — load the one that matches your test type | Test type | L…

\n\n# Upgrade.\nupgrade_package_latest -profile ${CONFIG_PROFILES}/${PROFILE}\nstdout 'upgraded package '${PACKAGE_NAME}\n\n# Verify data survives the upgrade.\nget_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 2 -timeout 2m ${DATA_STREAM_NAME}\ncp stdout got_post_upgrade.json\nexec jq '.hits.total.value >= 2' got_post_upgrade.json\nstdout '^true

integration-testing When to use Load this skill whenever tasks include: - authoring or debugging pipeline test fixtures ( / inputs, output, config files) - setting up or debugging system tests ( , , mock APIs, 0-hits failures) - writing script tests for failure paths, API error handling, or package upgrades When not to use Do not use this skill as the primary guide for: - ingest pipeline processor design and architecture ( ) - CEL program development ( ) - broad elastic-package command selection and stack lifecycle ( ) Reference files — load the one that matches your test type | Test type | L…

\n\nremove_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} ${DATA_STREAM_NAME}\nuninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -timeout 1m\ndocker_down o365-mock\n```\n\n### Mock config.yml — o365 full example\n\nHandles OAuth token exchange, subscription endpoints with per-content-type routing, content listing with variable-capture, and content fetch:\n\n```yaml\nrules:\n # OAuth token endpoint.\n - path: /test-cel-tenant-id/oauth2/v2.0/token\n methods: [POST]\n query_params:\n client_id: test-cel-client-id\n client_secret: test-cel-client-secret\n grant_type: client_credentials\n scope: https://manage.office.com/.default\n request_headers:\n Content-Type:\n - \"application/x-www-form-urlencoded\"\n responses:\n - status_code: 200\n headers:\n Content-Type:\n - \"application/json\"\n body: |-\n {\"access_token\":\"test-token\",\"token_type\":\"Bearer\",\"expires_in\":3600}\n\n # Subscribe -- success.\n - path: /api/v1.0/test-cel-tenant-id/activity/feed/subscriptions/start\n methods: [POST]\n query_params:\n contentType: \"Audit.SharePoint\"\n PublisherIdentifier: test-cel-tenant-id\n request_headers:\n Authorization:\n - \"Bearer test-token\"\n responses:\n - status_code: 200\n headers:\n Content-Type:\n - \"application/json\"\n body: |-\n {\"contentType\":\"Audit.SharePoint\",\"status\":\"enabled\",\"webhook\":null}\n\n # Subscribe -- error (permission denied for this content type).\n - path: /api/v1.0/test-cel-tenant-id/activity/feed/subscriptions/start\n methods: [POST]\n query_params:\n contentType: \"Audit.TypeRequiringAdditionalPermissions\"\n PublisherIdentifier: test-cel-tenant-id\n request_headers:\n Authorization:\n - \"Bearer test-token\"\n responses:\n - status_code: 401\n headers:\n Content-Type:\n - \"application/json\"\n body: |-\n {\"error\":{\"code\":\"AF10001\",\"message\":\"Permission denied.\"}}\n\n # List content -- variable-capture for dynamic time params + self-referential URL.\n - path: /api/v1.0/test-cel-tenant-id/activity/feed/subscriptions/content\n methods: [GET]\n query_params:\n contentType: \"Audit.SharePoint\"\n startTime: \"{startTime:.*}\"\n endTime: \"{endTime:.*}\"\n PublisherIdentifier: test-cel-tenant-id\n request_headers:\n Authorization:\n - \"Bearer test-token\"\n responses:\n - status_code: 200\n headers:\n Content-Type:\n - \"application/json\"\n body: |-\n [{\"contentType\":\"Audit.SharePoint\",\"contentId\":\"sp-1\",\"contentUri\":\"http://{{ hostname }}:{{ env \"PORT\" }}/api/v1.0/test-cel-tenant-id/activity/feed/audit/sp-1\",\"contentCreated\":\"{{ .request.vars.endTime }}\",\"contentExpiration\":\"2199-12-31T23:59:59.000Z\"}]\n\n # Fetch content.\n - path: /api/v1.0/test-cel-tenant-id/activity/feed/audit/sp-1\n methods: [GET]\n request_headers:\n Authorization:\n - \"Bearer test-token\"\n responses:\n - status_code: 200\n headers:\n Content-Type:\n - \"application/json\"\n body: |-\n [{\"Id\":\"sp-event-001\",\"CreationTime\":\"2020-02-07T16:43:53\",\"Workload\":\"SharePoint\",\"Operation\":\"PageViewed\",\"RecordType\":4}]\n```\n\nKey patterns:\n- **`{{ hostname }}`** — resolves to the Docker container hostname (set via `hostname:` in docker-compose). Required for self-referential URLs such as next-page links.\n- **`{{ env \"PORT\" }}`** — resolves to the PORT environment variable.\n- **`{varName:regex}`** — captures variable request values in query params.\n- **`{{ .request.vars.endTime }}`** — references a captured query param in the response body.\n- Rules are matched top-down; put more specific rules before less specific ones on the same path.\n\n### test_config.yaml — o365 example\n\n```yaml\ninput: cel\nvars: ~\ndata_stream:\n vars:\n url: http://o365-mock:8080\n token_url: http://o365-mock:8080\n preserve_original_event: true\n client_id: test-cel-client-id\n client_secret: test-cel-client-secret\n azure_tenant_id: test-cel-tenant-id\n content_types: \"Audit.SharePoint, Audit.General\"\n interval: 30s\n initial_interval: 1h\n enable_request_tracer: false\n```\n\n`initial_interval: 1h` with `interval: 30s` gives one listing batch per content type per cycle. Longer values generate many batches and produce more documents than the assertion expects.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":19625,"content_sha256":"edb66faaeb43615ca550498f701a25644ad7e89adcdc39807c101cdc03d6bf2e"},{"filename":"references/system-testing-cel.md","content":"# system testing — CEL input\n\nInput-specific guidance for system-testing data streams that use the `cel` input. Load `system-testing.md` (generic) first.\n\n## Overview\n\nCEL system tests require a mock HTTP API to stand in for the real vendor API. The mock is an `elastic/stream` http-server container defined in `_dev/deploy/docker/docker-compose.yml` with a rule-based config file.\n\n## Docker Compose pattern\n\n```yaml\nversion: '2.3'\nservices:\n \u003cpackage>-\u003cstream>-mock:\n image: docker.elastic.co/observability/stream:v0.22.0\n volumes:\n - ./files:/files:ro\n command: http-server --addr=:8090 --config=/files/config-\u003cstream>.yml\n ports:\n - 8090\n```\n\nThe mock config file at `_dev/deploy/docker/files/config-\u003cstream>.yml` contains rule-based request matching with response definitions.\n\n## Test config pattern\n\n```yaml\nwait_for_data_timeout: 1m\ninput: cel\nservice: \u003cpackage>-\u003cstream>-mock\ndata_stream:\n vars:\n url: http://{{Hostname}}:{{Port}}\nassert:\n hit_count: \u003cexpected_count>\n```\n\n## Key patterns\n\n- **Rule-based mock config**: rules match on `path`, `methods`, `query_params`, and `request_headers` (first match wins, top-down ordering)\n- **Variable-capture patterns**: use `{varName:regex}` in query param values for dynamic params like timestamps — without these, the mock returns the same data for every time window, causing infinite loops\n- **Two-round cursor testing**: set `interval: 2s` in test config so the agent completes one pagination cycle, persists the cursor, then fires a second cycle to verify cursor persistence\n- **`assert.hit_count`**: must account for events from both evaluation rounds\n\n## Debugging 0 hits (CEL-specific)\n\n### Step 1: Check the mock container log (always first)\n\n```\nbuild/container-logs/\u003cpackage_name>-\u003cdatastream_name>-\u003cDIGIT>.log\n```\n\nThis log shows every HTTP request the CEL program makes against the mock API, including which rules matched. **Tail the last 100–200 lines** as the first diagnostic step.\n\nCommon patterns:\n- **Hundreds of requests with advancing timestamps**: infinite time-window looping — the mock lacks variable-capture catchall rules\n- **No requests at all**: the agent never contacted the mock — check `url` and `service:` in the test config\n- **Requests with no rule matches**: query param or header mismatch between CEL program and mock rules\n- **401/403 responses**: auth rule mismatch in mock config\n\n### Step 2: Check agent log for dropped events\n\nFollow the generic debugging steps in `system-testing.md` → \"Debugging system test failures — general\".\n\nA CEL-specific rejection reason:\n- `Duplicate field '@timestamp'`: the CEL program sets `@timestamp` in event output while the framework also adds it. Fix: only emit `{\"message\": e.encode_json()}`.\n\n## Additional CEL-specific failure patterns\n\n- **0 hits with CEL input**: check container logs first (Step 1) before any other debugging\n- **Infinite time-window looping**: mock config lacks variable-capture catchall rules — fix by adding `startTime: \"{startTime:.*}\"` patterns\n\n## Detailed mock setup reference\n\nFor comprehensive mock API flow design, variable-capture syntax, pagination mocking, and two-round cursor persistence patterns, see the `cel-programs` skill → `references/cel-system-tests.md`.\n\n## Reference integrations\n\n- Any CEL integration in `elastic/integrations` — e.g. `wiz`, `ti_otx`, `canva` — uses `elastic/stream` http-server mocks with rule configs\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3448,"content_sha256":"9601b5b53e4847f1e1d22ee06f6d057d461422c1f3a5ce57588a1ac3546896d4"},{"filename":"references/system-testing-cloud-skip.md","content":"# system testing — cloud storage inputs (skip guidance)\n\nGuidance for data streams that use `aws-s3`, `gcs`, `azure-blob-storage`, or `azure-eventhub` inputs. Load `system-testing.md` (generic) first.\n\n## When to skip system tests\n\nCloud storage and cloud message bus inputs (`aws-s3`, `gcs`, `azure-blob-storage`, `azure-eventhub`) currently **do not have a standard docker-based system test pattern**. These inputs require cloud infrastructure (S3 buckets, SQS queues, GCS buckets, Azure Blob containers, Event Hubs) that cannot be reliably emulated in a local Docker environment.\n\nSome integrations in the official repo use Terraform-based system tests (`_dev/deploy/tf/`) that provision real AWS/GCP/Azure resources in CI, but these are:\n- Not part of the standard `elastic-package test system` docker workflow\n- Require cloud credentials and infrastructure\n- Not suitable for local development or standard CI pipelines\n\n## What to do instead\n\n1. **Focus on pipeline tests**: create thorough pipeline test fixtures (`_dev/test/pipeline/`) covering all event types, edge cases, and error paths. Pipeline tests validate the ingest pipeline independently of the input.\n\n2. **Skip system test setup**: do not create `_dev/deploy/docker/docker-compose.yml` or `_dev/test/system/` test configs for cloud storage inputs unless a docker-based mock is available.\n\n3. **`sample_event.json` generation**: since `elastic-package test system --generate` cannot run without a system test, `sample_event.json` must be created through alternative means:\n - Run the pipeline against a representative fixture using `elastic-package test pipeline`, then construct `sample_event.json` from the expected output\n - Or set up a temporary local stack run with real cloud credentials to generate it\n\n4. **Note in the report**: when the orchestrator skips system tests for cloud inputs, include this in the final report: \"System tests skipped for `\u003cinput>` data stream — no docker-based mock available for cloud storage inputs. Pipeline tests provide coverage.\"\n\n## Integrations with Terraform-based tests (for reference)\n\nSome integrations use `_dev/deploy/tf/` with Terraform and real cloud resources:\n\n- `crowdstrike` (FDR) — uses `{{TF_OUTPUT_queue_url}}` with real AWS SQS\n- `github` (audit) — combines TF for S3 with Docker for GCS mock and Azurite\n\nThese patterns require cloud credentials (`AWS_ACCESS_KEY_ID`, etc.) and are only suitable for CI environments with cloud access.\n\n## Partial docker-based mocks (advanced)\n\nA few integrations use docker-based mocks for cloud storage:\n\n- **GCS**: `shourieg/gcs-mock-service` provides a basic GCS-compatible API\n- **Azure Blob**: `mcr.microsoft.com/azure-storage/azurite` emulates Azure Blob Storage locally\n\nThese are not widely adopted and require additional setup (uploading test data, configuring endpoints). If the integration warrants it, examine `google_cloud_storage` or `symantec_endpoint_security` in the official repo for working examples.\n\n## Reference integrations\n\n- [`crowdstrike`](https://github.com/elastic/integrations/tree/main/packages/crowdstrike) (FDR) — TF-based aws-s3 tests\n- [`google_cloud_storage`](https://github.com/elastic/integrations/tree/main/packages/google_cloud_storage) — docker GCS mock\n- [`symantec_endpoint_security`](https://github.com/elastic/integrations/tree/main/packages/symantec_endpoint_security) — aws-s3, GCS, and Azure Blob test configs\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3438,"content_sha256":"bfd8ae497cf09bb6d0540f705d909e10ea38449b7ce8961b757fa36ca18f5956"},{"filename":"references/system-testing-http-endpoint.md","content":"# system testing — HTTP endpoint (webhook) input\n\nInput-specific guidance for system-testing data streams that use the `http_endpoint` input. Load `system-testing.md` (generic) first.\n\n## Overview\n\nHTTP endpoint system tests use `elastic/stream` as an HTTP client that posts NDJSON log data to the Elastic Agent's `http_endpoint` listener. The `stream` tool sends data as webhook-style HTTP POST requests using `STREAM_PROTOCOL=webhook`.\n\n## Docker Compose pattern\n\n### HTTP\n\n```yaml\nversion: '2.3'\nservices:\n \u003cpackage>-\u003cstream>-webhook-http:\n image: docker.elastic.co/observability/stream:v0.22.0\n volumes:\n - ./sample_logs:/sample_logs:ro\n environment:\n - STREAM_PROTOCOL=webhook\n - STREAM_WEBHOOK_PROBE=false\n - STREAM_ADDR=http://elastic-agent:\u003cport>/\u003curl_path>\n - STREAM_WEBHOOK_HEADER=Authorization=\u003cauth_value>\n command: log --start-signal=SIGHUP --delay=5s /sample_logs/\u003clogfile>.log\n```\n\n### HTTPS\n\n```yaml\n \u003cpackage>-\u003cstream>-webhook-https:\n image: docker.elastic.co/observability/stream:v0.22.0\n volumes:\n - ./sample_logs:/sample_logs:ro\n environment:\n - STREAM_PROTOCOL=webhook\n - STREAM_WEBHOOK_PROBE=false\n - STREAM_INSECURE=true\n - STREAM_ADDR=https://elastic-agent:\u003cport>/\u003curl_path>\n - STREAM_WEBHOOK_HEADER=Authorization=\u003cauth_value>\n command: log --start-signal=SIGHUP --delay=5s /sample_logs/\u003clogfile>.log\n```\n\n### With basic auth\n\n```yaml\n environment:\n - STREAM_PROTOCOL=webhook\n - STREAM_WEBHOOK_PROBE=false\n - STREAM_ADDR=http://elastic-agent:\u003cport>/\u003curl_path>\n - STREAM_WEBHOOK_USERNAME=abc123\n - STREAM_WEBHOOK_PASSWORD=abc123\n```\n\n## Test config pattern\n\n```yaml\nwait_for_data_timeout: 1m\nservice: \u003cpackage>-\u003cstream>-webhook-http\nservice_notify_signal: SIGHUP\ninput: http_endpoint\ndata_stream:\n vars:\n listen_address: 0.0.0.0\n listen_port: \u003cport>\n url: /\u003curl_path>\n secret_value: \u003cauth_value>\n preserve_original_event: true\nassert:\n hit_count: \u003cline_count>\n```\n\n## Key patterns\n\n- **`STREAM_PROTOCOL=webhook`**: tells `stream` to POST data as HTTP webhook requests instead of raw TCP/UDP\n- **`STREAM_WEBHOOK_PROBE=false`**: disables the probe request that some webhook receivers expect\n- **`STREAM_ADDR`**: full URL including protocol, host (`elastic-agent`), port, and path. The path must match `url` in the test config vars.\n- **`STREAM_WEBHOOK_HEADER`**: sets custom headers on each POST request — used for auth tokens, API keys. Format: `HeaderName=value`.\n- **`STREAM_INSECURE=true`**: required for HTTPS when using self-signed certificates\n- **Auth alignment**: the auth value in `STREAM_WEBHOOK_HEADER` (or `STREAM_WEBHOOK_USERNAME`/`PASSWORD`) must match the auth config in the test config vars (`secret_value`, `basic_auth`, etc.)\n- **Sample log file**: place NDJSON files in `_dev/deploy/docker/sample_logs/`. Each line is posted as one HTTP request body.\n- **Content-Type**: set `STREAM_WEBHOOK_HEADER=Content-Type=application/json` when the endpoint expects JSON\n\n## Reference integrations\n\n- [`zoom`](https://github.com/elastic/integrations/tree/main/packages/zoom) — HTTP and HTTPS webhook test configs\n- [`cloudflare_logpush`](https://github.com/elastic/integrations/tree/main/packages/cloudflare_logpush) — multiple data streams with webhook tests\n- [`http_endpoint`](https://github.com/elastic/integrations/tree/main/packages/http_endpoint) — reference implementation with basic auth and ack modes\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3477,"content_sha256":"2d6c48b9e4d5ec02ad65c275b60e90ae21dd0fa8c02bf9de0e4a1213ee8671fa"},{"filename":"references/system-testing-kafka-pubsub.md","content":"# system testing — Kafka and Pub/Sub inputs\n\nInput-specific guidance for system-testing data streams that use `kafka` or `gcp-pubsub` inputs. Load `system-testing.md` (generic) first.\n\n## Kafka\n\n### Overview\n\nKafka system tests run a real Kafka broker (Kraft mode) alongside an `elastic/stream` producer that publishes sample log lines to a topic. The Elastic Agent consumes from the topic using the `kafka` input.\n\n### Docker Compose pattern\n\n```yaml\nversion: '2.3'\nservices:\n kafka-service:\n image: bashj79/kafka-kraft\n healthcheck:\n test: nc -z kafka-service 9094 || exit -1\n interval: 10s\n timeout: 5s\n retries: 15\n environment:\n KAFKA_LISTENERS: \"INTERNAL://kafka-service:9092,EXTERNAL://:9094,CONTROLLER://:9093\"\n KAFKA_ADVERTISED_LISTENERS: \"INTERNAL://kafka-service:9092,EXTERNAL://kafka-service:9094\"\n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: \"CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT\"\n KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL\n ports:\n - 9094\n\n \u003cpackage>-\u003cstream>-producer:\n image: docker.elastic.co/observability/stream:v0.22.0\n volumes:\n - ./sample_logs:/sample_logs:ro\n command:\n - log\n - --retry=30\n - --addr=kafka-service:9094\n - -p=kafka\n - --kafka-topic=\u003ctopic_name>\n - /sample_logs/\u003clogfile>.log\n depends_on:\n kafka-service:\n condition: service_healthy\n```\n\n### Test config pattern\n\n```yaml\nwait_for_data_timeout: 1m\nservice: kafka-service\ninput: kafka\ndata_stream:\n vars:\n topics:\n - \u003ctopic_name>\n hosts:\n - \"{{Hostname}}:{{Port}}\"\n group_id: system_test\n```\n\n### Key patterns\n\n- **`service: kafka-service`**: the test config references the broker service, not the producer\n- **`{{Hostname}}:{{Port}}`**: resolved by the test runner to the broker's advertised address\n- **`--retry=30`**: the producer retries connecting to the broker since it may take time to become healthy\n- **`depends_on` with `service_healthy`**: ensures the broker is ready before the producer starts\n- **`--kafka-topic`**: must match the `topics` list in the test config\n\n## GCP Pub/Sub\n\n### Overview\n\nPub/Sub system tests use the Google Cloud Pub/Sub emulator alongside an `elastic/stream` publisher. The Elastic Agent consumes messages using the `gcp-pubsub` input pointed at the emulator.\n\n### Docker Compose pattern\n\n```yaml\nversion: '2.3'\nservices:\n gcppubsub-emulator:\n image: google/cloud-sdk:emulators\n command: gcloud beta emulators pubsub start --host-port=0.0.0.0:8681\n ports:\n - \"8681/tcp\"\n\n \u003cpackage>-\u003cstream>-publisher:\n image: docker.elastic.co/observability/stream:v0.22.0\n volumes:\n - ./sample_logs:/sample_logs:ro\n command:\n - log\n - --retry=30\n - --addr=gcppubsub-emulator:8681\n - -p=gcppubsub\n - --gcppubsub-clear=true\n - --gcppubsub-project=\u003cproject_id>\n - /sample_logs/\u003clogfile>.log\n depends_on:\n - gcppubsub-emulator\n```\n\n### Test config pattern\n\n```yaml\nwait_for_data_timeout: 1m\nservice: gcppubsub-emulator\ninput: gcp-pubsub\nvars:\n alternative_host: \"{{Hostname}}:{{Port}}\"\n credentials_json: >-\n {\"type\":\"service_account\",\"project_id\":\"\u003cproject_id>\"}\n project_id: \u003cproject_id>\n subscription_name: subscription\n topic: topic\n```\n\n### Key patterns\n\n- **`alternative_host`**: points the agent at the emulator instead of real GCP\n- **`--gcppubsub-clear=true`**: resets the emulator state before publishing\n- **`credentials_json`**: a stub service account credential sufficient for the emulator\n\n## Reference integrations\n\n- [`kafka_log`](https://github.com/elastic/integrations/tree/main/packages/kafka_log) — Kafka Kraft broker with stream producer\n- [`gcp_pubsub`](https://github.com/elastic/integrations/tree/main/packages/gcp_pubsub) — Pub/Sub emulator with stream publisher\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3835,"content_sha256":"972403f5f76c530e464d9f2a418beb25f214e205cf8b488cded891f94369a9d9"},{"filename":"references/system-testing-logfile.md","content":"# system testing — logfile/filestream input\n\nInput-specific guidance for system-testing data streams that use `logfile` or `filestream` inputs. Load `system-testing.md` (generic) first.\n\n## Overview\n\nLog file system tests use a lightweight Alpine container that copies sample log files into a shared volume (`SERVICE_LOGS_DIR`). The Elastic Agent then reads the files using the `logfile` or `filestream` input with glob paths.\n\n## Docker Compose pattern\n\n```yaml\nversion: '2.3'\nservices:\n \u003cpackage>-\u003cstream>-logfile:\n image: alpine\n volumes:\n - ./sample_logs:/sample_logs:ro\n - ${SERVICE_LOGS_DIR}:/var/log\n command: /bin/sh -c \"cp /sample_logs/* /var/log/\"\n```\n\nThe Alpine container runs `cp` to copy all sample log files into the `SERVICE_LOGS_DIR` volume, then exits. The Elastic Agent detects the new files and processes them.\n\nFor integrations that also support network inputs (TCP/UDP), combine both service types in the same docker-compose:\n\n```yaml\nversion: '2.3'\nservices:\n \u003cpackage>-\u003cstream>-logfile:\n image: alpine\n volumes:\n - ./sample_logs:/sample_logs:ro\n - ${SERVICE_LOGS_DIR}:/var/log\n command: /bin/sh -c \"cp /sample_logs/* /var/log/\"\n\n \u003cpackage>-\u003cstream>-tcp:\n image: docker.elastic.co/observability/stream:v0.22.0\n volumes:\n - ./sample_logs:/sample_logs:ro\n command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:\u003cport> -p=tcp /sample_logs/\u003clogfile>.log\n```\n\n## Test config pattern\n\n```yaml\nwait_for_data_timeout: 1m\nservice: \u003cpackage>-\u003cstream>-logfile\ninput: logfile\nvars:\n paths:\n - \"{{SERVICE_LOGS_DIR}}/*.log\"\n```\n\nWith additional options:\n\n```yaml\nwait_for_data_timeout: 1m\nservice: \u003cpackage>-\u003cstream>-logfile\ninput: logfile\ndata_stream:\n vars:\n preserve_original_event: true\nvars:\n paths:\n - \"{{SERVICE_LOGS_DIR}}/*\u003cstream>*.log\"\n tz_offset: \"+0500\"\n```\n\n## Key patterns\n\n- **`${SERVICE_LOGS_DIR}`**: environment variable populated by the test runner, points to a shared volume between the Alpine container and the Elastic Agent\n- **`{{SERVICE_LOGS_DIR}}`**: the Handlebars placeholder used in the test config (resolves to the same path at runtime)\n- **Alpine `cp` command**: the simplest approach — just copies files. The container exits after the copy, which is fine since the agent reads the files afterward.\n- **Glob paths**: use `{{SERVICE_LOGS_DIR}}/*.log` for all log files, or `{{SERVICE_LOGS_DIR}}/*\u003cname>*.log` to select specific files\n- **Sample log placement**: place log files in `_dev/deploy/docker/sample_logs/`\n- **No `service_notify_signal`**: unlike TCP/UDP, logfile tests do not need signal coordination — the agent detects files via filesystem polling\n- **`tz_offset`**: if the data stream manifest includes `tz_offset` for timezone handling, include it in test vars\n\n## Reference integrations\n\n- [`checkpoint`](https://github.com/elastic/integrations/tree/main/packages/checkpoint) — Alpine container pattern with TCP/UDP/TLS alongside\n- [`panw`](https://github.com/elastic/integrations/tree/main/packages/panw) — logfile with specific glob patterns\n- [`f5_bigip`](https://github.com/elastic/integrations/tree/main/packages/f5_bigip) — combines logfile (Alpine) and http_endpoint in one compose\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3233,"content_sha256":"445b034c79f04a53c26babfe221bbec08a8a5b793cb904c4db04d3bb8170b651"},{"filename":"references/system-testing-tcp-udp.md","content":"# system testing — TCP/UDP input\n\nInput-specific guidance for system-testing data streams that use `tcp` or `udp` inputs. Load `system-testing.md` (generic) first.\n\n## Overview\n\nTCP/UDP system tests use `elastic/stream` as a log sender that replays sample log files to the Elastic Agent's listening port. The `stream` tool sends lines over TCP, UDP, or TLS, coordinated with `service_notify_signal: SIGHUP` so the agent starts listening before the sender starts transmitting.\n\n## Docker Compose pattern\n\n```yaml\nversion: '2.3'\nservices:\n \u003cpackage>-\u003cstream>-tcp:\n image: docker.elastic.co/observability/stream:v0.22.0\n volumes:\n - ./sample_logs:/sample_logs:ro\n command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:\u003cport> -p=tcp /sample_logs/\u003clogfile>.log\n\n \u003cpackage>-\u003cstream>-udp:\n image: docker.elastic.co/observability/stream:v0.22.0\n volumes:\n - ./sample_logs:/sample_logs:ro\n command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:\u003cport> -p=udp /sample_logs/\u003clogfile>.log\n```\n\nFor TLS-over-TCP, use `-p=tls --insecure`:\n\n```yaml\n \u003cpackage>-\u003cstream>-tls:\n image: docker.elastic.co/observability/stream:v0.22.0\n volumes:\n - ./sample_logs:/sample_logs:ro\n command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:\u003cport> -p=tls --insecure /sample_logs/\u003clogfile>.log\n```\n\n## Test config pattern\n\n### TCP\n\n```yaml\nwait_for_data_timeout: 1m\nservice: \u003cpackage>-\u003cstream>-tcp\nservice_notify_signal: SIGHUP\ninput: tcp\ndata_stream:\n vars:\n listen_address: 0.0.0.0\n listen_port: \u003cport>\n preserve_original_event: true\nassert:\n hit_count: \u003cline_count>\n```\n\n### UDP\n\n```yaml\nwait_for_data_timeout: 1m\nservice: \u003cpackage>-\u003cstream>-udp\nservice_notify_signal: SIGHUP\ninput: udp\ndata_stream:\n vars:\n listen_address: 0.0.0.0\n listen_port: \u003cport>\nassert:\n hit_count: \u003cline_count>\n```\n\n## Key patterns\n\n- **`service_notify_signal: SIGHUP`**: required so the `stream` tool waits for the agent to be ready before sending. The `--start-signal=SIGHUP` flag on the `stream` command listens for this signal.\n- **`--delay=5s`**: adds a small delay after receiving the signal before sending, giving the agent time to fully initialize.\n- **Port alignment**: the port in `--addr elastic-agent:\u003cport>` must match `listen_port` in the test config. Use unique ports per service to avoid collisions.\n- **Sample log file**: place log files in `_dev/deploy/docker/sample_logs/`. Each line in the file becomes one event.\n- **`assert.hit_count`**: should match the number of non-empty lines in the sample log file.\n- **Multiple test configs**: create separate `test-tcp-config.yml` and `test-udp-config.yml` files to test both protocols against the same data stream.\n\n## Skipping tests\n\nIf sample logs are not yet available, a test config can be marked as skipped:\n\n```yaml\nservice: \u003cpackage>-\u003cstream>-tcp\nskip:\n reason: \"No sample logs available\"\n link: https://github.com/elastic/integrations/issues/\u003cnumber>\nservice_notify_signal: SIGHUP\ninput: tcp\ndata_stream:\n vars:\n listen_address: 0.0.0.0\n listen_port: \u003cport>\n```\n\n## Reference integrations\n\n- [`vectra_detect`](https://github.com/elastic/integrations/tree/main/packages/vectra_detect) — TCP, UDP, and TLS test configs\n- [`watchguard_firebox`](https://github.com/elastic/integrations/tree/main/packages/watchguard_firebox) — UDP with `assert.hit_count`\n- [`zscaler_zia`](https://github.com/elastic/integrations/tree/main/packages/zscaler_zia) — TCP across multiple data streams\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":3515,"content_sha256":"c724cfbde388954369461fdbc8e31b21668e289b672ec7f47085581e94dc4ca6"},{"filename":"references/system-testing.md","content":"# system testing\n\nEverything needed to set up, run, and debug `elastic-package test system` for full end-to-end ingest validation.\n\n## Purpose\n\nSystem tests validate the complete ingest path: service deployment → Elastic Agent policy → ingest pipeline → indexed documents in Elasticsearch. They also generate `sample_event.json`.\n\n## Required layout\n\nService deployment config (package-level or stream-level):\n- `_dev/deploy/docker/` — Docker Compose service definition\n\nSystem test configs (per data stream):\n- `data_stream/\u003cstream>/_dev/test/system/test-\u003cscenario>-config.yml`\n\nFor multi-version or environment variants:\n- `_dev/deploy/variants.yml`\n\n## System test config fields\n\n| Field | Description |\n|-------|-------------|\n| `wait_for_data_timeout` | Max time the test runner waits for expected documents in Elasticsearch. **Always set to `1m`** in every system test config. |\n| `input` | Select integration input when multiple exist (e.g. `tcp`, `http_endpoint`, `cel`) |\n| `service` | Maps test case to deploy service name |\n| `service_notify_signal` | Signal used when service config reload is needed |\n| `vars` | Package-level vars for integration policy |\n| `data_stream.vars` | Stream-level vars (e.g. `paths`, `hosts`, `listen_port`) |\n| `assert.hit_count` | Expected number of indexed events/documents |\n\nAvailable placeholders in config files:\n- `{{Hostname}}`\n- `{{Port}}`\n- `{{Ports}}` (or indexed forms like `{{Ports.0}}`)\n- `{{SERVICE_LOGS_DIR}}`\n\n## Core commands\n\n```bash\n# Start stack once\nelastic-package stack up -d\n\n# Run all system tests in current package\nelastic-package test system\n\n# Scope to specific streams\nelastic-package test system --data-streams \u003cstream1>[,\u003cstream2>]\n\n# Run one test config\nelastic-package test system --data-streams \u003cstream> --test-config test-default-config.yml\n\n# Run with a specific deploy variant\nelastic-package test system --variant \u003cvariant-name>\n\n# Generate sample_event.json\nelastic-package test system --generate\n\n# Keep resources around for debugging\nelastic-package test system --defer-cleanup 10m\n```\n\n## Input-type-specific guidance\n\nSystem test setup varies significantly by input type. Load the appropriate input-specific reference file alongside this generic reference:\n\n| Input type | Reference file |\n|------------|----------------|\n| `cel` | `system-testing-cel.md` |\n| `tcp`, `udp` | `system-testing-tcp-udp.md` |\n| `http_endpoint` | `system-testing-http-endpoint.md` |\n| `logfile`, `filestream` | `system-testing-logfile.md` |\n| `kafka`, `gcp-pubsub` | `system-testing-kafka-pubsub.md` |\n| `aws-s3`, `gcs`, `azure-blob-storage`, `azure-eventhub` | `system-testing-cloud-skip.md` |\n\nWhen an integration supports multiple input types, load the generic reference plus each applicable input-type reference.\n\n## Teardown failures (Fleet / agent policy conflicts)\n\nIf system test **teardown** fails with an error that an **agent policy** (or similar Fleet resource) is still in use and cannot be removed, **do not try to fix Fleet state by hand**.\n\nReset the whole local stack:\n\n```bash\nelastic-package stack down\nelastic-package stack up -d -v\n```\n\nThen rerun `elastic-package test system` (after `elastic-package build` if the package changed). This clears stale policies and agents tied to the previous run.\n\n## Verifying generated `sample_event.json`\n\n`--generate` produces `data_stream/\u003cstream>/sample_event.json` containing one representative event from the indexed results. This is a snapshot of current behavior, not a correctness guarantee.\n\nAfter generation, verify:\n1. Document shape matches expectations (correct ECS fields, correct nesting, expected values)\n2. Geo fields appear under the correct parent entity (`source.geo`, `destination.geo`, etc.) — not at document root\n3. Dotted field names from source data appear as properly nested objects\n4. If the sample event contains unexpected fields or missing values, fix the pipeline and regenerate — do not edit the file manually\n\n## Debugging system test failures — general\n\n### Check the agent log for dropped events\n\n```\nbuild/container-logs/elastic-agent-\u003cID>.log\n```\n\nIf the service container log shows successful responses but you still get 0 hits, check the agent log for:\n\n```\n\"events were dropped! Look at the event log to view the event and cause\"\n```\n\nThis means events were produced by the input and sent to Elasticsearch via bulk API, but **Elasticsearch rejected every document**.\n\n### Check the elastic-agent event log inside the running container\n\nWhen events are being dropped, the **only** way to see the actual rejection reason is the event log inside the running elastic-agent container:\n\n```\n/usr/share/elastic-agent/state/data/logs/events/elastic-agent-event-log-*.ndjson\n```\n\nTo inspect it during a running system test:\n\n1. **Increase `wait_for_data_timeout`** in the test config (e.g., to `5m`) so the system test stays running long enough to inspect the container.\n2. Start the system test and wait for the first \"events were dropped\" warnings.\n3. Find the elastic-agent container:\n ```bash\n docker ps --format '{{.Names}}' | grep elastic-agent\n ```\n4. Read the event log:\n ```bash\n docker exec \u003ccontainer-name> tail -20 /usr/share/elastic-agent/state/data/logs/events/elastic-agent-event-log-*.ndjson\n ```\n5. Look for entries with `\"Cannot index event\"` — these contain the **exact Elasticsearch rejection reason**.\n\nCommon rejection reasons:\n\n| Rejection | Root cause | Fix |\n|-----------|-----------|-----|\n| `mapper_parsing_exception` | Field type conflict (e.g. a field mapped as `keyword` receives an object) | Fix field definitions or pipeline |\n| `illegal_argument_exception` | Invalid field value (e.g. malformed IP, date parse failure) | Fix the ingest pipeline |\n| `Duplicate field '@timestamp'` | Input sets `@timestamp` in event output; framework also adds `@timestamp` → duplicate key | Remove `@timestamp` from event map — only set `{\"message\": ...}` |\n\nAfter diagnosing, reduce `wait_for_data_timeout` back to `1m` for the final test run.\n\n### Elasticsearch debugging (last resort)\n\nOnly if the above steps reveal nothing: check whether documents arrived but were routed incorrectly or failed in the pipeline. This is rarely needed — the container log and event log almost always reveal the root cause.\n\n## Common failure patterns\n\n- **\"field X is undefined\" for ECS fields** (e.g. `field \"destination.ip\" is undefined`):\n - Missing or outdated `_dev/build/build.yml`\n - Fix: add `dependencies.ecs.reference: \"[email protected]\"`\n- **Service not reachable**: wrong ports or placeholders in test config\n- **No events ingested**: wrong paths/hosts/input selection\n- **Mapping conflicts**: field type mismatches between pipeline output and field definitions\n- **Teardown fails**: agent policy still referenced — run `elastic-package stack down` then `elastic-package stack up -d -v` and rerun (see Teardown failures above)\n\n## Data anonymization\n\n**All test data must be fully anonymized before committing.** No real production data, customer data, or identifiable information may appear in system test sample logs (`_dev/deploy/docker/sample_logs/`), mock API response configs (`_dev/deploy/docker/files/`), test config files, or generated `sample_event.json`.\n\nReplace every identifying value with a synthetic example of the same format — IP addresses, hostnames, email addresses, usernames, organization names, account IDs, tokens, and any other value traceable to a real entity. Use RFC 5737 documentation IP ranges, `example.com` domains, and realistic placeholder names. Refer to the `anonymize-logs` skill for the full anonymization policy and placeholder conventions.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":7683,"content_sha256":"09fcb23fa6f3f14706ef2cf458558f4ec36cb14bb0832b2e26f6e995cb845fb3"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"integration-testing","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"When to use","type":"text"}]},{"type":"paragraph","content":[{"text":"Load this skill whenever tasks include:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"authoring or debugging pipeline test fixtures (","type":"text"},{"text":".log","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":".json","type":"text","marks":[{"type":"code_inline"}]},{"text":" inputs, ","type":"text"},{"text":"*-expected.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" output, config files)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"setting up or debugging system tests (","type":"text"},{"text":"_dev/test/system/","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"_dev/deploy/","type":"text","marks":[{"type":"code_inline"}]},{"text":", mock APIs, 0-hits failures)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"writing script tests for failure paths, API error handling, or package upgrades","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"When not to use","type":"text"}]},{"type":"paragraph","content":[{"text":"Do not use this skill as the primary guide for:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"ingest pipeline processor design and architecture (","type":"text"},{"text":"ingest-pipelines","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"CEL program development (","type":"text"},{"text":"cel-programs","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"broad elastic-package command selection and stack lifecycle (","type":"text"},{"text":"elastic-package-cli","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Reference files — load the one that matches your test type","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Test type","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Load when","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Reference file","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Pipeline tests","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Writing fixtures, config files, expected output, debugging pipeline test failures","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/pipeline-testing.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"System tests (generic)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Always load for any system test work — config fields, commands, teardown, debugging","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/system-testing.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Script tests","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"txtar failure/error tests, upgrade tests, mock services embedded in txtar","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/script-testing.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"System test input-specific references","type":"text"}]},{"type":"paragraph","content":[{"text":"In addition to the generic ","type":"text"},{"text":"system-testing.md","type":"text","marks":[{"type":"code_inline"}]},{"text":", load the reference file matching your data stream's input type:","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Input type","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Reference file","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"cel","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/system-testing-cel.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"tcp","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"udp","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/system-testing-tcp-udp.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"http_endpoint","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/system-testing-http-endpoint.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"logfile","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"filestream","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/system-testing-logfile.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"kafka","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"gcp-pubsub","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/system-testing-kafka-pubsub.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"aws-s3","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"gcs","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"azure-blob-storage","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"azure-eventhub","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"references/system-testing-cloud-skip.md","type":"text","marks":[{"type":"code_inline"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"When an integration supports multiple input types, load the generic reference plus each applicable input-type reference.","type":"text"}]},{"type":"paragraph","content":[{"text":"When working across multiple test types in one task (e.g. creating a new data stream end-to-end), load all applicable reference files.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"References","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/pipeline-testing.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" — directory layout, naming conventions, fixture formats, config options, expected output format and review, core workflow, fixture scenario coverage, data anonymization, troubleshooting","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/system-testing.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" — generic system test reference: required layout, config fields, core commands, teardown failures, ","type":"text"},{"text":"sample_event.json","type":"text","marks":[{"type":"code_inline"}]},{"text":" verification, general debugging","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/system-testing-cel.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" — CEL mock API wiring, 0-hits debugging for CEL, variable-capture patterns","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/system-testing-tcp-udp.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" — TCP/UDP log sender pattern with ","type":"text"},{"text":"elastic/stream","type":"text","marks":[{"type":"code_inline"}]},{"text":", signal coordination, port alignment","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/system-testing-http-endpoint.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" — webhook/HTTP endpoint testing with ","type":"text"},{"text":"STREAM_PROTOCOL=webhook","type":"text","marks":[{"type":"code_inline"}]},{"text":", auth headers","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/system-testing-logfile.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" — Alpine container + ","type":"text"},{"text":"SERVICE_LOGS_DIR","type":"text","marks":[{"type":"code_inline"}]},{"text":" pattern for logfile/filestream inputs","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/system-testing-kafka-pubsub.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" — Kafka broker + stream producer, Pub/Sub emulator patterns","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/system-testing-cloud-skip.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" — when and why to skip system tests for cloud storage inputs (aws-s3, gcs, azure-blob-storage, azure-eventhub)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/script-testing.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" — txtar format, env smoke test, system-level skeleton, mock service docker-compose and config.yml, upgrade test pattern, pitfalls, full o365 examples","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/builder-setup-subagent-guidance.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" — subagent operating manual for wiring data collection (docker-compose, sample logs, agent stream template, system test config, manifest var cleanup) for non-CEL data streams. The orchestrator dispatches subagents by passing this file's ","type":"text"},{"text":"path","type":"text","marks":[{"type":"strong"}]},{"text":" in the task prompt; the subagent reads it itself in its own fresh context. Do NOT embed/paste its contents into the task prompt.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"references/builder-system-test-subagent-guidance.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" — subagent operating manual for running ","type":"text"},{"text":"elastic-package test system --generate","type":"text","marks":[{"type":"code_inline"}]},{"text":" after pipeline work completes (any testable input). Same dispatch rule as above: orchestrators pass the path, the subagent reads the file itself.","type":"text"}]}]}]}]},"metadata":{"date":"2026-11-02","name":"integration-testing","author":"@skillopedia","source":{"stars":16,"repo_name":"integration-skills","origin_url":"https://github.com/elastic/integration-skills/blob/HEAD/skills/integration-testing/SKILL.md","repo_owner":"elastic","body_sha256":"3b07764d70d7deb1cdb5a2adc0b1743521a909d579696257a0ac28f4e2c9bda1","cluster_key":"dc0d0483034cbd261560312e4ebc246b5b9ac3e7ab1b0a1282c8db8355e209ab","clean_bundle":{"format":"clean-skill-bundle-v1","source":"elastic/integration-skills/skills/integration-testing/SKILL.md","text_bytes":84328,"source_commit":"39792cf81750ee54cba5b653fbfb69706871d42b","skills_sh_rank":19313,"attachment_count":11,"text_attachments":11,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"skills/integration-testing/SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"v1","category":"testing-qa","category_label":"Testing"},"exact_dupes_collapsed_into_this":0},"license":"Apache-2.0","version":"v1","category":"testing-qa","metadata":{"author":"elastic","version":"1.0"},"import_tag":"skills-sh-top20k-2026-11","description":"Use when creating, running, or debugging elastic-package tests — pipeline fixture authoring and expected output, system tests with mock API wiring, and script tests for failure paths and upgrades. Load the reference file for the test type you are working on."}},"renderedAt":1793661395910}

integration-testing When to use Load this skill whenever tasks include: - authoring or debugging pipeline test fixtures ( / inputs, output, config files) - setting up or debugging system tests ( , , mock APIs, 0-hits failures) - writing script tests for failure paths, API error handling, or package upgrades When not to use Do not use this skill as the primary guide for: - ingest pipeline processor design and architecture ( ) - CEL program development ( ) - broad elastic-package command selection and stack lifecycle ( ) Reference files — load the one that matches your test type | Test type | L…