configuring-exports — Skillopedia

<!-- TIER:1 -- Configuring Exports An export is the data source in a Celigo integration. It connects to an external system and pulls data into the pipeline. Exports serve two roles: - Source -- the starting point that fetches the primary batch of records - Lookup -- a mid-flow enrichment step ( ) that fetches additional data per-record during processing Both roles are used across flows, APIs, and tools. Beyond fetching data, exports also handle post-retrieval processing before records enter the pipeline: - Output filter -- expression-based filtering to skip records that don't match criteria -…

— the record root. An object record is addressed as\n '$.field'; a grouped (array) record as '$[0].field' / '$[*].field'\n - Used for precisely targeting data in structured JSON objects\n - Examples: '$.customer.firstName', '$.items[0].price', '$.addresses[*].street',\n '$[*].Tax' (every row of a grouped record)\n - Wildcards like [*] extract multiple values/objects\n - A key that contains a dot needs bracket notation: `$['P.O. Number']` (or\n `$[\"P.O. Number\"]`, `$.[P.O. Number]`). `$.P.O. Number` is read as a nested\n path: it maps nothing and the record still succeeds. Spaces, hyphens and `#`\n need no brackets (`$.Ship To Name`, `$.Tracking #`)\n - Filter expressions such as `$.items[?(@.active)]` fail the record with\n \"Eval [?(expr)] prevented in JSONPath expression.\"\n\n ```json\n \"extract\": \"$.customer.addresses[*]\" // Extracts all addresses\n ```\n\n **2. Handlebars Template Syntax**\n - Contains '{{' and '}}' pattern\n - Evaluated by the AFE 2.0 handlebars template engine\n - Can include logic, formatting, and computation\n - Access input record fields with {{record.fieldName}} notation; a grouped\n (array) record binds as `rows` instead — {{rows.0.fieldName}} / {{#each rows}}\n - Examples: \"{{record.firstName}} {{record.lastName}}\", \"{{#if record.isActive}}Active{{else}}Inactive{{/if}}\"\n - Valid on scalars and on `object` rows, where the rendered output must\n parse into an object. On an array row the template goes in a\n `buildArrayHelper` entry instead: on an `objectarray` row's own\n `extract` it is refused with 422 (\"buildArrayHelper\" or \"hardCodedValue\"\n must be present for object array datatype.), and on a primitive-array\n row's own `extract` it yields nothing (the field is left out) without an\n error\n\n ```json\n \"extract\": \"{{record.price}} {{record.currency}}\" // Combines two fields\n ```\n\n **3. Hard-Coded Value (literal string)**\n - Does NOT start with '

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

\n - Does NOT contain handlebars '{{' syntax\n - System treats it as a literal string value\n - VERY COMMON for setting static/constant values\n - Examples: \"Active\", \"USD\", \"Completed\", \"true\"\n\n ```json\n \"extract\": \"primary\" // Sets field value to the literal string \"primary\"\n \"extract\": \"true\" // Sets field value to the literal string \"true\"\n \"extract\": \"N/A\" // Sets field value to the literal string \"N/A\"\n ```\n\n This third pattern is the simplest and most efficient way to set hard-coded values in your mappings.\n AI agents should use this pattern whenever a field needs a static value that doesn't come from\n the input record or require computation.\n\n **Important implementation details**\n\n - JSON path patterns ALWAYS execute from the TOP-LEVEL root of the input record\n - The system maintains this context even in deeply nested mappings\n - For object mappings without child mappings, extract should return a complete object\n - An `object` row takes `extract` (to copy a whole object) or child `mappings`, not\n both: a row with both fails with 422 (`\"extract\" should not be present if\n \"mappings\" field is present`). A `buildArrayHelper` entry is different: there\n `extract` and `mappings` work together.\n - The root is the record as it arrives: an object record roots at `$.field`; a\n grouped record (an array of rows — grouped exports, file key columns, NetSuite\n grouped saved searches) roots at the array, so its rows are addressed as\n `$[0].field` (one row) or `$[*].field` (every row). A root that contradicts the\n record's shape resolves to nothing without an error.\n\n For most simple field-to-field mappings, prefer JSON path syntax for its clarity and performance.\n For hard-coded values, simply use the literal string as the extract value.\n x-celigo-ai-guidance:\n - |-\n ## GROUPED (ARRAY) RECORDS\n Decide the JSONPath root from the record's shape before writing\n any extract. When the record is an array of rows, `$.field`,\n `$.properties.field` and `$.rows[*].field` all resolve to\n nothing — the processor still reports success and the request\n body ships as `{}`. Address rows as `$[0].field` (that row's\n value, typed) or `$[*].field` (every row: joined with commas\n into a `string` destination; rejected with\n `incompatible_destination_conversion` into a `number`\n destination). Column labels with spaces work as\n `$[*][Order Number]`, `$[*].[Order Number]`, or\n `$[*]['Order Number']`. For one destination element per row\n the two array kinds take different helper forms: a primitive\n array (`stringarray` / `numberarray`) uses `buildArrayHelper:\n [{\"extract\": \"$[*].field\"}]`; an `objectarray` uses\n `buildArrayHelper: [{\"extract\": \"$[*]\", \"mappings\": [...]}]`\n with the children addressing the row relatively (`$.field`,\n `{{rows.field}}`). An `extract` placed directly on the\n array-typed field yields nothing. Handlebars\n extracts bind the array record as `rows` (`{{rows.0.field}}`),\n an object record as `record`. The reverse holds too: `$[*]` on\n an object record iterates the object's values, not rows.\n - |-\n ## SIZE LIMIT\n The platform's import save validator caps this field at **1024\n characters**. Saves with a longer `extract` are rejected\n with `[size_limit_exceeded] The maximum size of the field:\n mappings.\u003cindex>.extract in the Import document should be\n 1024 characters`. This typically only matters for\n Handlebars templates that build complex strings inline -- if\n a template is approaching the cap, prefer one of:\n\n - **Decompose into nested mappings** (set `dataType:\n \"object\"` and use child `mappings` -- each child gets\n its own 1024-char budget).\n - **Move the logic to a `preMap` hook** so the heavy\n transformation runs in script and the mapping just reads\n `record.\u003ccomputed>`.\n - **Use a lookup** (`lookupName` + a small `lookups` entry)\n for value translation that would otherwise need a long\n `{{#if ...}}{{else if ...}}` chain.\n examples: [\"$.fullName\", \"{{record.fullName}}\", \"primary\", \"$.lineItems[*].unitPrice\"]\n extractDateFormat:\n type: string\n description: |\n Specifies the format pattern of the input date string to ensure proper parsing.\n\n Used on string-typed mappings whose `extract` yields a date. Uses\n Moment.js-compatible formatting tokens to describe how the incoming date\n string is structured.\n examples: [\"YYYY-MM-DD\", \"MM/DD/YYYY\", \"DD-MMM-YYYY HH:mm:ss\"]\n extractDateTimezone:\n type: string\n description: |\n Specifies the timezone of the input date string using Olson/IANA timezone identifiers.\n\n Used on string-typed mappings whose `extract` yields a date; tells the system\n how to interpret timestamp values from the input system.\n examples: [\"UTC\", \"America/New_York\", \"Europe/London\"]\n generateDateFormat:\n type: string\n description: |\n Specifies the output format pattern when generating a date string or converting\n from a Date type to String type.\n\n Uses Moment.js-compatible formatting tokens to define the structure of the resulting\n date string.\n examples: [\"YYYY-MM-DD\", \"MM/DD/YYYY\", \"DD-MMM-YYYY HH:mm:ss\"]\n generateDateTimezone:\n type: string\n description: |\n Specifies the timezone to apply when generating or converting timestamp values\n using Olson/IANA timezone identifiers.\n\n Controls timezone conversion when producing date output.\n examples: [\"UTC\", \"America/New_York\", \"Europe/London\"]\n default:\n type: [string, 'null']\n description: |\n Specifies a fallback value to use when extract returns empty/null or when conditional\n logic fails and no other mapping supplies a value.\n\n Explicit JSON `null` is itself a valid fallback: the destination field is written as\n JSON null when the extract yields nothing (the Mapper UI's \"Use null as default value\"\n action). Omitting the key entirely means no fallback — the field is left out of the\n output when the extract is missing or an empty string (the UI's \"Do nothing\" action),\n while a null source is still written as null unless `conditional.when` is\n `extract_not_empty`. Failing the record instead is `conditional.when:\n fail_record_when_empty`, which excludes a default.\n x-celigo-ai-guidance:\n - |-\n `\"default\": null` and `\"default\": \"\"` are distinct saved values, not\n absent keys: null renders the destination field as JSON null and the\n empty string renders it as `\"\"` when the extract yields nothing.\n Preserve them verbatim when editing or migrating a mapping — dropping\n the key changes runtime behavior from \"use null/empty string as\n default value\" to \"do nothing\" (a missing or `\"\"` source is left out\n of the output, while a null source is written as null unless\n `conditional.when` is `extract_not_empty`).\n - |-\n The four \"Action to take if source field has no value\" choices map to:\n do nothing → no `default` key (add `conditional.when:\n extract_not_empty` to leave a null source out too; without it a null\n source is written as null); use null → `\"default\": null`; use empty string or\n a custom value → `\"default\": \"\u003cvalue>\"`; cause the record to fail →\n `conditional.when: fail_record_when_empty` with no `default` key.\n examples: [\"N/A\", \"0\", \"false\"]\n lookupName:\n type: string\n description: |\n **Purpose**\n References a lookup table for transforming values during the mapping process.\n\n **Usage**\n\n The lookupName refers to a named lookup defined in the lookups array of the same resource.\n On an HTTP import that is `http.lookups`: a `lookups` array sent only at the top level\n is dropped on save. On an import, a name that matches no lookup fails every record\n with \"Unable to find lookup in mapping configs - \u003cname>\", and a key the lookup\n doesn't hold follows the lookup's `allowFailures`: false fails the record; true, or\n the field left out, lets the record succeed with the lookup's `default`, or without\n the mapped field when there is none.\n\n ```json\n {\n \"generate\": \"countryName\",\n \"dataType\": \"string\",\n \"extract\": \"$.countryCode\",\n \"lookupName\": \"countryCodeToName\"\n }\n ```\n\n During processing, the system:\n 1. Extracts the value from the input record (e.g., \"US\")\n 2. Finds the lookup table with the specified name\n 3. Uses the extracted value as a key in the lookup\n 4. Returns the corresponding value (e.g., \"United States\")\n\n **Benefits**\n\n - **Standardization**: Ensures consistent value translation across mappings\n - **Centralization**: Define translations once and reference them in multiple places\n - **Maintainability**: Update all mappings by changing the lookup definition\n - **Readability**: Makes mappings more descriptive and self-documenting\n\n The specific lookup capabilities depend on the context where mappings are used.\n examples: [\"countryCodeToName\", \"statusMapping\", \"departmentCodes\"]\n hardCodedValue:\n type: string\n description: |\n Constant written to the destination instead of an extracted value. On a primitive\n array (`stringarray`, `numberarray`, `booleanarray`) the string is split on commas,\n verbatim and untrimmed, into elements of the array's type (`\"A,B\"` gives `[\"A\", \"B\"]`).\n On an `object` it must be a JSON object string and on an `objectarray` a JSON array\n string (an object string becomes one element); the save parses it and refuses non-JSON\n or the wrong shape with 422, and an array row with a constant needs no `buildArrayHelper`.\n examples: [\"USD\", \"A,B\", \"[{\\\"code\\\": \\\"A\\\"}, {\\\"code\\\": \\\"B\\\"}]\"]\n description:\n type: string\n description: |\n Optional free-text annotation that appears in the Mapper sidebar to provide context about\n the mapping's purpose for collaboration and documentation.\n\n Has no functional impact on the mapping behavior.\n examples: [\"Maps customer name\", \"Converts price to target currency\", \"Formats date for target system\"]\n sourceDataType:\n type: string\n description: |\n Declares the JSON type of the value produced by `extract`, before any\n conversion to `dataType`. Same enum as `dataType`. Set on leaf mappings\n only — parent mappings (with child `mappings` or `buildArrayHelper`)\n have no extracted value of their own; the children carry their own\n `sourceDataType`.\n\n For date fields use `string` (JSON represents dates as strings); the\n parsing/formatting lives in `extractDateFormat` / `generateDateFormat` /\n `extractDateTimezone` / `generateDateTimezone`.\n enum:\n - string\n - number\n - boolean\n - object\n - stringarray\n - numberarray\n - booleanarray\n - objectarray\n - arrayarray\n x-enumDescriptions:\n string: Source value is a text string.\n number: Source value is numeric.\n boolean: Source value is true or false.\n object: Source value is a JSON object.\n stringarray: Source value is an array of strings.\n numberarray: Source value is an array of numbers.\n booleanarray: Source value is an array of booleans.\n objectarray: Source value is an array of objects.\n arrayarray: Source value is an array of arrays.\n examples: [\"string\", \"number\", \"boolean\", \"object\"]\n mappings:\n type: array\n description: |\n **Purpose**\n Enables recursive definition of nested object structures through child mapping objects.\n\n **Guidance**\n This is the KEY FIELD that implements the recursive pattern at the core of this schema:\n\n **When to Use**\n - REQUIRED when dataType = \"object\" (unless you are copying an entire object from the input record)\n - REQUIRED in buildArrayHelper.mappings when defining complex object array elements\n - NEVER used with simple types (string, number, boolean)\n\n **Behavior**\n - Each mapping in this array becomes a property of the parent object\n - The full Mappings schema is repeated recursively at each level\n - Can be nested to any depth for complex hierarchical structures\n\n **Context Handling**\n - Each level of nesting changes the mapping CONTEXT for 'generate'\n - The extraction CONTEXT always remains the original input record\n - This means child mappings can pull data from anywhere in the input record\n\n **Common Patterns**\n\n **Nested Objects**\n ```json\n {\n \"generate\": \"customer\",\n \"dataType\": \"object\",\n \"mappings\": [\n {\n \"generate\": \"contact\",\n \"dataType\": \"object\",\n \"mappings\": [\n {\"generate\": \"email\", \"dataType\": \"string\", \"extract\": \"$.customerEmail\"}\n ]\n }\n ]\n }\n ```\n\n **Multiple Fields in Object**\n ```json\n {\n \"generate\": \"address\",\n \"dataType\": \"object\",\n \"mappings\": [\n {\"generate\": \"street\", \"dataType\": \"string\", \"extract\": \"$.address.line1\"},\n {\"generate\": \"city\", \"dataType\": \"string\", \"extract\": \"$.address.city\"},\n {\"generate\": \"country\", \"dataType\": \"string\", \"extract\": \"$.address.country\"}\n ]\n }\n ```\n\n IMPORTANT: When analyzing or generating mappings, ALWAYS check if parent.dataType = \"object\"\n or if you're inside buildArrayHelper.mappings for objectarray elements. These are the only\n valid contexts for the mappings array.\n items:\n $ref: '#/Mappings/items'\n examples: [[{\"generate\": \"firstName\", \"extract\": \"$.name.first\", \"dataType\": \"string\", \"status\": \"Active\"}, {\"generate\": \"lastName\", \"extract\": \"$.name.last\", \"dataType\": \"string\", \"status\": \"Active\"}]]\n buildArrayHelper:\n type: array\n x-celigo-ai-guidance:\n - |-\n Entry-level `extract` accepts an iterating array path\n (`$.items[*]`), `\"$\"` (the record root), or omission — a\n constructed entry (child `mappings` with the extract omitted or\n `\"$\"`; the two spellings are equivalent) contributes exactly one\n element, and entries append in order. On a grouped (array)\n record the record itself is the array to iterate: `\"$[*]\"`\n yields one `objectarray` element per row (children address the\n row as `$.field`), and `\"$[*].field\"` yields one primitive-array\n value per row. Indexed or comma-joined\n entry extracts (`\"$[0]\"`, `\"$[0],$[0]\"`) are not valid: the\n processor renders the entire destination array as null with no\n validation error. To emit a fixed number of static elements, add\n that many sibling constructed entries — never index the extract.\n - |-\n An entry's `extract` can be a Handlebars template. On an\n `objectarray` it must render a JSON object (one element) or a JSON\n array (one element per item). On a primitive array each entry\n renders one element, so a serialized array becomes one string\n element. A bare literal entry extract is refused with 422 (\"extract\n only supports json path or handlebars expression\"); set a constant\n with the row's `hardCodedValue` instead.\n description: |\n **Purpose**\n Configures how to construct arrays in the output record, handling various array types and inputs.\n\n **Guidance**\n This is the mechanism for building array data types:\n\n **When to Use**\n - REQUIRED when dataType ends with \"array\" (stringarray, objectarray, etc.), unless the\n row sets `hardCodedValue`: an array row's own `extract` is never read (see `dataType`)\n - Each entry in this array contributes elements to the output array\n - Multiple entries allow combining data from different input arrays\n\n **Array Type Handling**\n\n **For Simple Arrays (stringarray, numberarray, booleanarray)**\n - Only the 'extract' field is used to pull values\n - JSON path with wildcards (e.g., $.items[*].name) returns multiple values;\n on a grouped (array) record the record itself is the array, so the\n path roots at it ($[*].name — one value per row)\n - Each result is converted to the appropriate primitive type\n ```json\n {\n \"generate\": \"productNames\",\n \"dataType\": \"stringarray\",\n \"buildArrayHelper\": [\n {\"extract\": \"$.products[*].name\"}\n ]\n }\n ```\n\n **For Object Arrays (objectarray) - three patterns**\n\n 1. Extract Only (existing objects):\n ```json\n {\n \"generate\": \"contacts\",\n \"dataType\": \"objectarray\",\n \"buildArrayHelper\": [\n {\"extract\": \"$.account.primaryContacts[*]\"}, // Pull primary contact objects\n {\"extract\": \"$.account.secondaryContacts[*]\"}, // Pull secondary contact objects\n {\"extract\": \"$.vendor.contactPersons[*]\"}, // Pull vendor contact objects\n {\"extract\": \"$.subsidiaries[*].mainContact\"} // Pull main contact from each subsidiary\n ]\n }\n ```\n\n 2. Mappings Only (constructed object):\n ```json\n {\n \"generate\": \"contactInfo\",\n \"dataType\": \"objectarray\",\n \"buildArrayHelper\": [\n {\n \"mappings\": [ // Creates one object in the array\n {\"generate\": \"type\", \"dataType\": \"string\", \"extract\": \"primary\"},\n {\"generate\": \"email\", \"dataType\": \"string\", \"extract\": \"$.primaryEmail\"}\n ]\n },\n {\n \"mappings\": [ // Creates another object in the array\n {\"generate\": \"type\", \"dataType\": \"string\", \"extract\": \"secondary\"},\n {\"generate\": \"email\", \"dataType\": \"string\", \"extract\": \"$.secondaryEmail\"}\n ]\n }\n ]\n }\n ```\n Each constructed entry contributes exactly one element, in entry\n order; writing the entry with `\"extract\": \"$\"` is equivalent to\n omitting the extract (both anchor the element at the record root).\n A fixed number of static elements is built from that many sibling\n constructed entries.\n\n 3. Extract AND Mappings (transform input arrays):\n ```json\n {\n \"generate\": \"lineItems\",\n \"dataType\": \"objectarray\",\n \"buildArrayHelper\": [\n {\n \"extract\": \"$.order.items[*]\", // For each item in the array\n \"mappings\": [ // Transform to this structure using the composite object\n {\"generate\": \"sku\", \"dataType\": \"string\", \"extract\": \"$.order.items.productId\"}, // Notice: items is singular\n {\"generate\": \"quantity\", \"dataType\": \"number\", \"extract\": \"$.order.items.qty\"}, // Notice: items is singular\n {\"generate\": \"orderNumber\", \"dataType\": \"string\", \"extract\": \"$.order.id\"}, // Access parent data\n {\"generate\": \"customerName\", \"dataType\": \"string\", \"extract\": \"$.customerName\"} // Access root data\n ]\n }\n ]\n }\n ```\n\n **Arrays of Arrays**\n - The mapper has no working type for them: `arrayarray` saves, but every record\n fails with `Destination datatype is a required field.` Build an array of arrays\n in a script hook instead.\n\n **Important details**\n\n - When both extract and mappings are provided, the system creates special composite objects\n that maintain hierarchical context during processing\n - This enables accessing both the current array element AND its parent context\n - An ITERATING entry's extract MUST be a JSON path that iterates an array:\n '$.items[*]' when the array is a field of an object record, or '$[*]'\n (objectarray) / '$[*].field' (primitive arrays) when the record itself is a\n grouped array of rows. A CONSTRUCTED entry (child mappings building one\n element) instead omits extract or sets it to exactly \"$\" — never rewrite \"$\"\n into an indexed or comma-joined form ('$[0]', '$[0],$[0]'): those render the\n entire destination array as null without raising a validation error\n - Each array helper entry acts independently, potentially adding multiple elements\n\n The buildArrayHelper is the most complex part of the mappings system - always analyze the\n dataType first to understand which pattern is appropriate.\n items:\n type: object\n properties:\n extract:\n type: string\n description: |\n JSON path expression that identifies the input array or values to extract.\n\n For objectarray with mappings, this defines which input objects to iterate through.\n The JSON path must return either a single object or an array of objects.\n\n For a CONSTRUCTED entry (child mappings building exactly one element from\n record-root fields and constants), omit this field or set it to exactly \"$\" —\n the two spellings are equivalent. Indexed or comma-joined forms ('$[0]',\n '$[0],$[0]') are not valid at the entry level: the processor renders the\n entire destination array as null without raising a validation error.\n\n The system creates special composite objects during processing to maintain\n hierarchical relationships, allowing easy access to both the current array item\n and its parent contexts.\n\n An iterating entry keeps the `[*]`: `$.line_items` without it yields one empty\n element per item, with no error. Child extracts drop the `[*]` but keep the\n array's path (`$.line_items.sku`); `$.line_items[*].sku` maps nothing, and an\n element-relative `$.sku` reads the record root instead of the element.\n examples: [\"$.items[*]\", \"$.order.lineItems[*]\", \"$.parents[*].children[*]\", \"$\"]\n name:\n type: string\n maxLength: 200\n description: |\n Display label for this source object in the mapper, so the sources of\n an object array can be told apart instead of all showing as ` configuring-exports — Skillopedia .\n Has no effect at runtime; stored only when non-empty.\n x-celigo-ai-guidance:\n - |-\n Carry an existing `name` through every edit — it is the user's label\n for the tab, and an entry sent without it loses the label. Duplicate\n names across entries are allowed. Set one only when the user names\n the source object; never invent labels.\n examples: [\"Billing address\", \"Shipping address\"]\n sourceDataType:\n type: string\n description: |\n Declares the JSON type of the input array being iterated, to ensure\n proper type handling during array construction. Same enum as `dataType`.\n enum:\n - string\n - number\n - boolean\n - object\n - stringarray\n - numberarray\n - booleanarray\n - objectarray\n - arrayarray\n x-enumDescriptions:\n string: Input array contains string elements.\n number: Input array contains numeric elements.\n boolean: Input array contains boolean elements.\n object: Input array contains object elements.\n stringarray: Input array contains arrays of strings.\n numberarray: Input array contains arrays of numbers.\n booleanarray: Input array contains arrays of booleans.\n objectarray: Input array contains arrays of objects.\n arrayarray: Input array contains arrays of arrays.\n examples: [\"string\", \"number\", \"boolean\", \"object\"]\n default:\n type: [string, 'null']\n description: |\n Specifies a fallback value when the extracted array element is empty or\n not found in the input data.\n\n Explicit JSON `null` is itself a valid fallback (the element is written as\n JSON null); omitting the key means no fallback is applied.\n examples: [\"N/A\", \"0\", \"false\"]\n conditional:\n type: object\n description: |\n Defines conditional rules for including each array element in the result.\n x-celigo-ai-guidance:\n - |-\n Judged against this entry's own `extract` (the array source). The\n design scopes the UI's tab-level \"Action to take if source field has\n no value\" to copy-as-is tabs (no child `mappings`); the API accepts\n `fail_record_when_empty` on a mapped entry as well, and the mapper\n processor honors it there — the record fails when that array is\n missing or empty (`errors[]` entry `code: extract_empty`, `fatal:\n true`, message `\u003cgenerate>: Source field has no value`; probed\n 2026-10-09). Say so when writing it on a mapped entry: the user may\n not find the setting in the mapper's tab. To fail on an empty field\n inside each element, set the action on that child mapping instead.\n properties:\n when:\n type: string\n description: |\n Condition judged on the entry's extract — skip the elements the\n extract does not return, or fail the whole record when the source\n array is missing or empty.\n enum:\n - extract_not_empty\n - fail_record_when_empty\n x-enumDescriptions:\n extract_not_empty: Include an array element only when the extract returns a non-empty value for it\n fail_record_when_empty: Fail the record when the entry's source array is missing or empty\n examples: [\"extract_not_empty\"]\n mappings:\n type: array\n description: |\n Contains recursive mapping definitions for complex array element transformations.\n\n **Composite object mechanism**\n\n When both 'extract' and 'mappings' are used together, the system implements a sophisticated\n \"composite object\" approach that is crucial for AI agents to understand:\n\n 1. The system starts with the complete input record\n\n 2. For each array element matched by the extract path, it creates a modified version of\n the input record where:\n - Array paths in the extract JSON path are REPLACED with single objects\n - Each array ([]) in the path is converted to a single object ({})\n - This preserves the hierarchical relationship between nested arrays\n\n **Example**\n\n Given an input record:\n ```json\n {\n \"customer\": {\n \"name\": \"John Doe\",\n \"orders\": [\n {\n \"id\": \"O-001\",\n \"items\": [\n {\"sku\": \"ABC\", \"qty\": 2},\n {\"sku\": \"XYZ\", \"qty\": 1}\n ]\n },\n {\n \"id\": \"O-002\",\n \"items\": [\n {\"sku\": \"DEF\", \"qty\": 3}\n ]\n }\n ]\n }\n }\n ```\n\n With extract path: `$.customer.orders[*].items[*]`\n\n For each item, the system creates a composite object like:\n ```json\n {\n \"customer\": {\n \"name\": \"John Doe\",\n \"orders\": { // Note: Array replaced with single object\n \"id\": \"O-001\",\n \"items\": { // Note: Array replaced with single object\n \"sku\": \"ABC\",\n \"qty\": 2\n }\n }\n }\n }\n ```\n\n Then in your mappings, you can access:\n - The current item: `$.customer.orders.items.sku`\n - The parent order: `$.customer.orders.id`\n - Top-level data: `$.customer.name`\n\n This approach allows for precise mapping from deeply nested structures while maintaining\n access to all contextual parent data, without requiring complex array index management.\n\n **Implementation guidance**\n\n When implementing the composite object mechanism:\n\n 1. Analyze the extract path to identify all array patterns (`[*]` or `[number]`)\n 2. For each array in the path, understand that it will be replaced with a single object\n 3. In the mappings, use paths that reference these arrays as if they were objects\n 4. Remember that every mapping still has access to the full input record context\n 5. This mechanism is especially powerful when mapping hierarchical data like:\n - Order → Line Items → Taxes/Discounts\n - Customer → Addresses → Address Lines\n - Invoice → Line Items → Serial Numbers\n\n The extract path effectively tells the system \"iterate through these arrays\",\n while the composite object mechanism ensures you can still access both the\n current array item AND its parent context during mapping.\n items:\n $ref: '#/Mappings/items'\n examples: [[{\"generate\": \"itemName\", \"extract\": \"$.name\", \"dataType\": \"string\", \"status\": \"Active\"}, {\"generate\": \"itemPrice\", \"extract\": \"$.price\", \"dataType\": \"number\", \"status\": \"Active\"}]]\n # The save rejects the fail-record action paired with a default on an\n # entry ([unsupported_mapping]); the entry schema has no lookupName or\n # hardCodedValue, so this is the only pair at this level.\n if:\n properties:\n conditional:\n properties:\n when:\n const: fail_record_when_empty\n required: [when]\n required: [conditional]\n then:\n not:\n required: [default]\n status:\n type: string\n description: |\n **Purpose**\n Controls whether the mapping is applied. Send it on every mapping entry.\n\n **Guidance**\n Only `Active` rows run. `Draft` and `Inactive` rows keep their\n configuration, are saved with minimal validation, and are skipped at\n runtime and in preview; a non-Active parent skips its whole subtree\n whatever its children say. Imports (on save and in the mapper processor)\n and API response mappings reject a row missing this field (validation\n error: \"Mapping object must have status field present.\"); a Tool's\n output mapping saves and runs one, which is why `required` doesn't\n list it.\n x-celigo-ai-guidance:\n - |-\n Emit `\"Active\"` for mappings that should run; `\"Draft\"` is also\n accepted and saves the mapping while bypassing the Active-only\n field validations. The value is case-sensitive. The API rejects:\n - Missing field: `[status_field_absent] Mapping object must have status field present.`\n - Wrong case (e.g. `\"active\"`): `[invalid_status_enums] active is not a valid enum for status field.`\n - |-\n `\"Inactive\"` is the user's \"Set inactive\" on a row: use it only when\n asked to disable or park a row, never to mean incomplete (that is\n `\"Draft\"`). Setting a parent `Inactive` means no `Active` row may\n remain under it — the save rejects an `Active` row under an\n `Inactive` parent (`[inactive_parent_active_child]` on `mappings`,\n no message text) and never cascades or promotes on its own; `Draft`\n descendants are kept as they are, and a `Draft` parent does not gate\n its children. The UI's Set inactive on a parent marks the parent and\n every child `Inactive`. Re-enabling a child\n means re-enabling its ancestors in the same payload. Edits that keep\n existing rows must carry their stored status forward — rewriting a\n `Draft` or `Inactive` row to `Active` turns a row the user parked\n back on.\n enum:\n - Active\n - Draft\n - Inactive\n x-enumDescriptions:\n Active: The mapping is enabled and applied during processing\n Draft: In-progress mapping; saved without the Active-only field validations and skipped at runtime\n Inactive: Disabled by the user; configuration kept, skipped at runtime and in preview\n examples: [\"Active\"]\n conditional:\n type: object\n description: |\n **Purpose**\n Defines conditional processing rules for the entire mapping.\n\n **Guidance**\n These conditions determine whether the mapping is applied based on record\n state or field content:\n\n **When to Use**\n - When a mapping should only be applied in specific circumstances\n - To implement conditional logic without using complex handlebars expressions\n - For creating mappings that only run during create or update operations\n\n **Available Conditions**\n\n - `record_created`: Apply only when creating a new record\n Useful for setting initial values that should not be overwritten during updates\n\n - `record_updated`: Apply only when updating an existing record\n Useful for transformation logic that should only run during updates\n\n - `extract_not_empty`: Apply only when the extract field returns a value\n Useful for conditional mapping based on input data availability\n\n - `fail_record_when_empty`: Fail the record when the extract field returns\n no value (the UI's \"Cause record to fail\" action when the source field\n has no value)\n\n **Example**\n ```json\n {\n \"generate\": \"statusMessage\",\n \"dataType\": \"string\",\n \"status\": \"Active\",\n \"extract\": \"$.status.message\",\n \"conditional\": {\n \"when\": \"extract_not_empty\" // Only map when status.message exists\n }\n }\n ```\n x-celigo-ai-guidance:\n - |-\n `when` holds one value, and two different settings share it: the\n composite-import operation gate (`record_created` / `record_updated`,\n the UI's \"Only perform mapping when\") and the empty-source action\n (`extract_not_empty` / `fail_record_when_empty`, the UI's \"Action to\n take if source field has no value\"). A row cannot carry both — writing\n one replaces the other and the save succeeds, so a composite import\n row that needs both has to choose. Never author `fail_record_when_empty`\n on a row that already carries `record_created` or `record_updated`.\n properties:\n when:\n type: string\n description: |\n Condition that gates this mapping — either an operation gate on a\n composite (create-or-update) import or the action to take when the\n extract returns no value. Empty means null, undefined or an empty\n string; `0`, `false` and whitespace are values.\n x-celigo-ai-guidance:\n - |-\n `fail_record_when_empty` is the fourth \"Action to take if source field\n has no value\" and is mutually exclusive with the other three: send it\n with no `default` key. A `default`, a `lookupName` or a `hardCodedValue`\n on the same row — or a `default` on a `buildArrayHelper` entry — is\n rejected at save with `[unsupported_mapping] \"\u003ckey>\" and conditional.when\n \"fail_record_when_empty\" cannot be present in mapping simultaneously.`\n (the lookup variant adds `Use lookup \"allowFailures\" instead.`);\n lookups fail through `allowFailures: false`, and a hard-coded row\n always has a value. At runtime an empty extract fails the record with\n an `errors[]` entry `{code: \"extract_empty\", fatal: true, source:\n \"mapping\", message: \"\u003cgenerate>: Source field has no value\"}`\n (\"Input field has no value\" in a Transformation 2.0); `0` and `false`\n are values and map through.\n enum:\n - record_created\n - record_updated\n - extract_not_empty\n - fail_record_when_empty\n x-enumDescriptions:\n record_created: Apply this mapping only when creating a new record\n record_updated: Apply this mapping only when updating an existing record\n extract_not_empty: Apply this mapping only when the extract field returns a non-empty value\n fail_record_when_empty: Fail the record when the extract field returns no value\n examples: [\"record_created\", \"record_updated\", \"extract_not_empty\", \"fail_record_when_empty\"]\n required:\n - dataType\n if:\n properties:\n conditional:\n properties:\n when:\n const: fail_record_when_empty\n required: [when]\n required: [conditional]\n then:\n not:\n anyOf:\n - required: [default]\n - required: [lookupName]\n - required: [hardCodedValue]\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":52991,"content_sha256":"52bc801b81ac5f934dda6ea0e194361e1aad6592840f911cf948db2fbb4ca41d"},{"filename":"references/common-schemas/one-to-many.yml","content":"OneToMany:\n type: boolean\n description: |\n When true, the step runs once per child record instead of once per incoming record.\n `pathToMany` names the array field that holds the children inside an object record; when\n the incoming record is itself an array (grouped or row-based data), leave `pathToMany` blank\n and each element becomes a record. The fan-out is scoped to this step: afterwards the children\n are re-joined into the record's original shape — the object with its array, or the array of rows —\n carrying any response-mapping enrichment, and that re-joined record is what the next step\n receives. Not for locating records in an export's HTTP response; use\n `http.response.resourcePath` for that. Applies to steps that receive a record — imports and\n lookups (an export referenced as a page processor of a flow, API, or Tool). A standalone export\n (a flow's page generator) has no incoming record, so the fields are saved but have no effect on\n the records it produces; to emit one record per array element from a standalone export, use a\n `hooks.preSavePage` script, or for a file-definition export make the repeating segment the\n record boundary in the definition's rules.\n x-celigo-ai-guidance:\n - |-\n Only imports and lookups (exports placed as page processors) fan out. On a standalone export\n (a flow's page generator) the API saves ``oneToMany`` / ``pathToMany`` and applies neither:\n the records reach the next step unchanged and no error is raised. Live-verified 2026-09-18 on\n a flow test run — an order record with three ``lines`` reached the import as ONE record with\n the setting on the export, and as THREE child records with the same setting on the import.\n Never propose these fields as the way to split a standalone export's records; that is\n ``hooks.preSavePage`` work (or, for a file definition, the definition's record boundary).\n The rules below apply only on a step that receives a record.\n - |-\n Two modes, decided by the SHAPE of the record entering the step. Object record with a child\n array under a field → set ``pathToMany`` to that field. Array record (a ``groupByFields``\n export, a CSV/file group, a script that returns an array) → leave ``pathToMany`` blank; each\n element becomes one record. Live-verified 2026-09-10 on flows, Tools, and APIs.\n - |-\n Blank ``pathToMany`` on an OBJECT record fails every record with code ``oneToMany_error``:\n \"Failed to extract child records from oneToMany import data. Record at index 0 was not an\n array, and no PathToMany was provided\". Builder previews return only the code, with no\n message. The fix is never \"turn oneToMany off\" by default — first determine whether the\n record arriving at the step is an object (set a path) or an array (blank is correct; a\n preview error on an array-fed step is the preview panel re-applying one-to-many to the\n already-fanned-out record — verify with a test run or a real run).\n - |-\n Child record shape differs by mode. Path mode: each child is the array element plus\n ``_PARENT`` (the parent record minus the array). Blank/row mode: each child is the element\n itself, with no ``_PARENT``. In both modes templates on the step reference the child as\n ``{{record.\u003cfield>}}``; a response mapping on a one-to-many lookup merges into each child and\n the downstream step receives the re-joined parent (path mode) or the re-joined array (row\n mode) with every element enriched.\n - |-\n One-to-many never batches by itself: ``batch_of_records`` appears as the HTTP body template\n root only when ``http.batchSize`` is greater than 1 (children are batched after fan-out).\n With the default batch size the template root is ``record``.\n default: false\n examples: [true, false]\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":3900,"content_sha256":"9c57c8003be6b14dd6d1f0f1320ddf2a1cf1bc95f496b5cb968d599408650b15"},{"filename":"references/common-schemas/parsers.yml","content":"Parsers:\n type: array\n description: |\n Configuration for parsing XML payloads (i.e. files, HTTP responses, etc.). Use this field when you need to process XML data\n and transform it into a JSON records.\n\n **Implementation notes**\n\n - This is where you configure how to parse XML data in your resource\n - Although defined as an array, you typically only need a single parser configuration\n - Currently only XML parsing is supported\n - Only configure this field when working with XML data that needs structured parsing\n items:\n type: object\n properties:\n version:\n type: string\n description: |\n Version identifier for the parser configuration format. Currently only version \"1\" is supported.\n\n Always set this field to \"1\" as it's the only supported version at this time.\n enum: [\"1\"]\n x-enumDescriptions:\n \"1\": Current and only supported parser configuration version.\n examples: [\"1\"]\n type:\n type: string\n description: |\n Defines the type of parser to use. Currently only \"xml\" is supported.\n\n While the system is designed to potentially support multiple parser types in the future,\n at this time only XML parsing is implemented, so this field must be set to \"xml\".\n enum: [\"xml\"]\n x-enumDescriptions:\n xml: Parses XML payloads into JSON records.\n examples: [\"xml\"]\n name:\n type: string\n description: |\n Optional identifier for the parser configuration. This field is primarily for documentation\n purposes and is not functionally used by the system.\n\n This field can be omitted in most cases as it's not required for parser functionality.\n examples: [\"XMLCustomParser\", \"SpecializedXMLParser\"]\n rules:\n type: object\n description: |\n Configuration rules that determine how XML data is parsed and converted to JSON.\n These settings control the structure and format of the resulting JSON records.\n\n **Parsing options**\n\n There are two main parsing strategies available:\n - **Automatic parsing**: Simple but produces more complex output\n - **Custom parsing**: More control over the resulting JSON structure\n x-celigo-ai-guidance:\n - |-\n With V0_json false, an empty element (`\u003cFirstName>\u003c/FirstName>`, `\u003cFirstName/>`) parses to\n an empty object `{}` under every setting here, not an empty string. A Handlebars template\n that outputs the field renders `[object Object]`; guard it with\n `{{#if (eq (typeOf FirstName) \"string\")}}{{FirstName}}{{/if}}` — a bare `{{#if FirstName}}`\n does not skip it because `{}` is truthy.\n properties:\n V0_json:\n type: boolean\n description: |\n Controls the XML parsing strategy.\n\n - When set to **true** (Automatic): XML data is automatically converted to JSON without\n additional configuration. This is simpler to set up but typically produces more complex\n and deeply nested JSON that may be harder to work with.\n\n - When set to **false** (Custom): Gives you more control over how the XML is converted to JSON.\n This requires additional configuration (like listNodes) but produces cleaner, more\n predictable JSON output.\n\n Most implementations use the Custom approach (false) for better control over the output format.\n examples: [false, true]\n listNodes:\n type: array\n description: |\n Paths of elements that always parse as arrays. An element that can repeat parses as an\n object when it occurs once and as an array when it occurs more than once, so list every\n repeating element here to give each record the same shape regardless of item count.\n Each entry is the element's full path from the record element selected by the resource\n path (from the document root when there is no resource path), with an optional leading\n slash. Only relevant when V0_json is set to false (Custom parsing).\n items:\n type: string\n examples: [\n [\"/product-lineitems/product-lineitem\",\n \"/shipping-lineitems/shipping-lineitem\",\n \"/shipments/shipment\"]\n ]\n x-celigo-ai-guidance:\n - |-\n Add every element that can repeat. Left out, a one-item record parses it as an object:\n a Handlebars `{{#each}}` over it iterates the object's fields (each field reference\n renders empty) and mappings or filters written against an array find nothing.\n - |-\n Write the full path from the record element: `Items/LineItemInfo` for records at\n resource path `/Orders/Order`. A bare descendant name (`LineItemInfo`) or a\n document-rooted path (`/Orders/Order/Items/LineItemInfo`) matches nothing while a\n resource path is set; without a resource path the path must start at the document\n root. `//Element` descendant paths never match.\n includeNodes:\n type: array\n description: |\n Limits which XML nodes are included in the output JSON.\n\n For large XML documents, you can use this field to extract only the nodes you need,\n reducing the size and complexity of the resulting JSON. Only nodes specified here\n (and their children) will be included in the output.\n\n Each entry should be a simplified XPath expression pointing to nodes to include.\n\n Only relevant when V0_json is set to false (Custom parsing).\n items:\n type: string\n examples: [[\"/order/customer\", \"/order/items\", \"/order/shipping\"]]\n excludeNodes:\n type: array\n description: |\n Specifies which XML nodes should be excluded from the output JSON.\n\n Sometimes it's easier to specify which nodes to exclude rather than which to include.\n Use this field to identify nodes that should be omitted from the output JSON.\n\n Each entry should be a simplified XPath expression pointing to nodes to exclude.\n\n Only relevant when V0_json is set to false (Custom parsing).\n items:\n type: string\n examples: [[\"/order/metadata\", \"/order/system-info\", \"/order/audit-trail\"]]\n stripNewLineChars:\n type: boolean\n description: |\n Controls whether newline characters are removed from text values.\n\n When set to true, all newline characters (\\n, \\r, etc.) will be removed from\n text content in the XML before conversion to JSON.\n default: false\n examples: [false, true]\n trimSpaces:\n type: boolean\n description: |\n Controls whether leading and trailing whitespace is trimmed from text values.\n\n When set to true, all values will have leading and trailing whitespace removed\n before conversion to JSON.\n default: false\n examples: [false, true]\n attributePrefix:\n type: string\n description: |\n Specifies a character sequence to prepend to XML attribute names when converted to JSON properties.\n\n In XML, both elements and attributes can exist at the same level, but in JSON this distinction is lost.\n To maintain the distinction between element data and attribute data in the resulting JSON, this prefix\n is added to attribute names during conversion.\n\n For example, with attributePrefix set to \"Att-\" and an XML element like:\n ```xml\n \u003cproduct id=\"123\">Laptop\u003c/product>\n ```\n\n The resulting JSON would be:\n ```json\n {\n \"product\": \"Laptop\",\n \"Att-id\": \"123\"\n }\n ```\n\n This helps maintain the distinction between element content and attribute values in the\n converted JSON, making it easier to reference specific data in downstream processing steps.\n examples: [\"Att-\", \"attrib_\", \"@\", \"_\"]\n textNodeName:\n type: string\n description: |\n Specifies the property name to use for element text content when an element has both\n text content and child elements or attributes.\n\n When an XML element contains both text content and other nested elements or attributes,\n this field determines what property name will hold the text content in the resulting JSON.\n\n For example, with textNodeName set to \"value\" and an XML element like:\n ```xml\n \u003citem id=\"123\">\n Laptop\n \u003ccategory>Electronics\u003c/category>\n \u003c/item>\n ```\n\n The resulting JSON would be:\n ```json\n {\n \"item\": {\n \"value\": \"Laptop\",\n \"category\": \"Electronics\",\n \"id\": \"123\"\n }\n }\n ```\n\n This allows for unambiguous parsing of complex XML structures that mix text content with\n child elements. Choose a name that's unlikely to conflict with actual element names in your XML.\n examples: [\"txt\", \"value\", \"text\", \"#text\"]","content_type":"application/yaml; charset=utf-8","language":"yaml","size":9766,"content_sha256":"05ba3821a5f4c8ec97f65cf46c084860604a7e98aa04ae16606ab819aab197c7"},{"filename":"references/common-schemas/path-to-many.yml","content":"PathToMany:\n type: string\n description: |\n Path to the array of child records inside an object record when `oneToMany` is true, in dot\n notation (`items`, `lines.lineItems`). Leave blank when the incoming record is itself an array\n (grouped or row-based data) — each element is then a child record. A path that does not resolve\n to an array processes zero records and reports success. Read only on steps that receive a\n record (imports, lookup exports); on a standalone export it is saved and ignored — see `oneToMany`.\n x-celigo-ai-guidance:\n - |-\n Inert on a standalone export (a flow's page generator) exactly like ``oneToMany`` — the value\n is saved, the export's records reach the next step unchanged, and nothing errors\n (live-verified 2026-09-18). Splitting a standalone export's records into one per element is\n ``hooks.preSavePage`` work, or the file definition's record boundary; never this field.\n - |-\n Set this ONLY when the record entering the step is an object whose children sit under a field.\n On an array record (a ``groupByFields`` export, a CSV/file group, a script returning an array)\n any path resolves to nothing: the step processes ZERO children, builds no request, and reports\n success with no error. Live-verified 2026-09-10 — the failure is silent, so confirm the arriving\n record's shape (test run stage data, or the flow builder's input panel: ``rows`` means array).\n - |-\n Blank with an object record is the ``oneToMany_error`` failure (\"Record at index 0 was not an\n array, and no PathToMany was provided\"); blank with an array record is the correct row-based\n configuration. An empty string behaves exactly like an absent field.\n - |-\n Use the bare field name or dotted path (``lineItems``, ``details.items``), not a JSONPath\n expression. Each child in path mode carries ``_PARENT`` (the parent minus the array), so\n parent-level fields are read as ``{{record._PARENT.\u003cfield>}}``.\n examples: [\"items\", \"line_items\", \"orderLines\", \"lines.lineItems\"]\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":2095,"content_sha256":"d2e78377a11ab6f901a8834c658d80da27216888de4fe915e592546b2737b938"},{"filename":"references/common-schemas/pre-save.yml","content":"PreSave:\n type: object\n description: |\n Defines a JavaScript hook that executes before the resource is saved.\n\n This hook allows for programmatic validation, transformation, or enrichment of the\n resource itself before it is persisted. It can be used to enforce business rules,\n set derived properties, or implement cross-field validations that can't be expressed\n through the standard UI.\n\n **Function signature**\n\n The preSave function is invoked with a single 'options' argument containing:\n ```javascript\n function preSave(options) {\n // Process options and return the modified resource\n return options.newResource;\n }\n ```\n\n **Available context**\n\n The 'options' argument provides access to:\n - `options.newResource` - The resource being saved (with pending changes)\n - `options.oldResource` - The previous version of the resource (before changes)\n - `options.sandbox` - Boolean flag indicating whether the script is running in a sandbox environment\n\n\n **Common uses**\n\n - Enforcing complex business rules across multiple fields\n - Automatically deriving field values based on other configuration\n - Performing validation that depends on external systems or data\n - Normalizing or standardizing configuration values\n - Adding computed or derived properties\n - Implementing versioning or change tracking\n - Dynamically looking up data using the Celigo API module to enrich configuration\n\n **Return value**\n\n The function must return the newResource object (potentially modified) to be saved.\n Throwing an exception will prevent saving and signal an error to the user.\n properties:\n function:\n type: string\n description: |\n The name of the function to execute within the referenced script.\n\n This property specifies which function to invoke from the script\n referenced by _scriptId. The function will be called just before\n the resource is saved.\n\n The function must follow the expected signature and return the resource object.\n examples: [\"preSave\", \"validateResource\", \"prepareResourceForSave\"]\n _scriptId:\n type: string\n format: objectId\n x-celigo-refModel: scripts\n description: |\n Reference to a predefined script resource containing the preSave function.\n\n The referenced script should contain the function specified in the\n 'function' property. This script must be accessible within the user's account\n and have appropriate permissions.\n examples: [\"60a2c4e6f321d800129a1a3c\", \"5f8e7d2c1b54e6003a9c7e12\"]","content_type":"application/yaml; charset=utf-8","language":"yaml","size":2618,"content_sha256":"60e1b1de4be3c2404f72c6701b986109a3c3aaabf69b27dba14820cf519a361f"},{"filename":"references/common-schemas/resource-response.yml","content":"ResourceResponse:\n type: object\n description: Response\n x-celigo-ai-guidance:\n - Core response fields shared by all Celigo resources\n properties:\n _id:\n type: string\n format: objectId\n readOnly: true\n description: Unique identifier for the resource. Format is a 24-character hexadecimal string.\n x-celigo-ai-guidance:\n - |-\n The _id is used in:\n - API endpoints that operate on a specific resource (e.g., GET, PUT, DELETE)\n - References from other resources (e.g., flows that use this resource)\n - Job history and error tracking\n examples: [\"5f8d43a1b9e5a80011a35f2c\"]\n createdAt:\n type: string\n format: date-time\n readOnly: true\n description: Timestamp when the resource was created. Set automatically and cannot be modified.\n x-celigo-ai-guidance:\n - Recorded in ISO 8601 with a UTC timezone (Z suffix).\n examples: [\"2023-04-01T09:15:32Z\"]\n lastModified:\n type: string\n format: date-time\n readOnly: true\n description: Timestamp when the resource was last updated. Changes whenever any property is modified.\n x-celigo-ai-guidance:\n - Recorded in ISO 8601 with a UTC timezone (Z suffix); always equal to or later than `createdAt`.\n - |-\n Records that something was written, not what — a settings edit, a flag flip, or a\n platform workflow all move it. What changed and when is in the audit log: the\n per-resource route `GET /v1/{resourceType}/{_id}/audit` for the types its\n `resourceType` enum lists, or the account-wide\n `GET /v1/audit?resourceType=\u003csingular>&_resourceId={_id}` for the further types that\n endpoint's enum accepts (users, environments, notifications among them). A type absent\n from both enums — jobs, connectors, templates — has only this timestamp.\n examples: [\"2023-04-15T14:30:15Z\"]\n deletedAt:\n type: [string, 'null']\n format: date-time\n readOnly: true\n description: Timestamp when the resource was soft-deleted. When null or absent, the resource is active.\n x-celigo-ai-guidance:\n - Non-null means the resource is soft-deleted (in the recycle bin) — recoverable within the retention period before permanent removal.\n examples: [\"2023-05-20T11:45:32Z\"]\n required:\n - _id\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":2380,"content_sha256":"545364158eb63d4407ea79b5957cc357b55411ee852257bb240a60d88741bb70"},{"filename":"references/common-schemas/settings.yml","content":"Settings:\n type: object\n description: |\n Configuration settings that can be accessed by hooks, filters, mappings and handlebars templates at runtime.\n\n It enables customization of the resource's logic, allowing hooks, mappings, filters, and\n handlebars to access and apply the settings at runtime.\n\n **Usage**\n\n The settings object can store arbitrary JSON data that you want to save with the resource.\n While it's often populated through a form defined in the `settingsForm` field, you can also:\n\n - Directly provide JSON settings without using a form\n - Store configuration values used by hooks and templates\n - Create resource-specific constants and parameters\n - Maintain lookup tables or mapping structures\n - Define conditional logic parameters\n\n **Accessibility**\n\n Settings are available in:\n - All handlebars fields for building dynamic payloads\n - Field mapping expressions\n - JavaScript hooks via the options object\n - Filters and transformations\n\n At runtime, the settings objects in the step's execution chain are gathered into a\n single `settings` context keyed by fixed scope keys. Flow runs populate the full\n chain: `settings.integration.*`, `settings.flowGrouping.*` (the flow group the flow\n belongs to), `settings.flow.*`, `settings.connection.*`, `settings.iClient.*`, and\n the running step's own scope — `settings.export.*` on exports/lookups,\n `settings.import.*` on imports (the key matches the step type). Steps executing\n inside a My API or a Tool receive only the step's own scope plus\n `settings.connection.*` / `settings.iClient.*`; the integration, flowGrouping, and\n flow scopes resolve empty there. The scope key is literal — resource display names\n and settingsForm section labels are never part of the path. The segments after the\n scope mirror the stored settings JSON key path exactly: a top-level field is\n `settings.\u003cscope>.\u003cfieldId>`, and nested objects add one segment per JSON key\n (e.g. `settings.flowGrouping.Customer.region` when the group's settings JSON nests\n `region` under `Customer`). References without a scope key\n (e.g. `{{settings.myField}}`) resolve to empty strings.\n\n **Best practices**\n\n For non-technical users, create a custom form instead of editing the JSON directly.\n This provides a user-friendly interface for updating settings without requiring JSON knowledge.\n additionalProperties: true\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":2467,"content_sha256":"7fb3eb881109b6912af008c054c5d5f237fdb1c36430c16e3e43d18114bee12b"},{"filename":"references/common-schemas/transform.yml","content":"Transform:\n type: object\n description: |\n Configuration for transforming data during processing operations. This object enables\n reshaping of records.\n\n **Transformation capabilities**\n\n Celigo's transformation engine offers powerful features for data manipulation:\n - Precise field mapping with JSONPath expressions\n - Support for any level of nested arrays\n - Formula-based field value generation\n - Dynamic references to flow and integration settings\n\n **Implementation approaches**\n\n There are two distinct transformation mechanisms available:\n\n **Rule-Based Transformation (`type: \"expression\"`)**\n - **Best For**: Most transformation scenarios from simple to complex\n - **Capabilities**: Field mapping, formula calculations, lookups, nested data handling\n - **Advantages**: Visual configuration, no coding required, intuitive interface\n - **Configuration**: Define rules in the `expression` object\n - **Use When**: You have clear mapping requirements or need to reshape data structure\n\n **Script-Based Transformation (`type: \"script\"`)**\n - **Best For**: Extremely complex logic or proprietary algorithms\n - **Capabilities**: Full programmatic control, custom processing, complex business rules\n - **Advantages**: Maximum flexibility, can implement any transformation logic\n - **Configuration**: Reference a script in the `script` object\n - **Use When**: Visual transformation tools aren't sufficient for your use case\n properties:\n type:\n type: string\n description: |\n Determines which transformation mechanism to use. This choice affects which properties\n must be configured and how transformation logic is implemented.\n\n **Available types**\n\n **Rule-Based Transformation (`\"expression\"`)**\n - **Required Config**: The `expression` object with mapping definitions\n - **Behavior**: Applies declarative rules to reshape data\n - **Best For**: Most transformation scenarios from simple to complex\n - **Advantages**: Visual configuration, no coding required\n\n **Script-Based Transformation (`\"script\"`)**\n - **Required Config**: The `script` object with _scriptId and function\n - **Behavior**: Executes custom JavaScript to transform data\n - **Best For**: Extremely complex logic or proprietary algorithms\n - **Advantages**: Maximum flexibility, can implement any logic\n\n **Implementation guidance**\n\n 1. For standard data transformations, use `\"expression\"`\n 2. For complex logic or specialized processing, use `\"script\"`\n 3. When selecting a type, you must configure the corresponding object:\n - `type: \"expression\"` requires the `expression` object\n - `type: \"script\"` requires the `script` object\n enum: [\"expression\", \"script\"]\n x-enumDescriptions:\n expression: Rule-based transformation using declarative mapping definitions.\n script: Script-based transformation using custom JavaScript logic.\n examples: [\"expression\", \"script\"]\n expression:\n type: object\n description: |\n Configuration for declarative rule-based transformations. This object enables reshaping data\n without requiring custom code.\n\n **Usage context**\n\n This object is REQUIRED when `transform.type` is set to \"expression\" and should not be\n configured otherwise. It provides a standardized way to define transformation rules that\n can map, modify, and generate data elements.\n\n **Implementation guidance**\n\n The expression system uses a rule-based approach where:\n - Field mappings define how input data is transformed to target fields\n - Formulas can be used to calculate or generate new values\n - Lookups can enrich data by fetching related information\n - Mode determines how records are processed (create new or modify existing)\n properties:\n version:\n type: string\n description: |\n Version of the expression format. Determines which rules\n property contains the transformation logic.\n enum: [\"1\", \"2\"]\n x-enumDescriptions:\n \"1\": \"Legacy format. Rules are in the `rules` array (array of arrays of extract/generate objects).\"\n \"2\": \"Current format. Rules are in the `rulesTwoDotZero` object with mode, mappings, and lookups.\"\n examples: [\"1\", \"2\"]\n rules:\n type: array\n description: |\n Transformation rules for version 1 expressions. An array of\n rule groups; each group is an array of field-mapping objects.\n Most transforms have a single group. Present when `version`\n is `\"1\"`. The output record contains ONLY the generated\n fields — every unmapped field is dropped (v1 has no\n equivalent of Transform 2.0's `modify` mode), and the\n record's trace key does not survive the rebuild.\n x-celigo-ai-guidance:\n - |-\n v1 rules REPLACE the record: a one-field rule group drops\n every other field, silently breaking downstream filters,\n router branch rules, and templates that referenced the\n dropped fields. Either map every field downstream logic\n needs forward, or author Transform 2.0 with\n ``mode: \"modify\"`` instead.\n - |-\n An import's ``responseTransform`` is the exception: it accepts\n version ``\"1\"`` only (``\"2\"`` is refused with 422, verified live\n 2026-10-21), so there, map every response field that response\n mapping needs.\n items:\n type: array\n items:\n type: object\n properties:\n extract:\n type: string\n description: |\n Source field path to read from. Supports multiple\n syntaxes: bare field names (`id`), dot notation\n (`fulfillment.shipment_id`), slash-prefixed paths\n for XML (`/FeedProcessingStatus`), wildcards (`*.id`,\n `*.[Internal ID]`), and array indexing (`SDF[0]`).\n generate:\n type: string\n description: |\n Target field name to write to. Typically a bare name\n (`id`) or dot path (`SDF.Filter.ID`).\n key:\n type: string\n description: |\n Auto-generated identifier for this rule, used by the\n UI to track individual rules for editing and reordering.\n required:\n - extract\n - generate\n rulesTwoDotZero:\n type: object\n description: |\n Configuration for version 2 transformation rules. This object contains the core logic\n for how data is mapped, enriched, and transformed.\n\n **Capabilities**\n\n Transformation 2.0 provides:\n - Precise field mapping with JSONPath expressions\n - Support for deeply nested data structures\n - Formula-based field generation\n - Dynamic lookups for data enrichment\n - Multiple operating modes to fit different scenarios\n properties:\n mode:\n type: string\n description: |\n Transformation mode that determines how records are handled during processing.\n\n **Available modes**\n\n **Create Mode (`\"create\"`)**\n - **Behavior**: Builds entirely new output records from inputs\n - **Use When**: Output structure differs significantly from input\n - **Advantage**: Clean slate approach, no field inheritance\n\n **Modify Mode (`\"modify\"`)**\n - **Behavior**: Makes targeted edits to existing records\n - **Use When**: Output structure should remain similar to input\n - **Advantage**: Preserves unmapped fields from the original record\n enum: [\"create\", \"modify\"]\n x-enumDescriptions:\n create: Builds entirely new output records from inputs, ignoring unmapped fields.\n modify: Makes targeted edits to existing records, preserving unmapped fields.\n x-celigo-ai-guidance:\n - |-\n On a grouped record (the export groups rows, so the record is an\n array) only `create` can author output — build it from the rows with\n `$[0].field`, `$[*].field`, or a `buildArrayHelper` over `$[*]`.\n `modify` has no object to overlay onto an array: the record passes\n through untouched and the processor reports success. Either mode\n reshapes the inside of one record; neither changes how many records\n the page carries (that is the export's `preSavePage` hook) or whether\n the export groups at all (that is its grouping setting).\n examples: [\"create\", \"modify\"]\n mappings:\n $ref: './mappings.yml#/Mappings'\n lookups:\n allOf:\n - description: |\n Shared lookup tables used across all mappings defined in the transformation rules.\n\n **Purpose**\n\n Lookups provide centralized value translation that can be referenced from any mapping\n in your transformation configuration. They enable consistent translation of codes, IDs,\n and values between systems without duplicating translation logic.\n\n **Usage in transformations**\n\n Lookups are particularly valuable in transformations for:\n\n - **Data Normalization**: Standardizing values from diverse source systems\n - **Code Translation**: Converting between different coding systems (e.g., status codes)\n - **Field Enrichment**: Adding descriptive values based on ID or code lookups\n - **Cross-Reference Resolution**: Mapping identifiers between integrated systems\n\n **Implementation**\n\n Lookups are defined once in this array and referenced by name in mappings:\n\n ```json\n \"lookups\": [\n {\n \"name\": \"statusMapping\",\n \"map\": {\n \"A\": \"Active\",\n \"I\": \"Inactive\",\n \"P\": \"Pending\"\n },\n \"default\": \"Unknown Status\"\n }\n ]\n ```\n\n Then referenced in mappings using the lookupName property:\n\n ```json\n {\n \"generate\": \"status\",\n \"dataType\": \"string\",\n \"extract\": \"$.statusCode\",\n \"lookupName\": \"statusMapping\"\n }\n ```\n\n The system automatically applies the lookup during transformation processing.\n\n For complete details on lookup properties and behavior, see the Lookups schema.\n - $ref: './lookups.yml#/Lookups'\n inputContext:\n type: string\n enum:\n - record\n - envelope\n x-enumDescriptions:\n record: The transform processor evaluates `mappings[].extract` JSONPath against the flat record itself (`$.field`, no `record.` segment) — the default when absent.\n envelope: The transform processor evaluates JSONPath against the wrapped runtime envelope `{record, job, testMode, settings, headers, queryParams}` (file-provider steps also carry `fileMeta`) — record fields need the `record.` segment (`$.record.field`), while `headers` and `queryParams` sit beside `record` (`$.headers['X-Id']`, `$.queryParams.src`).\n description: |\n Controls the JSON shape the transformTwoDotZero processor\n evaluates `mappings[].extract` JSONPath values against at\n flow runtime. Applies only to Transform 2.0 (v2,\n `rulesTwoDotZero`); v1 transforms (the `rules` array on\n `transform.expression.rules`) and script-mode transforms\n ignore this field.\n x-celigo-ai-guidance:\n - |-\n ## Extract anchoring by mode\n\n - `record` (default — absent / empty / undefined): the\n processor sees the FLAT record as its evaluation root.\n `mappings[].extract` JSONPath MUST anchor directly at\n ` configuring-exports — Skillopedia with NO `record.` segment — e.g. `\"$.orderId\"`,\n `\"$.customer.email\"`; PLACEHOLDER extracts use\n `\"$.PLACEHOLDER_\u003cfield>\"`.\n This is the dominant Celigo idiom and the right choice for new transforms.\n - `envelope`: the processor sees a wrapped object\n `{record: {...}, job, testMode, settings, headers,\n queryParams}` (plus `fileMeta` on file-provider steps).\n `mappings[].extract` JSONPath MUST include the\n `record.` segment for record fields — e.g.\n `\"$.record.orderId\"` — and reads `headers` and\n `queryParams` beside it (`$.headers['X-Id']`);\n PLACEHOLDER extracts use `\"$.record.PLACEHOLDER_\u003cfield>\"`.\n Pick this only when the transform was authored against\n the envelope shape on purpose.\n - |-\n Handlebars expressions inside `extract`\n (`{{record.field}}`, `{{dateFormat ... record.field}}`)\n ALWAYS reference the `record` root regardless of\n `inputContext` — only JSONPath anchoring flips.\n - |-\n In `envelope` mode the output is still a bare record: a\n `modify` transform returns the record with its mapped\n fields merged in (no `record` wrapper, no `job` or\n `settings` keys), and a `create` transform returns only\n the mapped fields.\n - |-\n Emit `inputContext` explicitly on new v2 transforms:\n unset is read as `record`, but stating it makes the\n extract-anchoring contract self-documenting and immune\n to a platform default change.\n - |-\n ## Allowed values\n\n - ``record`` (default — emitted when the field is absent\n / empty / undefined): the transform processor sees\n the FLAT record itself as its evaluation root.\n ``mappings[].extract`` JSONPath values MUST anchor\n directly at `` configuring-exports — Skillopedia ` with NO ``record.`` segment — for\n example ``\"$.orderId\"`` or\n ``\"$.customer.email\"``. PLACEHOLDER extracts use\n ``\"$.PLACEHOLDER_\u003cfield>\"``. This is the dominant\n Celigo idiom and the right choice for new transforms.\n\n - ``envelope``: the transform processor sees a wrapped\n object of the shape\n ``{record: {...}, job, testMode, settings, headers,\n queryParams}`` (plus ``fileMeta`` on file-provider\n steps; ``headers`` and ``queryParams`` sit beside\n ``record``) — the\n same runtime-context envelope Handlebars / SQL /\n GraphQL templates render against. ``mappings[].extract``\n JSONPath values for record fields MUST include the\n ``record.`` segment — for example ``\"$.record.orderId\"``\n or ``\"$.record.customer.email\"`` — while ``headers`` and\n ``queryParams`` are read beside it\n (``\"$.headers['X-Id']\"``). PLACEHOLDER extracts\n use ``\"$.record.PLACEHOLDER_\u003cfield>\"``. Pick this\n mode only when the export was authored against the\n envelope shape on purpose (typically because an\n upstream step intentionally feeds the wrapped\n runtime context through to the transform).\n\n ## Handlebars expressions are NOT affected\n\n Handlebars expressions inside ``extract``\n (``{{record.field}}``,\n ``{{dateFormat ... record.field}}``, etc.) ALWAYS\n reference the runtime context's ``record`` root\n regardless of ``inputContext`` — the Handlebars runtime\n always sees the wrapped envelope. Only JSONPath\n behavior flips with this field.\n\n ## Agent guidance\n\n Set this field explicitly on every new v2 transform\n the agent authors. Leaving it unset is interpreted as\n ``record`` by the platform, but emitting\n ``inputContext: record`` makes the contract\n self-documenting and prevents the transform from\n drifting into envelope mode if an upstream change ever\n changes the platform default. Read this value before\n writing ``mappings[].extract`` so the JSONPath form\n matches the runtime root.\n - |-\n ## fileMeta on file-provider steps\n\n File-provider exports and lookups populate a\n ``fileMeta`` envelope key describing each record's\n source file: ``fileName``, ``fileSize``, and\n ``lastModifiedTime`` (epoch milliseconds). Reach it\n as ``$.fileMeta.\u003cfield>`` in envelope mode, or as\n ``{{fileMeta.\u003cfield>}}`` from a Handlebars extract in\n either mode. The keys are exactly these three —\n lowercase variants such as ``filesize`` or\n ``filename`` resolve empty. The standalone transform\n processor substitutes sample fileMeta values when the\n posted data lacks them, so a resolving editor render\n is not proof the step provides fileMeta — only\n file-provider steps populate it at flow runtime.\n examples: [\"record\", \"envelope\"]\n script:\n type: object\n description: |\n Configuration for programmable script-based transformations. This object enables complex, custom\n transformation logic beyond what expression-based transformations can provide.\n\n **Usage context**\n\n This object is REQUIRED when `transform.type` is set to \"script\" and should not be configured\n otherwise. It provides a way to execute custom JavaScript code to transform data according to\n specialized business rules or complex algorithms.\n\n **Implementation approach**\n\n Script-based transformation works by:\n 1. Executing the specified function from the referenced script\n 2. Passing input data to the function\n 3. Using the function's return value as the transformed output\n\n **Common use cases**\n\n Script transformation is ideal for:\n - Complex business logic that can't be expressed through mappings\n - Algorithmic transformations requiring computation\n - Dynamic transformations based on external factors\n - Legacy system data format compatibility\n - Multi-stage processing with intermediate steps\n\n Only use script-based transformation when expression-based transformation is insufficient.\n Script transformation requires maintaining custom code, which adds complexity to the integration.\n properties:\n _scriptId:\n type: string\n description: |\n Reference to a predefined script resource containing the transformation logic.\n\n The referenced script should contain the function specified in the\n 'function' property. It must be a script in this account: any other id fails the save\n with 422 (\"Only a script owned by the user or a script shared by the Integration App can\n be used at transform.script._scriptId.\").\n format: objectid\n x-celigo-refModel: scripts\n examples: [\"60a2c4e6f321d800129a1a3c\", \"5f8d43a1b9e5a80011a35f2c\"]\n function:\n type: string\n description: |\n Name of the function within the script to execute for transformation. This function\n must exist in the script referenced by _scriptId.\n examples: [\"transformData\", \"convertRecord\", \"applyMapping\"]\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":21239,"content_sha256":"0f2a1bdcf3fd4c5a271d5fc97ba73ce416e3c9a25c3767407d9d0eac9738578f"},{"filename":"references/netsuite.md","content":"# Diagnosing NetSuite Steps\n\nHow to configure NetSuite exports, imports and connections is in [configuring-exports](../SKILL.md), [configuring-imports](../../configuring-imports/SKILL.md) and [configuring-connections](../../configuring-connections/SKILL.md), with their schemas. This reference is for a NetSuite step that fails, stalls or returns nothing: the checks that reach into the NetSuite account, the errors NetSuite steps return, and the traps that fail without an error.\n\n- **CLI:** `celigo connections ping`, `celigo connections suiteapp-status`, `celigo metadata types|fields`, plus the run and error commands in [troubleshooting-flows](../../troubleshooting-flows/SKILL.md).\n- **MCP:** the deployed server has no tool for a connection ping, the SuiteApp check or metadata. Run them with the CLI, or through the REST API: `GET /v1/connections/{_id}/ping`, `GET /v1/connections/{_id}/distributed` and `GET /v1/connections/{_id}/distributedApps`.\n\n## Check the NetSuite Side First\n\nWhen every NetSuite step on a connection fails, or a listener or a NetSuite DA import never gets anywhere, check the account before the step:\n\n```bash\nceligo connections ping \u003cconnectionId> # credentials and reachability; answers 200 either way, so read the body\nceligo connections ping \u003cconnectionId> --distributed # the Celigo SuiteApp or bundle RESTlet in the account\nceligo connections suiteapp-status \u003cconnectionId> # which of the SuiteApp and the bundle are installed, and their versions\n```\n\n- **The distributed ping** is a live round trip to the Celigo SuiteApp or bundle RESTlet that NetSuite listeners and NetSuite DA imports run through. It takes a few seconds and returns `{success, statusCode, resBody, bundleURL}`. Run it before building either.\n- **`suiteapp-status`** reports the SuiteApp and the legacy bundle separately (`{bundle, suiteapp}`). Trust it over the connection's `netsuite.suiteAppInstalled`, which only records what was detected the last time the connection authenticated.\n\n## Errors on NetSuite Steps\n\n| Error | Where | What it means | What to do |\n|---|---|---|---|\n| HTTP 400 `SSS_REQUEST_LIMIT_EXCEEDED` (RESTlets), HTTP 429 (REST web services), `ExceededConcurrentRequestLimitFault` or `WS_REQUEST_BLOCKED` (SOAP), \"Only one request may be made against a session at a time\" | Any step on the NetSuite account | The account's concurrency limit is spent. Every connection to the account and every caller outside Celigo share it, and NetSuite refuses all of them at once, so a step can fail with nothing changed on its own connection. On RESTlet and SOAP calls the platform retries `SSS_REQUEST_LIMIT_EXCEEDED`, `WS_REQUEST_BLOCKED` and `WS_CONCUR_SESSION_DISALLWD` once a second, for up to about five minutes by default, before the step fails, so a run can sit still that long first | Total the ceilings of every connection to that account and keep them, with the outside callers, within its limit, or have the connections borrow from one parent ([Connections as concurrency queues](../../configuring-connections/SKILL.md#connections-as-concurrency-queues----throughput-and-governance)) |\n| 403 `FEATURE_DISABLED` \"The 'REST Web Services' feature is not enabled\" | Every SuiteTalk REST and SuiteQL call | NetSuite's REST Web Services feature is off | A NetSuite administrator enables it (Setup > Company > Enable Features > SuiteCloud); custom RESTlets don't need it ([gotcha 12](../SKILL.md#gotchas)) |\n| `sss_invalid_srch_operator` | A saved-search or lookup criterion | A numeric operator (`equalto`, `greaterthan`, `between` and the like) on a text column: the platform turns a templated `searchValue` into a number for them | Compare text and checkbox columns with `is` / `isnot`, and list or select columns with `anyof` / `noneof` and an array `searchValue` |\n| `[search_not_sorted]` | An ad-hoc RESTlet search (criteria and columns, no saved search) | Grouping needs the search sorted by internal id | Set `netsuite.skipGrouping: true` ([gotcha 6](../SKILL.md#gotchas)) |\n| `STATIC_MAPPING_ERROR` \"Exported value is empty/undefined for the field: $.\u003cpath>\" | A NetSuite DA lookup set with `searchField` and `operator` | The record's value for the search is empty | Fill the value upstream, or set `allowFailures: true` with a `default` |\n| `VALUE_LOOKUP_FAILED` | A NetSuite DA lookup set with an `expression` | No match, or more than one | [configuring-imports > Common Errors](../../configuring-imports/SKILL.md#common-errors); a single match can be wrong too (below) |\n| `Failed to create, update/transform record because \u003cmessage>` or `Failed to save record because \u003cmessage>` | A NetSuite DA import | The Celigo SuiteApp or bundle puts NetSuite's refusal in one of these sentences and keeps NetSuite's code. *create, update/transform* means NetSuite refused while the record was being created, loaded or transformed and its fields set, before any save; *save* means NetSuite refused the save itself. `because null` means NetSuite's error carried no message | Read the code and the text after `because` as NetSuite's error. With `null`, read the calls the SuiteApp made for that record ([below](#seeing-what-a-da-import-asked-of-netsuite)) |\n| `UNEXPECTED_ERROR`, followed by `Possible reasons include` and three general guesses | A NetSuite DA import | The SuiteApp or bundle adds that same list to every `UNEXPECTED_ERROR`; it isn't a diagnosis, and NetSuite named no cause | Read the calls the SuiteApp made for the record ([below](#seeing-what-a-da-import-asked-of-netsuite)) -- the last one is where NetSuite failed -- and the values in the failed record's retry data. A case that repeats goes to NetSuite support, with those calls |\n| `INVALID_INITIALIZE_REF`, or `INVALID_TRANSFORMATION` \"Invalid transaction id …\" | A NetSuite DA import that maps `Created From` (`celigo_nlobjTransformId`) | A mapped `Created From` makes the import transform that source transaction into the new record (an invoice or cash sale from a sales order, say) instead of creating one from scratch. `INVALID_INITIALIZE_REF` is NetSuite refusing that transform for this source; `INVALID_TRANSFORMATION` means the value isn't a transaction's internal id | Check the source transaction in NetSuite -- whether NetSuite offers this transform for it in its current status -- and that the row maps its internal id, not its document number. Then retry the failed records |\n| `… Invalid Field Value NaN/NaN/NaN for the following field: \u003cfield>` (bundle and `suiteapp1.0` steps) or `… Invalid Field Value '\u003cvalue>' for the following field: \u003cfield>` (`suiteapp2.0` steps) | A NetSuite DA import, on a date or date-time field | The value mapped to the field isn't a date the import can read: text in the date format of the NetSuite user the connection signs in as, or ISO 8601 and the other forms JavaScript's date parser reads. SuiteScript 1.0 steps then send `NaN/NaN/NaN`; a `suiteapp2.0` step refuses the value before NetSuite sees it. A date mapping that couldn't read its source sends the text `Invalid date` ([configuring-imports > Common Errors](../../configuring-imports/SKILL.md#common-errors)) | Send ISO 8601 (`YYYY-MM-DDTHH:mm:ss.SSSZ`), or set the row's `extractDateFormat` to the source's real format; the failed record's retry data shows the value. Then retry the failed records |\n| `io_suiteapp_not_installed` \"Integrator IO SuiteApp is not installed. …\", or `sss_invalid_scriptlet_id` \"We are currently unable to communicate with the 'Celigo integrator.io' …\" (source `connection`) | NetSuite DA imports and lookups, RESTlet exports, a listener's setup | NetSuite answered that the Celigo RESTlet the step calls isn't there for this connection: the SuiteApp or bundle isn't installed in that account (or was removed), or the connection's role can't run its scripts | `celigo connections suiteapp-status \u003cconnectionId>` and the distributed ping ([above](#check-the-netsuite-side-first)) show which. A NetSuite administrator installs or updates it, or gives the role access |\n| `INVALID_KEY_OR_REF` \"Invalid `\u003cfield>` reference key `\u003cvalue>`.\" | A NetSuite import row marked `internalId: true` | The row sends its value as an internal id, and NetSuite has no record it accepts for that value on this record: a name or other text where the id belongs, or an id that no longer resolves there (a deleted or inactive record, or one outside the transaction's subsidiary). The message names the field and quotes the value | Compare the quoted value with what the records that succeed send. Turn a name into its internal id with a lookup on the row (`lookupName`): a static `map` from name to id, or a search on the name with `resultField: internalid`, kept in `netsuite_da.lookups` and the same entry in the top-level `lookups`. Correct a stale id at the source. Then retry the failed records |\n| `SSS_INVALID_SUBLIST_OPERATION` \"…add or remove lines from a static sublist\" (the SuiteApp adds \"Tip: Please make sure that all the line items that you're trying to use exist on the record.\") | A NetSuite DA import that writes an item fulfillment | A mapped line matched none of the fulfillment's lines on its key (the `isKey` columns, or the item when none is marked), so the import tried to add a line, which a fulfillment can't: its lines are its order's ([writing-mappings > Mapper 1.0](../../writing-mappings/SKILL.md#mapper-10-reference-netsuite-and-salesforce)) | Find the line whose key value isn't on the order and fix the row or lookup that produces it; a retry before that fails the same way |\n| `MULTIPLE_LINES_WITH_SAME_KEYS` \"Multiple lines found with keys \u003cfield>=\u003cvalue> …\" | A NetSuite DA import with a sublist | Two of the record's lines hold the mapped line's key values -- on a fulfillment keyed on the item, an order with that item on two lines | Key the line on what only it holds, such as the order line's id mapped to `orderline` and marked `isKey` |\n| `USER_ERROR`, with `source: application` | A NetSuite import | NetSuite refused the save: a required field is empty, or a rule in the account -- often its own user event script or workflow -- rejected the record. Celigo's hooks report their own `source` (`pre_map_hook` and the like for a JavaScript hook; `pre_map_hook_ss`, `post_map_hook_ss` or `post_submit_hook_ss` for the import's SuiteScript hooks), so NetSuite's code with `source: application` means NetSuite raised it. A postSubmit SuiteScript hook that rewords an error -- as an Integration App does when it puts its own sentence first -- changes that error's `source` to `post_submit_hook_ss` and keeps NetSuite's code ([Reading an App's Errors](../../managing-integrations/references/integration-apps.md#reading-an-apps-errors)) | Read the message for the field or condition it requires, and check whether the import sends it: the failed record's retry data shows what the step received, and the mapping what it writes. Map the field, or have a NetSuite administrator change the rule; don't edit a Celigo hook for it. Then retry the failed records |\n| `SSS_TIME_LIMIT_EXCEEDED` or `SSS_USAGE_LIMIT_EXCEEDED` (possibly lowercase) | A NetSuite RESTlet (saved-search) export, partway through a run | One RESTlet call passed NetSuite's limit for a single execution: 300 seconds, or 5,000 usage units. Each call reads `netsuite.restlet.batchSize` records (default 1000), so a search that grew in NetSuite -- more rows, columns or formulas -- can push a call over with nothing changed in Celigo | Lower `netsuite.restlet.batchSize` ([below](#batch-size-previews-and-metadata)); `pageSize` doesn't size the call. Setting `restlet.columns` doesn't trim a saved search: the RESTlet adds those columns after the search's own and never drops one. If small batches still fail, the search's owner simplifies the search itself in NetSuite |\n| `SSS_TIME_LIMIT_EXCEEDED` or `SSS_USAGE_LIMIT_EXCEEDED` on a batch of records at once | A NetSuite DA import | One call to the SuiteApp or bundle passed NetSuite's limit for a single execution. Each call carries `netsuite_da.batchSize` records (100 when unset), one call after another, and the SuiteApp or bundle saves them one at a time; a call that fails returns no result per record, so every record in it gets the same error -- including any it had already saved | Lower `netsuite_da.batchSize`. Before retrying an `add`, look in NetSuite for the records that call saved: a retry adds them again |\n| `INVALID_LOGIN_ATTEMPT` \"Invalid login attempt.\" | Every step on a token-based connection (`netsuite.authType` `token` or `token-auto`), and its test | NetSuite refused the connection's token. The Detail column of NetSuite's Login Audit Trail says why: `token_rejected` (the token was revoked, made inactive or can't be found -- TBA tokens don't expire), `permission_denied` (the user or role is inactive, the role was taken off the user, or it lacks the *Log in using Access Tokens* permission), `consumer_key_refused` (the integration record is blocked). A token created in another account fails too: production tokens aren't copied to a sandbox, and a refreshed sandbox needs new ones | Name the account, role and token on the connection (`netsuite.account`, `environment`, `roleName`/`roleId`, `tokenName`). A NetSuite administrator fixes that cause or creates a new access token for the same user, role and application; a person enters it on the connection in the Celigo UI (a `token-auto` connection authorizes again there, signing in with that role), tests the connection, then retries the failed records. Never take the token in chat ([configuring-connections > Minimum Required Fields](../../configuring-connections/SKILL.md#minimum-required-fields)) |\n\nWhat `INVALID_KEY_OR_REF`, `USER_ERROR`, `SSS_TIME_LIMIT_EXCEEDED` / `SSS_USAGE_LIMIT_EXCEEDED`, `INVALID_INITIALIZE_REF` and `INVALID_LOGIN_ATTEMPT` mean is NetSuite's own documented meaning; no live call reached NetSuite to confirm it.\n\n### Checking a refused token\n\nFor `INVALID_LOGIN_ATTEMPT` on a token-based connection, the NetSuite administrator works from the account, role and token the connection names (`netsuite.account` and `environment`, `roleName` / `roleId`, `tokenName`):\n\n- **The reason**: the Login Audit Trail -- Setup > Users/Roles > User Management > View Login Audit Trail, with Use Advanced Search and the Detail, Token-based Access Token Name and Token-based Application Name result columns. `token_rejected` is a revoked, inactive or unknown token; `permission_denied` is the user or the role; `consumer_key_refused` is a blocked integration record.\n- **The token**: Setup > Users/Roles > Access Tokens. TBA tokens don't expire; a revoked one can't be restored, so a new token is created for the same user, role and application. Deleting an integration record revokes its tokens.\n- **The role**: taking a role off the user leaves its token unusable until the role is added back. Token login needs *Log in using Access Tokens* (or *User Access Tokens*, which includes it); a role with only *Access Token Management* can't log in with a token.\n- **The integration record**: Setup > Integration > Manage Integrations, State Enabled rather than Blocked.\n- **The account**: tokens aren't copied from production to a sandbox, Release Preview or Next Preview account, and a refreshed sandbox needs new tokens; non-production account ids carry a suffix such as `123456_SB1`.\n\n## Seeing What a DA Import Asked of NetSuite\n\n- **The connection's debug log** lists, for each record a NetSuite DA import sends while it's on, the NetSuite calls the SuiteApp or bundle made to build and save the record (`nlapi calls registered for $R`), in order; the last one before an error is where NetSuite failed. Turn it on for a short window, run or retry the records, read it, and turn it off:\n\n ```bash\n celigo connections enable-debug \u003cconnectionId> --duration 15\n celigo connections debug-logs \u003cconnectionId>\n celigo connections disable-debug \u003cconnectionId>\n ```\n\n Over the REST API that's `PATCH /v1/connections/{_id}` on `/debugDate`, then `GET /v1/connections/{_id}/debug`.\n- **`Print Nlapi Debug Logs`** -- a checkbox the field list offers on every record type (`celigo_write_log`) -- mapped to `true` also writes a debug entry for each of those calls into the Celigo RESTlet's own execution log in NetSuite.\n\n## Wrong Results With No Error\n\n- **A DA lookup whose key renders empty searches `isempty`.** When the source path a criterion reads is null or missing, the lookup searches that field for blank values instead of failing: two or more blank records fail the record with `VALUE_LOOKUP_FAILED` (\"More than one match for `\"[\u003cfield>,\"isempty\"]\"`\"), none fail it with \"Could not find a match\", and exactly one silently becomes the match. Branch on the key inside the expression: send a record without one to a fallback record, or render a value that can't match and set `allowFailures: true` with a `default`.\n- **An update that repeats the record's lines.** A NetSuite DA update whose sublist has no `isKey` column adds every mapped line after the existing ones on each run, except on sublists with a default key: the item on fulfillment-style transactions, the applied transaction on a payment's or credit's. Mark the row that identifies a line `isKey`, or map `celigo_replaceAllLines_\u003csublist id>` to `true` to replace the lines ([writing-mappings > Mapper 1.0](../../writing-mappings/SKILL.md#mapper-10-reference-netsuite-and-salesforce)).\n- **Saved-search columns become fields by their Custom Label.** A column without one takes its column id, and an unlabeled formula column its type and position (`formulatext_txt_3`), so adding or moving a column in the search can rename the field the export's mappings read, and they find nothing. A select column's field holds the value's display text, not its internal id, and two columns with the same label become one field holding an array. Give every column a mapping reads its own Custom Label.\n- **A saved search edited in NetSuite changes the export.** The export runs the search's current definition, so its output changes with no change in Celigo. When the output changed and the export didn't, open the search in NetSuite.\n- **A formula criterion that's false for every record returns no rows** and no error. Check the formula with a preview that should return records.\n- **A listener that never fires** is usually scoped away from the save: records a Celigo NetSuite import writes arrive in the `restlet` context, which the defaults leave out ([gotchas 14 and 15](../SKILL.md#gotchas)), and the listener's deployment follows its flow being enabled ([Listeners](../SKILL.md#listeners)).\n- **The mapping Preview of a NetSuite DA import can't evaluate `[*]` inside a block-helper argument** (`{{#if line_items[*].x}}`, `{{#contains line_items[*].sku \"_\"}}`): the sublist comes back empty with no error, and a record using `\u003carray>.@index.\u003cfield>` is dropped. An empty preview says nothing about such an expression; check it with a run.\n\n## Batch Size, Previews and Metadata\n\n- **`netsuite.restlet.batchSize`** (default 1000) sets how many records each NetSuite call returns; the export's `pageSize` doesn't. The RESTlet caps a call at 1000, so a larger value acts as 1000. Larger batches mean fewer round trips but more NetSuite governance and memory per call, so lower it for large records, or when a call stops at NetSuite's per-call limit (`SSS_TIME_LIMIT_EXCEEDED`, above). A test run or preview caps each request at the export's `pageSize` (default 20) instead, so it never exercises the run's batch size.\n- **Metadata is cached.** `celigo metadata types \u003cconnectionId>` lists record types and saved searches, and `celigo metadata fields \u003cconnectionId> \u003crecordType>` a record type's fields; after a field, record type or saved search is created in NetSuite, read again with `--refresh` ([step 4](../SKILL.md#4-query-metadata-for-the-target-system)).\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":19919,"content_sha256":"3d366a26aab6652878d0920ac1c2139a2416be2b2423532d42e94e6a5ec6c882"},{"filename":"references/schemas/as2.yml","content":"AS2:\n type: object\n description: |-\n Configures AS2 exports and listeners. Optional — the transport settings live on the AS2\n connection, and an AS2 export is valid with only `file` (parse settings); set this object\n to link a Trading Partner Connector or to transfer raw files as blobs. An AS2 listener\n acts as the flow's source, receiving trading-partner transmissions in near real-time and\n handling decryption, signature verification, and MDN generation; AS2 (Applicability\n Statement 2) transmits EDI and other data securely over HTTP/S using S/MIME encryption\n and digital signatures.\n x-celigo-ai-guidance:\n - |-\n ## WHAT IS AS2?\n\n Applicability Statement 2 (AS2) is a widely adopted protocol for securely and reliably transmitting\n EDI and other data types over the internet using HTTP/S, S/MIME encryption, and digital signatures.\n AS2 provides:\n\n - **Message integrity** through digital signature validation\n - **Confidentiality** via encryption with X.509 certificates\n - **Non-repudiation** via Message Disposition Notifications (MDNs)\n - |-\n ## AS2 EXPORT CONFIGURATION\n\n Do not populate this object on every AS2 export — the transport settings live on the\n AS2 connection, and an export with only `file` (parse settings) is valid. Set it only\n to link a Trading Partner Connector (`_tpConnectorId`) or to enable raw-file blob\n transfer.\n - |-\n ## AS2 LISTENER FUNCTIONALITY\n\n An AS2 listener is a flow step in Celigo designed to receive incoming AS2 transmissions\n and deliver them into a defined integration flow. It acts as the \"source\" of a flow—similar to\n how a webhook listener works—except it specifically handles AS2 protocol requirements, including\n decryption, signature verification, and MDN generation.\n\n Unlike periodic polling or scheduled exports, an AS2 listener functions in near real-time—when\n a trading partner pushes an AS2 message, Celigo's listener step processes it instantly,\n generating an MDN in response to acknowledge receipt. This ensures low-latency, event-driven\n processing where each inbound AS2 transmission triggers the integration flow automatically.\n properties:\n _tpConnectorId:\n type: string\n format: objectId\n x-celigo-refModel: tradingpartnerconnectors\n description: |-\n Trading Partner Connector that supplies the partner-specific EDI configuration —\n communication protocol, document schemas, mappings, validation rules, and endpoint\n details. Set this to link the export to all settings required for AS2 communication\n with that partner.\n x-celigo-ai-guidance:\n - Reference to a TradingPartnerConnector document.\n - |-\n ## TRADING PARTNER CONNECTOR OVERVIEW\n\n A Trading Partner Connector in Celigo's integrator.io is a prebuilt, partner-specific integration\n template that streamlines the setup and management of Electronic Data Interchange (EDI) transactions\n with a designated trading partner. It encapsulates all requisite configurations:\n\n - Communication protocol (e.g., AS2, FTP/SFTP, VAN)\n - Document schemas (such as ANSI X12 or EDIFACT)\n - Mappings\n - Validation rules\n - Endpoint details\n - |-\n ## BENEFITS\n\n By referencing a Trading Partner Connector through this field, organizations:\n\n - Reduce manual setup time\n - Ensure compliance with specific partner requirements\n - Take advantage of Celigo's out-of-the-box EDI capabilities\n - Process transactions reliably and securely\n - Onboard new partners rapidly without building flows from scratch\n\n This field is crucial for AS2 configurations as it links the export to all partner-specific\n settings required for successful AS2 communication.\n examples: [\"60a2c4e6f321d800129a1a3c\"]\n blob:\n type: boolean\n description: |-\n When true, retrieves raw files without parsing them into structured records (rendered as\n a \"Transfer\" step in the flow UI). Use only when the file contents are not needed in\n subsequent steps — for binary files or when parsing is handled downstream. Only\n available on AS2 and VAN exports.\n x-celigo-ai-guidance:\n - |-\n - **Behavior**: Retrieves raw files without parsing them into structured data records. Should only be used when the contents of the file will not be used in subsequent steps.\n - **UI Appearance**: \"Transfer\" flow step\n - **Required Config**: Configuration only available on AS2 and VAN exports (as2.blob = true)\n - **Use Case**: Raw file transfers for binary files or when parsing is handled downstream\n - **Important Note**: Use this when you want to handle the file as a raw blob without automatic parsing\n required: []\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":4985,"content_sha256":"64c9d36b3ee8bc4817d441d165d5796a24faa603013e90ad5460d9152b043105"},{"filename":"references/schemas/base.yml","content":"ExportBase:\n type: object\n description: Writable export fields shared by the request and response schemas.\n properties:\n name:\n type: string\n maxLength: 100\n description: |-\n Display name for the export, shown in the flow builder, job history, and error logs.\n Descriptive, unique names indicating the data source and purpose make large accounts easier to manage.\n examples: [\"Salesforce Contacts Export\", \"NetSuite Items Export\", \"Amazon Orders Export\"]\n description:\n type: [string, 'null']\n description: |-\n Free-text summary of what the export retrieves and why. Shown in the UI and available\n to AI agents for context; has no effect on execution.\n maxLength: 5120\n examples: [\"Exports contacts from Salesforce for syncing to NetSuite\", \"Retrieves item inventory data from NetSuite\"]\n _connectionId:\n format: objectId\n x-celigo-refModel: connections\n type: [string, 'null']\n description: |-\n Connection this export uses to reach the source system. The connection's type must be\n compatible with the export's `adaptorType` (e.g. an `HTTPExport` needs an `http` connection;\n `NetSuiteExport` and `NetSuiteHTTPExport` both need a `netsuite` connection).\n Server-required unless `type` is `webhook` or `simple` (those receive data instead of\n fetching it) or `adaptorType` is `TopicExport` (Event Streams listeners are delivered\n by the platform itself and must omit it). A `WebhookExport` saved without `type: webhook`\n is refused with 422 \"_connectionId is required unless type is [webhook|simple]\"; a webhook\n export that does carry a connection saves and updates normally.\n examples: [\"60a2c4e6f321d800129a1a3c\"]\n adaptorType:\n type: string\n description: |-\n Selects the adaptor technology that executes this export, which determines the compatible\n connection types and which adaptor-specific configuration object must also be supplied\n (e.g. set `salesforce` when using `SalesforceExport`).\n enum:\n - HTTPExport\n - FTPExport\n - AS2Export\n - S3Export\n - NetSuiteExport\n - NetSuiteHTTPExport\n - SalesforceExport\n - JDBCExport\n - RDBMSExport\n - MongodbExport\n - DynamodbExport\n - WrapperExport\n - SimpleExport\n - WebhookExport\n - FileSystemExport\n - TopicExport\n - RESTExport\n x-enumDescriptions:\n HTTPExport: Exports data from generic REST or SOAP APIs over HTTP.\n FTPExport: Exports files from FTP or SFTP servers.\n AS2Export: Exports data using the AS2 (Applicability Statement 2) secure file transfer protocol.\n S3Export: Exports files from Amazon S3 cloud storage buckets.\n NetSuiteExport: Exports data from Oracle NetSuite through the Celigo SuiteApp/bundle RESTlets (saved searches or search criteria), SOAP web services, real-time listeners, or the file cabinet.\n NetSuiteHTTPExport: Exports records from Oracle NetSuite over its REST APIs — SuiteTalk REST record endpoints and SuiteQL, or a custom RESTlet — on a netsuite connection, with no saved search and no Celigo SuiteApp involved.\n SalesforceExport: Exports data from Salesforce using SOQL queries or real-time events.\n JDBCExport: Exports data from databases using JDBC connections.\n RDBMSExport: Exports data from relational databases using SQL queries.\n MongodbExport: Exports documents from MongoDB collections.\n DynamodbExport: Exports items from Amazon DynamoDB tables.\n WrapperExport: Exports data using a wrapper connector that delegates to another export.\n SimpleExport: Exports data via manual file uploads through the data loader UI.\n WebhookExport: Listens for incoming HTTP requests as a real-time event endpoint.\n FileSystemExport: Exports files from a local or mounted file system.\n TopicExport: Listens to an Event Streams topic and delivers its messages in batches, configured through `topic`; needs no connection.\n RESTExport: Legacy REST adaptor, superseded by HTTPExport — still accepted and present on older exports.\n examples: [\"HTTPExport\", \"NetSuiteExport\", \"NetSuiteHTTPExport\", \"SalesforceExport\", \"TopicExport\"]\n x-celigo-ai-guidance:\n - |-\n On a netsuite connection, default to `NetSuiteHTTPExport` (with `nsDomainType` and an\n `http` object) for exports and lookups: it reads records over NetSuite's REST APIs with\n nothing to create in NetSuite first — SuiteQL for filtered or multi-table extracts, the\n record endpoint `GET /services/rest/record/v1/\u003ctype>/{id}` for one record. Choose\n `NetSuiteExport` when the user names an existing saved search or asks for one, for\n real-time listeners (`type: distributed`), and for file-cabinet exports; never convert an\n existing `NetSuiteExport` to `NetSuiteHTTPExport` unless asked.\n - |-\n When authoring a NetSuite REST export, send `adaptorType: \"NetSuiteHTTPExport\"` explicitly:\n on a netsuite connection the server otherwise infers `NetSuiteExport` and rejects the\n document with 422 \"netsuite subschema not defined\", even though `nsDomainType` and `http`\n are present.\n nsDomainType:\n type: string\n enum:\n - suitetalk\n - restlet\n x-lowercase: true\n x-enumDescriptions:\n suitetalk: NetSuite SuiteTalk REST web services — record endpoints under /services/rest/record/v1 and SuiteQL under /services/rest/query/v1\n restlet: A custom RESTlet script deployed in the NetSuite account, called at /app/site/hosting/restlet.nl?script=\u003cid>&deploy=\u003cn>\n description: |-\n Selects which NetSuite REST host a `NetSuiteHTTPExport` calls. The server derives the\n full base URL from the connection's NetSuite account\n (`https://\u003caccount>.suitetalk.api.netsuite.com` or `https://\u003caccount>.restlets.api.netsuite.com`)\n and signs each request with the connection's token-based credentials, so `http.relativeURI`\n carries the complete path starting at `/services/rest/...` or `/app/site/hosting/restlet.nl`.\n Ignored on other adaptor types.\n x-celigo-ai-guidance:\n - |-\n Required on `NetSuiteHTTPExport`: omitting it fails the save with 422 \"Path `nsDomainType`\n is required.\" and any other value fails with 422 \"is not a valid enum value for path\n `nsDomainType`\". The export's `_connectionId` must be a netsuite connection — an http\n connection fails with 422 invalid_connection_type.\n - |-\n With `suitetalk`, saving links `http._httpConnectorVersionId`, `_httpConnectorResourceId`\n and `_httpConnectorEndpointId` to the platform's SuiteTalk connector from `http.method` +\n `http.relativeURI`, so the UI's Simple view renders an API-authored step; `restlet` steps\n are not linked.\n - |-\n The record list endpoint (`GET /services/rest/record/v1/\u003ctype>?limit=\u003cn>&offset={{{export.http.paging.skip}}}`)\n returns only `{id, links}` per item, so it yields no business fields on its own. For\n records with fields use SuiteQL: `POST /services/rest/query/v1/suiteql?limit=\u003cn>&offset={{{export.http.paging.skip}}}`,\n header `Prefer: transient`, `http.body` `{\"q\": \"\u003cSuiteQL>\"}`, `http.paging.method: skip`,\n `http.response.resourcePath: items`; the response envelope carries `hasMore` and\n `totalResults`. SuiteQL column names are the Records Catalog names (lowercase, e.g.\n `entityid`, `lastmodifieddate`), not the REST record field names.\n - |-\n For incremental runs set `type: delta`, embed `{{lastExportDateTime}}` in the SuiteQL\n `WHERE` clause (or the record endpoint's `q` parameter) and set `delta.dateFormat` to the\n format the query compares against, e.g. `to_char(lastmodifieddate,'YYYY-MM-DD HH24:MI:SS') >= '{{lastExportDateTime}}'`\n with `delta.dateFormat: \"YYYY-MM-DD HH:mm:ss\"`. In a preview the placeholder renders\n empty, so preview results do not reflect the delta window.\n - |-\n NetSuite must have the REST Web Services feature enabled (Setup > Company > Enable\n Features > SuiteCloud); otherwise every SuiteTalk call fails with NetSuite 403\n FEATURE_DISABLED \"The 'REST Web Services' feature is not enabled\". Custom RESTlets do\n not need it.\n - |-\n `http.type` file, blob, and csvstream are each rejected with 422 unsupported_type\n (\"NetSuiteHTTPExport does not support type: \u003cvalue>\") — REST steps are record-based only.\n type:\n type: [string, 'null']\n description: |-\n Operational mode of the export. When omitted, the export retrieves all available records —\n the standard batch mode, which is also the right choice for parsing structured files\n (CSV/XML/JSON) into records. Most modes take their matching configuration object (e.g. a\n `webhook` object when `type` is `webhook`); an HTTP or SQL `delta` export saves without a\n `delta` object because its request carries `{{lastExportDateTime}}`, though\n `delta.startDate` sets its first window. It still needs `type: delta`: on an export\n with no `type`, `{{lastExportDateTime}}` and `{{currentExportDateTime}}` render\n empty on every run, so a request that carries them goes out with blank values each time\n (live-verified 2026-10-22). Use `blob` only to transfer raw files without parsing their\n contents.\n x-celigo-ai-guidance:\n - |-\n ## HOW TO PICK THE TYPE VALUE\n\n The type value is determined by USER INTENT, not by the endpoint's\n shape or by which fields the endpoint happens to expose. Resolve\n the user's intent to one of the enum values FIRST, then author the\n matching config block. Each enum value below corresponds to a\n distinct retrieval semantic; any phrasing the user uses to express\n that semantic resolves to the same value:\n\n - **Process only records changed / created since the last run**\n -- the incremental / watermark / delta semantic, in whatever\n phrasing the user chose -> `delta`\n - **Limited subset for development or smoke-testing** -> `test`\n - **Process records once and have the source mark them as\n handled** (e.g. flip a boolean field after each record is\n read) -> `once`\n - **Listen for inbound HTTP events from an external system**\n -> `webhook`\n - **Listen for inbound events from NetSuite or Salesforce**\n -> `distributed`\n - **Stream database changes in real time as they happen**\n (change data capture from MongoDB, SQL Server, or\n PostgreSQL — \"when a row changes\", \"listen to my table\")\n -> `stream`, configured via the `cdc` object\n - **Subscribe to an Event Streams topic** (\"listen to topic X\",\n \"consume messages published to …\") -> `stream` with\n `adaptorType: TopicExport`, configured via the `topic` object\n (no `cdc`, no `_connectionId`)\n - **Transfer raw files WITHOUT parsing their contents into\n records** (images, PDFs, binaries) -> `blob`\n - **Manual data uploads via the data loader UI** -> `simple`\n - **Otherwise** (full / one-shot fetch with no incremental\n tracking) -> leave undefined / null\n - |-\n ### Type is an INTENT decision, not an endpoint-capability decision\n\n Uncertainty about which specific endpoint field implements the\n chosen semantic is NEVER a reason to skip the matching type\n value. If the user's intent maps to `delta` but you cannot\n verify which timestamp field the endpoint exposes, the answer\n is STILL `type: \"delta\"` with the closest-available delta\n configuration -- not \"omit delta because I'm uncertain about\n field names\". Field-name choice INSIDE the resulting filter /\n delta config is a separate concern that scopes how you fill in\n the chosen config, not whether to use it.\n - |-\n `stream` is accepted only on MongoDB and RDBMS exports (SQL Server\n or PostgreSQL connections) and on Event Streams listeners\n (`adaptorType: TopicExport`); every other adaptor rejects the save\n with `unsupported_export_type`. A MongoDB or RDBMS stream export\n must carry `cdc.snapshotMode` (and `cdc.captureMode` on MongoDB) —\n omitting them fails with 422 `missing_required_field` on those paths.\n enum:\n - webhook\n - test\n - delta\n - once\n - tranlinedelta\n - simple\n - blob\n - distributed\n - stream\n - all\n x-enumDescriptions:\n webhook: Creates an HTTP endpoint that listens for incoming requests in real time.\n test: Retrieves a limited subset of records for testing and validation purposes.\n delta: Retrieves only records changed since the last export execution.\n once: Retrieves records one time and marks them as processed in the source system.\n tranlinedelta: Retrieves only transaction line-level changes since the last export execution.\n simple: Allows manual file uploads through the data loader UI.\n blob: Retrieves raw files without parsing them into structured data records.\n distributed: Listens for real-time events from NetSuite or Salesforce.\n stream: Streams events in real time — database change data capture (MongoDB change streams, SQL Server CDC, PostgreSQL logical replication) configured through the `cdc` object, or an Event Streams topic subscription (`TopicExport`) configured through `topic`.\n all: Explicitly retrieves all available records — equivalent to omitting `type`.\n examples: [\"webhook\", \"delta\", \"test\", \"once\", \"simple\", \"blob\"]\n pageSize:\n type: [integer, 'null']\n description: |-\n Number of records per page streamed to downstream flow steps when the export is the\n flow's source step. Pages are additionally capped at 5 MB regardless of record count, so\n very large records may produce smaller pages. The server does not validate the value.\n A lookup (`isLookup: true`) runs to completion and hands every result to the incoming\n record as one response, capped at 5 MB in total, so the value does not change how many\n results a lookup returns; a result set larger than that belongs in a standalone export\n step. In a test run or preview the value also caps the records fetched (default 20).\n On a webhook listener the records of each incoming request are paged separately, before\n any hook or downstream step runs.\n default: 20\n examples: [100, 250, 500]\n dataURITemplate:\n type: string\n x-celigo-agent:\n - handlebars\n description: |-\n Handlebars template that builds a link back to each record in the source application's UI\n (e.g. `https://my.salesforce.com/lightning/r/Contact/{{record.Id}}/view`). The resolved\n URL is stored with error records in job history so users can jump straight to the record.\n examples: [\n \"https://your-store.myshopify.com/admin/customers/{{record.id}}\",\n \"https://my.salesforce.com/lightning/r/Contact/{{record.Id}}/view\",\n \"https://system.netsuite.com/app/common/entity/custjob.nl?id={{record.internalId}}\",\n \"{{record.customer_id}}, {{record.email}}, {{record.order_number}}\"\n ]\n traceKeyTemplate:\n type: [string, 'null']\n x-celigo-agent:\n - handlebars\n description: |-\n Handlebars template that overrides how each record's unique trace key is generated, used\n to track records through the flow and match retries to prior errors (e.g.\n `{{join \"_\" record.customerId record.orderId}}`). When omitted, the system picks a unique\n field automatically from the record's top level, by exact field name (`GET\n /v1/flows/{_id}/traceKey` serves the pattern). The rendered value of a template is stamped\n on each record as `io_custom_trace_key` and travels downstream with it. If a transform\n reshapes the data first, omit the `record.` prefix. Trace keys are capped at 256 characters;\n a longer key is stored truncated from the middle, keeping the beginning and end of the\n value. Downstream imports use the trace key as their default concurrency lock key (a blank\n key means no lock); it never de-duplicates — a record emitted twice with the same key is\n processed twice.\n x-celigo-ai-guidance:\n - |-\n The default derivation reads these top-level fields in order and takes the first present\n (a record carrying `id` and `internalId` is keyed on `id` alone) — `id`, `Id`, `ID`,\n `internalid`, `Internalid`, `internalId`, `InternalId`, `INTERNALID`, `_id`, `_Id`, `_ID`,\n `internal_id`, `internal_Id`, `Internal_id`, `Internal_Id`, `INTERNAL_ID`, `identifier`,\n `IDENTIFIER` — then a `*anything` rule the platform does not document.\n - |-\n A record in flight through the flow carries `io_custom_trace_key` when a template is set\n — that value is the downstream imports' default lock key; without a template the record\n carries no trace-key field (the key is derived, not stored on the record).\n examples: [\"{{record.id}}\", \"{{join \\\"_\\\" record.customerId record.orderId}}\", \"{{record.field1}}\", \"{{record.externalId}}\"]\n skipRetries:\n type: boolean\n description: |-\n When true, the platform does not retain the source data needed to retry failed records,\n reducing stored data at the cost of being unable to reprocess errors from this export.\n Shown as \"Do not store retry data\" in the UI. Defaults to false.\n examples: [true, false]\n oneToMany:\n $ref: '../common-schemas/one-to-many.yml#/OneToMany'\n pathToMany:\n $ref: '../common-schemas/path-to-many.yml#/PathToMany'\n isLookup:\n type: boolean\n description: |-\n When true, the export runs as a mid-flow lookup: it executes once per incoming record,\n using the input record's fields to parameterize the request, and passes the results to\n subsequent steps. When false, the export runs as a standalone data source. A lookup's\n results are not paged through the flow: every page the adaptor fetches is merged into one\n response for the record, up to 5 MB, so `pageSize` does not limit or extend what a lookup\n returns. A lookup that must return more than that is better modeled as a standalone\n export step.\n x-celigo-ai-guidance:\n - |-\n Not supported together with `type: \"stream\"`: a topic listener (`adaptorType:\n TopicExport`) with `isLookup: true` fails the save with 422 `not_supported`\n (\"Lookup functionality is not available for Stream Export\"). Enrich from a topic by\n listening to it in its own flow.\n examples: [true, false]\n groupByFields:\n $ref: './group-by.yml#/GroupBy'\n delta:\n $ref: './delta.yml#/Delta'\n test:\n $ref: './test.yml#/Test'\n once:\n $ref: './once.yml#/Once'\n webhook:\n $ref: './webhook.yml#/Webhook'\n simple:\n $ref: './simple.yml#/Simple'\n x-celigo-ai-guidance:\n - |-\n Configuration for data loader exports that only run in data loader specific flows.\n Note: This field and all its properties are only relevant when the 'type' field is set to 'simple'.\n distributed:\n $ref: './distributed.yml#/Distributed'\n cdc:\n $ref: './cdc.yml#/Cdc'\n topic:\n $ref: './topic.yml#/Topic'\n filesystem:\n $ref: './filesystem.yml#/FileSystem'\n http:\n $ref: './http.yml#/Http'\n file:\n $ref: './file.yml#/File'\n salesforce:\n $ref: './salesforce.yml#/Salesforce'\n as2:\n $ref: './as2.yml#/AS2'\n dynamodb:\n $ref: './dynamodb.yml#/DynamoDB'\n ftp:\n $ref: './ftp.yml#/FTP'\n jdbc:\n $ref: './jdbc.yml#/JDBC'\n mongodb:\n $ref: './mongodb.yml#/MongoDB'\n netsuite:\n $ref: './netsuite.yml#/NetSuite'\n rdbms:\n $ref: './rdbms.yml#/RDBMS'\n s3:\n $ref: './s3.yml#/S3'\n wrapper:\n $ref: './wrapper.yml#/Wrapper'\n parsers:\n $ref: '../common-schemas/parsers.yml#/Parsers'\n filter:\n x-celigo-agent:\n - filter\n description: |-\n Filter applied to each record after the export's transform and before its preSavePage\n hook. Records that match continue through the flow; records that don't are silently\n dropped. Filter expressions reference the transformed record's fields.\n allOf:\n - $ref: '../common-schemas/filter.yml#/Filter'\n x-celigo-ai-guidance:\n - |-\n Configuration for selectively processing records from an export based on their field values.\n This object enables precise control over which records continue through the flow.\n\n ## FILTER BEHAVIOR\n\n When configured, the filter is applied to each record after the export's transform and before the preSavePage hook:\n - Records that match the filter criteria continue through the flow\n - Records that don't match are silently dropped\n - No partial record processing is performed\n\n ## AVAILABLE FILTER FIELDS\n The fields available for filtering are those of each record as the export's transform leaves it (the retrieved record when no transform is set).\n inputFilter:\n x-celigo-agent:\n - filter\n description: |-\n Filter applied to incoming records before a lookup export queries the external system.\n Only matching input records trigger lookup calls; non-matching records pass through the\n step unenriched, which cuts unnecessary API traffic. Only relevant when `isLookup` is true.\n allOf:\n - $ref: '../common-schemas/filter.yml#/Filter'\n x-celigo-ai-guidance:\n - |-\n This filter is only relevant for exports where `isLookup` is set to `true`, meaning\n the export is being used as a flow step to retrieve additional data for records\n processed in previous steps.\n\n ## INPUT FILTER BEHAVIOR\n\n When configured in a lookup export, this filter is applied to the incoming records \n before they are used to query the external system:\n - Only input records that match the filter criteria will trigger lookup operations\n - Records that don't match will pass through the step without being enriched\n - This can significantly improve performance by reducing unnecessary API calls\n\n ## USE CASES\n\n Common scenarios for using inputFilter include:\n - Only looking up additional data for records that meet certain criteria\n - Preventing API calls for records that already have the required data\n - Implementing conditional lookup logic based on record properties\n - Reducing API call volume to stay within rate limits\n\n ## AVAILABLE FILTER FIELDS\n The fields available for filtering are the data fields from the input records\n passed to this lookup export from previous flow steps.\n mappings:\n x-celigo-agent:\n - mapping\n description: |-\n Field mappings applied to each incoming record before a lookup HTTP request is made,\n reshaping the upstream record into the structure the lookup target API expects. Only\n supported when `isLookup` is true and `adaptorType` is `HTTPExport` or\n `NetSuiteHTTPExport`; do not set it on source exports or other lookup adaptors.\n allOf:\n - $ref: '../common-schemas/mappings.yml#/Mappings'\n x-celigo-ai-guidance:\n - |-\n ## WHEN THIS FIELD IS VALID\n\n Celigo only supports ``mappings`` on exports that meet **BOTH**\n of the following conditions:\n\n 1. ``isLookup`` is ``true`` (the export is being used as a\n lookup step, not a source export), AND\n 2. ``adaptorType`` is ``\"HTTPExport\"`` (generic REST/SOAP\n HTTP lookup) or ``\"NetSuiteHTTPExport\"`` (NetSuite REST\n APIs) — not the saved-search NetSuite adaptor, Salesforce,\n RDBMS, file-based, or any other adaptor.\n\n For any other combination (source exports, non-HTTP lookup\n exports), do **not** emit this field. Explain the constraint\n to the user.\n\n ## BEHAVIOR WHEN VALID\n\n When used on a lookup HTTP export, ``mappings`` transforms\n each incoming record from the upstream flow step before the\n lookup HTTP call is made:\n\n - Input records are reshaped according to the mapping rules.\n - The transformed record flows into the HTTP request — either\n directly as the request body, or further shaped by an\n ``http.body`` Handlebars template which then renders\n against the post-mapped record.\n - Useful when the lookup target API expects a request\n structure that differs from the upstream record shape.\n transform:\n x-celigo-agent:\n - transform\n description: |-\n Transformation that reshapes records before they leave this step — source exports\n transform the retrieved records, lookup exports transform the lookup results. Commonly\n used to flatten nested lookup responses (e.g. `results[0].id` → `id`); merging results\n back into source records is handled separately by the flow's response mapping.\n allOf:\n - $ref: '../common-schemas/transform.yml#/Transform'\n x-celigo-ai-guidance:\n - |-\n ## EXPORT-SPECIFIC BEHAVIOR\n\n **Source Exports**: Transforms records retrieved from the external system before they are passed to downstream flow steps.\n\n **Lookup Exports (isLookup: true)**: Transforms the lookup results returned by the external system.\n\n ## LOOKUPS\n\n Optional on lookups; use one to flatten nested results (e.g., `results[0].id` -> `id`). A lookup's response mapping reads the transformed records, so its extracts name the transformed fields (`data[0].id`), not the raw response paths.\n\n Transformed results are not merged back into source records automatically; the flow's response mapping does that.\n hooks:\n type: object\n description: |-\n Custom JavaScript hooks that run at fixed points in the export lifecycle for\n transformations, validation, and filtering beyond what configuration alone can express.\n A hook name not listed here (an import's `preMap`, for example) is dropped on save\n without an error, so read `hooks` back after saving.\n properties:\n preSavePage:\n type: object\n x-celigo-agent:\n - script\n description: |-\n Hook that runs on each page of records after the export's transform and output\n filter, before the page is sent to downstream steps; commonly used to flatten\n structures, drop unwanted records, or add computed fields. It receives the page as\n `options.data` (with `errors`, `pageIndex`, `configuration` and `settings`) and returns\n `{data, errors, abort, newErrorsAndRetryData}`. Return that object, not the array: a\n hook that returns `options.data` itself, or a filtered copy, drops the page without an\n error, and the run completes with 0 records, 0 errors and 0 pages. An exception thrown\n here fails every record on that page, each with its own retry data, and later pages\n still run.\n properties:\n function:\n type: string\n description: Function to invoke within the referenced script.\n examples: [\"processPage\", \"transformData\", \"filterRecords\"]\n _scriptId:\n type: string\n format: objectId\n x-celigo-refModel: scripts\n description: |-\n Script containing the hook function named in `function`. Must be a script in this\n account: any other id fails the save with 422 (\"Only a script owned by the user or\n a script shared by the Integration App can be used at hooks.preSavePage._scriptId.\").\n examples: [\"60a2c4e6f321d800129a1a3c\", \"5f8e7d2c1b54e6003a9c7e12\"]\n _stackId:\n type: string\n x-celigo-refModel: stacks\n format: objectId\n description: Stack that hosts the hook logic, used instead of a script for stack-based deployments.\n configuration:\n type: object\n description: |-\n Static parameters passed to the hook function at runtime, letting one script be\n reused across exports with different settings.\n example: { \"threshold\": 100, \"prefix\": \"EXP-\" }\n preParse:\n type: object\n x-celigo-agent:\n - script\n description: |-\n Hook that runs on file-based exports after the file is downloaded and before it is\n parsed: it receives the raw file text as `preParseData` together with `fileMeta` and\n returns both, modified or not, for the configured parser to read. Used to normalize a\n non-standard file — fix a header row, strip a preamble, convert a format — before\n parsing; an exception thrown here fails the run.\n properties:\n function:\n type: string\n description: Function to invoke within the referenced script.\n examples: [\"fixHeader\", \"normalizeFile\", \"convertToCsv\"]\n _scriptId:\n type: string\n format: objectId\n x-celigo-refModel: scripts\n description: Script containing the hook function named in `function`.\n examples: [\"60a2c4e6f321d800129a1a3c\", \"5f8e7d2c1b54e6003a9c7e12\"]\n _stackId:\n type: string\n x-celigo-refModel: stacks\n format: objectId\n description: Stack that hosts the hook logic, used instead of a script for stack-based deployments.\n configuration:\n type: object\n description: |-\n Static parameters passed to the hook function at runtime, letting one script be\n reused across exports with different settings.\n example: { \"headerRow\": 1, \"encoding\": \"utf8\" }\n settingsForm:\n x-celigo-agent:\n - settings_form\n $ref: '../common-schemas/form.yml#/Form'\n settings:\n $ref: '../common-schemas/settings.yml#/Settings'\n mockOutput:\n $ref: './mock-output.yml#/MockOutput'\n _ediProfileId:\n type: string\n format: objectId\n x-celigo-refModel: ediprofiles\n description: |-\n EDI profile this export uses to parse incoming X12 or EDIFACT documents into structured\n JSON — it supplies the envelope qualifiers, delimiters, version, and validation rules.\n Set it when the export parses or validates EDI files (a save with a `delimited/x12` file\n definition at `version` `\"2\"` and no profile fails with 422 `missing_ediProfile`); omit\n it otherwise. Accepted on the file-reading exports — FTP, AS2, S3 and HTTP file mode\n (`http.type: \"file\"`).\n x-celigo-ai-guidance:\n - >-\n An HTTP export accepts this field only in file mode (`http.type: \"file\"`, the folder\n mode of file-provider connectors such as Google Drive, Box, Dropbox, Azure Blob, Google\n Cloud Storage and Celigo Storage). On a REST/records HTTP export the save is rejected\n with `not_supported`: \"EDIProfile can only be associated with the export of type FTP,\n AS2, S3 or HTTP file.\"\n examples: [\"60a2c4e6f321d800129a1a3c\"]\n _postParseListenerId:\n type: string\n format: objectId\n x-celigo-refModel: exports\n description: |-\n Webhook export invoked once per file after EDI parsing, on both success and failure\n (errors are included in the payload when parsing fails). Primarily used to send\n functional acknowledgements (997/999) to trading partners. Accepted on the file-reading\n exports that parse EDI — FTP, AS2, S3 and HTTP file mode (`http.type: \"file\"`).\n x-celigo-ai-guidance:\n - >-\n The listener must be an export with `type: \"webhook\"` and `webhook.provider:\n \"integrator-extension\"`; any other export id is rejected with\n `invalid_post_parse_listener`. On an HTTP export outside file mode the save is rejected\n with `post_parse_listener_not_supported`: \"Listener for functional acknowledgment is\n only supported for AS2, FTP, S3 and HTTP file export.\"\n examples: [\"60a2c4e6f321d800129a1a3c\"]\n externalId:\n type: [string, 'null']\n description: External identifier for correlating the export with a record in another system.\n _integrationId:\n type: [string, 'null']\n format: objectId\n x-celigo-refModel: integrations\n description: |-\n Not stored on an export: a value sent on create or on a full update is dropped, and a GET\n doesn't return the field. An export isn't owned by an integration; to find the flows and\n other resources that use it, call `GET /v1/exports/{_id}/dependencies`.\n _connectorId:\n type: string\n format: objectId\n x-celigo-refModel: connectors\n description: |-\n Connector this export was created from, set when the export is part of an installed\n integration app. The app's install sets it: a create that sends it is refused with 403\n (\"Cannot create an Integration App resource\").\n unencrypted:\n type: object\n description: Custom configuration values stored without encryption and returned in API responses.\n useTechAdaptorForm:\n type: boolean\n description: |-\n When true, the UI presents the full technical adaptor form for this export instead of\n the simplified assistant form.\n rawData:\n type: string\n description: Key referencing the raw sample payload captured for preview and testing.\n rest:\n type: object\n deprecated: true\n additionalProperties: true\n description: |-\n Legacy REST adaptor configuration, present on exports created before the REST adaptor\n was folded into `http`. Superseded by the `http` object.\n preSave:\n $ref: '../common-schemas/pre-save.yml#/PreSave'\n assistant:\n type: string\n description: |-\n Identifier for the connector assistant used to configure this export. For the cloud\n file providers it also selects the provider-specific location rules —\n `azuredatalakestoragegen2` (container in `file.directory.name`, path in\n `http.relativeURI`) and `googlecloudstorage` (bucket in `file.directory.storageRootId`,\n prefix in `file.directory.id`).\n examples: [\"shopify\", \"azuredatalakestoragegen2\", \"googlecloudstorage\"]\n assistantMetadata:\n type: [object, string]\n additionalProperties: true\n description: Metadata associated with the connector assistant configuration.\n sampleData:\n type: [string, object, array, 'null']\n description: Sample data payload used for previewing and testing the export.\n sampleHeaders:\n type: array\n description: Sample HTTP headers used for previewing and testing the export.\n items:\n type: object\n properties:\n name:\n type: string\n description: Header name.\n value:\n type: string\n description: Header value.\n sampleQueryParams:\n type: array\n description: Sample query parameters used for previewing and testing the export.\n items:\n type: object\n properties:\n name:\n type: string\n description: Query parameter name.\n value:\n type: string\n description: Query parameter value.\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":36677,"content_sha256":"7e15493996463726aa8eccffe56db8166b5b1a956e43f9e2092b84d1ad6fc9c9"},{"filename":"references/schemas/cdc.yml","content":"Cdc:\n type: object\n description: |-\n Change-data-capture configuration for real-time listener exports (the \"Listen for real-time\n data\" source step). Present only when `type` is `stream`. Set on `MongodbExport` (MongoDB change\n streams) and `RDBMSExport` (SQL Server CDC and PostgreSQL logical replication); the listener\n streams change events continuously instead of fetching records on a schedule.\n\n On save the API normalizes the `properties` grid into the typed fields: a `snapshot.mode`\n entry populates `snapshotMode` and a `capture.mode` entry populates `captureMode`. Every\n stream export must resolve a snapshot mode from one of those two places.\n x-celigo-ai-guidance:\n - |-\n A stream export that resolves no snapshot mode — neither `snapshotMode` nor a\n `snapshot.mode` entry in `properties` — is rejected 422 (\"Expected field: snapshotMode\n to be present\").\n properties:\n captureMode:\n type: string\n enum:\n - change_streams_update_full\n - change_streams_update_full_with_pre_image\n x-enumDescriptions:\n change_streams_update_full: Emits the full current document on each change, without the prior state.\n change_streams_update_full_with_pre_image: Emits the full current document plus its pre-image (the document before the change); requires pre-images enabled on the MongoDB collection.\n default: change_streams_update_full\n description: |-\n How MongoDB change-stream events are materialized into records. Use\n `change_streams_update_full_with_pre_image` when downstream steps need the document's prior\n state. MongoDB listeners only — `RDBMSExport` (SQL Server CDC, PostgreSQL) does not set this.\n examples: [\"change_streams_update_full\", \"change_streams_update_full_with_pre_image\"]\n snapshotMode:\n type: string\n enum:\n - initial\n - no_data\n - when_needed\n - initial_only\n x-enumDescriptions:\n initial: Takes a full snapshot of existing data on first run, then streams ongoing changes.\n no_data: Skips the initial snapshot and streams only changes that occur after the listener starts.\n when_needed: Takes a snapshot only when the connector determines one is required to avoid data loss.\n initial_only: Takes the initial snapshot and then stops, without streaming ongoing changes.\n default: no_data\n description: |-\n Whether the listener seeds existing rows before streaming changes. Use `initial` to backfill\n existing data on the first run; `no_data` to stream only new changes.\n x-celigo-ai-guidance:\n - |-\n The PostgreSQL listener form offers only `no_data` and `when_needed`. The API stores other\n values on a PostgreSQL export without rejecting them, but they are outside the supported\n PostgreSQL matrix — stay within the form's subset.\n examples: [\"initial\", \"no_data\", \"when_needed\"]\n slotName:\n type: string\n pattern: \"^[a-z_][a-z0-9_]{0,62}$\"\n description: |-\n Name of the PostgreSQL logical replication slot the listener consumes. PostgreSQL listeners\n only. When omitted, the platform manages the slot name. Must be a valid PostgreSQL\n identifier (lowercase letters, numbers, or underscores, starting with a letter or\n underscore).\n x-celigo-ai-guidance:\n - |-\n PostgreSQL connections only: on any other connection the save is rejected 422\n `not_allowed` (\"CDC slotName can only be used with postgresql adaptor type\").\n - |-\n The identifier pattern is enforced server-side — an invalid name (uppercase, dashes) is\n rejected 422 `invalid_field_value` at save time.\n examples: [\"celigo_orders_slot\", \"cdc_slot\"]\n publicationName:\n type: string\n pattern: \"^[a-z_][a-z0-9_]{0,62}$\"\n description: |-\n Name of the PostgreSQL publication that defines which tables stream changes. PostgreSQL\n listeners only. When set, the publication is the sole table filter and `rdbms.tables`\n stays unset. Must be a valid PostgreSQL identifier.\n x-celigo-ai-guidance:\n - |-\n Author EITHER `publicationName` OR `rdbms.tables`, never both. The API stores both if\n sent, so nothing catches the ambiguity at save time — the exclusivity is the listener\n form's rule (publication wins as the sole table filter).\n - |-\n PostgreSQL connections only: rejected 422 `not_allowed` on other connections, and the\n identifier pattern is enforced server-side like `slotName`.\n examples: [\"celigo_pub\", \"orders_publication\"]\n cursor:\n type: string\n description: |-\n Server-managed stream position marker (where the listener resumes reading change events).\n Maintained by the platform as events are consumed; the UI exposes a reset workflow for it\n under Cursor Management.\n x-celigo-ai-guidance:\n - Never author or modify this field — the platform owns the stream position.\n paths:\n type: array\n items:\n type: string\n description: |-\n Dot-notation paths within each raw change event to retain in the emitted record. MongoDB\n listeners default to `[\"payload\"]` (the whole change document); SQL Server and PostgreSQL\n listeners commonly select `payload.after`, `payload.op`, and related fields to expose the\n changed row and operation type.\n examples: [[\"payload\"], [\"payload.after\", \"payload.op\"]]\n intervalTime:\n type: integer\n default: 300\n description: |-\n Polling interval, in seconds, at which the listener checks the source for new change events.\n examples: [300, 30]\n properties:\n type: array\n description: |-\n Connector-engine settings passed through to the underlying CDC connector as name/value\n pairs. Mirrors the form's additional-properties grid: the required keys (`capture.mode`\n and `snapshot.mode` for MongoDB, `snapshot.mode` for SQL Server and PostgreSQL) are\n serialized first, followed by optional keys such as `field.exclude.list`,\n `column.include.list`, or `snapshot.collection.filter.overrides.\u003cdb>.\u003ccollection>`.\n On save the API also lifts `snapshot.mode` / `capture.mode` entries into the typed\n `snapshotMode` / `captureMode` fields.\n items:\n type: object\n required: [name, value]\n properties:\n name:\n type: string\n description: Connector property key (e.g. `capture.mode`, `snapshot.mode`, `field.exclude.list`).\n examples: [\"snapshot.mode\", \"field.exclude.list\"]\n value:\n type: string\n description: Connector property value.\n examples: [\"no_data\", \"schema.table.column1,schema.table.column2\"]\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":6914,"content_sha256":"e72c87d4a356bb47deffe66d0fa563624dc894b91dc501624d45b4788369f3e4"},{"filename":"references/schemas/clone-request.yml","content":"CloneRequest:\n type: object\n description: Request body for cloning an export.\n properties:\n name:\n type: string\n description: Optional name for the cloned resource. If omitted, the server may generate a default clone name.\n examples: [\"Clone - Get Accounts\"]\n connectionMap:\n type: object\n description: |\n Mapping of original connection ids to replacement connection ids.\n Keys are source connection ids on the original resource; values are target connection ids.\n For an export that uses a connection, omitting the map or sending an empty one fails\n with 400 `invalid_connectionMap`; map the connection to itself to reuse it in the clone.\n additionalProperties:\n type: string\n example:\n 5f7c579b6411271af4e7cefa: 64a1234567890abcdef12345\n 5f7c579b6411271af4e7cefb: 64a1234567890abcdef12346\n _stackId:\n type: string\n format: objectId\n x-celigo-refModel: stacks\n description: |-\n Stack that every stack reference in the cloned resources (hooks and wrapper\n connections running on a stack) points at. Required when the clone preview reports\n `stackRequired: true`: without it the clone fails with 400 `stack_not_found`. The\n stack must belong to the target environment or be shared with it (404\n `stack_not_found` for an unknown id, 400 `invalid_stack` otherwise).\n examples: [\"5f8d43a1b9e5a80011a35f2c\"]\n sandbox:\n type: boolean\n deprecated: true\n description: Optional flag copied from the original resource when applicable.\n additionalProperties: true\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":1628,"content_sha256":"61eea040886ea5298273a4c44b7239fff251a9a8a1194996e5be640a46627f29"},{"filename":"references/schemas/clone-response.yml","content":"CloneResponse:\n description: Response body for a clone operation. Some clone endpoints return the cloned resource, while others may return a list of related created resources.\n oneOf:\n - $ref: \"./response.yml#/Export\"\n - type: array\n items:\n type: object\n properties:\n model:\n type: string\n description: Model name of the created resource (e.g., Flow, Export, Import).\n _id:\n type: string\n format: objectId\n description: Unique id of the created resource.\n name:\n type: string\n description: Optional name of the created resource.\n required:\n - _id\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":695,"content_sha256":"673c8ea0763dd8d3312f4833515789e1ae440e37f801d56f9f4350b26e4b932e"},{"filename":"references/schemas/delta.yml","content":"Delta:\n type: object\n description: |-\n Configures incremental exports that retrieve only records created or modified since the\n flow's delta checkpoint; omit it for other export types. Adaptors that compare a record field\n name it in `dateField`, while HTTP exports and SQL queries carry {{lastExportDateTime}} in the\n request and save without a delta object. Each run reads the window from {{lastExportDateTime}}\n to {{currentExportDateTime}}: the flow's checkpoint and the time the run started, both less\n `lagOffset` when it is set. Until a run has exported records the flow has no checkpoint, and\n {{lastExportDateTime}} is `startDate` itself, unshifted. A first run is not a full export: with\n no `startDate`, its {{lastExportDateTime}} is the run's own timestamp, so it picks up no\n earlier changes. The checkpoint moves to a run's start time only when the run\n exports records, so a run that exports nothing leaves it where it was, and after a failed run\n the next run reuses the last checkpoint. On success, when `POST /v1/flows/{_id}/run` sends\n `export.endDate`, the flow's checkpoint is left at that value, so an end date in the past\n moves the next run's lower bound back.\n x-celigo-ai-guidance:\n - |-\n Omit this object for export types other than \"delta\". An HTTP export with type\n \"delta\" saves without it, since the request carries {{lastExportDateTime}}, but\n set ``startDate`` anyway: without it the first run's window is empty.\n Delta exports are designed for efficient synchronization by retrieving only\n records that have been created or modified since the last execution.\n - |-\n ## REQUIRED FIELDS\n - ``startDate`` -- the initial cutoff timestamp the platform uses on the\n first run (and as the floor for ``{{lastExportDateTime}}`` thereafter).\n Without it, a new flow's first run renders ``{{lastExportDateTime}}`` as\n the run's own time, so the first run exports nothing (verified on HTTP\n and RDBMS delta exports). Adapter-specific validators (e.g. NetSuite's\n ``transformTwoDotZero``) reject delta configs that omit ``startDate``\n with ``missing_delta_startDate``, so set it on every delta export.\n - |-\n ## DEFAULT CUTOFF BEHAVIOR (NO USER-SUPPLIED CUTOFF)\n When the user prompt does not specify an explicit cutoff timestamp,\n default ``startDate`` to **today at midnight UTC** in ISO-8601 format\n (e.g. ``\"2026-05-14T00:00:00.000Z\"``). This means the first run picks\n up records changed today; subsequent runs use the platform-managed\n ``{{lastExportDateTime}}`` (the start of the last run that exported records, less ``lagOffset``).\n\n - First run: filters records changed since ``startDate``.\n - Subsequent runs: uses ``{{lastExportDateTime}}`` as the lower bound;\n ``startDate`` becomes the floor when the platform timestamp is absent\n (e.g. after a reset).\n\n For optimal AI agent implementation, consider these guidelines:\n\n 1. Primary configuration method depends on adapter type:\n - For HTTP exports: Use {{lastExportDateTime}} variable in relativeURI or body\n - For specific application adapters: Use dateField to specify timestamp fields\n\n 2. The system automatically maintains the checkpoint\n - No need to store or manage timestamps in your own code\n - A first run without ``startDate`` fetches nothing older than the run itself\n - A run that exports records moves the checkpoint to its start time. The next run\n reads from the checkpoint less ``lagOffset``, which is this run's\n {{currentExportDateTime}} while ``lagOffset`` is unchanged\n - Use {{lastExportDateTime}} as the lower bound. {{currentExportDateTime}} as the\n lower bound asks only for records changed after this run's start less ``lagOffset``\n (nothing when it is 0), so the changes from {{lastExportDateTime}} up to that bound are\n never exported\n\n 3. Error handling and recovery:\n - If an export fails, the next run uses the last successful timestamp\n - Records created/modified during a failed run will be included in the next run\n - A run that exports nothing leaves the checkpoint where it was, so after a stretch of\n empty runs the next run that finds records reaches back to the last run that exported\n any: once the cause is fixed, no backfill run is needed\n - A run given its own window (export.startDate / export.endDate on\n POST /v1/flows/{_id}/run) that exports records leaves the checkpoint at endDate,\n even a past one, so the next run re-reads everything after it\n - ``lagOffset`` delays every window by its value, for a source that shows changes late\n - |-\n Omit this object for export types other than \"delta\". An HTTP export with type\n \"delta\" saves without it, since the request carries {{lastExportDateTime}}, but\n set ``startDate`` anyway: without it the first run's window is empty.\n Delta exports are designed for efficient synchronization by retrieving only\n records that have been created or modified since the last execution.\n properties:\n dateField:\n type: string\n x-celigo-excludeAdaptorTypes:\n - http\n x-celigo-ui-override: >-\n Required by the export form when the export type is \"delta\" (delta.dateField is\n required:true, visibleWhen type=delta). Encoded to mirror the form so builders produce\n connectable configurations.\n description: |-\n Record timestamp field(s) compared against the last successful run time to identify\n changed records. Accepts a single field or multiple comma-separated fields, processed\n sequentially — useful when different operations update different timestamp fields.\n If the flow's own downstream steps update the exported records, add export criteria\n that exclude already-processed records (or use a creation-time field for process-once\n flows) so each run doesn't re-export what the previous run wrote. Not supported on\n HTTP exports: embed {{lastExportDateTime}} in the relativeURI or body instead; an HTTP\n export saved with dateField is refused with 422 \"Unsupported dateField in\n export.delta.dateField\". For Salesforce this\n field is required and defaults to the standard timestamp fields (LastModifiedDate,\n CreatedDate, SystemModstamp, LastActivityDate, LastViewedDate, LastReferencedDate) plus\n any custom timestamp fields.\n x-celigo-ai-guidance:\n - Specifies one or more timestamp fields to filter records by modification date.\n - |-\n ## FIELD BEHAVIOR\n\n This field determines which record timestamp(s) are compared against the last successful run time\n to identify changed records. Key characteristics:\n\n - REQUIRED for most adapter types (except HTTP and REST where this field is not supported)\n - Can reference a single field or multiple comma-separated fields\n - Field(s) must exist in the source system and contain valid date/time values\n - When multiple fields are specified, they are processed sequentially\n - |-\n ## IMPLEMENTATION PATTERNS\n\n ### Single Field Pattern\n ```\n \"dateField\": \"lastModifiedDate\"\n ```\n - Records where lastModifiedDate > last run time are exported\n - Most common pattern, suitable for most applications\n - Works when a single field reliably tracks all changes\n\n ### Multiple Field Pattern\n ```\n \"dateField\": \"createdAt,lastModified\"\n ```\n - First exports records where createdAt > last run time\n - Then exports records where lastModified > last run time\n - Useful when different operations update different timestamp fields\n - Handles cases where some records only have creation timestamps\n\n ###CRITICAL ADAPTOR-SPECIFIC INSTRUCTION:\n - The adaptor type is HTTP. For HTTP exports, the \"dateField\" property MUST NOT be included in the delta configuration.\n - HTTP exports use the {{lastExportDateTime}} variable directly in the relativeURI or body instead of dateField.\n - DO NOT include \"dateField\" in your response. If you include it, the save is refused with\n 422 \"Unsupported dateField in export.delta.dateField\".\n\n Example HTTP query with implicit delta:\n ```\n \"/api/v1/users?modified_since={{lastExportDateTime}}\"\n ```\n\n Example (nested resource with OData-style $filter):\n ```\n \"/api/v1/parents/{{record.parentId}}/children?$filter=\u003cmodifiedTimestampField> gt {{lastExportDateTime}}\"\n ```\n\n Substitute ``\u003cmodifiedTimestampField>`` with the actual\n timestamp field the target entity exposes in its response. Do\n NOT invent or copy field names from an example authored for a\n different endpoint -- an unknown field name causes an\n \"unknown property\" / \"invalid column\" error that surfaces only\n at run time.\n\n For Salesforce, this field is required and has the following default values:\n - LastModifiedDate\n - CreatedDate\n - SystemModstamp\n - LastActivityDate\n - LastViewedDate\n - LastReferencedDate\n Also, any custom fields that are not listed above but are timestamp fields will be added to the default values.\n ```\n - |-\n ## SELF-TRIGGERING FLOWS (WRITE-BACK TO SOURCE)\n\n When a downstream step in the same flow saves the exported records\n (status stamps, processed flags, enrichment writes), a\n modification-time dateField re-qualifies every record the flow\n touches — each run re-exports everything the previous run\n processed, duplicating downstream effects and re-billing\n per-record steps such as AI agents. Remedies, in order of\n preference:\n\n 1. Keep the modification-time field and add export criteria that\n exclude records already carrying the flow's own write — usually\n the very field the write-back sets (e.g. \"sync status is\n empty\"). This preserves update-capture semantics while making\n the flow idempotent.\n 2. Switch to a creation-time field ONLY when the intent is to\n process each record exactly once — this changes semantics, and\n genuine edits to records will no longer be picked up.\n\n Writes that do not save the exported record do not re-qualify it —\n for example, attaching a NetSuite User Note through the Celigo\n bundle's usernotes mapping does not save the parent record, so its\n modification timestamp is unchanged.\n examples: [\"lastModifiedDate\", \"updatedAt\", \"modifiedOn\", \"createdAt,lastModified\"]\n dateFormat:\n type: string\n description: |-\n Moment.js format string applied to both delta variables, {{lastExportDateTime}} and\n {{currentExportDateTime}}, in HTTP requests and SQL queries alike. Leave unset unless the\n source system requires a non-ISO8601 format; by default both render as ISO 8601 UTC with\n milliseconds. Date-only formats truncate the time portion, widening the filter window.\n x-celigo-ai-guidance:\n - Defines the date/time format expected by the source system's API.\n - |-\n ## FIELD BEHAVIOR\n\n This field controls how the system formats the timestamp used for filtering:\n\n - OPTIONAL: Only needed when the source system doesn't support ISO8601\n - Default: ISO8601 format (YYYY-MM-DDTHH:mm:ss.sssZ)\n - Uses Moment.js formatting tokens\n - Formats both {{lastExportDateTime}} and {{currentExportDateTime}}, in HTTP requests and in\n SQL queries (an RDBMS delta query receives the formatted value as a quoted literal)\n - |-\n ## IMPLEMENTATION PATTERNS\n\n ### Standard Date Format\n ```\n \"dateFormat\": \"YYYY-MM-DD\" // 2023-04-15\n ```\n - For APIs that accept date-only values\n - Will truncate time portion (potentially creating a wider filter window)\n\n ### Custom DateTime Format\n ```\n \"dateFormat\": \"MM/DD/YYYY HH:mm:ss\" // 04/15/2023 14:30:00\n ```\n - For APIs with specific formatting requirements\n - Especially common with older or proprietary systems\n\n ### Localized Format\n ```\n \"dateFormat\": \"DD-MMM-YYYY HH:mm:ss\" // 15-Apr-2023 14:30:00\n ```\n - For systems requiring locale-specific representations\n - Often needed for ERP systems or regional applications\n\n Leave this field unset unless the source system explicitly requires a non-ISO8601 format.\n examples: [\"YYYY-MM-DD\", \"MM/DD/YYYY\", \"DD-MMM-YYYY HH:mm:ss\"]\n lagOffset:\n type: integer\n description: |-\n Milliseconds taken off a run's start time to give `{{currentExportDateTime}}`, the window's\n upper bound, and off the flow's checkpoint, the start time of the last run that exported\n records, to give `{{lastExportDateTime}}`, its lower bound. Until a run has exported records\n there is no checkpoint, and the lower bound is `startDate` itself, unshifted, so a\n `startDate` within the last `lagOffset` gives the run an empty window. While the value stays the same, windows stay back to back and nothing is read twice:\n each record just reaches a run `lagOffset` later, after it has had that long to become\n visible. Changing it moves the next window's start by the change: raising it re-reads the\n added span once, and lowering it skips the removed span. Set it when records\n changed shortly before a run are sometimes missing because the source shows them with a\n delay (indexing, replication), and keep it no larger than that delay, since every record\n waits that much longer. Never set a negative value: once the flow has a checkpoint it moves\n both ends into the future, so a record changed in the `lagOffset` after each run falls in no\n window and is never exported (before a checkpoint only `{{currentExportDateTime}}` moves).\n x-celigo-ai-guidance:\n - |-\n Verified live (2026-10-23, an HTTP delta export run twice with `lagOffset` 3600000 and\n then twice with -3600000): each run's `{{currentExportDateTime}}` was its start minus\n `lagOffset`; the next run's `{{lastExportDateTime}}` was the previous run's start minus\n `lagOffset`, which equals the previous `{{currentExportDateTime}}` while the value is\n unchanged; the first successful run's `{{lastExportDateTime}}` was `startDate` unshifted,\n after two failed runs that hadn't moved it. With -3600000 both ends were an hour after\n the run, and the switch from 3600000 moved the next window's start two hours past the\n previous `{{currentExportDateTime}}`, skipping that span.\n - |-\n ## DIAGNOSIS\n\n Set it when records created or modified near the run time are occasionally missing from\n a delta export because the source makes them visible late. Start from the observed delay\n (15000 to 300000 is typical) and raise it only as far as the misses stop: each record\n arrives `lagOffset` later, and each raise re-reads the added span once.\n examples: [15000, 30000, 60000]\n startDate:\n type: [string, 'null']\n format: date-time\n description: |-\n Lower-bound cutoff for the first run, and for every run until one exports records. Omitting it doesn't export history: the first run\n then starts at its own timestamp, an empty window. Set it to the earliest change the first\n run should pick up. Later runs start from the flow's checkpoint, the start of the last run\n that exported records (less `lagOffset`).\n x-celigo-ai-guidance:\n - |-\n Initial cutoff timestamp for the first export run, in ISO-8601 UTC format (e.g.\n ``\"2026-05-14T00:00:00.000Z\"``).\n Set it on every delta export: without it the first run's window is empty (verified\n on HTTP and RDBMS delta exports), and adapter-specific validators reject delta\n configs without it (e.g.\n NetSuite returns ``missing_delta_startDate`` from the ``transformTwoDotZero`` processor).\n - |-\n ## DEFAULT\n When the user prompt does not specify an explicit cutoff, default\n to **today at midnight UTC** -- i.e. the current date with the time\n portion zeroed out:\n ``\"YYYY-MM-DDT00:00:00.000Z\"`` where ``YYYY-MM-DD`` is the date the\n flow is being authored.\n - |-\n ## SUBSEQUENT RUNS\n After the first run that exports records, the platform keeps the\n flow's checkpoint at the start of the last run that exported records\n and exposes it, less ``lagOffset``, as ``{{lastExportDateTime}}``; runs that export\n nothing leave it unchanged. ``startDate`` becomes the floor used\n only when the platform timestamp is absent (e.g. after a reset).\n - |-\n ## EXAMPLES\n - ``\"2026-05-14T00:00:00.000Z\"`` -- today at midnight UTC.\n - ``\"2025-01-01T00:00:00.000Z\"`` -- explicit cutoff supplied by\n the user (e.g. \"only export sales orders created since 2025\").\n - |-\n Initial cutoff timestamp for the first export run, in ISO-8601 UTC format (e.g.\n ``\"2026-05-14T00:00:00.000Z\"``).\n Set it on every delta export: without it the first run's window is empty (verified\n on HTTP and RDBMS delta exports), and adapter-specific validators reject delta\n configs without it (e.g.\n NetSuite returns ``missing_delta_startDate`` from the ``transformTwoDotZero`` processor).\n examples:\n - \"2024-01-01T00:00:00.000Z\"\n - '2026-05-14T00:00:00.000Z'\n - '2025-01-01T00:00:00.000Z'\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":18370,"content_sha256":"9628b6ada041275732160e068e654edd4593e87da500bfe5ab5138c74b3f43c1"},{"filename":"references/schemas/distributed.yml","content":"Distributed:\n type: object\n description: Authentication settings for distributed (real-time listener) exports.\n x-celigo-ai-guidance:\n - Configuration object for distributed exports that require authentication.\n - This object contains authentication credentials needed for distributed processing.\n properties:\n bearerToken:\n type: string\n description: |-\n Bearer token that a system pushing data into this export sends as\n `Authorization: Bearer \u003ctoken>`. Optional — most distributed listeners (NetSuite,\n Salesforce real-time) use no token. Write-only and stored encrypted; treat it as a\n sensitive credential.\n format: password\n examples: [\"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...\", \"sk-1234567890abcdef\"]\n examples: [\n {\n \"bearerToken\": \"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.EXAMPLEpayload.EXAMPLEsignature\"\n }\n ]\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":895,"content_sha256":"55b8b37984b26f391687ef21acba6bed8cb93cd63645b44e61a0627ce528030f"},{"filename":"references/schemas/dynamodb.yml","content":"DynamoDB:\n type: object\n description: |-\n Defines how records are queried from DynamoDB tables. Required when the _connectionId field\n references a DynamoDB connection; must not be included for other connection types. Basic\n exports need region, method, tableName, keyConditionExpression, expressionAttributeNames,\n and expressionAttributeValues; once exports (export type \"once\") additionally need\n onceExportPartitionKey, plus onceExportSortKey for composite-key tables.\n x-celigo-ai-guidance:\n - Configuration object for Amazon DynamoDB data integration exports.\n - |-\n This object is REQUIRED when the _connectionId field references a DynamoDB\n connection and must not be included for other connection types.\n It defines how data is extracted from DynamoDB tables,\n using query operations against NoSQL data structures.\n required:\n - region\n - tableName\n - keyConditionExpression\n - expressionAttributeNames\n - expressionAttributeValues\n properties:\n region:\n type: string\n enum: [\"us-east-1\", \"us-east-2\", \"us-west-1\", \"us-west-2\", \"af-south-1\", \"ap-east-1\", \"ap-south-1\", \"ap-northeast-1\", \"ap-northeast-2\", \"ap-northeast-3\", \"ap-southeast-1\", \"ap-southeast-2\", \"ca-central-1\", \"eu-central-1\", \"eu-west-1\", \"eu-west-2\", \"eu-west-3\", \"eu-south-1\", \"eu-north-1\", \"me-south-1\", \"sa-east-1\"]\n x-enumDescriptions:\n us-east-1: US East (N. Virginia).\n us-east-2: US East (Ohio).\n us-west-1: US West (N. California).\n us-west-2: US West (Oregon).\n af-south-1: Africa (Cape Town).\n ap-east-1: Asia Pacific (Hong Kong).\n ap-south-1: Asia Pacific (Mumbai).\n ap-northeast-1: Asia Pacific (Tokyo).\n ap-northeast-2: Asia Pacific (Seoul).\n ap-northeast-3: Asia Pacific (Osaka).\n ap-southeast-1: Asia Pacific (Singapore).\n ap-southeast-2: Asia Pacific (Sydney).\n ca-central-1: Canada (Central).\n eu-central-1: Europe (Frankfurt).\n eu-west-1: Europe (Ireland).\n eu-west-2: Europe (London).\n eu-west-3: Europe (Paris).\n eu-south-1: Europe (Milan).\n eu-north-1: Europe (Stockholm).\n me-south-1: Middle East (Bahrain).\n sa-east-1: South America (Sao Paulo).\n description: |-\n AWS region hosting the DynamoDB table. Must match the region where the table is\n deployed so the integration can reach it.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field determines where to connect to DynamoDB:\n\n - REQUIRED for all DynamoDB exports\n - Must match the region where your DynamoDB table is deployed\n - Select the same AWS region used in your database configuration\n - Ensures the integration can access your table\n default: \"us-east-1\"\n examples: [\"us-east-1\", \"eu-west-1\", \"ap-southeast-2\"]\n method:\n type: string\n enum: [\"query\"]\n x-enumDescriptions:\n query: Retrieves items from DynamoDB using a key condition expression against a partition key.\n description: |-\n DynamoDB operation used to retrieve items. Only \"query\" is currently supported.\n x-celigo-ai-guidance:\n - Defines the DynamoDB operation method used to retrieve data.\n - |-\n ## FIELD BEHAVIOR\n\n - REQUIRED for all DynamoDB exports\n - Currently only supports \"query\" operations\n - Always set this value to \"query\"\n - Additional methods may be supported in future versions\n examples: [\"query\"]\n tableName:\n type: string\n x-celigo-ui-override: >-\n Required by the DynamoDB export form (dynamodb.tableName is required:true). Encoded to\n mirror the form so builders produce connectable configurations.\n description: |-\n Name of the DynamoDB table to query. Must exactly match an existing table (case-sensitive).\n x-celigo-ai-guidance:\n - Specifies the DynamoDB table from which to retrieve data.\n - |-\n ## FIELD BEHAVIOR\n\n This field identifies the data source:\n\n - REQUIRED for all DynamoDB exports\n - Must be an exact match to an existing table name\n - Case-sensitive as per AWS naming conventions\n - Cannot be changed without recreating the export\n - |-\n ## IMPLEMENTATION PATTERNS\n\n ### Standard Table Names\n ```\n \"tableName\": \"Customers\"\n ```\n examples: [\"Customers\", \"Orders\", \"Product-Catalog\", \"Transactions-2023\"]\n keyConditionExpression:\n type: string\n description: |-\n Key condition determining which items the query retrieves. Must include a condition on\n the partition key and may add sort-key conditions (equality, BETWEEN, begins_with).\n Reference attribute names with \"#\" placeholders defined in expressionAttributeNames and\n values with \":\" placeholders defined in expressionAttributeValues.\n x-celigo-ai-guidance:\n - Defines the search criteria to determine which items to retrieve from DynamoDB.\n - |-\n ## FIELD BEHAVIOR\n\n - REQUIRED when method=\"query\"\n - Must include a condition on the partition key\n - Can optionally include conditions on the sort key\n - Uses placeholders defined in expressionAttributeNames and expressionAttributeValues\n - |-\n ## COMMON PATTERNS\n\n ```\n \"#pk = :pkValue\" // Partition key only\n \"#pk = :pkValue AND #sk = :skValue\" // Exact match on partition and sort key\n \"#pk = :pkValue AND #sk BETWEEN :start AND :end\" // Range query on sort key\n \"#pk = :pkValue AND begins_with(#sk, :prefix)\" // Prefix match on sort key\n ```\n\n Placeholders with '#' reference attribute names, while ':' reference values.\n examples: [\"#pk = :customerId\", \"#pk = :orderId AND #sk BETWEEN :startDate AND :endDate\", \"#pk = :productId AND begins_with(#sk, :prefix)\"]\n filterExpression:\n type: string\n description: |-\n Filters query results on non-key attributes after the key condition is applied; omit to\n return all items matching the key condition. Uses the same \"#\" and \":\" placeholders\n defined in expressionAttributeNames and expressionAttributeValues.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n - OPTIONAL: If omitted, all items matching the key condition are returned\n - Applied after the key condition but before returning results\n - Can reference any non-key attributes to further refine results\n - Uses placeholders defined in expressionAttributeNames and expressionAttributeValues\n - |-\n ## EXAMPLES\n\n ```\n \"#status = :active\"\n \"#status = :active AND #price > :minPrice\"\n \"contains(#tags, :tagValue)\"\n ```\n\n Refer to the DynamoDB documentation for the complete list of valid operators and syntax.\n examples: [\"#status = :active AND #price > :threshold\", \"attribute_exists(#timestamp) AND #quantity > :minQuantity\", \"contains(#categories, :category)\"]\n projectionExpression:\n type: array\n items:\n type: string\n description: |-\n Attributes to return from each item, reducing data transfer; omit to return all\n attributes. Each array element is one field, referenced via \"#\" placeholders defined in\n expressionAttributeNames.\n x-celigo-ai-guidance:\n - Specifies which fields to return from each item in the results.\n - |-\n ## FIELD BEHAVIOR\n\n - OPTIONAL: If omitted, all fields are returned\n - Each array element represents a field to include\n - References attribute names defined in expressionAttributeNames\n - Reduces data transfer by returning only needed fields\n - |-\n ## EXAMPLES\n\n ```\n [\"#id\", \"#name\", \"#email\"] // Basic fields\n [\"#id\", \"#profile.#firstName\"] // Nested fields\n [\"#id\", \"#items[0]\", \"#items[1]\"] // List elements\n ```\n\n Refer to the DynamoDB documentation for more details on projection syntax.\n examples: [[\"#id\", \"#name\", \"#email\"], [\"#customerId\", \"#orderDate\", \"#items\"], [\"#partitionKey\", \"#sortKey\", \"#lastModified\"]]\n expressionAttributeNames:\n type: string\n description: |-\n JSON string mapping \"#\" placeholders to actual attribute names, e.g.\n {\"#pk\": \"customerId\"}. Placeholders defined here are used in keyConditionExpression,\n filterExpression, and projectionExpression.\n x-celigo-ai-guidance:\n - Defines placeholders for attribute names used in expressions.\n - |-\n ## FIELD BEHAVIOR\n\n - REQUIRED when using expressions that reference attribute names\n - Must be a valid JSON string mapping placeholders to actual attribute names\n - Each placeholder must begin with a pound sign (#) followed by alphanumeric characters\n - Used in keyConditionExpression, filterExpression, and projectionExpression\n - |-\n ## EXAMPLE\n\n ```\n \"{\\\"#pk\\\": \\\"customerId\\\", \\\"#status\\\": \\\"status\\\"}\"\n ```\n\n This maps the placeholder #pk to the actual attribute name \"customerId\" and #status to \"status\".\n\n Refer to the DynamoDB documentation for more details.\n examples: [\"{\\\"#pk\\\": \\\"id\\\", \\\"#sk\\\": \\\"timestamp\\\"}\", \"{\\\"#name\\\": \\\"name\\\", \\\"#price\\\": \\\"price\\\"}\", \"{\\\"#status\\\": \\\"status\\\", \\\"#date\\\": \\\"date\\\"}\"]\n expressionAttributeValues:\n type: string\n description: |-\n JSON string mapping \":\" placeholders to comparison values, e.g. {\":status\": \"ACTIVE\"}.\n Values can be static or dynamic handlebars expressions such as {{lastExportDateTime}},\n and are referenced from keyConditionExpression and filterExpression.\n x-celigo-ai-guidance:\n - Defines placeholder values used in expressions for comparison.\n - |-\n ## FIELD BEHAVIOR\n\n - REQUIRED when using expressions that compare attribute values\n - Must be a valid JSON string mapping placeholders to actual values\n - Each placeholder must begin with a colon (:) followed by alphanumeric characters\n - Used in keyConditionExpression and filterExpression\n - Can contain static values or dynamic values with handlebars syntax\n - |-\n ## EXAMPLE\n\n ```\n \"{\\\":customerId\\\": \\\"12345\\\", \\\":status\\\": \\\"ACTIVE\\\"}\"\n ```\n\n This maps the placeholder :customerId to the value \"12345\" and :status to \"ACTIVE\".\n\n Refer to the DynamoDB documentation for more details.\n examples: [\"{\\\":val\\\": \\\"12345\\\"}\", \"{\\\":start\\\": \\\"2023-01-01\\\", \\\":end\\\": \\\"{{lastExportDateTime}}\\\"}\", \"{\\\":status\\\": \\\"ACTIVE\\\", \\\":minValue\\\": 100}\"]\n onceExportPartitionKey:\n type: string\n x-celigo-ui-override: >-\n Required by the DynamoDB export form when the export type is \"once\" (dynamodb.onceExportPartitionKey\n is required, visibleWhen type=once). Encoded to mirror the form so builders produce\n connectable configurations.\n description: |-\n Partition key attribute that uniquely identifies items when the export's type is \"once\".\n Celigo uses it to mark items as processed after a successful export, preventing the same\n items from being exported again on subsequent runs.\n x-celigo-ai-guidance:\n - Specifies the partition key attribute for identifying items in once exports.\n - |-\n ## FIELD BEHAVIOR\n\n - REQUIRED when export.type=\"once\"\n - Must specify the primary key that uniquely identifies each item in the table\n - Celigo uses this to track which items have been processed\n - After successful export, Celigo updates a tracking field in the database\n\n This is needed for once exports to prevent duplicate processing of the same items\n in subsequent runs by marking them as processed.\n\n Refer to the DynamoDB documentation for more details on partition keys.\n examples: [\"id\", \"customerId\", \"orderNumber\", \"eventId\"]\n onceExportSortKey:\n type: string\n description: |-\n Sort key attribute used together with onceExportPartitionKey to identify processed items\n when the table has a composite primary key. Omit for tables keyed by a partition key\n alone.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n - OPTIONAL: Only needed for tables with composite primary keys\n - Used together with onceExportPartitionKey for tables where items are identified by both keys\n - Celigo uses both keys to uniquely identify items that have been processed\n - For tables with only a partition key (simple primary key), leave this empty\n\n This is only required if your DynamoDB table uses a composite primary key\n (partition key + sort key) to uniquely identify items.\n\n Refer to the DynamoDB documentation for more details on sort keys.\n examples: [\"timestamp\", \"email\", \"version\", \"sequenceNumber\"]\n pathToRecords:\n $ref: './record-extraction.yml#/PathToRecords'\n includeParentData:\n $ref: './record-extraction.yml#/IncludeParentData'\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":13489,"content_sha256":"2d3ff009b229747aba316dec2f0cb6ddde4d4be12306c85e3a61dd933c649f81"},{"filename":"references/schemas/file.yml","content":"File:\n type: object\n description: |-\n Controls how files are parsed, filtered, and processed across all file-based exports\n (FTP/SFTP, Amazon S3, simple file uploads, and other file sources). The type field selects\n the file format and determines which format-specific object (csv, json, xlsx, xml, or\n fileDefinition) must be configured; the output field selects whether files are parsed into\n records, transferred as blobs, or listed as metadata only. The filter object selectively\n skips files before processing.\n x-celigo-ai-guidance:\n - |-\n Configuration for file processing in exports.\n This object defines how files are parsed, filtered,\n and processed across all file-based export operations within Celigo.\n - |-\n ## EXPORT CONTEXTS\n\n This schema applies to multiple file-based export scenarios:\n\n 1. **Source System Types**:\n - Simple exports with file uploads through the UI\n - HTTP exports retrieving files from web sources\n - FTP/SFTP exports downloading files from servers\n - Amazon S3\n - Azure Blob Storage\n - Google Cloud Storage\n - And other file-based source systems\n - |-\n ## IMPLEMENTATION GUIDELINES\n\n AI agents should consider these key decision points when configuring file processing for exports:\n\n 1. **File Format Selection**: Set the `type` field to match the format of the files being processed\n (csv, json, xlsx, xml). This determines which format-specific configuration object to populate.\n\n 2. **Processing Mode**: Set the `output` field based on whether you need to:\n - Parse file contents into records (`\"records\"`)\n - Transfer files without parsing (`\"blobKeys\"`)\n - Only retrieve metadata about files (`\"metadata\"`)\n\n 3. **File Filtering**: Use the `filter` object to selectively process files based on criteria\n like file names, sizes, or custom logic.\n\n 4. **Format-Specific Configuration**: Configure the corresponding object (csv, json, xlsx, xml)\n based on the selected file type.\n - |-\n ## EXPORT-SPECIFIC CONSIDERATIONS\n\n While the file processing configuration remains consistent, different export types may have\n additional requirements:\n\n - **HTTP Exports**: May need authentication and specific endpoint configurations\n - **FTP/SFTP Exports**: Require server credentials and path information\n - **Cloud Storage Exports**: Need bucket/container details and access credentials\n\n The File schema focuses specifically on how files are processed once they are\n retrieved from the source system, regardless of which export type is used.\n properties:\n encoding:\n type: string\n description: |-\n Character encoding used to read and parse file content. If the encoding is unknown, try\n utf8 first (the default), then win1252 for Western-language files with garbled\n characters; consider the geographic origin of the data when selecting.\n x-celigo-ai-guidance:\n - |-\n Character encoding used for reading and parsing file content.\n This setting is critical for ensuring proper character interpretation,\n especially for international data and special characters.\n - |-\n ## ENCODING OPTIONS AND USAGE GUIDANCE\n\n ### UTF-8 (`\"utf8\"`)\n - **Default Setting**: Used if no encoding is specified\n - **Best For**: Modern text files, international character sets, XML/JSON files\n - **Compatibility**: Universally supported; standard for web applications\n - **When to Use**: First choice for most new integrations; handles most languages\n\n ### Windows-1252 (`\"win1252\"`)\n - **Best For**: Legacy Windows system files, older Western European data\n - **Compatibility**: Common in Windows-based exports, especially older systems\n - **When to Use**: When files originate from older Windows systems or contain certain special characters not rendering properly in utf8\n\n ### UTF-16LE (`\"utf-16le\"`)\n - **Best For**: Unicode text with extensive character requirements\n - **Compatibility**: Microsoft Word documents, some database exports\n - **When to Use**: When files have Byte Order Mark (BOM) or are known to be 16-bit Unicode\n\n ### GB18030 (`\"gb18030\"`)\n - **Best For**: Chinese character sets\n - **Compatibility**: Official character set standard for China\n - **When to Use**: For files containing simplified or traditional Chinese characters\n\n ### Mac Roman (`\"macroman\"`)\n - **Best For**: Legacy Mac system files (pre-OS X)\n - **Compatibility**: Older Apple systems and applications\n - **When to Use**: For older files created on Apple systems\n\n ### ISO-8859-1 (`\"iso88591\"`)\n - **Best For**: Western European languages\n - **Compatibility**: Widely supported in older systems\n - **When to Use**: For legacy European language content\n\n ### Shift JIS (`\"shiftjis\"`)\n - **Best For**: Japanese character sets\n - **Compatibility**: Common in Japanese Windows and older systems\n - **When to Use**: For files containing Japanese text\n - |-\n ## IMPLEMENTATION GUIDANCE FOR AI AGENTS\n\n 1. **Detection Strategy**: If encoding is unknown, first try utf8 (default), then try win1252 for Western language files with errors\n\n 2. **Encoding Selection Process**:\n - Check source system documentation for encoding specifications\n - For files with corrupt/missing characters, test alternative encodings\n - Consider geographic origin of data (Asian languages often require specific encodings)\n\n 3. **Common Issues to Watch For**:\n - Mojibake (garbled text): Indicates wrong encoding selection\n - Question marks or boxes: Character conversion failures\n - BOM markers appearing as visible characters: Consider utf-16le\n enum: [\"utf8\", \"win1252\", \"utf-16le\", \"gb18030\", \"macroman\", \"iso88591\", \"shiftjis\"]\n x-enumDescriptions:\n utf8: Universal character encoding supporting all languages (default, recommended for most files).\n win1252: Windows-1252 encoding for legacy Western European text files from Windows systems.\n utf-16le: UTF-16 Little Endian encoding for Unicode text, common in some Microsoft file exports.\n gb18030: Chinese national standard encoding supporting simplified and traditional Chinese characters.\n macroman: Mac Roman encoding for legacy files created on pre-OS X Apple systems.\n iso88591: ISO 8859-1 encoding for Western European languages in older systems.\n shiftjis: Shift JIS encoding for Japanese text, common in Japanese Windows systems.\n examples: [\"utf8\", \"win1252\"]\n type:\n type: string\n description: |-\n Format of the files being processed; determines which format-specific configuration\n object (csv, json, xlsx, xml, or fileDefinition) must be populated — other format\n objects are ignored. Required for all file-based exports except blob exports (export\n type \"blob\" or output \"blobKeys\").\n x-celigo-ai-guidance:\n - |-\n This field creates a critical dependency that determines which format-specific\n configuration object must be populated.\n - |-\n ## FORMAT OPTIONS AND REQUIREMENTS\n\n ### CSV Files (`\"csv\"`)\n - **Use For**: Tabular data with delimiter-separated values\n - **Required Config**: The `csv` object with settings like delimiters and header options\n - **Best For**: Simple tabular data, exports from spreadsheets, flat data structures\n - **Example Sources**: Exported reports, data extracts, simple database dumps\n\n ### JSON Files (`\"json\"`)\n - **Use For**: Hierarchical data in JavaScript Object Notation\n - **Required Config**: The `json` object, especially the `resourcePath` to locate records\n - **Best For**: Nested data structures, API responses, complex object representations\n - **Example Sources**: REST APIs, document databases, configuration files\n\n ### Excel Files (`\"xlsx\"`)\n - **Use For**: Microsoft Excel spreadsheets\n - **Required Config**: The `xlsx` object with Excel-specific settings\n - **Best For**: Business reports, formatted tabular data, multi-sheet documents\n - **Example Sources**: Financial reports, manually created spreadsheets\n\n ### XML Files (`\"xml\"`)\n - **Use For**: Extensible Markup Language documents\n - **Required Config**: The `xml` object, critically the `resourcePath` using XPath\n - **Best For**: Document-oriented data, SOAP responses, EDI formats\n - **Example Sources**: SOAP APIs, legacy system exports, industry standard formats\n\n ### File Definition (`\"filedefinition\"`)\n - **Use For**: Complex proprietary formats requiring custom parsing logic\n - **Required Config**: The `fileDefinition` object with the _fileDefinitionId\n - **Best For**: Legacy formats, fixed-width files, complex multi-record formats\n - **Example Sources**: Mainframe exports, proprietary formats, EDI documents\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n 1. Determine the file format from the source system or documentation\n 2. Select the matching type from the enum values\n 3. Configure ONLY the corresponding format-specific object\n 4. Other format-specific objects will be ignored\n\n For AI agents: This field creates a critical dependency chain - selecting a type\n commits you to using the corresponding configuration object.\n enum: [\"csv\", \"json\", \"xlsx\", \"xml\", \"filedefinition\"]\n x-enumDescriptions:\n csv: Parses delimiter-separated text files (CSV, TSV, pipe-delimited) into records.\n json: Parses JSON files containing hierarchical or array-based data into records.\n xlsx: Parses Microsoft Excel Open XML (.xlsx) spreadsheet files into records.\n xml: Parses XML documents using XPath expressions to extract records.\n filedefinition: Parses complex or proprietary file formats using a predefined file definition resource.\n examples: [\"csv\", \"json\", \"xlsx\", \"filedefinition\"]\n output:\n type: string\n description: |-\n Processing mode for retrieved files: parse contents into records, transfer files as\n unparsed blobs, or return only file metadata. Determines what data is passed to\n subsequent flow steps.\n x-celigo-ai-guidance:\n - |-\n Defines the fundamental processing mode for file data.\n This critical field determines how files are handled and what data is passed to\n subsequent flow steps.\n - |-\n ## PROCESSING MODES\n\n ### Content Processing (`\"records\"`)\n - **Behavior**: Files are parsed into structured records based on their format\n - **Use When**: You need to access and manipulate the data inside files\n - **Output**: Array of record objects reflecting the file's content\n - **Example Flow**: CSV files → Parse into records → Transform → Import to target system\n - **Best For**: Data synchronization, ETL processes, content-based workflows\n - **Technical Impact**: Requires format-specific parsing; higher processing overhead\n\n ### File Transfer (`\"blobKeys\"`)\n - **Behavior**: Files are treated as binary objects and transferred without parsing\n - **Use When**: You need to move files between systems without modifying content\n - **Output**: References to the binary file objects (blobKeys)\n - **Example Flow**: Image files → Transfer as blobs → Upload to cloud storage\n - **Best For**: Binary files, images, documents, any non-textual content\n - **Technical Impact**: Lower processing overhead; maintains file integrity\n\n ### File Discovery (`\"metadata\"`)\n - **Behavior**: Only file metadata is retrieved (name, size, dates) without content\n - **Use When**: You need to inventory files before deciding which to process\n - **Output**: Array of file metadata objects\n - **Example Flow**: Scan FTP folder → Get metadata → Filter by date → Process selected files\n - **Best For**: File inventory, selective processing, large directory scanning\n - **Technical Impact**: Minimal processing overhead; fastest operation mode\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n This setting profoundly affects flow architecture:\n\n 1. For data integration: Use `\"records\"` to work with the file contents\n 2. For file movement: Use `\"blobKeys\"` to preserve binary integrity\n 3. For file discovery: Use `\"metadata\"` as a first step before selective processing\n\n AI agents should select this value based on whether the integration needs to\n work with the file's content or just move/manage the files themselves.\n enum: [\"records\", \"metadata\", \"blobKeys\"]\n x-enumDescriptions:\n records: Parses file contents into structured data records for processing in the flow.\n metadata: Retrieves only file metadata (name, size, dates) without downloading or parsing content.\n blobKeys: Transfers files as binary objects without parsing, preserving the original file content.\n examples: [\"records\", \"blobKeys\", \"metadata\"]\n skipDelete:\n type: boolean\n default: false\n description: |-\n When true, source files remain on the file server after processing; when false (the\n default), files are deleted after successful processing. Files that fail processing are\n never deleted, regardless of this setting. Enable retention when files must be processed\n by other flows or systems, or kept for compliance.\n x-celigo-ai-guidance:\n - |-\n Controls whether source files are retained or deleted after successful processing.\n This setting has significant implications for data lifecycle management and system storage.\n - |-\n ## BEHAVIOR\n\n - **When true**: Source files remain on the file server after processing\n - **When false** (default): Source files are automatically deleted after successful processing\n - **Error Handling**: Files are only deleted after SUCCESSFUL processing; failed files remain intact\n - |-\n ## DECISION FACTORS FOR AI AGENTS\n\n Consider recommending `skipDelete: true` when:\n\n 1. **Compliance Requirements**:\n - Regulatory frameworks require source file retention (GDPR, HIPAA, SOX)\n - Audit trails need to maintain original file evidence\n - Data retention policies mandate preserving source files\n\n 2. **Operational Needs**:\n - Files need to be processed by multiple different flows\n - Source files serve as disaster recovery backups\n - Re-processing might be required (for testing or validation)\n - Source systems do not maintain their own copy of the files\n\n Consider recommending `skipDelete: false` (default) when:\n\n 1. **Storage Optimization**:\n - Working with large files that would consume significant storage\n - High volume of files processed frequently\n - Files are already backed up elsewhere\n - Storage costs are a concern\n\n 2. **Security Considerations**:\n - Files contain sensitive data that should be minimized\n - \"Clean workspace\" policies are in place\n - Source files represent a potential security liability\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n - **Storage Planning**: When `skipDelete: true`, ensure sufficient storage is available for file accumulation\n - **File Organization**: Consider implementing an archiving strategy for retained files\n - **Monitoring**: Set up space monitoring when retaining files to prevent storage exhaustion\n - **Cleanup Automation**: If files must be retained but eventually deleted, consider a separate cleanup job\n - |-\n ## INTEGRATION PATTERNS\n\n - **Multi-stage Processing**: Set to `true` for files that need multi-step processing in separate flows\n - **Extract-Transform-Archive**: Set to `true` when original files need archiving after extraction\n - **Single-use Import**: Set to `false` for one-time imports where originals have no further value\n - |-\n ## TECHNICAL CONSIDERATIONS\n\n This setting only affects the source file server. Records extracted from the files and processed through the flow are not affected by this setting - they continue through your integration regardless of this value.\n examples: [true, false]\n compressionFormat:\n type: string\n description: |-\n Compression format of incoming files, which are decompressed before any other processing\n (parsing, filtering). Set this only when the source always delivers compressed files —\n if a file marked as compressed is not actually compressed, processing fails. Leave unset\n when files arrive uncompressed or only sometimes compressed.\n x-celigo-ai-guidance:\n - |-\n Specifies the compression format of the files being processed.\n This setting enables the system to automatically decompress files before parsing their contents.\n - |-\n ## COMPRESSION OPTIONS\n\n ### GZIP (`\"gzip\"`)\n - **File Extension**: Typically .gz, .gzip\n - **Characteristics**: Single-file compression, maintains original file name in metadata\n - **Compression Ratio**: Moderate to high, depends on file type (5-75% size reduction)\n - **Common Sources**: Linux/Unix systems, database exports, API response payloads\n - **Use Cases**: Individual file transfers, API response handling, log files\n\n ### ZIP (`\"zip\"`)\n - **File Extension**: .zip\n - **Characteristics**: Archive format that can contain multiple files/directories\n - **Compression Ratio**: Moderate (usually 30-60% size reduction)\n - **Common Sources**: Windows systems, manual exports, email attachments\n - **Use Cases**: Multi-file packages, email attachments, mixed-format content\n - |-\n ## IMPLEMENTATION GUIDANCE FOR AI AGENTS\n\n ### When to Configure Compression\n\n 1. **Source System Behavior**:\n - Set when the source system always delivers compressed files\n - Leave blank when files are delivered uncompressed\n - NEVER set when files are sometimes compressed, sometimes not (this will cause errors)\n\n 2. **Selection Criteria**:\n - Examine file extensions (.zip, .gz) in the source system\n - Check source system documentation for compression specifications\n - Consider typical OS of the source (.zip for Windows, .gz for Unix/Linux)\n\n 3. **Multi-file Considerations**:\n - For .zip files containing multiple files, all files will be processed individually\n - For nested compression (e.g., .gz files inside .zip), only the outer compression is handled\n\n ### Technical Implementation Notes\n\n - **Processing Flow**: Files are decompressed before any other processing (parsing, filtering)\n - **Filename Handling**: After decompression, the contained file name(s) are used for subsequent operations\n - **Error Conditions**: If a file is marked as compressed but isn't actually compressed, processing will fail\n - **Mixed Formats**: If files might arrive in different compression formats, use separate exports for each format\n\n ### Performance Considerations\n\n - **Network Efficiency**: Compressed files reduce transfer time from source to integration platform\n - **Processing Overhead**: Decompression adds slight processing time but saves network time\n - **Storage Impact**: Compressed files use less storage in transit but are decompressed for processing\n\n Leave this field blank if files are not compressed. Setting an incorrect compression format will cause processing errors.\n enum: [\"gzip\", \"zip\"]\n x-enumDescriptions:\n gzip: Decompresses single-file gzip (.gz) compressed files before parsing.\n zip: Decompresses zip (.zip) archive files, processing each contained file individually.\n examples: [\"gzip\", \"zip\"]\n purgeInternalBackup:\n type: boolean\n description: |-\n When true, Celigo keeps no internal backup copies of files processed by this export;\n when false (the default), copies are retained for your account's retention period and\n are available for reprocessing or troubleshooting. Applies only to this export and only\n to Celigo's internal copies — source files are governed by skipDelete. Enable for highly\n sensitive data or zero-retention policies; without backups, recovery may require\n re-obtaining files from the source system.\n x-celigo-ai-guidance:\n - |-\n Controls whether Celigo's internal backup system retains copies of processed files.\n This setting affects data retention, recovery capabilities,\n and compliance posture.\n - |-\n ## BEHAVIOR\n\n - **When false** (default): Celigo maintains copies of all processed files for the duration of your account's retention policy\n - **When true**: Celigo will NOT keep internal backup copies of files processed by this specific export\n - **Scope**: This setting applies only to this specific export configuration; other exports are unaffected\n - |-\n ## DECISION FACTORS FOR AI AGENTS\n\n ### Scenarios to Recommend `purgeInternalBackup: true`\n\n 1. **Data Sensitivity Requirements**:\n - Files contain highly sensitive information (PII, PHI, financial, etc.)\n - Data residency/sovereignty requirements prohibit additional copies\n - Zero-retention policies mandate immediate deletion after processing\n - Compliance frameworks require minimizing data copies (GDPR, HIPAA)\n\n 2. **Technical Considerations**:\n - Very large files where storage costs are significant\n - Files that are already reliably backed up in source systems\n - Files with very short-lived relevance (e.g., temporary processing files)\n - Processing of non-production/test data that doesn't require retention\n\n ### Scenarios to Recommend `purgeInternalBackup: false` (Default)\n\n 1. **Recovery Requirements**:\n - Files represent critical business data with recovery needs\n - Source systems don't maintain reliable backups\n - Reprocessing capabilities are needed for disaster recovery\n - Audit trails require evidence of processed files\n\n 2. **Operational Benefits**:\n - Troubleshooting integration issues requires access to source files\n - Files might need reprocessing in case of downstream errors\n - Historical analysis or validation may be required\n - Protection against source system data loss\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n ### Governance Considerations\n\n - **Data Lifecycle**: Setting to `true` permanently removes files from Celigo after processing\n - **Recovery Impact**: Without backups, recovery from certain errors may require re-obtaining files from source systems\n - **Audit Trail**: Consider if processed files need to be available for future audits or investigations\n\n ### Best Practices\n\n - **Document Decision**: When setting to `true`, document the rationale for disabling backups\n - **Retention Alignment**: Ensure this setting aligns with overall data retention policies\n - **Risk Assessment**: Evaluate recovery needs against data minimization requirements\n - **Consistency**: Apply consistent backup settings across similar data types\n\n ### System Impact\n\n This setting does NOT affect:\n - The processing of files during integration runs\n - Source files on their original servers (see `skipDelete` for that)\n - Storage of processed data records in the target system\n\n It ONLY controls whether Celigo maintains internal copies of the original files.\n examples: [true, false]\n decrypt:\n type: string\n description: |-\n Decryption applied to incoming files before any other processing; decryption runs before\n decompression, and a decryption failure fails the file's processing entirely. The\n connection must already be configured with the private key (and passphrase, if\n applicable) matching the public key used to encrypt the files. Only PGP/GPG encryption\n is currently supported.\n x-celigo-ai-guidance:\n - |-\n Specifies the decryption method to apply to incoming files before processing.\n This setting enables handling of encrypted files that require decryption before\n their contents can be parsed.\n - |-\n ## SUPPORTED ENCRYPTION\n\n ### PGP/GPG Encryption (`\"pgp\"`)\n - **File Extensions**: Typically .pgp, .gpg, or .asc\n - **Encryption Standard**: OpenPGP (RFC 4880)\n - **Key Requirements**: Private key must be configured on the connection\n - **Common Sources**: Secure file transfers, encrypted backups, confidential data exchanges\n - |-\n ## IMPLEMENTATION REQUIREMENTS\n\n 1. **Connection Configuration Prerequisites**:\n - This field assumes the connection has already been configured with appropriate cryptographic settings\n - Private key must be uploaded to the connection configuration\n - Passphrase (if applicable) must be configured on the connection\n - For asymmetric encryption, the corresponding public key must have been used to encrypt the files\n\n 2. **File Processing Flow**:\n - Encrypted files are first retrieved from the source\n - Decryption is applied using the configured connection's cryptographic settings\n - After successful decryption, normal file processing continues (parsing, filtering, etc.)\n - If decryption fails, the file processing will error out completely\n - |-\n ## GUIDANCE FOR AI AGENTS\n\n ### When to Configure Decryption\n\n 1. **Security Requirements**:\n - Set to \"pgp\" when source files are PGP/GPG encrypted\n - Required for end-to-end encrypted data transfers\n - Common in financial, healthcare, and other industries with sensitive data\n - Essential for compliance with certain data protection regulations\n\n 2. **Technical Indicators**:\n - File extensions indicate encryption (.pgp, .gpg, .asc)\n - Source system documentation mentions PGP encryption\n - Files cannot be opened with standard text editors\n - Source system provides a public key for encryption\n\n ### Implementation Considerations\n\n - **Key Management**: Ensure private keys are securely stored and properly configured\n - **Error Handling**: Decryption failures will cause the entire file processing to fail\n - **Performance Impact**: Decryption adds processing overhead before file parsing begins\n - **Debugging Challenges**: Encrypted files cannot be easily examined for troubleshooting\n\n ### Security Best Practices\n\n - **Key Rotation**: Recommend periodic key rotation according to security policies\n - **Passphrase Protection**: Use strong passphrases for private keys when possible\n - **Access Control**: Limit access to connections with decryption capabilities\n - **Audit Logging**: Enable detailed logging for decryption operations when available\n - |-\n ## INTEGRATION WITH OTHER SETTINGS\n\n - If files are both encrypted AND compressed, decryption happens before decompression\n - Subsequent processing (based on file type settings) occurs after decryption\n - Internal backups (controlled by purgeInternalBackup) store the decrypted files unless configured otherwise\n\n Currently, only PGP/GPG encryption is supported. For other encryption methods, custom preprocessing may be required.\n enum: [\"pgp\"]\n x-enumDescriptions:\n pgp: Decrypts files encrypted with PGP/GPG (OpenPGP standard) before processing.\n examples: [\"pgp\"]\n batchSize:\n type: [integer, 'null']\n description: |-\n Number of files retrieved per batch; if a batch fails, the whole batch is retried\n (stored as `null` when not configured). Use\n lower values (10-50) for large files to reduce timeout and memory pressure, and higher\n values for many small files to improve throughput. Controls file retrieval only — record\n paging is governed by the export's pageSize.\n x-celigo-ai-guidance:\n - |-\n Controls the number of files processed in a single batch operation.\n This setting allows fine-tuning of performance and resource utilization during file processing.\n - |-\n ## BEHAVIOR AND PURPOSE\n\n - **Function**: Limits the number of files processed in a single batch request\n - **Maximum**: 1000 files per batch (hard system limit)\n - **Impact**: Affects performance, memory usage, and error resilience, but NOT total processing capacity\n - |-\n ## PERFORMANCE OPTIMIZATION GUIDANCE\n\n ### Large File Optimization (Set Lower Values: 10-50)\n\n When working with large files (>10MB each), smaller batch sizes are recommended:\n\n - **Network Benefits**: Reduces timeout risks during file transfer\n - **Memory Usage**: Prevents excessive memory consumption\n - **Error Isolation**: Limits the impact of processing failures\n - **Example Scenarios**: Document processing, image files, complex spreadsheets\n\n ```\n \"batchSize\": 20 // Good setting for large PDF or image files\n ```\n\n ### Small File Optimization (Set Higher Values: 100-1000)\n\n When working with small files (\u003c1MB each), larger batch sizes improve efficiency:\n\n - **Throughput**: Processes more files with less overhead\n - **API Efficiency**: Reduces the number of API calls\n - **Resource Utilization**: Maximizes processing efficiency\n - **Example Scenarios**: Small CSV files, transaction records, simple data files\n\n ```\n \"batchSize\": 500 // Efficient for small data files\n ```\n - |-\n ## IMPLEMENTATION GUIDANCE FOR AI AGENTS\n\n ### Recommendation Framework\n\n 1. **File Size Assessment**:\n - For files averaging >10MB: Recommend 10-20\n - For files averaging 1-10MB: Recommend 20-100\n - For files averaging \u003c1MB: Recommend 100-500\n - For very small files (\u003c100KB): Consider maximum (1000)\n\n 2. **Reliability Factors**:\n - For critical data with no retry capability: Recommend lower values\n - For unstable network connections: Recommend lower values\n - For production environments: Start conservative (lower) and increase based on performance\n - For development/testing: Can use higher values for efficiency\n\n 3. **System Constraints**:\n - Consider available memory in the integration environment\n - Evaluate network bandwidth and stability\n - Account for source system rate limits or concurrent connection limits\n - |-\n ## TECHNICAL CONSIDERATIONS\n\n - **Error Handling**: If a batch fails, only that batch is retried (not individual files)\n - **Parallelism**: Batch size affects concurrent processing but within system limits\n - **Monitoring**: Larger batch sizes make monitoring individual file progress more difficult\n - **Resource Scaling**: Higher batch sizes require more memory but can complete faster\n - |-\n ## RELATIONSHIP TO OTHER SETTINGS\n\n - This setting controls file retrieval batching, not record processing batch size\n - Works in conjunction with compression and decryption settings\n - Separate from and complementary to the main flow's pageSize setting\n\n Consider starting with more conservative (lower) values and increasing based on performance monitoring.\n maximum: 1000\n examples: [10, 20, 50, 1000]\n sortByFields:\n type: array\n description: |-\n Sorts the records parsed from each file before they are processed, establishing a\n deterministic processing order (for example, chronological or priority-based). Sorting\n happens after parsing but before any filtering or grouping, and does not modify the\n source files. Sorting by the same fields used in groupByFields improves grouping\n performance.\n x-celigo-ai-guidance:\n - |-\n Allows you to sort all records in a file before processing them.\n This configuration enables deterministic ordering of records,\n which can be critical for maintaining data consistency and enabling specific\n processing patterns.\n - |-\n ## FUNCTIONALITY OVERVIEW\n\n - **Purpose**: Establishes a specific processing order for records within files\n - **Timing**: Sorting is applied after file parsing but before any filtering or grouping\n - **Scope**: Affects only the in-memory representation of records (doesn't modify source files)\n - **Performance**: Has computational cost proportional to number of records × log(number of records)\n - |-\n ## STRATEGIC USES FOR AI AGENTS\n\n ### Business Process Optimization\n\n 1. **Chronological Processing**:\n - Sort by date/timestamp fields to process events in time order\n - Essential for financial transactions, audit logs, event sequences\n - Example: `[{\"field\": \"transactionDate\", \"descending\": false}]`\n\n 2. **Hierarchical Data Handling**:\n - Sort by parent records before children\n - Ensures referential integrity in relational data\n - Example: `[{\"field\": \"parentId\", \"descending\": false}, {\"field\": \"lineNumber\", \"descending\": false}]`\n\n 3. **Priority-Based Processing**:\n - Sort by importance/priority fields to handle critical items first\n - Useful for SLA-driven processes, tiered operations\n - Example: `[{\"field\": \"priority\", \"descending\": true}, {\"field\": \"createdDate\", \"descending\": false}]`\n\n ### Technical Optimization\n\n 1. **Grouping Efficiency**:\n - Sorting by the same fields used in groupByFields improves grouping performance\n - Reduces memory usage when processing large files\n - Example: `[{\"field\": \"customerId\", \"descending\": false}]` with corresponding groupByFields\n\n 2. **Lookup Optimization**:\n - Sorting by reference fields enhances performance of subsequent lookups\n - Minimizes database calls by enabling batch lookups\n - Example: `[{\"field\": \"productSku\", \"descending\": false}]`\n\n 3. **Error Reduction**:\n - Sorting can ensure dependencies are processed in correct order\n - Reduces failures from out-of-sequence processing\n - Example: `[{\"field\": \"sequenceNumber\", \"descending\": false}]`\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n ### Field Selection Considerations\n\n - **Data Type Compatibility**: Fields must contain comparable values (dates, numbers, strings)\n - **Nulls Handling**: Null values are typically sorted last (after all non-null values)\n - **Nested Fields**: Use dot notation for accessing nested properties (`customer.region`)\n - **Performance Impact**: Each additional sort field increases computational cost\n\n ### Common Implementation Patterns\n\n ```json\n // Simple single-field ascending sort (most common)\n [\n {\"field\": \"orderDate\", \"descending\": false}\n ]\n\n // Multi-field sort with primary and secondary criteria\n [\n {\"field\": \"region\", \"descending\": false},\n {\"field\": \"revenue\", \"descending\": true}\n ]\n\n // Descending priority sort with tie-breaker\n [\n {\"field\": \"priority\", \"descending\": true},\n {\"field\": \"createdDate\", \"descending\": false}\n ]\n ```\n\n ### Limitations and Constraints\n\n - Sorting large datasets has memory implications; consider record volume\n - Maximum recommended number of sort fields: 3-5 (performance considerations)\n - Sorting effectiveness depends on data consistency in source files\n - Complex sorting logic might be better implemented in custom scripts\n items:\n type: object\n properties:\n field:\n type: string\n description: |-\n Record field to sort by; use dot notation for nested properties (e.g.\n customer.name). Field names are case-sensitive.\n x-celigo-ai-guidance:\n - |-\n Specifies the record field to use as a sort key.\n This field name identifies which property of each record will be used for\n comparison when establishing processing order.\n - |-\n ## FIELD SELECTION GUIDELINES\n\n ### Data Type Considerations\n\n - **Date/Time Fields**: Provide chronological sorting (`createdDate`, `timestamp`)\n - **Numeric Fields**: Enable quantitative ordering (`amount`, `sequenceNumber`, `priority`)\n - **String Fields**: Sort alphabetically (`name`, `status`, `category`)\n - **Boolean Fields**: Group records by true/false values (`isActive`, `isProcessed`)\n\n ### Accessing Field Paths\n\n - **Top-level Properties**: Direct field names (`orderNumber`, `date`)\n - **Nested Objects**: Use dot notation (`customer.name`, `address.country`)\n - **Array Elements**: Not directly supported in basic sorting; use preprocessing\n\n ### Common Field Patterns by Domain\n\n 1. **Order Processing**:\n - `orderDate`, `orderNumber`, `customerId`, `lineNumber`\n\n 2. **Financial Data**:\n - `transactionDate`, `accountNumber`, `amount`, `documentNumber`\n\n 3. **Customer Records**:\n - `lastName`, `firstName`, `customerType`, `region`\n\n 4. **Inventory/Products**:\n - `productCategory`, `itemNumber`, `stockLevel`, `reorderDate`\n\n 5. **Event Logs**:\n - `timestamp`, `severity`, `eventType`, `sourceSystem`\n - |-\n ## IMPLEMENTATION NOTES\n\n - Field names are case-sensitive\n - Fields must exist in all records (or have consistent representation when missing)\n - Non-existent fields or null values are typically sorted last\n - Maximum recommended field name length: 64 characters\n examples: [\"date\", \"priority\", \"orderNumber\", \"customer.name\"]\n descending:\n type: boolean\n description: |-\n When true, sorts this field in descending order (newest/highest first); when false\n or omitted, sorts ascending. Directions can be mixed across fields in a\n multi-field sort.\n x-celigo-ai-guidance:\n - |-\n Controls the sort direction for the specified field.\n This setting determines whether records will be arranged in ascending (lowest to\n highest) or descending (highest to lowest) order.\n - |-\n ## BEHAVIOR\n\n - **When false or omitted**: Sorts in ascending order (A→Z, 0→9, oldest→newest)\n - **When true**: Sorts in descending order (Z→A, 9→0, newest→oldest)\n - |-\n ## STRATEGIC DIRECTION SELECTION\n\n ### Ascending Order (descending: false)\n\n Recommended for:\n - Chronological event processing (earliest first)\n - Sequential operations with dependencies\n - Reference data that builds on previous records\n - Incremental ID or sequence numbers\n\n Example use cases:\n - Processing dated transactions in chronological order\n - Handling items in order of creation\n - Incrementally building state that depends on previous records\n\n ### Descending Order (descending: true)\n\n Recommended for:\n - Priority-based processing (highest first)\n - Recent-first temporal processing\n - Most significant items first\n - Limited processing where only top N items matter\n\n Example use cases:\n - Processing high-priority items before low-priority\n - Handling most recent updates first\n - Focusing on highest-value transactions first\n - |-\n ## IMPLEMENTATION PATTERNS\n\n ### Single Field Direction\n\n ```json\n {\"field\": \"createdDate\", \"descending\": false} // Oldest first\n {\"field\": \"createdDate\", \"descending\": true} // Newest first\n ```\n\n ### Mixed Directions in Multi-field Sorts\n\n ```json\n // Group by category (A→Z) but show highest priority first in each category\n [\n {\"field\": \"category\", \"descending\": false},\n {\"field\": \"priority\", \"descending\": true}\n ]\n ```\n - |-\n ## TECHNICAL CONSIDERATIONS\n\n - Default value is `false` if omitted (ascending sort)\n - For date fields, ascending means oldest first\n - For numeric fields, ascending means smallest first\n - For string fields, ascending means alphabetical order\n examples: [true, false]\n groupByFields:\n $ref: './group-by.yml#/GroupBy'\n groupEmptyValues:\n type: boolean\n description: |-\n When true, records whose groupByFields values are empty are still grouped together rather\n than each forming its own group; when false (the default), empty-keyed records are not\n grouped. Only relevant when groupByFields is set.\n examples: [true, false]\n csv:\n type: object\n description: |-\n Parsing settings for delimiter-separated text files (CSV, TSV, pipe-delimited, and\n similar). Configure when type is \"csv\".\n\n Required when type is csv.\n x-celigo-ai-guidance:\n - |-\n Configuration settings for parsing CSV (Comma-Separated Values) files.\n This object defines how the system interprets delimited text files,\n handling variations in format, structure, and content.\n - |-\n ## WHEN TO USE\n\n Configure this object when the `type` field is set to \"csv\". This configuration is required for properly parsing:\n - Standard CSV files (.csv)\n - Tab-delimited files (.tsv, .tab)\n - Other character-delimited files (semicolon, pipe, etc.)\n - Fixed-width text files converted to delimited format\n - |-\n ## IMPLEMENTATION STRATEGY FOR AI AGENTS\n\n 1. **Format Analysis**:\n - Examine sample files to identify delimiter pattern\n - Check for presence/absence of header row\n - Look for whitespace or quote pattern inconsistencies\n - Identify any rows that should be skipped (headers, metadata, etc.)\n\n 2. **Configuration Priority**:\n - `columnDelimiter`: Most critical setting; incorrect delimiter causes parsing failures\n - `hasHeaderRow`: Affects field mapping and identification\n - `rowDelimiter`: Usually auto-detected but important for non-standard files\n - `trimSpaces`: Important for inconsistent formatting\n - `rowsToSkip`: Necessary when files contain metadata/comments before data\n\n 3. **Common File Source Patterns**:\n\n | Source System | Typical Delimiter | Header Row | Common Issues |\n |--------------|-------------------|------------|---------------|\n | Excel (US) | Comma (,) | Yes | Quoted fields with embedded commas |\n | Excel (EU) | Semicolon (;) | Yes | Decimal separator conflicts |\n | Legacy Systems | Pipe (\\|) or Tab | Varies | Inconsistent field counts |\n | POS Systems | Comma or Tab | Often No | Trailing delimiters |\n | ERP Exports | Varies widely | Usually Yes| Fixed field counts with padding |\n - |-\n ## ERROR PREVENTION\n\n - **Misaligned Columns**: Usually caused by incorrect delimiter or quotes handling\n - **Truncated Data**: Can result from wrong row delimiter settings\n - **Field Misinterpretation**: Often caused by incorrect header row setting\n - **Character Encoding Issues**: Address with the parent `encoding` setting\n - **Whitespace Problems**: Resolve with `trimSpaces` setting\n - |-\n ## OPTIMIZATION OPPORTUNITIES\n\n - For maximum parsing speed, set only the minimal required settings\n - For problematic files with inconsistent formatting, use more restrictive settings\n - Balance between permissive parsing (more data accepted) and strict validation (cleaner data)\n properties:\n columnDelimiter:\n type: string\n description: |-\n Character sequence separating fields within each row; comma when omitted. Use \"\\t\"\n for tab-delimited files; European-locale exports often use semicolons. A wrong\n delimiter doesn't fail the parse: with a header row, each row becomes one field named\n after the whole header line (`{\"order_id;sku;qty\": \"1001;TEE-M;2\"}`), so the failure\n shows up in later steps.\n x-celigo-ai-guidance:\n - |-\n Specifies the character sequence that separates individual fields (columns)\n within each row of the CSV file.\n - |-\n ## BEHAVIOR\n\n - Controls how the parser identifies individual fields in each row\n - Default value: comma (,) if not specified\n - Special characters may need to be escaped\n - |-\n ## COMMON DELIMITER PATTERNS\n\n ### Standard CSV (`,`)\n ```\n \"columnDelimiter\": \",\"\n ```\n - Most common format in US/UK systems\n - Default for most spreadsheet exports\n - Used by: Microsoft Excel (US), Google Sheets, many database exports\n\n ### European CSV (`;`)\n ```\n \"columnDelimiter\": \";\"\n ```\n - Common in European locales where comma is the decimal separator\n - Standard format in many EU countries\n - Used by: Microsoft Excel (many EU locales), European business systems\n\n ### Tab-Delimited (`\\t`)\n ```\n \"columnDelimiter\": \"\\t\"\n ```\n - Used for tab-separated values (TSV) files\n - Better for data containing commas\n - Used by: Database exports, scientific data, legacy systems\n\n ### Other Common Delimiters\n - Pipe: `\"columnDelimiter\": \"|\"` (used in mainframes, legacy systems)\n - Colon: `\"columnDelimiter\": \":\"` (less common, specialized formats)\n - Space: `\"columnDelimiter\": \" \"` (uncommon, problematic with text fields)\n - |-\n ## DETERMINATION STRATEGY FOR AI AGENTS\n\n 1. **File Extension Check**:\n - .csv → Usually comma (,)\n - .tsv → Always tab (\\t)\n - .txt → Could be any delimiter; needs inspection\n\n 2. **Source System Analysis**:\n - EU-based systems often use semicolon (;)\n - Legacy/mainframe systems often use pipe (|)\n - Scientific/statistical data often uses tab (\\t)\n\n 3. **File Content Inspection**:\n - Open file in text editor to identify separating character\n - Check for character frequency patterns\n - Look for consistent character between data elements\n\n 4. **System Documentation**:\n - Check export settings in source system\n - Review file specifications if available\n - |-\n ## IMPLEMENTATION NOTES\n\n - For tab delimiter, use `\"\\t\"` (escape sequence for tab)\n - If file contains the delimiter within text fields, ensure proper quoting\n - Multi-character delimiters are supported but rare\n - Setting the wrong delimiter is the most common parsing error\n examples: [\",\", \";\", \"\\t\"]\n rowDelimiter:\n type: string\n description: |-\n Character sequence marking the end of each record; auto-detected from file content\n when omitted. Set explicitly (\"\\n\", \"\\r\\n\", or \"\\r\") only when auto-detection fails,\n such as for files with mixed line endings — an incorrect value merges or splits\n records.\n x-celigo-ai-guidance:\n - Specifies the character sequence that indicates the end of each record (row) in the CSV file.\n - |-\n ## BEHAVIOR\n\n - Controls how the parser identifies the boundaries between records\n - Default: Auto-detect (system attempts to determine from file content)\n - Common values: newline (`\\n`), carriage return + newline (`\\r\\n`)\n - |-\n ## COMMON ROW DELIMITER PATTERNS\n\n ### Windows-Style (`\\r\\n`)\n ```\n \"rowDelimiter\": \"\\r\\n\"\n ```\n - CRLF (Carriage Return + Line Feed) sequence\n - Standard for files created on Windows systems\n - Used by: Microsoft Office, Windows-based applications\n\n ### Unix-Style (`\\n`)\n ```\n \"rowDelimiter\": \"\\n\"\n ```\n - LF (Line Feed) character only\n - Standard for files created on Unix/Linux/macOS (modern) systems\n - Used by: Linux applications, macOS applications, web exports\n\n ### Classic Mac-Style (`\\r`)\n ```\n \"rowDelimiter\": \"\\r\"\n ```\n - CR (Carriage Return) character only\n - Legacy format used by older Mac systems (pre-OSX)\n - Rare in modern files but still found in some legacy systems\n - |-\n ## WHEN TO SPECIFY EXPLICITLY\n\n In most cases, the auto-detection works well, but explicitly set this when:\n\n 1. **Mixed Line Endings**: Files containing inconsistent line ending styles\n 2. **Custom Record Separators**: Files using unconventional record delimiters\n 3. **Parsing Errors**: When auto-detection fails to correctly separate records\n 4. **Performance Optimization**: To avoid detection overhead in high-volume processing\n - |-\n ## DETERMINATION STRATEGY FOR AI AGENTS\n\n 1. **Source System Analysis**:\n - Windows systems typically use `\\r\\n`\n - Unix/Linux/macOS typically use `\\n`\n - Web downloads could use either format\n\n 2. **Troubleshooting Guidance**:\n - If records are merged or split incorrectly, check for proper row delimiter\n - If file opens correctly in text editor but parsing fails, row delimiter may be the issue\n - For files with unusual record counts, examine row delimiter setting\n - |-\n ## IMPLEMENTATION NOTES\n\n - Use escape sequences (`\\n`, `\\r\\n`, `\\r`) to represent control characters\n - Setting incorrect row delimiter may result in merged records or split records\n - When in doubt, leave unspecified to use auto-detection\n - Multi-character delimiters beyond standard line endings are supported but rare\n examples: [\"\\n\", \"\\r\\n\"]\n hasHeaderRow:\n type: boolean\n description: |-\n When true (the default), the first row is read as field names rather than data, and\n those names are available in mappings. When false, every row is treated as a data\n record and fields are referenced by position.\n x-celigo-ai-guidance:\n - Indicates whether the CSV file contains a header row with field names as the first row.\n - |-\n ## BEHAVIOR\n\n - **When true** (default): First row is treated as field names, not data\n - **When false**: All rows including the first are treated as data records\n - Impacts field mapping, validation, and record counting\n - |-\n ## IMPLEMENTATION IMPACT\n\n ### With Header Row (true)\n\n - Field names from the header row can be referenced in mappings\n - Record count excludes the header row\n - First row of data is the second physical row in the file\n - Provides self-documenting data structure\n\n ### Without Header Row (false)\n\n - Fields are referenced by position/index (e.g., Column1, Column2)\n - Record count includes all rows in the file\n - First row of data is the first physical row in the file\n - Requires external schema or position-based mapping\n - |-\n ## DETERMINATION STRATEGY FOR AI AGENTS\n\n 1. **Visual Inspection**:\n - Check if the first row contains descriptive labels rather than actual data\n - Look for data type consistency (headers are typically text, while data may be mixed)\n - Headers often use camelCase, PascalCase, or snake_case formatting\n\n 2. **Source System Analysis**:\n - Most business systems include headers by default\n - Legacy/mainframe systems may omit headers\n - Data extracts intended for human use typically include headers\n\n 3. **Content Patterns**:\n - Headers typically don't match the pattern of subsequent data rows\n - Headers often contain special characters not found in data (spaces, symbols)\n - Data rows typically have consistent patterns while headers may differ\n - |-\n ## COMMON CONFIGURATIONS BY SOURCE\n\n | Source Type | Typical Setting | Notes |\n |-------------|-----------------|-------|\n | Business Reports | true | Headers provide field context |\n | Database Exports | true | Column names as headers |\n | Legacy System Feeds | false | Often position-based fixed formats |\n | IoT/Sensor Data | false | Often compact, headerless formats |\n | Manual Data Entry | true | Helps maintain field alignment |\n - |-\n ## BEST PRACTICES\n\n - Always explicitly set this value rather than relying on the default\n - For data without headers, consider adding them in preprocessing if possible\n - When headers exist but should be ignored, use `hasHeaderRow: true` and `rowsToSkip: 1`\n - Document field positions when working with headerless files\n examples: [true, false]\n trimSpaces:\n type: boolean\n description: |-\n When true, removes leading and trailing whitespace from every field value during\n parsing; when false (the default), whitespace is preserved exactly as in the source.\n Header row values are always trimmed regardless of this setting, and spaces between\n words are never affected.\n x-celigo-ai-guidance:\n - |-\n Controls whether leading and trailing whitespace should be removed from field\n values during parsing.\n - |-\n ## IMPLEMENTATION IMPACT\n\n ### With Trimming Enabled (true)\n\n - More consistent data for comparison and matching operations\n - Prevents issues with invisible whitespace affecting equality checks\n - Reduces storage space for text-heavy datasets\n - Helps normalize data from inconsistent sources\n\n ### With Trimming Disabled (false)\n\n - Preserves exact data as represented in the source file\n - Required when whitespace is semantically meaningful\n - Maintains original field lengths exactly\n - Necessary for certain data validation scenarios\n - |-\n ## USAGE GUIDANCE FOR AI AGENTS\n\n ### Recommend `trimSpaces: true` when:\n\n 1. **Data Consistency Issues**:\n - Source systems are known to have inconsistent spacing\n - Data will be used for matching or comparison operations\n - Files are generated by multiple different systems\n - Human-entered data is present (prone to spacing errors)\n\n 2. **Data Type Considerations**:\n - Fields contain numeric values (where spaces are not meaningful)\n - Fields contain codes, IDs, or reference values\n - Fields will be used in lookups or joins\n - Normalization is more important than exact representation\n\n ### Recommend `trimSpaces: false` when:\n\n 1. **Data Fidelity Requirements**:\n - Working with fixed-width fields where spaces matter\n - Dealing with formatted data where spacing is semantic\n - Legal or compliance scenarios requiring exact preservation\n - Scientific data where precision of representation matters\n\n 2. **Content Characteristics**:\n - Working with text fields where leading/trailing spaces could be intentional\n - Processing creative content, addresses, or formatted text\n - Source system uses space padding for alignment purposes\n - |-\n ## IMPLEMENTATION NOTES\n\n - This setting affects all fields consistently (cannot be applied to select fields)\n - Only affects leading and trailing spaces, not spaces between words\n - Has no effect on empty fields (empty remains empty)\n - For selective trimming, use transformation rules after parsing\n examples: [true, false]\n rowsToSkip:\n type: integer\n description: |-\n Number of rows at the top of the file to ignore before parsing begins — useful for\n report titles, timestamps, or other metadata above the data. The header row (when\n hasHeaderRow is true) is expected after the skipped rows.\n x-celigo-ai-guidance:\n - |-\n Specifies the number of rows at the beginning of the file to ignore before\n starting data processing.\n - |-\n ## BEHAVIOR\n\n - Skips the specified number of rows from the beginning of the file\n - These rows are completely ignored and not processed as data\n - The header row (if present) is counted after the skipped rows\n - Default value is 0 (no rows skipped)\n - |-\n ## IMPLEMENTATION IMPACT\n\n ### Common Use Cases\n\n 1. **Metadata Headers**:\n - Skip report titles, generated timestamps, system information\n - Skip explanatory text at the beginning of files\n - Skip company letterhead or report identification rows\n\n 2. **Multi-Header Files**:\n - Skip category headers or section titles\n - Skip nested headers or hierarchy information\n - Skip column grouping indicators\n\n 3. **Technical Requirements**:\n - Skip binary file markers or encoding identifiers\n - Skip non-data content like instructions or disclaimers\n - Skip inconsistent early rows before standardized data begins\n - |-\n ## CALCULATION GUIDANCE FOR AI AGENTS\n\n When determining the correct value for `rowsToSkip`:\n\n 1. **Count from Zero**:\n - Row 1 = 0, Row 2 = 1, Row 3 = 2, etc.\n\n 2. **For Files with Headers**:\n - Set rowsToSkip = (first data row position - 1) - (hasHeaderRow ? 1 : 0)\n - Example: If data starts on row 5, and file has a header row:\n rowsToSkip = (5 - 1) - 1 = 3\n\n 3. **For Files without Headers**:\n - Set rowsToSkip = (first data row position - 1)\n - Example: If data starts on row 3, and file has no header row:\n rowsToSkip = (3 - 1) = 2\n - |-\n ## DETERMINATION STRATEGY\n\n 1. **Visual Inspection**:\n - Open file in text editor and count non-data rows at the top\n - Identify the first row containing actual data values\n - Note if a header row exists separately from skipped content\n\n 2. **Common Patterns by Source**:\n - ERP Reports: Often 2-5 rows of report metadata\n - Exported Spreadsheets: May have title rows, date stamps\n - Database Extracts: Usually minimal (0-1) skipped rows\n - Legacy Systems: May have control records or job information\n - |-\n ## IMPLEMENTATION NOTES\n\n - Setting too high skips valid data; setting too low includes non-data as records\n - When in doubt, visually inspect the file to confirm correct skip count\n - Remember that header row (if hasHeaderRow=true) is counted AFTER skipped rows\n - Maximum recommended value: 100 (larger values may indicate format misunderstanding)\n examples: [0, 1, 2]\n disableQuoteAndStripEnclosingQuotes:\n type: boolean\n description: |-\n When true, disables CSV quote processing: quotes are treated as literal characters,\n enclosing quotes are stripped, and delimiters inside quoted text split the field.\n When false (the default), standard RFC 4180 quoting applies and quoted fields\n protect embedded delimiters. Enable only for files with non-standard or malformed\n quoting; review this setting first when field counts vary unexpectedly between rows.\n x-celigo-ai-guidance:\n - |-\n Controls the handling of quoted fields in CSV files,\n specifically how the parser manages quotation marks around field values.\n - |-\n ## BEHAVIOR\n\n - **When false** (default): Standard CSV quoting rules are applied\n - Quotation marks around fields protect embedded delimiters\n - Parser intelligently handles escaped quotes within quoted fields\n - Follows RFC 4180 CSV specifications for quote handling\n\n - **When true**: Quote detection and processing is disabled\n - All quotes are treated as literal characters, not field delimiters\n - Any quotes surrounding field values are removed\n - Embedded delimiters in quoted fields will cause field splitting\n - |-\n ## IMPLEMENTATION IMPACT\n\n ### Standard Quote Handling (false)\n\n Example input: `\"Smith, John\",42,\"Notes with \"\"quotes\"\" inside\"`\n\n Result:\n - Field 1: `Smith, John` (comma preserved inside quotes)\n - Field 2: `42`\n - Field 3: `Notes with \"quotes\" inside` (embedded quotes normalized)\n\n ### Disabled Quote Handling (true)\n\n Example input: `\"Smith, John\",42,\"Notes with \"\"quotes\"\" inside\"`\n\n Result:\n - Field 1: `\"Smith`\n - Field 2: ` John\"`\n - Field 3: `42`\n - Field 4: `\"Notes with \"\"quotes\"\" inside\"`\n - |-\n ## USAGE GUIDANCE FOR AI AGENTS\n\n ### Recommend `disableQuoteAndStripEnclosingQuotes: true` when:\n\n 1. **Quote-Related Parsing Problems**:\n - Files contain inconsistent or malformed quote usage\n - Source system doesn't follow standard CSV quoting rules\n - Quotes appear as literal data rather than field delimiters\n - Quotes are present but delimiters are never embedded in fields\n\n 2. **Special Data Formats**:\n - Working with custom delimited formats that don't use quotes for escaping\n - Files use alternate escaping mechanisms for embedded delimiters\n - Source system adds quotes to all fields regardless of content\n\n ### Recommend `disableQuoteAndStripEnclosingQuotes: false` (default) when:\n\n 1. **Standard CSV Compliance**:\n - Files follow RFC 4180 or similar CSV standards\n - Fields contain embedded delimiters that must be preserved\n - Quotes are used properly to enclose fields with special characters\n - Source is a standard database, spreadsheet, or business system export\n\n 2. **Data Content Characteristics**:\n - Fields contain embedded commas, newlines, or other delimiters\n - Text fields might contain quotation marks as part of the content\n - Preserving the exact structure of complex text fields is important\n - |-\n ## TROUBLESHOOTING INDICATORS\n\n Consider changing this setting when encountering these issues:\n\n - Field counts vary unexpectedly between rows\n - Text with embedded delimiters is being split into multiple fields\n - Quotes appearing at the beginning and end of every field in the result\n - Extra quote characters appearing within field values\n - |-\n ## IMPLEMENTATION NOTES\n\n - This setting significantly changes parsing behavior - test thoroughly\n - Affects all fields in the file consistently\n - Incorrect setting can cause severe data misalignment\n - When field count inconsistency occurs, review this setting first\n examples: [true, false]\n keyColumns:\n type: array\n description: |-\n Field names whose values together identify a record's group when consecutive rows\n share the same key — used to merge multi-line records (e.g. an order header repeated\n across line-item rows) into a single record. Leave empty for one record per row; the\n columns must exist in the parsed header.\n items:\n type: string\n examples: [[\"Event Type\"], [\"settlement-id\", \"order-id\", \"transaction-type\", \"marketplace-name\"]]\n json:\n type: object\n description: |-\n Parsing settings for JSON files. Configure when type is \"json\". resourcePath locates the\n array of records when they are nested inside a container object; malformed JSON fails\n the entire file's processing.\n\n Required when type is json.\n x-celigo-ai-guidance:\n - |-\n ## WHEN TO USE\n\n Configure this object when the `type` field is set to \"json\". This configuration is required for properly parsing:\n - Standard JSON files (.json)\n - JSON data exports from APIs or databases\n - JSON Lines format (newline-delimited JSON)\n - Nested or hierarchical data structures\n - |-\n Configuration settings for parsing JSON (JavaScript Object Notation) files.\n This object defines how the system interprets and processes hierarchical data\n contained in JSON-formatted files.\n - |-\n ## JSON PARSING CHARACTERISTICS\n\n - **Hierarchical Data**: JSON naturally supports nested objects and arrays\n - **Type Preservation**: Numbers, booleans, nulls, and strings are correctly typed\n - **Flexible Structure**: Can handle varying record structures\n - **Tree Navigation**: Supports complex object traversal via path expressions\n - |-\n ## IMPLEMENTATION STRATEGY FOR AI AGENTS\n\n 1. **Data Structure Analysis**:\n - Examine sample files to understand the object hierarchy\n - Identify where the actual records/rows are located in the structure\n - Determine if records are at the root or nested within containers\n - Check for array structures that contain the target records\n\n 2. **Common JSON Data Patterns**:\n\n ### Root Array Pattern\n ```json\n [\n {\"id\": 1, \"name\": \"Product 1\"},\n {\"id\": 2, \"name\": \"Product 2\"}\n ]\n ```\n - Records are directly at the root as an array\n - No resourcePath needed (leave blank)\n - Most straightforward structure for processing\n\n ### Container Object Pattern\n ```json\n {\n \"data\": [\n {\"id\": 1, \"name\": \"Product 1\"},\n {\"id\": 2, \"name\": \"Product 2\"}\n ],\n \"metadata\": {\n \"count\": 2,\n \"page\": 1\n }\n }\n ```\n - Records are in an array inside a container object\n - Requires resourcePath (e.g., \"data\")\n - Common in API responses with metadata\n\n ### Nested Container Pattern\n ```json\n {\n \"response\": {\n \"results\": [\n {\"id\": 1, \"name\": \"Product 1\"},\n {\"id\": 2, \"name\": \"Product 2\"}\n ],\n \"pagination\": {\n \"nextPage\": 2\n }\n },\n \"status\": \"success\"\n }\n ```\n - Records are deeply nested in the hierarchy\n - Requires dot notation in resourcePath (e.g., \"response.results\")\n - Common in complex API responses\n - |-\n ## ERROR PREVENTION\n\n - **Invalid Path**: Incorrectly specified resourcePath results in zero records found\n - **Type Mismatch**: resourcePath must point to an array of objects for proper record processing\n - **Empty Results**: If path resolves to null or non-existent field, no error is thrown but no records are processed\n - **Parsing Failures**: Malformed JSON will cause the entire file processing to fail\n - |-\n ## OPTIMIZATION OPPORTUNITIES\n\n - For large JSON files, consider preprocessing to extract only relevant sections\n - For files with complex structures, validate the resourcePath with sample data\n - When processing API responses, coordinate resourcePath with the API documentation\n - For very large datasets, consider using streaming JSON parsing (NDJSON format)\n properties:\n resourcePath:\n type: string\n description: |-\n Dot-notation path to the array of records within the JSON structure (e.g.\n \"response.data.customers\"); leave empty when the file's root is already the record\n array. The path must resolve to an array of objects — array indexing and wildcards\n are not supported. A path that resolves to nothing produces zero records without\n raising an error.\n x-celigo-ai-guidance:\n - |-\n Specifies the path to the array of records within the JSON structure.\n This field helps the system locate and extract the target records when they're\n nested inside a larger JSON object hierarchy.\n - |-\n ## BEHAVIOR\n\n - **Purpose**: Identifies where the array of records is located in the JSON structure\n - **Format**: Dot notation path to navigate nested objects (e.g., \"data.records\")\n - **When Empty**: System expects records to be at the root level as an array\n - **Result**: Array found at this path is processed as individual records\n - |-\n ## PATH NOTATION GUIDELINES\n\n ### Basic Path Patterns\n\n - **Root Level Array**: Leave empty or null when records are a direct array at root\n - **Single Level Nesting**: Use the property name (e.g., \"data\", \"results\", \"items\")\n - **Multi-Level Nesting**: Use dot notation (e.g., \"response.data.items\")\n\n ### Path Construction Rules\n\n 1. **Object Navigation**:\n - Use dots to traverse object properties: \"parent.child.grandchild\"\n - Each segment must be a valid property name in the JSON\n\n 2. **Target Requirements**:\n - The path MUST resolve to an array of objects\n - Each object in the array will be processed as one record\n - The array must be the final element in the path\n\n 3. **Limitations**:\n - Array indexing is not supported in the path (e.g., \"data[0]\")\n - Wildcard selectors are not supported\n - Regular expressions are not supported\n - |-\n ## DETERMINATION STRATEGY FOR AI AGENTS\n\n To identify the correct resourcePath:\n\n 1. **Examine Sample Data**:\n - Open a sample JSON file or API response\n - Locate the array containing the actual data records\n - Note the full path from root to this array\n\n 2. **Common Patterns by Source**:\n\n | Source Type | Common Paths | Example |\n |-------------|--------------|---------|\n | REST APIs | \"data\", \"results\", \"items\" | \"data\" |\n | Complex APIs | \"response.data\", \"data.items\" | \"response.data\" |\n | Database Exports | \"rows\", \"records\", \"exports\" | \"rows\" |\n | CRM Systems | \"contacts\", \"accounts\", \"opportunities\" | \"contacts\" |\n | Analytics APIs | \"data.rows\", \"response.data.rows\" | \"data.rows\" |\n\n 3. **Verification Approach**:\n - The path should resolve to an array (typically with square brackets in the JSON)\n - Each element in this array should represent one complete record\n - The array should not be a property array (like tags or categories)\n - |-\n ## IMPLEMENTATION EXAMPLES\n\n ### Root Array (No Path Needed)\n\n JSON Structure:\n ```json\n [\n {\"id\": 1, \"name\": \"Record 1\"},\n {\"id\": 2, \"name\": \"Record 2\"}\n ]\n ```\n\n Configuration:\n ```json\n \"resourcePath\": \"\" // or omit entirely\n ```\n\n ### Single-Level Nesting\n\n JSON Structure:\n ```json\n {\n \"orders\": [\n {\"id\": \"A001\", \"customer\": \"John\"},\n {\"id\": \"A002\", \"customer\": \"Jane\"}\n ],\n \"count\": 2\n }\n ```\n\n Configuration:\n ```json\n \"resourcePath\": \"orders\"\n ```\n\n ### Multi-Level Nesting\n\n JSON Structure:\n ```json\n {\n \"response\": {\n \"data\": {\n \"customers\": [\n {\"id\": 1, \"name\": \"Acme Corp\"},\n {\"id\": 2, \"name\": \"Globex Inc\"}\n ]\n },\n \"status\": \"success\"\n }\n }\n ```\n\n Configuration:\n ```json\n \"resourcePath\": \"response.data.customers\"\n ```\n - |-\n ## TROUBLESHOOTING INDICATORS\n\n If you encounter these issues, review the resourcePath setting:\n\n - Export completes successfully but processes 0 records\n - \"Cannot read property 'forEach' of undefined\" errors\n - \"Expected array but got object/string/number\" errors\n - Records appear flattened or with unexpected structure\n - |-\n ## BEST PRACTICES\n\n - Always verify the path with sample data before deployment\n - Use the simplest path that reaches the target array\n - Document the expected JSON structure alongside the configuration\n - For APIs with changing response structures, implement validation checks\n examples: [\"data.orders\", \"response.results\", \"items\", \"data.customers.records\"]\n xlsx:\n type: object\n description: |-\n Parsing settings for Microsoft Excel (.xlsx) workbooks. Configure when type is \"xlsx\".\n Calculated cell values are extracted rather than formulas; legacy .xls files are not\n supported — use the modern Open XML format.\n\n Required when type is xlsx.\n x-celigo-ai-guidance:\n - |-\n ## WHEN TO USE\n\n Configure this object when the `type` field is set to \"xlsx\". This configuration is required for properly parsing:\n - Modern Excel files (.xlsx) using the Open XML format\n - Excel workbooks with multiple sheets\n - Files exported from Microsoft Excel or compatible applications\n - Spreadsheet data with formatting, formulas, or multiple worksheets\n - |-\n Configuration settings for parsing Microsoft Excel (XLSX) files.\n This object defines how the system interprets and extracts data from Excel workbooks,\n handling their unique structures and formatting.\n - |-\n ## EXCEL PARSING CHARACTERISTICS\n\n - **Multiple Worksheets**: Can access data from specific sheets within workbooks\n - **Cell Formatting**: Handles various data types (text, numbers, dates, etc.)\n - **Formula Resolution**: Retrieves calculated values rather than formulas\n - **Data Extraction**: Converts tabular Excel data to structured records\n - |-\n ## IMPLEMENTATION STRATEGY FOR AI AGENTS\n\n 1. **File Analysis**:\n - Determine if the source file is an actual .xlsx format (not .xls, .csv, etc.)\n - Identify which worksheet contains the target data\n - Check for header rows, merged cells, or other special formatting\n - Note any preprocessing required (hidden rows, filtered data, etc.)\n\n 2. **Common Excel File Patterns**:\n\n ### Standard Data Table\n - Data organized in clear rows and columns\n - First row contains headers\n - No merged cells or complex formatting\n - Most straightforward to process\n\n ### Report-Style Workbook\n - Contains titles, headers, and possibly footers\n - May have merged cells for headings\n - Could have multiple tables on a single sheet\n - May require specific sheet selection or row skipping\n\n ### Multi-Sheet Workbook\n - Data distributed across multiple worksheets\n - May require multiple export configurations\n - Often needs sheet name specification (via pre-processing)\n - Common in financial or complex business reports\n - |-\n ## LIMITATIONS AND CONSIDERATIONS\n\n - **Hidden Data**: Hidden rows/columns are still processed unless filtered\n - **Formatting Loss**: Visual formatting and styles are ignored\n - **Formula Handling**: Only calculated values are extracted, not formulas\n - **Non-Tabular Data**: Pivot tables and non-tabular layouts may cause issues\n - **Large Files**: Very large Excel files may require additional memory\n - |-\n ## ERROR PREVENTION\n\n - **Format Compatibility**: Ensure the file is modern .xlsx format, not legacy .xls\n - **Data Structure**: Verify data is in a consistent tabular format\n - **Special Characters**: Watch for special characters in header rows\n - **Empty Sheets**: Check that target worksheets contain actual data\n - |-\n ## OPTIMIZATION OPPORTUNITIES\n\n - For complex workbooks, consider pre-processing to simplify structure\n - For large files, extract only necessary worksheets/ranges before processing\n - When possible, use files with consistent tabular layouts\n - Consider converting Excel data to CSV format for simpler processing\n properties:\n hasHeaderRow:\n type: boolean\n description: |-\n When true (the default), the first row is read as field names rather than data;\n blank header cells are auto-named and duplicate names are made unique with suffixes.\n When false, every row is treated as data and fields receive generic positional names\n (Column1, Column2, ...).\n x-celigo-ai-guidance:\n - |-\n ## BEHAVIOR\n\n - **When true** (default): First row is treated as field names, not data\n - **When false**: All rows including the first are treated as data records\n - Impacts field mapping, validation, and record counting\n - |-\n ## IMPLEMENTATION IMPACT\n\n ### With Header Row (true)\n\n - Field names from the header row can be referenced in mappings\n - Record count excludes the header row\n - First row of data is the second physical row in the spreadsheet\n - Column names are derived from the first row text values\n - Blank header cells may be auto-named (Column1, Column2, etc.)\n\n ### Without Header Row (false)\n\n - Fields are referenced by position/Excel column letters (A, B, C, etc.)\n - Record count includes all rows in the sheet\n - First row of data is the first physical row in the spreadsheet\n - Requires external schema or position-based mapping\n - All fields are given generic names (Column1, Column2, etc.)\n - Indicates whether the Excel file contains a header row with field names as the first row.\n - |-\n ## DETERMINATION STRATEGY FOR AI AGENTS\n\n To determine if a header row exists and should be configured:\n\n 1. **Visual Inspection**:\n - Open the Excel file and examine the first row\n - Look for descriptive labels rather than actual data values\n - Check for formatting differences between the first row and others\n - Header rows often use bold formatting or different background colors\n\n 2. **Content Analysis**:\n - Headers typically contain text while data rows may contain mixed types\n - Headers often use naming conventions (camelCase, Title Case, etc.)\n - Headers don't follow the pattern/format of subsequent data rows\n - Headers rarely contain numeric-only values (unless they're codes)\n\n 3. **Source Context**:\n - Business reports almost always include headers\n - Data exports from systems typically include column names\n - Machine-generated data might skip headers\n - Scientific or technical data sometimes omits headers\n - |-\n ## USAGE GUIDANCE FOR AI AGENTS\n\n ### Recommend `hasHeaderRow: true` when:\n\n 1. **Standard Business Data**:\n - Most business Excel files include headers\n - Reports and exports from business systems\n - Files intended for human readability\n - When column names provide important context\n\n 2. **Integration Requirements**:\n - When field names are needed for mapping\n - When data needs to be self-describing\n - When header names match target system fields\n - For maintaining field identity across systems\n\n ### Recommend `hasHeaderRow: false` when:\n\n 1. **Special Data Types**:\n - Scientific or sensor data without labels\n - Machine-generated output files\n - Legacy system exports with position-based fields\n - When all rows contain actual data values\n\n 2. **Technical Scenarios**:\n - When the first row contains required data\n - When column positions are used for mapping\n - When headers are inconsistent or misleading\n - For maximum data extraction with minimal configuration\n - |-\n ## IMPLEMENTATION NOTES\n\n - This setting affects all worksheets in multi-sheet processing\n - Excel column names with spaces or special characters may be normalized\n - Duplicate header names will be made unique with suffixes\n - Empty header cells will get automatically generated names\n - Maximum recommended header length: 64 characters\n - Consider pre-processing files without headers to add them for clarity\n examples: [true, false]\n xml:\n type: object\n description: |-\n Parsing settings for XML documents. Configure when type is \"xml\". resourcePath is an\n XPath expression selecting the elements treated as records; namespaces are handled\n automatically.\n\n Required when type is xml.\n x-celigo-ai-guidance:\n - |-\n ## WHEN TO USE\n\n Configure this object when the `type` field is set to \"xml\". This configuration is required for properly parsing:\n - Standard XML files (.xml)\n - SOAP API responses and web service outputs\n - Industry-specific XML formats (EDI, NIEM, UBL, etc.)\n - Document-oriented data with hierarchical structure\n - |-\n Configuration settings for parsing XML (Extensible Markup Language) files.\n This object defines how the system navigates and extracts hierarchical data from XML documents,\n enabling processing of structured markup data.\n - |-\n ## XML PARSING CHARACTERISTICS\n\n - **Hierarchical Structure**: Processes nested elements and attributes\n - **Schema Independence**: Works with or without formal XML schemas\n - **Node Selection**: Uses XPath to precisely target record elements\n - **Namespace Support**: Handles XML namespaces in complex documents\n - |-\n ## IMPLEMENTATION STRATEGY FOR AI AGENTS\n\n 1. **Document Analysis**:\n - Examine the XML structure to identify repeating elements (records)\n - Determine the hierarchical level where target records exist\n - Identify any namespaces that must be addressed\n - Note attributes vs. element content patterns\n\n 2. **Common XML Data Patterns**:\n\n ### Simple Element List\n ```xml\n \u003cRecords>\n \u003cRecord id=\"1\">\n \u003cName>Product 1\u003c/Name>\n \u003cPrice>10.99\u003c/Price>\n \u003c/Record>\n \u003cRecord id=\"2\">\n \u003cName>Product 2\u003c/Name>\n \u003cPrice>20.99\u003c/Price>\n \u003c/Record>\n \u003c/Records>\n ```\n - Records are identical element types with similar structure\n - Direct children of a container element\n - XPath: `/Records/Record`\n\n ### Namespaced XML\n ```xml\n \u003csoap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n \u003csoap:Body>\n \u003cns:GetCustomersResponse xmlns:ns=\"http://example.com/api\">\n \u003cns:Customer id=\"1\">\n \u003cns:Name>Acme Corp\u003c/ns:Name>\n \u003c/ns:Customer>\n \u003cns:Customer id=\"2\">\n \u003cns:Name>Globex Inc\u003c/ns:Name>\n \u003c/ns:Customer>\n \u003c/ns:GetCustomersResponse>\n \u003c/soap:Body>\n \u003c/soap:Envelope>\n ```\n - Elements use XML namespaces\n - Records are nested within service response structures\n - XPath: `//ns:Customer` or `/soap:Envelope/soap:Body/ns:GetCustomersResponse/ns:Customer`\n\n ### Heterogeneous Records\n ```xml\n \u003cFeed>\n \u003cEntry type=\"product\">\n \u003cProductId>123\u003c/ProductId>\n \u003cName>Widget\u003c/Name>\n \u003c/Entry>\n \u003cEntry type=\"category\">\n \u003cCategoryId>A5\u003c/CategoryId>\n \u003cLabel>Supplies\u003c/Label>\n \u003c/Entry>\n \u003c/Feed>\n ```\n - Same element type may have different internal structures\n - Usually identified by an attribute or child element type\n - May require multiple export configurations\n - XPath: `/Feed/Entry[@type=\"product\"]`\n - |-\n ## XPATH QUERY FORMULATION\n\n XPath is a powerful language for selecting nodes in XML documents. When formulating a resourcePath:\n\n - **Absolute Paths** (starting with `/`): Select from the document root\n - **Relative Paths** (no leading `/`): Select from the current context\n - **Any-Level Selection** (`//`): Select matching nodes regardless of location\n - **Predicates** (`[]`): Filter elements based on attributes or content\n - **Attribute Selection** (`@`): Select attribute values instead of elements\n - |-\n ## ERROR PREVENTION\n\n - **Invalid XPath**: Test the resourcePath against sample data before deployment\n - **Namespace Issues**: Ensure proper namespace handling in complex documents\n - **Empty Results**: Verify that the XPath selects the intended nodes and not an empty set\n - **Encoding Problems**: Use the correct encoding setting for international content\n - |-\n ## OPTIMIZATION OPPORTUNITIES\n\n - For large XML files, use more specific XPaths to reduce processing overhead\n - For complex structures, consider preprocessing to simplify before parsing\n - For SOAP responses, extract just the response body before processing\n - For repeating integration, document the exact XPath with examples\n properties:\n resourcePath:\n type: string\n description: |-\n XPath expression selecting the elements treated as records — each matching element\n becomes one record, with its child elements as fields. Required for XML parsing;\n there is no default. Use an absolute path (/Root/Order) when the structure is fixed,\n //Element to match at any depth, or predicates (//Element[@type=\"product\"]) to\n filter; XPath is case-sensitive, and a non-matching path yields zero records.\n x-celigo-ai-guidance:\n - |-\n Specifies the XPath expression used to locate record elements within the XML document.\n This critical field determines which XML nodes are treated as individual records for processing.\n - |-\n ## BEHAVIOR\n\n - **Purpose**: Identifies which elements in the XML represent individual records\n - **Format**: Uses XPath syntax to select nodes from the document structure\n - **Requirement**: MANDATORY for XML processing - no default value exists\n - **Result**: Each XML element matching the XPath is processed as one record\n - |-\n ## XPATH SYNTAX GUIDANCE\n\n ### Core XPath Patterns\n\n 1. **Direct Child Selection** (`/Root/Element`):\n ```xml\n \u003cRoot>\n \u003cElement>Record 1\u003c/Element>\n \u003cElement>Record 2\u003c/Element>\n \u003c/Root>\n ```\n - XPath: `/Root/Element`\n - Selects elements that are direct children following exact path\n - Most precise, requires exact hierarchy knowledge\n - Recommended when structure is consistent and well-known\n\n 2. **Any-Level Selection** (`//Element`):\n ```xml\n \u003cRoot>\n \u003cSection>\n \u003cElement>Record 1\u003c/Element>\n \u003c/Section>\n \u003cContainer>\n \u003cElement>Record 2\u003c/Element>\n \u003c/Container>\n \u003c/Root>\n ```\n - XPath: `//Element`\n - Selects all matching elements regardless of location\n - More flexible, works across varying structures\n - Use when element hierarchy may vary or is unknown\n\n 3. **Filtered Selection** (`//Element[@attr=\"value\"]`):\n ```xml\n \u003cRoot>\n \u003cElement type=\"product\">Record 1\u003c/Element>\n \u003cElement type=\"category\">Not a record\u003c/Element>\n \u003cElement type=\"product\">Record 2\u003c/Element>\n \u003c/Root>\n ```\n - XPath: `//Element[@type=\"product\"]`\n - Selects only elements matching both name and attribute criteria\n - Precise targeting when elements have identifying attributes\n - Useful for heterogeneous XML with type indicators\n\n ### Advanced Selection Techniques\n\n 1. **Position-Based** (`/Root/Element[1]`):\n - Selects first element only\n - Use when only certain occurrences should be processed\n\n 2. **Content-Based** (`//Element[contains(text(),\"Value\")]`):\n - Selects elements containing specific text\n - Useful for filtering based on content\n\n 3. **Parent-Relative** (`//Parent[Child=\"Value\"]/Element`):\n - Selects elements with specific sibling or parent conditions\n - Powerful for complex structural conditions\n - |-\n ## NAMESPACE HANDLING\n\n When working with namespaced XML:\n\n ```xml\n \u003csoap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\"\n xmlns:ns=\"http://example.com/api\">\n \u003csoap:Body>\n \u003cns:Response>\n \u003cns:Customer>Record 1\u003c/ns:Customer>\n \u003cns:Customer>Record 2\u003c/ns:Customer>\n \u003c/ns:Response>\n \u003c/soap:Body>\n \u003c/soap:Envelope>\n ```\n\n The system automatically handles namespaces, but for clarity and precision:\n\n 1. **Namespace-Aware Path**:\n - XPath: `/soap:Envelope/soap:Body/ns:Response/ns:Customer`\n - Include namespace prefixes as they appear in the document\n\n 2. **Namespace-Agnostic Path**:\n - XPath: `//Customer` or `//*[local-name()=\"Customer\"]`\n - Use when you want to ignore namespaces entirely\n - |-\n ## DETERMINATION STRATEGY FOR AI AGENTS\n\n 1. **Identify Record Elements**:\n - Look for repeating elements that represent individual \"rows\" of data\n - These elements typically have the same name and similar structure\n - They often contain multiple child elements representing \"fields\"\n\n 2. **Analyze Element Hierarchy**:\n - Note the path from root to record elements\n - Determine if records appear at consistent locations or vary\n - Check if they need to be filtered by attributes or position\n\n 3. **Test Path Specificity**:\n - More specific paths reduce processing overhead but are less flexible\n - More general paths (with `//`) are robust to structure changes but less efficient\n - Balance specificity with flexibility based on source stability\n - |-\n ## COMMON XPATH PATTERNS BY SOURCE\n\n | Source Type | Common XPath Pattern | Example |\n |-------------|----------------------|---------|\n | SOAP APIs | `/Envelope/Body/*/Response/*` | `/soap:Envelope/soap:Body/ns:GetOrdersResponse/ns:Order` |\n | REST XML | `/Response/Results/*` | `/ApiResponse/Results/Customer` |\n | Feeds | `/Feed/Entry` or `/Feed/Item` | `/rss/channel/item` |\n | Documents | `//Section/Item` | `//Chapter/Paragraph` |\n | EDI/Business | `/Document/Transaction/Line` | `/Invoice/LineItems/Item` |\n - |-\n ## TROUBLESHOOTING INDICATORS\n\n If you encounter these issues, review the resourcePath:\n\n - Export completes successfully but processes 0 records\n - Records contain unexpected or partial data\n - Only first level of data is extracted (missing nested content)\n - Namespace-related \"element not found\" errors\n - |-\n ## IMPLEMENTATION NOTES\n\n - XPath is case-sensitive; element and attribute names must match exactly\n - Each matching element becomes a separate record for processing\n - Child elements become fields in the processed record\n - Attributes can be included in field data if needed\n - Namespaces are handled automatically but may require explicit prefixes\n - Testing with an XPath tool on sample data is highly recommended\n examples: [\"/Root/Orders/Order\", \"//Order\", \"//Invoice\", \"/Customers/Customer\", \"//Products/Product\"]\n includeParentData:\n $ref: './record-extraction.yml#/IncludeParentData'\n fileDefinition:\n type: object\n description: |-\n Parsing via a predefined file definition resource, for formats the standard parsers\n cannot handle — fixed-width files, EDI documents (X12, EDIFACT), and multi-record-type\n or proprietary formats. Configure when type is \"filedefinition\".\n\n Required when type is filedefinition.\n x-celigo-ai-guidance:\n - |-\n ## WHEN TO USE\n\n Configure this object when the `type` field is set to \"filedefinition\". This approach is required for properly handling:\n - Legacy or proprietary file formats with complex structures\n - Fixed-width text files where field positions are defined by character positions\n - Electronic Data Interchange (EDI) documents (X12, EDIFACT, etc.)\n - Multi-record type files where different lines have different formats\n - Files requiring complex preprocessing or custom parsing logic\n - |-\n Configuration settings for parsing files using a predefined file definition.\n This object enables processing of complex, non-standard,\n or proprietary file formats that require specialized parsing logic beyond what\n the standard parsers (CSV,\n JSON, XML, etc.) can handle.\n - |-\n ## FILE DEFINITION CHARACTERISTICS\n\n - **Custom Parsing Rules**: Applies predefined parsing logic to complex file formats\n - **Reusable Configurations**: References externally defined parsing rules that can be reused\n - **Complex Format Support**: Handles formats that standard parsers cannot process\n - **Specialized Processing**: Often used for industry-specific or legacy formats\n - |-\n ## IMPLEMENTATION STRATEGY FOR AI AGENTS\n\n 1. **Format Analysis**:\n - Determine if the file format is standard (CSV, JSON, XML) or requires custom parsing\n - Check if the format follows industry standards like EDI, SWIFT, or fixed-width\n - Assess if there are multiple record types within the same file\n - Identify if specialized logic is needed to interpret the file structure\n\n 2. **File Definition Selection**:\n - Verify that a suitable file definition has already been created in the system\n - Check if existing file definitions match the format requirements\n - Confirm the file definition ID from system administrators if needed\n - Ensure the file definition is compatible with the export's needs\n - |-\n ## USE CASE SCENARIOS\n\n ### Fixed-Width Files\n\n Files where each field has a specific starting position and length:\n ```\n CUST00001JOHN DOE 123 MAIN ST\n CUST00002JANE SMITH 456 OAK AVE\n ```\n - Fields are positioned by character count rather than delimiters\n - Requires precise position and length definitions\n - Common in legacy mainframe and banking systems\n\n ### EDI Documents\n\n Electronic Data Interchange formats for business transactions:\n ```\n ISA*00* *00* *ZZ*SENDER *ZZ*RECEIVER *...\n GS*PO*SENDER*RECEIVER*20210101*1200*1*X*004010\n ST*850*0001\n BEG*00*SA*123456**20210101\n ...\n ```\n - Highly structured with segment identifiers and element separators\n - Contains multiple record types with different structures\n - Requires complex parsing rules and validation\n\n ### Multi-Record Files\n\n Files containing different record types identified by indicators:\n ```\n H|SHIPMENT|20210115|PRIORITY\n D|ITEM001|5|WIDGET|RED\n D|ITEM002|10|GADGET|BLUE\n T|2|15|COMPLETE\n ```\n - Each line starts with a record type indicator\n - Different record types have different field structures\n - Requires conditional processing based on record type\n - |-\n ## ERROR PREVENTION\n\n - **Definition Mismatch**: Ensure the file definition matches the actual file format\n - **Missing Definition**: Verify the file definition exists before referencing it\n - **Access Issues**: Confirm the integration has permission to use the file definition\n - **Version Compatibility**: Check if file definition version matches current file format\n - |-\n ## OPTIMIZATION OPPORTUNITIES\n\n - Document which file definition is used and why it's appropriate for the file format\n - Consider creating purpose-specific file definitions for complex formats\n - Test file definitions with sample files before deploying in production\n - Maintain documentation of the file structure alongside the file definition reference\n properties:\n _fileDefinitionId:\n type: string\n format: objectId\n x-celigo-refModel: filedefinitions\n description: |-\n File definition resource containing the parsing rules for this format. Must\n reference an existing, accessible file definition — never guess or fabricate the ID.\n Definitions are reusable across exports, so changing one affects every export that\n uses it.\n x-celigo-ai-guidance:\n - |-\n The unique identifier of the file definition to use for parsing the file.\n This ID references a preconfigured file definition resource that contains the\n detailed parsing instructions for a specific file format.\n - |-\n ## FIELD BEHAVIOR\n\n - **Purpose**: References an existing file definition resource in the system\n - **Format**: MongoDB ObjectId (24-character hexadecimal string)\n - **Requirement**: MANDATORY when type=\"filedefinition\"\n - **Validation**: Must reference a valid, accessible file definition\n - |-\n ## UNDERSTANDING FILE DEFINITIONS\n\n A file definition is a separate resource that defines:\n\n 1. **Record Structure**:\n - Field names, positions, and data types\n - Record identifiers and format specifications\n - Parsing rules and field extraction logic\n\n 2. **Processing Rules**:\n - How to identify different record types\n - How to handle headers, footers, and details\n - Data validation and transformation rules\n\n 3. **Format-Specific Settings**:\n - For fixed-width: Character positions and field lengths\n - For EDI: Segment identifiers and element separators\n - For proprietary formats: Custom parsing instructions\n - |-\n ## OBTAINING THE CORRECT ID\n\n To identify the appropriate file definition ID:\n\n 1. **System Administration**:\n - Check with system administrators for a list of available file definitions\n - Request the specific ID for the file format you need to process\n - Verify the file definition's compatibility with your file format\n\n 2. **File Definition Catalog**:\n - If available, consult the file definition catalog in the system\n - Search for definitions matching your file format requirements\n - Note the ObjectId of the appropriate definition\n\n 3. **Custom Definition Creation**:\n - If no suitable definition exists, request creation of a new one\n - Provide sample files and format specifications\n - Obtain the new file definition's ID after creation\n - |-\n ## IMPLEMENTATION GUIDANCE FOR AI AGENTS\n\n ### Recommendation Framework\n\n When implementing a file definition-based export:\n\n 1. **Verify Definition Existence**:\n - Confirm the file definition exists before configuration\n - Do not guess or generate random IDs\n - Request specific ID from system administrators\n\n 2. **Documentation Requirements**:\n - Document which file definition is being used and why\n - Note any specific requirements or limitations of the definition\n - Record the mapping between file fields and integration needs\n\n 3. **Testing Approach**:\n - Recommend testing with sample files before production use\n - Verify all required fields are correctly extracted\n - Validate the parsing results meet integration requirements\n\n ### Common File Definition Categories\n\n | Category | Description | Example Formats |\n |----------|-------------|----------------|\n | Fixed-Width | Fields defined by character positions | Banking transactions, government reports |\n | EDI | Electronic Data Interchange standards | X12, EDIFACT, TRADACOMS |\n | Hierarchical | Complex parent-child structures | Specialized industry formats |\n | Multi-Record | Different record types in one file | Inventory systems, financial exports |\n | Proprietary | Custom or legacy system formats | Mainframe exports, specialized software |\n - |-\n ## TECHNICAL CONSIDERATIONS\n\n - File definitions are reusable across multiple exports\n - Changes to a file definition affect all exports using it\n - File definitions may have version dependencies\n - Some file definitions may require specific pre-processing settings\n - Performance impact varies based on definition complexity\n - |-\n ## TROUBLESHOOTING INDICATORS\n\n If you encounter these issues, verify the file definition ID:\n\n - \"File definition not found\" errors\n - Unexpected field mapping or missing fields\n - Data type conversion errors\n - Parsing failures with specific record types\n\n Always document the exact file definition ID with its purpose to facilitate troubleshooting and maintenance.\n examples: [\"60a2c4e6f321d800129a1a3c\", \"5f8d43a1b9e5a80011a35f2c\", \"61b7d2e8c912e500118c4d9f\"]\n allowPartialSuccess:\n type: boolean\n description: |-\n When true, a file whose records partially fail parsing against the file definition\n still emits the records that did parse, instead of failing the entire file. Applies to\n file-definition (fixed-width, EDI) parsing only. For X12 and EDIFACT the unit is the\n transaction set (ST–SE, UNH–UNT), and it takes effect only with an EDI profile linked:\n a failed set is set aside and reported as an error while the file's other sets go on\n as records. Without this flag, one failed set fails the whole file.\n examples: [true, false]\n filter:\n description: |-\n Selects which files are processed: files matching the filter criteria are exported,\n and non-matching files are skipped entirely before processing begins. Filterable\n fields come from each file's listing entry: the name as `fileName`, the size as\n `filesize`, and the modified time as `lastModifiedTime` on FTP, SFTP and S3 or\n `modifiedTime` on cloud-storage folders (`http.type: \"file\"`)\n (e.g. [\"endswith\", [\"string\", [\"extract\", \"fileName\"]], \".csv\"]). There is no\n `lastmodified` key.\n allOf:\n - $ref: '../common-schemas/filter.yml#/Filter'\n x-celigo-ai-guidance:\n - |-\n ## AVAILABLE FILTER FIELDS\n\n The fields a file filter reads are on each file's listing entry, before the file is\n downloaded. They are not the records' `fileMeta`, though some names match.\n\n ### Common Filter Fields\n\n These are the most commonly available fields across most file providers:\n\n 1. **fileName**: The name of the file (with extension)\n - Example filter: Match files with specific extensions or naming patterns\n - Usage: `[\"endswith\", [\"string\", [\"extract\", \"fileName\"]], \".csv\"]`\n\n 2. **filesize**: The size of the file in bytes\n - Example filter: Skip files that are too large or too small\n - Usage: `[\"lessthan\", [\"number\", [\"extract\", \"filesize\"]], 1000000]`\n\n 3. **lastModifiedTime** (FTP, SFTP, S3) or **modifiedTime** (cloud-storage folders): The\n last modification timestamp of the file. There is no `lastmodified` key.\n - Example filter: Process only files created/modified within a specific date range\n - Usage (FTP, SFTP, S3): `[\"greaterthan\", [\"epochtime\", [\"extract\", \"lastModifiedTime\"]], 1672531200000]` (2023-01-01T00:00:00Z in epoch milliseconds)\n - Usage (cloud-storage folders): `[\"greaterthan\", [\"epochtime\", [\"extract\", \"modifiedTime\"]], 1672531200000]`\n backupPath:\n type: string\n description: |-\n Path where backup copies of source files are stored before processing.\n x-celigo-ai-guidance:\n - |-\n The file system path where backup files will be stored before processing.\n This path specifies a directory location where the system will create backup\n copies of files before they are processed by the export flow.\n - |-\n ## BACKUP MECHANISM OVERVIEW\n\n The backup mechanism creates a copy of source files in the specified location before processing begins. This provides:\n\n - **Data Safety**: Preserves original files in case of processing errors\n - **Audit Trail**: Maintains historical record of exported data\n - **Recovery Option**: Enables reprocessing from original files if needed\n - **Compliance Support**: Helps meet data retention requirements\n - |-\n ## PATH CONFIGURATION GUIDELINES\n\n The path format must follow these conventions:\n\n - **Absolute Paths**: Must start with \"/\" (Unix/Linux) or include drive letter (Windows)\n - **Relative Paths**: Interpreted relative to the application's working directory\n - **Network Paths**: Can use UNC format (\\\\server\\share\\path) or mounted network drives\n - **Access Requirements**: The path must be writable by the service account running the integration\n - |-\n ## IMPLEMENTATION STRATEGY FOR AI AGENTS\n\n When configuring the backup path, consider these factors:\n\n 1. **Storage Capacity Planning**:\n - Estimate average file sizes and volumes\n - Calculate required storage based on retention period\n - Implement monitoring for storage utilization\n - Plan for storage growth based on business projections\n\n 2. **Path Selection Criteria**:\n - Choose locations with sufficient disk space\n - Ensure appropriate read/write permissions\n - Select paths with reliable access (avoid temporary or volatile storage)\n - Consider network latency for remote locations\n\n 3. **Backup Naming Convention**:\n - Default: Original filename with timestamp suffix\n - Custom: Can be controlled through integration settings\n - Avoid paths that may contain special characters that need escaping\n - Consider filename length limitations of target filesystem\n\n 4. **Security Considerations**:\n - Restrict access to backup location to authorized personnel only\n - Avoid public-facing directories\n - Consider encryption for sensitive data backups\n - Implement appropriate file permissions\n - |-\n ## BACKUP STRATEGY RECOMMENDATIONS\n\n | Data Sensitivity | Recommended Approach | Path Considerations |\n |------------------|----------------------|---------------------|\n | Low | Local directory backup | Fast access, limited protection |\n | Medium | Network share with permissions | Balanced access/protection |\n | High | Secure storage with encryption | Highest protection, potential performance impact |\n | Regulated | Compliant storage with audit trail | Must meet specific regulatory requirements |\n - |-\n ## INTEGRATION PATTERNS\n\n ### Temporary Processing Pattern\n\n For short-term processing needs:\n ```\n /tmp/exports/backups\n ```\n - Files stored temporarily during processing\n - Limited retention period\n - Optimized for processing speed\n - May be automatically cleaned up\n\n ### Long-term Archival Pattern\n\n For regulatory or business retention requirements:\n ```\n /archive/exports/2023/Q4\n ```\n - Organized by time period\n - Structured for easy retrieval\n - May include additional metadata\n - Designed for long-term storage\n\n ### Cloud Storage Pattern\n\n For scalable, managed storage:\n ```\n /mnt/cloud/exports/client123\n ```\n - Mounted cloud storage location\n - Potentially unlimited capacity\n - May include built-in versioning\n - Often includes automatic replication\n - |-\n ## ERROR HANDLING GUIDANCE\n\n When configuring backup paths, anticipate these common issues:\n\n - **Permission Denied**: Ensure service account has write access\n - **Path Not Found**: Verify directory exists or create it programmatically\n - **Disk Full**: Monitor storage capacity and implement alerts\n - **Path Too Long**: Be aware of filesystem path length limitations\n - |-\n ## TECHNICAL CONSIDERATIONS\n\n - Backup operations may impact performance for large files\n - Network paths may introduce latency and availability concerns\n - Some filesystems have case sensitivity differences (important for path matching)\n - Path separators vary by platform (/ vs \\)\n - Special characters in paths may require escaping in certain contexts\n - Consider implementing automatic cleanup policies for backups\n - |-\n ## SYSTEM ADMINISTRATION NOTES\n\n - Backup paths should be included in system backup procedures\n - Monitor space utilization on backup volumes\n - Implement appropriate retention policies\n - Document backup path locations in system configuration\n - Consider periodic validation of backup file integrity\n examples: [\"/var/backups/exports\", \"C:\\\\integrations\\\\backups\", \"/mnt/nas/backups/clients/acme\", \"./backups\", \"~/integrations/backups\"]\n includeSubdirectories:\n type: boolean\n default: false\n x-celigo-ui-override: >-\n Defaulted by the Azure Data Lake Storage Gen2 export form\n (fieldDefinitions/resources/exports/azuredatalakestoragegen2.js: 'file.includeSubdirectories'\n defaultValue r?.file?.includeSubdirectories === true, visible only for that assistant).\n The API stores the flag on any file export; the form offers it for ADLS Gen2 only.\n description: |-\n When true, an Azure Data Lake Storage Gen2 transfer (`assistant: azuredatalakestoragegen2`,\n `http.type: file`) also picks up files in the subdirectories below the directory path in\n `http.relativeURI`; when false or omitted only that directory's own files are transferred.\n examples: [true, false]\n directory:\n type: object\n required:\n - pathMode\n description: |-\n Structured source location for cloud file-provider sources (Google Drive shared drives,\n Box, Dropbox, Google Cloud Storage, Azure Data Lake Storage Gen2), replacing the flat path\n used by classic FTP/S3 sources. Select the location by folder ID or by a path relative to\n a configured storage root via pathMode. For the drive-style providers, directoryId mode\n addresses the folder by id alone — the flat relative-path field is not consulted at\n runtime. Google Cloud Storage puts the bucket in `storageRootId` and the object prefix in\n `id` with `pathMode: directoryId`; Azure Data Lake Storage Gen2 puts the container in\n `name` and the directory path in `http.relativeURI`.\n x-celigo-ai-guidance:\n - >-\n Save-time validation accepts any string for id without checking it against the\n provider — set it only from a folder ID the user supplied or the UI folder browser\n stored; never fabricate one. To address a folder by path instead, set pathMode to\n relativePath and put the path in the adaptor's flat path field.\n - >-\n `id` and `pathMode` are validated as a pair: `pathMode: directoryId` without `id` fails\n with 422 `missing_required_field` (\"File exports require the file.directory.id to have a\n value.\"), and `id` with any other pathMode fails with 422 `missing_required_field`\n (\"File exports require the file.directory.pathMode to have a value.\").\n properties:\n pathMode:\n type: string\n enum: [\"relativePath\", \"directoryId\"]\n x-enumDescriptions:\n relativePath: Address the location by a path relative to the storage root identified by storageRootId.\n directoryId: Address the location directly by the provider's folder ID set in id.\n description: How the directory is addressed — by provider folder ID, or by a path relative to a configured storage root.\n examples: [\"relativePath\", \"directoryId\"]\n id:\n type: string\n description: |-\n Provider-native folder ID of the source directory (e.g. a Google Drive folder ID); set\n when pathMode is directoryId. This is the file provider's own ID, not a Celigo resource ID.\n For Google Cloud Storage it is the object prefix inside the bucket, ending with `/`.\n examples: [\"1KppvTTr4jKhJqwPAhGWN6SRwk5m45DL2\", \"inbound/orders/\"]\n name:\n type: string\n description: |-\n Display name of the folder identified by id, kept for readability in the UI; the\n folder browser fills it when the location is picked visually. For Azure Data Lake\n Storage Gen2 it is the authoritative container name instead — the form saves the\n container here and no id.\n examples: [\"Invoices - Inbound\", \"invoices-inbound\"]\n storageRootId:\n type: string\n description: |-\n Provider-native ID of the storage root (e.g. a Google Drive shared-drive ID) the relative\n path resolves against; set when pathMode is relativePath. The provider's own ID, not a Celigo ID.\n For Google Cloud Storage it is the bucket name.\n examples: [\"1qU1hvFarBFMb4-twqKm1rR5o_6j9vs9g\", \"erp-exports\"]\n storageRootName:\n type: string\n description: Display name of the storage root identified by storageRootId, retained for reference in the UI.\n examples: [\"Shared drive - Finance\"]\n backupDirectory:\n type: object\n required:\n - pathMode\n description: |-\n Structured destination for backup copies of source files on cloud file providers — the\n file-provider counterpart to backupPath. Addressed the same way as directory.\n properties:\n pathMode:\n type: string\n enum: [\"relativePath\", \"directoryId\"]\n x-enumDescriptions:\n relativePath: Address the backup location by a path relative to the storage root identified by storageRootId.\n directoryId: Address the backup location directly by the provider's folder ID set in id.\n description: How the backup location is addressed — by provider folder ID, or by a path relative to a configured storage root.\n examples: [\"relativePath\", \"directoryId\"]\n id:\n type: string\n description: Provider-native folder ID of the backup directory; set when pathMode is directoryId.\n examples: [\"1KppvTTr4jKhJqwPAhGWN6SRwk5m45DL2\"]\n name:\n type: string\n description: |-\n Display name of the backup folder identified by id, kept for readability in the UI;\n the folder browser fills it when the location is picked visually.\n examples: [\"Processed archive\"]\n storageRootId:\n type: string\n description: Provider-native ID of the storage root the backup relative path resolves against; set when pathMode is relativePath.\n examples: [\"1qU1hvFarBFMb4-twqKm1rR5o_6j9vs9g\"]\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":126117,"content_sha256":"2d341703e09e6f1aa582624a36b4c5f023c96509d68fe68943480b67f9625adb"},{"filename":"references/schemas/filesystem.yml","content":"FileSystem:\n type: object\n description: |-\n Defines which files to read from a local or mounted folder on the host running the\n on-premise agent. Required when the _connectionId field references a file-system (on-premise)\n connection; must not be included for other connection types.\n x-celigo-ai-guidance:\n - Configuration for FileSystem exports\n required:\n - directoryPath\n properties:\n directoryPath:\n type: string\n x-celigo-agent:\n - handlebars\n description: |-\n Folder on the on-premise agent's host to read files from; the agent's OS account must\n have read permission on it. Accepts a local OS path or a UNC network share, and supports\n handlebars templates for dynamic folders.\n x-celigo-ai-guidance:\n - Directory path to retrieve files from (required)\n examples: [\"C:\\\\Celigo\\\\Inbound\", \"D:\\\\exports\\\\orders\", \"\\\\\\\\NAS\\\\Accounting\\\\AP\"]\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":929,"content_sha256":"de5455828c18b4fb66dd3fe05e7162341077dd1834a07e6c6868939a508951a0"},{"filename":"references/schemas/ftp.yml","content":"FTP:\n type: object\n description: |-\n Defines which files to retrieve from an FTP, FTPS, or SFTP server. Required when the\n _connectionId field references an FTP/SFTP connection; must not be included for other\n connection types. directoryPath selects the folder, fileNameStartsWith/fileNameEndsWith\n filter files by name, and backupDirectoryPath controls where files are moved after\n retrieval.\n x-celigo-ai-guidance:\n - Configuration object for FTP/SFTP connection settings in export integrations.\n - |-\n The FTP export object has the following requirements:\n\n - Required fields: directoryPath\n - Optional fields: fileNameStartsWith, fileNameEndsWith, backupDirectoryPath, _tpConnectorId\n - |-\n ## PURPOSE\n\n This configuration specifies:\n - Which directory to retrieve files from\n - How to filter files by name patterns\n - Where to move files after retrieval (optional)\n - Any trading partner-specific connection settings\n required:\n - directoryPath\n properties:\n _tpConnectorId:\n type: string\n format: objectId\n x-celigo-refModel: tradingpartnerconnectors\n description: |-\n Trading Partner Connector that supplies partner-specific B2B settings for this export.\n When set, the export inherits the connector's pre-configured settings; omit to use only\n the FTP connection details.\n x-celigo-ai-guidance:\n - References a Trading Partner Connector for standardized B2B integrations.\n - |-\n ## FIELD BEHAVIOR\n\n This field links to pre-configured trading partner settings:\n\n - OPTIONAL: If omitted, uses only the FTP connection details\n - References a Celigo Trading Partner Connector by _id\n - When specified, inherits partner-specific configurations\n examples: [\"60a2c4e6f321d800129a1a3c\", \"5f8e7d2c1b54e6003a9c7e12\"]\n directoryPath:\n type: string\n x-celigo-agent:\n - handlebars\n x-celigo-ui-override: >-\n Required by the FTP export form (fieldDefinitions/resources/exports/ftp.js: ftp.directoryPath\n is required:true). Encoded to mirror the form so builders produce connectable configurations.\n description: |-\n Directory on the server to retrieve files from, either absolute or relative to the login\n directory; the FTP user must have read permission on it. Use forward slashes regardless\n of server OS — paths are case-sensitive on UNIX/Linux servers. Supports handlebars\n templates, e.g. archive/{{timestamp 'YYYY-MM-DD'}}.\n examples: [\"users/dave\", \"incoming/orders\", \"/var/ftp/public\", \"EDI/850\"]\n fileNameStartsWith:\n type: string\n x-celigo-agent:\n - handlebars\n description: |-\n Only retrieves files whose names start with this value (case-sensitive on most servers);\n accepts static text or handlebars templates. When combined with fileNameEndsWith, files\n must match both.\n x-celigo-ai-guidance:\n - Optional prefix filter for filenames.\n - |-\n - Filters files based on starting characters\n - Case-sensitive on most FTP servers\n - Can use static text or handlebars templates\n - Examples:\n - `\"ORDER_\"` - matches ORDER_123.csv but not order_123.csv\n - `\"INV_{{timestamp 'YYYYMMDD'}}\"` - matches current date's invoices\n examples: [\"ORDER_\", \"INV_\", \"EXPORT_\", \"PO_{{timestamp 'YYYYMMDD'}}\"]\n fileNameEndsWith:\n type: string\n x-celigo-agent:\n - handlebars\n description: |-\n Only retrieves files whose names end with this value, commonly a file extension\n (case-sensitive on most servers). When combined with fileNameStartsWith, files must\n match both.\n x-celigo-ai-guidance:\n - Optional suffix filter for filenames.\n - |-\n - Commonly used to filter by file extension\n - Case-sensitive on most FTP servers\n - Examples:\n - `\".csv\"` - retrieves only CSV files\n - `\"_FINAL.xml\"` - retrieves only XML files with _FINAL suffix\n - `\"_READY\"` - retrieves files with status indicator\n examples: [\".csv\", \".xml\", \".json\", \"_READY\", \"_FINAL.txt\"]\n backupDirectoryPath:\n type: string\n description: |-\n Directory on the same server where files are moved after successful export, giving you\n an independent backup; if omitted, files are simply deleted from the source directory\n after successful export unless `file.skipDelete` is true (Celigo also keeps its own copy\n of processed files for a set period). Supports static paths or handlebars templates.\n x-celigo-ai-guidance:\n - Optional directory where files are moved before deletion.\n - |-\n IMPORTANT:\n Celigo automatically deletes files from the source directory after successful export,\n unless `file.skipDelete` is true. Moving them to a backup directory also takes them out\n of the source directory, so when another system still reads them there, set\n `file.skipDelete: true` instead.\n The backup directory is for users who want to maintain their own independent\n backup of exported files.\n Celigo also maintains its own backup of processed files for a set period of time.\n - |-\n IMPORTANT:\n Celigo automatically deletes files from the source directory after successful export,\n unless `file.skipDelete` is true. Moving them to a backup directory also takes them out\n of the source directory, so when another system still reads them there, set\n `file.skipDelete: true` instead.\n The backup directory is for users who want to maintain their own independent\n backup of exported files.\n Celigo also maintains its own backup of processed files for a set period of time.\n examples: [\"processed\", \"archive\", \"backup/{{timestamp 'YYYY-MM'}}\", \"/var/ftp/archive\"]\n x-celigo-agent:\n - handlebars\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":6090,"content_sha256":"fdf761427c3c020786beb62d8f52a850092e21c92773f0113d0758cbc67dea2d"},{"filename":"references/schemas/group-by.yml","content":"GroupBy:\n type: array\n description: |\n Specifies which fields to use for grouping records in the export results. When configured, records with\n the same values in these fields will be grouped together and treated as a single record by downstream\n steps in your flow.\n\n For example:\n - Group sales orders by customer ID to process all orders for each customer together\n - Group journal entries by accounting period to consolidate related transactions\n - Group inventory items by location to process inventory by warehouse\n\n When grouping is used, the export's page size determines the maximum number of groups per page, not individual\n records. Note that effective grouping typically requires that records with the same group field values appear\n together in the export data.\n\n Not a webhook listener feature: on a `webhook` export the platform applies it only within each page, after\n the request has been paged, and only to adjacent rows — the field is not offered for listeners in the UI.\n x-celigo-ai-guidance:\n - |-\n Only adjacent rows with equal values merge — on an HTTP export and on a database export\n alike, a row for key A, then key B, then key A again yields three groups. The platform's\n `sortByFields` setting exists on file exports (`file.sortByFields`) and is ignored as a\n top-level field on an HTTP or database export, so those sources must return same-key rows\n already adjacent — on a database export the query's `ORDER BY` is what turns the three\n groups into two.\n - |-\n Not accepted on a NetSuite export: the platform rejects it with `[invalid_config]\n netsuite connection is not supported for groupByFields property`. A NetSuite saved\n search groups through `netsuite.skipGrouping` (the UI's *Group rows*), keyed on the\n record's internal id.\n - |-\n In `POST /v1/exports/preview` the grouped records are the `group` stage that follows\n `parse` (mirrored in top-level `data`); the `parse` stage still holds the ungrouped\n rows, so read `group` when it is present.\n - |-\n Never set this on a webhook listener to keep a request's rows together —\n the listener pages the request's records at ``pageSize`` FIRST and then\n merges only adjacent same-key rows inside each page, so rows that cross a\n page boundary still arrive as separate records and unsorted rows are not\n grouped at all. Keep a listener's request together by having the sender\n post an object (one record) or by raising the listener's ``pageSize``.\n - |-\n A grouped record's wire shape is a BARE ARRAY of the grouped rows\n (``[{row}, {row}]``) — there is no ``rows`` key on the record\n itself. ``rows`` is the NAME Handlebars and editor contexts bind\n array-records to (templates reference ``{{rows.0.\u003cfield>}}`` /\n iterate ``{{#each rows}}``); hook code sees the array as the\n record element and must not assume object shape.\n - |-\n A grouped record is ONE record: Mapper 2.0 and Transform 2.0 root\n JSONPath at the array (``$[0].\u003cfield>`` / ``$[*].\u003cfield>``;\n ``$.\u003cfield>`` resolves nothing), and a Transform 2.0 in ``mode:\n \"create\"`` reshapes it in place — header fields from ``$[0]``, a\n lines array from a ``buildArrayHelper`` over ``$[*]`` (``mode:\n \"modify\"`` has no object to overlay and passes the array through\n untouched). Changing how many records the page carries — each\n row becoming its own record for every downstream step — is the\n ``preSavePage`` hook's job alone.\n items:\n type: string\n examples: [[\"customerId\", \"orderId\"], [\"accountId\", \"locationId\"], [\"journalId\"], [\"department\", \"class\"]]","content_type":"application/yaml; charset=utf-8","language":"yaml","size":3745,"content_sha256":"c3fbd744b9208bb4f0f85dcdcced3464a548ba490a81b6f8e5e4fd18c4717f70"},{"filename":"references/schemas/http.yml","content":"Http:\n type: object\n description: |-\n Configuration for HTTP exports. Required when the export's `_connectionId` references a\n connection of type `http`, and on `NetSuiteHTTPExport` (NetSuite's REST APIs on a `netsuite`\n connection, where `nsDomainType` supplies the host).\n x-celigo-ai-guidance:\n - |-\n IMPORTANT:\n When the _connectionId field points to a connection where the type is http,\n this object MUST be populated for the export to function properly.\n This is a required configuration for all HTTP based exports,\n as determined by the connection associated with the export.\n - |-\n A `NetSuiteHTTPExport` uses this same object on a netsuite connection: `relativeURI`\n carries the full path from `/services/rest/...` (SuiteTalk) or\n `/app/site/hosting/restlet.nl?script=\u003cid>&deploy=\u003cn>` (custom RESTlet), the host and the\n token signature come from the connection, and `type` must stay unset (file, blob and\n csvstream are rejected with 422 unsupported_type).\n properties:\n type:\n type: string\n enum:\n - file\n - blob\n - csvstream\n x-enumDescriptions:\n file: Works with files instead of a record endpoint — a raw file download (PDFs, images, binary data), or a file-provider folder whose files the file object parses into records (csv, json, xml, xlsx, filedefinition incl. EDI)\n blob: Transfers the raw response body downstream as a blob without parsing it into records.\n csvstream: Streams a large CSV response, parsing it row-by-row instead of loading the whole body into memory.\n description: |-\n Set to `file` when the export works with files rather than a record-returning endpoint: raw file downloads (PDFs, images, binary data) and the folder mode of the cloud file providers (Google Drive, Box, Dropbox, Azure Blob, Google Cloud Storage, Celigo Storage), where each file in the folder is parsed by the `file` object's `type` (csv, json, xml, xlsx, filedefinition — including EDI) into records; the `file` object must also be configured. Leave undefined for standard data exports that return structured records (JSON, XML, GraphQL, SOAP responses) — the response is then parsed into records for downstream steps. When set to `file` or `blob`, the export appears as a \"Transfer\" step in Flow Builder instead of a standard \"Export\" step.\n x-celigo-ai-guidance:\n - |-\n ## IMPORTANT: This field should be LEFT UNDEFINED for the vast majority of HTTP exports.\n\n This is an OPTIONAL field that should only be set in rare, specific cases. For standard REST API exports\n (Shopify, Salesforce, NetSuite, custom REST APIs, etc.), this field MUST be left undefined.\n - |-\n ## WHEN TO LEAVE THIS FIELD UNDEFINED (MOST COMMON CASE):\n\n Leave this field undefined for ALL standard data exports, including:\n - REST API exports that return JSON records\n - APIs that return XML records or structured data\n - Any export that retrieves business records, entities, or data objects\n - Standard CRUD operations that return record collections\n - GraphQL queries that return structured data\n - SOAP APIs that return structured responses\n\n Examples of exports that should have this field undefined:\n - \"Export all Shopify Customers\" → undefined (returns JSON customer records)\n - \"Retrieve orders from custom REST API\" → undefined (returns JSON order records)\n - |-\n ## WHEN TO SET THIS FIELD TO 'file' (the export works with FILES, not a record endpoint):\n\n Set this field to 'file' in two situations:\n - The HTTP endpoint is specifically designed to download files — it returns raw binary\n file content (PDFs, images, ZIP files) or is a file download service (invoices,\n reports, attachments) whose body is saved as a file rather than parsed as records\n - The connection is a cloud file provider used in FOLDER mode (Google Drive, Box, Dropbox,\n Azure Blob Storage, Google Cloud Storage, Celigo Storage): the export reads the files\n in a folder and the 'file' object's `type` (csv, json, xml, xlsx, filedefinition)\n parses each one into records — the same file layer as an FTP or S3 export. In folder\n mode `relativeURI` is a folder path: an API endpoint path there fails the run with\n \"Directory is not found. Please create one and provide the path.\" To download one\n file per record through the provider's API instead, leave `http.type` unset, set the\n export's top-level `type` to `blob` (typically on a lookup) and point `relativeURI`\n at the file-download endpoint\n\n Examples of when to set type: \"file\":\n - \"Download PDF invoices from the API\" → type: \"file\"\n - \"Retrieve image files from a file server\" → type: \"file\"\n - \"Read the CSV files dropped in our Google Drive folder\" → type: \"file\", with file.type \"csv\"\n - \"Parse EDI X12 850 files from a Google Drive / Celigo Storage folder\" → type: \"file\",\n with file.type \"filedefinition\" and a top-level _ediProfileId (EDI fields are\n accepted on HTTP exports only in file mode)\n - |-\n ## IMPLEMENTATION DETAILS\n\n When this field is set to 'file':\n - The 'file' object property MUST also be configured\n - The export appears as a \"Transfer\" step in the Flow Builder UI\n - The system applies file-specific processing to the HTTP response\n - Downstream steps receive records when the 'file' object's `type` parses the files\n (csv, json, xml, xlsx, filedefinition with `file.output: \"records\"`), or the file\n content itself when `file.output` is \"blobKeys\"\n\n When this field is undefined (default for most exports):\n - The export appears as a standard \"Export\" step in the Flow Builder UI\n - The system parses the HTTP response as structured data (JSON, XML, etc.)\n - Downstream steps receive record data that can be mapped and transformed\n - |-\n ## DECISION FLOWCHART\n\n 1. Does the API endpoint return business records/entities (customers, orders, products, etc.)?\n → YES: Leave this field undefined\n 2. Does the API endpoint return structured data (JSON objects, XML records)?\n → YES: Leave this field undefined\n 3. Does the API endpoint return raw file content (PDFs, images, binary data)?\n → YES: Set this field to \"file\" (and configure the 'file' property)\n 4. Is the connection a cloud file provider and the export reads a FOLDER of files\n (CSV, JSON, XML, XLSX, or EDI via a file definition)?\n → YES: Set this field to \"file\" (and configure the 'file' property with the parsing type)\n\n Remember: When in doubt, leave this field undefined. Most HTTP exports are standard data exports.\n examples:\n - file\n - blob\n formType:\n # Server-enforced enum: values outside this set are rejected with a 422\n # (\"not a valid enum value for path `http.formType`\").\n type: string\n enum:\n - http\n - rest\n - graph_ql\n - assistant\n - assistant_graphql\n x-enumDescriptions:\n http: Standard HTTP request form — the default authoring mode.\n rest: Legacy REST-adaptor authoring mode, present on exports migrated from the REST adaptor.\n graph_ql: GraphQL authoring mode; the query is composed in GraphQL-specific fields.\n assistant: Connector-assistant authoring mode driven by the `assistant` metadata.\n assistant_graphql: Connector-assistant authoring mode for GraphQL-based assistants.\n description: |-\n Authoring mode of the HTTP request form. Controls which editor the UI presents\n (standard HTTP, legacy REST, GraphQL, or connector-assistant driven) and how the\n request fields are interpreted. When omitted, the standard HTTP form is used.\n x-celigo-ai-guidance:\n - |-\n The value picks the editor and changes one thing in a GraphQL request: under `graph_ql`\n and `assistant_graphql`, a `variables` member stored as a JSON string is parsed and sent\n as an object, while under `http` it is sent as the string. A body whose `variables` is\n already an object goes out unchanged under `graph_ql` and `http`. Live-verified 2026-10-22 with\n export previews against an echo endpoint: a string `variables` under `graph_ql`,\n `assistant_graphql` and `http`, and an object `variables` under `graph_ql` and `http`.\n - |-\n The GraphQL form reads `http.body` as the JSON envelope `{\"query\", \"operationName\",\n \"variables\"}` and shows each member in its own editor, recompiling the envelope on save\n (`http.paging.body` likewise from its paging fields). A body that is not parseable JSON at\n rest — Handlebars block helpers outside a string value — renders every field EMPTY in that\n form, so keep templated variables inside a JSON string value:\n `\"variables\": \"{\\\"after\\\": {{#if export.http.paging.token}}\\\"{{export.http.paging.token}}\\\"{{else}}null{{/if}} }\"`.\n - |-\n The GraphQL form has no relative URI field: the connection `baseURI` is expected to be the\n GraphQL endpoint (a saved `relativeURI` still applies but is not shown). Create GraphQL\n exports on a `graph_ql` connection — that is the setup where the form offers the Form view\n toggle between the GraphQL editor and the raw HTTP form; a `graph_ql` export on an `http`\n connection works but cannot be switched back in the UI.\n examples: [\"http\", \"graph_ql\"]\n method:\n type: string\n description: |-\n HTTP method used to request data from the target API. Consult the target API's\n documentation to determine the appropriate method. `HEAD` exists for connector endpoints\n that return metadata in response headers only (Azure Data Lake Storage Gen2 path\n properties).\n enum:\n - GET\n - POST\n - PUT\n - PATCH\n - DELETE\n - HEAD\n x-enumDescriptions:\n GET: Retrieves data from the target API without modifying resources.\n POST: Sends a request body to the target API when body criteria are needed, commonly used for RPC or SOAP/XML-based data retrieval\n PUT: Sends a full replacement payload, available for APIs that support it for data retrieval.\n PATCH: Sends a partial update payload, less common for exports but available for specialized APIs.\n DELETE: Requests resource deletion, rarely used for exports but available for specialized use cases.\n HEAD: Requests only the response headers, for connector endpoints that expose metadata that way (e.g. ADLS Gen2 path properties); there is no body to parse\n x-celigo-ai-guidance:\n - |-\n Any other verb fails the save with 422 `enum` (\"`OPTIONS` is not a valid enum value for\n path `http.method`\"). Import `http.method` does not accept `HEAD`.\n examples:\n - GET\n - POST\n - HEAD\n followRedirects:\n type: boolean\n default: true\n description: |-\n When explicitly false, 3xx responses are not followed — the redirect\n response itself (status code, `Location` header, body) becomes the\n record. Omitted or true follows allowed redirects (the default\n behavior).\n examples: [false]\n maxRedirects:\n type: integer\n minimum: 1\n maximum: 10\n description: |-\n Caps how many consecutive 3xx redirects are followed. Only applies\n when `followRedirects` is not false; to not follow at all, set\n `followRedirects: false` rather than `maxRedirects: 0`. Decimal\n values are rejected on save.\n examples: [3]\n relativeURI:\n type: string\n x-celigo-agent:\n - handlebars\n x-celigo-handlebars-context: pre-mapped\n description: |-\n Resource path appended to the connection's `baseURI` to form the complete request URL. Path segments, query parameters, or the entire value can be built with handlebars expressions for endpoints determined at runtime. For lookup exports (`isLookup: true`) with mappings configured, handlebars always render against the original pre-mapped input record, so fields removed or renamed by mappings remain available for URI construction. On `NetSuiteHTTPExport` the base URL is derived from the NetSuite connection and `nsDomainType`, so the path starts at `/services/rest/...` or `/app/site/hosting/restlet.nl`.\n\n Leave it empty to call the connection's `baseURI` exactly; `/` appends a trailing slash, which is a different URL. An export saved with an empty relative URI and no `http.body` comes back from reads without its `http` block, so a PUT built from such a read loses the stored settings: send the complete `http` block with every PUT.\n x-celigo-ai-guidance:\n - The resource path portion of the API endpoint used for this export.\n - |-\n Examples:\n - Simple resource paths: \"/products\", \"/orders\", \"/customers\"\n - With query parameters: \"/orders?status=pending\", \"/products?category=electronics&limit=100\"\n - With path parameters: \"/customers/{{record.customerId}}/orders\", \"/accounts/{{record.accountId}}/transactions\"\n - With dynamic query values: \"/orders?since={{lastExportDateTime}}\"\n - Fully dynamic path: \"{{record.dynamicPath}}\"\n - |-\n Path parameters, query parameters,\n or the entire URI can be dynamically generated using handlebars syntax.\n This is particularly useful for parameterized API calls or when the endpoint\n needs to be determined at runtime based on data or context.\n - |-\n ## LOOKUP EXPORT BEHAVIOR WITH MAPPINGS\n\n **CRITICAL**: For lookup exports (isLookup: true) that have mappings configured, the handlebars template evaluation for relativeURI always uses the **original input record** before any mapping transformations are applied.\n\n This design ensures that:\n - Mappings can transform the record structure for the request body without affecting URI construction\n - Essential fields like record IDs remain accessible for building dynamic endpoints\n - The request body can be optimized for the target API while preserving URI parameters\n\n **Example Scenario:**\n ```\n Input record: {\"customerId\": \"12345\", \"name\": \"John Doe\", \"email\": \"[email protected]\"}\n Mappings: Transform to {\"customer_name\": \"John Doe\", \"contact_email\": \"[email protected]\"}\n relativeURI: \"/customers/{{record.customerId}}/details\"\n Result: \"/customers/12345/details\" (uses original customerId, not mapped version)\n ```\n\n This prevents situations where mapping transformations would remove or rename fields needed for endpoint construction, ensuring reliable API calls regardless of how the request body is structured.\n - 'The entire relativeURI can be defined using handlebars expressions to create dynamic paths:'\n - |-\n This value is combined with the baseURI defined in the associated connection to\n form the complete API endpoint URL.\n - |-\n On a GraphQL-form export (`formType` `graph_ql` / `assistant_graphql`) the form owns\n this field: it shows no relative-URI field and, on every save in the editor, rewrites\n it to the GET query string it compiles from the Query / Operation name / Variables\n fields, because the endpoint is expected to be the connection's `baseURI` — a\n `graph_ql` connection, or a connector connection bound to its GraphQL API. A path set\n here through the API on such an export is invisible in the editor and does not survive\n the next save there; when the connection's base URI stops at an API root, move the\n export to a connection that supplies the GraphQL endpoint instead of patching this\n field.\n examples:\n - /api/v1/contacts\n - /customers/{{record.customerId}}\n - /orders?since={{lastExportDateTime}}\n - \"{{record.dynamicEndpoint}}\"\n - /users/{{record.userId}}/transactions?from={{record.startDate}}\n headers:\n type: array\n description: |-\n Headers specific to this export, merged with (and able to override) headers defined on the connection. Define common headers such as authentication on the connection instead. Values support handlebars expressions; for lookup exports with mappings, values render against the pre-mapped input record.\n x-celigo-ai-guidance:\n - |-\n Export-specific HTTP headers to include with API requests.\n Note that common headers like authentication are typically defined on the\n connection record rather than here.\n - |-\n Examples of export-specific headers:\n - Accept: To request specific content format for this export only\n - X-Custom-Filter: Export-specific filtering parameters\n - |-\n Header values can be defined using handlebars expressions if you need to\n reference any dynamic data or configurations.\n - |-\n For lookup exports (isLookup: true) with mappings configured,\n header value templates render against the **pre-mapped** record (the original\n input record from the upstream flow step) — mappings do not rewrite header\n evaluation.\n - |-\n Use this field only for headers that are specific to this particular export operation.\n Headers defined here will be merged with (and can override) headers from the connection.\n items:\n type: object\n properties:\n name:\n type: string\n description: Name of the HTTP header to send; a header defined here overrides a same-named header from the connection.\n examples:\n - Accept\n - X-Custom-Filter\n value:\n type: string\n description: Value sent for the header. Supports handlebars expressions; for lookup exports with mappings, the value renders against the pre-mapped input record.\n x-celigo-agent:\n - handlebars\n x-celigo-handlebars-context: pre-mapped\n examples:\n - application/json\n - \"{{record.customValue}}\"\n - \"{{lastExportDateTime}}\"\n requestMediaType:\n type: string\n description: |-\n Overrides the connection-level request media type for this export. Set only when this endpoint requires a different format than the connection default.\n x-celigo-ai-guidance:\n - |-\n Override request media type.\n Use this field to handle the use case where the HTTP request requires a\n different media type than what is configured on the connection.\n - |-\n Most APIs use a consistent media type across all endpoints, which should be configured at the connection resource. Use this field only when:\n\n - This specific endpoint requires a different format than other endpoints in the API\n - You need to override the connection-level setting for this particular export only\n enum:\n - json\n - xml\n - urlencoded\n - form-data\n - plaintext\n x-enumDescriptions:\n json: \"Sends the request body as JSON (Content-Type: application/json).\"\n xml: \"Sends the request body as XML (Content-Type: application/xml).\"\n urlencoded: \"Sends the request body as URL-encoded form data (Content-Type: application/x-www-form-urlencoded).\"\n form-data: Sends the request body as multipart form data, typically used for file uploads.\n plaintext: \"Sends the request body as plain text (Content-Type: text/plain).\"\n examples:\n - json\n - xml\n - urlencoded\n body:\n type: string\n x-celigo-agent:\n - handlebars\n - graphql\n description: |-\n Request body sent with POST, PUT, or PATCH requests, typically carrying query or filter criteria for APIs (such as GraphQL or SOAP) that expect them in the body. The content must match the format set by `requestMediaType` and supports handlebars expressions for dynamic values.\n x-celigo-ai-guidance:\n - |-\n The HTTP request body to send with POST, PUT, or PATCH requests. This field is typically used to:\n\n 1. Send query parameters to APIs that require them in the request body (e.g., GraphQL or SOAP APIs)\n 2. Provide filtering criteria for data exports\n - |-\n The body content must match the format specified in the requestMediaType field (JSON,\n XML, etc.).\n - |-\n A GraphQL request is this field holding the stringified JSON envelope `{\"query\":\n \"\u003cdocument>\", \"operationName\": \"...\", \"variables\": ...}`. `variables` may be an object\n or a JSON-encoded string, and a string is sent as an object only under the GraphQL\n form types (see `formType`). Variables with unquoted Handlebars (a number, `null`, an\n `{{#if}}` block) must be a string so the body remains parseable JSON at rest; string\n values inside that string sit two JSON levels deep, so write them as\n `\\\"{{jsonEncode record.x}}\\\"` with double braces: a plain `{{record.x}}`, or a\n triple-brace `jsonEncode`, fails a value holding a quote with 422 `invalid_json`.\n An object or array there is `{{jsonSerialize record.x}}`, also with double braces;\n triple braces fail the same way.\n - |-\n You can use handlebars expressions to create dynamic content:\n ```\n {\n \"query\": \"SELECT Id, Name FROM Account WHERE LastModifiedDate > {{lastExportDateTime}}\",\n \"parameters\": {\n \"customerId\": \"{{record.customerId}}\",\n \"limit\": 100\n }\n }\n ```\n - |-\n For XML or SOAP requests:\n ```\n \u003crequest>\n \u003cfilter>\n \u003cupdatedSince>{{lastExportDateTime}}\u003c/updatedSince>\n \u003ctype>{{record.type}}\u003c/type>\n \u003c/filter>\n \u003c/request>\n ```\n examples:\n - '{\"query\": \"SELECT * FROM Contacts WHERE LastModifiedDate > {{lastExportDateTime}}\"}'\n - \u003crequest>\u003cfilters>\u003cfilter>\u003cupdatedSince>{{lastExportDateTime}}\u003c/updatedSince>\u003c/filter>\u003c/filters>\u003c/request>\n - '{\"ids\": [{{record.id}}], \"includeDetails\": true}'\n - '{\"lastSyncDate\": \"{{lastExportDateTime}}\", \"limit\": 100}'\n successMediaType:\n type: string\n description: |-\n Media type used to parse successful response bodies. Set only when the response format differs from the request format.\n x-celigo-ai-guidance:\n - |-\n Specifies the media type (content type) expected in successful responses for this specific export. This field should only be used when:\n\n 1. The response format differs from the request format\n - |-\n Most APIs return responses in the same format as the request,\n so this field is often unnecessary.\n - |-\n Common values:\n - \"json\": For JSON responses (typically with Content-Type: application/json)\n - \"xml\": For XML responses (typically with Content-Type: application/xml)\n - \"csv\": For CSV data (typically with Content-Type: text/csv)\n - \"plaintext\": For plain text responses\n enum:\n - json\n - xml\n - csv\n - plaintext\n x-enumDescriptions:\n json: \"Parses the successful response body as JSON (Content-Type: application/json).\"\n xml: \"Parses the successful response body as XML (Content-Type: application/xml).\"\n csv: \"Parses the successful response body as CSV (Content-Type: text/csv).\"\n plaintext: Treats the successful response body as plain text.\n examples:\n - json\n - xml\n errorMediaType:\n type: string\n description: |-\n Media type used to parse error response bodies. Set only when error responses use a different format than the request.\n x-celigo-ai-guidance:\n - |-\n Most APIs return responses in the same format as the request,\n so this field is often unnecessary.\n - |-\n Specifies the media type (content type) expected in error responses for this specific export. This field should only be used when:\n\n 1. Error response format differs from the request format\n - |-\n Common values:\n - \"json\": For JSON error responses (most common in modern APIs)\n - \"xml\": For XML error responses (common in SOAP and older REST APIs)\n - \"plaintext\": For plain text error messages\n enum:\n - json\n - xml\n - plaintext\n x-enumDescriptions:\n json: Parses the error response body as JSON, the most common error format for modern APIs\n xml: Parses the error response body as XML.\n plaintext: Treats the error response body as plain text.\n examples:\n - json\n - xml\n _asyncHelperId:\n type: string\n format: objectId\n x-celigo-refModel: asynchelpers\n description: |-\n AsyncHelper resource that handles polling for long-running operations on APIs that process requests asynchronously (HTTP 202 responses, job tickets, feed or document IDs). Set when the export must submit a request, poll for status, and retrieve results once the external process completes — for example Amazon SP-API feeds or large report generators.\n x-celigo-ai-guidance:\n - |-\n Reference to an AsyncHelper resource that polls an asynchronous source API on\n this export's behalf.\n - |-\n Set this ONLY when the source API is genuinely asynchronous — it acknowledges\n the request (HTTP 202,\n a job ticket, a feed/document id) and processes it in the background,\n so results must be polled for before they can be retrieved (e.g.\n Amazon SP-API feeds, large report generators).\n Most exports are synchronous and need NO async helper;\n adding one to a synchronous source just adds polling overhead plus a status and\n result export to maintain.\n When in doubt, leave it unset.\n - |-\n The referenced helper bundles the polling config plus a required status export\n (polled to check progress) and a result export (fetches the final payload).\n An export configured with an async helper cannot carry its own transform,\n output filter,\n or preSavePage hook — put that processing in the result export instead.\n examples:\n - 60a2c4e6f321d800129a1a3c\n - 5f8d43a1b9e5a80011a35f2c\n x-celigo-agent:\n - async_helper\n once:\n type: object\n description: |-\n Callback configuration for once exports, used to mark records as exported in the source system after successful processing.\n Called only while the export's `type` is `once`: an export with no `type` keeps a saved `once` and never calls it, so every\n run reads every record again (live-verified 2026-10-22).\n properties:\n relativeURI:\n type: string\n x-celigo-agent:\n - handlebars\n x-celigo-handlebars-context: pre-mapped\n description: |-\n Relative path (starting with `/`) called on the source system to mark each record as exported after successful processing. Supports handlebars variables and renders against the pre-mapped record — mappings do not apply to the callback URI.\n x-celigo-ai-guidance:\n - |-\n - Must be a relative path starting with \"/\"\n - Can include Handlebars variables: \"/orders/{{record.Id}}/exported\"\n - Common patterns: dedicated status endpoint or record-specific updates\n - Renders against the **pre-mapped** record (original extracted record); mappings do not apply to the callback URI.\n examples:\n - /api/v1/mark-exported\n - /export/flag-records\n - /orders/update-status\n - /v2/transactions/{{record.id}}/exported\n method:\n type: string\n description: HTTP method used for the mark-as-exported callback request.\n enum:\n - GET\n - PUT\n - POST\n - PATCH\n - DELETE\n x-enumDescriptions:\n GET: Uses an HTTP GET request to mark records as exported.\n PUT: Uses an HTTP PUT request to mark records as exported.\n POST: Uses an HTTP POST request to mark records as exported.\n PATCH: Uses an HTTP PATCH request to mark records as exported.\n DELETE: Uses an HTTP DELETE request to mark records as exported.\n examples:\n - POST\n - PUT\n body:\n type: string\n x-celigo-agent:\n - handlebars\n description: Request body sent with the mark-as-exported callback. Supports handlebars expressions for dynamic values.\n x-celigo-ai-guidance:\n - |-\n The HTTP request body used when calling back to mark records as exported.\n Can include Handlebars expressions for dynamic values.\n examples:\n - '{\"status\": \"exported\", \"exportId\": \"{{_exportId}}\"}'\n - '{\"records\": [{\"id\": \"{{record.id}}\", \"exported\": true}]}'\n paging:\n type: object\n description: |-\n Controls how the export requests subsequent pages when the API returns multi-page responses. The `method` field determines which companion fields are required and how each next page is requested. For the page, skip, and token methods, reference the matching pagination variable (`{{export.http.paging.page}}`, `{{export.http.paging.skip}}`, or `{{export.http.paging.token}}`) in the relative URI or request body — misconfigured pagination is a common cause of incomplete data retrieval.\n\n When this export is the first lookup of a builder-mode API with `pagination.enabled` set, it is that API's volume driver: saving the export (even unchanged) invalidates the API's issued cursors, so a caller mid-walk gets an empty final page, and removing `paging` is accepted on save but makes every call to the API fail with `400 api_pagination_source_not_paged` until paging is restored or the API's flag is turned off.\n\n When a next page's response is identical to the previous one, paging stops: the export records an open error with code `page_response_error` (\"Identical response received from next page request. No more page requests will be made.\") and the run completes with the pages already read, so later pages are never exported. A request that doesn't change between pages produces this, as observed with `method: page` and the pagination variable missing from the relative URI and body; reference the variable so each request asks for the next page.\n x-celigo-ai-guidance:\n - Configuration object for navigating through multi-page API responses.\n - |-\n ## OVERVIEW FOR AI AGENTS\n\n This object is critical for retrieving large datasets that cannot be returned in a single API response.\n The pagination implementation determines how the system will retrieve subsequent pages of data after\n the first request, enabling complete data collection regardless of volume.\n - |-\n ## KEY DECISION POINTS\n\n 1. **Identify the API's pagination mechanism** (check API documentation)\n 2. **Select the corresponding method** value (most important field)\n 3. **Configure the required fields** based on your selected method\n 4. **Add pagination variables** to your request configuration\n 5. **Consider last page detection** options if needed\n - |-\n ## FIELD DEPENDENCIES BY PAGINATION METHOD\n\n 1. **page**: Page number-based pagination (e.g., ?page=2)\n - Required: Set `method` to \"page\"\n - Optional: `page` - Set if first page index is not 0 (e.g., set to 1 for APIs that start at page 1)\n - Optional: `maxPagePath` - Path to find total pages in response\n - Optional: `maxCountPath` - Path to find total records in response\n - Optional: `relativeURI` - Only if subsequent page URLs differ from first page\n - Optional: `resourcePath` - Only if records location changes in follow-up responses\n\n 2. **skip**: Offset/limit pagination (e.g., ?offset=100&limit=50)\n - Required: Set `method` to \"skip\"\n - Optional: `skip` - Set if first skip index is not 0\n - Optional: `maxPagePath` - Path to find total pages in response\n - Optional: `maxCountPath` - Path to find total records in response\n - Optional: `relativeURI` - Only if subsequent page URLs differ from first page\n - Optional: `resourcePath` - Only if records location changes in follow-up responses\n\n 3. **token**: Token-based pagination (e.g., ?page_token=abc123)\n - Required: Set `method` to \"token\"\n - Required: `path` - Location of the token in the response\n - Required: `pathLocation` - Whether token is in \"body\" or \"header\"\n - Optional: `token` - Set to provide initial token (rare)\n - Optional: `pathAfterFirstRequest` - Only if token location changes after first page\n - Optional: `relativeURI` - Only if subsequent page URLs differ from first page\n - Optional: `resourcePath` - Only if records location changes in follow-up responses\n\n 4. **linkheader**: Link header pagination (uses HTTP Link header with rel values)\n - Required: Set `method` to \"linkheader\"\n - Optional: `linkHeaderRelation` - Set if relation is not the default \"next\"\n - Optional: `resourcePath` - Only if records location changes in follow-up responses\n\n 5. **nextpageurl**: Complete next URL in response\n - Required: Set `method` to \"nextpageurl\"\n - Required: `path` - Location of the next URL in the response\n - Optional: `resourcePath` - Only if records location changes in follow-up responses\n\n 6. **relativeuri**: Custom relative URI pagination\n - Required: Set `method` to \"relativeuri\"\n - Required: `relativeURI` - Configure using handlebars with previous_page context\n - Optional: `resourcePath` - Only if records location changes in follow-up responses\n\n 7. **body**: Custom request body pagination\n - Required: Set `method` to \"body\"\n - Required: `body` - Configure using handlebars with previous_page context\n - Optional: `resourcePath` - Only if records location changes in follow-up responses\n - |-\n ## PAGINATION VARIABLES\n\n Based on your selected method, you MUST add one of these variables to your request configuration:\n\n - For page-based: Add `{{export.http.paging.page}}` to the URI or body\n - For offset-based: Add `{{export.http.paging.skip}}` to the URI or body\n - For token-based: Add `{{export.http.paging.token}}` to the URI or body\n - |-\n ## LAST PAGE DETECTION OPTIONS\n\n These fields can be used with any pagination method to detect the last page:\n\n - `lastPageStatusCode` - Detect last page by HTTP status code\n - `lastPagePath` - JSON path to check for last page indicator\n - `lastPageValues` - Values at lastPagePath that indicate last page\n - |-\n ## COMMON IMPLEMENTATION PATTERNS\n\n Most APIs require only 2-3 fields to be configured. The most common patterns are:\n\n ```json\n // Page-based pagination (starting at page 1)\n {\n \"method\": \"page\",\n \"page\": 1\n }\n\n // Token-based pagination\n {\n \"method\": \"token\",\n \"path\": \"meta.nextToken\",\n \"pathLocation\": \"body\"\n }\n\n // Link header pagination (simplest to configure)\n {\n \"method\": \"linkheader\"\n }\n ```\n\n IMPORTANT: Incorrect pagination configuration is one of the most common causes of incomplete data retrieval. Take time to properly identify and configure the correct pagination method for your API.\n properties:\n method:\n type: string\n description: |-\n Pagination strategy used to request each subsequent page; match it to the mechanism documented by the target API. Determines which companion fields are required: `token` needs `path` and `pathLocation`, `url` needs `path`, `relativeuri` needs `relativeURI`, `body` needs `body`, while `linkheader` typically needs no extra configuration. Using the wrong method results in errors or incomplete data retrieval.\n x-celigo-ai-guidance:\n - Defines the pagination strategy that will be used to retrieve all data pages.\n - |-\n ## PAGINATION METHODS AND THEIR REQUIREMENTS\n\n ### Page-Based Pagination (`\"page\"`)\n ```\n \"method\": \"page\"\n ```\n - **Implementation**: Uses increasing page numbers (e.g., ?page=1, ?page=2)\n - **Required Setup**: Add `{{export.http.paging.page}}` to your URI or body\n - **Common Fields**: page (if starting at 1 instead of 0)\n - **API Examples**: Most REST APIs, Shopify, WordPress\n - **When to Use**: APIs that accept a page number parameter\n\n ### Offset/Skip Pagination (`\"skip\"`)\n ```\n \"method\": \"skip\"\n ```\n - **Implementation**: Uses increasing offset values (e.g., ?offset=0, ?offset=100)\n - **Required Setup**: Add `{{export.http.paging.skip}}` to your URI or body\n - **Common Fields**: Usually none (system handles offset increments)\n - **API Examples**: MongoDB, SQL-based APIs\n - **When to Use**: APIs that use offset/limit or skip/limit parameters\n\n ### Token-Based Pagination (`\"token\"`)\n ```\n \"method\": \"token\"\n ```\n - **Implementation**: Passes tokens from previous responses to get next pages\n - **Required Setup**:\n 1. Add `{{export.http.paging.token}}` to your URI or body\n 2. Set path to location of token in response\n 3. Set pathLocation to \"body\" or \"header\"\n - **API Examples**: AWS, Google Cloud, modern REST APIs\n - **When to Use**: APIs that provide continuation tokens/cursors\n\n ### Link Header Pagination (`\"linkheader\"`)\n ```\n \"method\": \"linkheader\"\n ```\n - **Implementation**: Follows URLs in HTTP Link headers automatically\n - **Required Setup**: None (simplest to configure)\n - **Common Fields**: Usually none (automatic)\n - **API Examples**: GitHub, GitLab, any API following RFC 5988\n - **When to Use**: APIs that return Link headers with rel=\"next\"\n\n ### Next Page URL (`\"nextpageurl\"`)\n ```\n \"method\": \"nextpageurl\"\n ```\n - **Implementation**: Uses complete URLs returned in response body\n - **Required Setup**: Set path to location of next URL in response\n - **API Examples**: Some social media APIs, GraphQL implementations\n - **When to Use**: APIs that include complete next page URLs in responses\n\n ### Custom Relative URI (`\"relativeuri\"`)\n ```\n \"method\": \"relativeuri\"\n ```\n - **Implementation**: Builds custom URIs based on previous responses\n - **Required Setup**: Configure relativeURI with handlebars templates\n - **When to Use**: Non-standard pagination requiring custom logic\n\n ### Custom Request Body (`\"body\"`)\n ```\n \"method\": \"body\"\n ```\n - **Implementation**: Creates custom request bodies for pagination\n - **Required Setup**: Configure body with handlebars templates\n - **API Examples**: GraphQL, SOAP, RPC APIs\n - **When to Use**: APIs requiring POST requests with pagination in body\n - |-\n ## SELECTION GUIDANCE\n\n To determine the correct method:\n 1. Check the API documentation for pagination instructions\n 2. Look for examples of multi-page requests in API samples\n 3. Test with a small request to observe pagination mechanics\n 4. Choose the method matching the API's expected behavior\n\n IMPORTANT: Using the wrong pagination method will result in either errors or incomplete data retrieval.\n - |-\n ## IMPORTANCE FOR AI AGENTS\n\n This is the MOST CRITICAL field in pagination configuration. It determines:\n - Which other fields are required vs. optional\n - How subsequent pages will be requested\n - Which pagination variables must be used in requests\n - How the system detects the last page\n enum:\n - linkheader\n - page\n - skip\n - token\n - url\n - relativeuri\n - body\n x-enumDescriptions:\n linkheader: Follows URLs in the HTTP Link header with rel=\"next\" for pagination.\n page: Uses incrementing page numbers for pagination (e.g., ?page=1, ?page=2).\n skip: Uses incrementing offset values for pagination (e.g., ?offset=0, ?offset=100).\n token: Passes a continuation token from the previous response to fetch the next page.\n url: Follows a complete next-page URL returned in the response body.\n relativeuri: Builds a custom relative URI for each subsequent page using handlebars templates.\n body: Sends a custom request body for each subsequent page using handlebars templates.\n examples:\n - page\n - token\n - linkheader\n - relativeuri\n page:\n type: integer\n description: |-\n Starting page number for `method: page`. Set to 1 for APIs whose first page is not zero-indexed; when omitted, paging starts at 0. The value is incremented automatically for each subsequent page request.\n x-celigo-ai-guidance:\n - Specifies the starting page number for page-based pagination.\n - |-\n ## FIELD BEHAVIOR\n\n - RELEVANT ONLY for method=\"page\"\n - OPTIONAL: Defaults to 0 if not provided\n - COMMON VALUES: 1 (most APIs), 0 (zero-indexed APIs)\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n This field should be set when the API's first page is not zero-indexed. Most APIs use 1 as\n their first page number, in which case you should set:\n\n ```json\n {\n \"method\": \"page\",\n \"page\": 1\n }\n ```\n\n The system will automatically increment this value for each subsequent page request.\n - |-\n ## EXAMPLES\n\n - Shopify uses page=1 for first page\n - Some GraphQL APIs use page=0 for first page\n examples:\n - 1\n skip:\n type: integer\n description: |-\n Starting offset for `method: skip`. Rarely needed — most APIs start at 0, and the value is incremented by the page size automatically for each subsequent request. Set only when the API requires a non-zero starting offset.\n x-celigo-ai-guidance:\n - Specifies the starting offset value for offset/skip-based pagination.\n - |-\n ## FIELD BEHAVIOR\n\n - RELEVANT ONLY for method=\"skip\"\n - OPTIONAL: Defaults to 0 if not provided\n - COMMON VALUES: 0 (vast majority of APIs)\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n This field rarely needs to be set since most APIs use 0 as the starting offset.\n The system will automatically increment this value by the pageSize for each subsequent request.\n\n Example calculation for page transitions:\n - First page: offset=0 (or your configured value)\n - Second page: offset=pageSize\n - Third page: offset=pageSize*2\n - |-\n ## WHEN TO USE\n\n Only set this if the API requires a non-zero starting offset value, which is very uncommon.\n examples:\n - 0\n - 100\n token:\n type: string\n description: |-\n Initial token for `method: token`. Leave empty for normal pagination — the first request is sent without a token, and subsequent tokens are extracted from each response via `path`. Set only to resume from a known token or for APIs that require a token on the first request.\n x-celigo-ai-guidance:\n - Specifies an initial token value for token-based pagination.\n - |-\n ## FIELD BEHAVIOR\n\n - RELEVANT ONLY for method=\"token\"\n - OPTIONAL: Leave empty for normal pagination from the beginning\n - ADVANCED USE ONLY: Most implementations should NOT set this\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n Token-based pagination normally works by:\n 1. Making the first request with no token\n 2. Extracting a token from the response (using the path field)\n 3. Using that token for the next request\n\n This field should ONLY be set in rare scenarios:\n - Resuming a previous pagination sequence from a known token\n - APIs that require a token value even for the first request\n - Testing specific pagination scenarios\n - |-\n ## EXAMPLE SCENARIOS\n\n ```json\n // To resume pagination from a specific point:\n {\n \"method\": \"token\",\n \"path\": \"meta.nextToken\",\n \"pathLocation\": \"body\",\n \"token\": \"\u003cnext-page-token>\"\n }\n\n // For APIs requiring an initial token:\n {\n \"method\": \"token\",\n \"path\": \"pagination.nextToken\",\n \"pathLocation\": \"body\",\n \"token\": \"start\"\n }\n ```\n examples:\n - next_token_123\n path:\n type: string\n description: |-\n Location of the pagination value in each response — the continuation token for `method: token`, or the complete next-page URL for `method: url`. When `pathLocation` is `body`, use a dot-notation JSON path (e.g. `meta.nextToken`); when `header`, use the exact case-sensitive header name. Not used by other pagination methods.\n x-celigo-ai-guidance:\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n ### For token-based pagination (method=\"token\"):\n\n 1. When pathLocation=\"body\":\n - Set to a JSON path that points to the token in the response body\n - Uses dot notation to navigate JSON objects\n\n Example response:\n ```json\n {\n \"data\": [...],\n \"meta\": {\n \"nextToken\": \"abc123\"\n }\n }\n ```\n Correct path: \"meta.nextToken\"\n\n 2. When pathLocation=\"header\":\n - Set to the exact name of the HTTP header containing the token\n - Case-sensitive, must match the header exactly\n\n Example header:\n ```\n X-Pagination-Token: \u003cnext-page-token>\n ```\n Correct path: \"X-Pagination-Token\"\n\n ### For next page URL pagination (method=\"nextpageurl\"):\n\n - Set to a JSON path that points to the complete URL in the response\n\n Example response:\n ```json\n {\n \"data\": [...],\n \"pagination\": {\n \"next_url\": \"https://api.example.com/data?page=2\"\n }\n }\n ```\n Correct path: \"pagination.next_url\"\n - Specifies the location of pagination information in API responses.\n - |-\n ## COMMON ERROR PATTERNS\n\n 1. Missing dot notation: \"meta.nextToken\" not \"meta/nextToken\"\n 2. Incorrect case: \"Meta.NextToken\" when API returns \"meta.nextToken\"\n 3. Missing array indices when needed: \"items[0].next\" not \"items.next\"\n examples:\n - meta.nextPage\n - pagination.next\n - X-Next-Token\n - pagination.nextUrl\n pathLocation:\n type: string\n description: |-\n Where the export looks for the pagination token referenced by `path`. Required for `method: token`; not used by other pagination methods.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n - REQUIRED for method=\"token\"\n - NOT USED for other pagination methods\n - LIMITED to two possible values: \"body\" or \"header\"\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n When using token-based pagination, you must:\n 1. Set method=\"token\"\n 2. Set path to locate the token\n 3. Set pathLocation to indicate where the token is found\n\n ### When to use \"body\":\n\n Set to \"body\" when the token is contained in the JSON response body.\n This is the most common scenario for modern APIs.\n\n Example configuration:\n ```json\n {\n \"method\": \"token\",\n \"path\": \"metadata.nextToken\",\n \"pathLocation\": \"body\"\n }\n ```\n\n ### When to use \"header\":\n\n Set to \"header\" when the token is returned as an HTTP header.\n\n Example configuration:\n ```json\n {\n \"method\": \"token\",\n \"path\": \"X-Next-Page-Token\",\n \"pathLocation\": \"header\"\n }\n ```\n - Specifies where to find the pagination token in the API response.\n - |-\n ## DEPENDENCY CHAIN\n\n This field participates in a critical dependency chain:\n\n 1. Set method=\"token\"\n 2. Set pathLocation=\"body\" or \"header\"\n 3. Set path to token location based on pathLocation value\n 4. Add {{export.http.paging.token}} to URI or body parameters\n\n All four elements must be properly configured for token pagination to work.\n enum:\n - body\n - header\n x-enumDescriptions:\n body: The pagination token is located in the JSON response body.\n header: The pagination token is located in an HTTP response header.\n examples:\n - body\n - header\n pathAfterFirstRequest:\n type: string\n description: |-\n Alternative token location used for responses after the first page, in the same format as `path`. Set only when the API moves the token to a different location in subsequent responses — setting it unnecessarily can cause pagination to fail.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n - RELEVANT ONLY for method=\"token\"\n - OPTIONAL: Only needed when token location changes after first page\n - Uses same format as the path field (JSON path or header name)\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n This field should only be set when the API changes its response structure between\n the first page and subsequent pages. Most APIs maintain consistent structure, but\n some APIs may:\n\n 1. Use different response formats for first vs. subsequent pages\n 2. Move the token to a different location after the initial response\n 3. Change the field name for the token in follow-up responses\n\n Example scenario where this is needed:\n ```json\n // First page response:\n {\n \"data\": [...],\n \"meta\": {\n \"initialNextToken\": \"abc123\"\n }\n }\n\n // Subsequent page responses:\n {\n \"data\": [...],\n \"pagination\": {\n \"nextToken\": \"def456\"\n }\n }\n ```\n\n In this case:\n - path = \"meta.initialNextToken\" (for first page)\n - pathAfterFirstRequest = \"pagination.nextToken\" (for subsequent pages)\n - |-\n ## DEPENDENCY CHAIN\n\n This field works in conjunction with the main path field:\n 1. First request: token is extracted using the path field\n 2. Subsequent requests: token is extracted using pathAfterFirstRequest\n\n IMPORTANT: Only set this field if you've verified that the API actually changes\n its response structure. Setting it unnecessarily can cause pagination to fail.\n examples:\n - pagination.nextToken\n - meta.continueToken\n - X-Next-Page-Token\n relativeURI:\n type: string\n x-celigo-agent:\n - handlebars\n description: |-\n Overrides the main relative URI for second and later page requests; leave empty when the main relative URI works for all pages. Build it with handlebars using the `previous_page` context — `previous_page.full_response` (the prior response body), `previous_page.last_record` (the last record of the prior page), and `previous_page.headers` (the prior response headers).\n x-celigo-ai-guidance:\n - |-\n Override relative URI for subsequent page requests.\n This field appears as \"Override relative URI for subsequent page requests\" in the UI.\n - |-\n This field only needs to be set if subsequent page requests require a different\n relative URI than what is configured in the primary relative URI field.\n Most APIs use the same endpoint for all pages and vary only the query parameters,\n but some may require a completely different path for subsequent requests.\n - |-\n You can use handlebars expressions to reference data from the previous API response using the `previous_page` context object, which contains:\n\n - `previous_page.full_response` - The entire JSON response body from the previous request\n - `previous_page.last_record` - The last record from the previous page of results\n - `previous_page.headers` - All HTTP headers from the previous response\n - |-\n Common patterns include:\n - `{{previous_page.full_response.next_page}}` - Use a complete next page URL returned by the API\n - `/customers?page={{previous_page.full_response.page_count}}` - Use a page number from the response\n - `/orders?cursor={{previous_page.full_response.next_cursor}}` - Use a cursor/token from the response\n - The exact structure of data available depends on your specific API's response format.\n examples:\n - \"{{previous_page.full_response.next_page}}\"\n - /customers?page={{previous_page.full_response.page}}\n - /orders?token={{previous_page.full_response.next_token}}\n - /items?after={{previous_page.last_record.id}}&limit=100\n x-celigo-handlebars-context: previous_page\n body:\n type: string\n x-celigo-agent:\n - handlebars\n - graphql\n description: |-\n Overrides the main request body for second and later page requests, typically for GraphQL or SOAP APIs that paginate through the body; leave empty when the main body works for all pages. Build it with handlebars using the `previous_page` context — `previous_page.full_response`, `previous_page.last_record`, and `previous_page.headers`.\n x-celigo-ai-guidance:\n - |-\n This field only needs to be set if subsequent page requests require a different\n HTTP request body than what is configured in the primary HTTP request body\n field.\n Most APIs use query parameters for pagination,\n but some (especially GraphQL or SOAP APIs) may require pagination parameters to\n be sent in the request body.\n - |-\n You can use handlebars expressions to reference data from the previous API response using the `previous_page` context object, which contains:\n\n - `previous_page.full_response` - The entire JSON response body from the previous request\n - `previous_page.last_record` - The last record from the previous page of results\n - `previous_page.headers` - All HTTP headers from the previous response\n - The exact structure of data available depends on your specific API's response format.\n - |-\n Override HTTP request body for subsequent page requests.\n This field appears as \"Override HTTP request body for subsequent page requests\" in the UI.\n - |-\n Common patterns include:\n - Including the next cursor in a GraphQL query: `{\"query\": \"...\", \"variables\": {\"cursor\": \"{{previous_page.full_response.pageInfo.endCursor}}\"}}`\n - Using the last record's ID: `{\"after\": \"{{previous_page.last_record.id}}\", \"limit\": 100}`\n - Including a page number: `{\"page\": {{previous_page.full_response.meta.next_page}}, \"pageSize\": 50}`\n - Leave this field empty if the main HTTP request body can be used for all page requests.\n examples:\n - '{\"query\": \"query($cursor: String) { items(after: $cursor) { edges { node { id name } } pageInfo { endCursor } } }\", \"variables\": {\"cursor\": \"{{previous_page.full_response.data.items.pageInfo.endCursor}}\"}}'\n - '{\"pageToken\": \"{{previous_page.full_response.nextPageToken}}\"}'\n - '{\"startIndex\": {{previous_page.full_response.nextStartIndex}}, \"maxResults\": 100}'\n mergeBodyParamsToPagingBody:\n type: boolean\n description: |-\n Only applies when `paging.method` is `body`. When true, the body\n parameters produced by the export's mappings at runtime are merged\n into the evaluated `paging.body` for page 2+ requests, so paging\n requests keep the same mapped parameters as the first page.\n Omitted/false keeps the original behavior — page 2+ requests send\n only the evaluated `paging.body`.\n x-celigo-handlebars-context: previous_page\n linkHeaderRelation:\n type: string\n description: |-\n Link header relation followed for `method: linkheader` when the API uses a value other than the default `next`. Case-sensitive and must exactly match the `rel` value in the Link header, without the `rel=` prefix.\n x-celigo-ai-guidance:\n - Specifies which relation in the Link header to use for pagination.\n - |-\n ## FIELD BEHAVIOR\n\n - RELEVANT ONLY for method=\"linkheader\"\n - OPTIONAL: Defaults to \"next\" if not provided\n - Case-sensitive value matching the rel attribute in Link header\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n Link header pagination follows the RFC 5988 standard where pagination links\n are provided in HTTP headers. A typical Link header looks like:\n\n ```\n Link: \u003chttps://api.example.com/items?page=2>; rel=\"next\", \u003chttps://api.example.com/items?page=1>; rel=\"prev\"\n ```\n\n This field allows you to specify which relation type to follow for pagination:\n\n ```\n \"linkHeaderRelation\": \"next\" // Default value\n ```\n\n Some APIs use non-standard relation names, which is when you'd need to change this:\n\n ```\n \"linkHeaderRelation\": \"successor\" // Custom relation name\n ```\n - |-\n ## COMMON VALUES\n\n - \"next\" (default): Standard for most RFC 5988 compliant APIs\n - \"successor\": Alternative used by some APIs\n - \"forward\": Alternative used by some APIs\n - \"nextpage\": Non-standard but used by some implementations\n\n IMPORTANT: This is case-sensitive and must exactly match the relation value in\n the Link header. If the API includes the prefix \"rel=\" in the header, do NOT\n include it here.\n examples:\n - next\n - successor\n - forward\n - nextpage\n resourcePath:\n type: string\n description: |-\n Overrides the path to records for second and later page responses. Set only when follow-up pages place records at a different location than the first response; leave empty when all pages share the same structure.\n x-celigo-ai-guidance:\n - |-\n Override path to records for subsequent page requests.\n This field appears as \"Override path to records for subsequent page requests\" in the UI.\n - |-\n This field only needs to be set if subsequent page requests return a different\n response structure,\n and the records are located in a different place than the original request.\n - |-\n For example, if the first request returns records in a structure like {\"data\":\n [...]} but subsequent page responses have records in {\"results\": [...]} instead,\n you would set this field to \"results\" to correctly extract data from the follow-up pages.\n examples:\n - results\n - data.items\n - response.records\n lastPageStatusCode:\n type: integer\n description: |-\n HTTP status code that signals the last page, replacing the default behavior of treating 404 as the end of pagination. When this status is received (any code but `200` paired with `lastPagePath`), paging stops and the response is not treated as an error. Set it when the API uses a non-404 code (such as 204 or 400) to indicate no more pages, or to `200` together with `lastPagePath` when the last page is an ordinary 200 response: then the response is parsed, its records are kept, and paging stops only when `lastPagePath` matches `lastPageValues`.\n x-celigo-ai-guidance:\n - Specifies a custom HTTP status code that indicates the last page of results.\n - |-\n ## FIELD BEHAVIOR\n\n - OPTIONAL: Only needed for APIs with non-standard last page indicators\n - Applies to all pagination methods\n - Overrides the default 404 end-of-pagination detection\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n By default, the system treats a 404 status code as an indicator that\n pagination is complete. This field allows you to specify a different\n status code if your API uses an alternative convention.\n\n Common scenarios where this is needed:\n\n 1. APIs that return 204 (No Content) for empty result sets\n ```\n \"lastPageStatusCode\": 204\n ```\n\n 2. APIs that return 400 (Bad Request) when requesting beyond available pages\n ```\n \"lastPageStatusCode\": 400\n ```\n\n 3. APIs with custom error codes for pagination completion\n ```\n \"lastPageStatusCode\": 499\n ```\n - |-\n ## TECHNICAL DETAILS\n\n When this status code is received (any code but `200` paired with\n `lastPagePath`), the system:\n - Stops the pagination process\n - Considers the data collection complete\n - Does not treat the response as an error\n - Does not attempt to process any response body\n\n IMPORTANT: Only set this if your API explicitly uses a non-404 status code\n to indicate the end of pagination, or to `200` with `lastPagePath` (below).\n Setting this incorrectly could cause premature termination of data\n collection or error handling issues.\n - |-\n `200` is the exception, for an API whose last page is an ordinary 200 response:\n together with `lastPagePath`, paging stops on the first 200 whose `lastPagePath`\n value matches `lastPageValues`, and that page's records are kept. Live-verified\n 2026-10-22 with token paging.\n examples:\n - 204\n - 400\n - 500\n lastPagePath:\n type: string\n description: |-\n JSON path to a response-body field that signals the end of pagination, such as a \"has more\" flag or a cursor that empties on the last page. Must be used with `lastPageValues`, which lists the values at this path that stop paging. If the path does not exist in a response, the condition is not considered met. With `method: token` the path is read only on a response whose status code equals `lastPageStatusCode`, so pair it with `lastPageStatusCode: 200` for an ordinary last page; with `method: body` it is read on every response.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n - OPTIONAL: Only needed for APIs with field-based pagination completion signals\n - Accepted on all pagination methods; when it is read depends on the method (see\n the method note below)\n - Used in conjunction with lastPageValues\n - JSON path notation to a field in the response body\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n This field is used when an API indicates the last page through a field\n in the response body rather than using HTTP status codes. Which responses\n the system checks it on depends on the paging method (see the method note\n below).\n\n Common patterns include:\n\n 1. Boolean flag fields\n ```\n \"lastPagePath\": \"meta.isLastPage\"\n ```\n\n 2. \"Has more\" indicators\n ```\n \"lastPagePath\": \"pagination.hasMore\"\n ```\n\n 3. Cursor/token fields that are null/empty on the last page\n ```\n \"lastPagePath\": \"meta.nextCursor\"\n ```\n\n 4. Error message fields\n ```\n \"lastPagePath\": \"error.message\"\n ```\n - |-\n ## DEPENDENCY CHAIN\n\n This field must be used with lastPageValues, which specifies the value(s)\n at this path that indicate pagination is complete. For example:\n\n ```json\n \"lastPagePath\": \"pagination.hasMore\",\n \"lastPageValues\": [\"false\", \"0\"]\n ```\n\n IMPORTANT: The path is evaluated using JSON path notation, on the responses the\n method note below describes. If the path doesn't exist in the response, the\n condition is not considered met.\n - |-\n On `method: token` the path is checked only on a response whose status code equals\n `lastPageStatusCode`. For an API whose last page is an ordinary 200 (a GraphQL\n `pageInfo.hasNextPage: false`), set `lastPageStatusCode: 200` with it; without that,\n paging goes on while the token at `path` is non-empty, so a last page that still\n carries a cursor costs one more request. On `method: body` the path is read on\n every response, with no status code needed. Live-verified 2026-10-22 against an echo\n endpoint for `token` (two requests without `lastPageStatusCode`, one with 200) and\n `body` (stopped on the second response); `page`, `skip` and `url` paging were not\n verified.\n examples:\n - meta.isLastPage\n - pagination.hasMore\n - meta.nextCursor\n - error.message\n lastPageValues:\n type: array\n description: |-\n Values at `lastPagePath` that stop pagination; a match on any entry ends paging. All entries are compared as exact, case-sensitive strings, even for boolean or numeric fields — use `\"true\"`, `\"false\"`, `\"null\"` for JSON null, or `\"\"` for an empty string. Required when `lastPagePath` is set.\n x-celigo-ai-guidance:\n - Specifies which value(s) at the lastPagePath indicate the end of pagination.\n - |-\n ## FIELD BEHAVIOR\n\n - REQUIRED when lastPagePath is used\n - Array of string values (even for boolean or numeric comparisons)\n - Case-sensitive matching against the value at lastPagePath\n - Multiple values create an OR condition (any match indicates last page)\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n This field works in conjunction with lastPagePath to determine when\n pagination is complete. The system looks for the field specified by\n lastPagePath and compares its value against each entry in this array.\n\n Common patterns include:\n\n 1. For boolean \"isLastPage\" flags (true means last page)\n ```json\n \"lastPagePath\": \"meta.isLastPage\",\n \"lastPageValues\": [\"true\"]\n ```\n\n 2. For \"hasMore\" flags (false means last page)\n ```json\n \"lastPagePath\": \"pagination.hasMore\",\n \"lastPageValues\": [\"false\", \"0\"]\n ```\n\n 3. For empty cursors (null/empty string means last page)\n ```json\n \"lastPagePath\": \"meta.nextCursor\",\n \"lastPageValues\": [\"null\", \"\"]\n ```\n\n 4. For specific error messages\n ```json\n \"lastPagePath\": \"error.message\",\n \"lastPageValues\": [\"No more pages\", \"End of results\"]\n ```\n - |-\n ## TECHNICAL DETAILS\n\n - All values must be specified as strings, even for boolean or numeric comparisons\n - JSON null should be represented as the string \"null\"\n - Empty string is represented as \"\"\n - The comparison is exact and case-sensitive\n\n IMPORTANT: This field is only considered when the lastPagePath exists in the\n response. Both lastPagePath and lastPageValues must be configured correctly\n for proper pagination termination.\n items:\n type: string\n examples:\n - - \"true\"\n - - \"false\"\n - \"0\"\n - - \"null\"\n - \"\"\n - - No more pages\n - End of results\n maxPagePath:\n type: string\n description: |-\n JSON path to the total page count in the response, used to stop paging once the last page is reached. Only applies to the `page` and `skip` methods. Point it at the total number of pages, not the current page number.\n x-celigo-ai-guidance:\n - Specifies a JSON path to a field containing the total number of pages available.\n - |-\n ## FIELD BEHAVIOR\n\n - OPTIONAL: Only relevant for \"page\" and \"skip\" pagination methods\n - JSON path to a numeric field in the response\n - Used to optimize pagination by detecting the last page early\n - Ignored for other pagination methods\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n This field enables pagination optimization when an API includes metadata\n about the total number of pages. When configured, the system:\n\n 1. Extracts the total page count from each response\n 2. Compares the current page number against this total\n 3. Stops pagination when the maximum page is reached\n\n Common API response patterns include:\n\n ```json\n // Pattern 1: Metadata section with page counts\n {\n \"data\": [...],\n \"meta\": {\n \"totalPages\": 5,\n \"currentPage\": 2\n }\n }\n\n // Pattern 2: Pagination object\n {\n \"results\": [...],\n \"pagination\": {\n \"pageCount\": 5,\n \"page\": 2\n }\n }\n\n // Pattern 3: Root level pagination info\n {\n \"items\": [...],\n \"pages\": 5,\n \"current\": 2\n }\n ```\n - |-\n ## USAGE SCENARIOS\n\n Most useful when:\n - The API reliably includes total page counts\n - You want to prevent unnecessary requests after the last page\n - The 404/last page detection mechanisms aren't suitable\n\n IMPORTANT: This field should point to the TOTAL number of pages,\n not the current page number. The value must be numeric (integer).\n examples:\n - meta.totalPages\n - pagination.pageCount\n - response.paging.total\n - pages\n maxCountPath:\n type: string\n description: |-\n JSON path to the total record count in the response, used to stop paging once all records have been retrieved. Only applies to the `page` and `skip` methods; when both are set, `maxPagePath` takes precedence. Point it at the total number of records, not the count in the current page.\n x-celigo-ai-guidance:\n - Specifies a JSON path to a field containing the total number of records available.\n - |-\n ## FIELD BEHAVIOR\n\n - OPTIONAL: Only relevant for \"page\" and \"skip\" pagination methods\n - JSON path to a numeric field in the response\n - Alternative to maxPagePath for record-based termination\n - Used when APIs provide total record count instead of page count\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n This field enables pagination optimization when an API includes metadata\n about the total number of records rather than pages. When configured,\n the system:\n\n 1. Extracts the total record count from each response\n 2. Tracks the total number of records processed so far\n 3. Stops pagination when all records have been processed\n\n Common API response patterns include:\n\n ```json\n // Pattern 1: Metadata section with record counts\n {\n \"data\": [...],\n \"meta\": {\n \"totalCount\": 42,\n \"page\": 2,\n \"pageSize\": 10\n }\n }\n\n // Pattern 2: Pagination object\n {\n \"results\": [...],\n \"pagination\": {\n \"total\": 42,\n \"offset\": 20,\n \"limit\": 10\n }\n }\n\n // Pattern 3: Root level count info\n {\n \"items\": [...],\n \"count\": 42,\n \"page\": 2\n }\n ```\n - |-\n ## RELATIONSHIP WITH maxPagePath\n\n This field is an alternative to maxPagePath:\n - Use maxPagePath when the API provides a total page count\n - Use maxCountPath when the API provides a total record count\n - If both are provided, maxPagePath takes precedence\n\n IMPORTANT: This field should point to the TOTAL number of records,\n not the number of records in the current page. The value must be\n numeric (integer).\n examples:\n - meta.totalCount\n - pagination.total\n - response.totalResults\n - count\n if: { properties: { method: { const: body } }, required: [method] }\n then: { required: [body] }\n else:\n # token paging requires path AND pathLocation. The server defaults an omitted\n # pathLocation to body (and most stored docs omit it), but declaring where the\n # token is read from is enforced for new configs — a token that actually arrives\n # in a header fails silently under the implicit default.\n if: { properties: { method: { const: token } }, required: [method] }\n then: { required: [path, pathLocation] }\n else:\n if: { properties: { method: { const: relativeuri } }, required: [method] }\n then: { required: [relativeURI] }\n else:\n if: { properties: { method: { const: url } }, required: [method] }\n then: { required: [path] }\n response:\n type: object\n description: |-\n Controls how records are extracted from the API response and how success or failure is detected at the response level. When the API wraps records in an envelope object, set `resourcePath` to the path of the records array — without it, the entire response body is treated as a single record. Leave this object undefined when the API returns a bare JSON array.\n x-celigo-ai-guidance:\n - Configuration for parsing and interpreting HTTP responses returned by the source API.\n - |-\n ## MOST IMPORTANT FIELD: resourcePath\n\n `resourcePath` is the single most commonly needed field in this object. When an API\n wraps its records inside a JSON envelope, you MUST set resourcePath to the dot-path\n (or ` configuring-exports — Skillopedia -prefixed JSONPath expression) that points to the array of records. Without it,\n the export treats the entire response as a single record.\n\n Example API response:\n ```json\n {\n \"status\": \"ok\",\n \"data\": {\n \"customers\": [\n {\"id\": 1, \"name\": \"Alice\"},\n {\"id\": 2, \"name\": \"Bob\"}\n ]\n }\n }\n ```\n → Set `resourcePath` to `data.customers` so the export produces 2 records.\n - |-\n ## WHEN TO LEAVE THIS OBJECT UNDEFINED\n\n If the API returns a bare JSON array (e.g. `[{\"id\":1}, {\"id\":2}]`) with no\n wrapper object, you do not need this object at all.\n properties:\n resourcePath:\n type: string\n description: |-\n Dot-separated path to the array of records inside the response body (e.g. `data.customers` for `{\"data\": {\"customers\": [...]}}`); without it, a wrapped response is treated as a single record. Values starting with ` configuring-exports — Skillopedia are evaluated as JSONPath instead of dot notation — including unions (`$['orders','invoices'][*]`) and filter expressions (`$.items[?(@.qty>3)]`); all other values keep the existing dot-notation behavior unchanged. Leave undefined when the API returns a bare JSON array. This extracts records from the API response — not to be confused with `oneToMany`/`pathToMany`, which unwrap arrays from input records, or `paging.resourcePath`, which applies only to subsequent page responses.\n x-celigo-ai-guidance:\n - |-\n ## CRITICAL FIELD FOR CORRECT DATA EXTRACTION\n\n Most APIs wrap their data in an envelope object. This field tells the export\n where to find the actual records within that envelope. Without this field,\n the export treats the entire response body as a single record, which is\n almost never the desired behavior when the response has a wrapper.\n - |-\n ## HOW IT WORKS\n\n Given an API response like:\n ```json\n {\n \"meta\": {\"page\": 1, \"total\": 42},\n \"results\": [\n {\"id\": \"A\", \"value\": 10},\n {\"id\": \"B\", \"value\": 20}\n ]\n }\n ```\n Setting `resourcePath` to `results` causes the export to produce 2 records\n (`{\"id\":\"A\",\"value\":10}` and `{\"id\":\"B\",\"value\":20}`).\n\n For deeply nested responses:\n ```json\n {\n \"slideshow\": {\n \"slides\": [{\"title\": \"Slide 1\"}, {\"title\": \"Slide 2\"}]\n }\n }\n ```\n Set `resourcePath` to `slideshow.slides` to get each slide as a record.\n - |-\n ## WHEN TO SET THIS FIELD\n\n - The API response is a JSON object (not a bare array) and the records are\n nested inside it → set this to the path\n - The API response is a bare JSON array → leave undefined (records are\n already at the top level)\n - |-\n ## JSONPATH SYNTAX (values starting with ` configuring-exports — Skillopedia )\n\n Values starting with ` configuring-exports — Skillopedia are evaluated as JSONPath instead of dot notation:\n - `$.data.orders` — equivalent to dot notation `data.orders`\n - `$['orders','invoices'][*]` — union: concatenates the elements of several\n arrays into one record stream, in listed order (the trailing `[*]` is\n required; without it each array arrives as a single record)\n - `$.items[?(@.qty>3)]` — filter: emits only the elements matching the\n expression (filters may only access the current node `@`)\n - A path that resolves to nothing emits zero records for that response;\n evaluation failures surface as record-level errors\n Dot-notation values (no ` configuring-exports — Skillopedia prefix) keep the existing behavior unchanged.\n - |-\n ## COMMON PATTERNS\n\n | API response structure | resourcePath value |\n |---|---|\n | `{\"data\": [...]}` | `data` |\n | `{\"results\": [...]}` | `results` |\n | `{\"items\": [...]}` | `items` |\n | `{\"records\": [...]}` | `records` |\n | `{\"response\": {\"data\": [...]}}` | `response.data` |\n | `{\"slideshow\": {\"slides\": [...]}}` | `slideshow.slides` |\n | `[...]` (bare array) | leave undefined |\n - |-\n ## IMPORTANT DISTINCTION\n\n This field extracts records from the **API response**. Do NOT confuse it with:\n - `oneToMany` + `pathToMany` — which unwrap child arrays from *input records*\n in lookup/import steps (a completely different mechanism)\n - `paging.resourcePath` — which overrides the record location for *subsequent*\n page requests only (when follow-up pages use a different response structure)\n examples:\n - data\n - results\n - items\n - records\n - slideshow.slides\n - response.data\n - data.customers\n - $.data.orders\n includeParentData:\n $ref: './record-extraction.yml#/IncludeParentData'\n resourceIdPath:\n type: string\n description: |-\n Path to the unique identifier within each record, used primarily when processing results of asynchronous import responses. When omitted, the system looks for standard `id` or `_id` fields automatically.\n examples:\n - id\n - _id\n - recordId\n successPath:\n type: string\n description: |-\n Path to a response field that signals whether the call succeeded, for APIs that return HTTP 200 even on failure. With `successValues`, the value at this path must match one of them; without `successValues`, the path only has to resolve — a response in which it is absent fails with `response_failure` (\"Unable to extract paths from response body using path: \u003cpath>, Response: …\").\n x-celigo-ai-guidance:\n - |-\n Use this when the API returns HTTP 200 for all requests but signals success or\n failure through a field in the response body.\n - |-\n Example: If the API returns `{\"status\": \"ok\", \"data\": [...]}`,\n set `successPath` to `status` and `successValues` to `[\"ok\"]`.\n - |-\n On a LOOKUP step this is also the way to make \"no match\" fail the record. A lookup\n whose `resourcePath` resolves to nothing returns zero records and the platform counts\n that as success: the record continues to the next step with nothing merged, and the\n flow's `pageProcessors[].proceedOnFailure` for that step has no bearing on it. Set\n `successPath` (no `successValues`) to a path that exists only when a match was found,\n e.g. `results[0].id`: an empty result then fails the record with `response_failure`\n (`Unable to extract paths from response body using path: results[0].id, Response:\n {…}`), and `proceedOnFailure: true` on that step lets the failed record continue.\n Live-verified 2026-09-24 in real flow runs (listener -> HTTP lookup -> import, one\n record each): empty result with no `successPath` -> lookup 1 success, import 1\n success, with or without `proceedOnFailure`; `successPath` on an absent path ->\n lookup 1 error with that message, import 0 (1 with `proceedOnFailure: true`);\n `successPath` on a present path -> success. The error text is generic; for a custom\n message a `preSavePage` hook on the lookup that returns one `newErrorsAndRetryData`\n entry (`{errors: [{code: 404, message: \"Product not found\"}]}`) when `options.data`\n and `options.errors` are both empty failed the record with \"Product not found\" in\n the same runs. Lookups on adaptors without HTTP response settings (NetSuite, for\n example) have no `successPath`.\n examples:\n - status\n - success\n - result.code\n - meta.status\n successValues:\n type: array\n items:\n type: string\n description: |-\n Values at `successPath` that mark the response as successful; any other value is treated as an error. All comparisons are string-based — use `\"true\"` or `\"false\"` for boolean fields.\n x-celigo-ai-guidance:\n - Values at the `successPath` location that indicate the API call was successful.\n - |-\n When the value at `successPath` matches any entry in this array,\n the response is treated as successful.\n If the value does not match, the response is treated as an error.\n examples:\n - [\"ok\"]\n - [\"true\", \"success\"]\n - [\"200\", \"201\"]\n errorPath:\n type: string\n description: |-\n Path to the error message in the response body. The value at this path is included in error logs and error records when the API returns an error; without it, the message is the whole response body.\n examples:\n - error.message\n - errorMessage\n - errors[0].detail\n - errors[0].message\n failPath:\n type: string\n description: |-\n Path to a response field that signals failure even when the HTTP status code is 200 — the inverse of `successPath`. With `failValues`, the response fails when the value there matches one of them; without, when the path holds a truthy value. A path that resolves to an array matches only when the array has exactly one element.\n x-celigo-ai-guidance:\n - |-\n Similar to `successPath` but inverted logic — checks for failure indicators.\n - |-\n A path that resolves to an array matches only when the array has exactly one\n element. GraphQL APIs usually report a failed query with HTTP 200 and an `errors` array:\n `failPath: errors` fails the response for one error and lets two or more through,\n so set `failPath` (and `errorPath`) to `errors[0].message`. Without a fail path the\n response counts as a success, and an export whose `resourcePath` points into\n `data` returns zero records with no error. Live-verified 2026-10-22 with export\n and import previews against an echo endpoint.\n examples:\n - error\n - status\n - result.error\n - errors[0].message\n failValues:\n type: array\n items:\n type: string\n description: |-\n Values at `failPath` that mark the response as failed, even when the HTTP status code is 200.\n x-celigo-ai-guidance:\n - Values at the `failPath` location that indicate the API call failed.\n - |-\n When the value at `failPath` matches any entry in this array,\n the response is treated as a failure even if the HTTP status code was 200.\n examples:\n - [\"error\", \"failed\"]\n - [\"false\"]\n allowArrayforSuccessPath:\n type: boolean\n description: |-\n When true, treats the value at `successPath` as an array and counts the response as successful if any element matches `successValues`, rather than requiring a single scalar match. Set it for APIs that return per-record status arrays in a batch response.\n twoDArray:\n type: object\n description: |-\n Parsing options for endpoints that return tabular data as a two-dimensional array (rows of cells) rather than an array of objects, such as spreadsheet-style or report APIs.\n properties:\n hasHeader:\n type: boolean\n description: When true, the first row is treated as column headers and used to name the fields of each generated record.\n doNotNormalize:\n type: boolean\n description: When true, the rows are passed through as raw arrays instead of being normalized into keyed records.\n blobFormat:\n type: string\n description: |-\n Controls how the binary response body is decoded for blob exports. Only relevant when `http.type` is `file` or the export type is `blob`.\n enum:\n - utf8\n - ucs2\n - utf-16le\n - ascii\n - binary\n - base64\n - hex\n x-enumDescriptions:\n utf8: Decodes the blob response using UTF-8 character encoding.\n ucs2: Decodes the blob response using UCS-2 (two-byte Unicode) encoding.\n utf-16le: Decodes the blob response using UTF-16 Little Endian encoding.\n ascii: Decodes the blob response using 7-bit ASCII encoding.\n binary: Treats the blob response as raw binary data without character decoding.\n base64: Decodes the blob response from Base64-encoded text.\n hex: Decodes the blob response from hexadecimal-encoded text.\n examples:\n - utf8\n - base64\n # No successPath->successValues / failPath->failValues pairing: the paths are\n # meaningful alone (existence/error-extraction semantics) and the value lists are\n # optional refinements.\n _httpConnectorVersionId:\n type: string\n format: objectId\n description: |-\n Pins the export to a version of the published HTTP connector its connection is bound to\n (`connection.http._httpConnectorId`), so the UI renders that connector's endpoint form instead\n of a hand-authored request. Linked on save together with `_httpConnectorResourceId` and\n `_httpConnectorEndpointId` — see that field for the rule.\n _httpConnectorResourceId:\n type: string\n format: objectId\n x-celigo-refModel: httpconnectorresources\n description: |-\n The connector resource (a group of related endpoints, such as Orders) that this export's\n endpoint belongs to. Linked on save together with `_httpConnectorEndpointId` — see that field\n for the rule.\n sendAuthForFileDownloads:\n type: boolean\n description: When true, includes authentication headers when downloading files.\n isRest:\n type: boolean\n description: |-\n Internal flag marking an export built with the legacy REST form and stored in the unified\n HTTP framework. Set by the platform when it converts a legacy `rest` body; not configured\n directly. Returned on every HTTP export, `false` unless the export was converted.\n _httpConnectorEndpointId:\n type: string\n format: objectId\n x-celigo-refModel: httpconnectorendpoints\n description: |-\n The connector endpoint this export calls, shown in the UI as the selected endpoint. On save the\n platform derives it (with the version and resource ids) from `relativeURI` and `method` when\n they match an endpoint of the connector the connection is bound to — path parameters, literal\n ids and query strings still match — and replaces ids that belong to another connector. An id for\n the connection's own connector is kept as sent even when the URI names a different endpoint;\n when nothing matches, or the connection has no connector, whatever was sent is kept and nothing\n is derived.\n x-celigo-ai-guidance:\n - |-\n To build an export on a prebuilt connector instead of a hand-rolled HTTP request, bind the\n CONNECTION (`http._httpConnectorId` — find the connector in `GET /v1/httpconnectors` by `name`\n or `legacyId`) and write the endpoint's `relativeURI` and `method`; the platform links the\n export to that endpoint on save, so the three pointer fields may be omitted. Read the saved\n export back: no pointers means the URI matched no endpoint of the connection's connector. Set\n `_httpConnectorVersionId`, `_httpConnectorResourceId` and `_httpConnectorEndpointId` together\n only to pin a same-connector endpoint the URI would not match. Endpoint discovery through the\n published operations: `GET /v1/httpconnectors/{_id}` (`versions`, under `apis[]` for\n multi-API connectors), `GET /v1/httpconnectors/{_id}/httpconnectorresources`,\n `GET /v1/httpconnectors/{_id}/{_httpConnectorResourceId}/httpconnectorendpoints`, and\n `GET /v1/httpconnectors/{_id}/{_httpConnectorResourceId}/{_id}` for an endpoint's `method` and\n `relativeURI`. A top-level `_httpConnectorId` on the export itself is dropped. Verified against\n the live API 2026-09-21.\n # The file-mode rule (`type: file` outside an assistant form requires the export-level `file`\n # object) lives on the request schema, where `file` is in scope — see request.yml.\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":97056,"content_sha256":"8fdea6e951f36bc6ff2f21a2660a21b54853ce913920b9efe9579973c66c69b2"},{"filename":"references/schemas/jdbc.yml","content":"JDBC:\n type: object\n description: |-\n Configuration object for JDBC (Java Database Connectivity) data integration exports.\n\n This object is REQUIRED when the _connectionId field references a JDBC database connection\n and must not be included for other connection types. It defines how data is extracted\n from relational databases using SQL queries.\n\n **Jdbc export capabilities**\n - Execute custom SQL SELECT statements\n - Support for joins, aggregations, and functions\n - Flexible data retrieval from any accessible tables or views\n - Compatible with all major database systems\n **Critical:** WHAT BELONGS IN THIS OBJECT\n - `query` - **ALWAYS REQUIRED** - The SQL SELECT statement\n - `once` - **REQUIRED** when the export's Object Type is `\"once\"` (set _include_once: true)\n - **DO NOT** put `delta` inside this object - delta is handled via the query\n\n **Delta exports (type: \"delta\")**\n For delta/incremental exports, do NOT populate a `delta` object inside `jdbc`.\n Instead, use `{{lastExportDateTime}}` or `{{currentExportDateTime}}` directly in the query:\n ```json\n {\n \"type\": \"delta\",\n \"jdbc\": {\n \"query\": \"SELECT * FROM customers WHERE updatedAt > {{lastExportDateTime}}\"\n }\n }\n ```\n\n **Once exports (type: \"once\")**\n For once exports (mark records as processed), populate `jdbc.once.query`:\n ```json\n {\n \"type\": \"once\",\n \"jdbc\": {\n \"query\": \"SELECT * FROM customers WHERE exported = false\",\n \"once\": {\n \"query\": \"UPDATE customers SET exported = true WHERE id = {{record.id}}\"\n }\n }\n }\n ```\n\n **Standard exports (type: null or not specified)**\n Just provide the query:\n ```json\n {\n \"jdbc\": {\n \"query\": \"SELECT * FROM customers WHERE status = 'ACTIVE'\"\n }\n }\n ```\n required:\n - query\n properties:\n formType:\n # Nullable: the UI persists an explicit null until a mode is picked;\n # null means the 'sql' default.\n type: [string, 'null']\n enum:\n - sql\n - simple\n - null\n x-enumDescriptions:\n sql: The query is authored as raw SQL in the `query` field.\n simple: The query is built visually from `simple.table`, `simple.columns`, and `simple.filter` instead of raw SQL.\n description: |-\n Selects how the export's query is authored. Use `sql` for a hand-written SQL statement in\n `query`; use `simple` to build the query from the `simple` object's table/columns/filter.\n examples: [\"sql\", \"simple\"]\n query:\n type: string\n x-celigo-agent:\n - sql\n x-celigo-ui-override: >-\n Required by the JDBC export form (jdbc.query is required:true). Encoded to mirror the form\n so builders produce connectable configurations.\n description: |-\n SQL SELECT statement executed to retrieve data, from simple table selections to joins and\n aggregations. For delta exports, include {{lastExportDateTime}}/{{currentExportDateTime}}\n in the WHERE clause rather than configuring a separate delta object.\n x-celigo-ai-guidance:\n - |-\n This field contains the SQL SELECT statement that will be executed to fetch data\n from the database.\n The query can range from simple table selections to complex joins and aggregations.\n - |-\n Examples:\n - Basic: `SELECT id, name, email FROM customers WHERE status = 'ACTIVE'`\n - Join: `SELECT o.id, c.name, o.amount FROM orders o JOIN customers c ON o.customer_id = c.id`\n - Aggregate: `SELECT category, COUNT(*) as count FROM orders GROUP BY category`\n - Parameterized: `SELECT * FROM orders WHERE customer_id = {{record.customer_id}}`\n - |-\n ## FOR DELTA EXPORTS (when top-level type is \"delta\")\n Include `{{lastExportDateTime}}` or `{{currentExportDateTime}}` in the WHERE clause:\n - `SELECT * FROM customers WHERE updatedAt > {{lastExportDateTime}}`\n - `SELECT * FROM orders WHERE modified_date >= {{lastExportDateTime}}`\n - |-\n This field contains the SQL SELECT statement that will be executed to fetch data\n from the database.\n The query can range from simple table selections to complex joins and aggregations.\n examples: [\"SELECT * FROM customers WHERE updatedAt > {{lastExportDateTime}}\", \"SELECT * FROM orders WHERE exported = false\", \"SELECT * FROM customers WHERE region = 'Northeast'\", \"SELECT o.id, c.name, o.total FROM orders o JOIN customers c ON o.customer_id = c.id WHERE o.status = 'SHIPPED'\", \"SELECT product_category, SUM(quantity) as total_sold FROM order_items GROUP BY product_category\", \"SELECT * FROM customer_orders WHERE customer_id = {{record.id}}\"]\n once:\n type: object\n description: |\n **CRITICAL: REQUIRED when the export's Object Type is \"once\".**\n\n If Object Type is \"once\", you MUST set _include_once to true (or include this object).\n\n This object has ONLY ONE property: \"query\" (a SQL UPDATE string).\n DO NOT create any other properties like \"update\", \"table\", \"set\", \"where\", etc.\n\n CORRECT format:\n ```json\n {\"query\": \"UPDATE customers SET exported=true WHERE id={{record.id}}\"}\n ```\n\n WRONG format (DO NOT DO THIS):\n ```json\n {\"update\": {\"table\": \"customers\", \"set\": {...}}}\n ```\n properties:\n query:\n type: string\n x-celigo-agent:\n - sql\n description: |\n **REQUIRED** - A SQL UPDATE statement string to mark records as processed.\n\n This is a plain SQL UPDATE query string, NOT a structured object.\n\n The query runs AFTER each record is successfully exported, setting a flag\n to indicate the record has been processed.\n\n Format: \"UPDATE \u003ctable> SET \u003ccolumn>=\u003cvalue> WHERE \u003cid_column>={{record.\u003cid_field>}}\"\n\n Example: \"UPDATE customers SET exported=true WHERE id={{record.id}}\"\n\n The {{record.id}} placeholder is replaced with the actual record ID from each exported row.\n examples: [\"UPDATE orders SET exported=true WHERE id={{record.id}}\", \"UPDATE customers SET exported=true WHERE customer_id={{record.customer_id}}\", \"UPDATE inventory SET is_exported=1 WHERE sku={{record.sku}}\"]\n simple:\n type: object\n description: |-\n Visual query-builder configuration used when `formType` is `simple`; the platform builds\n the SELECT from these instead of raw SQL. Only relevant when `formType` is `simple`.\n properties:\n table:\n type: string\n description: Table or view to select rows from.\n examples: [\"dbo.Orders\", \"public.customers\"]\n columns:\n type: array\n items:\n type: string\n description: Columns to return; an empty list selects all columns.\n examples: [[\"id\", \"name\", \"total\"]]\n filter:\n # Stored as the platform's structured filter object, never a string.\n x-celigo-agent:\n - filter\n description: |-\n Filter applied as the WHERE clause for the generated query, stored as the\n platform's structured filter-rules object.\n allOf:\n - $ref: '../common-schemas/filter.yml#/Filter'\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":7384,"content_sha256":"fe1d705a65622452f44c67af85174fa9675b0290512fc7f7974e10629d0d7330"},{"filename":"references/schemas/mock-output.yml","content":"MockOutput:\n # Nullable: the UI clears mock output by writing null.\n type: [object, 'null']\n description: |\n Sample data that simulates the output from an export for testing and configuration purposes.\n\n Mock output allows you to configure and test flows without executing the actual export or\n waiting for real-time data to arrive. This is particularly useful for:\n - Initial flow configuration and testing\n - Mapping development without requiring live data\n - Generating metadata for downstream flow steps\n - Creating realistic test scenarios\n - Documenting expected data structures\n\n **Structure**\n\n The mock output must follow the integrator.io canonical format, which consists of a\n `page_of_records` array containing record objects. Each record object has a `record`\n property that contains the actual data fields.\n\n ```json\n {\n \"page_of_records\": [\n {\n \"record\": {\n \"field1\": \"value1\",\n \"field2\": \"value2\",\n ...\n }\n },\n ...\n ]\n }\n ```\n\n **Usage**\n\n In a flow test run, and for an upstream lookup during a page-processor preview\n (`previewPageProcessor`), integrator.io substitutes this mock output for the export's live\n response instead of calling the source. A saved mock is not read anywhere else: a target\n lookup in that preview runs its live query, invoking the export\n (`POST /v1/exports/{_id}/invoke`) reads the source, and real runs ignore it. This allows you\n to:\n - Test mappings with representative data\n - Configure downstream flow steps without waiting for real data\n - Simulate various data scenarios\n\n Where it is substituted, the records stand in for the RAW parsed response: the export's own\n `transform` and output `filter` still run on top of them, so downstream steps — and a\n lookup's results mapping — see the transformed, filtered records, not the shape stored here.\n\n In a flow test run of an HTTP export with a saved mock (live probe, 2026-10-21), the export\n still listed `request` and `response` stages: `request` held the request the export composes,\n and `response` held the mock under the source's URL, as did the test run's request log, which\n recorded it as a 200. Read those stages as the substituted mock, not as a live call.\n\n `POST /v1/exports/preview` reads the `mockOutput` in its request body, not the saved one. For\n an export the preview supports (an HTTP export, for example), a body that carries it (as an\n export read back with GET does) returns that snapshot as a lone `parse` stage without calling\n the source; remove `mockOutput` from the body to preview live records.\n\n **Limitations**\n\n - Maximum of 10 records\n - Maximum size of 1 MB\n - Must follow the canonical format shown above\n\n Mock output can be populated automatically from preview data (which is the parsed response,\n before `transform` / `filter`) or entered manually.\n x-celigo-ai-guidance:\n - |-\n Store the RAW parsed response records here, never transformed ones — the platform runs\n the export's `transform` and `filter` on the mock exactly as on live data, and a\n pre-transformed mock would be transformed twice. When you read `mockOutput` to learn\n what a lookup returns, apply the transform first; a results mapping authored against\n the stored shape walks paths that do not exist at runtime.\n - |-\n A saved mockOutput is read in exactly two places — a flow test run, and an upstream lookup\n during a page-processor preview. A target lookup's preview, `POST /v1/exports/{_id}/invoke`\n and every real run call the source and never read it.\n properties:\n page_of_records:\n type: array\n description: |\n Array of record objects in the integrator.io canonical format.\n\n Each item in this array represents one record that would be processed\n by the flow during execution.\n items:\n type: object\n properties:\n record:\n type: object\n description: |\n Container for the actual record data fields.\n\n The structure of this object will vary depending on the specific\n export configuration and the source system's data structure.\n additionalProperties: true\n example:\n id: \"12345\"\n name: \"Sample Product\"\n price: 99.99\n inStock: true\n categories: [\"Electronics\", \"Accessories\"]\n example:\n page_of_records:\n - record:\n id: \"12345\"\n name: \"Sample Product\"\n price: 99.99\n inStock: true\n categories: [\"Electronics\", \"Accessories\"]\n - record:\n id: \"67890\"\n name: \"Another Product\"\n price: 49.99\n inStock: false\n categories: [\"Home\", \"Kitchen\"]","content_type":"application/yaml; charset=utf-8","language":"yaml","size":4932,"content_sha256":"a1637c1e5edb7ff95a797a051bd4633132f3a065c17a122356dab93193861940"},{"filename":"references/schemas/mongodb.yml","content":"MongoDB:\n type: object\n description: |-\n Defines how documents are retrieved from MongoDB collections. Required, with `method` and\n `collection`, when the _connectionId field references a MongoDB connection and the export\n fetches documents; must not be included for other connection types. A change-stream\n listener (`type: stream`) is configured through `cdc` instead and may omit this object, or\n set only `collection` to scope the listener. Supports find queries with optional filter\n criteria and field projections.\n x-celigo-ai-guidance:\n - Configuration object for MongoDB data integration exports.\n - |-\n MongoDB exports currently support the following operational modes:\n - Retrieves documents from specified collections\n - Filters documents based on query criteria\n - Selects specific fields with projections\n - Provides NoSQL flexibility with JSON query syntax\n - |-\n On a find export both `method` and `collection` are validator-enforced: omitting either\n fails the save with 422 `missing_required_field` (\"Expected field: collection to be\n present.\" / \"Expected field: method to be present.\"). On a `type: stream` listener neither\n is required and the object is returned only when `collection` is set.\n properties:\n method:\n type: string\n enum: [\"find\"]\n x-enumDescriptions:\n find: Retrieves documents from a MongoDB collection using filter and projection criteria.\n description: |-\n MongoDB operation used to retrieve documents. Only \"find\" is currently supported,\n equivalent to db.collection.find(filter, projection).\n x-celigo-ai-guidance:\n - Specifies the MongoDB operation to perform when retrieving data.\n - |-\n ## FIELD BEHAVIOR\n\n This field defines the query approach:\n\n - REQUIRED for find exports (not for `type: stream` listeners)\n - Currently only supports \"find\" operations\n - Determines how other parameters are interpreted\n - Corresponds to MongoDB's db.collection.find() method\n - Future versions may support additional methods\n - |-\n ## QUERY METHOD TYPES\n\n ### Find Method\n ```\n \"method\": \"find\"\n ```\n\n - **Behavior**: Retrieves documents from a collection based on filter criteria\n - **MongoDB Equivalent**: db.collection.find(filter, projection)\n - **Required Parameters**: collection\n - **Optional Parameters**: filter, projection\n - **Use Cases**: Standard document retrieval, filtered queries, field selection\n - |-\n ## TECHNICAL CONSIDERATIONS\n\n The method selection influences:\n - What other fields must be provided\n - How the query will be executed against MongoDB\n - What indexing strategies should be applied\n - Performance characteristics of the operation\n\n IMPORTANT: While only \"find\" is currently supported, the schema is designed\n for future expansion to include other MongoDB operations like \"aggregate\"\n for more complex data transformations and aggregations.\n examples: [\"find\"]\n collection:\n type: string\n description: |-\n Name of the MongoDB collection to query. Case-sensitive and must reference an existing\n collection in the connected database. Required on find exports; on a change-stream\n listener (`type: stream`) it is optional and may list several collections separated by\n commas, matching the listener form's \"Collections\" field.\n x-celigo-ai-guidance:\n - Specifies the MongoDB collection to query for documents.\n - |-\n ## FIELD BEHAVIOR\n\n This field identifies the data source:\n\n - REQUIRED for find exports; optional (single or comma-separated) on `type: stream` listeners\n - Must reference a valid collection in the MongoDB database\n - Case-sensitive according to MongoDB collection naming\n - The primary container for documents to be retrieved\n examples: [\"customers\", \"orders\", \"product_catalog\", \"users, orders, products\"]\n filter:\n type: string\n description: |-\n MongoDB query document, as a JSON string, that selects which documents to export; omit\n to return every document in the collection. Supports standard MongoDB query operators\n and handlebars variables for dynamic values — for example\n {\"lastModified\": {\"$gt\": \"{{lastRun}}\"}} for incremental processing.\n x-celigo-ai-guidance:\n - Defines query criteria for selecting documents from the collection.\n - |-\n ## FIELD BEHAVIOR\n\n This field narrows document selection:\n\n - OPTIONAL: If omitted, all documents in the collection are returned\n - Contains a MongoDB query document as a JSON string\n - Supports all standard MongoDB query operators\n - Provides precise control over which documents are retrieved\n - |-\n ## QUERY PATTERNS\n\n ### Simple Equality Query\n ```\n \"filter\": \"{\"status\": \"active\"}\"\n ```\n\n - **Behavior**: Returns only documents where status equals \"active\"\n - **MongoDB Equivalent**: db.collection.find({\"status\": \"active\"})\n - **Matching Documents**: {\"_id\": 1, \"status\": \"active\", \"name\": \"Example\"}\n - **Use Cases**: Status filtering, category selection, type filtering\n\n ### Comparison Operator Query\n ```\n \"filter\": \"{\"createdDate\": {\"$gt\": \"2023-01-01T00:00:00Z\"}}\"\n ```\n\n - **Behavior**: Returns documents created after January 1, 2023\n - **MongoDB Equivalent**: db.collection.find({\"createdDate\": {\"$gt\": \"2023-01-01T00:00:00Z\"}})\n - **Operators**: $eq, $gt, $gte, $lt, $lte, $ne, $in, $nin\n - **Use Cases**: Date ranges, numeric thresholds, incremental processing\n\n ### Logical Operator Query\n ```\n \"filter\": \"{\"$or\": [{\"status\": \"pending\"}, {\"status\": \"processing\"}]}\"\n ```\n\n - **Behavior**: Returns documents with either pending or processing status\n - **MongoDB Equivalent**: db.collection.find({\"$or\": [{\"status\": \"pending\"}, {\"status\": \"processing\"}]})\n - **Operators**: $and, $or, $nor, $not\n - **Use Cases**: Multiple conditions, alternative criteria, complex filtering\n\n ### Nested Document Query\n ```\n \"filter\": \"{\"address.country\": \"USA\"}\"\n ```\n\n - **Behavior**: Returns documents where the nested country field equals \"USA\"\n - **MongoDB Equivalent**: db.collection.find({\"address.country\": \"USA\"})\n - **Dot Notation**: Accesses nested document fields\n - **Use Cases**: Nested data filtering, object property matching\n\n ### Handlebars Template Query\n ```\n \"filter\": \"{\"customerId\": \"{{record.customer_id}}\", \"status\": \"{{record.status}}\"}\"\n ```\n\n - **Behavior**: Dynamically filters based on record field values\n - **MongoDB Equivalent**: db.collection.find({\"customerId\": \"123\", \"status\": \"active\"})\n - **Template Variables**: Values replaced at runtime with actual record data\n - **Use Cases**: Dynamic filtering, context-aware queries, relational lookups\n\n ### Incremental Processing Query\n ```\n \"filter\": \"{\"lastModified\": {\"$gt\": \"{{lastRun}}\"}}\"\n ```\n\n - **Behavior**: Returns only documents modified since last execution\n - **MongoDB Equivalent**: db.collection.find({\"lastModified\": {\"$gt\": \"2023-06-15T10:30:00Z\"}})\n - **System Variables**: {{lastRun}} replaced with timestamp of previous execution\n - **Use Cases**: Change data capture, delta synchronization, incremental updates\n examples: ['{\"status\": \"active\"}', '{\"createdDate\": {\"$gt\": \"{{lastRun}}\"}}', '{\"$or\": [{\"status\": \"pending\"}, {\"status\": \"processing\"}]}', '{\"address.country\": \"USA\", \"age\": {\"$gte\": 21}}', '{\"customerId\": \"{{record.customer_id}}\", \"orderStatus\": \"{{record.status}}\"}', '{\"productId\": \"{{record.product.id}}\", \"category\": \"{{record.product.category}}\"}']\n projection:\n type: string\n description: |-\n MongoDB projection document, as a JSON string, that limits which fields are returned;\n omit to return all fields. Use 1 to include fields or 0 to exclude them — the two modes\n cannot be mixed except for _id, which is included by default unless explicitly excluded.\n Projection affects only the shape of returned documents, not which documents match.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field optimizes data retrieval:\n\n - OPTIONAL: If omitted, all fields are returned\n - Contains a MongoDB projection document as a JSON string\n - Can include fields (1) or exclude fields (0), but not both (except _id)\n - Helps minimize data transfer by selecting only needed fields\n - |-\n ## PROJECTION PATTERNS\n\n ### Field Inclusion Projection\n ```\n \"projection\": \"{\"name\": 1, \"email\": 1, \"_id\": 0}\"\n ```\n\n - **Behavior**: Returns only name and email fields, excludes _id\n - **MongoDB Equivalent**: db.collection.find({}, {\"name\": 1, \"email\": 1, \"_id\": 0})\n - **Result Format**: {\"name\": \"Example\", \"email\": \"[email protected]\"}\n - **Use Cases**: Specific field selection, minimizing payload size\n\n ### Field Exclusion Projection\n ```\n \"projection\": \"{\"password\": 0, \"internal_notes\": 0}\"\n ```\n\n - **Behavior**: Returns all fields except password and internal_notes\n - **MongoDB Equivalent**: db.collection.find({}, {\"password\": 0, \"internal_notes\": 0})\n - **Result Impact**: Removes sensitive or unnecessary fields\n - **Use Cases**: Security filtering, removing large fields, data protection\n\n ### Nested Field Projection\n ```\n \"projection\": \"{\"profile.firstName\": 1, \"profile.lastName\": 1, \"orders\": 1, \"_id\": 0}\"\n ```\n\n - **Behavior**: Returns only specific nested fields and the orders array\n - **MongoDB Equivalent**: db.collection.find({}, {\"profile.firstName\": 1, \"profile.lastName\": 1, \"orders\": 1, \"_id\": 0})\n - **Dot Notation**: Accesses specific nested document fields\n - **Use Cases**: Partial nested document selection, specific array inclusion\n - |-\n ## TECHNICAL CONSIDERATIONS\n\n - Maximum size: 128KB\n - Must be a valid JSON string representing a MongoDB projection\n - Cannot mix inclusion and exclusion modes (except _id field)\n - _id field is included by default unless explicitly excluded\n - Projection does not affect which documents are returned, only their fields\n\n IMPORTANT: When working with nested documents or arrays, be aware that including\n a specific field path does not automatically include parent documents or arrays.\n For example, including \"addresses.zipcode\" will only return that specific field,\n not the entire addresses array or documents within it.\n examples: ['{\"name\": 1, \"email\": 1, \"_id\": 0}', '{\"password\": 0, \"internal_notes\": 0}', '{\"profile.firstName\": 1, \"profile.lastName\": 1, \"orders\": 1, \"_id\": 0}', '{\"metadata\": 0, \"system_tags\": 0}']\n pipeline:\n type: string\n description: |-\n MongoDB aggregation pipeline, as a JSON-array string of stages, used instead of a plain\n find when documents need to be transformed, joined, or grouped server-side (e.g. `$match`,\n `$lookup`, `$group`). When set, it takes the place of `filter`/`projection`. Supports\n handlebars variables for incremental processing.\n examples: ['[{\"$match\": {\"status\": \"active\"}}, {\"$project\": {\"name\": 1, \"email\": 1}}]', '[{\"$match\": {\"lastModified\": {\"$gt\": \"{{lastRun}}\"}}}]']\n readPreference:\n type: string\n enum:\n - doNotOverride\n - primary\n - primaryPreferred\n - secondary\n - secondaryPreferred\n - nearest\n x-enumDescriptions:\n doNotOverride: Uses the read preference configured on the connection (the default).\n primary: Reads only from the replica-set primary.\n primaryPreferred: Reads from the primary when available, otherwise a secondary.\n secondary: Reads only from a replica-set secondary.\n secondaryPreferred: Reads from a secondary when available, otherwise the primary.\n nearest: Reads from the member with the lowest network latency.\n default: doNotOverride\n description: |-\n Overrides which replica-set member this export reads from, trading consistency for load\n distribution. Leave at `doNotOverride` to inherit the connection's setting.\n pathToRecords:\n $ref: './record-extraction.yml#/PathToRecords'\n includeParentData:\n $ref: './record-extraction.yml#/IncludeParentData'\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":13177,"content_sha256":"33e0639755f7e76d2739b5d65b7dd01f60b7119dedead4b43a5732e0e8591e7e"},{"filename":"references/schemas/netsuite.yml","content":"NetSuite:\n type: object\n description: |-\n NetSuite-specific export configuration. Required when `_connectionId` references a NetSuite\n connection; omit for all other connection types. Supports saved-search, RESTlet, and\n distributed (SuiteApp) exports, plus two file cabinet modes: blob exports transfer files\n as-is (set the export's top-level `type` to `blob` and configure `netsuite.blob`), while\n file exports parse file contents into records (leave the export's top-level `type` unset\n and configure `netsuite.file`).\n x-celigo-ai-guidance:\n - Configuration object for NetSuite data integration exports.\n - |-\n This object is REQUIRED when the _connectionId field references a NetSuite\n connection and must not be included for other connection types.\n It defines how data is extracted from NetSuite, including saved searches,\n RESTlets, and distributed/SuiteApp exports.\n - |-\n ## NETSUITE EXPORT MODES\n\n NetSuite exports support several operating modes:\n\n 1. **Saved Search Exports** - Uses NetSuite saved searches to retrieve data\n 2. **RESTlet Exports** - Uses custom RESTlet scripts for data retrieval\n 3. **Distributed Exports** - Uses SuiteApp for real-time or batch processing\n 4. **Blob Exports** - Retrieves files from the NetSuite file cabinet and transfers them WITHOUT parsing them into records (raw binary transfer)\n 5. **File Exports** - Retrieves files from the NetSuite file cabinet and PARSES them into records (CSV, XML, JSON, etc.)\n - |-\n ## CRITICAL: Blob vs File Export Configuration\n\n The export `type` field at the top level determines whether file content is parsed:\n\n - **For Blob Exports (no parsing)**: Set the export's `type: \"blob\"` AND configure `netsuite.blob`\n - **For File Exports (with parsing)**: Leave the export's `type` as null/undefined AND configure `netsuite.file`\n\n Do NOT set `type: \"blob\"` when you want file content parsed into records. The \"blob\" type is specifically for raw file transfers without any parsing.\n - |-\n ## IMPLEMENTATION REQUIREMENTS\n\n - For saved search exports: Configure the `searches` or `type` properties\n - For RESTlet exports: Configure the `restlet` property with script details\n - For distributed exports: Configure the `distributed` property\n - For blob exports (no parsing): Set export `type: \"blob\"` and configure `netsuite.blob`\n - For file exports (with parsing): Leave export `type` null and configure `netsuite.file`\n properties:\n type:\n type: string\n enum:\n - search\n - basicSearch\n - metadata\n - selectoption\n - restlet\n - getList\n - getServerTime\n - distributed\n - file\n x-enumDescriptions:\n search: Retrieves records with a NetSuite saved search over SOAP web services; the export form marks this path deprecated in favour of the REST APIs for new exports\n basicSearch: Retrieves records using a basic NetSuite search query.\n metadata: Retrieves metadata about a NetSuite record type.\n selectoption: Retrieves the available select options for a specific field.\n restlet: Retrieves data using a custom NetSuite RESTlet script.\n getList: Retrieves a list of records by their internal IDs.\n getServerTime: Retrieves the current NetSuite server time.\n distributed: Uses the SuiteApp for real-time, event-driven, or batch exports\n file: Exports and parses files from the NetSuite file cabinet into records.\n description: |-\n Controls how data is retrieved from NetSuite and which sibling object must be configured:\n `search` pairs with `netsuite.searches`, `restlet` with `netsuite.restlet`, `distributed`\n with `netsuite.distributed`, and `file` with `netsuite.file.folderInternalId`. For blob\n exports (raw file transfer without parsing), leave this field unset, set the export's\n top-level `type` to `blob`, and configure `netsuite.internalId` instead. For lookups\n (`isLookup: true`), prefer `restlet`, which supports `suiteapp2.0` saved searches with\n dynamic inputs; `search` is limited for dynamic lookups.\n examples: [\"file\", \"search\"]\n x-celigo-ai-guidance:\n - |-\n ## CRITICAL: File exports vs Blob exports\n\n - **File exports (with parsing)**: Set netsuite.type to \"file\" and configure netsuite.file.folderInternalId\n - **Blob exports (raw transfer, no parsing)**: Leave netsuite.type BLANK/null, set the export's top-level type to \"blob\", and configure netsuite.internalId\n\n Do NOT set netsuite.type to \"file\" for blob exports. For blob exports, this property should be omitted or null.\n - |-\n ## RECOMMENDED TYPES:\n\n - **For Lookups (isLookup: true)**:\n - **PREFER \"restlet\"**: This allows you to use `suiteapp2.0` saved searches with dynamic inputs easily.\n - **AVOID \"search\"**: Standard search type is often limited for dynamic lookups.\n - |-\n ## IMPLEMENTATION GUIDANCE:\n - For file exports WITH parsing: Set netsuite.type to \"file\" and configure netsuite.file.folderInternalId\n - For blob exports (no parsing): Leave netsuite.type blank, set export type to \"blob\", configure netsuite.internalId\n - For saved search exports: Set type to \"search\" and configure netsuite.searches\n - For RESTlet exports: Set type to \"restlet\" and configure netsuite.restlet\n - For distributed/real-time exports: Set type to \"distributed\" and configure netsuite.distributed\n - |-\n ## EXAMPLES:\n - \"file\" - For file cabinet exports with parsing\n - \"search\" - For saved search exports\n - null - For blob exports (raw file transfer without parsing)\n searches:\n type: array\n description: |-\n Saved search configurations that query NetSuite for records. Each entry references a\n saved search by internal ID, names the record type being searched, and can add filter\n criteria.\n x-celigo-ai-guidance:\n - |-\n An array of search configurations used to query and retrieve data from NetSuite.\n Each search object defines a saved search or ad-hoc query configuration.\n - |-\n ## STRUCTURE\n Each item in the array is an object with the following properties:\n - savedSearchId: The internal ID of a saved search in NetSuite (string)\n - recordType: The NetSuite record type being searched (string, e.g., \"customer\", \"salesorder\")\n - criteria: Array of search criteria/filters (optional)\n - |-\n ## EXAMPLES\n ```json\n [\n {\n \"savedSearchId\": \"10\",\n \"recordType\": \"customer\",\n \"criteria\": []\n }\n ]\n ```\n - |-\n ## IMPLEMENTATION GUIDANCE\n - Use savedSearchId to reference an existing saved search in NetSuite\n - recordType should match a valid NetSuite record type\n - criteria can be used to add additional filters to the search\n items:\n type: object\n properties:\n savedSearchId:\n type: string\n description: Internal ID of the NetSuite saved search to run.\n recordType:\n type: string\n description: |-\n NetSuite record type being searched, as its exact lowercase script ID\n (e.g. \"customer\", \"salesorder\") — not the display name.\n x-celigo-ai-guidance:\n - |-\n Must be the exact lowercase script ID as defined in NetSuite (e.g., \"customer\",\n \"salesorder\", \"invoice\", \"vendorbill\").\n This is NOT the display name - use the script ID which is always lowercase with no spaces.\n criteria:\n type: array\n description: Additional filter criteria applied on top of the saved search.\n x-celigo-ai-guidance:\n - |-\n Array of search filters to apply.\n Each criterion becomes a `search.createFilter()` call inside Celigo's NetSuite\n SuiteScript runtime.\n - |-\n ## REQUIRED FIELDS PER CRITERION\n The Celigo backend translates each criterion into a\n `search.createFilter({ name, operator, values })` call.\n `field` (script-id of the search column) and `operator`\n are mandatory -- omitting either crashes the saved search\n at runtime with `search.createFilter: Missing a required\n argument: name`.\n - |-\n ## EXAMPLE\n ```json\n [\n {\"field\": \"trandate\", \"operator\": \"after\", \"searchValue\": \"2025-01-01\"},\n {\"field\": \"status\", \"operator\": \"anyof\", \"searchValue\": [\"SalesOrd:B\"]}\n ]\n ```\n - |-\n ## FIELD vs SEARCH COLUMN NAMING\n The `field` value MUST be a valid search-column `id` for\n the chosen `recordType` (the ones listed in the\n SEARCH COLUMNS metadata block) -- NOT a body-field name.\n For example, Sales Order shipping address line 1 is\n `shipaddress1` (search column), not `shipaddr1` (body\n field).\n items:\n type: object\n properties:\n field:\n type: string\n description: |-\n Script-id of the search column to filter on. REQUIRED.\n Must come from the record type's SEARCH COLUMNS list\n (NOT the body-fields list). Becomes the `name`\n argument of NetSuite's `search.createFilter()`.\n operator:\n type: string\n description: |-\n Comparison operator. REQUIRED. Supported values\n depend on the column's data type (e.g. `is`,\n `isnot`, `after`, `before`, `anyof`,\n `noneof`, `contains`, `startswith`,\n `greaterthan`, `lessthan`). Must be a string\n NetSuite recognises -- mismatched operators cause\n runtime rejection.\n join:\n type: string\n description: |-\n Optional join name when filtering through a related\n record (e.g. `customer` from a sales order). Use\n only when the column lives on a related record and\n NetSuite supports the join from the base record type.\n searchValue:\n description: |-\n Value (or array of values for `anyof`-style\n operators) to compare against the column. Required\n for most operators; omit for the few unary operators\n (e.g. `isempty`, `isnotempty`).\n required:\n - field\n - operator\n metadata:\n type: object\n description: Supplemental metadata associated with the NetSuite export.\n selectoption:\n type: object\n description: |-\n Configuration for `selectoption` exports, which retrieve the available select options\n for a NetSuite field.\n x-celigo-ai-guidance:\n - |-\n selectoption: > Represents a selectable option within a NetSuite field,\n typically used in dropdown menus, radio buttons, or other selection controls.\n Each selectoption consists of a user-friendly label and an associated value that\n uniquely identifies the option internally.\n This structure enables consistent data entry, filtering,\n and categorization within NetSuite forms and records.\n - |-\n ## FIELD BEHAVIOR:\n - Defines a single, discrete choice available to users in selection interfaces such as dropdowns, radio buttons, or multi-select lists.\n - Can be part of a collection of options presented to the user for making a selection.\n - Includes both a display label (visible to users) and a corresponding value (used internally or in API interactions).\n - Supports filtering, categorization, and conditional logic based on the selected option.\n - May be dynamically generated or statically defined depending on the field configuration.\n - |-\n ## IMPLEMENTATION GUIDANCE:\n - Assign a unique and stable value to each selectoption to prevent ambiguity and maintain data integrity.\n - Use clear, concise, and user-friendly labels that accurately describe the option's meaning.\n - Validate option values against expected data types and formats to ensure compatibility with backend processing.\n - Implement localization strategies for labels to support multiple languages without altering the underlying values.\n - Consistently apply selectoption structures across all fields requiring predefined choices to standardize user experience.\n - Consider accessibility best practices when designing labels and selection controls.\n - |-\n ## EXAMPLES:\n - { label: \"Active\", value: \"1\" }\n - { label: \"Inactive\", value: \"2\" }\n - { label: \"Pending Approval\", value: \"3\" }\n - { label: \"High Priority\", value: \"high\" }\n - { label: \"Low Priority\", value: \"low\" }\n - |-\n ## IMPORTANT NOTES:\n - The label is intended for display purposes and may be localized; the value is the definitive identifier used in data processing and API calls.\n - Values should remain consistent over time to avoid breaking integrations or corrupting data.\n - When supporting multiple languages, labels should be translated appropriately while keeping values unchanged.\n - Changes to selectoption values or labels should be managed carefully to prevent unintended side effects.\n - Selectoption entries may be influenced by the context of the parent record, user roles, or permissions.\n - |-\n ## DEPENDENCY CHAIN:\n - Utilized within field definitions that support selection inputs (e\n customFieldMetadata:\n type: object\n description: Metadata describing the custom fields defined in the NetSuite account.\n x-celigo-ai-guidance:\n - |-\n customFieldMetadata:\n Metadata information related to custom fields defined within the NetSuite environment,\n providing comprehensive details about each custom field's configuration,\n behavior,\n and constraints to facilitate accurate data handling and UI generation.\n - |-\n ## FIELD BEHAVIOR:\n - Contains detailed metadata about custom fields, including their definitions, types, configurations, and constraints.\n - Provides contextual information necessary for understanding, validating, and manipulating custom fields programmatically.\n - May include attributes such as field ID, label, data type, default values, validation rules, display settings, sourcing information, and field dependencies.\n - Used to dynamically interpret or generate UI elements, data validation logic, or data structures based on custom field configurations.\n - Reflects the current state of custom fields as defined in the NetSuite account, enabling synchronization between the API consumer and the NetSuite environment.\n - |-\n ## IMPLEMENTATION GUIDANCE:\n - Ensure that the metadata accurately reflects the current state of custom fields in the NetSuite account by synchronizing regularly or on configuration changes.\n - Update the metadata whenever custom fields are added, modified, or removed to maintain consistency and prevent data integrity issues.\n - Use this metadata to validate input data against custom field constraints (e.g., data type, required status, allowed values) before processing or submission.\n - Consider caching metadata for performance optimization but implement mechanisms to refresh it periodically or on-demand to capture updates.\n - Handle cases where customFieldMetadata might be null, incomplete, or partially loaded gracefully, including fallback logic or error handling.\n - Respect user permissions and access controls when retrieving or exposing custom field metadata to ensure compliance with security policies.\n - |-\n ## EXAMPLES:\n - A custom field metadata object describing a custom checkbox field with ID \"custfield_123\", label \"Approved\", default value false, and display type \"inline\".\n - Metadata for a custom list/record field specifying the list of valid options, their internal IDs, and whether multiple selections are allowed.\n - Information about a custom date field including its date format, minimum and maximum allowed dates, and any validation rules applied.\n - Metadata describing a custom currency field with precision settings and default currency.\n - |-\n ## IMPORTANT NOTES:\n - The structure and content of customFieldMetadata may vary depending on the NetSuite configuration, customizations, and API version.\n - Access to custom field metadata may require appropriate permissions within the NetSuite environment; unauthorized access may result in incomplete or no metadata.\n skipGrouping:\n type: boolean\n description: |-\n When true, each saved-search result row becomes its own record. When false or omitted,\n consecutive result rows that share the record's internal id are grouped into one record\n shaped as an array of rows — the *Group rows* option in the UI.\n x-celigo-ai-guidance:\n - |-\n A saved search that returns several rows per record (a sublist or multi-select column)\n is grouped by leaving this false or omitted: each record arrives as an array of its\n rows, every row carrying the same internal id as the record's trace key, and the\n preview reports no duplicate trace keys. With `true` the same search emits one object\n per row and the repeated internal ids surface as duplicate trace keys\n (`traceKeysDuplicate` in the preview). Grouping keys on the internal id only — a\n different key needs a `preSavePage` hook, which sees one page at a time.\n - |-\n Celigo's documentation requires the saved search to be sorted by Internal ID for\n grouping to apply.\n - |-\n The generic `groupByFields` setting is not the NetSuite mechanism: the platform rejects\n it on a NetSuite export with `[invalid_config] netsuite connection is not supported for\n groupByFields property`.\n - |-\n Set `true` for an ad-hoc RESTlet search (criteria and columns without a saved\n search): grouping needs the search sorted by internal id, and an unsorted ad-hoc\n search fails with `[search_not_sorted]`.\n - |-\n For `exportSelect` / `refreshableSelect` dropdowns backed by a NetSuite RESTlet, set `skipGrouping: true`: the UI's option mapper expects flat per-row results, and the platform otherwise wraps each row in its own grouping list (every option's `label` then renders as `undefined`).\n statsOnly:\n type: boolean\n description: |-\n When true, returns only summary statistics about matching records instead of the\n detailed records themselves.\n x-celigo-ai-guidance:\n - |-\n statsOnly indicates whether the API response should include only aggregated\n statistical summary data without any detailed individual records.\n This property is used to optimize response size and improve performance when\n detailed data is unnecessary.\n - |-\n ## FIELD BEHAVIOR:\n - When set to true, the API returns only summary statistics such as counts, averages, sums, or other aggregate metrics.\n - When set to false or omitted, the response includes both detailed data records and the associated statistical summaries.\n - Helps reduce network bandwidth and processing time by excluding verbose record-level data.\n - Primarily intended for use cases like dashboards, reports, or monitoring tools where only high-level metrics are required.\n - |-\n ## IMPLEMENTATION GUIDANCE:\n - Default the value to false to ensure full data retrieval unless explicitly requesting summary-only data.\n - Validate that the input is a boolean to prevent unexpected API behavior.\n - Use this flag selectively in scenarios where detailed records are not needed to avoid loss of critical information.\n - Ensure the API endpoint supports this flag before usage, as some endpoints may not implement statsOnly functionality.\n - Adjust client-side logic to handle different response structures depending on the flag's value.\n - |-\n ## EXAMPLES:\n - statsOnly: true — returns only aggregated statistics such as total counts, averages, or sums without any detailed entries.\n - statsOnly: false — returns full detailed data records along with statistical summaries.\n - statsOnly omitted — defaults to false, returning detailed data and statistics.\n - |-\n ## IMPORTANT NOTES:\n - Enabling statsOnly disables access to individual record details, which may limit in-depth data analysis.\n - The response schema changes significantly when statsOnly is true; clients must handle these differences gracefully.\n - Some API endpoints may not support this property; verify compatibility in the API documentation.\n - Pagination parameters may be ignored or behave differently when statsOnly is enabled, since detailed records are excluded.\n - |-\n ## DEPENDENCY CHAIN:\n - May interact with filtering, sorting, or date range parameters that influence the statistical data returned.\n - Can affect pagination logic because detailed records are omitted when statsOnly is true.\n - Dependent on the API endpoint's support for summary-only responses.\n - |-\n ## TECHNICAL DETAILS:\n - Data type: Boolean.\n - Default value: false.\n - Typically implemented as a query parameter or part of the request payload depending on API design.\n - Alters the response payload structure by excluding detailed record arrays and including only aggregated metrics.\n - Helps optimize API performance and reduce response payload size in scenarios where detailed data is unnecessary.\n internalId:\n type: string\n description: |-\n Internal ID of the file in the NetSuite file cabinet to export. Required for blob\n exports (the export's top-level `type` is `blob`); the file is transferred as-is\n without parsing. For parsed file exports, use `netsuite.file.folderInternalId` instead.\n examples: [\"12345\", \"67890\"]\n x-celigo-ai-guidance:\n - |-\n ## CRITICAL: Required for blob exports\n\n This property is REQUIRED when the export type is \"blob\". For blob exports, you must specify the internalId of the file to export from the NetSuite file cabinet.\n - |-\n ## FIELD BEHAVIOR:\n - Identifies a specific file in the NetSuite file cabinet by its internal ID\n - Required for blob exports (raw binary file transfers without parsing)\n - The file at this internal ID will be exported as-is without parsing\n - |-\n ## IMPLEMENTATION GUIDANCE:\n - For blob exports: Set netsuite.type to \"blob\" (on the export, not netsuite) and provide netsuite.internalId\n - Obtain the file's internalId from NetSuite's file cabinet or via API\n - Validate the internalId corresponds to an existing file before export\n - |-\n ## EXAMPLES:\n - \"12345\" - Internal ID of a specific file\n - \"67890\" - Another file internal ID\n - |-\n ## IMPORTANT NOTES:\n - This is different from netsuite.file.folderInternalId which specifies a folder for file exports with parsing\n - For blob exports: Use netsuite.internalId (file ID)\n - For file exports with parsing: Use netsuite.file.folderInternalId (folder ID)\n blob:\n type: object\n properties:\n purgeFileAfterExport:\n type: boolean\n description: |-\n When true, permanently deletes the file from the NetSuite file cabinet after a\n successful export. When false or omitted, the file is left in place.\n x-celigo-ai-guidance:\n - |-\n purgeFileAfterExport:\n Whether to delete the file from the system after it has been successfully exported.\n This property controls the automatic removal of the source file post-export to\n help manage storage and maintain system cleanliness.\n - |-\n ## FIELD BEHAVIOR:\n - Determines if the exported file should be removed from the source storage immediately after a successful export operation.\n - When set to `true`, the system deletes the file as soon as the export completes without errors.\n - When set to `false` or omitted, the file remains intact in its original location after export.\n - Facilitates automated cleanup of files to prevent unnecessary storage consumption.\n - Does not affect the export process itself; deletion occurs only after confirming export success.\n - |-\n ## IMPLEMENTATION GUIDANCE:\n - Confirm that the export operation has fully completed and succeeded before initiating file deletion to avoid data loss.\n - Verify that the executing user or system has sufficient permissions to delete files from the source location.\n - Assess downstream workflows or processes that might require access to the file after export before enabling purging.\n - Implement logging or notification mechanisms to record when files are purged for audit trails and troubleshooting.\n - Consider integrating with retention policies or backup systems to prevent accidental loss of important data.\n - |-\n ## EXAMPLES:\n - `true` — The file will be deleted immediately after a successful export.\n - `false` — The file will remain in the source location after export.\n - Omitted or `null` — Defaults to `false`, meaning the file is retained post-export.\n - |-\n ## IMPORTANT NOTES:\n - File deletion is permanent and cannot be undone; ensure that the file is no longer needed before enabling this option.\n - Use caution in multi-user or multi-process environments where files may be shared or required beyond the export operation.\n - Immediate purging may interfere with backup, archival, or compliance requirements if files are deleted too soon.\n - Consider implementing safeguards or confirmation steps if enabling automatic purging in production environments.\n - |-\n ## DEPENDENCY CHAIN:\n - Relies on the successful completion of the export operation to trigger file deletion.\n - Dependent on file system permissions and access controls to allow deletion.\n - May be affected by other system settings related to file retention, archival, or cleanup policies.\n - Could interact with error handling mechanisms to prevent deletion if export fails or is incomplete.\n - |-\n ## TECHNICAL DETAILS:\n - Typically represented as a boolean value (`true` or `false`).\n - Default behavior is to retain files unless explicitly set to `true`.\n - Deletion should be performed using secure\n description: |-\n Configuration for blob exports, which transfer files from the NetSuite file cabinet\n as-is without parsing them into records. To export a blob, set the export's top-level\n `type` to `blob`, set `netsuite.internalId` to the file's internal ID, and leave\n `netsuite.type` unset. Use `netsuite.file` instead when file contents should be parsed\n into records.\n x-celigo-ai-guidance:\n - |-\n blob:\n Configuration for retrieving raw binary files from NetSuite file cabinet WITHOUT\n parsing them into records.\n Use this for binary file transfers (images, PDFs,\n executables) where the file content should be transferred as-is.\n - |-\n ## CRITICAL: Blob export configuration\n\n For blob exports, configure:\n 1. Set the export's top-level `type` to \"blob\"\n 2. Set `netsuite.internalId` to the file's internal ID\n 3. Leave `netsuite.type` blank/null (do NOT set it to \"file\")\n 4. Optionally configure `netsuite.blob.purgeFileAfterExport`\n - |-\n ## When to use blob vs file\n - Blob exports: Raw binary transfer WITHOUT parsing - leave netsuite.type blank\n - File exports: Parse file contents into records - set netsuite.type to \"file\"\n\n Do NOT use blob configuration when you want file content parsed into data records.\n - |-\n ## FIELD BEHAVIOR:\n - Stores raw binary data including files, images, audio, video, or any non-textual content.\n - Supports download operations for binary content from the NetSuite file cabinet.\n - File content is transferred as-is without any parsing or transformation.\n - May be immutable or mutable depending on the specific NetSuite entity and operation.\n - Requires careful handling to maintain data integrity during transmission and storage.\n - |-\n ## IMPLEMENTATION GUIDANCE:\n - Always encode binary data (e.g., using base64) when transmitting over text-based protocols such as JSON or XML to ensure data integrity.\n - Validate the size of the blob against NetSuite API limits and storage constraints to prevent errors or truncation.\n - Implement secure handling practices, including encryption in transit and at rest, to protect sensitive binary data.\n - Use appropriate MIME/content-type headers when uploading or downloading blobs to correctly identify the data format.\n - Consider chunked uploads/downloads or streaming for large blobs to optimize performance and resource usage.\n - Ensure consistent encoding and decoding mechanisms between client and server to avoid data corruption.\n - |-\n ## EXAMPLES:\n - A base64-encoded PDF document attached to a NetSuite customer record.\n - An image file (PNG or JPEG) stored as a blob for product catalog entries.\n - A binary export of transaction data in a proprietary format used for integration with external systems.\n - Audio or video files associated with marketing campaigns or training materials.\n - Encrypted binary blobs containing sensitive configuration or credential data.\n - |-\n ## IMPORTANT NOTES:\n - Blob size may be limited by NetSuite API constraints or underlying storage capabilities; exceeding these limits can cause failures.\n - Encoding and decoding must be consistent and correctly implemented to prevent data corruption or loss.\n - Large blobs should be handled using chunked or streamed transfers to avoid memory issues and improve reliability.\n - Security is paramount; blobs may contain sensitive information requiring encryption and strict access controls.\n - Access to blob data typically requires proper authentication and authorization aligned with NetSuite's security model.\n - |-\n ## DEPENDENCY CHAIN:\n - Dependent on authentication and authorization mechanisms\n restlet:\n type: object\n properties:\n recordType:\n type: string\n description: |-\n NetSuite record type the RESTlet operates on, as its script ID. Custom record\n types use the \"customrecord_\" prefix.\n examples: [\"customer\", \"salesOrder\", \"invoice\", \"employee\", \"customrecord_myCustomRecord\", \"vendor\", \"purchaseOrder\"]\n x-celigo-ai-guidance:\n - |-\n Saved searches over any transaction type (sales orders, purchase orders,\n invoices, ...) run with `recordType: \"transaction\"` and narrow the type through the\n search or a `criteria` row such as `{\"field\": \"type\", \"operator\": \"anyof\",\n \"searchValue\": [\"PurchOrd\"]}`.\n batchSize:\n type: number\n description: |-\n Number of records the RESTlet returns per request (default 1000). This, not the\n export's `pageSize`, sizes each NetSuite call; the export keeps requesting until the\n search is exhausted. Larger batches reduce round-trips but raise NetSuite governance\n and memory use per call; tune to the record's size. The RESTlet caps a call at 1000,\n so a larger value acts as 1000. In a test run or preview the request is capped at the\n export's `pageSize` (default 20) instead.\n examples: [100, 200, 1000]\n searchId:\n type: [string, 'null']\n description: |-\n Internal ID of the NetSuite saved search the RESTlet executes. Results follow the\n saved search's current definition, so editing the search in NetSuite changes the\n export output without changing this ID.\n examples: [\"9013\", \"customsearch_plan_velocity\", \"-1\"]\n x-celigo-ai-guidance:\n - |-\n Accepts the saved search's internal id (`\"9013\"`) or script id\n (`\"customsearch_...\"`). `\"-1\"` means no saved search: the RESTlet builds an ad-hoc\n search from `recordType` + `criteria` + `columns` (see `columns` — required in\n that mode, along with `netsuite.skipGrouping: true`). Omitting both `searchId`\n and `columns` fails with `[celigo_config_error] searchId or columns are required`.\n useSS2Restlets:\n type: boolean\n description: |-\n When true, calls SuiteScript 2.0 RESTlets. When false or omitted, legacy\n SuiteScript 1.0 RESTlets are used.\n Set at step creation together with restletVersion — the same\n creation-time-only constraint applies (see restletVersion).\n x-celigo-ai-guidance:\n - |-\n An authoring-time choice: set it when creating the step (true unless the user\n explicitly needs a legacy version). It cannot be toggled on a saved step — moving\n an existing step to a different version means recreating or cloning it.\n - |-\n SuiteScript 2.0 steps need the integrator.io SuiteApp installed in the target\n NetSuite account, and any SuiteScript hooks must be written for the step's\n version — verify both before cutting a recreated step over.\n restletVersion:\n type: string\n enum:\n - suitebundle\n - suiteapp1.0\n - suiteapp2.0\n x-enumDescriptions:\n suitebundle: Legacy SuiteBundle RESTlet version; slated for deprecation — NetSuite no longer updates SuiteScript 1.0.\n suiteapp1.0: SuiteApp 1.0 RESTlet version using SuiteScript 1.0.\n suiteapp2.0: SuiteApp 2.0 RESTlet version using SuiteScript 2.0, the modern default.\n x-lowercase: true\n description: |-\n RESTlet version the export invokes. Defaults to `suiteapp2.0` when\n `useSS2Restlets` is true, `suitebundle` otherwise.\n The version is fixed when the step is created — the Advanced selector is\n disabled on existing steps, and migrating an existing export to a\n different version means recreating or cloning it.\n x-celigo-ai-guidance:\n - |-\n The NetSuite API version cannot be changed on an existing step — the\n UI disables the selector once the step is saved. When asked to migrate\n an existing export off the SuiteBundle (or between versions), do not\n update this field in place; the migration path is recreating or\n cloning the step with the new version. (Real-time listener steps are\n `type: distributed` exports — their version lives at\n `netsuite.distributed.frameworkVersion`, with the same constraint.)\n - |-\n SuiteScript hooks are a separate setting from the step's API version:\n hook code must be written for the same SuiteScript version as the step\n (1.0 and 2.x are not interchangeable), and migrating a hook is its own\n task — flag it whenever migrating a step that carries hooks.\n - |-\n Author new steps with suiteapp2.0 unless the user explicitly requires\n a legacy version — NetSuite no longer updates SuiteScript 1.0 and\n Celigo plans to deprecate the SuiteBundle option.\n criteria:\n type: array\n description: |-\n Filter conditions added on top of the saved search (the UI's \"Additional search\n criteria\"). Each condition pairs a search column with an operator and one or two\n comparison values, and the conditions are combined with AND — every one must match.\n x-celigo-ai-guidance:\n - |-\n A flat list: each entry becomes one `search.createFilter({ name, join, operator,\n values, formula })` call and NetSuite ANDs the resulting filters. There is no\n logical-combinator or grouping property (`join` is NetSuite's related-record\n join, e.g. `customer`, not an AND/OR operator) — a group object such as\n `{\"operator\": \"OR\", \"criteria\": [...]}` is rejected with a 422 (`OR` is not a\n valid enum value for path `operator`).\n - |-\n OR or grouped logic belongs in ONE formula condition: `{\"field\": \"formulanumeric\",\n \"operator\": \"equalto\", \"searchValue\": 1, \"formula\": \"CASE WHEN {tranid} =\n '{{poNumber}}' OR ({shipzip} = '{{zip}}' AND {shipaddress1} = '{{address1}}')\n THEN 1 ELSE 0 END\"}`. The formula is a NetSuite (Oracle SQL) expression over\n `{searchColumn}` references and can carry Handlebars values from the incoming\n record. The alternative is a second lookup step filtered to run only when this\n one returned no record. Do not describe OR between rows as configurable.\n - |-\n ## EXAMPLE\n ```json\n [\n {\"field\": \"type\", \"operator\": \"anyof\", \"searchValue\": [\"PurchOrd\"]},\n {\"field\": \"mainline\", \"operator\": \"is\", \"searchValue\": \"F\"},\n {\"field\": \"tranid\", \"operator\": \"is\", \"searchValue\": \"{{poNumber}}\"}\n ]\n ```\n `field` must be a search-column id for the record type (the SEARCH COLUMNS\n metadata block), not a body-field name — e.g. `shipaddress1`, not `shipaddr1`.\n items:\n type: object\n properties:\n field:\n type: string\n description: |-\n Script-id of the search column to filter on. This is the primary\n field the criterion is evaluated against (e.g. \"trandate\", \"status\").\n join:\n type: string\n description: |-\n Join relationship that applies this filter to a related record's fields instead\n of the base record. Must match a join name NetSuite defines for the record type.\n operator:\n type: string\n enum:\n - after\n - allof\n - any\n - anyof\n - before\n - between\n - contains\n - doesnotcontain\n - doesnotstartwith\n - equalto\n - greaterthan\n - greaterthanorequalto\n - haskeywords\n - is\n - isempty\n - isnot\n - isnotempty\n - lessthan\n - lessthanorequalto\n - noneof\n - notafter\n - notallof\n - notbefore\n - notbetween\n - notequalto\n - notgreaterthan\n - notlessthan\n - notlessthanorequalto\n - noton\n - notonorafter\n - notonorbefore\n - notwithin\n - \"on\"\n - onorafter\n - onorbefore\n - startswith\n - within\n x-enumDescriptions:\n after: Date column is after the value\n allof: Multi-select column contains every value in the array\n any: NetSuite's any operator for list and multi-select columns\n anyof: List or select column equals any value in the array\n before: Date column is before the value\n between: Numeric column lies between searchValue and searchValue2\n contains: Text column contains the substring\n doesnotcontain: Text column does not contain the substring\n doesnotstartwith: Text column does not start with the value\n equalto: Numeric column equals the value\n greaterthan: Numeric column is greater than the value\n greaterthanorequalto: Numeric column is greater than or equal to the value\n haskeywords: Text column contains the keywords\n is: Text or checkbox column equals the value exactly (checkboxes use T or F)\n isempty: Column has no value\n isnot: Text or checkbox column does not equal the value\n isnotempty: Column has a value\n lessthan: Numeric column is less than the value\n lessthanorequalto: Numeric column is less than or equal to the value\n noneof: List or select column equals none of the values in the array\n notafter: Date column is not after the value\n notallof: Multi-select column does not contain every value in the array\n notbefore: Date column is not before the value\n notbetween: Numeric column lies outside searchValue and searchValue2\n notequalto: Numeric column does not equal the value\n notgreaterthan: Numeric column is not greater than the value\n notlessthan: Numeric column is not less than the value\n notlessthanorequalto: Numeric column is not less than or equal to the value\n noton: Date column is not on the value\n notonorafter: Date column is not on or after the value\n notonorbefore: Date column is not on or before the value\n notwithin: Date column is outside the range\n \"on\": Date column is on the value\n onorafter: Date column is on or after the value\n onorbefore: Date column is on or before the value\n startswith: Text column starts with the value\n within: Date column is within the range\n description: |-\n Comparison operator applied between the column and the search value, using\n NetSuite's lowercase search operator names. Which operators a column accepts\n depends on its data type.\n examples: [\"is\", \"anyof\", \"equalto\"]\n x-celigo-ai-guidance:\n - |-\n The server validates the value against this list and rejects anything else\n with a 422 enum error — including logical words such as `OR`; conditions are\n never joined through this property.\n - |-\n Numeric operators (`equalto`, `notequalto`, `greaterthan`, `lessthan`, the\n `...orequalto` / `not...` variants, `between`, `notbetween`) make the platform\n coerce a templated `searchValue` to a number, and NetSuite rejects them on text\n columns with `sss_invalid_srch_operator`. Compare text and checkbox columns\n with `is` / `isnot` and list or select columns with `anyof` / `noneof` (array\n `searchValue`).\n searchValue:\n description: |-\n Value the column is compared against — a string, a number, or an array of\n values for `anyof`-style operators. On lookup steps it may be a Handlebars\n expression rendered from the incoming record (for example `{{poNumber}}`).\n examples: [\"{{poNumber}}\", \"F\", [\"PurchOrd\"]]\n x-celigo-ai-guidance:\n - |-\n The incoming record is the Handlebars root here: write `{{poNumber}}`, not\n `{{record.poNumber}}`. The Flow Builder's step preview rejects the\n `record.`-rooted form with `handlebars_template_parse_error` (\"poNumber\" not\n defined in undefined) while a flow run still renders it — write the bare path\n so the criterion can be step-previewed.\n - |-\n A Handlebars path the incoming record does not carry fails the render with\n `handlebars_template_parse_error` (\"Failed to generate criteria.N.searchValue\n from template\"). A standalone preview of a lookup that uses `{{...}}` here\n fails the same way — outside a flow run there is no record to render from;\n preview with a literal value or through a flow test run instead.\n - |-\n With a numeric `operator` the rendered value is coerced to a number before the\n search runs, so a template must resolve to something numeric.\n searchValue2:\n description: |-\n Second comparison value for the two-value operators `between` and `notbetween`\n (the upper bound). Omit for every other operator.\n x-celigo-ai-guidance:\n - |-\n Same typing as `searchValue`: a literal or a Handlebars expression, coerced to\n a number for the numeric two-value operators.\n formula:\n type: string\n description: |-\n NetSuite formula expression used as the filter condition, for criteria that\n standard field-operator-value comparisons can't express.\n examples: [\"CASE WHEN {quantity} > 0 THEN 1 ELSE 0 END\", \"NVL({amount}, 0)\", \"TO_DATE({trandate})\"]\n x-celigo-ai-guidance:\n - |-\n Used with a formula column in `field` (`formulanumeric` or `formulatext`):\n NetSuite evaluates this expression per record and `operator` / `searchValue`\n compare its result. `formulanumeric` `equalto` `1` with `CASE WHEN \u003ccondition>\n THEN 1 ELSE 0 END` turns any boolean condition — including OR and grouping\n across columns — into one criterion, which is the only way to express OR in\n this array.\n - |-\n Oracle SQL syntax: `{searchColumn}` references, single-quoted string literals,\n `CASE` / `NVL` / `TO_DATE`; a Handlebars value from the incoming record can be\n embedded (`{tranid} = '{{poNumber}}'`). A formula that evaluates false for\n every record returns no rows with no error, so verify the expression with a\n preview that is expected to return records.\n columns:\n type: array\n description: |-\n Search columns the RESTlet returns for each matching record. With a saved search,\n the RESTlet returns the search's own result columns followed by any listed here: a\n list adds columns and never removes one of the search's, so remove a column in the\n saved search itself. An ad-hoc search (no `searchId`) must list its columns here.\n x-celigo-ai-guidance:\n - |-\n Entries are objects keyed by `name` — `[{\"name\": \"internalid\"}, {\"name\":\n \"tranid\"}]`. Plain strings fail with a 422 `ObjectParameterError`; objects keyed by\n `_id` fail with a 422 `BSONError` (`_id` is the server's own subdocument id).\n Omitting both `columns` and `searchId` fails with `[celigo_config_error] searchId\n or columns are required`.\n - |-\n Each `name` must be a search column id for the `recordType` (the SEARCH COLUMNS\n metadata block), not a body-field id — `shipaddress1`, not `shipaddr1`. Set\n `label` to control the property name in the exported record.\n items:\n type: object\n properties:\n _id:\n type: string\n readOnly: true\n description: |-\n Subdocument id the server assigns to the column entry on save. Not a NetSuite\n field id — never set it when authoring.\n name:\n type: string\n description: Column field name as referenced in the search.\n x-celigo-ai-guidance:\n - |-\n REQUIRED. The NetSuite field ID (script ID) of the column to return -- e.g.\n `\"internalid\"`, `\"tranid\"`, `\"entity\"`, `\"trandate\"`, `\"status\"`,\n `\"shipaddress1\"`, `\"item\"`.\n This is the same identifier that appears in the NetSuite UI's \"Field\" picker on saved searches,\n NOT a label or display name (use `label` for that).\n - |-\n ## PLATFORM ENFORCEMENT\n NetSuite's `search.createColumn` rejects entries\n without a `name` argument with\n `[application · stage=apiCall] search.createColumn:\n Missing a required argument: name`. The platform\n validation runs before any column data is read, so\n the export fails fast.\n - |-\n ## NAME MUST BE A SEARCH COLUMN, NOT A BODY FIELD\n NetSuite distinguishes between body fields (the\n record's intrinsic fields like `shipaddr1`) and\n search columns (the fields exposed for filtering /\n sorting in saved searches, like `shipaddress1`).\n `columns[].name` MUST be a search column ID, not\n a body field ID: take it from the chosen\n `recordType`'s SEARCH COLUMNS list.\n - |-\n ## EXAMPLES\n - `\"internalid\"` -- the record's internal id\n - `\"tranid\"` -- transaction number\n - `\"entity\"` -- customer / vendor reference\n - `\"datecreated\"`, `\"lastmodifieddate\"`\n - `\"status\"` -- transaction status\n - `\"shipaddress1\"` -- shipping address line 1\n - `\"item\"` -- line item field\n join:\n type: string\n description: |-\n Join relationship that sources this column from a related record instead of\n the base record. Supports dot notation for nested joins (e.g. \"employee.manager\").\n examples: [\"customer\", \"item\", \"employee.manager\", \"vendor\"]\n summary:\n type: string\n enum: [group, sum, count, avg, min, max]\n x-enumDescriptions:\n group: Group results by this column's value\n sum: Sum of the column across the grouped rows\n count: Count of grouped rows\n avg: Average of the column across the grouped rows\n min: Minimum value across the grouped rows\n max: Maximum value across the grouped rows\n x-lowercase: true\n description: Summary type applied to this column, turning the search into a summary search.\n formula:\n type: string\n description: |-\n NetSuite formula expression that computes this column's value at runtime, such\n as a CASE statement or date formatting, instead of reading a stored field.\n examples: [\"CASE WHEN {status} = 'Open' THEN 1 ELSE 0 END\", \"NVL({amount}, 0) * 0.1\", \"TO_CHAR({trandate}, 'YYYY-MM-DD')\"]\n x-celigo-ai-guidance:\n - |-\n Use with a formula column in `name` (`formulatext`, `formulanumeric`,\n `formuladate`, ...) and give the entry a `label` — the label becomes the\n exported record's property name. Oracle SQL syntax over `{searchColumn}`\n references; string literals use single quotes.\n label:\n type: string\n description: |-\n Display name for the column; it becomes the property name of the column's value\n in each exported record.\n examples: [\"Customer Name\", \"Invoice Date\", \"Total Amount\", \"Status\"]\n sort:\n type: boolean\n description: When true, query results are sorted by this column's values.\n markExportedBatchSize:\n type: integer\n description: |-\n Number of records updated per RESTlet call when marking records as exported in\n NetSuite (live: an integer count, e.g. 100).\n maximum: 100\n minimum: 1\n x-celigo-ui-override: >-\n min 1 / max 100 mirror the integrator-ui NetSuite export form\n (netsuite.restlet.markExportedBatchSize, fallsWithinNumericalRange min 1 / max 100).\n Encoded to mirror the form so builders produce valid configurations.\n hooks:\n type: [object, 'null']\n properties:\n batchSize:\n type: number\n description: |-\n Legacy location of the RESTlet page size; `restlet.batchSize` is the current\n field and takes precedence when both are set.\n preSend:\n type: object\n properties:\n fileInternalId:\n type: string\n description: |-\n Internal ID of the JavaScript file in the NetSuite File Cabinet that defines\n the hook function. Must be readable by the role behind the export's connection.\n examples: [\"12345\", \"6398\"]\n function:\n type: string\n description: |-\n Name of the global function in that file that NetSuite invokes as the pre\n send hook.\n examples: [\"preSendHookScheduledExport\", \"appendCustomerEmail\"]\n configuration:\n type: object\n description: |-\n Free-form JSON stored with the hook reference for the script's own use. Custom\n settings in scope for the export are passed to the function separately as\n `options.settings`.\n description: |-\n SuiteScript hook that runs inside NetSuite after a page of records is collected\n and before it is sent to the Celigo platform.\n x-celigo-ai-guidance:\n - |-\n The function receives one `options` argument — `data` (the page of records),\n `recordType`, `searchId`, `skipGrouping`, `delta`, `once`, `_exportId`,\n `_connectionId`, `_flowId`, `_integrationId`, `settings`, `testMode` — and must\n return `{ data, errors, errorsAndRetryData }` (`errorsAndRetryData` items are\n `{retryData: \u003crecord>, errors: [\u003cerror>]}`). Throwing stops the flow.\n - |-\n Use it to enrich or filter records with SuiteScript searches and record loads\n before they leave NetSuite (no extra API round-trips). The script must be\n written for the step's SuiteScript version (1.0 vs 2.x are not interchangeable);\n the SuiteScript 1.0 real-time variant receives a single record object rather\n than an array.\n description: |-\n SuiteScript hooks for the export. Only `preSend` exists on exports; imports carry\n the pre map, post map, and post submit hooks.\n cLocked:\n type: object\n # TODO(verify): 0 occurrences across all stored exports (restlet/distributed/top-level); semantics unverified\n description: Lock state that prevents modification of the configuration when set.\n description: |-\n Configuration for `restlet` exports. Identifies the RESTlet script that retrieves data\n from NetSuite, along with the saved search, criteria, and columns it executes with.\n x-celigo-ai-guidance:\n - |-\n Two shapes: a saved-search export (`searchId` set; `criteria` optionally narrows it;\n `columns` omitted to return the search's own columns) or an ad-hoc search\n (`searchId: \"-1\"` or omitted, `recordType` + `criteria` + `columns` supplied, and\n `netsuite.skipGrouping: true` — without it the run fails with `[search_not_sorted]`).\n distributed:\n type: object\n x-celigo-ai-guidance:\n - |-\n A real-time listener is an export with top-level `type: distributed` and\n `netsuite.type: distributed`. Saving it creates a user event script deployment on the\n record type that runs after submit, so records arrive only when NetSuite saves one and\n `POST /v1/exports/preview` cannot sample it.\n properties:\n recordType:\n type: string\n description: |-\n NetSuite record type the listener is deployed on, as its lowercase script ID\n (`customer`, `salesorder`, `customrecord_\u003cscriptid>`) — never the display name or a\n custom record's numeric internal ID. Types the platform's record-type catalog marks\n `doesNotSupportRTExport` cannot host a listener and are not offered in the UI.\n examples: [\"customer\", \"invoice\", \"salesorder\", \"itemfulfillment\", \"vendorbill\", \"employee\", \"purchaseorder\", \"creditmemo\"]\n x-celigo-ai-guidance:\n - |-\n Take the value from the `recordTypes` metadata catalog row's `scriptId`, lower-cased:\n custom records and lists carry NetSuite's numeric internal ID in `id` and the script\n ID in `scriptId`, so `id` is wrong for them. Skip rows with `doesNotSupportRTExport: true`.\n executionContext:\n type: array\n description: |-\n NetSuite execution contexts that trigger this distributed export. A record change\n fires the export only when it occurs in one of the listed contexts.\n x-celigo-ai-guidance:\n - |-\n Defaults to `userinterface` and `webstore` — the manual, time-sensitive saves a\n real-time export exists for. Bulk contexts such as `csvimport`, `custommassupdate`\n and `mapreduce` turn every record of a mass update into an outbound call and flood\n the flow with records a scheduled export would batch; add them only when asked.\n default:\n - userinterface\n - webstore\n items:\n type: string\n enum:\n - userinterface\n - webservices\n - csvimport\n - offlineclient\n - portlet\n - scheduled\n - suitelet\n - custommassupdate\n - workflow\n - webstore\n - userevent\n - mapreduce\n - restlet\n - webapplication\n - restwebservices\n x-enumDescriptions:\n userinterface: Triggered by user interactions in the NetSuite UI.\n webservices: Triggered by SOAP web services API calls.\n csvimport: Triggered by CSV import operations.\n offlineclient: Triggered by offline client synchronization.\n portlet: Triggered by portlet interactions on dashboards.\n scheduled: Triggered by scheduled script executions.\n suitelet: Triggered by Suitelet script executions.\n custommassupdate: Triggered by custom mass update operations.\n workflow: Triggered by SuiteFlow workflow actions.\n webstore: Triggered by SuiteCommerce web store transactions.\n userevent: Triggered by user event script executions.\n mapreduce: Triggered by Map/Reduce script operations.\n restlet: Triggered by RESTlet API calls, including the records a NetSuite Distributed import writes.\n webapplication: Triggered by web application interactions.\n restwebservices: Triggered by REST web services API calls.\n disabled:\n type: boolean\n description: |-\n When true, the listener's NetSuite deployment is inactive and record saves no longer\n trigger the export. Maintained by the platform as the flow is disabled and re-enabled;\n re-enabling the flow re-registers the deployment. A listener is created disabled, and\n a flow created already enabled leaves it that way until a flow update sets\n `disabled: false`, which activates it within seconds.\n x-celigo-ai-guidance:\n - |-\n Pause a listener by disabling its flow, not by setting this field. A listener that has\n stopped firing is re-registered by disabling and re-enabling the flow.\n executionType:\n type: array\n description: |-\n Record operations that trigger this distributed export. A record event fires the\n export only when its operation matches one of the listed types.\n x-celigo-ai-guidance:\n - |-\n Defaults to `create`, `edit` and `xedit` (inline edit) — new records plus every way\n an existing record changes. Narrow it for event-driven work: Celigo's own example\n exports a sales order only on `approve` to notify a chat channel.\n - |-\n The UI offers thirteen of these values (`create`, `edit`, `delete`, `xedit`,\n `approve`, `cancel`, `reject`, `pack`, `ship`, `dropship`, `specialorder`,\n `orderitems`, `paybills`); the API accepts all twenty.\n default:\n - create\n - edit\n - xedit\n items:\n type: string\n enum:\n - create\n - edit\n - delete\n - xedit\n - copy\n - view\n - cancel\n - approve\n - reject\n - pack\n - ship\n - markcomplete\n - reassign\n - editforecast\n - dropship\n - specialorder\n - orderitems\n - paybills\n - print\n - email\n x-enumDescriptions:\n create: Triggers when a new record is created.\n edit: Triggers when a record is edited via the UI.\n delete: Triggers when a record is deleted.\n xedit: Triggers when a record is inline-edited without opening it.\n copy: Triggers when a record is duplicated.\n view: Triggers when a record is viewed.\n cancel: Triggers when a transaction is cancelled.\n approve: Triggers when a record is approved.\n reject: Triggers when a record is rejected.\n pack: Triggers when a fulfillment record is packed.\n ship: Triggers when a fulfillment record is shipped.\n markcomplete: Triggers when a record is marked as complete.\n reassign: Triggers when a record is reassigned to another user.\n editforecast: Triggers when a forecast is edited.\n dropship: Triggers when a drop ship operation is performed.\n specialorder: Triggers when a special order operation is performed.\n orderitems: Triggers when an order items action is performed.\n paybills: Triggers when a pay bills action is performed.\n print: Triggers when a print action is performed on a record.\n email: Triggers when an email action is performed on a record.\n qualifier:\n # No object form: the previous 'type: object' matched no stored document.\n type: [array, string, 'null']\n description: |-\n Field-specific qualification criteria the SuiteApp evaluates against the record before\n exporting it; saves that do not match are discarded. Stored as a filter-expression array\n of `[field, operator, value]` rules joined by lowercase `\"and\"` / `\"or\"`, as a legacy\n expression string, or null when no criteria are set.\n examples:\n - [\"custrecord_ai_entitlements_status_code\", \"!=\", \"200\"]\n - [[\"val:entity\", \"=\", \"1397\"], \"and\", [\"shipstatus\", \"=\", \"C\"]]\n x-celigo-ai-guidance:\n - |-\n Rules combine strictly pairwise: three conditions are stored as\n `[A, \"and\", [B, \"and\", C]]`, never as a flat `[A, \"and\", B, \"and\", C]`.\n - |-\n The left operand is a body field ID; sublist (line) fields cannot be used. A select\n field takes a prefix: `text:\u003cfield>` compares the option's label (the UI's \"(Name)\"\n row) and `val:\u003cfield>` the internal ID (the \"(InternalId)\" row — `entity.internalid`\n is stored as `val:entity`); multiselects use `texts:` / `vals:`.\n - |-\n Operator tokens are `=`, `!=`, `?` (like), `contains`, `doesNotContain`, `>` and\n `<` (the HTML entities, literally); emptiness is the rule `[field, \"empty\", true]`\n and non-emptiness `[field, \"empty\", false]`. Checkbox values are `\"T\"` / `\"F\"` on\n SuiteBundle and SuiteApp 1.0 steps and boolean `true` / `false` on `suiteapp2.0` steps.\n - |-\n The UI's value dropdown for a select field is NetSuite's own option list for that\n field on that record type, capped at the first 1,000 options alphabetically — a value\n missing from the dropdown can still exist in NetSuite. Read the list through the\n record-type metadata endpoint with `includeSelectOptions=true` before judging it.\n skipExportFieldId:\n type: string\n description: |-\n Script ID of a checkbox body field on the record type. A save with that box checked is\n skipped by the listener, which then clears the box itself — used to keep a mass update\n from fanning out and to stop a write-back import from re-triggering the listener.\n x-celigo-ai-guidance:\n - |-\n For a bidirectional sync, have the write-back import set this checkbox to true in the\n same save: the listener skips that save and unchecks the box, breaking the loop.\n hooks:\n type: object\n description: |-\n SuiteScript hook that runs inside NetSuite before the record leaves it.\n properties:\n preSend:\n type: object\n description: |-\n Pre-send hook: a SuiteScript function in the NetSuite file cabinet invoked before\n records leave NetSuite, with the full SuiteScript API available — for example to run\n dynamic searches against the record being exported.\n x-celigo-ai-guidance:\n - |-\n The hook's SuiteScript version must match the step's `frameworkVersion` (1.0 and\n 2.x are not interchangeable); migrating a step to another version means rewriting\n the hook as a separate task.\n properties:\n fileInternalId:\n type: string\n description: Internal ID of the file cabinet file that holds the hook's SuiteScript.\n examples: [\"12345\", \"987654\"]\n function:\n type: string\n description: Name of the function in that file to invoke for each export.\n examples: [\"preSendCustomer\", \"enrichOrderDetails\"]\n configuration:\n type: object\n description: Settings passed to the hook function.\n sublists:\n type: [array, 'null']\n description: |-\n IDs of the NetSuite sublists to include with each exported record (for example `item`\n for transaction line items). Records are exported with body fields only when no\n sublists are selected. Legacy documents may store null (equivalent to no sublists).\n items:\n type: string\n examples: [[\"item\"]]\n x-celigo-ai-guidance:\n - |-\n Each selected sublist is an extra query inside the NetSuite save, so select only the\n sublists the flow maps. Qualification criteria cannot reference sublist fields\n regardless of this setting.\n referencedFields:\n type: object\n description: |-\n Field references recorded on the listener by the platform. Not exposed in the listener\n form; leave as stored.\n relatedLists:\n type: object\n description: |-\n Related record lists recorded on the listener by the platform. Not exposed in the\n listener form; leave as stored.\n forceReload:\n type: boolean\n description: |-\n When true, the listener reloads the record from the database after the save before\n exporting it, so fields that are populated only once the save completes — such as\n transaction line IDs on new lines — are included. The reload slows every save of that\n record type.\n x-celigo-ai-guidance:\n - |-\n Leave false unless a field is missing from the exported payload that NetSuite fills\n in after submit (line IDs are the documented case); there is no published list of such\n fields.\n ioEnvironment:\n type: string\n description: |-\n integrator.io environment the listener's NetSuite deployment reports to. Written by the\n platform when the listener registers with NetSuite; live-observed values include\n `production` and `staging`.\n examples: [\"production\", \"staging\"]\n x-celigo-ai-guidance:\n - |-\n Never author or change it. When updating from a GET that returned a value, carry it\n through unchanged; when the GET returned none, send none.\n ioDomain:\n type: string\n description: |-\n integrator.io domain the listener's NetSuite deployment calls back to — the account's\n own regional domain, stamped by the platform on every save and echoed on create and\n update responses; not returned by GET.\n examples: [\"integrator.io\"]\n x-celigo-ai-guidance:\n - |-\n Never author it: the platform re-stamps the account's domain on save. Live-verified on\n one North America account, where a PUT that omitted it came back with that account's\n domain (`integrator.io`); other regions carry their own host.\n lastSyncedDate:\n type: string\n format: date-time\n description: |-\n When the listener's deployment last synchronized with NetSuite. Written by the platform;\n not set until the first successful synchronization.\n x-celigo-ai-guidance:\n - |-\n Never author or change it. When updating from a GET that returned a value, carry it\n through unchanged; when the GET returned none, send none.\n settings:\n type: object\n description: |-\n Additional key-value settings for the distributed export. Not exposed in the listener\n form.\n useSS2Framework:\n type: boolean\n description: |-\n Derived by the platform from `frameworkVersion` — true when it is `suiteapp2.0`, absent\n otherwise — and echoed on create and update responses; not returned by GET. A value\n sent on create or update is replaced by the derived one.\n x-celigo-ai-guidance:\n - |-\n Never set this; `frameworkVersion` is authoritative. Live-verified: a PUT sending the\n opposite value came back with the value derived from `frameworkVersion`.\n frameworkVersion:\n type: string\n enum:\n - suitebundle\n - suiteapp1.0\n - suiteapp2.0\n x-enumDescriptions:\n suitebundle: Legacy SuiteBundle framework; slated for deprecation — NetSuite no longer updates SuiteScript 1.0.\n suiteapp1.0: SuiteApp 1.0 framework using SuiteScript 1.0.\n suiteapp2.0: SuiteApp 2.0 framework using SuiteScript 2.0, the modern default.\n description: |-\n SuiteApp framework version used by the distributed export.\n Fixed when the step is created — the Advanced selector is disabled on\n existing steps, and migrating an existing listener to a different\n version means recreating or cloning it.\n x-celigo-ai-guidance:\n - |-\n The NetSuite API version cannot be changed on an existing step — the\n UI disables the selector once the step is saved. When asked to\n migrate an existing real-time listener off the SuiteBundle (or\n between versions), do not update this field in place; the migration\n path is recreating or cloning the step with the new version.\n - |-\n SuiteScript hooks are a separate setting from the step's API\n version: hook code must be written for the same SuiteScript version\n as the step (1.0 and 2.x are not interchangeable), and migrating a\n hook is its own task — flag it whenever migrating a step that\n carries hooks.\n - |-\n Author new steps with suiteapp2.0 unless the user explicitly\n requires a legacy version — NetSuite no longer updates SuiteScript\n 1.0 and Celigo plans to deprecate the SuiteBundle option.\n description: |-\n Configuration for `distributed` exports, which use the NetSuite SuiteApp to fire\n real-time, event-driven exports when records change. Define the record type to listen\n to and the execution contexts and operations that trigger it.\n getList:\n # Array, not object; stored occurrences are empty — the item shape below documents\n # the getList record-reference semantics but is not observed populated.\n type: array\n description: |-\n Configuration for `getList` exports — a list of record references, each retrieving one\n NetSuite record by its identifier. The export returns the fetched records in list order.\n x-celigo-ai-guidance:\n - |-\n Send getList as a bare array of record references, exactly as GET returns it. Never\n wrap it in an object (e.g. `\"getList\": {\"type\": [...]}`): the adaptor requires an\n array, and an export saved with the wrapper shape fails at run time with\n \"requires getList prop\".\n - |-\n Reference standard records with `type` plus `internalId` or `externalId`. Reference\n custom records with `typeId` instead of `type`; a `typeId` starting with\n \"customtransaction\" targets a custom transaction type. When both identifiers are\n present, `internalId` wins and `externalId` is ignored.\n items:\n type: object\n properties:\n type:\n type: string\n description: |-\n Standard NetSuite record type to retrieve (e.g. \"customer\", \"salesOrder\").\n Used when `typeId` is absent. Case-sensitive.\n typeId:\n type: string\n description: |-\n Script ID of a custom record type (\"customrecord_...\") or custom transaction\n type (\"customtransaction_...\"). Takes precedence over `type` when present.\n Case-sensitive.\n internalId:\n type: string\n description: |-\n Internal ID of the NetSuite record to retrieve. Assigned by NetSuite at record\n creation and unique within the record type. Takes precedence over `externalId`.\n externalId:\n type: string\n description: |-\n Identifier assigned by an external system, used to reference and synchronize the\n record across systems. Distinct from the NetSuite-assigned internal ID.\n searchPreferences:\n type: object\n properties:\n bodyFieldsOnly:\n type: boolean\n description: |-\n When true, search results include only the record's body fields and exclude fields\n from joined or related records. Reduces payload size when related data isn't needed.\n x-celigo-ai-guidance:\n - |-\n bodyFieldsOnly indicates whether the search results should include only the body\n fields of the records,\n excluding any joined or related record fields.\n This setting controls the scope of data returned by the search operation,\n allowing for more focused and efficient retrieval when only the main record's\n fields are necessary.\n - |-\n ## FIELD BEHAVIOR:\n - When set to true, search results will include only the fields that belong directly to the main record (body fields), excluding any fields from joined or related records.\n - When set to false or omitted, search results may include fields from both the main record and any joined or related records specified in the search.\n - Directly affects the volume and detail of data returned, potentially reducing payload size and improving performance.\n - Influences how the search engine processes and compiles the result set, limiting it to primary record data when enabled.\n - |-\n ## IMPLEMENTATION GUIDANCE:\n - Use this property to optimize search performance and reduce data transfer when only the main record's fields are required.\n - Set to true to minimize payload size, which is beneficial for large datasets or bandwidth-sensitive environments.\n - Verify that your search criteria and downstream processing do not require any joined or related record fields before enabling this option.\n - If joined fields are necessary for your application logic, keep this property false or unset to ensure complete data retrieval.\n - Consider this setting in conjunction with other search preferences like pageSize and returnSearchColumns for optimal results.\n - |-\n ## EXAMPLES:\n - `bodyFieldsOnly: true` — returns only the main record's body fields in search results, excluding any joined record fields.\n - `bodyFieldsOnly: false` — returns both body fields and fields from joined or related records as specified in the search.\n - Omitted `bodyFieldsOnly` property — defaults to false behavior, including joined fields if requested.\n - |-\n ## IMPORTANT NOTES:\n - Enabling bodyFieldsOnly may omit critical related data if your search logic depends on joined fields, potentially impacting application functionality.\n - This setting is particularly useful for improving performance and reducing data size in scenarios where joined data is unnecessary.\n - Not all record types or search operations may support this preference; verify compatibility with your specific use case.\n - Changes to this setting can affect the structure and completeness of search results, so test thoroughly when modifying.\n - |-\n ## DEPENDENCY CHAIN:\n - This property is part of the `searchPreferences` object within the NetSuite API request.\n - It influences the fields returned by the search operation, affecting both data scope and payload size.\n - May\n pageSize:\n type: number\n description: Number of search results NetSuite returns per page.\n x-celigo-ai-guidance:\n - |-\n pageSize specifies the number of search results to be returned per page in a\n paginated search response.\n This property controls the size of each page of results when performing searches,\n enabling efficient handling and retrieval of large datasets by dividing them\n into manageable chunks.\n By adjusting pageSize,\n clients can balance between the volume of data received per request and the\n performance implications of processing large result sets.\n - |-\n ## FIELD BEHAVIOR:\n - Determines the maximum number of records returned in a single page of search results.\n - Directly influences the pagination mechanism by setting how many items appear on each page.\n - Helps optimize network and client performance by limiting the amount of data transferred and processed per response.\n - Affects the total number of pages available, calculated based on the total number of search results divided by pageSize.\n - When pageSize is changed between requests, it may affect the consistency of pagination navigation.\n - |-\n ## IMPLEMENTATION GUIDANCE:\n - Assign pageSize a positive integer value that balances response payload size and system performance.\n - Ensure the value respects any minimum and maximum limits imposed by the API or backend system.\n - Maintain consistent pageSize values across paginated requests to provide predictable and stable navigation through result pages.\n - Consider client device capabilities, network bandwidth, and expected user interaction patterns when selecting pageSize.\n - Implement validation to prevent invalid or out-of-range values that could cause errors or degraded performance.\n - When dealing with very large datasets, consider smaller pageSize values to reduce memory consumption and improve responsiveness.\n - |-\n ## EXAMPLES:\n - pageSize: 25 — returns 25 search results per page, suitable for standard list views.\n - pageSize: 100 — returns 100 search results per page, useful for bulk data processing or export scenarios.\n - pageSize: 10 — returns 10 search results per page, ideal for quick previews or limited bandwidth environments.\n - pageSize: 50 — a moderate setting balancing data volume and performance for typical use cases.\n - |-\n ## IMPORTANT NOTES:\n - Excessively large pageSize values can increase response times, memory usage, and may lead to timeouts or throttling.\n - Very small pageSize values can cause a high number of API calls, increasing overall latency and server load.\n - The API may enforce maximum allowable pageSize limits; requests exceeding these limits may result in errors or automatic truncation.\n - Changing pageSize mid-pagination can disrupt user experience by altering the number of pages and item offsets.\n - Some APIs may have default pageSize values if none is specified; explicitly setting page\n returnSearchColumns:\n type: boolean\n description: |-\n When true, search results include the column data defined in the search. When\n false, column data is omitted and results contain only minimal record information.\n x-celigo-ai-guidance:\n - |-\n returnSearchColumns:\n Specifies whether the search operation should return the columns (fields)\n defined in the search results,\n providing detailed data for each record matching the search criteria.\n - |-\n ## FIELD BEHAVIOR:\n - Determines if the search response includes the columns specified in the search definition, such as field values and metadata.\n - When set to true, the search results will contain detailed column data for each record, enabling comprehensive data retrieval.\n - When set to false, the search results will omit column data, potentially returning only record identifiers or minimal information.\n - Directly influences the amount of data returned, impacting response payload size and processing time.\n - Affects how client applications can utilize the search results, depending on the presence or absence of column data.\n - |-\n ## IMPLEMENTATION GUIDANCE:\n - Set to true when detailed search result data is required for processing, reporting, or display purposes.\n - Set to false to optimize performance and reduce bandwidth usage when only record IDs or minimal data are needed.\n - Use in conjunction with other search preference settings (e.g., `pageSize`, `returnSearchRows`) to fine-tune search responses.\n - Ensure client applications are designed to handle both scenarios—presence or absence of column data—to avoid errors or incomplete processing.\n - Consider the trade-off between data completeness and performance when configuring this property.\n - |-\n ## EXAMPLES:\n - `returnSearchColumns: true` — The search results will include all defined columns for each record, such as names, dates, and custom fields.\n - `returnSearchColumns: false` — The search results will exclude column data, returning only basic record information like internal IDs.\n - |-\n ## IMPORTANT NOTES:\n - Enabling returnSearchColumns may significantly increase response size and processing time, especially for searches returning many records or columns.\n - Some search operations or API endpoints may require columns to be returned to function correctly or to provide meaningful results.\n - Disabling this option can improve performance but limits the detail available in search results, which may affect downstream processing or user interfaces.\n - Changes to this setting can impact caching, pagination, and sorting behaviors depending on the search implementation.\n - |-\n ## DEPENDENCY CHAIN:\n - Related to other `searchPreferences` properties such as `pageSize` (controls number of records per page) and `returnSearchRows` (controls whether search rows are returned).\n - Works in tandem with search definition settings that specify which columns are included in the search.\n - May affect or be affected by API-level configurations or limitations on data retrieval and response formatting.\n description: Preferences that control how NetSuite executes searches and shapes the results it returns.\n x-celigo-ai-guidance:\n - |-\n searchPreferences:\n Preferences that control the behavior and parameters of search operations within\n the NetSuite environment,\n enabling customization of how search queries are executed and how results are\n returned to optimize relevance,\n performance, and user experience.\n - |-\n ## FIELD BEHAVIOR:\n - Defines the execution parameters for search queries, including pagination, sorting, filtering, and result formatting.\n - Controls the scope, depth, and granularity of data retrieved during search operations.\n - Influences the performance, accuracy, and relevance of search results based on configured preferences.\n - Can be adjusted dynamically to tailor search behavior to specific user roles, contexts, or application requirements.\n - May include settings such as page size limits, sorting criteria, case sensitivity, and filter application.\n - |-\n ## IMPLEMENTATION GUIDANCE:\n - Utilize this property to fine-tune search operations to meet specific user or application needs, improving efficiency and relevance.\n - Validate all preference values against supported NetSuite search parameters to prevent errors or unexpected behavior.\n - Establish sensible default preferences to ensure consistent and predictable search results when explicit preferences are not provided.\n - Allow dynamic updates to preferences to adapt to changing contexts, such as different user roles or data volumes.\n - Ensure that preference configurations comply with user permissions and role-based access controls to maintain security and data integrity.\n - |-\n ## EXAMPLES:\n - Setting a page size of 50 to limit the number of records returned per search query for better performance.\n - Enabling case-insensitive search filters to broaden result matching.\n - Specifying sorting order by transaction date in descending order to show the most recent records first.\n - Applying filters to restrict search results to a particular customer segment or date range.\n - Configuring search to exclude inactive records to streamline results.\n - |-\n ## IMPORTANT NOTES:\n - Misconfiguration of searchPreferences can lead to incomplete, irrelevant, or inefficient search results, negatively impacting user experience.\n - Certain preferences may be restricted or overridden based on user roles, permissions, or API version constraints.\n - Changes to searchPreferences can affect system performance; excessive page sizes or complex filters may increase load times.\n - Always verify compatibility of preference settings with the specific NetSuite API version and environment in use.\n - Consider the impact of preferences on downstream processes that consume search results.\n - |-\n ## DEPENDENCY CHAIN:\n - Depends on the overall search operation configuration and the specific search type being performed.\n - Interacts with user authentication and authorization settings to enforce access controls on search results.\n - Influences and is influenced by data retrieval mechanisms and indexing strategies within NetSuite.\n file:\n type: object\n description: |-\n Configuration for file exports, which retrieve files from the NetSuite file cabinet\n and parse their contents (CSV, XML, JSON) into records. Leave the export's top-level\n `type` unset when using this object; for raw transfers without parsing, use\n `netsuite.blob` with the export's `type` set to `blob` instead.\n x-celigo-ai-guidance:\n - |-\n Configuration for retrieving files from NetSuite file cabinet and PARSING them into records.\n Use this for structured file exports (CSV, XML,\n JSON) where the file content should be parsed into data records.\n - |-\n ## CRITICAL: When to use file vs blob\n - Use `netsuite.file` WITH export `type: null/undefined` for file exports WITH parsing (CSV, XML, JSON)\n - Use `netsuite.blob` WITH export `type: \"blob\"` for raw binary transfers WITHOUT parsing\n\n When you want file content to be parsed into individual records, use this `file` configuration and leave the export's `type` field as null or undefined (standard export). Do NOT set `type: \"blob\"` when using this configuration.\n properties:\n folderInternalId:\n type: string\n description: |-\n Internal ID of the NetSuite file cabinet folder to export files from. Accepts a\n handlebars expression (e.g. `{{record.folderId}}`) when the folder must be\n selected dynamically from record data. The ID is stable across folder renames and\n moves, but may differ between non-production and production accounts.\n examples: [\"12345\", \"67890\", \"{{record.folderId}}\"]\n x-celigo-ai-guidance:\n - |-\n Specify the internal ID for the NetSuite File Cabinet folder from which you want\n to export your files.\n If the folder internal ID is required to be dynamic based on the data you are integrating,\n you can specify the JSON path to the field in your data containing the folder\n internal ID values instead — with DOUBLE braces (e.g. {{record.folderId}}).\n Default to a static numeric ID; dynamic substitution is the exception.\n - |-\n ## FIELD BEHAVIOR:\n - Identifies the specific folder in NetSuite's file cabinet to export files from\n - Must be a valid internal ID that exists in the NetSuite environment\n - Supports dynamic values using handlebars notation for data-driven folder selection\n - The internal ID is distinct from folder names or paths; it's a stable numeric identifier\n - |-\n ## IMPLEMENTATION GUIDANCE:\n - Obtain the folderInternalId via NetSuite's UI (File Cabinet > folder properties) or API\n - For static exports, use the numeric internal ID directly (e.g., \"12345\")\n - For dynamic exports, use handlebars syntax to reference a field in your data\n - Verify folder permissions - the integration user must have access to the folder\n - The folderInternalId is distinct from folder names or folder paths; it is a stable numeric identifier used internally by NetSuite.\n - Using an incorrect or non-existent folderInternalId will result in errors or unintended file placement.\n - Folder permissions and user roles can impact the ability to perform operations even with a valid folderInternalId.\n - Folder hierarchy changes do not affect the folderInternalId, ensuring persistent reference integrity.\n - |-\n ## DEPENDENCY CHAIN:\n - Depends on the existence of the folder within the NetSuite file cabinet.\n - Requires appropriate user permissions to access or modify the folder.\n - Often used in conjunction with file identifiers and other file metadata fields.\n - May be linked to folder creation or folder search operations to retrieve valid IDs.\n - |-\n ## EXAMPLES:\n - \"12345\" - Static folder internal ID (the usual form)\n - \"67890\" - Another valid folder internal ID\n - \"{{record.folderId}}\" - Dynamic folder ID from integration data (double braces)\n - |-\n ## IMPORTANT NOTES:\n - Using an incorrect or non-existent folderInternalId will result in errors\n - Folder hierarchy changes do not affect the folderInternalId\n - Internal IDs may differ between sandbox and production environments\n backupFolderInternalId:\n type: string\n description: |-\n Internal ID of the NetSuite file cabinet folder where backup files are stored.\n IDs are account-specific and do not transfer between non-production and production\n environments.\n examples: [\"12345\", \"67890\", \"112233\"]\n x-celigo-ai-guidance:\n - |-\n backupFolderInternalId is the internal identifier of the backup folder within\n the NetSuite file cabinet where backup files are stored.\n This ID uniquely identifies the folder location used for saving backup files programmatically,\n ensuring that backup operations target the correct directory within the NetSuite environment.\n - |-\n ## FIELD BEHAVIOR:\n - Represents a unique internal ID assigned by NetSuite to a specific folder in the file cabinet.\n - Directs backup operations to the designated folder location for storing backup files.\n - Must correspond to an existing and accessible folder within the NetSuite file cabinet.\n - Typically handled as an integer value in API requests and responses.\n - Immutable for a given folder; changing the folder requires updating this ID accordingly.\n - |-\n ## IMPLEMENTATION GUIDANCE:\n - Verify that the folder with this internal ID exists before initiating backup operations.\n - Confirm that the folder has the necessary permissions to allow writing and managing backup files.\n - Use NetSuite SuiteScript APIs or REST API calls to retrieve and validate folder internal IDs dynamically.\n - Avoid hardcoding the internal ID; instead, use configuration files, environment variables, or administrative settings to maintain flexibility across environments.\n - Implement error handling to manage cases where the folder ID is invalid, missing, or inaccessible.\n - Consider environment-specific IDs for sandbox versus production to prevent misdirected backups.\n - |-\n ## EXAMPLES:\n - 12345\n - 67890\n - 112233\n - |-\n ## IMPORTANT NOTES:\n - The internal ID is unique per NetSuite account and environment; IDs do not transfer between sandbox and production.\n - Deleting or renaming the folder associated with this ID will disrupt backup processes until updated.\n - Proper access rights and permissions are mandatory to write backup files to the specified folder.\n - Changes to folder structure or permissions should be coordinated with backup scheduling to avoid failures.\n - This property is critical for ensuring backup data integrity and recoverability within NetSuite.\n - |-\n ## DEPENDENCY CHAIN:\n - Depends on the existence and accessibility of the folder in the NetSuite file cabinet.\n - Interacts with backup scheduling, file naming conventions, and storage management properties.\n - May be linked with authentication and authorization mechanisms controlling file cabinet access.\n - Relies on NetSuite API capabilities to manage and reference file cabinet folders.\n - |-\n ## TECHNICAL DETAILS:\n - Data type: Integer (typically a 32-bit integer).\n - Represents the internal NetSuite folder ID, not the folder name or path.\n - Used in API payloads to specify backup destination folder.\n - Must be retrieved or confirmed via NetSuite SuiteScript\n fileNameStartsWith:\n type: string\n description: |-\n Optional prefix filter. Only files whose names start with this string are\n exported from the configured folder.\n examples: [\"ORDER_\", \"INV_\"]\n fileNameEndsWith:\n type: string\n description: |-\n Optional suffix filter. Only files whose names end with this string (e.g. a\n \".csv\" extension) are exported from the configured folder.\n examples: [\".csv\", \".xml\"]\n required: []\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":101939,"content_sha256":"a200a100183e6dd3415e5b929fa7282b99f1dd87b7ab5ae0fff2368c3ed888fc"},{"filename":"references/schemas/once.yml","content":"Once:\n type: object\n description: |-\n Configures flag-based exports that skip records an earlier run already processed. Required\n when the export's type is \"once\"; omit for other export types. Each run retrieves records\n where the tracking boolean field is false, then sets that field to true after successful\n processing so later runs skip them; if a run fails, the flags are left unchanged and the\n records are retried automatically on the next run.\n x-celigo-ai-guidance:\n - |-\n This object is REQUIRED when the export's type field is set to \"once\" and should\n not be included for other export types.\n Once exports use a boolean/checkbox field in the source system to track which\n records have been processed,\n creating a reliable idempotent data extraction pattern.\n - |-\n For optimal AI agent implementation, consider these guidelines:\n\n 1. System behavior during execution:\n - First, the export retrieves all records where the specified boolean field is false\n - After successfully processing these records, the system automatically sets the field to true\n - On subsequent runs, previously processed records are excluded\n - |-\n 2. Prerequisites in the source system:\n - The source must have a boolean/checkbox field that can be used as a processing flag\n - Your connection must have write access to update this field after export\n - The field should be indexed for optimal performance\n - |-\n 3. Common implementation scenarios:\n - One-time migrations where data should not be duplicated\n - Processing queues where records are marked as \"processed\"\n - Compliance scenarios requiring audit trails of exported records\n - |-\n 4. Error handling behavior:\n - If the export fails, the boolean fields remain unchanged\n - Records will be retried on the next run\n - No manual intervention is required for recovery\n - |-\n This object is REQUIRED when the export's type field is set to \"once\" and should\n not be included for other export types.\n Once exports use a boolean/checkbox field in the source system to track which\n records have been processed,\n creating a reliable idempotent data extraction pattern.\n properties:\n booleanField:\n type: string\n x-celigo-ui-override: >-\n Required by the export form when the export type is \"once\" (once.booleanField is\n required:true, visibleWhen type=once). Encoded to mirror the form so builders produce\n connectable configurations.\n description: |-\n API field name of the boolean/checkbox in the source system that tracks processed\n records: the export selects only records where this field is false, then sets it to true\n in batches after each successfully processed page. The field must be writeable by the\n export's connection. Ensure no other process updates the same field — use a separate\n flag per export process — or records may be skipped unexpectedly.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field identifies which boolean field in the source system controls the export filtering:\n\n - REQUIRED when the export's type field is set to \"once\"\n - Must reference a valid boolean/checkbox field in the source system\n - Must be writeable by the connection's authentication credentials\n - The system performs two operations with this field:\n 1. Filters to only include records where this field is false\n 2. Updates processed records by setting this field to true\n - |-\n ## IMPLEMENTATION PATTERNS\n\n ### Using dedicated tracking fields\n ```\n \"booleanField\": \"isExported\"\n ```\n - Create a dedicated field specifically for integration tracking\n - Provides clear separation between business and integration logic\n - Most maintainable approach for long-term operations\n\n ### Using existing status fields\n ```\n \"booleanField\": \"isProcessed\"\n ```\n - Leverage existing status fields if they align with your integration needs\n - Ensure the field's meaning is compatible with your integration logic\n - Consider potential conflicts with other processes using the same field\n\n ### Targeted export tracking\n ```\n \"booleanField\": \"exported_to_netsuite\"\n ```\n - For systems synchronizing to multiple destinations\n - Create separate tracking fields for each destination system\n - Enables independent control of different export processes\n - |-\n ## TECHNICAL CONSIDERATIONS\n\n - Field updates happen in batches after each successful page of records is processed\n - The field update uses the same connection as the export operation\n - For optimal performance, the boolean field should be indexed in the source database\n - Boolean values of 0/1, true/false, and yes/no are all properly interpreted\n\n IMPORTANT: Ensure the field is not being updated by other processes, as this could\n cause records to be skipped unexpectedly. If multiple processes need to track exports,\n use separate boolean fields for each process.\n examples: [\"isExported\", \"hasBeenSynced\", \"processedFlag\", \"exported_to_netsuite\"]\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":5513,"content_sha256":"d0e160807e0b53059f27494d8df981d9923fcf5363b8d0cd14063830e4b021cc"},{"filename":"references/schemas/preview-request.yml","content":"PreviewRequest:\n type: object\n description: Request body for previewing export data\n allOf:\n - $ref: './response.yml#/Export'\n - type: object\n properties:\n postData:\n type: object\n description: |-\n Delta window for this preview. Without it, a document preview renders\n `{{lastExportDateTime}}` and `{{currentExportDateTime}}` empty, so the preview doesn't\n reflect the export's delta window: an HTTP request gets empty strings, and an RDBMS\n query gets empty, unquoted text, so a delta query fails to parse (observed on Snowflake;\n `delta.startDate` doesn't set them either). With it, an HTTP export inserts each value\n exactly as given, and an RDBMS query receives each wrapped in single quotes\n (`'2026-01-15T00:00:00.000Z'`, observed on Snowflake); other adaptors weren't checked.\n On an export without `delta.dateFormat`, a real run renders both as ISO 8601 UTC with\n milliseconds, so pass them in that form to preview the request such a run would send.\n properties:\n currentExportDateTime:\n type: string\n description: Value `{{currentExportDateTime}}` renders as in this preview (the window's end).\n example: \"2026-01-16T00:00:00.000Z\"\n lastExportDateTime:\n type: string\n description: Value `{{lastExportDateTime}}` renders as in this preview (the window's start).\n example: \"2026-01-15T00:00:00.000Z\"\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":1554,"content_sha256":"1dbfab623b112b20c382619313fcdbed63da8dcfdf1659e04349617c39f32093"},{"filename":"references/schemas/preview-response.yml","content":"PreviewResponse:\n type: object\n description: Response body for export data preview\n properties:\n data:\n type: array\n description: The data exported from source app\n items:\n type: object\n description: Individual data record from the source app\n dataURIs:\n type: array\n description: URIs to the data in the source app\n items:\n type: string\n description: URI to a specific data record in the source app\n stages:\n type: array\n description: Processing stages information\n items:\n type: object\n description: Information about a processing stage\n properties:\n name:\n type: string\n description: Name of the processing stage\n errors:\n type: array\n description: Errors encountered during this stage\n items:\n type: object\n description: Error information\n data:\n type: array\n description: Data after being processed by this stage\n items:\n type: object\n description: Individual data record after being processed by this stage","content_type":"application/yaml; charset=utf-8","language":"yaml","size":1185,"content_sha256":"fb203371226c35db623c78a23a589d50b4d5d3c33518fd11d79185f3f2f9ab63"},{"filename":"references/schemas/rdbms.yml","content":"RDBMS:\n type: object\n description: |-\n Configuration object for Relational Database Management System (RDBMS) data integration exports.\n\n This object defines how data is read from a relational database and must not be included\n for other connection types. For query-type exports (standard, delta, once) it is REQUIRED\n when the _connectionId field references an RDBMS database connection and holds the SQL\n query. For real-time stream listeners (`type: \"stream\"`) it holds the CDC watch list\n (`tables`) instead — and a PostgreSQL listener scoped by a publication\n (`cdc.publicationName`) may omit this object entirely.\n\n **Rdbms export capabilities**\n - Execute custom SQL SELECT statements\n - Support for joins, aggregations, and functions\n - Flexible data retrieval from any accessible tables or views\n - Compatible with all major database systems\n\n **Critical:** WHAT BELONGS IN THIS OBJECT\n - `query` - The SQL SELECT statement - required for every query-type export (standard, delta, once)\n - `once` - **REQUIRED** when the export's Object Type is `\"once\"` (set _include_once: true)\n - `tables` - the CDC listener watch list - real-time stream listeners (`type: \"stream\"`) only\n - **DO NOT** put `delta` inside this object - delta is handled via the query\n - **DO NOT** author `query` on a stream listener - real-time CDC exports have no SQL query\n\n **Delta exports (type: \"delta\")**\n For delta/incremental exports, do NOT populate a `delta` object inside `rdbms`.\n Instead, use `{{lastExportDateTime}}` or `{{currentExportDateTime}}` directly in the query, and\n set the export's top-level `delta.startDate`: without it the first run's window is empty.\n ```json\n {\n \"type\": \"delta\",\n \"delta\": { \"startDate\": \"2026-01-01T00:00:00.000Z\" },\n \"rdbms\": {\n \"query\": \"SELECT * FROM customers WHERE updatedAt > {{lastExportDateTime}}\"\n }\n }\n ```\n\n **Once exports (type: \"once\")**\n For once exports (mark records as processed), populate `rdbms.once.query`:\n ```json\n {\n \"type\": \"once\",\n \"rdbms\": {\n \"query\": \"SELECT * FROM customers WHERE exported = false\",\n \"once\": {\n \"query\": \"UPDATE customers SET exported = true WHERE id = {{record.id}}\"\n }\n }\n }\n ```\n\n **Standard exports (type: null or not specified)**\n Just provide the query:\n ```json\n {\n \"rdbms\": {\n \"query\": \"SELECT * FROM customers WHERE status = 'ACTIVE'\"\n }\n }\n ```\n\n **Real-time CDC listener exports (type: \"stream\")**\n SQL Server and PostgreSQL connections support change-data-capture listeners that stream row\n changes continuously (the `cdc` object holds the listener settings). A stream export has NO\n `query` — it watches `tables`:\n ```json\n {\n \"type\": \"stream\",\n \"rdbms\": { \"tables\": \"public.orders,public.customers\" },\n \"cdc\": { \"properties\": [{ \"name\": \"snapshot.mode\", \"value\": \"no_data\" }] }\n }\n ```\n A PostgreSQL listener can instead scope tables through a publication\n (`cdc.publicationName`), in which case `tables` stays unset and the `rdbms` object may be\n omitted entirely.\n properties:\n tables:\n type: string\n description: |-\n Comma-separated list of tables the CDC listener watches for changes. Used only by real-time\n listener exports (`type` is `stream`); standard SQL exports use `query` instead. SQL Server\n listeners use fully-qualified three-part names (`database.schema.table`); PostgreSQL\n listeners use two-part names (`schema.table`).\n x-celigo-ai-guidance:\n - |-\n For a PostgreSQL listener, author EITHER this field OR `cdc.publicationName`, never both —\n when a publication is set it is the sole table filter and the listener form clears this\n field (the API stores both if sent, so nothing catches the conflict at save time).\n examples: [\"UKGODS.dbo.UKG_MetadataStaging\", \"dbo.WorkOrderT,dbo.OrderLineT\", \"public.orders,public.customers\"]\n query:\n type: string\n x-celigo-agent:\n - sql\n x-celigo-ui-override: >-\n Required by the standard RDBMS export form (rdbms.query is required:true) for query-type\n exports. Deliberately NOT encoded as a schema-level required[] entry: stream listener\n exports (type: \"stream\") save without a query — they watch `tables` or a PostgreSQL\n publication — and a publication-based PostgreSQL listener persists with no rdbms object at\n all, so any required[] here would reject valid listener configs. Do not re-add it.\n description: |-\n SQL SELECT statement executed to retrieve data, from simple table selections to joins,\n aggregations, and parameterized queries with Handlebars expressions. Query-type exports\n only — real-time stream listeners (`type` is `stream`) have no SQL query. For delta exports,\n reference {{lastExportDateTime}}/{{currentExportDateTime}} directly in the query rather\n than configuring a separate delta object.\n\n Double braces render a string as a quoted SQL literal with its apostrophes escaped, and\n triple braces insert a value raw. Observed in Snowflake queries:\n\n - String helper output (`dateFormat`, `jsonSerialize`) and both delta variables are\n quoted the same way; a number or boolean renders bare, whether it is a JSON value or a\n numeric helper's result (`{{add 2 3}}` renders `5`).\n - In a run, {{lastExportDateTime}} and {{currentExportDateTime}} arrive as quoted ISO 8601\n UTC literals with milliseconds (`'2026-01-15T00:00:00.000Z'`), or in `delta.dateFormat`\n when it is set: add no quotes of your own, and don't parse them with a format mask\n written for another format.\n - A preview of an unsaved export document renders both as empty, unquoted text, so a\n delta query fails to compile there unless the document carries both in `postData`.\n - The quoting leaves backslashes, which Snowflake reads as escape sequences, so a\n double-brace value holding a backslash arrives changed. Pass such a value in a\n dollar-quoted constant with triple braces (`${{{record.field}}}$`), which Snowflake\n reads as is, but only when it can't contain `$`: a `$` ends the constant, and the rest\n is read as SQL.\n x-celigo-ai-guidance:\n - |-\n This field contains the SQL SELECT statement that will be executed to fetch data\n from the database.\n The query can range from simple table selections to complex joins and aggregations.\n - |-\n Examples:\n - Basic: `SELECT id, name, email FROM customers WHERE status = 'ACTIVE'`\n - Join: `SELECT o.id, c.name, o.amount FROM orders o JOIN customers c ON o.customer_id = c.id`\n - Aggregate: `SELECT category, COUNT(*) as count FROM orders GROUP BY category`\n - Parameterized: `SELECT * FROM orders WHERE customer_id = {{record.customer_id}}`\n - |-\n This field contains the SQL SELECT statement that will be executed to fetch data\n from the database.\n The query can range from simple table selections to complex joins and aggregations.\n examples: [\"SELECT * FROM customers WHERE updatedAt > {{lastExportDateTime}}\", \"SELECT * FROM orders WHERE exported = false\", \"SELECT * FROM customers WHERE region = 'Northeast'\", \"SELECT o.id, c.name, o.total FROM orders o JOIN customers c ON o.customer_id = c.id WHERE o.status = 'SHIPPED'\", \"SELECT department, AVG(salary) as avg_salary FROM employees GROUP BY department\", \"SELECT * FROM customer_orders WHERE customer_id = {{record.id}}\"]\n once:\n type: object\n description: |\n **CRITICAL: REQUIRED when the export's Object Type is \"once\".**\n\n If Object Type is \"once\", you MUST set _include_once to true (or include this object).\n\n This object has ONLY ONE property: \"query\" (a SQL UPDATE string).\n DO NOT create any other properties like \"update\", \"table\", \"set\", \"where\", etc.\n\n CORRECT format:\n ```json\n {\"query\": \"UPDATE customers SET exported=true WHERE id={{record.id}}\"}\n ```\n\n WRONG format (DO NOT DO THIS):\n ```json\n {\"update\": {\"table\": \"customers\", \"set\": {...}}}\n ```\n properties:\n query:\n type: string\n x-celigo-agent:\n - sql\n description: |\n **REQUIRED** - A SQL UPDATE statement string to mark records as processed.\n\n This is a plain SQL UPDATE query string, NOT a structured object.\n\n The query runs AFTER each record is successfully exported, setting a flag\n to indicate the record has been processed.\n\n Format: \"UPDATE \u003ctable> SET \u003ccolumn>=\u003cvalue> WHERE \u003cid_column>={{record.\u003cid_field>}}\"\n\n Example: \"UPDATE customers SET exported=true WHERE id={{record.id}}\"\n\n The {{record.id}} placeholder is replaced with the actual record ID from each exported row.\n examples: [\"UPDATE orders SET exported=true WHERE id={{record.id}}\", \"UPDATE customers SET exported=true WHERE customer_id={{record.customer_id}}\", \"UPDATE inventory SET is_exported=1 WHERE sku={{record.sku}}\"]\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":9364,"content_sha256":"9870afe72920458ec93ecad6d142656d4c725b21ea3a9200505c5277e93f080e"},{"filename":"references/schemas/record-extraction.yml","content":"# Record extraction is one capability stored in per-adaptor locations:\n# `mongodb.pathToRecords`, `dynamodb.pathToRecords`, the existing\n# `http.response.resourcePath`, and the existing `file.json`/`file.xml`\n# `resourcePath` (paired with `file.includeParentData`). These shared schemas\n# hold the canonical field documentation once; those locations $ref them\n# instead of re-wording the docs per adaptor.\n# The shipped `webhook.pathToRecords`/`webhook.includeParentData` deliberately\n# keep their own inline docs until the webhook service adopts the shared\n# extraction util — until then, webhook JSONPath support differs\n# (no union support, and evaluation errors fall back to the whole payload).\nPathToRecords:\n type: string\n maxLength: 250\n description: |-\n Path to the array of records inside each document or item fetched from the source; each array element is emitted as an individual record, which keeps records within the 5 MB page-size limit even when the source document exceeds it. Accepts dot notation (e.g. `data.orders`) or JSONPath for values starting with ` configuring-exports — Skillopedia , including unions (`$['orders','invoices'][*]`) and filter expressions (`$.items[?(@.qty>3)]`); a path that resolves to nothing emits zero records for that document or item. When omitted, each document or item is emitted as one record, exactly as before — HTTP and file-based exports configure this same capability through their existing `http.response.resourcePath` and `file.json`/`file.xml` `resourcePath` fields.\n examples:\n - lineItems\n - order.lineItems\n - $.data.orders\n - \"$['orders','invoices'][*]\"\n - \"$.items[?(@.qty>3)]\"\nIncludeParentData:\n type: boolean\n description: |-\n When true and a path to records is configured on the export (`pathToRecords` for MongoDB and DynamoDB, `http.response.resourcePath` for HTTP, `file.json`/`file.xml` `resourcePath` for file-based exports), the data outside that path is attached to each extracted record as a `_PARENT` object, keeping the surrounding envelope available to downstream steps. When omitted or false, records are emitted without `_PARENT`, exactly as before. Constraints: on HTTP exports this option cannot be combined with `useXpathParser: true` (rejected at save with `EXPORT_INCLUDE_PARENT_DATA_XPATH_UNSUPPORTED`), and XML responses must resolve to the default converted-JSON parse mode — a run whose XML parser resolves to DOM/XPath mode (for example, because a connection-level field uses an advanced XPath expression) fails at start with the same error rather than silently omitting `_PARENT`. On file-based exports, it requires `file.type` of `json` with `file.json.resourcePath` configured — XML file envelope capture is not yet supported (the streaming XML parser emits no envelope event; XML support arrives with a parsers-package enhancement) — and any other file configuration is rejected at save with `EXPORT_INCLUDE_PARENT_DATA_UNSUPPORTED_FILE_TYPE`.\n examples:\n - true\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":2967,"content_sha256":"40cf8533020c051163ef280be97ecd0dafa343873f52de197770ef8a68bc1d48"},{"filename":"references/schemas/request.yml","content":"Request:\n type: object\n description: |-\n Fields that can be sent when creating or updating an export. Set the adaptor-specific\n configuration object matching `adaptorType`, and the mode-specific object matching `type`.\n `SimpleExport` (data-loader) is the exception — it needs no adaptor config object. An Event\n Streams listener (`TopicExport`) carries `topic` and `type: stream` and no `_connectionId`.\n required:\n - name\n allOf:\n - $ref: './base.yml#/ExportBase'\n # The config object matching `adaptorType` must be supplied. Nested if/else-if (no allOf),\n # connection-request style. API-enforced: POST/PUT without the object fails with 422\n # \"\u003ckey> subschema not defined\".\n # Request-only (not in base.yml): legacy persisted records violate these pairings, so the\n # response schema must not assert them.\n # No SimpleExport branch — POST 201 with no config object (data-loader exports carry `file`\n # at most; there is no `simple` object).\n if: { properties: { adaptorType: { const: HTTPExport } }, required: [adaptorType] }\n then:\n required: [http]\n # Cloud file providers add a location rule keyed on `assistant` (the export-level field the\n # provider forms set). Form-derived: the API saves either shape without a directory.\n # The generic file-mode rule (`http.type: file` outside an assistant form needs the\n # export-level `file` parse settings) sits last so the provider branches take precedence;\n # it used to live inside http.yml, where `required: [file]` pointed at a non-existent\n # `http.file`.\n if: { properties: { assistant: { const: azuredatalakestoragegen2 } }, required: [assistant] }\n then:\n x-celigo-ui-override: >-\n Required by the Azure Data Lake Storage Gen2 export form\n (fieldDefinitions/resources/exports/azuredatalakestoragegen2.js: 'file.directory.storageRootId_adls'\n is required:true and preSave persists it as file.directory.name; the directory path in\n http.relativeURI is required:false). Encoded to mirror the form so builders produce\n connectable configurations.\n required: [file]\n properties:\n file:\n required: [directory]\n properties:\n directory:\n required: [name]\n else:\n if: { properties: { assistant: { const: googlecloudstorage } }, required: [assistant] }\n then:\n x-celigo-ui-override: >-\n Required by the Google Cloud Storage export form\n (fieldDefinitions/resources/exports/googlecloudstorage.js: bucket 'file.directory.storageRootId_gcs'\n is required:true and prefix 'file.directory.id' is required:true; preSave forces\n file.directory.pathMode to directoryId). Encoded to mirror the form so builders\n produce connectable configurations.\n required: [file]\n properties:\n file:\n required: [directory]\n properties:\n directory:\n required: [storageRootId, id]\n else:\n if:\n required: [http]\n properties:\n http:\n required: [type]\n properties:\n type:\n const: file\n not:\n required: [formType]\n properties:\n formType:\n const: assistant\n then:\n required: [file]\n else:\n if: { properties: { adaptorType: { const: FTPExport } }, required: [adaptorType] }\n then: { required: [ftp] }\n else:\n # AS2 listeners require `file` (parse settings), not `as2` — POST with only `file`\n # returns 201, and typical AS2 exports have no `as2` object at all.\n # Exception: webhook-mode AS2 listeners (`type: webhook`) receive raw payloads and\n # carry no file parse settings, so they are not required to carry `file`.\n if: { properties: { adaptorType: { const: AS2Export } }, required: [adaptorType] }\n then:\n if:\n not: { properties: { type: { const: webhook } }, required: [type] }\n then: { required: [file] }\n else:\n if: { properties: { adaptorType: { const: S3Export } }, required: [adaptorType] }\n then: { required: [s3] }\n else:\n if: { properties: { adaptorType: { const: NetSuiteExport } }, required: [adaptorType] }\n then: { required: [netsuite] }\n else:\n # NetSuite REST APIs reuse the `http` object on a netsuite connection; `nsDomainType`\n # picks the host. Both are 422-required (\"Path `nsDomainType` is required.\").\n if: { properties: { adaptorType: { const: NetSuiteHTTPExport } }, required: [adaptorType] }\n then: { required: [http, nsDomainType] }\n else:\n if: { properties: { adaptorType: { const: SalesforceExport } }, required: [adaptorType] }\n then: { required: [salesforce] }\n else:\n if: { properties: { adaptorType: { const: JDBCExport } }, required: [adaptorType] }\n then: { required: [jdbc] }\n else:\n if: { properties: { adaptorType: { const: RDBMSExport } }, required: [adaptorType] }\n then: { required: [rdbms] }\n else:\n # A MongoDB find export is 422-required to carry `mongodb.method` and\n # `mongodb.collection`; a change-stream listener (`type: stream`) needs\n # neither — it may omit `mongodb` entirely — but must carry `cdc`.\n if: { properties: { adaptorType: { const: MongodbExport } }, required: [adaptorType] }\n then:\n if: { properties: { type: { const: stream } }, required: [type] }\n then: { required: [cdc] }\n else:\n required: [mongodb]\n properties:\n mongodb:\n required: [method, collection]\n else:\n if: { properties: { adaptorType: { const: DynamodbExport } }, required: [adaptorType] }\n then: { required: [dynamodb] }\n else:\n if: { properties: { adaptorType: { const: WrapperExport } }, required: [adaptorType] }\n then: { required: [wrapper] }\n else:\n # webhook.verify is 422-required when `type` is `webhook` (the only real\n # WebhookExport shape).\n if: { properties: { adaptorType: { const: WebhookExport } }, required: [adaptorType] }\n then: { required: [webhook] }\n else:\n if: { properties: { adaptorType: { const: FileSystemExport } }, required: [adaptorType] }\n then: { required: [filesystem] }\n else:\n # The legacy REST adaptor was folded into `http`: POST and PUT of a\n # RESTExport without `http` fail 422; the deprecated `rest` object\n # is not required.\n if: { properties: { adaptorType: { const: RESTExport } }, required: [adaptorType] }\n then: { required: [http] }\n else:\n # Event Streams listener: the API saves a TopicExport with an\n # empty `topic` (defaults only); the topic binding is the form's rule.\n if: { properties: { adaptorType: { const: TopicExport } }, required: [adaptorType] }\n then:\n x-celigo-ui-override: >-\n Required by the Event Streams listener form\n (fieldDefinitions/resources/exports/export.js: 'topic._topicId' is\n required:true). Encoded to mirror the form so builders produce\n listeners that are bound to a topic.\n required: [topic]\n properties:\n topic:\n required: [_topicId]\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":8401,"content_sha256":"8759c95b08c1ce3f76b087f37c3ffa67a3c1389880710ef4f3dfd55c96718c4a"},{"filename":"references/schemas/response.yml","content":"Export:\n type: object\n required:\n - _id\n - name\n - adaptorType\n - createdAt\n - lastModified\n description: Export object as returned by the API.\n allOf:\n - $ref: './base.yml#/ExportBase'\n - $ref: '../common-schemas/resource-response.yml#/ResourceResponse'\n - $ref: '../common-schemas/ia-resource-response.yml#/IAResourceResponse'\n - type: object\n properties:\n aiDescription:\n $ref: '../common-schemas/ai-description.yml#/AIDescription'\n apim:\n $ref: '../common-schemas/apim.yml#/APIM'\n apiIdentifier:\n type: string\n readOnly: true\n description: API identifier assigned to this export.\n asynchronous:\n type: boolean\n readOnly: true\n description: Server-managed execution-mode flag set on creation.\n __linkedLookupCacheIds:\n type: array\n readOnly: true\n description: Lookup caches linked to this export, managed by the platform.\n items:\n type: string\n format: objectId\n sandbox:\n type: boolean\n deprecated: true\n readOnly: true\n description: When true, this export belongs to a sandbox account.\n rest:\n type: object\n deprecated: true\n readOnly: true\n additionalProperties: true\n description: |-\n Legacy REST adaptor configuration, still returned on exports created before\n the REST-to-HTTP migration (`adaptorType: RESTExport`). Mirrors the shape of\n `http` (method, relativeURI, paging and response-path settings). On write the\n platform maintains the equivalent `http` configuration — configure new\n exports through `http` instead.\n _sourceId:\n type: string\n format: objectId\n readOnly: true\n description: Reference to the source resource this export was created from.\n _templateId:\n type: string\n format: objectId\n readOnly: true\n x-celigo-refModel: templates\n description: Template this export was created from.\n draft:\n type: boolean\n readOnly: true\n description: When true, this export is in draft state and has not been confirmed.\n draftExpiresAt:\n type: string\n format: date-time\n readOnly: true\n description: Timestamp when the draft version of this export expires.\n debugUntil:\n type: string\n format: date-time\n readOnly: true\n description: |-\n Turns on step debug capture until this time: while it lies in the future, every\n request this export sends is recorded in the step's debug log\n (`GET /v1/{flows|apis|tools}/{containerId}/{_id}/requests`). Read-only on PUT —\n written only by `PATCH /v1/exports/{_id}` on `/debugUntil`, at most 3600 seconds\n ahead (`422` `update_not_allowed` beyond that); `remove` stops capture.\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":3056,"content_sha256":"32da6e3ba9b62ced17baf6ea290c92c1a8f0a67ddc6eeeee54f70ed286e766a1"},{"filename":"references/schemas/s3.yml","content":"S3:\n type: object\n description: |-\n Defines which files to retrieve from an Amazon S3 bucket. Required when the _connectionId\n field references an AWS S3 connection; must not be included for other connection types.\n region and bucket locate the source, keyStartsWith/keyEndsWith filter objects by key, and\n backupBucket with keyPrefix controls where files are moved after retrieval.\n x-celigo-ai-guidance:\n - Configuration object for Amazon S3 (Simple Storage Service) data integration exports.\n - |-\n The S3 export object has the following requirements:\n\n - Required fields: region, bucket\n - Optional fields: keyStartsWith, keyEndsWith, backupBucket, keyPrefix\n - |-\n ## PURPOSE\n\n This configuration specifies:\n - Which S3 bucket to retrieve files from\n - How to filter files by key patterns\n - Where to move files after retrieval (optional)\n required:\n - region\n - bucket\n properties:\n region:\n type: string\n default: us-east-1\n x-celigo-ui-override: >-\n Required by the S3 export form (fieldDefinitions/resources/exports/s3.js: s3.region is\n required:true, default us-east-1). Encoded to mirror the form so builders produce\n connectable configurations.\n description: |-\n AWS region where the bucket is located. Case-insensitive; the value is normalized to\n lowercase.\n x-celigo-ai-guidance:\n - |-\n - REQUIRED for all S3 exports\n - Must be a valid AWS region identifier (e.g., us-east-1, eu-west-1)\n - Case-insensitive (will be normalized to lowercase)\n examples: [\"us-east-1\", \"eu-west-1\", \"ap-southeast-2\", \"us-west-2\", \"ap-northeast-1\"]\n bucket:\n type: string\n x-celigo-ui-override: >-\n Required by the S3 export form (fieldDefinitions/resources/exports/s3.js: s3.bucket is\n required:true). Encoded to mirror the form so builders produce connectable configurations.\n description: |-\n S3 bucket to retrieve files from. The connection's AWS credentials must have\n s3:ListBucket and s3:GetObject permissions on it.\n x-celigo-ai-guidance:\n - |-\n - REQUIRED for all S3 exports\n - Must be a valid existing S3 bucket name\n - Globally unique across all AWS accounts\n - AWS credentials must have s3:ListBucket and s3:GetObject permissions\n examples: [\"my-company-exports\", \"customer-data-integration\", \"celigo-etl-files\", \"financial-data-prod\"]\n keyStartsWith:\n type: string\n description: |-\n Only retrieves objects whose keys start with this value (case-sensitive), effectively\n selecting a folder in S3's flat key structure. When combined with keyEndsWith, objects\n must match both.\n x-celigo-ai-guidance:\n - Optional prefix filter for S3 object keys.\n - |-\n - Filters files based on the beginning of their keys\n - Functions as a directory path in S3's flat storage structure\n - Case-sensitive (S3 keys are case-sensitive)\n - Examples:\n - `\"exports/\"` - retrieves files in the exports \"directory\"\n - `\"customer/orders/2023/\"` - retrieves files in this nested path\n - `\"invoice_\"` - retrieves files starting with \"invoice_\"\n - When used with keyEndsWith, files must match both criteria.\n examples: [\"exports/\", \"customer/orders/\", \"data/2023/\", \"invoice_\", \"logs/application/\"]\n keyEndsWith:\n type: string\n description: |-\n Only retrieves objects whose keys end with this value (case-sensitive), commonly a file\n extension. When combined with keyStartsWith, objects must match both.\n x-celigo-ai-guidance:\n - Optional suffix filter for S3 object keys.\n - |-\n - Commonly used to filter by file extension\n - Case-sensitive (S3 keys are case-sensitive)\n - Examples:\n - `\".csv\"` - retrieves only CSV files\n - `\"_FINAL.xml\"` - retrieves only XML files with _FINAL suffix\n - `\"_READY\"` - retrieves files with status indicator\n - When used with keyStartsWith, files must match both criteria.\n examples: [\".csv\", \".json\", \".xml\", \"_READY\", \"_FINAL.txt\", \"-daily.parquet\"]\n backupBucket:\n type: string\n description: |-\n Bucket in the same region where files are moved after successful export, giving you an\n independent backup; if omitted, files are simply deleted from the source bucket after\n successful export unless `file.skipDelete` is true (Celigo also keeps its own copy of\n processed files for a set period). The connection's AWS credentials must have\n s3:PutObject permission on this bucket.\n x-celigo-ai-guidance:\n - Optional destination bucket where files are moved before deletion.\n - |-\n IMPORTANT:\n Celigo automatically deletes files from the source bucket after successful export,\n unless `file.skipDelete` is true. Moving them to a backup bucket also takes them out of\n the source bucket, so when another system still reads them there, set\n `file.skipDelete: true` instead.\n The backup bucket is for users who want to maintain their own independent backup\n of exported files.\n Celigo also maintains its own backup of processed files for a set period of time.\n - |-\n IMPORTANT:\n Celigo automatically deletes files from the source bucket after successful export,\n unless `file.skipDelete` is true. Moving them to a backup bucket also takes them out of\n the source bucket, so when another system still reads them there, set\n `file.skipDelete: true` instead.\n The backup bucket is for users who want to maintain their own independent backup\n of exported files.\n Celigo also maintains its own backup of processed files for a set period of time.\n examples: [\"my-company-processed\", \"archive-bucket\", \"integration-backup\", \"source-bucket-archive\"]\n keyPrefix:\n type: string\n description: |-\n Prefix prepended to each file's name when it is moved to the backup bucket; accepts\n static text or handlebars templates. The original directory structure is not preserved —\n only the filename is appended to this prefix. Ignored unless backupBucket is set.\n x-celigo-ai-guidance:\n - Optional prefix to prepend to keys when moving to backup bucket.\n - |-\n - Used only when backupBucket is specified\n - Prepended to the original filename when moved to backup\n - Can contain static text or handlebars templates\n - Examples:\n - `\"processed/\"` - places files under a processed folder\n - `\"archive/{{timestamp 'YYYY-MM-DD'}}/\"` - organizes by date\n examples: [\"processed/\", \"archive/{{timestamp 'YYYY-MM'}}/\", \"history/\", \"completed/{{timestamp 'YYYY/MM/DD'}}/\"]\n x-celigo-agent:\n - handlebars\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":7056,"content_sha256":"8d0ba3d6125e909bdf0c48cc79c3429599dfef0ac24e6ef0885d0a05f82c1e20"},{"filename":"references/schemas/salesforce.yml","content":"Salesforce:\n type: object\n description: |-\n Configuration for Salesforce exports. Required when the export's `_connectionId` references a Salesforce connection; omit for other connection types. The `type` field selects the extraction mode: `soql` runs batch queries and requires the `soql` object, while `distributed` listens for real-time events and requires the `distributed` object. File retrieval (when the export's `type` is `blob`) requires `sObjectType` and `id` instead.\n x-celigo-ai-guidance:\n - |-\n This object is REQUIRED when the _connectionId field references a Salesforce\n connection and must not be included for other connection types.\n It defines how data is extracted from Salesforce,\n either through queries or real-time events.\n - |-\n Set `type` on every Salesforce export you create (`soql` or `distributed`).\n Never read its absence on an existing export as a defect: a listener saved\n from the integrator.io form carries no `salesforce.type` and the platform\n fires it without the field, and file/byId retrievals (`id` + `sObjectType`,\n no `soql` object) carry `type: soql` without a `soql` object. A listener\n that never creates a job is diagnosed from `distributed` (below), not from\n this field.\n - |-\n ## SALESFORCE EXPORT MODES\n\n Salesforce exports offer two fundamentally different operating modes:\n\n 1. **SOQL Query-based Exports** (type=\"soql\")\n - Scheduled or on-demand batch processing\n - Uses SOQL queries to retrieve data\n - Supports both REST and Bulk API\n - Can be configured as lookups (isLookup=true)\n - Requires the \"soql\" object with query configuration\n\n 2. **Real-time Event Listeners** (type=\"distributed\")\n - Responds to Salesforce record creates, updates, and deletes as they happen\n - Fires through an Apex trigger on the object in the connected org (see `distributed`)\n - Always appears as a \"Listener\" in the flow builder UI\n - Requires the \"distributed\" object with event configuration\n\n 3. **File/Blob Exports** (when export.type=\"blob\")\n - Retrieves files stored in Salesforce\n - Requires sObjectType and id fields\n - Supports Attachments, ContentVersion, and Document objects\n - |-\n ## IMPLEMENTATION REQUIREMENTS\n\n The salesforce object has conditional requirements based on the selected type:\n\n - For SOQL exports (type=\"soql\"):\n Required fields: type, soql.query\n Optional fields: api, includeDeletedRecords, bulk (when api=\"bulk\")\n\n - For Distributed exports (real-time listeners):\n Set type=\"distributed\" with the distributed object and sObjectType when creating one.\n A listener saved from the integrator.io form carries the distributed object and\n sObjectType with no type, and the platform fires it — a valid listener, not a missing field.\n Optional fields: distributed.referencedFields, distributed.qualifier\n\n - For Blob exports (when export.type=\"blob\"):\n Required fields: sObjectType, id\n properties:\n type:\n type: string\n description: |-\n Selects the extraction mode and determines which configuration object is required. Use `soql` for scheduled or on-demand batch queries and lookups — it requires the `soql` object and works with both the `rest` and `bulk` APIs and the standard, delta, test, and once export types. Use `distributed` for real-time event handling — it requires the `distributed` object, always appears as a Listener in Flow Builder, ignores the `api` field, and supports only the standard export type.\n x-celigo-ai-guidance:\n - Defines the fundamental data extraction method for Salesforce exports.\n - |-\n ## FIELD BEHAVIOR\n\n This field determines the core operating mode of the Salesforce export:\n\n - Set it on every Salesforce export you create; a listener saved from the\n integrator.io form carries none and the platform fires it without the\n field, so an absent `type` beside a `distributed` object is the normal\n shape of a form-saved listener, not a missing setting\n - Controls which additional configuration objects must be provided\n - Affects how the export appears and functions in the flow builder UI\n - Cannot be changed after creation without significant reconfiguration\n - |-\n ## AVAILABLE TYPES\n\n ### SOQL Query-based Export\n ```\n \"type\": \"soql\"\n ```\n\n - **Behavior**: Executes SOQL queries against Salesforce on schedule or demand\n - **UI Appearance**: \"Export\" or \"Lookup\" based on isLookup value\n - **Required Config**: Must provide the \"soql\" object with a valid query\n - **Use Cases**: Batch data extraction, delta synchronization, data migration\n - **Dependencies**:\n - Compatible with both \"rest\" and \"bulk\" API options\n - Works with standard, delta, test, and once export types\n\n ### Real-time Event Listener\n ```\n \"type\": \"distributed\"\n ```\n\n - **Behavior**: Listens for real-time Salesforce events (create/update/delete)\n - **UI Appearance**: Always appears as a \"Listener\" in the flow builder\n - **Required Config**: Must provide the \"distributed\" object with event configuration\n - **Use Cases**: Real-time synchronization, event-driven integration\n - **Dependencies**:\n - Only uses REST API (api field is ignored)\n - Depends on an Apex trigger on the object in the connected org: Integration Apps create it at installation; for a custom flow the customer deploys it from the export page's Required trigger code (see `distributed`)\n - Only compatible with standard export type (ignores delta/test/once)\n - |-\n ## IMPLEMENTATION CONSIDERATIONS\n\n The type selection creates a fundamental difference in how data flows:\n\n - \"soql\" operates on a pull model where the integration initiates data retrieval\n - \"distributed\" operates on a push model where Salesforce events trigger the integration\n\n IMPORTANT: Choose \"soql\" for batch processing and lookups; choose \"distributed\" for\n real-time event handling. This decision affects all other configuration aspects.\n enum: [\"soql\", \"distributed\"]\n x-enumDescriptions:\n soql: Executes SOQL queries to retrieve data from Salesforce on a schedule or on demand.\n distributed: Listens for real-time Salesforce events (create, update, delete) as they occur.\n examples: [\"soql\", \"distributed\"]\n sObjectType:\n type: string\n description: |-\n API name of the Salesforce object the export operates on; the object must exist in the connected org and be accessible to the integration user. Required for distributed exports and for blob exports — blob exports accept only the file storage objects (`Attachment`, `ContentVersion`, `Document`) paired with the `id` field. Optional for SOQL exports, where the object can be inferred from the query.\n x-celigo-ai-guidance:\n - Specifies the Salesforce object type for the export operation.\n - |-\n ## FIELD BEHAVIOR\n\n This field determines which Salesforce object is being exported:\n\n - **REQUIRED** when the parent export's type is \"distributed\"\n - **REQUIRED** when the parent export's type is \"blob\"\n - Optional for \"soql\" exports (can be inferred from the SOQL query)\n - Must be a valid Salesforce object API name\n - |-\n ## USE CASES BY EXPORT TYPE\n\n ### Distributed Exports\n ```\n \"sObjectType\": \"Account\"\n \"sObjectType\": \"Contact\"\n \"sObjectType\": \"Opportunity\"\n \"sObjectType\": \"Custom_Object__c\"\n ```\n\n - **Purpose**: Specifies the primary object type being exported from Salesforce\n - **Valid Values**: Any standard or custom Salesforce object (Account, Contact, Opportunity, Lead, Case, Custom_Object__c, etc.)\n - **API Access**: Uses the specified object's metadata and SOQL/REST APIs\n - **Use Cases**: Real-time distributed processing of Salesforce records\n - **Requirements**: Object must exist in the connected Salesforce org\n\n ### Blob/File Exports\n ```\n \"sObjectType\": \"Attachment\"\n \"sObjectType\": \"ContentVersion\"\n \"sObjectType\": \"Document\"\n ```\n\n - **Purpose**: Specifies which Salesforce file storage object contains the file data\n - **Valid Values**: File storage objects only (Attachment, ContentVersion, Document)\n - **API Access**: Uses file-specific APIs for data retrieval\n - **Use Cases**: Extracting files and binary data from Salesforce\n - **Requirements**: Must be used with the \"id\" field to specify the file record\n\n ### SOQL Exports\n ```\n \"sObjectType\": \"Account\" // Optional - can be inferred from query\n ```\n\n - **Purpose**: Optional hint about the primary object in the SOQL query\n - **Valid Values**: Any Salesforce object referenced in the query\n - **Use Cases**: Query optimization and metadata context\n - |-\n ## IMPLEMENTATION NOTES\n\n For distributed exports, this field is essential for:\n - Setting up proper event listeners and triggers\n - Configuring field metadata and validation\n - Enabling related object processing\n - Determining appropriate API endpoints\n\n For blob exports, this field works with the \"id\" field to retrieve specific file records.\n\n IMPORTANT: The object specified must exist in the target Salesforce org and be accessible\n to the integration user account.\n examples: [\"Account\", \"Contact\", \"Opportunity\", \"Lead\", \"Case\", \"Custom_Object__c\", \"Attachment\", \"ContentVersion\", \"Document\"]\n id:\n type: string\n description: |-\n Salesforce record ID of the file to retrieve; required for blob exports and not used for `soql` or `distributed` exports. Accepts a static ID or a handlebars expression such as `{{record.Attachment_Id__c}}` to resolve the file at runtime — dynamic IDs require the export to run as a lookup (`isLookup: true`). The ID must belong to the object named in `sObjectType`.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field identifies the specific file record in Salesforce:\n\n - REQUIRED when the parent export's type is \"blob\"\n - Must be a valid Salesforce ID or a handlebars expression\n - Used in conjunction with sObjectType to retrieve the file\n - Not used for regular data exports (type=\"soql\" or \"distributed\")\n - |-\n ## IMPLEMENTATION PATTERNS\n\n ### Static File ID\n ```\n \"id\": \"00P5f00000ZQcTZEA1\"\n ```\n\n - References a specific, fixed file in Salesforce\n - Useful for retrieving standard documents or templates\n - Always retrieves the same file on each execution\n - Simple to configure but lacks flexibility\n\n ### Dynamic File ID (Handlebars)\n ```\n \"id\": \"{{record.Attachment_Id__c}}\"\n ```\n\n - References a file ID from input data using handlebars\n - Requires the export to be used as a lookup (isLookup=true)\n - Dynamically determines which file to retrieve at runtime\n - Allows for contextual file retrieval based on previous steps\n - |-\n ## TECHNICAL DETAILS\n\n - For ContentVersion objects, this should be the ContentVersion ID\n - For Attachment objects, this should be the Attachment ID\n - For Document objects, this should be the Document ID\n\n IMPORTANT: Salesforce IDs are 15 or 18 characters, case-sensitive for 15-character\n versions, and case-insensitive for 18-character versions. When using handlebars,\n ensure the referenced field contains a valid Salesforce ID.\n examples: [\"00P5f00000ZQcTZEA1\", \"{{record.Attachment_ID__c}}\", \"{{record.ContentVersion_ID__c}}\"]\n x-celigo-agent:\n - handlebars\n includeDeletedRecords:\n type: boolean\n description: |-\n When true, SOQL exports use Salesforce's `queryAll()` instead of `query()`, including Recycle Bin records deleted within the past 15 days; each record's `IsDeleted` field identifies deletions. Useful for synchronizing deletes to target systems or maintaining a complete audit trail. Ignored for distributed and blob exports.\n x-celigo-ai-guidance:\n - Controls whether the export retrieves records from the Salesforce Recycle Bin.\n - |-\n ## FIELD BEHAVIOR\n\n This field enables access to recently deleted records:\n\n - OPTIONAL: Defaults to false if not specified\n - Only relevant for SOQL exports (type=\"soql\")\n - Ignored for distributed exports and blob exports\n - Changes the underlying API method used for queries\n - |-\n ## IMPLEMENTATION IMPACT\n\n When set to true:\n - Salesforce's queryAll() API method is used instead of query()\n - Records in the Recycle Bin (deleted within the past 15 days) are included\n - Each record contains an \"IsDeleted\" field to identify deleted status\n - API usage may be higher as queryAll() counts differently against limits\n - |-\n ## USE CASES\n\n This field is particularly useful for:\n\n - Synchronizing deletion operations to target systems\n - Building data recovery/rollback mechanisms\n - Maintaining a complete audit trail including deleted records\n - Implementing soft-delete patterns across integrated systems\n - |-\n ## TECHNICAL CONSIDERATIONS\n\n - Records in the Recycle Bin are only available for up to 15 days\n - Hard-deleted records (emptied from Recycle Bin) are not accessible\n - The IsDeleted field should be checked to identify deleted records\n - May increase response size and processing time slightly\n\n IMPORTANT: This feature only works with SOQL exports (type=\"soql\") and is ignored\n for distributed exports (type=\"distributed\") since those operate on events rather\n than queries.\n default: false\n examples: [true, false]\n api:\n type: string\n description: |-\n Salesforce API used to run SOQL queries; ignored for distributed and blob exports. Use `rest` (the default when omitted) for smaller datasets (under 10,000 records) and for lookup exports — the Bulk API is not compatible with `isLookup: true`. Use `bulk` for large data volumes and higher throughput, optionally tuned through the `bulk` object.\n x-celigo-ai-guidance:\n - Specifies which Salesforce API to use for retrieving data.\n - |-\n ## FIELD BEHAVIOR\n\n This field controls the underlying API technology:\n\n - OPTIONAL: Defaults to \"rest\" if not specified\n - Only relevant for SOQL exports (type=\"soql\")\n - Ignored for distributed exports and blob exports\n - Determines performance characteristics and compatibility\n - |-\n ## AVAILABLE APIs\n\n ### REST API\n ```\n \"api\": \"rest\"\n ```\n\n - **Performance**: Optimized for immediate response and smaller datasets\n - **Concurrency**: Higher - multiple queries can run simultaneously\n - **Data Volume**: Best for \u003c10,000 records\n - **Use Cases**: Lookups, real-time queries, smaller datasets\n - **Special Features**: Required for lookup exports (isLookup=true)\n\n ### Bulk API 2.0\n ```\n \"api\": \"bulk\"\n ```\n\n - **Performance**: Optimized for large data volumes, higher throughput\n - **Concurrency**: Lower - utilizes a job queuing system\n - **Data Volume**: Best for >=10,000 records\n - **Use Cases**: Large data migrations, full dataset exports, reports\n - **Special Features**: Requires \"bulk\" object configuration for settings\n - |-\n ## DEPENDENCIES AND CONSTRAINTS\n\n - When isLookup=true, api must be set to \"rest\" (or left as default)\n - When api=\"bulk\", the bulk object can be configured for additional options\n - Bulk API introduces slight processing latency but handles larger volumes\n - REST API provides immediate results but may time out with very large queries\n - |-\n ## SELECTION GUIDANCE\n\n Choose based on your data volume and response time needs:\n\n - For smaller datasets (\u003c10,000 records) or lookups: use \"rest\"\n - For larger datasets or background processing: use \"bulk\"\n - When immediacy is critical: use \"rest\"\n - When throughput is critical: use \"bulk\"\n\n IMPORTANT: The Bulk API is not compatible with lookup exports (isLookup=true).\n If your export is configured as a lookup, you must use the REST API.\n enum: [\"rest\", \"bulk\"]\n x-enumDescriptions:\n rest: Uses the Salesforce REST API, optimized for smaller datasets and lookup exports.\n bulk: Uses the Salesforce Bulk API 2.0, optimized for large data volumes with higher throughput.\n examples: [\"rest\", \"bulk\"]\n bulk:\n type: object\n description: |-\n Settings for Salesforce Bulk API 2.0 jobs. Only applies when `api` is `bulk` on a SOQL export; ignored otherwise.\n x-celigo-ai-guidance:\n - Configuration parameters for Salesforce Bulk API 2.0 exports.\n - |-\n ## FIELD BEHAVIOR\n\n This object contains settings specific to Bulk API operations:\n\n - REQUIRED when api=\"bulk\" and type=\"soql\"\n - Ignored when api=\"rest\" or type=\"distributed\"\n - Controls behavior of Salesforce Bulk API jobs\n - Provides optimization options for large data volumes\n - |-\n ## IMPLEMENTATION CONTEXT\n\n The Bulk API operates differently from REST API:\n - Creates asynchronous jobs in Salesforce\n - Processes records in batches for higher throughput\n - Optimized for transferring large datasets\n - Has different governor limits and behavior\n properties:\n maxRecords:\n type: integer\n description: |-\n Caps how many records a single Bulk API job retrieves, which helps prevent timeouts with complex queries or large records. When omitted, Salesforce's default applies. Lower values suit complex or custom objects; higher values improve throughput for simple records.\n x-celigo-ai-guidance:\n - Specifies the maximum number of records to retrieve in a single Bulk API job.\n - |-\n ## FIELD BEHAVIOR\n\n This field controls query result size:\n\n - OPTIONAL: Uses Salesforce's default if not specified\n - Sets the `maxRecords` parameter on Bulk API requests\n - Only applicable when api=\"bulk\" and type=\"soql\"\n - Helps prevent timeouts with complex queries or large record sizes\n - |-\n ## TECHNICAL CONSIDERATIONS\n\n - Different Salesforce editions have different limits\n - Values too high may cause timeouts with complex records\n - Values too low may require multiple API calls\n - Standard objects typically support higher limits than custom objects\n - |-\n ## OPTIMIZATION GUIDANCE\n\n - For simple records (few fields): Higher values improve throughput\n - For complex records (many fields): Lower values prevent timeouts\n - For standard objects: 50,000 is usually safe\n - For custom objects: 10,000-25,000 is recommended\n\n IMPORTANT: The Salesforce Bulk API 2.0 has a hard limit of 100 million records\n per job, but practical limits are typically much lower based on record complexity\n and Salesforce instance capacity.\n minimum: 10000\n examples: [10000, 50000, 100000]\n purgeJobAfterExport:\n type: boolean\n default: true\n description: |-\n When true, deletes the Bulk API job in Salesforce after all data is retrieved, keeping the Bulk Data Load Jobs list clean — at the cost of making job details unavailable for later troubleshooting. Has no effect on the data retrieved or the success of the export.\n x-celigo-ai-guidance:\n - Controls whether Bulk API jobs are automatically deleted after completion.\n - |-\n ## FIELD BEHAVIOR\n\n This field manages job cleanup:\n\n - OPTIONAL: Defaults to true if not specified\n - When true, deletes the Bulk API job after all data is retrieved\n - Only applicable when api=\"bulk\" and type=\"soql\"\n - Has no effect on the actual data retrieval or results\n - |-\n ## IMPLEMENTATION IMPACT\n\n When enabled (true):\n - Reduces clutter in the Salesforce Bulk Data Load Jobs UI\n - Prevents accumulation of completed jobs\n - May help stay under job retention limits\n - Makes job details unavailable for later troubleshooting\n\n When disabled (false):\n - Preserves job history for troubleshooting\n - Allows reviewing job details in Salesforce\n - May accumulate many jobs over time\n - |-\n ## BEST PRACTICES\n\n - For production environments: Set to true for cleanliness\n - For testing/development: Set to false for easier debugging\n - For audit-heavy environments: Set to false if job history is needed\n\n IMPORTANT: This setting only affects job metadata cleanup in Salesforce.\n It has no impact on the actual data retrieved or the success of the export.\n examples: [true, false]\n soql:\n type: object\n description: |-\n SOQL query configuration; required when `type` is `soql` and not used otherwise. The query controls which objects, fields, and filter conditions the export retrieves, with either the REST or Bulk API.\n\n Required when type is soql.\n x-celigo-ai-guidance:\n - Configuration for SOQL query-based Salesforce exports.\n - |-\n ## FIELD BEHAVIOR\n\n This object contains the SOQL query settings:\n\n - REQUIRED when type=\"soql\"\n - Not used when type=\"distributed\" or for blob exports\n - Controls what data is retrieved from Salesforce\n - Works with both REST API and Bulk API methods\n - |-\n ## IMPLEMENTATION REQUIREMENTS\n\n The soql object must include a valid query that follows Salesforce SOQL syntax.\n The query determines:\n - Which objects are accessed\n - Which fields are retrieved\n - What filtering conditions are applied\n - How results are sorted and limited\n properties:\n query:\n type: string\n x-celigo-ui-override: >-\n Required by the Salesforce export form when type is soql (soql.query is required:true).\n Encoded to mirror the form so builders produce connectable configurations.\n description: |-\n SOQL statement passed directly to Salesforce that defines which objects, fields, and records the export retrieves. Supports handlebars — for delta exports, filter with `WHERE LastModifiedDate > {{lastExportDateTime}}` — and relationship queries that fetch parent and child records together. Select only the fields you need and use `ORDER BY` for consistent results across pages.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field contains the actual SOQL statement:\n\n - REQUIRED when type=\"soql\"\n - Must follow Salesforce Object Query Language syntax\n - Passed directly to Salesforce API endpoints\n - Can include dynamic values via handlebars\n - |-\n ## QUERY STRUCTURE ELEMENTS\n\n A complete SOQL query typically includes:\n\n ### Field Selection\n ```\n SELECT Id, Name, Email, Phone, Account.Name\n ```\n - List specific fields to retrieve\n - Include relationship fields using dot notation\n - Use * sparingly (only with specific sObjects that support it)\n\n ### Object Selection\n ```\n FROM Contact\n ```\n - Specifies the Salesforce object to query\n - Must be a valid API name (not label)\n - Case-sensitive (match Salesforce API names exactly)\n\n ### Filter Conditions\n ```\n WHERE LastModifiedDate > {{lastExportDateTime}}\n AND IsActive = true\n ```\n - Limits which records are returned\n - Can reference handlebars variables (e.g., for delta exports)\n - Supports standard operators (=, !=, >, \u003c, LIKE, IN, etc.)\n\n ### Relationship Queries\n ```\n SELECT Account.Id, (SELECT Id, FirstName FROM Contacts)\n FROM Account\n ```\n - Retrieves parent and child records in a single query\n - Helps reduce API calls for related data\n - Supports both lookup and master-detail relationships\n - |-\n ## IMPLEMENTATION BEST PRACTICES\n\n - Select only the fields you need (improves performance)\n - Use WHERE clauses to limit data volume\n - For delta exports, use LastModifiedDate with {{lastExportDateTime}}\n - Use ORDER BY for consistent results across multiple pages\n - Avoid SOQL functions in filters when using Bulk API\n - |-\n ## TECHNICAL LIMITS\n\n - Maximum query length: 20,000 characters\n - Maximum relationships traversed: 5 levels\n - Maximum subquery levels: 1 (no nested subqueries)\n - Maximum batch size varies by API (REST: 2,000, Bulk: 10,000+)\n\n IMPORTANT: When using relationship queries, child objects count against\n governor limits differently. For bulk processing of many parent-child records,\n consider separate queries or the oneToMany export setting.\n maxLength: 200000\n examples: [\"SELECT Id, Name FROM Account WHERE LastModifiedDate > {{lastExportDateTime}}\", \"SELECT Id, Name, Email FROM Contact WHERE IsActive = true\", \"SELECT Id, Name, (SELECT Id, FirstName, LastName FROM Contacts) FROM Account\"]\n distributed:\n type: object\n description: |-\n Real-time listener configuration — the object that makes an export a listener. Set it, with `type: distributed`, when creating one; a listener saved from the integrator.io form carries it with no `type`, and the platform fires it either way. Not used on SOQL or file exports. Records reach the flow only when an Apex trigger on `sObjectType` in the connected org calls the Celigo package (`integrator_da__.RealTimeExporter.processExport()`) — no scheduling or manual execution is involved. Integration Apps create their triggers at installation; for a custom flow the customer deploys the trigger in Salesforce, starting from the code the export page shows under Required trigger.\n x-celigo-ai-guidance:\n - |-\n After creating a listener on a custom flow, relay what the Help Center's \"Listen for real-time data\n from Salesforce\" states: an Apex trigger is required per sObject type and, for a custom flow, the\n customer creates it in Salesforce — copy the export page's Required trigger code, replace `\u003cname>`,\n and save it under Setup → Object Manager → \u003cobject> → Triggers → New; deploying to production also\n needs a test class with 75% coverage. Integration Apps create their triggers at installation.\n - |-\n When a listener on an enabled flow never creates a job for a create, update, or delete, the\n Celigo-side causes are `distributed.disabled` true, an offline connection, and a qualifier the\n record does not satisfy — and any qualifier drops delete events (see `qualifier`). Also verify the\n Help Center's trigger requirement in the connected org: an active Apex trigger on the object whose\n code calls `integrator_da__.RealTimeExporter`, and the Celigo package reachable through the\n connection (`GET /v1/connections/{_id}/distributed`, documented in this spec set).\n - |-\n Two things are never why a listener does not fire. The export preview cannot run a listener:\n it validates the body as a SOQL export and answers 422 `missing_required_field` for\n `soql.query` (and for `type` when the listener carries none) before the connection is used,\n so a failed preview is expected, not a configuration defect. And an absent `salesforce.type`\n is the normal shape of a listener saved from the integrator.io form (see `type`).\n properties:\n referencedFields:\n type: [array, 'null']\n description: |-\n Fields from related objects to include in the exported record, written in dot notation (e.g. `Account.Name`, `Owner.Email`, `Custom_Lookup__r.Field__c`). Works with lookup and master-detail relationships, up to 10 unique referenced relationships per export. Referenced fields are retrieved via separate API calls, so include only fields the integration actually needs.\n x-celigo-ai-guidance:\n - Specifies additional fields to retrieve from related objects via relationships.\n - |-\n ## FIELD BEHAVIOR\n\n This field extends the data retrieval beyond the primary object:\n\n - OPTIONAL: If omitted, only direct fields are retrieved\n - Each entry specifies a field on a related object using dot notation\n - Values are included in the exported record data\n - Only works with lookup and master-detail relationships\n - |-\n ## IMPLEMENTATION PATTERNS\n\n ### Parent Object Fields\n ```\n [\"Account.Name\", \"Account.Industry\", \"Account.BillingCity\"]\n ```\n - Retrieves fields from parent objects\n - Useful for including context from parent records\n - Common for child objects like Contacts, Opportunities\n\n ### User/Owner Fields\n ```\n [\"Owner.Email\", \"CreatedBy.Name\", \"LastModifiedBy.Username\"]\n ```\n - Retrieves fields from standard user relationship fields\n - Provides attribution information\n - Useful for auditing and notification scenarios\n\n ### Custom Relationship Fields\n ```\n [\"Custom_Lookup__r.Field_Name__c\", \"Another_Relation__r.Status__c\"]\n ```\n - Works with custom relationship fields\n - Uses __r suffix for the relationship name\n - Can access standard or custom fields on the related object\n - |-\n ## TECHNICAL CONSIDERATIONS\n\n - Maximum 10 unique referenced relationships per export\n - Each referenced field counts against Salesforce API limits\n - Fields must be accessible to the connected user\n - Performance impact increases with each additional relationship\n\n IMPORTANT: Referenced fields are retrieved via separate API calls,\n which can impact performance with large numbers of records or relationships.\n Only include fields that are actually needed by your integration.\n items:\n type: [string, 'null']\n examples:\n - Account.Name\n - Account.Industry\n - Owner.Email\n - Custom_Object__r.Status__c\n disabled:\n type: boolean\n description: |-\n When true, the listener stops processing events while its configuration and Salesforce triggers stay in place. Events that occur while disabled are ignored, not queued — they are not processed retroactively when re-enabled, so consider a delta export to catch up after extended pauses.\n x-celigo-ai-guidance:\n - Controls whether this real-time event listener is active.\n - |-\n ## FIELD BEHAVIOR\n\n This field enables/disables event processing:\n\n - OPTIONAL: Defaults to false if not specified\n - When true, prevents the export from processing any events\n - Preserves configuration while temporarily stopping execution\n - Can be toggled without removing the entire export\n - |-\n ## USE CASES\n\n This field is particularly useful for:\n\n - Temporarily pausing real-time integration during maintenance\n - Testing event configuration without processing\n - Creating standby event handlers for disaster recovery\n - Controlling traffic during peak business periods\n - |-\n ## IMPLEMENTATION NOTES\n\n When disabled (true):\n - Events are NOT queued - they are completely ignored\n - No data will flow through this export\n - The Salesforce triggers remain in place but are inactive\n - No impact on Salesforce performance or API limits\n\n IMPORTANT: When disabled, events that occur will NOT be processed retroactively\n when re-enabled. Consider using a delta export for catching up on missed changes\n after extended disabled periods.\n examples: [true, false]\n qualifier:\n type: [string, 'null']\n description: |-\n SOQL WHERE-clause condition that filters which record events the listener processes; when omitted (stored as `null`), every event for the object is processed. The Celigo Salesforce package evaluates it inside Salesforce as `SELECT Id FROM \u003csObjectType> WHERE (\u003cqualifier>) AND Id IN :ids` before anything is sent, so only fields of the monitored object and SOQL comparison syntax apply — Salesforce formula functions are not part of this grammar. A deleted record is never returned by that query, so a listener with any qualifier receives no delete events; leave it unset when removals must trigger the flow.\n x-celigo-ai-guidance:\n - |-\n Write SOQL WHERE syntax only: `field = value`, `!=`, `\u003c`, `>`, `\u003c=`, `>=`, `LIKE`,\n `IN (...)`, `AND` / `OR`, parentheses for grouping, `NOT ( \u003ccomparison> )`, and\n lowercase `true` / `false` literals.\n - |-\n Formula syntax is rejected at save with HTTP 422 `Either whereClause expression ...\n is invalid ... unexpected token` — observed for `AND(a, b)` argument lists,\n `NOT(field)` on a bare boolean, `NOT(ISNEW())`, `ISDELETED()`, and `ISBLANK(field)`;\n uppercase `= FALSE` has been rewritten to `NOT(field)` and rejected the same way.\n - |-\n The rejection surfaces on a real save (a create, or the user's Accept of a staged\n update) — dry-run validation does not run the Salesforce-side check, so a staged\n card carrying an invalid qualifier passes dry-run and fails at Accept.\n - |-\n `IsDeleted = true` in a qualifier admits nothing — the query runs against the live\n record, which no longer exists after a delete, so any qualified listener stops\n receiving deletions. To keep deletes flowing, leave the qualifier unset and drop\n unwanted events with the export's output filter, accepting that those events still\n start a run.\n - |-\n Compare a date field with an unquoted `YYYY-MM-DD` and a dateTime field (`CreatedDate`,\n `LastModifiedDate`, `SystemModstamp`, custom Date/Time fields) with an unquoted\n `YYYY-MM-DDThh:mm:ssZ` or `YYYY-MM-DDThh:mm:ss±hh:mm` — SOQL's date and dateTime literal\n grammar, which accepts neither a date-only value on a dateTime field nor a quoted date\n or dateTime literal.\n - |-\n The UI's \"Define qualification criteria\" rule builder (Field / Operator / Value rows)\n quotes every typed value when it builds this string and un-quotes only `YYYY-MM-DD`,\n `…Thh:mm:ss.sss±hhmm`, `…Thh:mm:ss±hh:mm` and `…Thh:mm:ssZ`; a timestamp typed in any\n other spelling (for example `2024-01-02T00:00:00.000Z`) is stored quoted. A user working\n in that dialog types `2024-01-02T00:00:00Z`; a value set through the API is stored\n exactly as written.\n examples: [\"Amount > 1000\", \"Status__c = 'Approved'\", \"LeadSource = 'Web' AND Competitor__c = 'Oracle'\", \"IsDirect = false\", \"StageName = 'Won' AND ( NOT ( Unresolved_Errors__c >= 1 ) )\", \"CreatedDate > 2024-01-02T00:00:00Z\", \"CloseDate >= 2024-01-01\"]\n batchSize:\n type: [integer, 'null']\n description: |-\n Controls how many event records are grouped into each real-time processing batch. Stored as `null` when not configured. Smaller batches lower latency for time-sensitive operations; larger batches improve throughput for high-volume objects. Does not limit how many records are processed in total — only how they are grouped.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field affects event processing efficiency:\n\n - OPTIONAL: Uses system default if not specified\n - Valid range: 4 to 200 records per batch\n - Affects how events are grouped before processing\n - Balance between latency and throughput\n - |-\n ## PERFORMANCE CONSIDERATIONS\n\n ### Smaller Batch Sizes (4-20)\n ```\n \"batchSize\": 10\n ```\n - Lower latency - events processed more immediately\n - More overhead for small numbers of records\n - Better for time-sensitive operations\n - More resilient for complex record processing\n\n ### Larger Batch Sizes (50-200)\n ```\n \"batchSize\": 100\n ```\n - Higher throughput - better efficiency for many records\n - Slight increase in processing delay\n - Better for high-volume operations\n - More efficient use of API calls and resources\n\n ## IMPLEMENTATION GUIDANCE\n\n Choose based on your volume and timing requirements:\n\n - For high-volume objects (many changes per minute): Use larger batches\n - For time-sensitive operations: Use smaller batches\n - For complex processing logic: Use smaller batches\n - For efficiency and throughput: Use larger batches\n\n IMPORTANT: The batch size doesn't limit how many records can be processed\n in total, only how they're grouped for processing. All events will eventually\n be processed regardless of batch size.\n minimum: 4\n maximum: 200\n examples: [10, 50, 100]\n skipExportFieldId:\n type: [string, 'null']\n description: |-\n API name of a Salesforce checkbox field used to prevent infinite loops in bidirectional syncs: when the integration updates a record, this field is set so the resulting event is ignored, then cleared automatically. Required for bidirectional sync scenarios. The field must be a checkbox dedicated to integration use and updateable by the integration user.\n x-celigo-ai-guidance:\n - Specifies a boolean field that prevents integration loops in bidirectional sync.\n - |-\n ## FIELD BEHAVIOR\n\n This field provides a loop prevention mechanism:\n\n - OPTIONAL: If omitted, no loop prevention is applied\n - Must reference a valid boolean/checkbox field on the object\n - Field must be updateable via the Salesforce API\n - System automatically manages the field's value\n - |-\n ## IMPLEMENTATION MECHANISM\n\n The loop prevention works as follows:\n\n 1. When your integration updates a record in Salesforce\n 2. The system temporarily sets this field to true\n 3. The update triggers Salesforce's normal event system\n 4. But events where this field is true are ignored\n 5. The system automatically clears the field afterward\n - |-\n ## USE CASES\n\n This field is critical for:\n\n - Bidirectional synchronization scenarios\n - Preventing infinite update loops\n - Implementing changes that flow both ways\n - Distinguishing between user changes and integration changes\n - |-\n ## FIELD REQUIREMENTS\n\n The field you specify must be:\n - A checkbox (Boolean) field in Salesforce\n - Created specifically for integration purposes\n - Not used by other business processes\n - Updateable by the integration user\n\n IMPORTANT: For bidirectional sync scenarios, this field is required.\n Without it, updates from your integration would trigger events that\n could create infinite loops between systems.\n examples: [\"Skip_Export__c\", \"Exclude_From_Integration__c\", \"Integration_Update__c\"]\n relatedLists:\n type: [array, 'null']\n description: |-\n Child record sets to include with the parent record when it changes, one entry per related list. When omitted (stored as `null`), only the primary record is processed. Each related list adds Salesforce API calls and increases payload size.\n x-celigo-ai-guidance:\n - Configuration for retrieving child records related to the primary object.\n - |-\n ## FIELD BEHAVIOR\n\n This field enables parent-child data synchronization:\n\n - OPTIONAL: If omitted, only the primary record is processed\n - Each array entry configures one related list/child object\n - Child records are included with their parent in the payload\n - Automatically retrieves child records when parent changes\n - |-\n ## IMPLEMENTATION CONTEXT\n\n This feature allows you to:\n - Synchronize complete object hierarchies in real-time\n - Include child records when a parent record changes\n - Process parent-child data together in a single flow\n - Maintain relationships between objects across systems\n - |-\n ## TECHNICAL IMPACT\n\n - Each related list requires additional Salesforce API calls\n - Performance impact increases with each related list\n - Data volume can increase significantly with many children\n - Parent-child structures may require special handling in flows\n items:\n type: object\n description: |-\n Defines how to retrieve one type of child record related to the primary object. Configure multiple entries to retrieve different types of children.\n x-celigo-ai-guidance:\n - Configuration for a single related list (child object) to include.\n - |-\n Each object in this array defines how to retrieve one type of child records\n related to the primary object.\n Multiple related lists can be configured to retrieve different types of children.\n properties:\n referencedFields:\n type: array\n description: |-\n API names of the child object fields to retrieve; only the listed fields are included, and an empty array retrieves only the `Id` field. Include only fields the integration needs — each field adds data volume and processing time.\n x-celigo-ai-guidance:\n - Specifies which fields to retrieve from the child records.\n - |-\n ## FIELD BEHAVIOR\n\n This field selects child record fields:\n\n - REQUIRED for each related list configuration\n - Must contain valid API field names for the child object\n - Only listed fields will be retrieved from child records\n - Empty array will retrieve only Id field\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n - Include only fields needed by your integration\n - Always include key identifier fields\n - Consider relationship fields if needed\n - Balance between completeness and performance\n\n IMPORTANT: Each field increases data volume and processing time.\n Only include fields that your integration actually needs to process.\n items:\n type: [string, 'null']\n examples:\n - FirstName\n - LastName\n - Email\n - LineItemNumber\n - Status__c\n parentField:\n type: string\n description: |-\n API name of the lookup or master-detail field on the child object that references the parent (e.g. `AccountId`, `Parent_Object__c`) — the field's API name, not the relationship name. Used to build the query that fetches children for each parent record.\n x-celigo-ai-guidance:\n - Specifies the field on the child object that relates back to the parent.\n - |-\n ## RELATIONSHIP FIELD PATTERNS\n\n ### Standard Relationships\n ```\n \"parentField\": \"AccountId\"\n ```\n - For standard parent-child relationships\n - Field name typically ends with \"Id\"\n - References standard objects\n\n ### Custom Relationships\n ```\n \"parentField\": \"Parent_Object__c\"\n ```\n - For custom parent-child relationships\n - Field name typically ends with \"__c\"\n - References custom objects\n - |-\n ## TECHNICAL DETAILS\n\n The system uses this field to construct a query like:\n ```\n SELECT [referencedFields] FROM [sObjectType]\n WHERE [parentField] = [parent record Id]\n ```\n\n IMPORTANT: This must be the exact API name of the field on the child\n object that creates the relationship to the parent, not the relationship\n name itself.\n examples: [\"AccountId\", \"OpportunityId\", \"Parent_Object__c\", \"Case__c\"]\n sObjectType:\n type: string\n description: |-\n Case-sensitive API name of the child object to retrieve (e.g. `Contact`, `Custom_Child__c`) — use the API name, not the label. The object must have a relationship field to the parent and be accessible to the connected user.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field identifies the child object type:\n\n - REQUIRED for each related list configuration\n - Must be a valid Salesforce API object name\n - Case-sensitive (match Salesforce naming exactly)\n - Can be standard or custom object\n - |-\n ## OBJECT NAME PATTERNS\n\n ### Standard Objects\n ```\n \"sObjectType\": \"Contact\"\n ```\n - Standard Salesforce objects\n - No namespace or suffix\n - First letter capitalized\n\n ### Custom Objects\n ```\n \"sObjectType\": \"Custom_Object__c\"\n ```\n - Custom Salesforce objects\n - API name with \"__c\" suffix\n - Case-sensitive, including underscores\n - |-\n ## RELATIONSHIP COMPATIBILITY\n\n The sObjectType must:\n - Have a relationship field to the parent object\n - Be accessible to the connected user\n - Support standard SOQL queries\n\n IMPORTANT: Use the exact API name of the object, not its label.\n This value is case-sensitive and must match Salesforce's naming exactly.\n examples: [\"Contact\", \"OpportunityLineItem\", \"Custom_Child__c\", \"Related_Object__c\"]\n filter:\n type: string\n description: |-\n SOQL condition that limits which child records are included; when omitted, all related children are retrieved. Provide only the condition expression without the `WHERE` keyword — it is combined automatically with the parent relationship filter.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field adds filtering to child record retrieval:\n\n - OPTIONAL: If omitted, all related child records are included\n - Contains only the condition expression (without \"WHERE\" keyword)\n - Uses standard SOQL syntax for conditions\n - Applied in addition to the parent relationship filter\n - |-\n ## FILTERING PATTERNS\n\n ### Simple Condition\n ```\n \"filter\": \"IsActive = true\"\n ```\n - Basic field comparison\n - Only active related records are included\n\n ### Multiple Conditions\n ```\n \"filter\": \"Status__c = 'Open' AND Priority = 'High'\"\n ```\n - Combined conditions with logical operators\n - Only records matching all conditions are included\n\n ### Complex Filtering\n ```\n \"filter\": \"CreatedDate > LAST_N_DAYS:30 OR IsClosed = false\"\n ```\n - Can use Salesforce date literals and functions\n - Can mix different types of conditions\n - |-\n ## TECHNICAL DETAILS\n\n The system appends this to the automatically generated relationship query:\n ```\n SELECT [fields] FROM [sObjectType]\n WHERE [parentField] = [parent ID] AND ([filter])\n ```\n\n IMPORTANT: Do not include the \"WHERE\" keyword in this field.\n Only include the condition expression itself, as it will be combined\n with the parent relationship condition automatically.\n examples: [\"IsActive = true\", \"Amount > 0\", \"Status__c = 'Open'\", \"CreatedDate > LAST_N_DAYS:30\"]\n orderBy:\n type: string\n description: |-\n Sort order for the retrieved child records, as field names with optional `ASC`/`DESC` directions (e.g. `CreatedDate DESC`). Provide only the fields and directions without the `ORDER BY` keywords. When omitted, Salesforce determines the order.\n x-celigo-ai-guidance:\n - |-\n ## TECHNICAL DETAILS\n\n The system appends this to the automatically generated relationship query:\n ```\n SELECT [fields] FROM [sObjectType]\n WHERE [parentField] = [parent ID]\n ORDER BY [orderBy]\n ```\n\n IMPORTANT: Do not include the \"ORDER BY\" keywords in this field.\n Only include the field names and sort directions, as they will be\n added to the query with the proper syntax automatically.\n - |-\n ## FIELD BEHAVIOR\n\n This field controls child record ordering:\n\n - OPTIONAL: If omitted, order is determined by Salesforce\n - Contains only field and direction (without \"ORDER BY\" keywords)\n - Uses standard SOQL syntax for sorting\n - Applied to the child records query\n - |-\n ## ORDERING PATTERNS\n\n ### Single Field Ascending (Default)\n ```\n \"orderBy\": \"Name\"\n ```\n - Sorts by a single field in ascending order\n - ASC is implied if not specified\n\n ### Single Field Descending\n ```\n \"orderBy\": \"CreatedDate DESC\"\n ```\n - Sorts by a single field in descending order\n - Must explicitly specify DESC\n\n ### Multiple Fields\n ```\n \"orderBy\": \"Priority DESC, CreatedDate ASC\"\n ```\n - Sorts by multiple fields in specified directions\n - Comma-separated list of fields with optional directions\n examples: [\"CreatedDate DESC\", \"Name ASC\", \"LineNumber\", \"Priority DESC, CreatedDate ASC\"]\n required:\n - parentField\n - referencedFields\n - sObjectType\n # Flavor gating:\n # - `type: distributed` docs carry `distributed` + `sObjectType`.\n # - Listeners saved from the integrator.io form carry no `salesforce.type` (the\n # real-time form does not write it): a `distributed` object with no `type` at all\n # still runs, so its presence alone selects the distributed flavor.\n # - Blob file exports carry `id` (+ `sObjectType`) and no `soql` even when a stray\n # `type: soql` is present, so `soql` must not be demanded for them.\n # - Metadata-only stubs (`{metadata: ...}` written onto non-Salesforce exports)\n # carry no flavor keys; requiring `type`/`soql` on them would reject server-written\n # documents, so the soql-flavor requireds are stub-gated.\n if:\n properties:\n type:\n const: distributed\n required:\n - type\n then:\n required:\n - distributed\n - sObjectType\n else:\n if:\n required:\n - distributed\n then: {}\n else:\n if:\n required:\n - id\n then:\n required:\n - sObjectType\n else:\n if:\n not:\n propertyNames:\n enum:\n - metadata\n then:\n required:\n - soql\n - type\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":56341,"content_sha256":"6b5d7d80c450ecb56889818d45146099255a170299d4d7483cf90df083d4e784"},{"filename":"references/schemas/simple.yml","content":"Simple:\n type: object\n description: |-\n Configuration for `simple` (data-loader) exports — exports that accept manually uploaded\n files through the data-loader UI instead of fetching data from a connection. Relevant only\n when the export's `type` is `simple`; the uploaded content is parsed using the same file\n settings as file-based exports.\n properties:\n file:\n $ref: ./file.yml#/File\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":413,"content_sha256":"e006221263ee72e849ecaaa33244acfe2f52a38eddf8812dc1db39b33128b5a2"},{"filename":"references/schemas/test.yml","content":"Test:\n type: object\n description: |-\n Configures test exports that cap the number of records retrieved, for safely developing and\n validating against small data samples. Required when the export's type is \"test\"; omit for\n other export types. A test export behaves like a standard export in every other way (filters,\n pagination, processing) and stores no state between runs.\n properties:\n limit:\n type: integer\n default: 1\n x-celigo-ui-override: >-\n Required by the export form when the export type is \"test\" (test.limit is required:true,\n default 1, visibleWhen type=test). Encoded to mirror the form so builders produce\n connectable configurations.\n description: |-\n Caps the total records a test run processes, counted on top-level records before\n oneToMany processing; the maximum exists to prevent accidentally processing large\n datasets during development. The cap applies across pages — processing stops once the\n limit is reached regardless of pageSize. When transitioning to production, leave this in\n place and change the export's type field; the limit then no longer applies.\n minimum: 1\n maximum: 100\n examples: [10, 25, 50]\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":1249,"content_sha256":"5ec22e14a79a8f89518c85496c3874e3a6f7caf679c22afd555eb3e0921ca4b5"},{"filename":"references/schemas/topic.yml","content":"Topic:\n type: object\n description: |-\n Subscription settings for an Event Streams listener (`adaptorType: TopicExport`). The\n listener consumes messages published to the topic and hands them to the flow in batches;\n the platform delivers them internally, so the export carries no `_connectionId`. Present\n when `adaptorType` is `TopicExport`; the server fills `startPosition` and `maxWaitTimeMs`\n with their defaults when they are omitted.\n x-celigo-ai-guidance:\n - |-\n Send `type: \"stream\"` together with this object, as the listener form always does. The\n save also succeeds without `type`, and the listener then runs as a batch export: its flow\n never starts when a message is published, a manual run re-reads every retained message\n without committing a position, and `GET /v1/topics/{_id}/exports/{_exportId}/offsets`\n reports `hasCursor: false`. `PATCH` cannot add the type afterwards (`/type` is not a\n whitelisted property): `PUT` the full document with `type: \"stream\"`, then disable and\n enable the flow. Sending this object without `adaptorType` is enough for the server to\n infer `TopicExport`.\n - |-\n The subscription is created when the listener's flow goes from disabled to enabled (a\n `PATCH` or `PUT` that changes `disabled`); the first run after a publish follows within\n about a minute. A flow created with `disabled: false` in one request has not subscribed,\n and a save that leaves `disabled` unchanged (a rename) does not subscribe it — toggle the\n flow. Under `startPosition: latest`, messages published before the subscription exists\n are skipped. A disabled flow keeps its position and resumes from it when enabled again.\n - |-\n A topic listener cannot be a lookup: `isLookup: true` with `type: \"stream\"` fails the save\n with 422 `not_supported` (\"Lookup functionality is not available for Stream Export\"), and a\n lookup-shaped TopicExport saved without `type` never runs.\n - |-\n Never add a `_connectionId`: the server then validates the document as that connection's\n adaptor instead (an http connection fails with 422 missing_required_field \"http subschema\n not defined\"). A TopicExport carries no `cdc` object either — sending `cdc.snapshotMode`\n fails with 422 `not_allowed` (\"CDC snapshotMode can only be used with stream export type\").\n properties:\n _topicId:\n type: string\n format: objectId\n x-celigo-refModel: topics\n x-celigo-ui-override: >-\n Required by the Event Streams listener form (fieldDefinitions/resources/exports/export.js:\n 'topic._topicId' is required:true, validWhen isNot ''). The API saves a TopicExport\n without it, so the requirement is encoded on the request schema to mirror the form and\n keep listeners bound to a topic.\n description: |-\n Topic this listener subscribes to. Create or look it up with `GET /v1/topics`; the\n topic must belong to the same account. Pointing an existing listener at a different\n topic starts a new subscription, read from `startPosition`. A clone made in the same\n environment — of this export, or of a flow that contains it — keeps `_topicId`, so the\n copy becomes another listener on the topic once its flow is enabled.\n x-celigo-ai-guidance:\n - |-\n An id that is not a topic in the account fails the save with 422 `invalid_ref`\n (\"\u003cid> is not a valid topic._topicId\"); a malformed id fails with 422 `invalid_field`.\n examples: [\"6ab1f48d38d7733b4599875c\"]\n startPosition:\n type: string\n enum: [latest, earliest]\n default: latest\n x-enumDescriptions:\n latest: Deliver only messages published after the listener subscribes\n earliest: Replay every retained message on the topic before delivering new ones\n description: |-\n Where the listener starts reading when it first subscribes to `_topicId`. Use `earliest`\n to backfill retained history, `latest` (the default) to process only new messages. Read\n on create and whenever `_topicId` changes; on other updates the stored value is kept even\n if a different value is sent.\n examples: [\"latest\", \"earliest\"]\n maxWaitTimeMs:\n type: integer\n minimum: 1000\n default: 30000\n description: |-\n Longest time, in milliseconds, the listener waits to fill a batch before delivering\n whatever it has collected. Lower values reduce latency for sparse topics; the default\n of 30 seconds favors fuller batches. The batch record cap is the export's `pageSize`.\n x-celigo-ai-guidance:\n - |-\n Values below 1000 (including 0 and negatives) fail the save with 422 `min`\n (\"Path `topic.maxWaitTimeMs` (0) is less than minimum allowed value (1000)\"). The form\n edits this as whole seconds and saves seconds × 1000.\n examples: [30000, 5000]\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":4986,"content_sha256":"56de91d4d6fb9a04e380fd4d3c8f865083aa1e13331111e2cdd0af035cf3dc9f"},{"filename":"references/schemas/webhook.yml","content":"Webhook:\n type: object\n description: |-\n Configuration for webhook listeners that receive data through incoming HTTP requests. Required when the export's `type` is `webhook`; omit for other export types. Senders POST to the export's listener URL, `https://api.integrator.io/v1/exports/{_id}/data` (or the same path on the account's regional API host); the platform validates each request using the configured `verify` method, passes the payload to subsequent flow steps, and returns a configurable HTTP response to the caller. A request is answered 204 by default when accepted, 401 when it fails verification, and 403 `unable_to_run_flows` while every flow the export feeds is disabled; a refused request is not queued, so the sender must resend it.\n x-celigo-ai-guidance:\n - Configuration object for real-time event listeners that receive data via incoming HTTP requests.\n - |-\n This object is REQUIRED when the export's type field is set to \"webhook\" and\n should not be included for other export types.\n Webhook exports create dedicated HTTP endpoints that can receive data from\n external systems in real-time,\n enabling event-driven integration architectures.\n - 'For optimal AI agent implementation, consider these guidelines:'\n - |-\n ## WEBHOOK SECURITY MODELS\n\n Webhooks support multiple security verification methods, each requiring different fields:\n\n 1. **HMAC Verification** (Most secure, recommended for production)\n - Required fields: verify=\"hmac\", key, algorithm, encoding, header\n - Verifies a cryptographic signature included with each request\n - Ensures data integrity and authenticity\n\n 2. **Token Verification** (Simple shared secret)\n - Required fields: verify=\"token\", token, path\n - Checks for a specific token value in the request\n - Simpler but less secure than HMAC\n\n 3. **Basic Authentication** (HTTP standard)\n - Required fields: verify=\"basic\", username, password\n - Uses HTTP Basic Authentication headers\n - Compatible with most HTTP clients\n\n 4. **Secret URL** (Simplest but least secure)\n - Required fields: verify=\"secret_url\", token\n - Relies solely on URL obscurity for security\n - The token is embedded in the webhook URL to create a unique, hard-to-guess endpoint\n - Suitable only for non-sensitive data or testing\n\n 5. **Public Key** (Advanced, for specific providers)\n - Required fields: verify=\"publickey\", key\n - Uses public key cryptography for verification\n - Only available for certain providers\n - |-\n ## RESPONSE CUSTOMIZATION\n\n You can customize how the webhook responds to callers with these field groups:\n\n 1. **Standard Success Response**\n - Fields: successStatusCode, successBody, successMediaType, successResponseHeaders\n - Controls how the webhook responds to valid requests\n\n 2. **Challenge Response** (For subscription verification)\n - Fields: challengeSuccessBody, challengeSuccessStatusCode, challengeSuccessMediaType, challengeResponseHeaders\n - Controls how the webhook responds to verification/challenge requests\n - |-\n ## IMPLEMENTATION SCENARIOS\n\n Webhooks are commonly used for:\n\n 1. **Real-time data synchronization**\n - E-commerce platforms sending order notifications\n - CRM systems delivering contact updates\n - Payment processors reporting transaction events\n\n 2. **Event-driven processes**\n - Triggering fulfillment when orders are placed\n - Initiating approval workflows on document submissions\n - Executing business logic when status changes occur\n\n 3. **System integration**\n - Connecting SaaS applications without polling\n - Building composite applications from microservices\n - Creating fan-out architectures for event distribution\n properties:\n provider:\n type: string\n description: |-\n Source application sending the webhook, used to pre-configure security settings and payload parsing for that platform. Use `custom` (the default when omitted) for unlisted providers or when you need full manual control over the security configuration. Provider-specific choices may require credentials (tokens, keys) mandated by that platform.\n x-celigo-ai-guidance:\n - |-\n Specifies the source application sending webhook data,\n enabling platform-specific optimizations.\n - |-\n ## FIELD BEHAVIOR\n\n This field determines how the webhook handles incoming requests:\n\n - OPTIONAL: Defaults to \"custom\" if not specified\n - When a specific provider is selected, the system:\n 1. Pre-configures appropriate security settings for that platform\n 2. Applies platform-specific payload parsing rules\n 3. May enable additional features only relevant to that provider\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n ### Provider-specific configurations\n\n When you know the exact source system, select its specific provider:\n ```\n \"provider\": \"shopify\"\n ```\n - Automatically configures proper HMAC verification settings\n - Optimizes payload parsing for Shopify's webhook format\n - May enable additional Shopify-specific features\n\n ### Custom configuration\n\n For generic webhooks or unlisted providers, use custom:\n ```\n \"provider\": \"custom\"\n ```\n - Requires manual configuration of all security settings\n - Maximum flexibility for handling any webhook format\n - Recommended for custom applications or newer platforms\n\n ### Selection criteria\n\n Choose a specific provider when:\n - The source system is explicitly listed in the enum values\n - You want to leverage pre-configured settings\n - The integration must follow platform-specific practices\n\n Choose \"custom\" when:\n - The source system is not listed\n - You need full control over webhook configuration\n - You're building a custom interface or protocol\n\n IMPORTANT: Some providers enforce specific security methods. When selecting a\n provider, ensure you have the necessary security credentials (tokens, keys, etc.)\n as required by that platform.\n enum:\n - github\n - shopify\n - travis\n - travis-org\n - slack\n - dropbox\n - onfleet\n - helpscout\n - errorception\n - box\n - stripe\n - aha\n - jira\n - pagerduty\n - postmark\n - mailchimp\n - intercom\n - activecampaign\n - segment\n - recurly\n - shipwire\n - surveymonkey\n - parseur\n - mailparser-io\n - hubspot\n - integrator-extension\n - custom\n - sapariba\n - happyreturns\n - typeform\n x-enumDescriptions:\n github: GitHub repository event webhooks with HMAC-SHA verification.\n shopify: Shopify store event webhooks with HMAC-SHA256 verification.\n travis: Travis CI build event webhooks (travis-ci.com).\n travis-org: Travis CI build event webhooks (travis-ci.org).\n slack: Slack workspace event webhooks with token verification.\n dropbox: Dropbox file change notification webhooks.\n onfleet: Onfleet delivery management event webhooks.\n helpscout: Help Scout customer support event webhooks.\n errorception: Errorception error tracking event webhooks.\n box: Box cloud storage event webhooks.\n stripe: Stripe payment event webhooks with signature verification.\n aha: Aha! product management event webhooks.\n jira: Jira issue tracking event webhooks.\n pagerduty: PagerDuty incident management event webhooks.\n postmark: Postmark email delivery event webhooks.\n mailchimp: Mailchimp email marketing event webhooks.\n intercom: Intercom customer messaging event webhooks.\n activecampaign: ActiveCampaign marketing automation event webhooks.\n segment: Segment analytics event webhooks.\n recurly: Recurly subscription billing event webhooks.\n shipwire: Shipwire fulfillment event webhooks.\n surveymonkey: SurveyMonkey survey response event webhooks.\n parseur: Parseur email parsing event webhooks.\n mailparser-io: Mailparser.io email parsing event webhooks.\n hubspot: HubSpot CRM event webhooks.\n integrator-extension: Celigo integrator extension internal event webhooks.\n custom: Generic webhook with fully manual security and payload configuration.\n sapariba: SAP Ariba procurement event webhooks.\n happyreturns: Happy Returns return management event webhooks.\n typeform: Typeform form submission event webhooks.\n examples: [\"shopify\", \"stripe\", \"custom\"]\n verify:\n type: string\n x-celigo-ui-override: >-\n Required by the webhook export form (verify is required:true). Encoded to mirror the form\n so builders produce connectable configurations.\n description: |-\n Verification method applied to every incoming request before processing; required for all webhook exports. Each method needs companion fields: `hmac` requires `key`, `algorithm`, `encoding`, and `header` (except on connector-backed webhooks, where the connector definition supplies them); `token` requires `token`, with `tokenLocation` defaulting to `body` and selecting which location-specific field applies; `basic` requires `username` and `password`; `secret_url` requires `token`. Prefer `hmac` when the source system supports it — it is the most secure option; `secret_url` relies only on URL obscurity and suits non-sensitive data or testing.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field is the primary control for webhook security:\n\n - REQUIRED for all webhook exports\n - Determines which additional security fields must be configured\n - Controls how incoming requests are validated before processing\n - |-\n ## VERIFICATION METHODS\n\n ### HMAC Verification\n ```\n \"verify\": \"hmac\"\n ```\n - Most secure method, cryptographically verifies request integrity\n - REQUIRES: key, algorithm, encoding, header fields\n - Validates a cryptographic signature included in the request header\n - Works well with providers that support HMAC (Shopify, Stripe, GitHub, etc.)\n\n ### Token Verification\n ```\n \"verify\": \"token\"\n ```\n - Simple verification using a shared secret token\n - REQUIRES: token, path fields\n - Checks for a specific token value in the request body or query params\n - Good for simple scenarios with trusted networks\n\n ### Basic Authentication\n ```\n \"verify\": \"basic\"\n ```\n - Standard HTTP Basic Authentication\n - REQUIRES: username, password fields\n - Validates credentials sent in the Authorization header\n - Compatible with most HTTP clients and tools\n\n ### Public Key\n ```\n \"verify\": \"publickey\"\n ```\n - Advanced verification using public key cryptography\n - REQUIRES: key field (containing the public key)\n - Only available for certain providers that use asymmetric cryptography\n - Highest security level but more complex to configure\n\n ### Secret URL\n ```\n \"verify\": \"secret_url\"\n ```\n - Simplest method, relies solely on the obscurity of the URL\n - REQUIRES: token field (the token is embedded in the webhook URL to create a unique, hard-to-guess endpoint)\n - Only suitable for non-sensitive data or testing environments\n - Not recommended for production use with sensitive data\n\n IMPORTANT: Choose the security method that matches your source system's capabilities.\n If the source system supports multiple verification methods, HMAC is generally the\n most secure option.\n enum: [\"token\", \"hmac\", \"basic\", \"secret_url\"]\n x-enumDescriptions:\n token: Verifies requests by matching a shared secret token found at a specified location in the request.\n hmac: Verifies requests using a cryptographic HMAC signature sent in a request header.\n basic: Verifies requests using standard HTTP Basic Authentication (username and password).\n secret_url: Relies on a unique, hard-to-guess token embedded in the webhook URL for security.\n examples: [\"token\", \"hmac\"]\n token:\n type: string\n description: |-\n Shared secret used when `verify` is `token` or `secret_url`. For `token` verification, the value found at `path` in each request must exactly match (case- and whitespace-sensitive) or the request is rejected with a 401. For `secret_url`, the token is embedded in the webhook URL to create a hard-to-guess endpoint — generate a random, high-entropy value and treat it as a sensitive credential.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field defines the expected token value:\n\n - REQUIRED when verify=\"token\" or verify=\"secret_url\"\n - When verify=\"token\": must be a string that exactly matches what the sender will provide. Used with the path field to locate and validate the token in the request.\n - When verify=\"secret_url\": the token is embedded in the webhook URL to create a unique, hard-to-guess endpoint. Generate a random, high-entropy value.\n - Case-sensitive and whitespace-sensitive\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n The token verification flow works as follows:\n 1. The webhook receives an incoming request\n 2. The system looks for the token at the location specified by the path field\n 3. If the found value exactly matches this token value, the request is processed\n 4. If no match is found, the request is rejected with a 401 error\n\n ### Security best practices\n\n For maximum security:\n - Use a random, high-entropy token (32+ characters)\n - Include a mix of uppercase, lowercase, numbers, and special characters\n - Don't use predictable values like company names or common words\n - Rotate tokens periodically for sensitive integrations\n\n ### Common implementations\n\n ```\n \"token\": \"\u003c32+ random characters>\"\n ```\n\n ```\n \"token\": \"whsec_\u003csigning secret from the sending system>\"\n ```\n\n IMPORTANT: Never share this token in public repositories or documentation.\n Treat it as a sensitive credential similar to a password.\n examples: [\"verification_token_abcdef\", \"whsec_8fb2e91a5c6b3e7dEXAMPLE\"]\n algorithm:\n type: string\n description: |-\n Hashing algorithm used to validate HMAC signatures when `verify` is `hmac`. Must match the algorithm the webhook sender uses — a mismatch causes every request to be rejected. Use `sha256` unless the provider explicitly requires another value.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field determines how signatures are validated:\n\n - REQUIRED when verify=\"hmac\"\n - Must match the algorithm used by the webhook sender\n - Affects security strength and compatibility\n - |-\n ## ALGORITHM SELECTION\n\n ### SHA-256 (Recommended)\n ```\n \"algorithm\": \"sha256\"\n ```\n - Modern, secure hash algorithm\n - Industry standard for most new webhook implementations\n - Preferred choice for all new integrations\n - Used by Shopify, Stripe, and many modern platforms\n\n ### SHA-1 (Legacy)\n ```\n \"algorithm\": \"sha1\"\n ```\n - Older, less secure algorithm\n - Still used by some legacy systems\n - Only select if the provider explicitly requires it\n - GitHub webhooks used this historically\n\n ### SHA-384/SHA-512 (High Security)\n ```\n \"algorithm\": \"sha384\"\n \"algorithm\": \"sha512\"\n ```\n - Higher security variants with longer digests\n - Use when specified by the provider or for sensitive data\n - Less common but supported by some security-focused systems\n\n IMPORTANT: This MUST match the algorithm used by the webhook sender.\n Mismatched algorithms will cause all webhook requests to be rejected.\n enum: [\"sha1\", \"sha256\", \"sha384\", \"sha512\"]\n x-enumDescriptions:\n sha1: Uses the SHA-1 hashing algorithm for HMAC signature verification (legacy, less secure).\n sha256: Uses the SHA-256 hashing algorithm for HMAC signature verification (recommended).\n sha384: Uses the SHA-384 hashing algorithm for HMAC signature verification (higher security, longer digest).\n sha512: Uses the SHA-512 hashing algorithm for HMAC signature verification (highest security, longer digest).\n examples: [\"sha256\", \"sha1\"]\n encoding:\n type: string\n description: |-\n Encoding of the HMAC signature value when `verify` is `hmac`. Must match the encoding the webhook sender uses — a mismatch causes requests to be rejected with a 401 even when the signature is otherwise correct. With `provider: custom`, `hex` is compared as lowercase hex: an uppercase hex signature is rejected.\n x-celigo-ai-guidance:\n - Specifies the encoding format used for the HMAC signature in webhook requests.\n - |-\n ## FIELD BEHAVIOR\n\n This field determines how signature values are encoded:\n\n - REQUIRED when verify=\"hmac\"\n - Must match the encoding used by the webhook sender\n - Affects how binary signature values are represented as strings\n - |-\n ## ENCODING OPTIONS\n\n ### Hexadecimal (hex)\n ```\n \"encoding\": \"hex\"\n ```\n - Represents the signature as a string of hexadecimal characters (0-9, a-f)\n - Most common encoding for web-based systems\n - Used by many platforms including Stripe and some Shopify implementations\n - Example output: \"8f7d56a32e1c9b47d882e3aa91341f64\"\n\n ### Base64\n ```\n \"encoding\": \"base64\"\n ```\n - Represents the signature using base64 encoding\n - More compact than hex (about 33% shorter)\n - Used by platforms like Shopify (newer implementations) and some GitHub scenarios\n - Example output: \"j31WozbhtrHYeC46qRNB9k==\"\n\n IMPORTANT: This MUST match the encoding used by the webhook sender.\n Mismatched encoding will cause all webhook requests to be rejected even if\n the signature is mathematically correct.\n enum: [\"hex\", \"base64\"]\n x-enumDescriptions:\n hex: The HMAC signature is encoded as a hexadecimal string.\n base64: The HMAC signature is encoded as a Base64 string.\n examples: [\"hex\", \"base64\"]\n key:\n type: string\n description: |-\n Secret used to validate signatures when `verify` is `hmac`. The system computes a signature of the request body using this key and the configured `algorithm`, then compares it with the signature sent in the request header named by `header`. Treat it as a highly sensitive credential — never expose it in repositories or logs.\n x-celigo-ai-guidance:\n - Specifies the secret key used to verify cryptographic signatures in incoming webhooks.\n - |-\n ## FIELD BEHAVIOR\n\n This field provides the shared secret for signature verification:\n\n - REQUIRED when verify=\"hmac\" or verify=\"publickey\"\n - Contains the secret value known to both sender and receiver\n - Used with the incoming payload to validate the signature\n - Highly sensitive security credential\n - |-\n ## IMPLEMENTATION GUIDANCE\n\n ### For HMAC verification\n\n The key is used in the following verification process:\n 1. The webhook receives an incoming request with a signature\n 2. The system computes an HMAC of the request body using this key and the specified algorithm\n 3. This computed signature is compared with the signature from the request header\n 4. If they match exactly, the request is authenticated and processed\n\n ### Security best practices\n\n For maximum security:\n - Use a random, high-entropy key (32+ characters)\n - Include a mix of characters and avoid dictionary words\n - Never share this key in code repositories or logs\n - Rotate keys periodically for sensitive integrations\n - Use environment variables or secure credential storage\n\n ### Common implementations\n\n ```\n \"key\": \"whsec_3a7c4f8b2e9d1a5c6b3e7dEXAMPLE\"\n ```\n\n ```\n \"key\": \"9d1a5c6b3e7d9f2a1c5b8e3aEXAMPLE\"\n ```\n\n IMPORTANT: This key should be treated as a highly sensitive credential,\n similar to a private key or password. It should never be exposed publicly\n or logged in application logs.\n examples: [\"whsec_3a7c4f8b2e9d1a5c6b3e7dEXAMPLE\", \"9d1a5c6b3e7d9f2a1c5b8e3aEXAMPLE\"]\n header:\n type: string\n description: |-\n Name of the request header that carries the HMAC signature when `verify` is `hmac`. Must match the header name the webhook sender uses (header names are case-insensitive); requests without this header are rejected with a 401. With `provider: custom` the header value must be the bare digest in the configured `encoding`: a value with a `sha256=` prefix is rejected with a 401.\n x-celigo-ai-guidance:\n - Specifies the HTTP header name that contains the signature for HMAC verification.\n - |-\n ## FIELD BEHAVIOR\n\n This field identifies where to find the signature in incoming requests:\n\n - REQUIRED when verify=\"hmac\"\n - Must exactly match the header name used by the webhook sender\n - Case-insensitive (HTTP headers are not case-sensitive)\n - |-\n ## COMMON HEADER PATTERNS\n\n ### Platform-specific headers\n\n Many platforms use standardized header names for their signatures:\n\n ```\n \"header\": \"X-Shopify-Hmac-SHA256\" // For Shopify webhooks\n ```\n\n ```\n \"header\": \"X-Hub-Signature-256\" // For GitHub webhooks\n ```\n\n ```\n \"header\": \"Stripe-Signature\" // For Stripe webhooks\n ```\n\n ### Generic signature headers\n\n For custom implementations or less common platforms:\n\n ```\n \"header\": \"X-Webhook-Signature\" // Common generic format\n ```\n\n ```\n \"header\": \"X-Signature\" // Simplified format\n ```\n - |-\n ## IMPLEMENTATION NOTES\n\n - The system will look for this exact header name in incoming requests\n - If the header is not found, the request will be rejected with a 401 error\n - Some platforms include a prefix in the header value (e.g., \"sha256=\"). With\n provider \"custom\" it is not stripped: a prefixed signature is rejected with a 401,\n so the sender must send the bare digest\n\n IMPORTANT: This must exactly match the header name used by the webhook sender.\n If you're unsure about the correct header name, consult the sender's documentation\n or use a tool like cURL with verbose output to inspect an example request.\n examples: [\"X-Webhook-Signature\", \"X-Shopify-Hmac-SHA256\", \"Stripe-Signature\", \"X-Hub-Signature-256\"]\n tokenLocation:\n type: string\n enum: [\"body\", \"header\", \"queryParam\"]\n x-enumDescriptions:\n body: The verification token is read from the request body at `path`.\n header: The verification token is read from the request header named by `tokenHeaderName`.\n queryParam: The verification token is read from the query parameter named by `tokenQueryParamName`.\n default: body\n description: |-\n Where the verification token is found when `verify` is `token`. Each location uses a different companion field: `body` uses `path`, `header` uses `tokenHeaderName` (and `tokenHeaderScheme`), `queryParam` uses `tokenQueryParamName`.\n examples: [\"body\", \"header\", \"queryParam\"]\n path:\n type: string\n description: |-\n JSON path into the request body holding the verification token when `verify` is `token` and `tokenLocation` is `body` (e.g. `meta.token`). The value at this path must exactly match `token` or the request is rejected.\n x-celigo-ai-guidance:\n - Specifies the location of the verification token in incoming webhook requests.\n - |-\n ## FIELD BEHAVIOR\n\n This field determines where to find the token for verification:\n\n - REQUIRED when verify=\"token\"\n - Defines a JSON path to locate the token in the request body\n - For query parameters, use the appropriate path format (typically at root level)\n - |-\n ## IMPLEMENTATION PATTERNS\n\n ### Token in request body\n\n For tokens embedded in JSON payloads:\n\n ```\n \"path\": \"meta.token\" // For { \"meta\": { \"token\": \"\u003ctoken>\" } }\n ```\n\n ```\n \"path\": \"verification.key\" // For { \"verification\": { \"key\": \"\u003ctoken>\" } }\n ```\n\n ### Token at root level\n\n For tokens in the top level of the request:\n\n ```\n \"path\": \"token\" // For { \"token\": \"\u003ctoken>\", \"data\": {...} }\n ```\n\n ### Token in query parameters\n\n For tokens sent as URL query parameters, use the parameter name:\n\n ```\n \"path\": \"verify_token\" // For /webhook?verify_token=\u003ctoken>\n ```\n - |-\n ## VERIFICATION PROCESS\n\n 1. The webhook receives an incoming request\n 2. The system uses this path to extract the token value\n 3. The extracted value is compared with the configured token\n 4. If they match exactly, the request is processed\n\n IMPORTANT: The path is case-sensitive and must exactly match the structure\n of incoming requests. For query parameters, the system automatically checks\n both the body and query string using the provided path.\n examples: [\"token\", \"verification_token\", \"meta.security.token\", \"auth.key\"]\n tokenHeaderName:\n type: string\n description: Request header carrying the verification token when `verify` is `token` and `tokenLocation` is `header`.\n examples: [\"X-Webhook-Token\", \"Authorization\"]\n tokenHeaderScheme:\n type: string\n enum: [\"bearer\", \"custom\", \"none\"]\n x-enumDescriptions:\n bearer: The header value is prefixed with `Bearer ` before the token.\n custom: The header value uses the custom prefix named in `customTokenScheme`.\n none: The header value is the bare token with no scheme prefix.\n default: bearer\n description: Scheme prefixing the token in the header when `tokenLocation` is `header`.\n examples: [\"bearer\", \"none\"]\n customTokenScheme:\n type: string\n description: Custom header scheme prefix used when `tokenHeaderScheme` is `custom`.\n examples: [\"Token\"]\n tokenQueryParamName:\n type: string\n description: Query parameter carrying the verification token when `verify` is `token` and `tokenLocation` is `queryParam`.\n examples: [\"token\", \"verify_token\"]\n _httpConnectorId:\n type: string\n format: objectId\n x-celigo-refModel: httpconnectors\n description: HTTP connector backing this listener when the webhook is provided by an assistant/connector (e.g. Slack); set by the platform for connector-backed listeners.\n requestMediaType:\n type: string\n enum: [\"json\", \"xml\", \"csv\", \"urlencoded\", \"plaintext\"]\n x-enumDescriptions:\n json: Parse the incoming request body as JSON.\n xml: Parse the incoming request body as XML.\n csv: Parse the incoming request body as CSV.\n urlencoded: Parse the incoming request body as URL-encoded form data.\n plaintext: Treat the incoming request body as plain text.\n description: Overrides how the incoming request body is parsed when the provider sends a non-standard content type. When omitted, the body is parsed by its Content-Type header.\n examples: [\"json\", \"csv\"]\n pathToRecords:\n type: string\n description: |-\n JSON path into an object payload to the array whose elements become the records. When omitted, a top-level JSON array is emitted as one record per element and an object payload is emitted as a single record.\n x-celigo-ai-guidance:\n - |-\n Set this only when the records sit inside an object (`{\"data\": {\"items\": [...]}}` → `data.items`). A sender that already posts a top-level array needs nothing here; a sender that posts an object with no path configured delivers exactly one record per request — the whole object — which is also the way to make one request one record on purpose.\n - |-\n Records are paged per request at the export's `pageSize` (default 20) before any hook, filter, or downstream step runs: a 25-element array is a page of 20 records followed by a page of 5, and a `preSavePage` hook receives one page (up to `pageSize` records) per call. A hook that groups rows by order therefore groups only the rows on the same page — an order whose rows span two pages comes out as two grouped records, the second after the first was already processed. To keep a request's rows together, have the sender wrap them in an object (one record, rows under `record.\u003cpath>`) or raise `pageSize` above the largest request — never `groupByFields`, which the listener applies per page after paging.\n - |-\n Requests that arrive together, or while the listener's current run is still collecting, become additional pages of that same run; a request that arrives after the run has stopped collecting starts a new run. Nothing on the listener de-duplicates: a sender that fires the same event twice (a retry, a button clicked twice) delivers two records, and the step that writes the effect has to be made idempotent — `ignoreExisting` or a keyed upsert under the import's `idLockTemplate`.\n examples: [\"events\", \"data.items\"]\n includeParentData:\n type: boolean\n description: When true and `pathToRecords` is set, each emitted record also carries the surrounding parent fields from the payload.\n username:\n type: string\n description: |-\n Username half of the credentials when `verify` is `basic`. Incoming requests must include an `Authorization: Basic` header carrying the base64-encoded `username:password` pair. Use only over HTTPS to prevent credential interception.\n x-celigo-ai-guidance:\n - Specifies the username for webhook HTTP Basic Authentication security.\n - |-\n ## FIELD BEHAVIOR\n\n This field defines one half of the Basic Authentication credentials:\n\n - REQUIRED when verify=\"basic\"\n - Used in conjunction with the password field\n - Case-sensitive string value\n - Encoded in the standard HTTP Basic Authentication format\n - |-\n ## IMPLEMENTATION NOTES\n\n When Basic Authentication is used, the webhook requires incoming requests to include\n an Authorization header containing \"Basic \" followed by a base64-encoded string of\n \"username:password\".\n\n Example header:\n ```\n Authorization: Basic base64(webhook_user:webhook_password)\n ```\n\n ### Security considerations\n\n Basic Authentication:\n - Is widely supported by HTTP clients and servers\n - Should ONLY be used over HTTPS to prevent credential interception\n - Provides a simple authentication mechanism but without integrity verification\n - Is less secure than HMAC verification for webhook scenarios\n\n IMPORTANT: Always use strong, unique credentials rather than generic or easily\n guessable values. Basic Authentication is less secure than HMAC for webhooks\n but can be appropriate for simple scenarios or when working with systems that\n don't support more advanced verification methods.\n examples: [\"webhook_user\", \"api_client_123\", \"integration_user\"]\n password:\n type: string\n description: |-\n Password half of the credentials when `verify` is `basic`, validated together with `username` from the request's `Authorization` header. Use a strong, unique value and treat it as a sensitive credential.\n x-celigo-ai-guidance:\n - Specifies the password for webhook HTTP Basic Authentication security.\n - |-\n ## FIELD BEHAVIOR\n\n This field defines the second half of the Basic Authentication credentials:\n\n - REQUIRED when verify=\"basic\"\n - Used in conjunction with the username field\n - Case-sensitive string value\n - Encoded in the standard HTTP Basic Authentication format\n - |-\n ## IMPLEMENTATION NOTES\n\n This password is combined with the username and encoded in base64 format for\n the HTTP Authorization header. The webhook verifies that incoming requests contain\n the correct encoded credentials before processing them.\n\n ### Security best practices\n\n For maximum security:\n - Use a strong, randomly generated password (16+ characters)\n - Include a mix of uppercase, lowercase, numbers, and special characters\n - Don't reuse passwords from other systems\n - Avoid dictionary words or predictable patterns\n - Rotate passwords periodically for sensitive integrations\n\n IMPORTANT: This password should be treated as a sensitive credential.\n Never share it in public repositories, documentation, or logs. Always use\n HTTPS for webhooks using Basic Authentication to prevent credential interception.\n examples: [\"xC7!rTp2@bN9$EXAMPLE\", \"webhook_secure_pass_123\"]\n successStatusCode:\n type: integer\n description: |-\n HTTP status code returned to the caller after successful processing; must be a valid 2xx code. The default 204 returns no response body and causes `successBody` to be ignored — set 200 or 202 when the caller needs a response body or expects a specific code.\n x-celigo-ai-guidance:\n - Specifies the HTTP status code sent back to webhook callers after successful processing.\n - |-\n ## FIELD BEHAVIOR\n\n This field controls the HTTP response status code:\n\n - OPTIONAL: Defaults to 204 (No Content) if not specified\n - Affects how webhook callers interpret the success response\n - Must be a valid HTTP status code in the 2xx range\n - |-\n ## COMMON STATUS CODES\n\n ### 204 No Content (Default)\n ```\n \"successStatusCode\": 204\n ```\n - Returns no response body\n - Most efficient option as it minimizes response size\n - Appropriate when the caller doesn't need confirmation details\n - Automatically disables successBody (even if specified)\n\n ### 200 OK\n ```\n \"successStatusCode\": 200\n ```\n - Standard success response\n - Allows returning a response body with details\n - Most widely used and recognized success code\n - Compatible with all HTTP clients\n\n ### 202 Accepted\n ```\n \"successStatusCode\": 202\n ```\n - Indicates request was accepted for processing but may not be complete\n - Appropriate for asynchronous processing scenarios\n - Signals that the webhook was received but full processing is pending\n - |-\n ## IMPLEMENTATION CONSIDERATIONS\n\n The appropriate status code depends on your webhook caller's expectations:\n\n - Some systems require specific status codes to consider the delivery successful\n - If the caller retries on anything other than 2xx, use 200 or 202\n - If the caller needs confirmation details, use 200 with a response body\n - If efficiency is paramount, use 204 (default)\n\n IMPORTANT: When using 204 No Content, any successBody configuration will be ignored\n as this status code specifically indicates no response body is being returned.\n default: 204\n examples: [200, 201, 202, 204]\n successBody:\n type: string\n description: |-\n Response body returned to the caller after successful processing; ignored when `successStatusCode` is 204. Content type is set by `successMediaType`, and the value can be static text or structured JSON/XML, including handlebars expressions for dynamic values.\n x-celigo-ai-guidance:\n - Specifies the HTTP response body sent back to webhook callers after successful processing.\n - |-\n ## FIELD BEHAVIOR\n\n This field controls the content returned to the webhook caller:\n\n - OPTIONAL: Defaults to empty (no body) if not specified\n - Ignored when successStatusCode is 204 (No Content)\n - Content type is determined by the successMediaType field\n - Can contain static text or structured data (JSON, XML)\n - |-\n ## IMPLEMENTATION PATTERNS\n\n ### Simple acknowledgment\n ```\n \"successBody\": \"OK\"\n ```\n - Minimal plaintext response\n - Confirms receipt without details\n - Most efficient for basic acknowledgment\n\n ### Structured response (JSON)\n ```\n \"successBody\": \"{\\\"success\\\":true,\\\"message\\\":\\\"Webhook received\\\"}\"\n ```\n - Provides structured data about the result\n - Can include more detailed status information\n - Compatible with programmatic processing by the caller\n - Remember to escape quotes in JSON strings\n\n ### Custom confirmation\n ```\n \"successBody\": \"{\\\"status\\\":\\\"received\\\",\\\"timestamp\\\":\\\"{{currentDateTime}}\\\"}\"\n ```\n - Can include dynamic values using handlebars templates\n - Useful for providing receipt confirmation with metadata\n - |-\n ## RESPONSE FLOW\n\n The response body is sent after the webhook payload has been:\n 1. Received and authenticated\n 2. Validated against any configured requirements\n 3. Accepted for processing by the system\n\n IMPORTANT: The successBody will only be returned if successStatusCode is NOT 204.\n If you want to return a body, make sure to set successStatusCode to 200, 201, or 202.\n examples: [\"{\\\"success\\\":true}\", \"OK\", \"{\\\"status\\\":\\\"received\\\",\\\"timestamp\\\":\\\"{{currentDateTime}}\\\"}\"]\n x-celigo-agent:\n - handlebars\n successMediaType:\n type: string\n description: |-\n Sets the Content-Type header on successful webhook responses. Only takes effect when a `successBody` is returned (a status code other than 204), and must match the actual format of that body.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field controls how the response body is interpreted:\n\n - OPTIONAL: Defaults to \"json\" if not specified\n - Only relevant when returning a successBody and not using status code 204\n - Determines the Content-Type header in the HTTP response\n - Must be consistent with the actual format of the successBody\n - |-\n ## MEDIA TYPE OPTIONS\n\n ### JSON (Default)\n ```\n \"successMediaType\": \"json\"\n ```\n - Sets Content-Type: application/json\n - Use when successBody contains valid JSON\n - Most common for API responses\n - Allows structured data that clients can parse programmatically\n\n ### XML\n ```\n \"successMediaType\": \"xml\"\n ```\n - Sets Content-Type: application/xml\n - Use when successBody contains valid XML\n - Necessary for systems expecting XML responses\n - Less common in modern APIs but still used in some enterprise systems\n\n ### Plain Text\n ```\n \"successMediaType\": \"plaintext\"\n ```\n - Sets Content-Type: text/plain\n - Use for simple string responses\n - Most compatible option for basic acknowledgments\n - Appropriate when successBody is just \"OK\" or similar\n - |-\n ## IMPLEMENTATION CONSIDERATIONS\n\n - The media type must match the actual content format in successBody\n - If returning JSON in successBody, use \"json\" (most common)\n - If returning a simple text acknowledgment, use \"plaintext\"\n - If the caller specifically requires XML, use \"xml\"\n\n IMPORTANT: When successStatusCode is 204 (No Content), this field has no effect\n since no body is returned, and therefore no Content-Type is needed.\n default: \"json\"\n enum: [\"json\", \"xml\", \"plaintext\"]\n x-enumDescriptions:\n json: \"Sets the success response Content-Type to application/json.\"\n xml: \"Sets the success response Content-Type to application/xml.\"\n plaintext: \"Sets the success response Content-Type to text/plain.\"\n examples: [\"json\", \"plaintext\", \"xml\"]\n successResponseHeaders:\n type: array\n description: |-\n Custom headers added to successful webhook responses — for example CORS headers or correlation IDs. Headers defined here take precedence over automatically set headers such as Content-Type, and values support handlebars expressions for dynamic content.\n x-celigo-ai-guidance:\n - Defines custom HTTP headers to include in successful webhook responses.\n - |-\n ## FIELD BEHAVIOR\n\n This field allows additional HTTP headers in the response:\n\n - OPTIONAL: If omitted, only standard headers are included\n - Each entry defines a name/value pair for a single header\n - Applied to all successful responses (regardless of status code)\n - Can override standard headers like Content-Type\n - |-\n ## IMPLEMENTATION PATTERNS\n\n ### Standard use cases\n\n Custom headers are useful for:\n - Providing metadata about the response\n - Enabling CORS for browser-based webhook callers\n - Including tracking or correlation IDs\n - Adding custom security headers\n\n ### Common header examples\n\n CORS support:\n ```json\n [\n {\"name\": \"Access-Control-Allow-Origin\", \"value\": \"*\"},\n {\"name\": \"Access-Control-Allow-Methods\", \"value\": \"POST, OPTIONS\"}\n ]\n ```\n\n Request tracking:\n ```json\n [\n {\"name\": \"X-Request-ID\", \"value\": \"{{jobId}}\"},\n {\"name\": \"X-Webhook-Received\", \"value\": \"{{currentDateTime}}\"}\n ]\n ```\n\n Custom application headers:\n ```json\n [\n {\"name\": \"X-API-Version\", \"value\": \"1.0\"},\n {\"name\": \"X-Processing-Status\", \"value\": \"accepted\"}\n ]\n ```\n - |-\n ## TECHNICAL DETAILS\n\n - Header names are case-insensitive as per HTTP specification\n - Some headers like Content-Type can be set via other fields (successMediaType)\n - Headers defined here take precedence over automatically set headers\n - The values can contain handlebars expressions for dynamic content\n\n IMPORTANT: Be careful when setting security-related headers like\n Access-Control-Allow-Origin, as improper values could create security vulnerabilities.\n items:\n type: object\n properties:\n name:\n type: string\n description: Name of the header to set on successful webhook responses; headers defined here take precedence over automatically set headers such as Content-Type.\n examples: [\"Content-Type\", \"X-Webhook-Received\", \"Access-Control-Allow-Origin\"]\n value:\n type: string\n description: Value sent for the header; supports handlebars expressions for dynamic content.\n examples: [\"application/json\", \"true\", \"*\"]\n x-celigo-agent:\n - handlebars\n challengeResponseHeaders:\n type: array\n description: |-\n Custom headers returned for webhook subscription verification (challenge) requests, which providers send before delivering real events. Required header values vary by provider — consult the provider's documentation, since incorrect challenge headers can prevent the subscription from being verified.\n x-celigo-ai-guidance:\n - Defines custom HTTP headers to include in webhook challenge responses.\n - |-\n ## FIELD BEHAVIOR\n\n This field configures headers for subscription verification:\n\n - OPTIONAL: If omitted, only standard headers are included\n - Only used for webhook verification/challenge requests\n - Each entry defines a name/value pair for a single header\n - Particularly important for platforms requiring specific verification headers\n - |-\n ## CHALLENGE VERIFICATION CONTEXT\n\n Many webhook providers implement a verification process:\n 1. Before sending real events, they send a \"challenge\" request\n 2. The webhook must respond with specific headers and/or body content\n 3. Only after successful verification will real webhook events be sent\n\n This field allows customizing the headers sent during this verification step.\n - |-\n ## COMMON PATTERNS BY PLATFORM\n\n ### Facebook/Instagram\n ```json\n [\n {\"name\": \"Content-Type\", \"value\": \"text/plain\"}\n ]\n ```\n\n ### Slack\n ```json\n [\n {\"name\": \"Content-Type\", \"value\": \"application/json\"}\n ]\n ```\n\n ### Custom implementations\n ```json\n [\n {\"name\": \"X-Challenge-Response\", \"value\": \"passed\"},\n {\"name\": \"X-Verification-Status\", \"value\": \"success\"}\n ]\n ```\n\n IMPORTANT: The specific headers required vary by platform. Consult the webhook\n provider's documentation for the exact verification requirements. Incorrect challenge\n response headers may prevent successful webhook subscription.\n items:\n type: object\n properties:\n name:\n type: string\n description: Name of the header to set on challenge (subscription verification) responses.\n examples: [\"Content-Type\", \"X-Challenge-Response\", \"X-Verification-Status\"]\n value:\n type: string\n description: Value sent for the header; consult the provider's documentation for required challenge header values.\n examples: [\"application/json\", \"passed\", \"success\"]\n challengeSuccessBody:\n type: string\n description: |-\n Response body returned for webhook subscription verification (challenge) requests. Many providers require echoing back a challenge value from the request, which handlebars expressions can access — for example `{{hub.challenge}}` (Facebook/Instagram) or `{\"challenge\":\"{{challenge}}\"}` (Slack). An incorrect challenge response prevents the subscription from being verified.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field defines the verification response content:\n\n - OPTIONAL: If omitted, a default empty response is sent\n - Only used for webhook subscription verification requests\n - Content type is determined by the challengeSuccessMediaType field\n - Often needs to contain specific values expected by the webhook provider\n - |-\n ## VERIFICATION PATTERNS BY PLATFORM\n\n Different webhook providers implement different verification mechanisms:\n\n ### Facebook/Instagram\n \"challengeSuccessBody\": \"{{hub.challenge}}\"\n ```\n - Must echo back the challenge value sent in the request\n - Uses handlebars expression to access the challenge parameter\n\n ### Slack\n ```\n \"challengeSuccessBody\": \"{\\\"challenge\\\":\\\"{{challenge}}\\\"}\"\n ```\n - Returns the challenge value in a JSON structure\n - Required for Slack's Events API verification\n\n ### Generic challenge-response\n ```\n \"challengeSuccessBody\": \"{\\\"verified\\\":true,\\\"timestamp\\\":\\\"{{currentDateTime}}\\\"}\"\n ```\n - Simple confirmation response for custom implementations\n - Can include additional metadata as needed\n\n ## IMPLEMENTATION CONSIDERATIONS\n\n - The exact format is dictated by the webhook provider's requirements\n - Some platforms require echoing back specific request parameters\n - Others require a structured response with specific fields\n - Handlebars expressions ({{variable}}) can access request data\n\n IMPORTANT: Incorrect challenge responses will prevent webhook subscription verification.\n Always consult the webhook provider's documentation for exact requirements.\n examples: [\"{\\\"challenge\\\":\\\"{{challenge}}\\\"}\", \"{{hub.challenge}}\", \"challenge-passed\"]\n challengeSuccessStatusCode:\n type: integer\n description: |-\n HTTP status code returned for webhook subscription verification (challenge) requests. Most providers expect the default 200; change it only when the provider's verification explicitly requires a different code.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field controls the verification response status:\n\n - OPTIONAL: Defaults to 200 (OK) if not specified\n - Only used for webhook subscription verification requests\n - Must match what the webhook provider expects for successful verification\n - Most platforms require a 200 OK response\n - |-\n ## COMMON STATUS CODES FOR VERIFICATION\n\n ### 200 OK (Default)\n ```\n \"challengeSuccessStatusCode\": 200\n ```\n - Standard success response\n - Most webhook platforms expect this status code\n - Generally the safest option for verification\n\n ### 201 Created\n ```\n \"challengeSuccessStatusCode\": 201\n ```\n - Used by some systems to indicate subscription was created\n - Less common for verification but used in some custom implementations\n - |-\n ## PLATFORM-SPECIFIC REQUIREMENTS\n\n Most major webhook providers require specific status codes:\n\n - Facebook/Instagram: 200\n - Slack: 200\n - GitHub: 200\n - Shopify: 200\n - Stripe: 200\n\n IMPORTANT: Using the wrong status code will cause the verification to fail.\n If you're unsure, keep the default 200 status code, as it's the most widely\n accepted for webhook verifications.\n default: 200\n examples: [200, 201, 202]\n challengeSuccessMediaType:\n type: string\n description: |-\n Sets the Content-Type header on challenge responses. Must match both the format of `challengeSuccessBody` and the provider's requirements — for example, Slack expects `json` while Facebook/Instagram verification expects `plaintext`.\n x-celigo-ai-guidance:\n - |-\n ## FIELD BEHAVIOR\n\n This field controls the challenge response format:\n\n - OPTIONAL: Defaults to \"json\" if not specified\n - Only used for webhook subscription verification requests\n - Determines the Content-Type header in the verification response\n - Must match the format of the challengeSuccessBody content\n - |-\n ## COMMON MEDIA TYPES FOR VERIFICATION\n\n ### JSON (Default)\n ```\n \"challengeSuccessMediaType\": \"json\"\n ```\n - Sets Content-Type: application/json\n - Required by Slack and many modern webhook providers\n - Use when returning structured verification data\n\n ### Plain Text\n ```\n \"challengeSuccessMediaType\": \"plaintext\"\n ```\n - Sets Content-Type: text/plain\n - Required by Facebook/Instagram webhook verification\n - Use when the challenge response is a simple string\n\n ### XML\n ```\n \"challengeSuccessMediaType\": \"xml\"\n ```\n - Sets Content-Type: application/xml\n - Less common but used by some enterprise systems\n - Use only when the webhook provider specifically requires XML\n - |-\n ## PLATFORM-SPECIFIC REQUIREMENTS\n\n - Facebook/Instagram: plaintext (when echoing hub.challenge)\n - Slack: json (for Events API verification)\n - Most modern APIs: json\n\n IMPORTANT: The media type must match both the format of your challengeSuccessBody\n and the requirements of the webhook provider. Mismatched content types can cause\n verification to fail even if the response body is correct.\n default: \"json\"\n enum: [\"json\", \"xml\", \"plaintext\"]\n x-enumDescriptions:\n json: \"Sets the challenge response Content-Type to application/json.\"\n xml: \"Sets the challenge response Content-Type to application/xml.\"\n plaintext: \"Sets the challenge response Content-Type to text/plain.\"\n examples: [\"json\", \"plaintext\", \"xml\"]\n # hmac companions are form-required only for form-managed webhooks; connector-backed\n # webhooks (_httpConnectorId set) store the signing secret per the connector definition.\n if:\n properties: { verify: { const: hmac } }\n required: [verify]\n not:\n required: [_httpConnectorId]\n then: { required: [key, algorithm, encoding, header] }\n else:\n if: { properties: { verify: { const: token } }, required: [verify] }\n then:\n # tokenLocation is optional: it was already declared with `default: body` before\n # the drift campaign, so requiring it would demand a defaulted field. But every\n # location needs its companion — including the implicit body default, hence the\n # final else requires `path` when tokenLocation is omitted.\n required: [token]\n if: { properties: { tokenLocation: { const: body } }, required: [tokenLocation] }\n then: { required: [path] }\n else:\n if: { properties: { tokenLocation: { const: header } }, required: [tokenLocation] }\n then: { required: [tokenHeaderName] }\n else:\n if: { properties: { tokenLocation: { const: queryParam } }, required: [tokenLocation] }\n then: { required: [tokenQueryParamName] }\n else: { required: [path] }\n else:\n if: { properties: { verify: { const: basic } }, required: [verify] }\n then: { required: [username, password] }\n else:\n if: { properties: { verify: { const: secret_url } }, required: [verify] }\n then: { required: [token] }\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":56102,"content_sha256":"39ad6184e0968ac4c2dcab65bf5332310361fc3e587ab743faa1353c108f230b"},{"filename":"references/schemas/wrapper.yml","content":"Wrapper:\n type: object\n description: |-\n Configuration for Wrapper exports, which delegate data retrieval to custom connector code\n (typically a stack-hosted function) rather than a built-in adaptor. Required when the\n _connectionId field references a wrapper connection.\n required:\n - function\n properties:\n function:\n type: string\n x-celigo-ui-override: >-\n Required by the wrapper export form (wrapper.function is required:true). Encoded to mirror\n the form so builders produce connectable configurations.\n description: |-\n Name of the function the wrapper invokes to retrieve records. Must match a callable\n function in the wrapper's execution context; names are case-sensitive.\n examples: [\"getOrdersFromWooCommerce\", \"processItemShipmentExport\", \"exportData\"]\n configuration:\n # Empty-array stubs accepted: the server writes the empty-array form when no\n # settings exist. A POPULATED array is still rejected (deliberately left failing).\n oneOf:\n - type: object\n additionalProperties: true\n - type: array\n maxItems: 0\n description: |-\n Free-form settings passed to the wrapper function at runtime (connector-specific keys such\n as method, apiVersion, headers, relativePath, or body). Structure is defined by the wrapper\n code, not by this schema. May be stored as an empty array when no settings exist.\n","content_type":"application/yaml; charset=utf-8","language":"yaml","size":1452,"content_sha256":"8ede14149b0644db12eab35e000689427bb20a7b77b5bb10c2f7b2571902c7ec"}],"content_json":{"type":"doc","content":[{"type":"paragraph","content":[{"text":"\u003c!-- TIER:1 -->","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Configuring Exports","type":"text"}]},{"type":"paragraph","content":[{"text":"An export is the ","type":"text"},{"text":"data source","type":"text","marks":[{"type":"strong"}]},{"text":" in a Celigo integration. It connects to an external system and pulls data into the pipeline. Exports serve two roles:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Source","type":"text","marks":[{"type":"strong"}]},{"text":" -- the starting point that fetches the primary batch of records","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Lookup","type":"text","marks":[{"type":"strong"}]},{"text":" -- a mid-flow enrichment step (","type":"text"},{"text":"isLookup: true","type":"text","marks":[{"type":"code_inline"}]},{"text":") that fetches additional data per-record during processing","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Both roles are used across flows, APIs, and tools.","type":"text"}]},{"type":"paragraph","content":[{"text":"Beyond fetching data, exports also handle post-retrieval processing before records enter the pipeline:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Output filter","type":"text","marks":[{"type":"strong"}]},{"text":" -- expression-based filtering to skip records that don't match criteria","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Transform","type":"text","marks":[{"type":"strong"}]},{"text":" -- Transformation 2.0 expression rules to reshape/flatten response data before mapping","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"preSavePage hook","type":"text","marks":[{"type":"strong"}]},{"text":" -- JavaScript processing on the full page of records before they enter the pipeline; the only stage that can change how many records a page carries (split one record into many, or merge several)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"One-to-many","type":"text","marks":[{"type":"strong"}]},{"text":" -- when used as a lookup, fan out a child array already on the input record. Set ","type":"text"},{"text":"oneToMany: true","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"pathToMany","type":"text","marks":[{"type":"code_inline"}]},{"text":" to that array path so each child triggers a separate lookup. Once fanned out, the array element itself is the record, and the parent (without the array) is ","type":"text"},{"text":"record._PARENT","type":"text","marks":[{"type":"code_inline"}]},{"text":". This is not ","type":"text"},{"text":"resourcePath","type":"text","marks":[{"type":"code_inline"}]},{"text":", which locates records in an API response -- see ","type":"text"},{"text":"One-to-many fan-out","type":"text","marks":[{"type":"link","attrs":{"href":"#one-to-many-fan-out----the-array-element-is-the-record","title":null}}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Response mapping","type":"text","marks":[{"type":"strong"}]},{"text":" -- when used as a lookup, extract fields from the lookup response back into the record. Configured on the flow's ","type":"text"},{"text":"pageProcessors[]","type":"text","marks":[{"type":"code_inline"}]},{"text":" entry, but planned when building the lookup export. The response contains a ","type":"text"},{"text":"data","type":"text","marks":[{"type":"code_inline"}]},{"text":" array and an ","type":"text"},{"text":"errors","type":"text","marks":[{"type":"code_inline"}]},{"text":" array. Use ","type":"text"},{"text":"data[0].fieldName","type":"text","marks":[{"type":"code_inline"}]},{"text":" when you expect a single result (e.g., fetching one order by ID). When multiple results are expected, map the whole array with ","type":"text"},{"text":"extract: \"data\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" (downstream steps then read it as a normal JSON array, or fan out over it with one-to-many), or use a ","type":"text"},{"text":"lists","type":"text","marks":[{"type":"code_inline"}]},{"text":" entry with ","type":"text"},{"text":"data[*].fieldName","type":"text","marks":[{"type":"code_inline"}]},{"text":" extracts to build a reshaped array. Do not put ","type":"text"},{"text":"data[*].fieldName","type":"text","marks":[{"type":"code_inline"}]},{"text":" in a top-level ","type":"text"},{"text":"fields[].extract","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- the wildcard is silently ignored there and nothing is merged. See ","type":"text"},{"text":"writing-mappings > Response Mapping Reference","type":"text","marks":[{"type":"link","attrs":{"href":"../writing-mappings/SKILL.md#response-mapping-reference-transformation-10","title":null}}]},{"text":". Response mapping uses Transformation 1.0 syntax (extract/generate pairs), not the newer expression-based transforms","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"postResponseMap hook","type":"text","marks":[{"type":"strong"}]},{"text":" -- JavaScript processing after response mapping merges the lookup response back into the record. Configured on the flow's ","type":"text"},{"text":"pageProcessors[]","type":"text","marks":[{"type":"code_inline"}]},{"text":" entry, but planned when building the lookup export","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Trace key","type":"text","marks":[{"type":"strong"}]},{"text":" -- each record's identity downstream: imports use it to match retries to their errors and as their default lock key, and it never de-duplicates. By default it's the first of the top-level fields ","type":"text"},{"text":"id","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"internalId","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"_id","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"internal_id","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"identifier","type":"text","marks":[{"type":"code_inline"}]},{"text":" (exact names, in a few capitalizations) that the record has -- never a nested field. Set ","type":"text"},{"text":"traceKeyTemplate","type":"text","marks":[{"type":"code_inline"}]},{"text":" when the identity is nested or composite (","type":"text"},{"text":"{{join \"_\" record.customerId record.orderId}}","type":"text","marks":[{"type":"code_inline"}]},{"text":", without the ","type":"text"},{"text":"record.","type":"text","marks":[{"type":"code_inline"}]},{"text":" prefix when a transform reshaped the record first). Keys are capped at 256 characters","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Export Execution Pipeline","type":"text"}]},{"type":"paragraph","content":[{"text":"When a flow runs, each export executes this pipeline in strict order:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"API request / query / file read","type":"text","marks":[{"type":"strong"}]},{"text":" -- fetches raw data from the external system. On a file export an optional ","type":"text"},{"text":"preParse","type":"text","marks":[{"type":"code_inline"}]},{"text":" hook then gets the raw file text (","type":"text"},{"text":"preParseData","type":"text","marks":[{"type":"code_inline"}]},{"text":", with ","type":"text"},{"text":"fileMeta","type":"text","marks":[{"type":"code_inline"}]},{"text":") and returns it for the parser; a throw there fails the run","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Response parsing","type":"text","marks":[{"type":"strong"}]},{"text":" -- ","type":"text"},{"text":"resourcePath","type":"text","marks":[{"type":"code_inline"}]},{"text":" extracts the record array from the response body or file (e.g., ","type":"text"},{"text":"http.response.resourcePath","type":"text","marks":[{"type":"code_inline"}]},{"text":" for HTTP, ","type":"text"},{"text":"file.json.resourcePath","type":"text","marks":[{"type":"code_inline"}]},{"text":" for JSON files, XPath for XML)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Grouping","type":"text","marks":[{"type":"strong"}]},{"text":" (optional) -- rows that share a key become one record, an array of those rows: ","type":"text"},{"text":"groupByFields","type":"text","marks":[{"type":"code_inline"}]},{"text":", or Group rows on a NetSuite saved search (see ","type":"text"},{"text":"Grouped records","type":"text","marks":[{"type":"link","attrs":{"href":"#grouped-records----one-record-many-rows","title":null}}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Transformation","type":"text","marks":[{"type":"strong"}]},{"text":" (optional) -- ","type":"text"},{"text":"transform","type":"text","marks":[{"type":"code_inline"}]},{"text":" reshapes individual records after extraction (Transformation 2.0). On an HTTP export it also sees the response's headers, by lowercase name, so it can carry a header into the records: ","type":"text"},{"text":"{{{headers.[x-total-count]}}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" as an extract, or ","type":"text"},{"text":"$.headers['x-total-count']","type":"text","marks":[{"type":"code_inline"}]},{"text":" with ","type":"text"},{"text":"inputContext: \"envelope\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" (live-verified in a preview and a test run, 2026-10-22). Use triple braces: double braces HTML-escape the value","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Output filter","type":"text","marks":[{"type":"strong"}]},{"text":" (optional) -- discards records that don't match filter expression rules","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"preSavePage hook","type":"text","marks":[{"type":"strong"}]},{"text":" (optional) -- JavaScript processing on the full page of records","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Key distinction:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"resourcePath","type":"text","marks":[{"type":"code_inline"}]},{"text":" tells the export WHERE to find records in the response. Transforms reshape WHAT each record looks like after extraction. When a user says \"extract records from X\" or \"treat each X as a separate record\", that's almost always a ","type":"text"},{"text":"resourcePath","type":"text","marks":[{"type":"code_inline"}]},{"text":" change, not a transform. Use transforms when you need to flatten nested objects, rename fields, or restructure individual records.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Three Categories of Export","type":"text"}]},{"type":"paragraph","content":[{"text":"Not all exports work the same way. Before building, understand which category you need:","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Listeners","type":"text"}]},{"type":"paragraph","content":[{"text":"Receive data pushed to Celigo from an external system. No polling, no scheduling -- the source system sends data when events happen.","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"WebhookExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" with ","type":"text"},{"text":"type: \"webhook\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- inbound HTTP listener (no connection required). URL, signature checks and responses: ","type":"text"},{"text":"Webhook listeners","type":"text","marks":[{"type":"link","attrs":{"href":"#webhook-listeners----type-url-signature-responses","title":null}}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"AS2Export","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- AS2 EDI file reception","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Distributed exports (","type":"text"},{"text":"type: \"distributed\"","type":"text","marks":[{"type":"code_inline"}]},{"text":") -- real-time listeners for NetSuite and Salesforce. NetSuite: saving the export deploys a Celigo user event script on the record type (the account needs the Celigo SuiteApp or bundle). Salesforce: saving creates nothing in Salesforce -- the org needs the Celigo managed package plus an Apex trigger on each object that calls it. Integration Apps create their triggers at installation; for a custom flow the customer deploys the trigger, starting from the code the export page shows under Required trigger. ","type":"text"},{"text":"celigo connections ping --distributed \u003cconnectionId>","type":"text","marks":[{"type":"code_inline"}]},{"text":" checks that the package or SuiteApp responds; its ","type":"text"},{"text":"success: false","type":"text","marks":[{"type":"code_inline"}]},{"text":" (still HTTP 200) also comes back for an offline connection, so read ","type":"text"},{"text":"resBody","type":"text","marks":[{"type":"code_inline"}]},{"text":" before telling the user to install anything.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Change data capture (","type":"text"},{"text":"type: \"stream\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" plus a ","type":"text"},{"text":"cdc","type":"text","marks":[{"type":"code_inline"}]},{"text":" object) -- streams database changes as they happen: MongoDB change streams (","type":"text"},{"text":"MongodbExport","type":"text","marks":[{"type":"code_inline"}]},{"text":"), and SQL Server CDC or PostgreSQL logical replication (","type":"text"},{"text":"RDBMSExport","type":"text","marks":[{"type":"code_inline"}]},{"text":"). A CDC export has no query. The RDBMS form watches ","type":"text"},{"text":"rdbms.tables","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"database.schema.table","type":"text","marks":[{"type":"code_inline"}]},{"text":" on SQL Server, ","type":"text"},{"text":"schema.table","type":"text","marks":[{"type":"code_inline"}]},{"text":" on PostgreSQL) or, on PostgreSQL only, ","type":"text"},{"text":"cdc.publicationName","type":"text","marks":[{"type":"code_inline"}]},{"text":". The save checks the export and the connection's database type, not the database: one whose own change capture isn't set up (PostgreSQL logical replication, SQL Server CDC on the database and each table, a MongoDB replica set) still saves, so when a CDC listener delivers nothing, check the database side first instead -- set one, not both. Every CDC export must resolve a snapshot mode (","type":"text"},{"text":"cdc.snapshotMode","type":"text","marks":[{"type":"code_inline"}]},{"text":", or a ","type":"text"},{"text":"snapshot.mode","type":"text","marks":[{"type":"code_inline"}]},{"text":" entry in ","type":"text"},{"text":"cdc.properties","type":"text","marks":[{"type":"code_inline"}]},{"text":"), and a MongoDB one a capture mode too (","type":"text"},{"text":"cdc.captureMode","type":"text","marks":[{"type":"code_inline"}]},{"text":", or a ","type":"text"},{"text":"capture.mode","type":"text","marks":[{"type":"code_inline"}]},{"text":" entry). See ","type":"text"},{"text":"cdc.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/cdc.yml","title":null}}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Topic listeners (","type":"text"},{"text":"adaptorType: \"TopicExport\"","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"type: \"stream\"","type":"text","marks":[{"type":"code_inline"}]},{"text":") -- consume what is published to the Event Streams topic in ","type":"text"},{"text":"topic._topicId","type":"text","marks":[{"type":"code_inline"}]},{"text":". No ","type":"text"},{"text":"_connectionId","type":"text","marks":[{"type":"code_inline"}]},{"text":" and no ","type":"text"},{"text":"cdc","type":"text","marks":[{"type":"code_inline"}]},{"text":" object: with a connection, the save validates the export as that connection's adaptor and fails. Without ","type":"text"},{"text":"type: \"stream\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" the export saves but runs as a batch export and never starts on a publish; a PATCH can't add the type later, so update the full document with it, then disable and re-enable the flow. A topic listener can't be a lookup. See ","type":"text"},{"text":"topic.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/topic.yml","title":null}}]},{"text":".","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"type: \"stream\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" is accepted only on those three adaptors (","type":"text"},{"text":"MongodbExport","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"RDBMSExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" on SQL Server or PostgreSQL, ","type":"text"},{"text":"TopicExport","type":"text","marks":[{"type":"code_inline"}]},{"text":"); every other adaptor rejects the save with ","type":"text"},{"text":"unsupported_export_type","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"A listener is switched by its flow.","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"netsuite.distributed.disabled","type":"text","marks":[{"type":"code_inline"}]},{"text":" is maintained by the platform; pause a NetSuite listener by disabling its flow. A topic subscription or a NetSuite listener's deployment becomes active when the flow goes from disabled to enabled -- a flow created already enabled doesn't activate it. Create the flow disabled and enable it afterwards. After repairing a listener export, or when a NetSuite listener stops firing, disable and re-enable the flow (","type":"text"},{"text":"celigo flows set \u003cflowId> disabled=true","type":"text","marks":[{"type":"code_inline"}]},{"text":", then ","type":"text"},{"text":"disabled=false","type":"text","marks":[{"type":"code_inline"}]},{"text":"). A webhook listener has no such step: it accepts requests as soon as a flow it feeds is enabled, a flow created enabled included, and refuses them with 403 while every such flow is disabled.","type":"text"}]},{"type":"paragraph","content":[{"text":"When to use:","type":"text","marks":[{"type":"strong"}]},{"text":" The source system supports outbound webhooks, push notifications, or change data capture, or the records arrive on an Event Streams topic, and you want real-time processing.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"File Transfers","type":"text"}]},{"type":"paragraph","content":[{"text":"Read files from a remote location, then either parse them into records or transfer them as blobs.","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"FTPExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"S3Export","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"FileSystemExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- fetch files from FTP/SFTP, S3, or local filesystem","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"HTTPExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" with ","type":"text"},{"text":"http.type: \"file\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- read the files in a cloud storage folder (Google Drive, Box, Dropbox, Azure Blob Storage, Google Cloud Storage, Azure Data Lake Storage Gen2, Celigo Storage). On these file-storage connectors it's folder mode: ","type":"text"},{"text":"relativeURI","type":"text","marks":[{"type":"code_inline"}]},{"text":" is a folder path the connector resolves, not an API endpoint -- an API URL there saves cleanly and fails at runtime with \"Directory is not found\". A folder picked in the UI's folder browser is stored in ","type":"text"},{"text":"file.directory","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"pathMode: \"directoryId\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" plus the provider's folder ","type":"text"},{"text":"id","type":"text","marks":[{"type":"code_inline"}]},{"text":"); on the drive-style providers that id alone selects the folder and ","type":"text"},{"text":"relativeURI","type":"text","marks":[{"type":"code_inline"}]},{"text":" isn't read, so retarget the export by changing ","type":"text"},{"text":"file.directory.id","type":"text","marks":[{"type":"code_inline"}]},{"text":" (Google Cloud Storage keeps the bucket in ","type":"text"},{"text":"storageRootId","type":"text","marks":[{"type":"code_inline"}]},{"text":" and the object prefix in ","type":"text"},{"text":"id","type":"text","marks":[{"type":"code_inline"}]},{"text":"). The connector handles auth; the ","type":"text"},{"text":"file{}","type":"text","marks":[{"type":"code_inline"}]},{"text":" config handles parsing. To fetch one file per record from an API endpoint instead, use a blob lookup (","type":"text"},{"text":"type: \"blob\"","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"isLookup: true","type":"text","marks":[{"type":"code_inline"}]},{"text":") without ","type":"text"},{"text":"http.type: \"file\"","type":"text","marks":[{"type":"code_inline"}]},{"text":", and response-map its ","type":"text"},{"text":"data.0.blobKey","type":"text","marks":[{"type":"code_inline"}]},{"text":" to ","type":"text"},{"text":"blobKey","type":"text","marks":[{"type":"code_inline"}]},{"text":", the field the upload import's ","type":"text"},{"text":"blobKeyPath","type":"text","marks":[{"type":"code_inline"}]},{"text":" names.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NetSuiteExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" with ","type":"text"},{"text":"netsuite.type: \"file\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- fetch and parse files (CSV, JSON, XLSX, XML, EDI) from the NetSuite file cabinet","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Parsed mode (","type":"text"},{"text":"file.output: \"records\"","type":"text","marks":[{"type":"code_inline"}]},{"text":") -- CSV, XML, JSON, XLSX, EDI files are parsed into individual records. Fixed-width files, and files whose rows carry a type marker (","type":"text"},{"text":"HDR","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"DTL","type":"text","marks":[{"type":"code_inline"}]},{"text":"), parse through a file definition (","type":"text"},{"text":"file.type: \"filedefinition\"","type":"text","marks":[{"type":"code_inline"}]},{"text":"); see ","type":"text"},{"text":"building-b2b","type":"text","marks":[{"type":"link","attrs":{"href":"../building-b2b/SKILL.md#5-create-file-definitions","title":null}}]},{"text":", since a few rule mistakes drop rows without an error. On a file-provider export or lookup, each record's source file is also ","type":"text"},{"text":"fileMeta","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- exactly ","type":"text"},{"text":"fileName","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"fileSize","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"lastModifiedTime","type":"text","marks":[{"type":"code_inline"}]},{"text":" (epoch milliseconds) -- read as ","type":"text"},{"text":"{{fileMeta.fileName}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" in Handlebars or ","type":"text"},{"text":"$.fileMeta.\u003cfield>","type":"text","marks":[{"type":"code_inline"}]},{"text":" in an envelope-mode transform. Only file-provider steps populate it at run time, even where an editor render resolves it","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Blob mode (","type":"text"},{"text":"type: \"blob\"","type":"text","marks":[{"type":"code_inline"}]},{"text":") -- binary files transferred as-is without parsing. Supported on HTTPExport, NetSuiteExport, SalesforceExport, FTPExport, and S3Export.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Source files are deleted after a successful read.","type":"text","marks":[{"type":"strong"}]},{"text":" An FTP/SFTP or S3 export removes each file it processed from the folder or bucket unless ","type":"text"},{"text":"file.skipDelete","type":"text","marks":[{"type":"code_inline"}]},{"text":" is ","type":"text"},{"text":"true","type":"text","marks":[{"type":"code_inline"}]},{"text":"; ","type":"text"},{"text":"ftp.backupDirectoryPath","type":"text","marks":[{"type":"code_inline"}]},{"text":" (S3: ","type":"text"},{"text":"s3.backupBucket","type":"text","marks":[{"type":"code_inline"}]},{"text":") moves them there instead, which also takes them out of the folder. Files that fail processing stay. When another system or flow reads the same files, set ","type":"text"},{"text":"file.skipDelete: true","type":"text","marks":[{"type":"code_inline"}]},{"text":" and narrow the export to its own files with ","type":"text"},{"text":"ftp.fileNameStartsWith","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"fileNameEndsWith","type":"text","marks":[{"type":"code_inline"}]},{"text":" (S3: ","type":"text"},{"text":"s3.keyStartsWith","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"keyEndsWith","type":"text","marks":[{"type":"code_inline"}]},{"text":").","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"A CSV export on SFTP that leaves the files for another reader (","type":"text"},{"text":"hasHeaderRow","type":"text","marks":[{"type":"code_inline"}]},{"text":" and a comma delimiter are the defaults, shown for clarity):","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"json"},"content":[{"text":"{\n \"name\": \"Read inventory files\",\n \"adaptorType\": \"FTPExport\",\n \"_connectionId\": \"\u003cftpConnectionId>\",\n \"ftp\": { \"directoryPath\": \"/outbound\", \"fileNameStartsWith\": \"inventory_\", \"fileNameEndsWith\": \".csv\" },\n \"file\": { \"type\": \"csv\", \"output\": \"records\", \"skipDelete\": true, \"csv\": { \"hasHeaderRow\": true, \"columnDelimiter\": \",\" } }\n}","type":"text"}]},{"type":"paragraph","content":[{"text":"When to use:","type":"text","marks":[{"type":"strong"}]},{"text":" The source system drops files (CSV, EDI, XML, etc.) into a directory, bucket, file cabinet, or cloud storage rather than exposing a record-based API.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Record-Based Exports","type":"text"}]},{"type":"paragraph","content":[{"text":"Actively fetch batches of records from an API or database on a schedule.","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"HTTPExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- REST and GraphQL APIs (for GraphQL, see ","type":"text"},{"text":"GraphQL APIs","type":"text","marks":[{"type":"link","attrs":{"href":"#graphql-apis----envelope-cursor-paging-errors-with-http-200","title":null}}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NetSuiteHTTPExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- NetSuite's REST APIs on a ","type":"text"},{"text":"netsuite","type":"text","marks":[{"type":"code_inline"}]},{"text":" connection: SuiteQL and REST record endpoints (","type":"text"},{"text":"nsDomainType: \"suitetalk\"","type":"text","marks":[{"type":"code_inline"}]},{"text":") or a custom RESTlet (","type":"text"},{"text":"nsDomainType: \"restlet\"","type":"text","marks":[{"type":"code_inline"}]},{"text":"). Nothing has to exist in NetSuite first, so it's the default for NetSuite exports and lookups","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NetSuiteExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- a saved search (or search criteria) run through Celigo's SuiteApp/bundle RESTlets. Use it when the user names or wants a saved search; it's also the adaptor for NetSuite listeners and file cabinet exports","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"SalesforceExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- SOQL/Bulk queries","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"RDBMSExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- SQL SELECT queries","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"MongodbExport","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"JDBCExport","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"DynamodbExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- other databases","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"WrapperExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- custom connector code, typically a function on a stack (","type":"text"},{"text":"wrapper.function","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"When to use:","type":"text","marks":[{"type":"strong"}]},{"text":" You need to poll an API or query a database for records on a schedule (full fetch or delta/incremental).","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Quick Reference","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Adaptor Decision Matrix","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Your data comes from...","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use adaptorType","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Category","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Read schema","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"REST or GraphQL API","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"HTTPExport","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Record-based","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"http.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/http.yml","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Files in a cloud storage folder (Google Drive, Box, Dropbox, Azure Blob, Google Cloud Storage, ADLS Gen2, Celigo Storage)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"HTTPExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" with ","type":"text"},{"text":"http.type: \"file\"","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"File transfer","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"http.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/http.yml","title":null}}]},{"text":" + ","type":"text"},{"text":"file.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/file.yml","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"NetSuite records (SuiteQL, REST record endpoints, custom RESTlet)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"NetSuiteHTTPExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" + ","type":"text"},{"text":"nsDomainType","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Record-based","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"http.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/http.yml","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"NetSuite saved search, listener, or file cabinet","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"NetSuiteExport","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Record-based, Listener, or File transfer","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"netsuite.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/netsuite.yml","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Salesforce objects","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"SalesforceExport","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Record-based","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"salesforce.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/salesforce.yml","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"SQL database (queries, or CDC on SQL Server and PostgreSQL)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"RDBMSExport","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Record-based or Listener","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"rdbms.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/rdbms.yml","title":null}}]},{"text":" (+ ","type":"text"},{"text":"cdc.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/cdc.yml","title":null}}]},{"text":" for CDC)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"MongoDB (queries or change streams)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"MongodbExport","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Record-based or Listener","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"mongodb.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/mongodb.yml","title":null}}]},{"text":" (+ ","type":"text"},{"text":"cdc.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/cdc.yml","title":null}}]},{"text":" for change streams)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"JDBC database, including NetSuite through SuiteAnalytics Connect (a ","type":"text"},{"text":"netsuitejdbc","type":"text","marks":[{"type":"code_inline"}]},{"text":" connection)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"JDBCExport","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Record-based","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"jdbc.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/jdbc.yml","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"DynamoDB","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"DynamodbExport","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Record-based","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"dynamodb.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/dynamodb.yml","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Files on FTP/SFTP","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"FTPExport","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"File transfer","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"ftp.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/ftp.yml","title":null}}]},{"text":" + ","type":"text"},{"text":"file.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/file.yml","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Files on S3","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"S3Export","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"File transfer","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"s3.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/s3.yml","title":null}}]},{"text":" + ","type":"text"},{"text":"file.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/file.yml","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Webhooks / push events","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"WebhookExport","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Listener","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"webhook.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/webhook.yml","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"AS2 EDI messages","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"AS2Export","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Listener","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"as2.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/as2.yml","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Event Streams topic","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"TopicExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"type: \"stream\"","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Listener","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"topic.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/topic.yml","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Manual file upload","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"SimpleExport","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"File transfer","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"simple.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/simple.yml","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Local filesystem","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"FileSystemExport","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"File transfer","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"filesystem.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/filesystem.yml","title":null}}]},{"text":" + ","type":"text"},{"text":"file.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/file.yml","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Custom connector code on a stack","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"WrapperExport","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Record-based","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"wrapper.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/wrapper.yml","title":null}}]}]}]}]}]},{"type":"paragraph","content":[{"text":"Raw HTTP is the fallback, not the default.","type":"text","marks":[{"type":"strong"}]},{"text":" Pick the most specific match, in order:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Native adaptor","type":"text","marks":[{"type":"strong"}]},{"text":" -- if the application has its own row (NetSuite, Salesforce, databases, FTP/S3), use it. Do not build an ","type":"text"},{"text":"HTTPExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" against that app's REST API.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Pre-built HTTP connector","type":"text","marks":[{"type":"strong"}]},{"text":" -- for any other REST/GraphQL app, check the 550+ connector catalog before writing HTTP config (see ","type":"text"},{"text":"Check for a pre-built connector","type":"text","marks":[{"type":"link","attrs":{"href":"#3-check-for-a-pre-built-connector","title":null}}]},{"text":"). The step is still an ","type":"text"},{"text":"HTTPExport","type":"text","marks":[{"type":"code_inline"}]},{"text":", but it runs on a connector-backed connection and takes its endpoint config from the connector.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Manual HTTP","type":"text","marks":[{"type":"strong"}]},{"text":" -- hand-write the config from public API docs only when no connector exists or it doesn't cover the endpoint you need.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"adaptorType","type":"text","marks":[{"type":"code_inline"}]},{"text":" is ","type":"text"},{"text":"case-sensitive","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"HTTPExport","type":"text","marks":[{"type":"code_inline"}]},{"text":", not ","type":"text"},{"text":"httpExport","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Minimum Required Fields","type":"text"}]},{"type":"paragraph","content":[{"text":"Every export needs at minimum:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"name","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- human-readable label","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"adaptorType","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- from the matrix above","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"_connectionId","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- except ","type":"text"},{"text":"WebhookExport","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"SimpleExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"TopicExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" (a connection on a topic listener fails the save)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Adaptor config block -- ","type":"text"},{"text":"http{}","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"netsuite{}","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"ftp{}","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"salesforce{}","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"rdbms{}","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"topic{}","type":"text","marks":[{"type":"code_inline"}]},{"text":", etc.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Which Schemas to Read","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Always:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"request.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/request.yml","title":null}}]},{"text":" (base fields for all exports)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Plus:","type":"text","marks":[{"type":"strong"}]},{"text":" the adaptor-specific file from the matrix above (e.g., ","type":"text"},{"text":"http.yml","type":"text","marks":[{"type":"code_inline"}]},{"text":" for HTTPExport)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"If file-based:","type":"text","marks":[{"type":"strong"}]},{"text":" also ","type":"text"},{"text":"file.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/file.yml","title":null}}]},{"text":" (CSV, XML, JSON, XLSX, EDI parsing config)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"If delta/incremental:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"type: \"delta\"","type":"text","marks":[{"type":"code_inline"}]},{"text":", plus ","type":"text"},{"text":"delta.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/delta.yml","title":null}}]},{"text":" or the Handlebars URI pattern (","type":"text"},{"text":"{{lastExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"If cloning:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"clone-request.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/clone-request.yml","title":null}}]},{"text":", ","type":"text"},{"text":"clone-response.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/clone-response.yml","title":null}}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Schema Index","type":"text"}]},{"type":"paragraph","content":[{"text":"All schemas are in ","type":"text"},{"text":"references/schemas/","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/","title":null}}]},{"text":":","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Base fields (all exports):","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"request.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/request.yml","title":null}}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Response shape:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"response.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/response.yml","title":null}}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Adaptor-specific config:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"http.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/http.yml","title":null}}]},{"text":" -- HTTP/REST/GraphQL","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"netsuite.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/netsuite.yml","title":null}}]},{"text":" -- ","type":"text"},{"text":"NetSuiteExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" (saved search, Celigo RESTlets, listener, file cabinet); ","type":"text"},{"text":"NetSuiteHTTPExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" uses ","type":"text"},{"text":"http.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/http.yml","title":null}}]},{"text":" plus ","type":"text"},{"text":"nsDomainType","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"salesforce.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/salesforce.yml","title":null}}]},{"text":" -- Salesforce (SOQL, bulk)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"ftp.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/ftp.yml","title":null}}]},{"text":" -- FTP/SFTP","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"s3.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/s3.yml","title":null}}]},{"text":" -- Amazon S3","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"rdbms.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/rdbms.yml","title":null}}]},{"text":" -- SQL databases","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"mongodb.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/mongodb.yml","title":null}}]},{"text":" -- MongoDB","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"jdbc.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/jdbc.yml","title":null}}]},{"text":" -- JDBC databases","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"dynamodb.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/dynamodb.yml","title":null}}]},{"text":" -- DynamoDB","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"as2.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/as2.yml","title":null}}]},{"text":" -- AS2 EDI","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"wrapper.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/wrapper.yml","title":null}}]},{"text":" -- custom stack connectors","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"filesystem.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/filesystem.yml","title":null}}]},{"text":" -- local filesystem","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"simple.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/simple.yml","title":null}}]},{"text":" -- data loader / manual upload","type":"text"}]}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"File parsing:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"file.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/file.yml","title":null}}]},{"text":" (CSV, XML, JSON, XLSX, EDI)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Grouping:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"group-by.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/group-by.yml","title":null}}]},{"text":" (","type":"text"},{"text":"groupByFields","type":"text","marks":[{"type":"code_inline"}]},{"text":"; NetSuite saved searches group with ","type":"text"},{"text":"netsuite.skipGrouping","type":"text","marks":[{"type":"code_inline"}]},{"text":" instead)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Operational modes:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"delta.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/delta.yml","title":null}}]},{"text":", ","type":"text"},{"text":"once.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/once.yml","title":null}}]},{"text":", ","type":"text"},{"text":"test.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/test.yml","title":null}}]},{"text":", ","type":"text"},{"text":"webhook.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/webhook.yml","title":null}}]},{"text":", ","type":"text"},{"text":"cdc.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/cdc.yml","title":null}}]},{"text":" (database listeners), ","type":"text"},{"text":"topic.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/topic.yml","title":null}}]},{"text":" (topic listeners). NetSuite and Salesforce listener settings live in ","type":"text"},{"text":"netsuite.distributed","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"salesforce.distributed","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"netsuite.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/netsuite.yml","title":null}}]},{"text":", ","type":"text"},{"text":"salesforce.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/salesforce.yml","title":null}}]},{"text":"); ","type":"text"},{"text":"distributed.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/distributed.yml","title":null}}]},{"text":" holds only the optional ","type":"text"},{"text":"bearerToken","type":"text","marks":[{"type":"code_inline"}]},{"text":" for pushing records into a distributed export","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Mock output:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"mock-output.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/mock-output.yml","title":null}}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Preview request:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"preview-request.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/preview-request.yml","title":null}}]},{"text":" (","type":"text"},{"text":"postData","type":"text","marks":[{"type":"code_inline"}]},{"text":" sets the delta variables in a preview)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Clone:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"clone-request.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/clone-request.yml","title":null}}]},{"text":", ","type":"text"},{"text":"clone-response.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/clone-response.yml","title":null}}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Related Skills","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"configuring-connections > Quick Reference","type":"text","marks":[{"type":"link","attrs":{"href":"../configuring-connections/SKILL.md#quick-reference","title":null}}]},{"text":" -- connection types, auth methods, iClients","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"managing-file-storage > Use stored files in a flow","type":"text","marks":[{"type":"link","attrs":{"href":"../managing-file-storage/SKILL.md#8-use-stored-files-in-a-flow","title":null}}]},{"text":" -- Celigo Storage folders as a file source","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"writing-mappings > Transformation 2.0","type":"text","marks":[{"type":"link","attrs":{"href":"../writing-mappings/SKILL.md#transformation-20-workflow","title":null}}]},{"text":" -- reshape export output before mapping","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"writing-scripts > Data Pipeline Hooks","type":"text","marks":[{"type":"link","attrs":{"href":"../writing-scripts/SKILL.md#data-pipeline-hooks","title":null}}]},{"text":" -- preSavePage, postResponseMap hooks","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"writing-handlebars > Quick Reference","type":"text","marks":[{"type":"link","attrs":{"href":"../writing-handlebars/SKILL.md#quick-reference","title":null}}]},{"text":" -- dynamic values in URIs, filters, delta tokens","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"building-flows > How to Build a Flow","type":"text","marks":[{"type":"link","attrs":{"href":"../building-flows/SKILL.md#how-to-build-a-flow","title":null}}]},{"text":" -- wiring exports into flows","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"configuring-topics > How to Build an Event Stream","type":"text","marks":[{"type":"link","attrs":{"href":"../configuring-topics/SKILL.md#how-to-build-an-event-stream","title":null}}]},{"text":" -- the topic a listener reads: registration, retention, messages, positions","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"building-syncs > Sources","type":"text","marks":[{"type":"link","attrs":{"href":"../building-syncs/SKILL.md#sources-catalog-datasets-and-export-backed-datasets","title":null}}]},{"text":" -- an export as the source of a sync's export-backed dataset","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"troubleshooting-flows > Diagnostic Workflow","type":"text","marks":[{"type":"link","attrs":{"href":"../troubleshooting-flows/SKILL.md#diagnostic-workflow","title":null}}]},{"text":" -- diagnosing export-related failures","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"\u003c!-- TIER:2 -->","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"How to Build an Export","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"1. Identify the target application","type":"text"}]},{"type":"paragraph","content":[{"text":"What system are you pulling data from? This determines everything -- adaptor type, connection type, and configuration shape.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"2. Check for existing patterns","type":"text"}]},{"type":"paragraph","content":[{"text":"Before building from scratch, look at what already exists:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Search across the entire account for related resources\nceligo account search \"\u003ckeyword>\"\n\n# Show what an existing export uses (connection) and what uses it (flows)\nceligo account dependencies export \u003cid>\n\n# Find orphaned exports not referenced by any flow\nceligo account lint\n\n# Check if a similar export already exists in the account\nceligo exports list | grep -i \"\u003capplication-name>\"\n\n# Search the marketplace for pre-built integration templates\nceligo templates marketplace\n\n# Preview a template to see its export configuration\nceligo templates preview \u003cid> --model Export\nceligo templates preview \u003cid> --summary","type":"text"}]},{"type":"paragraph","content":[{"text":"The account index auto-refreshes when stale (older than 15 minutes by default). Force a fresh snapshot with ","type":"text"},{"text":"celigo account snapshot","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"Existing exports in the account are the best reference -- they show proven patterns for that specific customer's setup. Marketplace templates may provide a complete pre-built integration you can install rather than building from scratch.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"3. Check for a pre-built connector","type":"text"}]},{"type":"paragraph","content":[{"text":"Always run this check before writing any HTTP config.","type":"text","marks":[{"type":"strong"}]},{"text":" Celigo maintains 550+ HTTP connector definitions and 590+ trading partner connectors. These provide pre-configured auth, base URLs, and endpoint definitions for common applications. Connectors are set on the ","type":"text"},{"text":"connection","type":"text","marks":[{"type":"strong"}]},{"text":", not the export -- but they determine what the export can do. Hand-write a manual ","type":"text"},{"text":"HTTPExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" from public API docs only when this search comes up empty or the connector doesn't cover the endpoint you need.","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Search HTTP connectors (REST APIs: Shopify, Stripe, HubSpot, etc.)\nceligo http-connectors list | grep -i \"\u003capplication-name>\"\nceligo http-connectors get \u003cid> --full # see endpoints, resources, auth config\n\n# Drill into the endpoints the connector defines for exports\nceligo http-connectors catalog \u003cid> --resource-type export --published-only\nceligo http-connectors endpoint-detail \u003cid> --resource-type export --resource-id \u003crid> --endpoint-id \u003cepid>\n\n# Search trading partner connectors (EDI, AS2, VAN)\nceligo tp-connectors list","type":"text"}]},{"type":"paragraph","content":[{"text":"If an HTTP connector exists for your target app, create the connection from it (","type":"text"},{"text":"http._httpConnectorId","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- see ","type":"text"},{"text":"configuring-connections > Check for a pre-built connector and global iClient","type":"text","marks":[{"type":"link","attrs":{"href":"../configuring-connections/SKILL.md#4-check-for-a-pre-built-connector-and-global-iclient","title":null}}]},{"text":") and take the export's ","type":"text"},{"text":"relativeURI","type":"text","marks":[{"type":"code_inline"}]},{"text":", method, pagination, and response paths from the connector's endpoint metadata rather than reconstructing them from public API docs. In ","type":"text"},{"text":"celigo http-connectors endpoint-detail","type":"text","marks":[{"type":"code_inline"}]},{"text":", the endpoint's ","type":"text"},{"text":"supportedBy","type":"text","marks":[{"type":"code_inline"}]},{"text":" lists what the connector's own form applies: ","type":"text"},{"text":"preConfiguredFields","type":"text","marks":[{"type":"code_inline"}]},{"text":" (values it sets), ","type":"text"},{"text":"fieldsUserMustSet","type":"text","marks":[{"type":"code_inline"}]},{"text":" (what it asks the user for -- often which ","type":"text"},{"text":"type","type":"text","marks":[{"type":"code_inline"}]},{"text":" the endpoint supports, such as ","type":"text"},{"text":"delta","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"test","type":"text","marks":[{"type":"code_inline"}]},{"text":") and ","type":"text"},{"text":"fieldsToUnset","type":"text","marks":[{"type":"code_inline"}]},{"text":". Their paths are the form's, not the export's: Shopify's list-customers endpoint presets ","type":"text"},{"text":"response","type":"text","marks":[{"type":"code_inline"}]},{"text":" to ","type":"text"},{"text":"{resourcePath: \"customers\"}","type":"text","marks":[{"type":"code_inline"}]},{"text":" (the export's ","type":"text"},{"text":"http.response.resourcePath","type":"text","marks":[{"type":"code_inline"}]},{"text":") and ","type":"text"},{"text":"delta","type":"text","marks":[{"type":"code_inline"}]},{"text":" to a ","type":"text"},{"text":"dateFormat","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"delta.dateFormat","type":"text","marks":[{"type":"code_inline"}]},{"text":") plus ","type":"text"},{"text":"defaults: {updated_at_min: \"{{{lastExportDateTime}}}\"}","type":"text","marks":[{"type":"code_inline"}]},{"text":" (a query parameter in ","type":"text"},{"text":"relativeURI","type":"text","marks":[{"type":"code_inline"}]},{"text":"). Saving links the export to the connector: when ","type":"text"},{"text":"relativeURI","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"method","type":"text","marks":[{"type":"code_inline"}]},{"text":" match one of its endpoints, the platform sets ","type":"text"},{"text":"http._httpConnectorVersionId","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"http._httpConnectorResourceId","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"http._httpConnectorEndpointId","type":"text","marks":[{"type":"code_inline"}]},{"text":", replacing pointers that belong to another connector. Pointers for the connection's own connector are kept exactly as sent, even when the URI names a different endpoint. Read the export back after saving (no pointers means the URI matched no endpoint), and when you retarget it to another endpoint, update or drop the old pointers in the same save: the request follows ","type":"text"},{"text":"relativeURI","type":"text","marks":[{"type":"code_inline"}]},{"text":", but stale pointers keep naming the old endpoint in the editor and in later edits.","type":"text"}]},{"type":"paragraph","content":[{"text":"If a trading partner connector exists (EDI/AS2), reference it on the export via ","type":"text"},{"text":"ftp._tpConnectorId","type":"text","marks":[{"type":"code_inline"}]},{"text":" (FTP exports) or ","type":"text"},{"text":"as2._tpConnectorId","type":"text","marks":[{"type":"code_inline"}]},{"text":" (AS2 exports). You may also need to set ","type":"text"},{"text":"_ediProfileId","type":"text","marks":[{"type":"code_inline"}]},{"text":" on the export for EDI document validation.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"4. Query metadata for the target system","type":"text"}]},{"type":"paragraph","content":[{"text":"For NetSuite, Salesforce, and RDBMS connections, you can discover available record types and fields directly from the live system:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# List available record types / sObjects / tables\n# NetSuite also returns saved searches alongside record types\nceligo metadata types \u003cconnectionId>\n\n# List fields for a specific entity type\nceligo metadata fields \u003cconnectionId> \u003centityType>","type":"text"}]},{"type":"paragraph","content":[{"text":"This tells you what data is available to export before you write any configuration. Both commands read a server-side cache: after a field, record type or saved search is created in the app, re-read with ","type":"text"},{"text":"--refresh","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NetSuite:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"metadata types","type":"text","marks":[{"type":"code_inline"}]},{"text":" returns both record types and saved searches (with IDs you need for ","type":"text"},{"text":"netsuite.restlet.searchId","type":"text","marks":[{"type":"code_inline"}]},{"text":"). Use a record type's lowercase ","type":"text"},{"text":"scriptId","type":"text","marks":[{"type":"code_inline"}]},{"text":" as ","type":"text"},{"text":"recordType","type":"text","marks":[{"type":"code_inline"}]},{"text":", not its ","type":"text"},{"text":"id","type":"text","marks":[{"type":"code_inline"}]},{"text":": for custom records and lists, ","type":"text"},{"text":"id","type":"text","marks":[{"type":"code_inline"}]},{"text":" is NetSuite's numeric internal id. A search over every transaction type uses ","type":"text"},{"text":"recordType: \"transaction\"","type":"text","marks":[{"type":"code_inline"}]},{"text":". ","type":"text"},{"text":"metadata fields","type":"text","marks":[{"type":"code_inline"}]},{"text":" returns field IDs, names, types, and group — including the sublist groups and fields you'll need for ","type":"text"},{"text":"netsuite_da.mapping.lists[]","type":"text","marks":[{"type":"code_inline"}]},{"text":" on a NetSuite DA import. Search criteria and columns take search-column ids instead, which can differ (","type":"text"},{"text":"shipaddress1","type":"text","marks":[{"type":"code_inline"}]},{"text":", not ","type":"text"},{"text":"shipaddr1","type":"text","marks":[{"type":"code_inline"}]},{"text":").","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Salesforce:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"metadata types","type":"text","marks":[{"type":"code_inline"}]},{"text":" returns sObjects with queryable/createable flags. ","type":"text"},{"text":"metadata fields","type":"text","marks":[{"type":"code_inline"}]},{"text":" returns fields, types, and relationship names — use these to discover child objects for ","type":"text"},{"text":"salesforce.distributed.relatedLists[]","type":"text","marks":[{"type":"code_inline"}]},{"text":" and relationship field names for cross-object queries.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"RDBMS:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"metadata types","type":"text","marks":[{"type":"code_inline"}]},{"text":" returns table names. ","type":"text"},{"text":"metadata fields","type":"text","marks":[{"type":"code_inline"}]},{"text":" returns column names and types for a given table — use these when writing SQL queries or building field mappings.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"5. Determine the category","type":"text"}]},{"type":"paragraph","content":[{"text":"Is this a ","type":"text"},{"text":"listener","type":"text","marks":[{"type":"strong"}]},{"text":" (real-time push from the source), a ","type":"text"},{"text":"file transfer","type":"text","marks":[{"type":"strong"}]},{"text":" (fetch and parse/transfer files), or a ","type":"text"},{"text":"record-based export","type":"text","marks":[{"type":"strong"}]},{"text":" (poll an API or query a database)? This narrows which adaptor types and modes apply.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"6. Choose the right adaptor type","type":"text"}]},{"type":"paragraph","content":[{"text":"Use the ","type":"text"},{"text":"Adaptor Decision Matrix","type":"text","marks":[{"type":"link","attrs":{"href":"#adaptor-decision-matrix","title":null}}]},{"text":" in Quick Reference above to select the correct ","type":"text"},{"text":"adaptorType","type":"text","marks":[{"type":"code_inline"}]},{"text":" for your target system. For NetSuite, ","type":"text"},{"text":"Diagnosing NetSuite Steps","type":"text","marks":[{"type":"link","attrs":{"href":"references/netsuite.md","title":null}}]},{"text":" has the checks to run against the NetSuite account and the errors its steps return.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"7. Build the export JSON","type":"text"}]},{"type":"paragraph","content":[{"text":"Use the ","type":"text"},{"text":"Schema Index","type":"text","marks":[{"type":"link","attrs":{"href":"#schema-index","title":null}}]},{"text":" and ","type":"text"},{"text":"Which Schemas to Read","type":"text","marks":[{"type":"link","attrs":{"href":"#which-schemas-to-read","title":null}}]},{"text":" in Quick Reference above. Read ","type":"text"},{"text":"request.yml","type":"text","marks":[{"type":"code_inline"}]},{"text":" for base fields, then the adaptor-specific schema, plus ","type":"text"},{"text":"file.yml","type":"text","marks":[{"type":"code_inline"}]},{"text":" if file-based and ","type":"text"},{"text":"delta.yml","type":"text","marks":[{"type":"code_inline"}]},{"text":" if incremental.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Export Design Decisions","type":"text"}]},{"type":"paragraph","content":[{"text":"A few design choices recur when building exports. Each has a defensible default once the framing is clear.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Delta vs one-time vs full sync","type":"text"}]},{"type":"paragraph","content":[{"text":"The export's ","type":"text"},{"text":"type","type":"text","marks":[{"type":"code_inline"}]},{"text":" field selects the sync behavior:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Delta","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"type: \"delta\"","type":"text","marks":[{"type":"code_inline"}]},{"text":") -- pulls only records created or modified since the flow's checkpoint (see ","type":"text"},{"text":"Delta windows and the checkpoint","type":"text","marks":[{"type":"link","attrs":{"href":"#delta-windows-and-the-checkpoint","title":null}}]},{"text":"). The default for ongoing scheduled syncs when the source exposes a usable \"last modified\" timestamp. Application adaptors such as NetSuite and Salesforce name the timestamp field in ","type":"text"},{"text":"delta.dateField","type":"text","marks":[{"type":"code_inline"}]},{"text":"; HTTP exports and SQL queries put ","type":"text"},{"text":"{{lastExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" in the request instead (","type":"text"},{"text":"relativeURI","type":"text","marks":[{"type":"code_inline"}]},{"text":", body or query), and an HTTP export with ","type":"text"},{"text":"delta.dateField","type":"text","marks":[{"type":"code_inline"}]},{"text":" is refused with 422 \"Unsupported dateField in export.delta.dateField\". See ","type":"text"},{"text":"delta.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/delta.yml","title":null}}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"One-time","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"type: \"once\"","type":"text","marks":[{"type":"code_inline"}]},{"text":") -- reads each record once by marking it in the source: each run selects the records not yet marked, then marks them after a page succeeds so later runs skip them. Adaptors that mark a checkbox name it in ","type":"text"},{"text":"once.booleanField","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"false","type":"text","marks":[{"type":"code_inline"}]},{"text":" to ","type":"text"},{"text":"true","type":"text","marks":[{"type":"code_inline"}]},{"text":"); an RDBMS export runs ","type":"text"},{"text":"rdbms.once.query","type":"text","marks":[{"type":"code_inline"}]},{"text":", an ","type":"text"},{"text":"UPDATE","type":"text","marks":[{"type":"code_inline"}]},{"text":" (see ","type":"text"},{"text":"writing-sql","type":"text","marks":[{"type":"link","attrs":{"href":"../writing-sql/SKILL.md","title":null}}]},{"text":"); an HTTP export calls ","type":"text"},{"text":"http.once","type":"text","marks":[{"type":"code_inline"}]},{"text":", a request rendered against each record as exported, before any mapping. Each mark runs only while the export's ","type":"text"},{"text":"type","type":"text","marks":[{"type":"code_inline"}]},{"text":" is ","type":"text"},{"text":"once","type":"text","marks":[{"type":"code_inline"}]},{"text":": on an export with no ","type":"text"},{"text":"type","type":"text","marks":[{"type":"code_inline"}]},{"text":", a saved ","type":"text"},{"text":"http.once","type":"text","marks":[{"type":"code_inline"}]},{"text":" is kept and never called, so every run reads everything again. The mark only stops this export from reading the record again; keeping a repeated record from writing twice is the import's job (","type":"text"},{"text":"configuring-imports > Writing Exactly Once","type":"text","marks":[{"type":"link","attrs":{"href":"../configuring-imports/SKILL.md#writing-exactly-once","title":null}}]},{"text":"). Use for backfills and migrations, or when the source has no reliable timestamp but its records can carry a processed flag. See ","type":"text"},{"text":"once.yml","type":"text","marks":[{"type":"link","attrs":{"href":"references/schemas/once.yml","title":null}}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Full","type":"text","marks":[{"type":"strong"}]},{"text":" (neither ","type":"text"},{"text":"delta","type":"text","marks":[{"type":"code_inline"}]},{"text":" nor ","type":"text"},{"text":"once","type":"text","marks":[{"type":"code_inline"}]},{"text":" mode) -- re-pulls the entire dataset every run. Use when the source has no usable modification timestamp, the dataset is small enough that re-pulling is cheap, or business logic requires a fresh snapshot each run.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"When the request is vague (\"sync customers\"), confirm which kind of sync is intended before building. Delta is a reasonable default when the source exposes a timestamp field; full is reasonable for small static datasets.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Delta windows and the checkpoint","type":"text"}]},{"type":"paragraph","content":[{"text":"Each delta run reads one window. ","type":"text"},{"text":"{{lastExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" is its start (the flow's checkpoint) and ","type":"text"},{"text":"{{currentExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" its end (the time the run started). Put ","type":"text"},{"text":"{{lastExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" in the request as the lower bound, such as ","type":"text"},{"text":"?updated_since={{lastExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":"; with ","type":"text"},{"text":"{{currentExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" there, every run asks for records changed after it started and exports nothing. Both render as ISO 8601 UTC with milliseconds (","type":"text"},{"text":"2026-10-01T00:00:00.000Z","type":"text","marks":[{"type":"code_inline"}]},{"text":") in HTTP requests and SQL queries alike, and ","type":"text"},{"text":"delta.dateFormat","type":"text","marks":[{"type":"code_inline"}]},{"text":" reformats both. ","type":"text"},{"text":"{{lastExportDateTimeUTC}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" isn't a variable: it renders empty. On an export with no ","type":"text"},{"text":"type","type":"text","marks":[{"type":"code_inline"}]},{"text":" neither is defined: a run fails with ","type":"text"},{"text":"handlebars_template_parse_error","type":"text","marks":[{"type":"code_inline"}]},{"text":" (\"","type":"text"},{"text":"lastExportDateTime","type":"text","marks":[{"type":"code_inline"}]},{"text":" not defined in the model\") before the request is sent, so set ","type":"text"},{"text":"type: \"delta\"","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Set ","type":"text","marks":[{"type":"strong"}]},{"text":"delta.startDate","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" for the first run.","type":"text","marks":[{"type":"strong"}]},{"text":" Without it, a new flow's first run renders ","type":"text"},{"text":"{{lastExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" as the run's own time -- an empty window, not a full export. With it, the first run starts there.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"The checkpoint moves only when a run exports records.","type":"text","marks":[{"type":"strong"}]},{"text":" Such a run sets it (","type":"text"},{"text":"GET /v1/flows/\u003cflowId>/lastExportDateTime","type":"text","marks":[{"type":"code_inline"}]},{"text":") to its window end. A run that exports nothing leaves it where it was, so the next run that finds records reaches back to the last run that exported any. After a stretch of empty runs (a broken filter token, an outage at the source), fix the cause and the next run picks up the missed records by itself: no backfill run is needed.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A run given its own window leaves the checkpoint at that window's end.","type":"text","marks":[{"type":"strong"}]},{"text":" A run with ","type":"text"},{"text":"export.startDate","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"endDate","type":"text","marks":[{"type":"code_inline"}]},{"text":" (CLI: ","type":"text"},{"text":"flows run --start-time/--end-time","type":"text","marks":[{"type":"code_inline"}]},{"text":") reads that window and leaves the checkpoint at ","type":"text"},{"text":"endDate","type":"text","marks":[{"type":"code_inline"}]},{"text":", even a past one, so the next scheduled run re-reads everything after it. A backfill with an old end date rewinds the export.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A document preview renders both variables empty","type":"text","marks":[{"type":"strong"}]},{"text":" unless the document supplies them in ","type":"text"},{"text":"postData","type":"text","marks":[{"type":"code_inline"}]},{"text":" (gotcha 18), so it doesn't show the delta window by itself.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Listener/webhook vs scheduled export","type":"text"}]},{"type":"paragraph","content":[{"text":"Both are starting steps (see ","type":"text"},{"text":"Three Categories of Export","type":"text","marks":[{"type":"link","attrs":{"href":"#three-categories-of-export","title":null}}]},{"text":"); the choice is driven by what the source supports and the latency budget, not preference:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Reach for a ","type":"text"},{"text":"listener","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"WebhookExport","type":"text","marks":[{"type":"code_inline"}]},{"text":", NetSuite/Salesforce ","type":"text"},{"text":"type: \"distributed\"","type":"text","marks":[{"type":"code_inline"}]},{"text":", or ","type":"text"},{"text":"type: \"stream\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" for database CDC and topic listeners) when the source pushes events or streams its changes and the flow needs to react quickly (\"when X happens, do Y\", \"when a row changes\").","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Reach for a ","type":"text"},{"text":"scheduled export","type":"text","marks":[{"type":"strong"}]},{"text":" when the source has no push mechanism, or when batch timing at off-peak hours is acceptable.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"NetSuite and Salesforce support both for many record types. Mixing them on one flow is a common, good pattern -- a listener handles low-latency reactions while a scheduled export runs as a safety net for backfills, end-of-day reconciliation, and catching up after a webhook outage.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Webhook listeners -- type, URL, signature, responses","type":"text"}]},{"type":"paragraph","content":[{"text":"A webhook listener is a ","type":"text"},{"text":"WebhookExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" with ","type":"text"},{"text":"type: \"webhook\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" and a ","type":"text"},{"text":"webhook","type":"text","marks":[{"type":"code_inline"}]},{"text":" object. Without the type, the save is refused with 422 \"_connectionId is required unless type is [webhook|simple]\". It needs no connection; one may be set, and it saves and updates without error. It runs only as a flow's source: put it in the flow's ","type":"text"},{"text":"pageGenerators","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"[{\"_exportId\": \"\u003cexportId>\"}]","type":"text","marks":[{"type":"code_inline"}]},{"text":") with a full update of the flow that keeps its ","type":"text"},{"text":"pageProcessors","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- the flow PATCH route can't set ","type":"text"},{"text":"pageGenerators","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- then enable the flow.","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"URL.","type":"text","marks":[{"type":"strong"}]},{"text":" The sender POSTs to ","type":"text"},{"text":"https://api.integrator.io/v1/exports/\u003cexportId>/data","type":"text","marks":[{"type":"code_inline"}]},{"text":", or the same path on the account's regional API host (","type":"text"},{"text":"api.eu.integrator.io","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"api.au.integrator.io","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"api.ca.integrator.io","type":"text","marks":[{"type":"code_inline"}]},{"text":"). The export's page in Celigo shows this URL as its public URL, ready to copy. There is no other listener path.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Signature.","type":"text","marks":[{"type":"strong"}]},{"text":" Configure exactly what the sender does. HMAC takes ","type":"text"},{"text":"verify: \"hmac\"","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"key","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"algorithm","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"sha256","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"sha1","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"sha384","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"sha512","type":"text","marks":[{"type":"code_inline"}]},{"text":"), ","type":"text"},{"text":"encoding","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"hex","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"base64","type":"text","marks":[{"type":"code_inline"}]},{"text":") and ","type":"text"},{"text":"header","type":"text","marks":[{"type":"code_inline"}]},{"text":" (the header that carries the signature). With ","type":"text"},{"text":"provider: \"custom\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" the platform computes the HMAC of the raw body and compares the header value with it character for character: a lowercase hex SHA-256 signature was accepted, while base64, uppercase hex, a ","type":"text"},{"text":"sha256=","type":"text","marks":[{"type":"code_inline"}]},{"text":" prefix, SHA-1 and a wrong key each got 401.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Secret.","type":"text","marks":[{"type":"strong"}]},{"text":" When the receiver picks the secret, as many senders allow, generate a random one of at least 32 characters (for example ","type":"text"},{"text":"openssl rand -hex 32","type":"text","marks":[{"type":"code_inline"}]},{"text":"), save it as the ","type":"text"},{"text":"key","type":"text","marks":[{"type":"code_inline"}]},{"text":" (or ","type":"text"},{"text":"token","type":"text","marks":[{"type":"code_inline"}]},{"text":"), and give it to the user to register with the sender. Never save a placeholder. Reads return it masked, so hand it over when you create it.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Responses.","type":"text","marks":[{"type":"strong"}]},{"text":" 204 when a request is accepted (the default ","type":"text"},{"text":"successStatusCode","type":"text","marks":[{"type":"code_inline"}]},{"text":"), 401 when it fails verification, and 403 ","type":"text"},{"text":"unable_to_run_flows","type":"text","marks":[{"type":"code_inline"}]},{"text":" while every flow the export feeds is disabled or not fully configured. A refused request isn't queued: once the flow is enabled again, the sender has to resend what it delivered in the meantime. Enabling the flow is the whole fix: the next delivery is accepted at once, and nothing needs to be run.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Updating the export needs the secret.","type":"text","marks":[{"type":"strong"}]},{"text":" Reads return ","type":"text"},{"text":"webhook.token","type":"text","marks":[{"type":"code_inline"}]},{"text":" masked, and an update without it is refused with 422 (\"token is required for token verification\"). When the flow is the problem, fix the flow and leave the export alone.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Lookup export vs separate scheduled export","type":"text"}]},{"type":"paragraph","content":[{"text":"The distinguishing question is when the data is needed:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A ","type":"text"},{"text":"lookup export","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"isLookup: true","type":"text","marks":[{"type":"code_inline"}]},{"text":") runs per in-flight record, mid-pipeline, keyed off the upstream record -- fetching the customer for a specific order, or inventory for a specific SKU.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A ","type":"text"},{"text":"scheduled export","type":"text","marks":[{"type":"strong"}]},{"text":" runs once per flow run as a starting point, producing the first batch of records the flow processes.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"If the request is \"for each X, look up Y\", it's a lookup. If it's \"every hour, pull all Y\", it's a scheduled export.","type":"text"}]},{"type":"paragraph","content":[{"text":"Four lookup behaviors to plan for before the first test:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Finding nothing isn't a failure.","type":"text","marks":[{"type":"strong"}]},{"text":" Zero results count as success: the record continues with nothing merged, so ","type":"text"},{"text":"proceedOnFailure","type":"text","marks":[{"type":"code_inline"}]},{"text":" doesn't apply. To fail the record instead on an HTTP lookup, set ","type":"text"},{"text":"http.response.successPath","type":"text","marks":[{"type":"code_inline"}]},{"text":" (without ","type":"text"},{"text":"successValues","type":"text","marks":[{"type":"code_inline"}]},{"text":") to a path only a match has, such as ","type":"text"},{"text":"results[0].id","type":"text","marks":[{"type":"code_inline"}]},{"text":"; an empty result then fails with ","type":"text"},{"text":"response_failure","type":"text","marks":[{"type":"code_inline"}]},{"text":". For your own error message, a ","type":"text"},{"text":"preSavePage","type":"text","marks":[{"type":"code_inline"}]},{"text":" hook on the lookup can return one ","type":"text"},{"text":"newErrorsAndRetryData","type":"text","marks":[{"type":"code_inline"}]},{"text":" entry when ","type":"text"},{"text":"options.data","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"options.errors","type":"text","marks":[{"type":"code_inline"}]},{"text":" are both empty. To route instead of fail, response-map ","type":"text"},{"text":"data","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"errors","type":"text","marks":[{"type":"code_inline"}]},{"text":" onto the record and branch on them.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"The response mapping reads the lookup's output, not its raw response.","type":"text","marks":[{"type":"strong"}]},{"text":" It sees the records after the lookup's own transform and output filter, so extract the field names the transform produces. A saved ","type":"text"},{"text":"mockOutput","type":"text","marks":[{"type":"code_inline"}]},{"text":" holds the raw response and a test run applies the transform on top of it, so don't write the mapping against the stored shape.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"More than one match needs a guard.","type":"text","marks":[{"type":"strong"}]},{"text":" When a lookup can match more than one record (a name, email or phone key), guard ","type":"text"},{"text":"data[0]","type":"text","marks":[{"type":"code_inline"}]},{"text":" before it feeds an update -- see ","type":"text"},{"text":"configuring-lookup-caches > Safe Lookup-to-Update","type":"text","marks":[{"type":"link","attrs":{"href":"../configuring-lookup-caches/SKILL.md#safe-lookup-to-update----never-write-off-an-unguarded-first-match","title":null}}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A standalone preview can't render the record.","type":"text","marks":[{"type":"strong"}]},{"text":" Any lookup whose request reads the arriving record (","type":"text"},{"text":"{{record.*}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" in a URI, body, SOQL or SQL query, or a ","type":"text"},{"text":"{{...}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" field in a NetSuite search criterion) fails a standalone preview with ","type":"text"},{"text":"handlebars_template_parse_error","type":"text","marks":[{"type":"code_inline"}]},{"text":" (\"... not defined in the model\"): neither the export preview nor a virtual export (","type":"text"},{"text":"exports invoke --connection","type":"text","marks":[{"type":"code_inline"}]},{"text":") binds a record, even when you send one. That's a preview limitation, not a broken template. Check the lookup in the flow builder's step preview with an upstream record, or in a flow test run (","type":"text"},{"text":"celigo flows test-run \u003cflowId> --export \u003csourceExportId>","type":"text","marks":[{"type":"code_inline"}]},{"text":"), which calls the lookup live unless it has a saved ","type":"text"},{"text":"mockOutput","type":"text","marks":[{"type":"code_inline"}]},{"text":" (a test run replays it).","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"One-to-many fan-out -- the array element IS the record","type":"text"}]},{"type":"paragraph","content":[{"text":"oneToMany","type":"text","marks":[{"type":"code_inline"}]},{"text":" plus ","type":"text"},{"text":"pathToMany","type":"text","marks":[{"type":"code_inline"}]},{"text":" unwraps a child array that is already on the input record. It does not locate records in an API response (","type":"text"},{"text":"resourcePath","type":"text","marks":[{"type":"code_inline"}]},{"text":" does that), and it does not split a lookup's multi-row response (map that onto the record with response mapping ","type":"text"},{"text":"extract: \"data\"","type":"text","marks":[{"type":"code_inline"}]},{"text":", then fan out over the mapped array in a later step). Only a step that receives a record fans out: on a flow's source export both fields save and do nothing, so split a source's records with a ","type":"text"},{"text":"preSavePage","type":"text","marks":[{"type":"code_inline"}]},{"text":" hook (or, for a file definition, the record boundary in its rules).","type":"text"}]},{"type":"paragraph","content":[{"text":"With ","type":"text"},{"text":"oneToMany: true","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"pathToMany","type":"text","marks":[{"type":"code_inline"}]},{"text":" set to a child array path, each element of that array triggers its own lookup. Once fanned out, ","type":"text"},{"text":"the element becomes the record","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Child fields are ","type":"text"},{"text":"{{record.variantId}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- not ","type":"text"},{"text":"{{variantId}}","type":"text","marks":[{"type":"code_inline"}]},{"text":", and not ","type":"text"},{"text":"{{record.lineItems.variantId}}","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"The parent's other fields stay on the child as ","type":"text"},{"text":"record._PARENT","type":"text","marks":[{"type":"code_inline"}]},{"text":" (the parent without the array): ","type":"text"},{"text":"{{record._PARENT.id}}","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Fan-out does not change the request root to ","type":"text"},{"text":"batch_of_records","type":"text","marks":[{"type":"code_inline"}]},{"text":". A lookup has no request batch size: it runs once per fanned-out child, and its request always renders against ","type":"text"},{"text":"record","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"When the arriving record is itself an array -- grouped rows from a ","type":"text"},{"text":"groupByFields","type":"text","marks":[{"type":"code_inline"}]},{"text":" export or a file group -- leave ","type":"text"},{"text":"pathToMany","type":"text","marks":[{"type":"code_inline"}]},{"text":" blank: each element becomes the record, with no ","type":"text"},{"text":"_PARENT","type":"text","marks":[{"type":"code_inline"}]},{"text":". A path on an array record fans out zero children and still reports success; a blank path on an object record fails with ","type":"text"},{"text":"oneToMany_error","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"Three consequences worth knowing before you debug the wrong thing:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"The build-time preview warning is expected.","type":"text","marks":[{"type":"strong"}]},{"text":" Previewing a fanned-out lookup in isolation reports \"","type":"text"},{"text":"\u003cfield>","type":"text","marks":[{"type":"code_inline"}]},{"text":" not defined in the model\" because no upstream record is bound, as for any lookup that reads the record (","type":"text"},{"text":"Lookup export vs separate scheduled export","type":"text","marks":[{"type":"link","attrs":{"href":"#lookup-export-vs-separate-scheduled-export","title":null}}]},{"text":"). That is not a broken template -- don't \"fix\" a correct ","type":"text"},{"text":"{{record.X}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" reference because of it.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Response mapping merges per element automatically.","type":"text","marks":[{"type":"strong"}]},{"text":" To get looked-up values back onto each element, author a normal top-level ","type":"text"},{"text":"fields","type":"text","marks":[{"type":"code_inline"}]},{"text":" response mapping; Celigo merges each result into its corresponding fanned-out element.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Two anti-patterns.","type":"text","marks":[{"type":"strong"}]},{"text":" Don't target the array with a ","type":"text"},{"text":"lists","type":"text","marks":[{"type":"code_inline"}]},{"text":" entry (that nests a new array inside each element), and don't attempt the per-element merge in ","type":"text"},{"text":"postResponseMap","type":"text","marks":[{"type":"code_inline"}]},{"text":" (it sees the page of parent records, not per-element results).","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Grouped records -- one record, many rows","type":"text"}]},{"type":"paragraph","content":[{"text":"Grouping runs after parsing and before the transform, and turns the rows that share a key into one record:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NetSuite saved searches","type":"text","marks":[{"type":"strong"}]},{"text":" group with Group rows (","type":"text"},{"text":"netsuite.skipGrouping","type":"text","marks":[{"type":"code_inline"}]},{"text":" false or omitted), keyed on the internal id, so sort the search by Internal ID. ","type":"text"},{"text":"groupByFields","type":"text","marks":[{"type":"code_inline"}]},{"text":" is refused on a NetSuite export (","type":"text"},{"text":"[invalid_config] netsuite connection is not supported for groupByFields property","type":"text","marks":[{"type":"code_inline"}]},{"text":"), and an ad-hoc search needs ","type":"text"},{"text":"skipGrouping: true","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Every other source","type":"text","marks":[{"type":"strong"}]},{"text":" uses ","type":"text"},{"text":"groupByFields","type":"text","marks":[{"type":"code_inline"}]},{"text":", which merges only adjacent rows with equal values, so return same-key rows together: the query's ","type":"text"},{"text":"ORDER BY","type":"text","marks":[{"type":"code_inline"}]},{"text":", or ","type":"text"},{"text":"file.sortByFields","type":"text","marks":[{"type":"code_inline"}]},{"text":" on a file export (it's ignored on HTTP and database exports). ","type":"text"},{"text":"pageSize","type":"text","marks":[{"type":"code_inline"}]},{"text":" then counts groups, not rows.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"The group is a bare array of rows","type":"text","marks":[{"type":"strong"}]},{"text":", not an object: ","type":"text"},{"text":"rows","type":"text","marks":[{"type":"code_inline"}]},{"text":" in Handlebars (","type":"text"},{"text":"{{rows.0.\u003cfield>}}","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"{{#each rows}}","type":"text","marks":[{"type":"code_inline"}]},{"text":"), ","type":"text"},{"text":"$[0].\u003cfield>","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"$[*].\u003cfield>","type":"text","marks":[{"type":"code_inline"}]},{"text":" in Mapper 2.0 and Transformation 2.0 (","type":"text"},{"text":"$.\u003cfield>","type":"text","marks":[{"type":"code_inline"}]},{"text":" resolves nothing), and an array in hooks. A document preview shows it as a ","type":"text"},{"text":"group","type":"text","marks":[{"type":"code_inline"}]},{"text":" stage after ","type":"text"},{"text":"parse","type":"text","marks":[{"type":"code_inline"}]},{"text":". A one-to-many step over it leaves ","type":"text"},{"text":"pathToMany","type":"text","marks":[{"type":"code_inline"}]},{"text":" blank.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A webhook listener","type":"text","marks":[{"type":"strong"}]},{"text":" makes one record of each element of a top-level JSON array and one record of an object (unless ","type":"text"},{"text":"webhook.pathToRecords","type":"text","marks":[{"type":"code_inline"}]},{"text":" names an inner array), then pages each request's records at ","type":"text"},{"text":"pageSize","type":"text","marks":[{"type":"code_inline"}]},{"text":" (default 20) before any hook or grouping runs. ","type":"text"},{"text":"groupByFields","type":"text","marks":[{"type":"code_inline"}]},{"text":" there merges only adjacent rows within a page, so keep one request's rows together by having the sender post an object, or by raising ","type":"text"},{"text":"pageSize","type":"text","marks":[{"type":"code_inline"}]},{"text":" above the largest request.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Source-side transform vs destination-side mapping","type":"text"}]},{"type":"paragraph","content":[{"text":"Both reshape data, but in opposite directions:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A ","type":"text"},{"text":"transform","type":"text","marks":[{"type":"strong"}]},{"text":" on a source export reshapes records as they enter the flow -- flattening nested responses, or aligning multiple sources to a common shape (see ","type":"text"},{"text":"Export Execution Pipeline","type":"text","marks":[{"type":"link","attrs":{"href":"#export-execution-pipeline","title":null}}]},{"text":").","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A ","type":"text"},{"text":"mapping","type":"text","marks":[{"type":"strong"}]},{"text":" on a downstream import reshapes records as they leave the flow toward a destination.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Don't add a transform to \"match a destination\" -- that's the destination import mapping's job. Transforms are for entry reshaping; mappings are for exit reshaping.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Async APIs (submit, poll, fetch)","type":"text"}]},{"type":"paragraph","content":[{"text":"Most APIs return data in the same call and need none of this. Some APIs only ","type":"text"},{"text":"acknowledge","type":"text","marks":[{"type":"em"}]},{"text":" a request (an HTTP 202, a job ticket, a feed or document id) and process it in the background -- Amazon SP-API feeds, large report generators, bulk extract and file-conversion jobs. For those, attach an ","type":"text"},{"text":"async helper","type":"text","marks":[{"type":"strong"}]},{"text":" to the export via ","type":"text"},{"text":"http._asyncHelperId","type":"text","marks":[{"type":"code_inline"}]},{"text":". The helper is its own resource (","type":"text"},{"text":"/v1/asynchelpers","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"celigo async-helpers","type":"text","marks":[{"type":"code_inline"}]},{"text":") that teaches the step the submit-poll-fetch pattern and does nothing until a step points at it. It bundles three pieces:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A status export","type":"text","marks":[{"type":"strong"}]},{"text":" (required) -- run on each poll to ask \"is it done yet?\". Configure the ","type":"text"},{"text":"status path","type":"text","marks":[{"type":"strong"}]},{"text":" to read in the response, the case-sensitive ","type":"text"},{"text":"in-progress / done / done-without-data / error","type":"text","marks":[{"type":"strong"}]},{"text":" value lists (taken from the API's docs), and the ","type":"text"},{"text":"initial wait","type":"text","marks":[{"type":"strong"}]},{"text":" and ","type":"text"},{"text":"poll wait","type":"text","marks":[{"type":"strong"}]},{"text":" intervals in minutes.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A result export","type":"text","marks":[{"type":"strong"}]},{"text":" (optional, usually present) -- fetches the final payload once status reports done.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Initial-submission handling","type":"text","marks":[{"type":"strong"}]},{"text":" -- where to find the job ticket in the first acknowledgement: \"same as status\" when the acknowledgement is itself shaped like a status response, otherwise a resource path (plus transform rules for non-JSON acknowledgements, e.g. Amazon's XML).","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Two constraints shape the design: the status and result exports must be ordinary synchronous exports (an async helper cannot nest another), and the async-configured step ","type":"text"},{"text":"cannot carry its own transform, output filter, or preSavePage hook","type":"text","marks":[{"type":"strong"}]},{"text":" -- put any reshaping or filtering on the dedicated result export instead. The same pattern applies symmetrically to imports writing to asynchronous destinations (","type":"text"},{"text":"_asyncHelperId","type":"text","marks":[{"type":"code_inline"}]},{"text":" on the import).","type":"text"}]},{"type":"paragraph","content":[{"text":"Build it bottom-up: the status export (and result export) first, then the helper that names them (","type":"text"},{"text":"http.status._exportId","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"http.result._exportId","type":"text","marks":[{"type":"code_inline"}]},{"text":"; ","type":"text"},{"text":"celigo async-helpers create","type":"text","marks":[{"type":"code_inline"}]},{"text":"), then ","type":"text"},{"text":"http._asyncHelperId","type":"text","marks":[{"type":"code_inline"}]},{"text":" on the step. A helper's delete is refused while any step still references it (","type":"text"},{"text":"celigo async-helpers dependencies \u003cid>","type":"text","marks":[{"type":"code_inline"}]},{"text":" lists them), and clearing a step's reference leaves the helper and its exports in place.","type":"text"}]},{"type":"paragraph","content":[{"text":"Reach for an async helper only when the API genuinely forces the fire-and-check-back shape. Adding one to a synchronous API is pure overhead -- extra polling plus a status and result export to maintain.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"GraphQL APIs -- envelope, cursor paging, errors with HTTP 200","type":"text"}]},{"type":"paragraph","content":[{"text":"A GraphQL export is an ordinary ","type":"text"},{"text":"HTTPExport","type":"text","marks":[{"type":"code_inline"}]},{"text":": ","type":"text"},{"text":"http.method: \"POST\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" and an ","type":"text"},{"text":"http.body","type":"text","marks":[{"type":"code_inline"}]},{"text":" holding the JSON envelope ","type":"text"},{"text":"{\"query\", \"operationName\", \"variables\"}","type":"text","marks":[{"type":"code_inline"}]},{"text":" serialized as one string (the schema types ","type":"text"},{"text":"http.body","type":"text","marks":[{"type":"code_inline"}]},{"text":" as a string, not an object), with ","type":"text"},{"text":"http.formType: \"graph_ql\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" (or ","type":"text"},{"text":"assistant_graphql","type":"text","marks":[{"type":"code_inline"}]},{"text":", the connector-assisted GraphQL form). The form type is set on the step, and a step that carries it works on a plain ","type":"text"},{"text":"http","type":"text","marks":[{"type":"code_inline"}]},{"text":" connection too; a ","type":"text"},{"text":"graph_ql","type":"text","marks":[{"type":"code_inline"}]},{"text":" connection only adds the editor toggle in the UI (","type":"text"},{"text":"configuring-connections > Connection Types","type":"text","marks":[{"type":"link","attrs":{"href":"../configuring-connections/SKILL.md#connection-types","title":null}}]},{"text":").","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"The form type changes the request.","type":"text","marks":[{"type":"strong"}]},{"text":" Under ","type":"text"},{"text":"graph_ql","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"assistant_graphql","type":"text","marks":[{"type":"code_inline"}]},{"text":", a ","type":"text"},{"text":"variables","type":"text","marks":[{"type":"code_inline"}]},{"text":" value stored as a JSON string is parsed and sent as an object; under ","type":"text"},{"text":"http","type":"text","marks":[{"type":"code_inline"}]},{"text":" it goes out as a string, not the object a GraphQL request carries. Keep the GraphQL form type on any step whose ","type":"text"},{"text":"variables","type":"text","marks":[{"type":"code_inline"}]},{"text":" is a string.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Handlebars in ","type":"text","marks":[{"type":"strong"}]},{"text":"variables","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":".","type":"text","marks":[{"type":"strong"}]},{"text":" The body renders as one template and is then parsed as JSON. When every expression sits inside a string value, store ","type":"text"},{"text":"variables","type":"text","marks":[{"type":"code_inline"}]},{"text":" as an object. An unquoted expression -- a number, ","type":"text"},{"text":"null","type":"text","marks":[{"type":"code_inline"}]},{"text":", a ","type":"text"},{"text":"{{#if}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" block -- needs ","type":"text"},{"text":"variables","type":"text","marks":[{"type":"code_inline"}]},{"text":" as a JSON string, which also keeps the body parseable at rest (otherwise the editor shows every field empty): ","type":"text"},{"text":"\"variables\": \"{\\\"first\\\": 50, \\\"after\\\": {{#if export.http.paging.token}}\\\"{{export.http.paging.token}}\\\"{{else}}null{{/if}} }\"","type":"text","marks":[{"type":"code_inline"}]},{"text":". A lookup's record values inside such a string: ","type":"text"},{"text":"configuring-imports > GraphQL imports","type":"text","marks":[{"type":"link","attrs":{"href":"../configuring-imports/SKILL.md#graphql-imports","title":null}}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Cursor paging.","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"http.paging.method: \"token\"","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"paging.path","type":"text","marks":[{"type":"code_inline"}]},{"text":" at the cursor (","type":"text"},{"text":"data.orders.pageInfo.endCursor","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"pathLocation: \"body\"","type":"text","marks":[{"type":"code_inline"}]},{"text":"), the token in ","type":"text"},{"text":"variables","type":"text","marks":[{"type":"code_inline"}]},{"text":" as above so the first page sends ","type":"text"},{"text":"null","type":"text","marks":[{"type":"code_inline"}]},{"text":", and ","type":"text"},{"text":"http.response.resourcePath","type":"text","marks":[{"type":"code_inline"}]},{"text":" at the records (","type":"text"},{"text":"data.orders.edges","type":"text","marks":[{"type":"code_inline"}]},{"text":"). Paging stops when the cursor comes back empty or null. ","type":"text"},{"text":"lastPagePath","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"lastPageValues","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"data.orders.pageInfo.hasNextPage","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"[\"false\"]","type":"text","marks":[{"type":"code_inline"}]},{"text":") stop token paging only together with ","type":"text"},{"text":"lastPageStatusCode: 200","type":"text","marks":[{"type":"code_inline"}]},{"text":"; without it the export sends one more request with the last cursor.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Errors come back as HTTP 200.","type":"text","marks":[{"type":"strong"}]},{"text":" A GraphQL server usually reports a failed query with 200, an ","type":"text"},{"text":"errors","type":"text","marks":[{"type":"code_inline"}]},{"text":" array and ","type":"text"},{"text":"data: null","type":"text","marks":[{"type":"code_inline"}]},{"text":", which the export reads as success: zero records, no error. Set ","type":"text"},{"text":"http.response.failPath","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"errorPath","type":"text","marks":[{"type":"code_inline"}]},{"text":" to ","type":"text"},{"text":"errors[0].message","type":"text","marks":[{"type":"code_inline"}]},{"text":" to fail the run with the server's message. ","type":"text"},{"text":"failPath: \"errors\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" catches a response with exactly one error and lets two or more through.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Connector GraphQL APIs.","type":"text","marks":[{"type":"strong"}]},{"text":" A connector flags GraphQL with ","type":"text"},{"text":"isGraphQL","type":"text","marks":[{"type":"code_inline"}]},{"text":", on the connector or on one of its ","type":"text"},{"text":"apis[]","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"celigo http-connectors get \u003cid>","type":"text","marks":[{"type":"code_inline"}]},{"text":"). Bind the connection to that API (","type":"text"},{"text":"http._httpConnectorApiId","type":"text","marks":[{"type":"code_inline"}]},{"text":") and its base URI is the GraphQL endpoint. A connector doesn't necessarily preset a fail path, so read the step's ","type":"text"},{"text":"http.response","type":"text","marks":[{"type":"code_inline"}]},{"text":" back and add one.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"The endpoint belongs to the connection.","type":"text","marks":[{"type":"strong"}]},{"text":" The GraphQL editor owns ","type":"text"},{"text":"http.relativeURI","type":"text","marks":[{"type":"code_inline"}]},{"text":": it shows no field for it and rewrites it whenever the export is saved there, so the GraphQL endpoint must be the connection's base URI -- a ","type":"text"},{"text":"graph_ql","type":"text","marks":[{"type":"code_inline"}]},{"text":" connection whose base URI is the endpoint, or a connector connection bound to the connector's GraphQL API. When the connection's base URI stops at an API root, move the export to a connection that supplies the GraphQL endpoint instead of patching ","type":"text"},{"text":"relativeURI","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"CLI Commands","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# CRUD\nceligo exports list\nceligo exports get \u003cid>\nceligo exports create \u003c export.json\nceligo exports update \u003cid> \u003c export.json\nceligo exports set \u003cid> key=value [key2=value2 ...]\nceligo exports delete \u003cid>\n\n# Invoke: with \u003cid>, run the saved export against the live source (no job, nothing saved);\n# with a piped document, preview it stage by stage -- parse, transform, output filter, preSavePage --\n# returning only the first record unless the document sets test.limit\nceligo exports invoke [id] [--all]\n# Run the saved export (or a piped document) through another connection: records only, {data, dataURIs}\nceligo exports invoke [id] --connection \u003cconnectionId>\n\n# Clone and connection management\necho '{\"connectionMap\":{\"oldConnId\":\"newConnId\"}}' | celigo exports clone \u003cid>\nceligo exports replace-connection \u003cid> \u003cnewConnectionId>\n\n# CDC listener: reset the stream position (refused for snapshotMode initial_only\n# unless --snapshot-mode changes the mode in the same save)\nceligo exports reset-cursor \u003cid> [--snapshot-mode when_needed]\n\n# Discovery\nceligo account search \"\u003ckeyword>\"\nceligo templates marketplace\nceligo templates preview \u003cid> --model Export\nceligo templates preview \u003cid> --summary\nceligo http-connectors list\nceligo http-connectors catalog \u003cid> --resource-type export --published-only\nceligo http-connectors endpoint-detail \u003cid> --resource-type export --resource-id \u003crid> --endpoint-id \u003cepid>\nceligo tp-connectors list\nceligo metadata types \u003cconnectionId>\nceligo metadata fields \u003cconnectionId> \u003centityType>\n\n# Debug\nceligo exports enable-debug \u003cid> [--duration \u003cminutes>]\nceligo exports disable-debug \u003cid>","type":"text"}]},{"type":"paragraph","content":[{"text":"\u003c!-- TIER:3 -->","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Pre-Submit Checklist","type":"text"}]},{"type":"paragraph","content":[{"text":"Before creating or updating an export, verify:","type":"text"}]},{"type":"checkbox_list","attrs":{"id":null},"content":[{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"adaptorType","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" is exact","type":"text","marks":[{"type":"strong"}]},{"text":" -- case-sensitive, matches the ","type":"text"},{"text":"Adaptor Decision Matrix","type":"text","marks":[{"type":"link","attrs":{"href":"#adaptor-decision-matrix","title":null}}]},{"text":" (e.g., ","type":"text"},{"text":"HTTPExport","type":"text","marks":[{"type":"code_inline"}]},{"text":", not ","type":"text"},{"text":"httpExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"HttpExport","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Pre-built connector was checked","type":"text","marks":[{"type":"strong"}]},{"text":" -- for HTTP exports, ","type":"text"},{"text":"celigo http-connectors list","type":"text","marks":[{"type":"code_inline"}]},{"text":" found no connector for the app (or the connector lacks the endpoint) before any hand-written ","type":"text"},{"text":"relativeURI","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"_connectionId","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" is valid","type":"text","marks":[{"type":"strong"}]},{"text":" -- points to an existing, online connection of the correct type. Not needed for ","type":"text"},{"text":"WebhookExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"SimpleExport","type":"text","marks":[{"type":"code_inline"}]},{"text":", and never set on ","type":"text"},{"text":"TopicExport","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Listener ","type":"text","marks":[{"type":"strong"}]},{"text":"type","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" is set","type":"text","marks":[{"type":"strong"}]},{"text":" -- ","type":"text"},{"text":"type: \"webhook\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" for ","type":"text"},{"text":"WebhookExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" (422 without it), ","type":"text"},{"text":"type: \"distributed\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" for NetSuite and Salesforce listeners, ","type":"text"},{"text":"type: \"stream\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" for CDC and topic listeners (a topic listener saved without it runs as a batch export), and the flow is created disabled, then enabled","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Adaptor config block is present","type":"text","marks":[{"type":"strong"}]},{"text":" -- ","type":"text"},{"text":"http{}","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"netsuite{}","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"ftp{}","type":"text","marks":[{"type":"code_inline"}]},{"text":", etc. matches the ","type":"text"},{"text":"adaptorType","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"resourcePath","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" or query is correct","type":"text","marks":[{"type":"strong"}]},{"text":" -- wrong path silently returns 0 records with no error","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Pagination is configured","type":"text","marks":[{"type":"strong"}]},{"text":" -- for HTTP exports, set ","type":"text"},{"text":"http.paging","type":"text","marks":[{"type":"code_inline"}]},{"text":" if the API returns paginated results","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Delta is configured for the adaptor","type":"text","marks":[{"type":"strong"}]},{"text":" -- ","type":"text"},{"text":"type: \"delta\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" is set (with no ","type":"text"},{"text":"type","type":"text","marks":[{"type":"code_inline"}]},{"text":", the delta variables render empty); HTTP and SQL: ","type":"text"},{"text":"{{lastExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" (not ","type":"text"},{"text":"{{currentExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":") as the lower bound in the request, and no ","type":"text"},{"text":"delta.dateField","type":"text","marks":[{"type":"code_inline"}]},{"text":" on HTTP; application adaptors: ","type":"text"},{"text":"delta.dateField","type":"text","marks":[{"type":"code_inline"}]},{"text":". ","type":"text"},{"text":"delta.startDate","type":"text","marks":[{"type":"code_inline"}]},{"text":" is set, or the first run exports nothing","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"File parsing matches the format","type":"text","marks":[{"type":"strong"}]},{"text":" -- if file-based, ","type":"text"},{"text":"file.type","type":"text","marks":[{"type":"code_inline"}]},{"text":" matches the actual file format (csv, json, xml, xlsx, edi)","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Files that must stay are kept","type":"text","marks":[{"type":"strong"}]},{"text":" -- a file export deletes each file it processed unless ","type":"text"},{"text":"file.skipDelete","type":"text","marks":[{"type":"code_inline"}]},{"text":" is ","type":"text"},{"text":"true","type":"text","marks":[{"type":"code_inline"}]},{"text":"; ","type":"text"},{"text":"backupDirectoryPath","type":"text","marks":[{"type":"code_inline"}]},{"text":" moves them away instead","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"A webhook secret is real","type":"text","marks":[{"type":"strong"}]},{"text":" -- a generated value of 32+ random characters handed to the user, never a placeholder, with the listener URL ","type":"text"},{"text":"https://\u003capi host>/v1/exports/\u003cexportId>/data","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"mockOutput","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" format is correct","type":"text","marks":[{"type":"strong"}]},{"text":" -- ","type":"text"},{"text":"{ \"page_of_records\": [{ \"record\": {...} }] }","type":"text","marks":[{"type":"code_inline"}]},{"text":", not a plain array","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"No ","type":"text","marks":[{"type":"strong"}]},{"text":"rest:","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" block","type":"text","marks":[{"type":"strong"}]},{"text":" -- ","type":"text"},{"text":"rest:","type":"text","marks":[{"type":"code_inline"}]},{"text":" creates a legacy RESTExport. Use only ","type":"text"},{"text":"http:","type":"text","marks":[{"type":"code_inline"}]},{"text":" for new exports","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Output filter syntax is valid","type":"text","marks":[{"type":"strong"}]},{"text":" -- if using an output filter expression, test it against sample data","type":"text"}]}]},{"type":"checkbox_item","attrs":{"checked":false},"content":[{"type":"paragraph","content":[{"text":"Lookup config is complete","type":"text","marks":[{"type":"strong"}]},{"text":" -- if ","type":"text"},{"text":"isLookup: true","type":"text","marks":[{"type":"code_inline"}]},{"text":", ensure response mapping is planned for the flow's ","type":"text"},{"text":"pageProcessors[]","type":"text","marks":[{"type":"code_inline"}]},{"text":" entry, and decide what a lookup that finds nothing, or more than one match, should do","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Gotchas","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"PUT erases omitted fields.","type":"text","marks":[{"type":"strong"}]},{"text":" Always GET first, modify, then PUT; ","type":"text"},{"text":"celigo exports set","type":"text","marks":[{"type":"code_inline"}]},{"text":" does this for every field except ","type":"text"},{"text":"assistantMetadata","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"debugUntil","type":"text","marks":[{"type":"code_inline"}]},{"text":", which it PATCHes. The export PATCH route accepts only those two paths, so every other change -- a URI, a filter, a delimiter -- is a full update. One read hides data: an ","type":"text"},{"text":"HTTPExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" saved with an empty ","type":"text"},{"text":"http.relativeURI","type":"text","marks":[{"type":"code_inline"}]},{"text":" and no ","type":"text"},{"text":"http.body","type":"text","marks":[{"type":"code_inline"}]},{"text":" reads back with no ","type":"text"},{"text":"http","type":"text","marks":[{"type":"code_inline"}]},{"text":" block, although the platform still stores and runs it. Don't build an update from that read (","type":"text"},{"text":"exports set","type":"text","marks":[{"type":"code_inline"}]},{"text":" included) -- it erases the stored request -- so send the complete ","type":"text"},{"text":"http","type":"text","marks":[{"type":"code_inline"}]},{"text":" block yourself. An empty relative URI calls the connection's base URI exactly; ","type":"text"},{"text":"\"/\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" adds a trailing slash, which is a different URL.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Including a ","type":"text","marks":[{"type":"strong"}]},{"text":"rest:","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" block creates a legacy RESTExport.","type":"text","marks":[{"type":"strong"}]},{"text":" Use only ","type":"text"},{"text":"http:","type":"text","marks":[{"type":"code_inline"}]},{"text":" for new exports.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Wrong ","type":"text","marks":[{"type":"strong"}]},{"text":"resourcePath","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" produces 0 records with no error.","type":"text","marks":[{"type":"strong"}]},{"text":" First thing to check when an export succeeds but returns nothing.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"mockOutput","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" format is ","type":"text","marks":[{"type":"strong"}]},{"text":"{ \"page_of_records\": [{ \"record\": {...} }] }","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":".","type":"text","marks":[{"type":"strong"}]},{"text":" Not a plain array.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"HTTP delta exports use Handlebars","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"{{lastExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" in ","type":"text"},{"text":"relativeURI","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- double braces, which URL-encode the value), not ","type":"text"},{"text":"delta.dateField","type":"text","marks":[{"type":"code_inline"}]},{"text":", which an HTTP export refuses with 422 \"Unsupported dateField in export.delta.dateField\". The ","type":"text"},{"text":"delta","type":"text","marks":[{"type":"code_inline"}]},{"text":" object is optional there, but ","type":"text"},{"text":"type: \"delta\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" isn't -- with no ","type":"text"},{"text":"type","type":"text","marks":[{"type":"code_inline"}]},{"text":" the token renders empty on every run -- and set ","type":"text"},{"text":"delta.startDate","type":"text","marks":[{"type":"code_inline"}]},{"text":": without it the first run's window is empty. See ","type":"text"},{"text":"Delta windows and the checkpoint","type":"text","marks":[{"type":"link","attrs":{"href":"#delta-windows-and-the-checkpoint","title":null}}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A NetSuite RESTlet export runs a saved search or an ad-hoc one.","type":"text","marks":[{"type":"strong"}]},{"text":" A saved search goes in ","type":"text"},{"text":"netsuite.restlet.searchId","type":"text","marks":[{"type":"code_inline"}]},{"text":" (its internal or script id; ","type":"text"},{"text":"celigo metadata types \u003cconnectionId>","type":"text","marks":[{"type":"code_inline"}]},{"text":" lists them). An ad-hoc search sets ","type":"text"},{"text":"searchId: \"-1\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" with ","type":"text"},{"text":"recordType","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"criteria","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"columns","type":"text","marks":[{"type":"code_inline"}]},{"text":", plus ","type":"text"},{"text":"netsuite.skipGrouping: true","type":"text","marks":[{"type":"code_inline"}]},{"text":" (without it the run fails with ","type":"text"},{"text":"[search_not_sorted]","type":"text","marks":[{"type":"code_inline"}]},{"text":"). Criteria rows are always ANDed, so put OR logic in one ","type":"text"},{"text":"formulanumeric","type":"text","marks":[{"type":"code_inline"}]},{"text":" criterion; inside criteria the incoming record is the Handlebars root: name the field as the record carries it, with no ","type":"text"},{"text":"record.","type":"text","marks":[{"type":"code_inline"}]},{"text":" prefix (","type":"text"},{"text":"{{customer_ref}}","type":"text","marks":[{"type":"code_inline"}]},{"text":", not ","type":"text"},{"text":"{{record.customer_ref}}","type":"text","marks":[{"type":"code_inline"}]},{"text":"). A path the record doesn't carry fails it before any search, with ","type":"text"},{"text":"handlebars_template_parse_error","type":"text","marks":[{"type":"code_inline"}]},{"text":" (\"Failed to generate criteria.0.searchValue from template\"): fix the template, then retry.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"File exports require the ","type":"text","marks":[{"type":"strong"}]},{"text":"file{}","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" block.","type":"text","marks":[{"type":"strong"}]},{"text":" Without it, file-based exports return raw bytes instead of parsed records.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A webhook export needs ","type":"text","marks":[{"type":"strong"}]},{"text":"type: \"webhook\"","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":", not a connection.","type":"text","marks":[{"type":"strong"}]},{"text":" Without the type, the save is refused with 422 \"_connectionId is required unless type is [webhook|simple]\". A ","type":"text"},{"text":"_connectionId","type":"text","marks":[{"type":"code_inline"}]},{"text":" on a webhook export is harmless: it saves and updates without error.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Listeners need ","type":"text","marks":[{"type":"strong"}]},{"text":"type: \"distributed\"","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" at the top level and on the adaptor block","type":"text","marks":[{"type":"strong"}]},{"text":" (","type":"text"},{"text":"netsuite.type","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"salesforce.type","type":"text","marks":[{"type":"code_inline"}]},{"text":"), plus the adaptor's ","type":"text"},{"text":"distributed","type":"text","marks":[{"type":"code_inline"}]},{"text":" object. Nothing is set on the connection; check the SuiteApp/bundle or Salesforce package with ","type":"text"},{"text":"celigo connections ping --distributed \u003cconnectionId>","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"type: \"once\"","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" needs a dedicated, writeable tracking flag.","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"once.booleanField","type":"text","marks":[{"type":"code_inline"}]},{"text":" must be writeable by the export's connection, and no other process may update the same field -- a shared flag causes records to be skipped.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"An async-helper export cannot carry its own transform, output filter, or preSavePage hook.","type":"text","marks":[{"type":"strong"}]},{"text":" Build that processing into the helper's result export instead. The status and result exports must themselves be plain synchronous exports -- an async helper cannot nest another. See ","type":"text"},{"text":"Async APIs (submit, poll, fetch)","type":"text","marks":[{"type":"link","attrs":{"href":"#async-apis-submit-poll-fetch","title":null}}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"NetSuite REST exports need ","type":"text","marks":[{"type":"strong"}]},{"text":"adaptorType: \"NetSuiteHTTPExport\"","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" and ","type":"text","marks":[{"type":"strong"}]},{"text":"nsDomainType","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" sent explicitly.","type":"text","marks":[{"type":"strong"}]},{"text":" On a NetSuite connection the server otherwise infers ","type":"text"},{"text":"NetSuiteExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" and rejects the document (422 \"netsuite subschema not defined\"). ","type":"text"},{"text":"http.relativeURI","type":"text","marks":[{"type":"code_inline"}]},{"text":" carries the full path starting at ","type":"text"},{"text":"/services/rest/...","type":"text","marks":[{"type":"code_inline"}]},{"text":", and the account needs NetSuite's REST Web Services feature (403 ","type":"text"},{"text":"FEATURE_DISABLED","type":"text","marks":[{"type":"code_inline"}]},{"text":" otherwise -- fall back to a saved search). The record list endpoint returns only ids and links; for records with fields use SuiteQL: ","type":"text"},{"text":"POST /services/rest/query/v1/suiteql?limit=\u003cn>&offset={{{export.http.paging.skip}}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" with header ","type":"text"},{"text":"Prefer: transient","type":"text","marks":[{"type":"code_inline"}]},{"text":", body ","type":"text"},{"text":"{\"q\": \"\u003cSuiteQL>\"}","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"http.paging.method: \"skip\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"http.response.resourcePath: \"items\"","type":"text","marks":[{"type":"code_inline"}]},{"text":". SuiteQL columns are the Records Catalog's lowercase names (","type":"text"},{"text":"entityid","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"lastmodifieddate","type":"text","marks":[{"type":"code_inline"}]},{"text":"). Don't convert an existing ","type":"text"},{"text":"NetSuiteExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" to REST unless asked.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A document preview runs the export's whole pipeline.","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"exports invoke","type":"text","marks":[{"type":"code_inline"}]},{"text":" with a piped document (","type":"text"},{"text":"POST /v1/exports/preview","type":"text","marks":[{"type":"code_inline"}]},{"text":") applies the transform, the output filter and the preSavePage hook, in that order, each as a stage after ","type":"text"},{"text":"parse","type":"text","marks":[{"type":"code_inline"}]},{"text":": ","type":"text"},{"text":"transformTwoDotZero","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"transform","type":"text","marks":[{"type":"code_inline"}]},{"text":" for version 1 rules), ","type":"text"},{"text":"outputFilter","type":"text","marks":[{"type":"code_inline"}]},{"text":", where each record it drops carries ","type":"text"},{"text":"isIgnored: true","type":"text","marks":[{"type":"code_inline"}]},{"text":", and ","type":"text"},{"text":"preSavePageHook","type":"text","marks":[{"type":"code_inline"}]},{"text":", which receives only the records the filter passed. Top-level ","type":"text"},{"text":"data","type":"text","marks":[{"type":"code_inline"}]},{"text":" is what the last of them returned. A preview body that still carries ","type":"text"},{"text":"mockOutput","type":"text","marks":[{"type":"code_inline"}]},{"text":", as a document read from the API does, returns that snapshot as a single ","type":"text"},{"text":"parse","type":"text","marks":[{"type":"code_inline"}]},{"text":" stage instead of calling the source: remove ","type":"text"},{"text":"mockOutput","type":"text","marks":[{"type":"code_inline"}]},{"text":" to read live, or invoke the saved export by id, which reads live.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A NetSuite listener fires only for the saves it's scoped to.","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"netsuite.distributed.executionContext","type":"text","marks":[{"type":"code_inline"}]},{"text":" defaults to ","type":"text"},{"text":"userinterface","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"webstore","type":"text","marks":[{"type":"code_inline"}]},{"text":", and ","type":"text"},{"text":"executionType","type":"text","marks":[{"type":"code_inline"}]},{"text":" to ","type":"text"},{"text":"create","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"edit","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"xedit","type":"text","marks":[{"type":"code_inline"}]},{"text":" (inline edit). Records a Celigo NetSuite import writes arrive in the ","type":"text"},{"text":"restlet","type":"text","marks":[{"type":"code_inline"}]},{"text":" context: list it to catch them, leave it out to ignore them. Bulk contexts such as ","type":"text"},{"text":"csvimport","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"custommassupdate","type":"text","marks":[{"type":"code_inline"}]},{"text":" turn each record of a mass update into its own event, so add them only when asked.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A NetSuite listener sends body fields only","type":"text","marks":[{"type":"strong"}]},{"text":", plus the sublists named in ","type":"text"},{"text":"netsuite.distributed.sublists","type":"text","marks":[{"type":"code_inline"}]},{"text":" (each one an extra query inside every save); ","type":"text"},{"text":"forceReload","type":"text","marks":[{"type":"code_inline"}]},{"text":" adds fields NetSuite fills after the save, such as new line ids. Its ","type":"text"},{"text":"qualifier","type":"text","marks":[{"type":"code_inline"}]},{"text":" rules nest in pairs (","type":"text"},{"text":"[A, \"and\", [B, \"and\", C]]","type":"text","marks":[{"type":"code_inline"}]},{"text":") and can't use sublist fields. To stop a two-way sync from looping, have the write-back import check the listener's ","type":"text"},{"text":"skipExportFieldId","type":"text","marks":[{"type":"code_inline"}]},{"text":" checkbox in the same save: the listener skips that save and clears the box. A scheduled export has no such field, so exclude the writer's own saves in its criteria instead.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A Salesforce listener with a ","type":"text","marks":[{"type":"strong"}]},{"text":"qualifier","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" receives no delete events.","type":"text","marks":[{"type":"strong"}]},{"text":" The package runs the qualifier as a query against the live record, which a delete removed. Leave it unset when deletes must reach the flow, and drop unwanted events with the output filter.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"XML sources need ","type":"text","marks":[{"type":"strong"}]},{"text":"listNodes","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":" for repeating elements.","type":"text","marks":[{"type":"strong"}]},{"text":" With custom parsing (","type":"text"},{"text":"parsers[].rules.V0_json: false","type":"text","marks":[{"type":"code_inline"}]},{"text":"), an element that occurs once parses as an object and more than once as an array, so list every element that can repeat in ","type":"text"},{"text":"listNodes","type":"text","marks":[{"type":"code_inline"}]},{"text":", as its full path from the record element. An empty element parses to ","type":"text"},{"text":"{}","type":"text","marks":[{"type":"code_inline"}]},{"text":", which renders ","type":"text"},{"text":"[object Object]","type":"text","marks":[{"type":"code_inline"}]},{"text":" and passes ","type":"text"},{"text":"{{#if}}","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A document preview reads one record and no delta window.","type":"text","marks":[{"type":"strong"}]},{"text":" Without ","type":"text"},{"text":"test.limit","type":"text","marks":[{"type":"code_inline"}]},{"text":" in the document, ","type":"text"},{"text":"exports invoke","type":"text","marks":[{"type":"code_inline"}]},{"text":" with a document returns only the first record (","type":"text"},{"text":"pageSize","type":"text","marks":[{"type":"code_inline"}]},{"text":" doesn't change that), while invoking the saved export by id returns them all. A delta export's ","type":"text"},{"text":"{{lastExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"{{currentExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" render empty in a document preview unless the document carries both in ","type":"text"},{"text":"postData","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"{\"lastExportDateTime\": \"2026-01-15T00:00:00.000Z\", \"currentExportDateTime\": \"2026-01-16T00:00:00.000Z\"}","type":"text","marks":[{"type":"code_inline"}]},{"text":"), which then render as given. A blob export's preview calls the source for real and returns a usable ","type":"text"},{"text":"{blobKey, fileMeta}","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A NetSuite RESTlet export stopped by NetSuite's per-call limit needs a smaller ","type":"text","marks":[{"type":"strong"}]},{"text":"netsuite.restlet.batchSize","type":"text","marks":[{"type":"code_inline"},{"type":"strong"}]},{"text":".","type":"text","marks":[{"type":"strong"}]},{"text":" Each RESTlet call reads ","type":"text"},{"text":"batchSize","type":"text","marks":[{"type":"code_inline"}]},{"text":" records (default 1000), and NetSuite stops any single RESTlet execution that runs past 300 seconds (","type":"text"},{"text":"SSS_TIME_LIMIT_EXCEEDED","type":"text","marks":[{"type":"code_inline"}]},{"text":") or 5,000 usage units (","type":"text"},{"text":"SSS_USAGE_LIMIT_EXCEEDED","type":"text","marks":[{"type":"code_inline"}]},{"text":"). A saved search that grew in NetSuite -- more rows, or added columns and formulas -- can push a call over, and the run fails partway with nothing changed in Celigo. Lower ","type":"text"},{"text":"restlet.batchSize","type":"text","marks":[{"type":"code_inline"}]},{"text":" in a full update of the export (gotcha 1): halving it halves the rows every call reads. In a flow run the export's ","type":"text"},{"text":"pageSize","type":"text","marks":[{"type":"code_inline"}]},{"text":" doesn't size the call; it pages records to the next step. Changing the export's columns isn't the fix either: on a saved-search export, ","type":"text"},{"text":"restlet.columns","type":"text","marks":[{"type":"code_inline"}]},{"text":" adds columns after the search's own and never removes one, so every call does more work. If small batches still fail, the search's owner simplifies the search itself in NetSuite.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Common Errors","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Error","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Likely Cause","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Fix","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"404 Not Found","type":"text","marks":[{"type":"code_inline"}]},{"text":" on export invoke","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Wrong ","type":"text"},{"text":"relativeURI","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"resourcePath","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Verify the endpoint path against the API docs; check for missing path parameters","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"401 Unauthorized","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Connection credentials expired or invalid","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Run ","type":"text"},{"text":"celigo connections ping \u003cconnId>","type":"text","marks":[{"type":"code_inline"}]},{"text":"; re-authorize OAuth connections","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"0 records exported","type":"text","marks":[{"type":"code_inline"}]},{"text":" (no error)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Wrong ","type":"text"},{"text":"resourcePath","type":"text","marks":[{"type":"code_inline"}]},{"text":", an overly restrictive filter, or an empty delta window: ","type":"text"},{"text":"{{currentExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" used as the lower bound, or a first run without ","type":"text"},{"text":"delta.startDate","type":"text","marks":[{"type":"code_inline"}]},{"text":". On a GraphQL export, also an ","type":"text"},{"text":"errors","type":"text","marks":[{"type":"code_inline"}]},{"text":" answer with HTTP 200. On a file export, also no file in the folder (","type":"text"},{"text":"ftp.directoryPath","type":"text","marks":[{"type":"code_inline"}]},{"text":") that passes the name filters (","type":"text"},{"text":"fileNameStartsWith","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"fileNameEndsWith","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"file.filter","type":"text","marks":[{"type":"code_inline"}]},{"text":"): files that don't match are skipped before download, silently, and the name match is case-sensitive on most servers (","type":"text"},{"text":"ORDER_","type":"text","marks":[{"type":"code_inline"}]},{"text":" skips ","type":"text"},{"text":"order_123.csv","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Compare a file export's folder and name filters with where and how the files actually land (an EDI profile or file definition doesn't set them). Check ","type":"text"},{"text":"resourcePath","type":"text","marks":[{"type":"code_inline"}]},{"text":" and the delta token (","type":"text"},{"text":"Delta windows and the checkpoint","type":"text","marks":[{"type":"link","attrs":{"href":"#delta-windows-and-the-checkpoint","title":null}}]},{"text":"), test without output filter. Empty runs don't move the checkpoint, so once fixed, the next run catches up without a backfill. GraphQL: read a preview's ","type":"text"},{"text":"raw","type":"text","marks":[{"type":"code_inline"}]},{"text":" stage, and set ","type":"text"},{"text":"failPath","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"GraphQL APIs","type":"text","marks":[{"type":"link","attrs":{"href":"#graphql-apis----envelope-cursor-paging-errors-with-http-200","title":null}}]},{"text":")","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"422 \"Unsupported dateField in export.delta.dateField\"","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"delta.dateField","type":"text","marks":[{"type":"code_inline"}]},{"text":" on an HTTP export","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Remove it; put ","type":"text"},{"text":"{{lastExportDateTime}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" in the request","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"422 \"_connectionId is required unless type is [webhook|simple]\"","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"A ","type":"text"},{"text":"WebhookExport","type":"text","marks":[{"type":"code_inline"}]},{"text":" saved without ","type":"text"},{"text":"type: \"webhook\"","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Add ","type":"text"},{"text":"type: \"webhook\"","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Webhook deliveries get 403 ","type":"text"},{"text":"unable_to_run_flows","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Every flow the listener feeds is disabled or not fully configured","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Enable the flow. Refused deliveries weren't queued, so the sender has to resend them","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Webhook deliveries get 401","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"The signature or token doesn't match: wrong ","type":"text"},{"text":"key","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"algorithm","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"encoding","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"header","type":"text","marks":[{"type":"code_inline"}]},{"text":", or a prefix such as ","type":"text"},{"text":"sha256=","type":"text","marks":[{"type":"code_inline"}]},{"text":" in the header","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Configure the sender's exact scheme; a ","type":"text"},{"text":"custom","type":"text","marks":[{"type":"code_inline"}]},{"text":" provider compares the bare digest as sent","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Cannot read property of undefined","type":"text","marks":[{"type":"code_inline"}]},{"text":" in preSavePage","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Script assumes a field exists that is missing from some records","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Add null checks: ","type":"text"},{"text":"if (record.field)","type":"text","marks":[{"type":"code_inline"}]},{"text":" before access","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"mockOutput is invalid","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Wrong format -- used array instead of object","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"{ \"page_of_records\": [{ \"record\": {...} }] }","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"handlebars_template_parse_error","type":"text","marks":[{"type":"code_inline"}]},{"text":" (\"... not defined in the model\") previewing a lookup","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"A standalone preview binds no record, so a request that reads the record can't render","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Not a broken template: check the lookup in the flow builder's step preview or a flow test run","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Invalid adaptorType","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Case mismatch or typo","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use exact casing from the Adaptor Decision Matrix","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Connection is offline","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Connection failed health check","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Fix credentials, re-authorize, then ","type":"text"},{"text":"celigo connections ping \u003cid>","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Rate limit exceeded","type":"text","marks":[{"type":"code_inline"}]},{"text":" / ","type":"text"},{"text":"429","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"The source API's rate limit was exceeded","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"With auto-recover on (the default), the connection throttles and retries by itself; to cap it, lower the connection's ","type":"text"},{"text":"targetConcurrencyLevel","type":"text","marks":[{"type":"code_inline"}]},{"text":" (","type":"text"},{"text":"concurrencyLevel","type":"text","marks":[{"type":"code_inline"}]},{"text":" is platform-managed while auto-recover is on). If the API signals limits another way (another status code or a body field), set the connection's ","type":"text"},{"text":"http.rateLimit","type":"text","marks":[{"type":"code_inline"}]},{"text":". See ","type":"text"},{"text":"configuring-connections > Auto-recover rate-limit errors","type":"text","marks":[{"type":"link","attrs":{"href":"../configuring-connections/SKILL.md#auto-recover-rate-limit-errors","title":null}}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Timeout","type":"text","marks":[{"type":"code_inline"}]},{"text":" on large exports","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Query returns too much data or API is slow","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Add pagination, narrow the date range, or increase timeout settings","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"SSS_TIME_LIMIT_EXCEEDED","type":"text","marks":[{"type":"code_inline"}]},{"text":" or ","type":"text"},{"text":"SSS_USAGE_LIMIT_EXCEEDED","type":"text","marks":[{"type":"code_inline"}]},{"text":" on a NetSuite RESTlet export, partway through a run","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"One RESTlet call ran past NetSuite's per-call time or usage limit; each call reads ","type":"text"},{"text":"netsuite.restlet.batchSize","type":"text","marks":[{"type":"code_inline"}]},{"text":" records (default 1000)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Lower ","type":"text"},{"text":"netsuite.restlet.batchSize","type":"text","marks":[{"type":"code_inline"}]},{"text":" -- not ","type":"text"},{"text":"pageSize","type":"text","marks":[{"type":"code_inline"}]},{"text":", and not ","type":"text"},{"text":"restlet.columns","type":"text","marks":[{"type":"code_inline"}]},{"text":", which adds to a saved search's columns (","type":"text"},{"text":"gotcha 19","type":"text","marks":[{"type":"link","attrs":{"href":"#gotchas","title":null}}]},{"text":")","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"File parsing error","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"file.type","type":"text","marks":[{"type":"code_inline"}]},{"text":" doesn't match actual file format, or delimiter/encoding mismatch","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Verify ","type":"text"},{"text":"file.type","type":"text","marks":[{"type":"code_inline"}]},{"text":", check ","type":"text"},{"text":"file.csv.columnDelimiter","type":"text","marks":[{"type":"code_inline"}]},{"text":", ensure correct encoding","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Every record of a CSV file fails downstream on missing fields; each parsed record has one field named after the whole header line (","type":"text"},{"text":"{\"order_id;sku;qty\": \"1001;TEE-M;2\"}","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"The file's column delimiter changed, so the parser no longer splits the columns","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Read one error's retry data: the field name shows the file's delimiter. Set ","type":"text"},{"text":"file.csv.columnDelimiter","type":"text","marks":[{"type":"code_inline"}]},{"text":" to it. Retrying won't help: the errors' retry data holds the mis-parsed records. Reprocess the file instead -- the export removes what it read unless ","type":"text"},{"text":"file.skipDelete","type":"text","marks":[{"type":"code_inline"}]},{"text":" is set, so the file has to be dropped in the folder again before the flow runs -- or correct each error's retry data before retrying","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"\"Directory is not found\"","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"http.type: \"file\"","type":"text","marks":[{"type":"code_inline"}]},{"text":" on a file-storage connector (folder mode) with an API URL in ","type":"text"},{"text":"relativeURI","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Put the folder path in ","type":"text"},{"text":"relativeURI","type":"text","marks":[{"type":"code_inline"}]},{"text":"; to fetch one file per record from an API, use a blob lookup without ","type":"text"},{"text":"http.type: \"file\"","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Preview returns ","type":"text"},{"text":"{}","type":"text","marks":[{"type":"code_inline"}]},{"text":", or only ","type":"text"},{"text":"request","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"raw","type":"text","marks":[{"type":"code_inline"}]},{"text":" stages (no ","type":"text"},{"text":"parse","type":"text","marks":[{"type":"code_inline"}]},{"text":", no ","type":"text"},{"text":"data","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Zero records: nothing matched, or ","type":"text"},{"text":"resourcePath","type":"text","marks":[{"type":"code_inline"}]},{"text":" points at nothing -- an HTTP preview looks the same either way","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Not a failed preview. Read the ","type":"text"},{"text":"raw","type":"text","marks":[{"type":"code_inline"}]},{"text":" stage to see where the records really are; a NetSuite search that matches nothing returns ","type":"text"},{"text":"{}","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"400 \"Virtual exports for export type: webhook are not supported.\"","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"A document preview of a webhook listener (a NetSuite listener can't be previewed either)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Listeners can't be previewed: test one with a real event, or save a ","type":"text"},{"text":"mockOutput","type":"text","marks":[{"type":"code_inline"}]},{"text":" on it for flow test runs to replay","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"422 from a document preview whose message is the source's own error body, such as ","type":"text"},{"text":"{\"errors\":[{\"code\":\"resource_not_found\",...}]}","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"The source system answered the request with an error","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Fix the URI, credentials or permissions at the source. A 422 that names a setting, such as ","type":"text"},{"text":"Invalid operation: isnotempty.","type":"text","marks":[{"type":"code_inline"}]},{"text":", is the platform refusing the document instead: the preview runs the output filter, so a rule it refuses, or a value an ","type":"text"},{"text":"epochtime","type":"text","marks":[{"type":"code_inline"}]},{"text":" rule can't read, fails the whole preview. The CLI prefixes every 422 with \"Validation error:\"","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"page_response_error","type":"text","marks":[{"type":"code_inline"}]},{"text":" \"Identical response received from next page request. No more page requests will be made.\"","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"A next page's response matched the previous one: the request doesn't change between pages (no paging variable in the URI or body), or the API repeats its last page","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"When the request lacks the paging variable, put ","type":"text"},{"text":"{{export.http.paging.page}}","type":"text","marks":[{"type":"code_inline"}]},{"text":" (or ","type":"text"},{"text":".skip","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":".token","type":"text","marks":[{"type":"code_inline"}]},{"text":") back in it; a stop rule (","type":"text"},{"text":"lastPagePath","type":"text","marks":[{"type":"code_inline"}]},{"text":" with ","type":"text"},{"text":"lastPageValues","type":"text","marks":[{"type":"code_inline"}]},{"text":", or ","type":"text"},{"text":"lastPageStatusCode","type":"text","marks":[{"type":"code_inline"}]},{"text":") would only end paging after the first page, and is for an API that repeats its last page. On an export with no ","type":"text"},{"text":"type","type":"text","marks":[{"type":"code_inline"}]},{"text":" the run completes with the pages already read; on a delta export it fails, nothing reaches the next steps and the checkpoint stays. After the fix, the next run reads every page (on a delta export, the whole window since the last run that exported records): say when that run is and what it will bring in, and offer to run it now","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Records changed just before a run are sometimes missing from a delta export","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"The source makes changes visible with a delay (indexing, replication), after the window that should have held them closed","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Set ","type":"text"},{"text":"delta.lagOffset","type":"text","marks":[{"type":"code_inline"}]},{"text":" to the delay: milliseconds taken off both ends of each window once the flow has a checkpoint, so each record reaches a run that much later, and while the value stays the same nothing is read twice. Changing it moves the next window's start by the change: raising it re-reads the added span once, lowering it skips the removed span. Never negative -- that moves the window into the future, and records changed just after each run are never exported","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Every call to a builder-mode API fails with 400 ","type":"text"},{"text":"api_pagination_source_not_paged","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"The API's first lookup, which drives its pagination, lost ","type":"text"},{"text":"http.paging","type":"text","marks":[{"type":"code_inline"}]},{"text":" (the export save accepts that)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Restore the lookup's paging, or turn off the API's ","type":"text"},{"text":"pagination.enabled","type":"text","marks":[{"type":"code_inline"}]},{"text":". Saving that export, even unchanged, also invalidates the cursors callers already hold","type":"text"}]}]}]}]}]},"metadata":{"date":"2026-11-02","name":"configuring-exports","author":"@skillopedia","source":{"stars":4,"repo_name":"ai","origin_url":"https://github.com/celigo/ai/blob/HEAD/skills/configuring-exports/SKILL.md","repo_owner":"celigo","body_sha256":"d878fce6421e6c5b2c714b58ba9c1a7f657cb8722ffcd238efbf7842dbedee91","cluster_key":"530879b4ff02318f170441300360fff614a32e6739f4ebdf66f2f06b0bb0d852","clean_bundle":{"format":"clean-skill-bundle-v1","source":"celigo/ai/skills/configuring-exports/SKILL.md","text_bytes":788907,"source_commit":"2f3853ed93eece259b8c78ee74ef19035dfae052","skills_sh_rank":6170,"attachment_count":46,"text_attachments":46,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"skills/configuring-exports/SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"v1","category":"integrations-apis","category_label":"Integrations"},"exact_dupes_collapsed_into_this":0},"version":"v1","category":"integrations-apis","import_tag":"skills-sh-top20k-2026-11","description":"Configure Celigo export resources -- the data source step that fetches records from external systems. Use when creating or editing exports, choosing the right adaptor type for a target application, setting up delta (incremental) exports, webhooks, file transfers, or lookup steps. To replicate tables into a data warehouse, use building-syncs."}},"renderedAt":1793671666988}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.