momentic-maintain — Skillopedia

Maintain Momentic tests Find the earliest point where a run entered a bad state, classify the cause from evidence, and repair the test when a durable test change is appropriate. Use MCP for targeted investigation, for a fresh classifier verdict, and for automated repair and verification. Do not weaken a test to hide an application bug, missing fixture, invalid credentials, outage, or Momentic defect. Report those causes instead of changing the test to pass. Common workflows - What happened on my branch? Call with the exact and bounded / . Start without a status filter, then narrow to , , or .…

\\t' read -r attempt url; do\n mkdir -p \"${RUN_INSPECTION_DIR}/attempts/${attempt}\"\n curl --fail --silent --show-error \"$url\" \\\n -o \"${RUN_INSPECTION_DIR}/attempt-${attempt}.zip\"\n unzip -q \"${RUN_INSPECTION_DIR}/attempt-${attempt}.zip\" \\\n -d \"${RUN_INSPECTION_DIR}/attempts/${attempt}\"\n done\n```\n\nInspect `metadata.json`, each attempt's `metadata.json`, screenshots, DOM\nsnapshots, console logs, HAR logs, video, resource usage, and crash archives.\nKeep attempt numbers intact. Do not commit the export response, run artifacts,\nsigned URLs, credentials, or customer data. Remove the temporary directory when\nthe investigation is complete.\n\n## Background\n\n### Test run result structure\n\nWhen momentic tests are run via the CLI, the results are stored in a \"run group\". The data for this run group is stored in a single directory within the momentic project. By default, the directory is called `test-results`, but can be changed in momentic project settings or on a single run of a run group. The run group results folder has the following structure:\n\n```\ntest-results/\n├── metadata.json data about the run group, including git metadata and timing info.\n└── runs/ On zip for each test run in the run group.\n ├── \u003crunId_1>.zip a zipped run directory containing data about this specific test run. Follows the structure described below.\n └── \u003crunId_2>.zip\n```\n\nWhen unzipped, run directories have the following structure:\n\n```\n\u003crunId>/\n├── metadata.json run-level metadata.\n└── attempts/\u003cn>/ one folder per attempt (1-based n).\n ├── metadata.json attempt outcome and step results.\n ├── console.json optional browser console output.\n └── assets/\n ├── \u003csnapshotId>.jpeg before/after screenshot for each step (see attempt metadata.json for snapshot ID).\n ├── \u003csnapshotId>.html before/after DOM snapshot for each step (see attempt metadata.json for snapshot ID).\n ├── har-pages.log HAR pages (ndjson).\n ├── har-entries.log HAR network entries (ndjson).\n ├── resource-usage.ndjson CPU/memory samples taken during the attempt.\n ├── \u003cvideoName> video recording (when video recording is enabled).\n └── browser-crash.zip browser crash dump (only present on crash).\n```\n\nWhen getting run results via the momentic MCP, tools such as `momentic_get_run` will return links to the MCP working directory (default `.momentic-mcp`). This directory will contain unzipped run result folders, following the structure above, named `run-result-\u003crunId>`.\n\n### Element locators\n\nCertain step types that interact with elements have a \"target\" property, or **locator**, that specifies which element the step should interact with.\n\n#### Locator caches\n\nLocators identify elements by sending the page state html/xml to an llm as well as a screenshot. The llm identifies which element on the page the user is referring to. Momentic will attempt to \"cache\" the answer from the llm so that future runs don't require AI calls. On future runs, the page state is checked against the cached element to determine whether the element is still usable, or the page has changed enough such that another AI call is required.\n\nA locator cache can bust for a variety of reasons:\n\n- the element description has changed, in which case we'll always bust the cache\n- the cached element could not be located in the current page state\n- the cached element was located in the page state, but fails certain checks specified on the cache entry, such as requiring a certain position, shape, or content.\n\nYou can find the `cacheBustReason` on the `trace` property in the results for a given step, but only when you explicitly request `includeTrace=true`. The `cache` property is also listed on the results, showing the full cache saved for that element.\n\n#### Identifying bad caches\n\nSometimes the element that was cached is not the element that the user intended to target. This can cause failures or unexpected behaviors in tests. In these cases, it helps to verify exactly why the wrong cache was saved in the first place. Only request `includeTrace=true` for these cache-debugging cases or when you suspect incorrect Momentic execution data. Use the `runId` property of the `targetUpdateLoggerTags` on the incorrect cache to get the details of the original run, calling `momentic_get_run` with this runId. This will return the run where the cache target was updated.\n\n### Module caching\n\nCached modules skip executing their steps when the module cache key and resolved inputs are unchanged, and reuse the cached return value from the module's last step.\n\nAuthentication modules can also save and restore browser auth state from the module cache, including cookies, localStorage, and IndexedDB. They may use a page-content check after restoring auth state to decide whether the cache is still valid.\n\n### File uploads\n\nA file upload step prepares one file for the next native file picker, so it must run before the action that opens the picker.\n\nSources can be remote URLs, `file://` references to earlier downloads, CLI-local paths, or uploaded user files. The step can also override the presented filename, and Momentic wires the prepared file into the browser's file chooser handling.\n\n## Using past runs\n\nPast runs are comparison evidence, not a substitute for reconstructing the current run. Use them when the current run does not answer:\n\n- When did this test start failing?\n- What differed vs the last passing run?\n- Did the same action behave differently on an earlier run?\n- Is this a test weakness, an application change, a real application bug, or a temporary slowdown?\n\nUse step results and screenshots on past runs to answer these questions. Do NOT rely only on summaries from `momentic_get_run` or `momentic_list_runs` to understand what happened in a test run. Look at the specific run details, including step results and screenshots, before citing a past run as evidence.\n\nWhen looking at past runs, use the following workflow:\n\n1. Call the `momentic_list_runs` tool to identify the runs you want more detail on. Always pass `gitBranchName` when it exists on the run in question. Omit it when you need runs from other branches.\n2. Call `momentic_get_run` for that specific run to get the run details.\n3. Call `momentic_get_step_result` for the same step/container or closest equivalent you are comparing, especially for screenshots.\n\nWhen past runs are irrelevant because the current run already proves the root cause, say that briefly instead of forcing historical evidence.\n\n### Multi-attempt runs\n\nWhen `momentic_list_runs` shows a passing run with `attempts > 1`, treat it as a partial failure worth investigating, not a clean passing run. Use the `attemptNumber` parameter to retrieve earlier failed attempt results for that run to understand what was going wrong before the retry succeeded.\n\n### Flakiness and intermittent failures\n\n- Call a test flaky only when the same app and test behavior fails intermittently.\n One failure is insufficient; rule out application and test changes using run\n evidence before attributing the pattern to flakiness.\n\n### Test temporality\n\n- Any past results may not necessarily match today’s test file. The test may have changed, meaning the result was on a different version of the test.\n- You can call `get_test_steps_for_run` to help you determine if the test itself changed between runs, although note that this tool returns a _summary_ of each test step. If you suspect that specific details on certain steps have changed between test runs, full step details are included in the response from `momentic_get_step_result`; only request `includeTrace=true` when those fields and screenshots still are not enough.\n\n## Common failure modes to watch for\n\n- A setup module appears to pass but leaves the wrong page, overlay, filter, search, selected row, or side peek open. Classify from the step/container that left the bad postcondition, not only from the next step that failed.\n- A click reports success and targets the intended element, but the application does not transition to the intended state. Verify the post-state; do not assume the click worked because the locator was correct.\n- A weak global assertion such as \"page contains X\" passes because unrelated text on the page matches. The next step may then type or click in the wrong context.\n- A type step without a specific target can silently type nowhere useful if the preceding action failed to focus the intended field.\n- A locator or cache can be technically valid but semantically wrong. Check the interacted element and, for bad caches, inspect the original cache-update run from `targetUpdateLoggerTags.runId`.\n- A recovered step can hide the first failure. Inspect failed child steps inside recovered modules and compare the recovered final state to the next step's expected baseline.\n- A timeout is not automatically `INFRA`. First rule out missing data, wrong page state, changed app flow, bad locator/assertion, and setup failure.\n\n## Identifying related vs unrelated issues\n\n- Determine intent from the test name, description, and simplified steps. Treat\n failures outside that intent, including most setup/teardown failures, as\n unrelated. Relatedness qualifies bugs and changes; it does not change an\n `INFRA` failure into another category.\n\n## Bug vs change\n\n- Bug: something very clearly went wrong when it should not have, such as an error message appearing. It is obvious just by looking at a single step or two that this is a bug.\n- Change: a clear change in the application behavior that you can prove through screenshots.\n\n## Recoverability\n\nAlong with the category, determine one recoverability value:\n\n- `RECOVERABLE` — The failure can be automatically fixed by updating the test itself so that future runs pass.\n - Examples: an application change that requires a test update; vague locators or assertions that can be rewritten to pass stably.\n- `ONE_TIME_RECOVERABLE` — The failure can be recovered for this specific run without persisting a test change.\n - Examples: a random modal that can be dismissed without affecting test purpose; a temporary delay where waiting or retrying would likely succeed.\n- `NON_RECOVERABLE` — The failure cannot be automatically addressed and requires manual intervention.\n - Examples: missing credentials; missing local files required for upload; outages likely caused by third-party systems where test steps cannot fix the issue.\n\n## Formal classification output\n\n- Exactly one category id — no new labels, no multi-label.\n- Ground your decision in data. Be sure that you've fully investigated the run before assigning the category.\n- Prefer human-readable references over UUIDs when the step/module can be identified colloquially: `module create-subpage-under-parent-page`, `the last invocation of module \u003cname>`, `substep 4 (0-indexed)`, `the failed setup assertion`, etc. Tool calls still require exact IDs, but final reasoning should be readable.\n- When referencing past runs in final output, use clickable Momentic URLs rather than bare UUIDs: `https://app.momentic.ai/runs/\u003crunId>`. Do not shorten UUIDs inside those URLs.\n- The reasoning must include the earliest divergent step/container and the broken postcondition it produced, not just the final failing step.\n\n```text\nReasoning: \u003ca few sentences tied to the earliest divergence, screenshots/traces, past runs if used, and test intent>\nCategory: \u003cone id from the list>\nRecoverable: \u003cRECOVERABLE | ONE_TIME_RECOVERABLE | NON_RECOVERABLE>\nConfidence: \u003chigh | medium | low>\n```\n\nConfidence levels:\n\n- `high` — direct evidence, such as a clear screenshot of a label change or crash\n- `medium` — strong inference from multiple signals but no single conclusive screenshot or data point\n- `low` — ambiguous evidence; the classification required significant inference or the root cause is unclear\n\n## Category ids\n\nUse these strings verbatim:\n\n- `NO_FAILURE` — The run had no failures; all attempts passed.\n- `APPLICATION_CHANGE` — The test is out of date because the application's flow or UI has changed; updating the test to match the new behavior would permanently fix the failure.\n- `BUG` — Something clearly went wrong in the application that shouldn't have, such as an error message appearing or expected content failing to render.\n- `TEST_AUTHORSHIP` — The test can be permanently updated to prevent the failure while still validating its original intent, and you can recommend a specific authorship change such as adding or modifying a step, rewriting a vague assertion, or making a locator description more specific. If you cannot name a concrete change, choose a different category. Timeouts, slow page loads, and any failure whose recommended fix is to \"wait longer\" or to increase a timeout are NOT authorship issues — those are `INFRA`, even when the test could technically be edited to wait longer.\n - Examples: race conditions that can be fixed by adding or modifying steps **other than** waits/timeouts (e.g. replacing a \"type with pressEnter\" step with an explicit \"select from list\" step so the test no longer races the application); vague assertions or locator descriptions that can be rewritten to be more specific.\n- `TEST_SETUP` — Missing test data or files necessary to run the test, where the fix requires user action outside of the test itself.\n - Examples: missing file for a file upload step; missing or incorrect credentials needed by the test.\n- `INFRA` — The failure was unrelated to the application or application code and was caused by an infrastructure outage, long load times, or some other issue due to outside factors.\n - Examples: browser crash; high resource usage; rate limiting; a step or assertion that timed out waiting for the page or application to reach a slow-but-eventual state.\n- `MOMENTIC_ISSUE` — Some issue occurred with the execution of the test or Momentic data was incorrect (e.g. cache is wrong, global locator redirect did something weird, AI hallucinations).\n - Examples: unexpected behavior when viewing the run trace; the AI clearly misread or hallucinated data that is unambiguous in the screenshot, and no reasonable test alternative exists to avoid the AI step.\n- `OTHER` — The failure doesn't fit any of the other categories.","attachment_filenames":[],"attachments":[],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"Maintain Momentic tests","type":"text"}]},{"type":"paragraph","content":[{"text":"Find the earliest point where a run entered a bad state, classify the cause from evidence, and repair the test when a durable test change is appropriate. Use MCP for targeted investigation, ","type":"text"},{"text":"momentic ai classify","type":"text","marks":[{"type":"code_inline"}]},{"text":" for a fresh classifier verdict, and ","type":"text"},{"text":"momentic ai triage","type":"text","marks":[{"type":"code_inline"}]},{"text":" for automated repair and verification.","type":"text"}]},{"type":"paragraph","content":[{"text":"Do not weaken a test to hide an application bug, missing fixture, invalid credentials, outage, or Momentic defect. Report those causes instead of changing the test to pass.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Common workflows","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"What happened on my branch?","type":"text","marks":[{"type":"strong"}]},{"text":" Call ","type":"text"},{"text":"momentic_list_runs","type":"text","marks":[{"type":"code_inline"}]},{"text":" with the exact ","type":"text"},{"text":"gitBranchName","type":"text","marks":[{"type":"code_inline"}]},{"text":" and bounded ","type":"text"},{"text":"start","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"end","type":"text","marks":[{"type":"code_inline"}]},{"text":". Start without a status filter, then narrow to ","type":"text"},{"text":"FAILED","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"CANCELLED","type":"text","marks":[{"type":"code_inline"}]},{"text":", or ","type":"text"},{"text":"recovered=true","type":"text","marks":[{"type":"code_inline"}]},{"text":". Open exact candidates and compare with main only for a concrete cross-branch question.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Why is main red?","type":"text","marks":[{"type":"strong"}]},{"text":" List ","type":"text"},{"text":"FAILED","type":"text","marks":[{"type":"code_inline"}]},{"text":" runs on the actual main branch over the incident window and paginate through every page needed for the claim. Narrow with ","type":"text"},{"text":"failureCategory","type":"text","marks":[{"type":"code_inline"}]},{"text":" when investigating a known category, but group symptoms only after inspecting representative runs and step evidence. Read each exact run's saved classification before optionally re-classifying it.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"De-flake and reduce recovery latency.","type":"text","marks":[{"type":"strong"}]},{"text":" Use ","type":"text"},{"text":"momentic_quarantine_list","type":"text","marks":[{"type":"code_inline"}]},{"text":" for quarantined tests, ","type":"text"},{"text":"recovered=true","type":"text","marks":[{"type":"code_inline"}]},{"text":" for successful runs that needed recovery, and ","type":"text"},{"text":"minAttempts","type":"text","marks":[{"type":"code_inline"}]},{"text":" for retry-heavy runs (","type":"text"},{"text":"2","type":"text","marks":[{"type":"code_inline"}]},{"text":" means at least one retry; ","type":"text"},{"text":"3","type":"text","marks":[{"type":"code_inline"}]},{"text":" means at least two). Inspect earlier attempts, prioritize repeated same-intent failures, fix the earliest unstable postcondition, and measure again.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Know the discovery limits:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"momentic_list_runs","type":"text","marks":[{"type":"code_inline"}]},{"text":" filters by test, branch, status, quarantine, recovery, effective failure category, minimum total attempts, and time. ","type":"text"},{"text":"failureCategory","type":"text","marks":[{"type":"code_inline"}]},{"text":" uses a manual classification when present, otherwise the automated classification.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"momentic_quarantine_list","type":"text","marks":[{"type":"code_inline"}]},{"text":" filters by quarantine date and returns each active quarantine's latest retained failed CLI run. It does not filter by branch or classification.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A category or retry threshold can still span multiple pages. Paginate through all ","type":"text"},{"text":"pageCount","type":"text","marks":[{"type":"code_inline"}]},{"text":" pages needed before calling an inventory exhaustive.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Maintenance workflow","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Start from the exact run, branch, run group, commit, or local results target.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Inspect existing automated/manual classifications and heal state first.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Reconstruct exact failures with MCP and only the necessary run history.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Explain the earliest divergence and broken postcondition before editing.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Classify afresh only when useful; persist a verdict only when requested.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Prepare the exact app and fixtures, then triage when repair is appropriate.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Review the diff and replay evidence; re-run inconclusive repairs.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Report diagnosis, repair, verification, delivery, and remaining blockers.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Inspect the existing classification","type":"text"}]},{"type":"paragraph","content":[{"text":"Fetch raw run metadata before MCP investigation when an API key is available:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"RUN_ID=\"\u003crun-id>\"\nMOMENTIC_SERVER_URL=\"${MOMENTIC_SERVER:-https://api.momentic.ai}\"\nRUN_INSPECTION_DIR=\"$(mktemp -d)\"\n\ncurl --fail --silent --show-error \\\n -H \"Authorization: Bearer ${MOMENTIC_API_KEY:?Set MOMENTIC_API_KEY}\" \\\n \"${MOMENTIC_SERVER_URL}/v1/runs/${RUN_ID}/metadata\" \\\n -o \"${RUN_INSPECTION_DIR}/metadata.json\"\n\njq '{\n runId: .id,\n status,\n failureReason,\n classification: .failureDetails.classification,\n manualClassification: .failureDetails.manualClassification,\n healStatus,\n healDetails\n}' \"${RUN_INSPECTION_DIR}/metadata.json\"","type":"text"}]},{"type":"paragraph","content":[{"text":"Do not treat a missing field as proof that no human reviewed the failure until the metadata request succeeds for the exact run and organization. Do not echo, log, or commit the API key. Treat downloaded metadata and signed artifact URLs as sensitive run data.","type":"text"}]},{"type":"paragraph","content":[{"text":"momentic_get_run","type":"text","marks":[{"type":"code_inline"}]},{"text":" intentionally removes prior classification and heal fields from the evidence it returns so a fresh investigation is not anchored by an old verdict. Use the metadata endpoint or dashboard for the existing verdict, then use MCP for the underlying evidence.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Helpful MCP tools","type":"text"}]},{"type":"paragraph","content":[{"text":"momentic_get_run","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Return run metadata and a summary of the full result tree. Use it to identify the attempt, failed section, failing step, and nested parent chain. If the current run is already loaded, call it again only for a different attempt or missing data.","type":"text"}]},{"type":"paragraph","content":[{"text":"momentic_list_runs","type":"text","marks":[{"type":"code_inline"}]},{"text":" - List recent runs for the test. Always pass ","type":"text"},{"text":"gitBranchName","type":"text","marks":[{"type":"code_inline"}]},{"text":" when it exists on the target run so comparisons are more likely to use the same test version. Omit it only when cross-branch history answers a specific question. Pass ","type":"text"},{"text":"recovered=true","type":"text","marks":[{"type":"code_inline"}]},{"text":" to inspect recovered runs. Pass ","type":"text"},{"text":"failureCategory","type":"text","marks":[{"type":"code_inline"}]},{"text":" to select effective manual/automated classifications and ","type":"text"},{"text":"minAttempts: 2","type":"text","marks":[{"type":"code_inline"}]},{"text":" or higher to find retried runs.","type":"text"}]},{"type":"paragraph","content":[{"text":"momentic_quarantine_list","type":"text","marks":[{"type":"code_inline"}]},{"text":" - List active local test quarantines and each test's latest retained failed CLI run. Use a bounded quarantine-date range.","type":"text"}]},{"type":"paragraph","content":[{"text":"momentic_get_step_result","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Return command details, before/after screenshots, and optionally the full trace for one step. Pass ","type":"text"},{"text":"parentStepIdChain","type":"text","marks":[{"type":"code_inline"}]},{"text":" for nested steps. Request ","type":"text"},{"text":"includeTrace=true","type":"text","marks":[{"type":"code_inline"}]},{"text":" only when screenshots and normal fields do not answer the question.","type":"text"}]},{"type":"paragraph","content":[{"text":"momentic_get_test_steps_for_run","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Return the simplified test definition recorded on a run. Use it when test intent remains unclear or when comparing whether the authored test changed between runs.","type":"text"}]},{"type":"paragraph","content":[{"text":"momentic_submit_result_classification","type":"text","marks":[{"type":"code_inline"}]},{"text":" - Persist a manual MCP classification. Call it only after the investigation is complete and only when the user asked to record the verdict. Do not call it in addition to ","type":"text"},{"text":"ai classify --save","type":"text","marks":[{"type":"code_inline"}]},{"text":" for the same conclusion.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Investigation workflow","type":"text"}]},{"type":"paragraph","content":[{"text":"Start with the current run before relying on history.","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Call ","type":"text"},{"text":"momentic_get_run","type":"text","marks":[{"type":"code_inline"}]},{"text":" and identify the failing attempt, section (","type":"text"},{"text":"beforeSteps","type":"text","marks":[{"type":"code_inline"}]},{"text":", main steps, or ","type":"text"},{"text":"afterSteps","type":"text","marks":[{"type":"code_inline"}]},{"text":"), failing step, and any ","type":"text"},{"text":"parentStepIdChain","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Pull the failing step result with screenshots and trace. If the step is nested, also pull the nearest parent container or module result.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Decide whether the failing step's before-screenshot is the correct baseline for that action. If it is already wrong, walk backward through the current run until you find the step/container that produced that bad state.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For repeated modules or repeated workflows, compare invocations inside the same current run before comparing older runs. The later failure is often caused by an earlier invocation that succeeded, recovered, or left an invalid postcondition.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Treat successful containers with failed or recovered child steps as partial failures until you inspect the container's final after-screenshot and URL.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Use past runs only for specific comparison questions once the current-run behavior is understood.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Before classifying, be able to answer:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"What is the test's intended behavior?","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"What is the earliest divergent step/container?","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"What did that step intend to do?","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Which element/state did it actually interact with or observe?","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"What changed in the screenshot, URL, DOM, trace, or recovery log after the step?","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Why is the later failure a consequence of that earlier divergence?","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Avoid vague root causes such as \"setup was unreliable\" or \"the page was in the wrong state.\" Name the broken postcondition directly: for example, \"the row-level plus button was clicked, but the app stayed on the parent page instead of opening the child-page editor; the following global ","type":"text"},{"text":"Add to","type":"text","marks":[{"type":"code_inline"}]},{"text":" assertion passed against unrelated page text, so the untargeted type step never entered the child title.\"","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Evidence standards","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Screenshots are the default truth source for page state. Use trace fields and DOM/HTML to explain why the screenshot changed or did not change.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Verify every causal claim. Do not say an overlay, side peek, modal, or menu was present unless the relevant before/after screenshot, URL, or DOM proves it.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Separate \"the target is missing now\" from \"the browser is in the state where that target should exist.\" A missing target is often a symptom of an earlier failed action.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For click/type/action steps, record the intended action, actual interacted element when available, before/after URL, and whether the expected UI state appeared.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For assertions, check whether the assertion is scoped enough to prove the intended state. A broad page-content assertion can pass for unrelated text.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"For recovery, inspect both the failed child step and the recovered container final state. Recovery can pass a retried assertion while leaving state that later steps did not expect.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Run AI classification","type":"text"}]},{"type":"paragraph","content":[{"text":"momentic ai classify","type":"text","marks":[{"type":"code_inline"}]},{"text":" runs Momentic's result-classification agent over failed runs. It reads the run artifacts, uses relevant run history and repository context, and returns a category, recoverability, confidence, reasoning, and citations. It explains the failure; it does not edit the test.","type":"text"}]},{"type":"paragraph","content":[{"text":"Use one selector per command:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"npx momentic ai classify --run-id \"$RUN_ID\" --output-format json\nnpx momentic ai classify --run-group-id \"\u003crun-group-id>\"\nnpx momentic ai classify --git-commit \"\u003ccommit-sha>\"","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Omit ","type":"text"},{"text":"--save","type":"text","marks":[{"type":"code_inline"}]},{"text":" for a diagnostic re-classification. Add ","type":"text"},{"text":"--save","type":"text","marks":[{"type":"code_inline"}]},{"text":" only when the new verdict should replace the saved classification on the local archive or cloud run.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"--no-cache","type":"text","marks":[{"type":"code_inline"}]},{"text":" when the existing cached classifier result is suspected to be stale and the user explicitly needs a fresh analysis.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"--skip-classified","type":"text","marks":[{"type":"code_inline"}]},{"text":" when resuming a batch without revisiting runs that already have saved classifications.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"--output-format json","type":"text","marks":[{"type":"code_inline"}]},{"text":" for automation and multi-run bookkeeping.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Classify multiple related run IDs together when useful, but keep each result mapped to its exact run.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"For one run in a real terminal, keep the classifier conversation open:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"npx momentic ai classify --run-id \"$RUN_ID\" --interactive","type":"text"}]},{"type":"paragraph","content":[{"text":"Use the chat to ask why it chose the category, what evidence it relied on, how the current run differs from a prior run, or what a durable fix would require. Interactive classification is single-run only, requires stdin to be a TTY, and cannot be combined with JSON output.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Prepare triage","type":"text"}]},{"type":"paragraph","content":[{"text":"Treat application readiness as a hard precondition. Triage replays the test in a real browser; it cannot repair a test against an unavailable app, missing fixture, wrong environment, unseeded database, invalid account, or cold compiler.","type":"text"}]},{"type":"paragraph","content":[{"text":"Before triage:","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Read ","type":"text"},{"text":"momentic.config.yaml","type":"text","marks":[{"type":"code_inline"}]},{"text":", the target test and its setup modules, repository scripts, and the CI workflow that normally runs the test. Use the same URL, environment, credentials, headers, files, services, and data fixtures.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Confirm ","type":"text"},{"text":"MOMENTIC_API_KEY","type":"text","marks":[{"type":"code_inline"}]},{"text":", project configuration, dependencies, and the requested browser are available.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Inspect ","type":"text"},{"text":"git status","type":"text","marks":[{"type":"code_inline"}]},{"text":" and preserve unrelated work. Decide the repair delivery before running the command; do not let an unknown dashboard default create a pull request or commit unexpectedly.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Build the application before starting triage. Prefer a production or otherwise precompiled build and a stable server over Next.js, webpack, Vite, or another development server that compiles routes incrementally.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Start every required service and wait for the actual readiness URL. Warm any unavoidable dev-only compilation path before triage and verify that the relevant route no longer shows a compile/loading screen or incurs a long first request.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Run a focused smoke or the failing test once when necessary to prove the app, fixture, authentication, and browser are ready. Fix setup failures before blaming triage.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Start with ","type":"text"},{"text":"--parallel 1","type":"text","marks":[{"type":"code_inline"}]},{"text":". Raise it only when the machine has enough CPU and memory for one browser and application workload per worker.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"If the repository exposes a stable server command, either start it separately or let triage own it with ","type":"text"},{"text":"--start","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"--wait-on","type":"text","marks":[{"type":"code_inline"}]},{"text":":","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"# Build first; use the repository's actual commands and readiness URL.\npnpm build\nnpx momentic ai triage --run-id \"$RUN_ID\" \\\n --start \"pnpm start\" \\\n --wait-on \"http://localhost:3000/health\" \\\n --parallel 1 \\\n --yes \\\n --on-heal-success nothing","type":"text"}]},{"type":"paragraph","content":[{"text":"Do not copy these example commands blindly. Discover the real build, start, and readiness contract from the repository and CI configuration.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Run AI triage","type":"text"}]},{"type":"paragraph","content":[{"text":"momentic ai triage","type":"text","marks":[{"type":"code_inline"}]},{"text":" groups failures by shared root cause, decides which ones are repairable, edits the relevant tests, replays them in a browser, and accepts only repairs that satisfy its verification. It can leave changes locally, produce a patch, commit, or open a pull request. It cannot fix an application bug, external outage, or missing fixture by changing a test.","type":"text"}]},{"type":"paragraph","content":[{"text":"Run classification first when the user wants an explicit diagnosis, then pass the same target to triage:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"npx momentic ai classify --run-id \"$RUN_ID\" --output-format json\nnpx momentic ai triage --run-id \"$RUN_ID\" \\\n --parallel 1 \\\n --yes \\\n --on-heal-success nothing","type":"text"}]},{"type":"paragraph","content":[{"text":"For a local run group, pass the results directory instead:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"npx momentic ai triage ./test-results \\\n --parallel 1 \\\n --yes \\\n --on-heal-success nothing","type":"text"}]},{"type":"paragraph","content":[{"text":"Best practices:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Pass multiple related run IDs after one ","type":"text"},{"text":"--run-id","type":"text","marks":[{"type":"code_inline"}]},{"text":" so the bucketing agent can group shared causes. Do not launch one competing triage command per run.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Set ","type":"text"},{"text":"--on-heal-success","type":"text","marks":[{"type":"code_inline"}]},{"text":" explicitly: use ","type":"text"},{"text":"nothing","type":"text","marks":[{"type":"code_inline"}]},{"text":" for local review, ","type":"text"},{"text":"patch","type":"text","marks":[{"type":"code_inline"}]},{"text":" for a portable diff, or a pull-request/commit behavior only when the user asked for that delivery.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Pass ","type":"text"},{"text":"--yes","type":"text","marks":[{"type":"code_inline"}]},{"text":" only after the environment and delivery choice are confirmed so dependency prompts do not stall an autonomous run.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"--dry-run --no-save","type":"text","marks":[{"type":"code_inline"}]},{"text":" to inspect bucketing without editing tests or saving triage metadata. Dry-run is not a repair attempt.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"--regenerate-heal","type":"text","marks":[{"type":"code_inline"}]},{"text":" only when a cached heal is stale or failed and a new repair should be generated from scratch.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Keep the command attached and monitor reasoning, tool calls, browser progress, CPU, and memory. Several minutes without browser progress can indicate app startup, cold compilation, resource pressure, or a missing fixture.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Inspect every resulting diff. Do not accept deleted assertions, broadened checks, arbitrary sleeps, or longer timeouts that merely conceal the cause.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Do not remove quarantine solely because triage produced a patch. Re-run the repaired test first.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Use interactive triage when the user wants to question the repair agent:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"npx momentic ai triage --run-id \"$RUN_ID\" \\\n --parallel 1 \\\n --yes \\\n --on-heal-success nothing \\\n --interactive","type":"text"}]},{"type":"paragraph","content":[{"text":"Interactive triage requires a TTY, cannot be combined with ","type":"text"},{"text":"--json","type":"text","marks":[{"type":"code_inline"}]},{"text":", and is available only when the command runs the bucketing agent. Use it to ask why a fix was accepted or rejected, what setup it observed, and what evidence remains unresolved.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Download run artifacts manually","type":"text"}]},{"type":"paragraph","content":[{"text":"Prefer MCP for ordinary inspection because ","type":"text"},{"text":"momentic_get_run","type":"text","marks":[{"type":"code_inline"}]},{"text":" downloads and extracts cloud attempts automatically. Use the API when MCP output is missing, when raw files must be grepped, or when independently verifying the agent.","type":"text"}]},{"type":"paragraph","content":[{"text":"The export endpoint returns run metadata plus one short-lived signed ZIP URL per attempt; it does not return one combined run ZIP. Download and extract them without printing the URLs:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"RUN_ID=\"\u003crun-id>\"\nMOMENTIC_SERVER_URL=\"${MOMENTIC_SERVER:-https://api.momentic.ai}\"\nRUN_INSPECTION_DIR=\"$(mktemp -d)\"\n\ncurl --fail --silent --show-error \\\n -H \"Authorization: Bearer ${MOMENTIC_API_KEY:?Set MOMENTIC_API_KEY}\" \\\n \"${MOMENTIC_SERVER_URL}/v1/runs/${RUN_ID}/export\" \\\n -o \"${RUN_INSPECTION_DIR}/export.json\"\n\njq '.run' \"${RUN_INSPECTION_DIR}/export.json\" \\\n > \"${RUN_INSPECTION_DIR}/metadata.json\"\n\njq -r '.attemptZipUrls | to_entries[] | select(.value != null) |\n [(.key + 1), .value] | @tsv' \"${RUN_INSPECTION_DIR}/export.json\" |\n while IFS=

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.