risk-based-testing — Skillopedia

<objective Equal coverage across all features wastes effort on low-risk areas while leaving critical paths under-tested — a 90% coverage target on a settings page is effort stolen from checkout. This skill discovers risk, quantifies it as impact × probability, maps test density to risk zones, and keeps the assessment current as the product evolves. Output is a scored risk matrix that feeds and . </objective Quick Route | Situation | Start at | |-----------|----------| | New product, no risk model yet | Phase 1 (Identification) → run the full 6 phases | | Post-incident reassessment | Phase 6 (…

| sort | uniq -c | sort -rn | head -20\n ```\n- **Defect clustering:** Which modules produce the most bugs? Track with issue labels.\n- **Near-miss frequency:** How often do staging/QA catches prevent production incidents?\n- **Dependency health:** Monitor status pages and uptime of critical third-party services.\n- **Coverage trends:** Is coverage increasing or decreasing in high-risk areas?\n\n---\n\n## Anti-Patterns\n\n### Testing everything equally\nApplying the same coverage target to every feature regardless of risk. A 90% target on a settings page wastes effort that should go to payments or auth. Let the risk model drive allocation.\n\n### One-time risk assessment\nCreating a matrix during planning and never updating it. The product, team, and dependencies all change. A model from 6 months ago is outdated and out of date the moment a dependency, feature, or incident shifts the picture — it does not reflect today. Schedule reassessment and enforce it.\n\n### Ignoring near-misses\nTreating bugs caught in staging as pure successes. If a critical bug was caught only by manual testing, the automated safety net has a gap. Document near-misses and adjust the model.\n\n### Risk theater\nGoing through the motions (filling matrices, drawing heatmaps) without changing test allocation. If the heatmap exists but coverage does not align to it, the exercise was wasted. Verify alignment quarterly. Bolton's \"Quality Engineering Is Not Testing\" (2026-04-20) warns of exactly this — building a heatmap and calling it \"QE done.\" Reference: https://developsense.com/blog/2026/04/quality-engineering-is-not-testing\n\n### Anchoring on historical risk\nOver-weighting past incidents and under-weighting new vectors. A module that failed 2 years ago and was since rewritten may no longer be high risk; a brand-new third-party integration has unknown risk that deserves attention.\n\n### Confusing severity with priority\nSeverity is how bad a failure is; priority is how urgently to test it. A catastrophic-but-rare failure (earthquake destroys data center) can be lower priority than a moderate-but-frequent one (search occasionally wrong). Use the composite score, not impact alone.\n\n---\n\n## Verification\n\nProve the matrix is real and aligned before calling it done — smallest check first:\n\n- **Artifact exists and is tracked:** `git ls-files | grep -E 'risk-matrix|qa-project-context'` returns the file. An untracked draft on someone's laptop is not a risk model.\n- **Every in-scope feature is scored:** grep the artifact for rows missing an impact or probability number. A feature with a name but no score is a gap, not a low risk.\n- **Top items have failure mode analysis:** every item scoring ≥ 10 has a block with all five fields (Trigger, Blast Radius, Detection Method, Current Mitigation, Gap). A failure mode with an empty Gap line was not actually analyzed.\n- **Coverage aligns to the heatmap:** for each CRITICAL/HIGH feature, confirm the prescribed coverage from the Phase-5 table actually exists (run the suite, check the coverage report's per-module numbers against the target). If the heatmap says CRITICAL but the module has 40% branch coverage and no E2E, the exercise was Risk Theater.\n- **Churn signal is current:** re-run the Phase-6 churn command; any module in the top 10 that is not scored ≥ Probability 4 is a model that has drifted from reality.\n\n## Done When\n\n- A scored risk matrix exists as a tracked artifact (`.agents/qa-project-context.md` risk section or a committed `risk-matrix.md`), with every in-scope feature scored on impact (1-5) and probability (1-5)\n- Each feature's composite score places it in a named zone (CRITICAL, HIGH, MEDIUM, or LOW) with a corresponding testing action assigned\n- Every feature scoring ≥ 10 has a completed failure mode analysis with trigger, blast radius, detection method, current mitigation, and gap documented\n- Coverage requirements per zone are mapped against current coverage, with each gap assigned a Priority (P0-P3), an Owner, and a Target Sprint\n- Reassessment triggers and cadence are recorded in the same artifact (quarterly minimum, plus within 48 hours of any production incident)\n\n## Reference Files (in `references/`)\n\n- **examples.md** — Four fully-scored worked examples (checkout, media platform, third-party API, auth): risk profile → failure modes → prescribed coverage.\n\n## Related Skills\n\n- **test-strategy** — The multi-quarter QA strategy this risk matrix feeds into; risk assessment is one input to a broader strategy.\n- **test-planning** — Single-sprint/release planning uses risk priorities to decide what to test this iteration.\n- **release-readiness** — Go/no-go decisions reference the heatmap to confirm critical areas are covered.\n- **qa-metrics** — Defect escape rate and defect clustering feed back into risk reassessment.\n- **ai-system-testing** — Building the eval suites that mitigate the Phase-3 AI/LLM failure classes.\n- **qa-project-context** — Captures the critical flows, fragile areas, and dependencies this skill consumes and writes the risk matrix back into.","attachment_filenames":["references/examples.md"],"attachments":[{"filename":"references/examples.md","content":"# Risk-Based Testing — Worked Examples\n\nFour fully-scored examples showing the path from risk score to prescribed coverage.\nEach follows the same shape: risk profile (Impact, Probability, composite score, zone),\nthe failure modes identified in Phase 3, and the coverage prescribed by the Phase-5\nby-zone table.\n\n---\n\n## Example 1: E-commerce Checkout\n\n**Risk profile:** Impact 5, Probability 4, Score 20 (CRITICAL)\n\n**Failure modes identified:**\n- Payment charged but order not created (race condition)\n- Discount stacking applies incorrect total\n- Inventory oversold under concurrent load\n- Shipping calculator returns wrong rate for international addresses\n- Tax calculation wrong for specific jurisdictions\n\n**Test coverage prescribed:**\n- Unit tests: discount calculation (all combinations), tax rules (per jurisdiction), inventory decrement logic\n- Integration tests: payment gateway communication (success, failure, timeout, duplicate), order creation pipeline, inventory reservation under concurrency\n- E2E tests: full checkout flow (guest + logged in), checkout with discount, checkout with international shipping, checkout retry after payment failure\n- Load tests: 100 concurrent checkouts for last-stock item\n- Monitoring: real-time order completion rate, payment-to-order reconciliation every 5 minutes, revenue anomaly detection\n\n---\n\n## Example 2: Content Loading (Media Platform)\n\n**Risk profile:** Impact 4, Probability 3, Score 12 (HIGH)\n\n**Failure modes identified:**\n- CDN cache miss causes origin overload\n- Video transcoding fails silently for specific codecs\n- Thumbnail generation timeout leaves blank images\n- Content recommendation engine returns stale or empty results\n\n**Test coverage prescribed:**\n- Unit tests: transcoding pipeline input validation, recommendation scoring algorithm\n- Integration tests: CDN purge/refresh flow, transcoding job queue processing, thumbnail generation for each supported format\n- E2E tests: content upload through playback, content discovery through recommendation click\n- Monitoring: CDN hit ratio, transcoding failure rate, thumbnail generation latency p99\n\n---\n\n## Example 3: Third-Party API Integration\n\n**Risk profile:** Impact 4, Probability 4, Score 16 (CRITICAL)\n\n**Failure modes identified:**\n- API rate limit exceeded during peak traffic\n- API response schema changes without notice (breaking deserialization)\n- API timeout causes cascade failure in synchronous call chain\n- API returns 200 with error body (non-standard error handling)\n\n**Test coverage prescribed:**\n- Unit tests: response parser for all known response shapes including malformed responses, rate limit backoff calculation, circuit breaker state transitions\n- Integration tests: API contract tests (validate response schema against expected shape), timeout handling, retry behavior, circuit breaker activation\n- E2E tests: user flow when API is slow (degraded but functional), user flow when API is down (graceful fallback)\n- Monitoring: API response time p50/p95/p99, error rate, rate limit proximity, circuit breaker state\n\n---\n\n## Example 4: Authentication Flows\n\n**Risk profile:** Impact 5, Probability 2, Score 10 (HIGH)\n\n**Failure modes identified:**\n- Session token not invalidated on password change\n- OAuth callback race condition allows account takeover\n- MFA bypass through API endpoint that skips MFA check\n- Rate limiting not enforced on login endpoint (brute force)\n\n**Test coverage prescribed:**\n- Unit tests: token generation and validation, password hashing, MFA code verification, rate limit counter logic\n- Integration tests: full auth flow (register, login, logout, password reset), session invalidation on credential change, OAuth flow with all supported providers, MFA enrollment and verification\n- E2E tests: login flow (valid credentials, invalid, locked account), password reset flow, MFA flow\n- Security tests: brute force attempt (verify rate limiting), session fixation, token reuse after logout\n- Monitoring: failed login rate spike, unusual session patterns, MFA bypass attempts\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":4047,"content_sha256":"17caa0b839f7cd5ced39852b6b03da5d4d166c07efa256f50ddc6d04f3f9d656"}],"content_json":{"type":"doc","content":[{"type":"paragraph","content":[{"text":"\u003cobjective> Equal coverage across all features wastes effort on low-risk areas while leaving critical paths under-tested — a 90% coverage target on a settings page is effort stolen from checkout. This skill discovers risk, quantifies it as impact × probability, maps test density to risk zones, and keeps the assessment current as the product evolves. Output is a scored risk matrix that feeds ","type":"text"},{"text":"test-strategy","type":"text","marks":[{"type":"code_inline"}]},{"text":" and ","type":"text"},{"text":"test-planning","type":"text","marks":[{"type":"code_inline"}]},{"text":". \u003c/objective>","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Quick Route","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Situation","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Start at","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"New product, no risk model yet","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Phase 1 (Identification) → run the full 6 phases","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Post-incident reassessment","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Phase 6 (Reassessment triggers), then re-score the affected items in Phase 2","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"AI/LLM feature to assess","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Phase 3 (AI/LLM failure classes), score each class in Phase 2","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Sprint refresh of an existing matrix","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Phases 4–5 (Heatmap + Coverage alignment) on changed features","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Verify an old heatmap is still true","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Phase 6 signals + Anti-Pattern \"Risk Theater\"","type":"text"}]}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Discovery Questions","type":"text"}]},{"type":"paragraph","content":[{"text":"Check ","type":"text"},{"text":".agents/qa-project-context.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" first — if it exists, use it as the foundation and skip questions already answered there. Gather the rest from stakeholders across engineering, product, and operations.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Revenue-Critical Flows","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Which user flows directly generate revenue? (checkout, subscription, billing, upgrades)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"What is the revenue impact per hour of downtime for each flow?","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Are there time-sensitive flows? (flash sales, market-hours trading, payroll deadlines)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Which flows have contractual SLAs with financial penalties?","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Recent Failures","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"What broke in the last 3 releases? What escaped to production?","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"What were the root causes? (code defect, config error, third-party failure, data migration)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"What was the blast radius of each incident? (users affected, revenue lost, reputation impact)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Were there near-misses caught late in testing that could have escaped?","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Fragile Areas","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Which parts of the codebase change most frequently? (high churn = high risk)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Which modules have the lowest test coverage today?","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Which areas have the most complex business logic or the most conditional branches?","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Which code was written by engineers who have since left the team?","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Third-Party Dependencies","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Which external services does the product depend on? (payment processors, auth providers, CDNs, APIs)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"What is the historical reliability of each dependency?","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"What happens when each dependency goes down? (graceful degradation or hard failure?)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Are there single points of failure with no fallback?","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Compliance and Data","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"What regulatory requirements apply? (GDPR, PCI-DSS, HIPAA, SOC2, SOX, EU AI Act)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"What data is most sensitive? (PII, financial, health, credentials)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"What are the legal consequences of a data breach or compliance violation?","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Are there audit requirements that mandate specific testing evidence?","type":"text"}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Core Principles","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Not all features are equal.","type":"text","marks":[{"type":"strong"}]},{"text":" A bug in checkout that blocks purchases is categorically different from a misaligned icon on a settings page. Equal coverage everywhere wastes resources on low-risk areas while leaving critical paths under-tested.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Risk = Impact × Probability.","type":"text","marks":[{"type":"strong"}]},{"text":" Risk is not a gut feeling. It is a product of two dimensions scored independently: how bad if this fails (impact), and how likely to fail (probability). Score both consistently across the product, then multiply.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Risk assessment is continuous.","type":"text","marks":[{"type":"strong"}]},{"text":" A risk model created once and never updated creates false confidence. Risk changes when the product changes, when dependencies change, when the team changes, and after every production incident. Build reassessment into the rhythm.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Near-misses are data.","type":"text","marks":[{"type":"strong"}]},{"text":" A catastrophic bug caught in staging is not a success story — it is a signal that the model underestimated that area. Track near-misses with the same rigor as production incidents.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Risk informs coverage, not the other way around.","type":"text","marks":[{"type":"strong"}]},{"text":" Do not start with \"we need 80% coverage everywhere.\" Start with \"where would a failure hurt most?\" and let the model drive coverage targets per module.","type":"text"}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Workflow","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"1. Identify → 2. Classify → 3. Analyze → 4. Heatmap → 5. Coverage → 6. Reassess → (repeat)\n ↑\n score ≥ 10 only; skip to 4 if no items reach threshold","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Phase 1: Risk Identification","type":"text"}]},{"type":"paragraph","content":[{"text":"Enumerate everything that could go wrong. Cast a wide net. Sources include:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Stakeholder interviews:","type":"text","marks":[{"type":"strong"}]},{"text":" Product managers know business-critical flows. Engineers know fragile code. Support knows recurring user complaints.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Incident history:","type":"text","marks":[{"type":"strong"}]},{"text":" Past failures predict future failures. Review post-mortems from the last 6–12 months.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Dependency mapping:","type":"text","marks":[{"type":"strong"}]},{"text":" List every external service, database, message queue, and third-party API. Each is a risk vector.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Change analysis:","type":"text","marks":[{"type":"strong"}]},{"text":" Areas with frequent code changes have higher defect probability. Use the ranked churn command in Phase 6 to find them; ","type":"text"},{"text":"git log --stat \u003cfile>","type":"text","marks":[{"type":"code_inline"}]},{"text":" is for per-commit inspection of a specific suspect, not for ranking.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Architecture review:","type":"text","marks":[{"type":"strong"}]},{"text":" Shared databases, single points of failure, synchronous chains, and tightly coupled modules amplify blast radius.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"HTSM v6.3","type":"text","marks":[{"type":"strong"}]},{"text":" (Heuristic Test Strategy Model, Bach) as a Phase-1 lens — its state-based and boundary heuristics surface risks a pure feature-list misses. Download: https://www.satisfice.com/download/heuristic-test-strategy-model","type":"text"}]},{"type":"paragraph","content":[{"text":"Output: a raw list of risk items, each describing what could fail and what the consequence would be.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Phase 2: Risk Classification","type":"text"}]},{"type":"paragraph","content":[{"text":"Categorize each risk item along two axes.","type":"text"}]},{"type":"paragraph","content":[{"text":"Impact categories (how bad is it):","type":"text","marks":[{"type":"strong"}]}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Score","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Level","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Definition","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Examples","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"5","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Catastrophic","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Revenue loss, data breach, legal action, user safety","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Payment processing fails, PII exposed","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"4","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Major","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Significant user impact, SLA violation, major feature broken","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Login broken for segment, data corruption","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"3","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Moderate","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Workflow disrupted, workaround exists","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Search returns wrong results, export fails","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"2","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Minor","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Cosmetic or minor UX issue","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Alignment bug, slow non-critical page","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"1","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Negligible","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"No user impact, internal only","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Admin tooltip wrong, log format issue","type":"text"}]}]}]}]},{"type":"paragraph","content":[{"text":"Probability categories (how likely is it):","type":"text","marks":[{"type":"strong"}]}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Score","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Level","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Definition","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Indicators","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"5","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Frequent","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Expected in most releases","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"High code churn, no tests, complex logic","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"4","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Likely","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Will probably happen within a quarter","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Recent changes, partial coverage, known tech debt","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"3","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Possible","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Could happen, has happened before","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Moderate complexity, some coverage","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"2","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Unlikely","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Improbable but not impossible","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Stable code, good coverage, simple logic","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"1","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Rare","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Requires exceptional circumstances","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Well-tested, rarely changed, simple","type":"text"}]}]}]}]},{"type":"paragraph","content":[{"text":"Composite score = Impact × Probability. Frequent changes indicate defect probability, so a Moderate-impact (3) feature under heavy churn scores Probability 5 → ","type":"text"},{"text":"Risk score: 15 → CRITICAL zone","type":"text","marks":[{"type":"strong"}]},{"text":", despite \"only\" moderate impact. The composite score drives priority, not impact alone.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Phase 3: Failure Mode Analysis","type":"text"}]},{"type":"paragraph","content":[{"text":"For each high-risk item (score ≥ 10), perform a detailed failure mode analysis.","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Feature/Component: [name]\nRisk Score: [impact × probability]\n\nFailure Mode 1: [what specifically can fail]\n Trigger: [what causes this failure]\n Blast Radius: [users affected, systems affected, data affected]\n Detection Method: [how would we know -- monitoring, user report, test]\n Current Mitigation: [existing tests, monitoring, feature flags, fallbacks]\n Gap: [what is missing from current mitigation]\n\nFailure Mode 2: ...","type":"text"}]},{"type":"paragraph","content":[{"text":"Example — E-commerce Checkout (Risk Score 20, Impact 5 × Probability 4):","type":"text","marks":[{"type":"strong"}]}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Failure Mode 1: Payment charge succeeds but order not recorded\n Trigger: Race condition between payment API callback and order write\n Blast Radius: Individual users; money charged but no order confirmation\n Detection Method: Payment reconciliation job (runs hourly), user complaint\n Current Mitigation: Idempotency key on payment, retry on order write\n Gap: No automated test for the race condition; reconciliation delay is 1 hour\n\nFailure Mode 2: Discount code applies incorrect amount\n Trigger: Percentage discount on already-discounted item\n Blast Radius: All users with stacked discounts; revenue leakage\n Detection Method: Margin monitoring alert (>5% deviation)\n Current Mitigation: Unit tests for single discounts\n Gap: No tests for discount stacking; no tests for rounding edge cases\n\nFailure Mode 3: Inventory not reserved during checkout\n Trigger: Concurrent purchases of last-stock item\n Blast Radius: Oversold items, fulfillment failure, customer trust\n Detection Method: Fulfillment team discovers during packing\n Current Mitigation: Database-level stock check on order creation\n Gap: No load test simulating concurrent last-item purchases","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"AI/LLM failure classes","type":"text"}]},{"type":"paragraph","content":[{"text":"For AI/LLM features, classify against these CT-GenAI classes and score Impact and Probability independently like any other risk. The mitigation is the existence of an automated ","type":"text"},{"text":"eval suite","type":"text","marks":[{"type":"strong"}]},{"text":", not a single manual test.","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"AI/LLM-specific failure classes","type":"text","marks":[{"type":"strong"}]},{"text":" (from ISTQB CT-GenAI v1.1, effective 27 April 2026):","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Hallucination / reasoning error","type":"text","marks":[{"type":"strong"}]},{"text":" — Impact: moderate to major; Probability: high without explicit prompt-eval coverage. Detection: golden-dataset evals, fact-check assertions (see ","type":"text"},{"text":"ai-system-testing","type":"text","marks":[{"type":"code_inline"}]},{"text":").","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Bias","type":"text","marks":[{"type":"strong"}]},{"text":" — Impact: catastrophic in regulated industries (finance, healthcare, hiring). Probability: dataset-dependent. Detection: counterfactual evals, demographic-parity checks.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Prompt injection / jailbreak","type":"text","marks":[{"type":"strong"}]},{"text":" — Impact: major (data exfiltration, prompt extraction). Probability: high for any externally-facing LLM feature. Detection: Garak, PyRIT, Promptfoo redteam.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Privacy leak","type":"text","marks":[{"type":"strong"}]},{"text":" — Impact: catastrophic under GDPR/CCPA/EU AI Act. Probability: dataset-dependent. Detection: PII scanning of training data and prompts.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"AI Act / regulatory non-compliance","type":"text","marks":[{"type":"strong"}]},{"text":" — Impact: catastrophic (fines, ban). Probability: high for EU-facing AI features. Detection: see ","type":"text"},{"text":"compliance-testing","type":"text","marks":[{"type":"code_inline"}]},{"text":".","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Tool freshness (mid-2026): PyRIT now lives at ","type":"text"},{"text":"microsoft/PyRIT","type":"text","marks":[{"type":"strong"}]},{"text":" — the old Azure-hosted repo was archived March 2026, so do not point new redteam work at the legacy Azure path. Promptfoo was acquired by OpenAI (March 2026) but remains MIT-licensed. Garak is current and unchanged.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Reference frameworks:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"CT-GenAI v1.1","type":"text","marks":[{"type":"strong"}]},{"text":" (ISTQB, effective 27 April 2026) codifies the AI/LLM classes above. ","type":"text"},{"text":"WQR 2025-26","type":"text","marks":[{"type":"strong"}]},{"text":" (Capgemini, 17th edition, Nov 2025) gives the adoption-stage framing for AI risk planning.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Phase 4: Risk Heatmap","type":"text"}]},{"type":"paragraph","content":[{"text":"Plot all risk items on a 5×5 matrix to communicate priorities and drive coverage decisions.","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":" PROBABILITY\n Rare(1) Unlikely(2) Possible(3) Likely(4) Frequent(5)\n +----------+-----------+-----------+----------+-----------+\n Catastrophic(5) | 5 MED | 10 HIGH | 15 CRIT | 20 CRIT | 25 CRIT |\n +----------+-----------+-----------+----------+-----------+\n Major(4) | 4 LOW | 8 MED | 12 HIGH | 16 CRIT | 20 CRIT |\nI +----------+-----------+-----------+----------+-----------+\nM Moderate(3) | 3 LOW | 6 MED | 9 MED | 12 HIGH | 15 CRIT |\nP +----------+-----------+-----------+----------+-----------+\nA Minor(2) | 2 LOW | 4 LOW | 6 MED | 8 MED | 10 HIGH |\nC +----------+-----------+-----------+----------+-----------+\nT Negligible(1) | 1 LOW | 2 LOW | 3 LOW | 4 LOW | 5 MED |\n +----------+-----------+-----------+----------+-----------+","type":"text"}]},{"type":"paragraph","content":[{"text":"Zone boundaries and action mapping:","type":"text","marks":[{"type":"strong"}]}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Zone","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Score Range","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Color","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Testing Action","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"CRITICAL","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"15-25","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Red","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Automate fully + monitor in production + load test + manual exploratory","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"HIGH","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"10-14","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Orange","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Automate fully + periodic manual review","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"MEDIUM","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"5-9","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Yellow","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Automate happy path + key error cases","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"LOW","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"1-4","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Green","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Manual testing on release or skip entirely","type":"text"}]}]}]}]},{"type":"paragraph","content":[{"text":"Populated example (where each named risk lands):","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":" Rare(1) Unlikely(2) Possible(3) Likely(4) Frequent(5)\n Catastrophic(5) Auth bypass Payments fail Checkout crash\n Major(4) Data export Search broken User upload\n Moderate(3) Report fmt Email deliver Profile edit\n Minor(2) Footer link Tooltip text Theme switch\n Negligible(1) Admin label","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Phase 5: Test Coverage Alignment","type":"text"}]},{"type":"paragraph","content":[{"text":"Map test density to risk level. Every zone gets a prescribed approach.","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Risk Zone","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Unit Tests","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Integration Tests","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"E2E Tests","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Manual Testing","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Monitoring","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"CRITICAL (15-25)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"90%+ branch coverage","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"All service boundaries","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Full user journey + error paths","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Exploratory each release","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Real-time alerts, synthetic checks","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"HIGH (10-14)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"80%+ branch coverage","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Key interactions","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Happy path + top 3 error paths","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Spot checks","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Dashboard + daily review","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"MEDIUM (5-9)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"70%+ branch coverage","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Happy path only","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Happy path only","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"On major changes","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Weekly review","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"LOW (1-4)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Basic happy path","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"None required","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"None required","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"On initial build","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"None required","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Gap Analysis Worksheet","type":"text"}]},{"type":"paragraph","content":[{"text":"Compare current coverage against required coverage per risk zone:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Feature: [name]\nRisk Zone: [CRITICAL / HIGH / MEDIUM / LOW] Risk Score: [number]\n\nRequired Coverage:\n Unit: [target %] Current: [actual %] Gap: [delta]\n Integration: [required?] Current: [exists? y/n] Gap: [missing scenarios]\n E2E: [required?] Current: [exists? y/n] Gap: [missing flows]\n Monitoring: [required?] Current: [exists? y/n] Gap: [missing alerts]\n\nPriority: [P0 / P1 / P2 / P3]\nEstimated Effort: [hours / story points]\nOwner: [name] Target Sprint: [sprint number]","type":"text"}]},{"type":"paragraph","content":[{"text":"A churn signal forces this worksheet open: a module that changed 47 times in 3 months (Probability → 5) with only 40% branch coverage and no integration tests jumps zones (e.g. MEDIUM → HIGH), and the new coverage target is justified by the churn, not picked arbitrarily.","type":"text"}]},{"type":"paragraph","content":[{"text":"See ","type":"text"},{"text":"references/examples.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" for four fully-scored examples (checkout, media platform, third-party API, auth) showing the path from risk score to prescribed coverage.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Phase 6: Monitoring and Reassessment","type":"text"}]},{"type":"paragraph","content":[{"text":"Risk assessment is not a one-time activity. Build reassessment into the team's rhythm.","type":"text"}]},{"type":"paragraph","content":[{"text":"Reassessment triggers:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"After every production incident (within 48 hours): re-score the affected items, check dependency health, and re-run the Phase-5 gap analysis to expose any coverage gap the incident revealed","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"When a new feature area is introduced","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"When a critical dependency changes (API version, provider switch)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"When team composition changes significantly","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Quarterly at minimum, even without triggers","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Continuous risk signals to monitor:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Code churn by module","type":"text","marks":[{"type":"strong"}]},{"text":" (ranked frequency table):","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"bash"},"content":[{"text":"git log --since=\"3 months ago\" --name-only --format= | grep -v '^

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

| sort | uniq -c | sort -rn | head -20","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Defect clustering:","type":"text","marks":[{"type":"strong"}]},{"text":" Which modules produce the most bugs? Track with issue labels.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Near-miss frequency:","type":"text","marks":[{"type":"strong"}]},{"text":" How often do staging/QA catches prevent production incidents?","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Dependency health:","type":"text","marks":[{"type":"strong"}]},{"text":" Monitor status pages and uptime of critical third-party services.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Coverage trends:","type":"text","marks":[{"type":"strong"}]},{"text":" Is coverage increasing or decreasing in high-risk areas?","type":"text"}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Anti-Patterns","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Testing everything equally","type":"text"}]},{"type":"paragraph","content":[{"text":"Applying the same coverage target to every feature regardless of risk. A 90% target on a settings page wastes effort that should go to payments or auth. Let the risk model drive allocation.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"One-time risk assessment","type":"text"}]},{"type":"paragraph","content":[{"text":"Creating a matrix during planning and never updating it. The product, team, and dependencies all change. A model from 6 months ago is outdated and out of date the moment a dependency, feature, or incident shifts the picture — it does not reflect today. Schedule reassessment and enforce it.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Ignoring near-misses","type":"text"}]},{"type":"paragraph","content":[{"text":"Treating bugs caught in staging as pure successes. If a critical bug was caught only by manual testing, the automated safety net has a gap. Document near-misses and adjust the model.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Risk theater","type":"text"}]},{"type":"paragraph","content":[{"text":"Going through the motions (filling matrices, drawing heatmaps) without changing test allocation. If the heatmap exists but coverage does not align to it, the exercise was wasted. Verify alignment quarterly. Bolton's \"Quality Engineering Is Not Testing\" (2026-04-20) warns of exactly this — building a heatmap and calling it \"QE done.\" Reference: https://developsense.com/blog/2026/04/quality-engineering-is-not-testing","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Anchoring on historical risk","type":"text"}]},{"type":"paragraph","content":[{"text":"Over-weighting past incidents and under-weighting new vectors. A module that failed 2 years ago and was since rewritten may no longer be high risk; a brand-new third-party integration has unknown risk that deserves attention.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Confusing severity with priority","type":"text"}]},{"type":"paragraph","content":[{"text":"Severity is how bad a failure is; priority is how urgently to test it. A catastrophic-but-rare failure (earthquake destroys data center) can be lower priority than a moderate-but-frequent one (search occasionally wrong). Use the composite score, not impact alone.","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Verification","type":"text"}]},{"type":"paragraph","content":[{"text":"Prove the matrix is real and aligned before calling it done — smallest check first:","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Artifact exists and is tracked:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"git ls-files | grep -E 'risk-matrix|qa-project-context'","type":"text","marks":[{"type":"code_inline"}]},{"text":" returns the file. An untracked draft on someone's laptop is not a risk model.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Every in-scope feature is scored:","type":"text","marks":[{"type":"strong"}]},{"text":" grep the artifact for rows missing an impact or probability number. A feature with a name but no score is a gap, not a low risk.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Top items have failure mode analysis:","type":"text","marks":[{"type":"strong"}]},{"text":" every item scoring ≥ 10 has a block with all five fields (Trigger, Blast Radius, Detection Method, Current Mitigation, Gap). A failure mode with an empty Gap line was not actually analyzed.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Coverage aligns to the heatmap:","type":"text","marks":[{"type":"strong"}]},{"text":" for each CRITICAL/HIGH feature, confirm the prescribed coverage from the Phase-5 table actually exists (run the suite, check the coverage report's per-module numbers against the target). If the heatmap says CRITICAL but the module has 40% branch coverage and no E2E, the exercise was Risk Theater.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Churn signal is current:","type":"text","marks":[{"type":"strong"}]},{"text":" re-run the Phase-6 churn command; any module in the top 10 that is not scored ≥ Probability 4 is a model that has drifted from reality.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Done When","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"A scored risk matrix exists as a tracked artifact (","type":"text"},{"text":".agents/qa-project-context.md","type":"text","marks":[{"type":"code_inline"}]},{"text":" risk section or a committed ","type":"text"},{"text":"risk-matrix.md","type":"text","marks":[{"type":"code_inline"}]},{"text":"), with every in-scope feature scored on impact (1-5) and probability (1-5)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Each feature's composite score places it in a named zone (CRITICAL, HIGH, MEDIUM, or LOW) with a corresponding testing action assigned","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Every feature scoring ≥ 10 has a completed failure mode analysis with trigger, blast radius, detection method, current mitigation, and gap documented","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Coverage requirements per zone are mapped against current coverage, with each gap assigned a Priority (P0-P3), an Owner, and a Target Sprint","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"Reassessment triggers and cadence are recorded in the same artifact (quarterly minimum, plus within 48 hours of any production incident)","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Reference Files (in ","type":"text"},{"text":"references/","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"examples.md","type":"text","marks":[{"type":"strong"}]},{"text":" — Four fully-scored worked examples (checkout, media platform, third-party API, auth): risk profile → failure modes → prescribed coverage.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Related Skills","type":"text"}]},{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"test-strategy","type":"text","marks":[{"type":"strong"}]},{"text":" — The multi-quarter QA strategy this risk matrix feeds into; risk assessment is one input to a broader strategy.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"test-planning","type":"text","marks":[{"type":"strong"}]},{"text":" — Single-sprint/release planning uses risk priorities to decide what to test this iteration.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"release-readiness","type":"text","marks":[{"type":"strong"}]},{"text":" — Go/no-go decisions reference the heatmap to confirm critical areas are covered.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"qa-metrics","type":"text","marks":[{"type":"strong"}]},{"text":" — Defect escape rate and defect clustering feed back into risk reassessment.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"ai-system-testing","type":"text","marks":[{"type":"strong"}]},{"text":" — Building the eval suites that mitigate the Phase-3 AI/LLM failure classes.","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"qa-project-context","type":"text","marks":[{"type":"strong"}]},{"text":" — Captures the critical flows, fragile areas, and dependencies this skill consumes and writes the risk matrix back into.","type":"text"}]}]}]}]},"metadata":{"date":"2026-11-02","name":"risk-based-testing","author":"@skillopedia","source":{"stars":239,"repo_name":"qa-skills","origin_url":"https://github.com/petrkindlmann/qa-skills/blob/HEAD/skills/risk-based-testing/SKILL.md","repo_owner":"petrkindlmann","body_sha256":"d39f564e2389b1de04024a40e0691feb100004f59e2616b2049632947ef0fe4a","cluster_key":"a1a20433c3c29b2ffe74dd6330e634382fa0bb8900fe63da41c9aee1fb789c81","clean_bundle":{"format":"clean-skill-bundle-v1","source":"petrkindlmann/qa-skills/skills/risk-based-testing/SKILL.md","text_bytes":4047,"source_commit":"62434253c2592f27b494609c27f224939c86346d","skills_sh_rank":9963,"attachment_count":1,"text_attachments":1,"attachment_storage":"skill_attachments","binary_attachments":0},"cluster_size":1,"skill_md_path":"skills/risk-based-testing/SKILL.md","import_metadata":{"date":"2026-11-02","author":"@skillopedia","version":"v1","category":"testing-qa","category_label":"Testing"},"exact_dupes_collapsed_into_this":0},"license":"MIT","version":"v1","category":"testing-qa","metadata":{"author":"kindlmann","version":"2.0","category":"strategy"},"import_tag":"skills-sh-top20k-2026-11","description":"Produce a risk matrix or heatmap that quantifies what could break by business impact × probability, runs failure mode analysis on the top items, and maps test coverage to risk zones. Includes stakeholder interview frameworks and continuous reassessment. Run this BEFORE test-strategy or test-planning. Use when: \"risk assessment,\" \"risk matrix,\" \"risk heatmap,\" \"what could break,\" \"critical paths,\" \"failure modes,\" \"where to focus testing.\" Not for: multi-quarter QA direction — use test-strategy. Not for: a single sprint/release test plan — use test-planning. Not for: hands-on session-based bug hunting — use exploratory-testing. Related: test-strategy, test-planning, release-readiness, qa-metrics."}},"renderedAt":1793666278501}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.