The Tests Rot. The Manifest Doesn't.

Let's Call It Grounded-Maintenance Engineering

A couple of weeks ago, we stood up an agentic test suite in an afternoon, behind a deterministic gate that decided what CI was allowed to trust. That project was about getting tests to exist. This one is the harder problem that starts the moment they do: keeping them alive while the application shifts underneath them.

A suite starts rotting the day it's written, not from any mistake, but because the app it tests keeps shipping. A button's accessible name changes and a selector goes dark. A checkout grows a shipping step and a four-click flow is now five. An API renames a field and a contract assertion fails against a response that's actually correct. None of these are bugs in the application; the suite is just falling out of sync with a reality that didn't ask permission to change. Left alone, a green suite goes red, and a red one nobody trusts gets quietly dropped from the gate: the worst of both worlds, the maintenance cost of automated tests and the safety of having none.

The panicked agentic answer is to point an agent at the broken suite and let it rewrite whatever's failing until the lights go green. It looks like exactly the kind of toil an agent should absorb. It's also the self-grading trap from Issue #7 and Issue #18 in a maintenance costume, with a second failure stacked on top. The self-grading part is familiar: an agent that certifies its own rewrite as passing is the grader and the graded collapsed into one process, optimizing for green with total confidence. The second failure is specific to maintenance, and it's worse: an agent staring at a broken test has no idea what it was supposed to verify. It sees a failing assertion and a stack trace, not intent. So it guesses, and a guess that turns the light green is the most expensive kind of wrong there is, because it looks exactly like a fix.

The disciplined answer has two parts, weighted evenly. First, an architecture that keeps the agent's creativity and the agent's authority in separate hands. Second, the thing that lets a maintenance agent stop guessing: a manifest that tells it what every test was supposed to verify before it touches a line, which you can adopt today, with any framework, before any of my tooling exists. Architecture first, then the part you can put in your repo today.

Companion script for this issue: intent-lint. It validates a test-intent manifest before a maintenance agent ever grounds on it, flagging tests with no declared intent, no criticality, or no reachable fallback selectors, plus orphaned references in both directions. Usable today, before any maintenance tooling ships. Hand-raiser keyword: INTENTLINT. Wiring and exit codes in the Quick Tip below.

For Further Reading

• Bashmatica! #018: Let's Get Started With Agentic Testing. The generation half of this story, and the gate-keeper script this maintenance sequel builds on.

• Bashmatica! #007: The Trust Decay Problem. Why the agent that writes a test can't be the thing that certifies it passed: the self-grading trap this issue's architecture avoids.

• Playwright Locators (Playwright docs). The recommended locator priority, role and accessible name first, raw CSS last, exactly the ranked fallback chain a healer should climb.

• Cypress: Best Practices (Cypress docs). Why selectors coupled to CSS and markup break on every UI change, and why dedicated test attributes don't.

• Healenium (open source). A runtime self-healing library for Selenium that swaps a broken locator mid-run against a stored fingerprint, the contrast to the maintenance-time, commit-the-fix approach here.

The Naive Fix vs The Disciplined Plan

Self-healing tests aren't a new idea. Tools like Healenium, Testim, and Mabl heal broken selectors at runtime: when a locator fails to resolve mid-run, the tool scores the nearest element against a stored fingerprint and swaps in a replacement, no human in the loop. It works, within a narrow band, and the band is the whole story: runtime self-healing fixes exactly one failure class, the broken locator, inside the runner where nobody reviews the heal and nothing records it. The test passed, but the selector it used isn't the one in your code, and that divergence compounds in the dark until a heal matches the wrong element and asserts against it. You've traded a loud, honest failure for a quiet, dishonest pass. And it only ever touches selectors; the new shipping step, the renamed API field, the stale test data, none of that is a locator problem, so none of it heals.

The discipline I want to argue for inverts both choices: put the intelligence at maintenance time, keep execution deterministic. The tool diagnoses the failure, writes a fix into the test code, and commits it (a branch, a PR, on terms you set), then gets out of the way. It doesn't insert itself into the runner. Tests still execute as plain Playwright, Cypress, or Selenium, except now the selector in the code is the selector that runs, and the fix is a diff a human can read in git history. AI at maintenance time, deterministic execution at runtime. That single line is the whole philosophy; everything below is mechanics.

The Architecture: Planner, Workers, Judge

The tool I'm designing to do this is testscout-maintain, and I'll be honest up front: it's on the spec board, early, not installable today. I'm building it in the open, so take this as a worked design, not a product pitch. It applies whether you build it yourself, wait for this, or assemble it from agents you already have. It runs as three roles, and the separation between them is the point.

The Planner reads the manifest and the failure report and does exactly one thing: it categorizes. Every failure gets sorted into a class: broken selector, timing or flakiness, flow change, API contract drift, stale test data, or a changed assertion value. It doesn't fix anything; it triages, skipping quarantined tests and prioritizing the critical path. A suite with forty broken tests comes out as a sorted work list: fifteen selector problems, ten timing, eight API, seven flow.

The Workers are specialists, one failure class each, and they're where the parallelism lives. A selector Worker heals locators by climbing the fallback chain the manifest gave it; a timing Worker adds waits and network-idle checks; an API Worker diffs the test's expectations against the current OpenAPI spec; a flow Worker detects the new or removed step and proposes the update. What makes this safe to run in parallel is that each Worker operates in its own git worktree: an isolated working directory with its own checked-out copy of the files, sharing the repository but never touching another Worker's files. Ten agents can rewrite ten tests at once and none can corrupt another's work, because at the filesystem level they aren't in the same place. That isolation is what makes "spin up a Worker per failure" a sane sentence.

The Judge is the second key, and it's a different agent from every Worker that proposed a fix. It merges the worktrees, runs the fixed suite for real, and decides each fix's fate by confidence: a high-confidence fix backed by a config selector that preserves the test's intent auto-commits; a lower-confidence fix opens a PR for a human; a fix that still fails escalates with a diagnosis and changes nothing. This is the exact two-key arrangement from Issue #18's gate-keeper, carried into maintenance. The proposer and the ratifier are different agents, by design, for the same reason a developer doesn't approve their own PR.

That's the architecture. None of it works, though, without the thing that lets every one of those agents stop guessing.

Do It Today: A Test-Intent Manifest

Everything turns on the question the architecture can't answer by itself: when a test fails, what was it trying to verify? An agent reading a broken checkout.spec.ts sees a failing assertion and a dead selector. It doesn't see that the test exists to confirm a promo code reduces the total price, so it optimizes to make the assertion pass, a different goal from preserving what the test was meant to prove, and the two diverge exactly when it matters most.

A test-intent manifest closes that gap. It's a config file in your repo root that declares, for every test, what the test verifies (the intent, in plain English, not the implementation), how critical it is, and the fallback selectors a healer can reach for, ranked by stability. This is retrieval-augmented grounding applied to test maintenance: instead of letting an agent infer what your tests mean, you hand it verified context up front and constrain it to that.

The best part: you can adopt it today, with any framework, and it pays off the moment any agent (mine or yours) reads it. The schema is progressive, the minimum to start being a project block and one test that declares its intent and criticality. Here's the entire getting-started config:

project:
  name: my-app
  framework: playwright
  test_dir: ./tests

tests:
  login:
    file: tests/login.spec.ts
    intent: "Verifyuserscanloginwithvalidcredentials"
    criticality: critical

That's enough to tell a maintenance agent which framework to use, where the test is, what it verifies, and that it's critical. Everything past that adds precision. The richer version adds the ranked fallback selectors that turn a broken-locator failure from a guess into a lookup:

pages:
  checkout:
    url: /checkout
    elements:
      promo_input:
        primary: "[data-testid=promo-code]"
        fallbacks:
          - "getByRole('textbox',{name:'Promocode'})"
          - "input[name=promo]"
      apply_promo_button:
        primary: "[data-testid=apply-promo]"
        fallbacks:
          - "getByRole('button',{name:'Apply'})"

tests:
  checkout_with_promo:
    file: tests/checkout.spec.ts
    intent: "Verifythatapplyingavalidpromocodereducesthetotalprice"
    criticality: high
    stability: flaky
    flaky_reason: "Promoservicesometimestimesoutinstaging"
    pages_involved: [checkout]

Now when the primary selector breaks, the healer doesn't invent a replacement from nothing (the lowest-confidence fix there is, and the one likeliest to match the wrong element). It climbs the declared chain: the test ID, then the accessible role and name, then the name attribute, exactly the locator priority Playwright and Cypress recommend, because that's the order that survives a redesign. The agent heals down a ladder you built instead of one it imagined.

The stability: flaky flag does quiet, important work too: it tells the Planner this test fails sometimes for reasons that aren't the app's fault, so a single red run isn't a signal to rewrite anything. Without it, an agent treats every failure as a defect, and "fixing" a flaky test usually means weakening its assertions until it stops failing, a regression dressed as a repair.

Quick Tip: Lint The Manifest Before You Trust It

A manifest only grounds an agent if it's complete and still matches what's on disk, so before any agent reads your testscout.config.yaml, lint it. That's intent-lint, this issue's companion script:

./intent-lint.sh --config ./testscout.config.yaml

It runs five checks: three for completeness (every test declares an intent, a criticality, and reaches at least one ranked fallback chain) and two for sync (every configured file still exists on disk, and every test file under your test directory has a config entry). The bundled demo seeds one of each:

./intent-lint.sh --example

# INTENT      missing on: checkout  FAIL
# CRITICALITY missing on: profile  FAIL
# FALLBACKS   zero fallback selectors reachable from: search  WARN
# ORPHAN-CFG  config entries point at missing files: legacy_flow(tests/legacy.spec.ts)  FAIL
# ORPHAN-DISK test files on disk with no config entry: ./tests/cart.spec.ts  WARN
#
# Verdict: NOT CLEAN (3 refusal(s), 2 warning(s))

Exit 0 means the manifest grounds an agent; exit 1 means it doesn't yet. Run it in CI before any maintenance step so an incomplete manifest can't quietly send an agent guessing. It parses the config in pure awk, so there's no YAML library to install. Full implementation, including --strict, in the bashmatica-scripts repo.

Quick Wins

🟢 Easy (20 min): Write a five-line manifest for your single most important test: the project block and one tests entry with an intent and a criticality. Writing the intent in one plain sentence is the whole exercise; if you can't say what a test verifies in a sentence, that's a finding about the test, not the manifest.

🟡 Medium (1 hour): Manifest your whole critical-path suite (intents and criticalities for every test, plus ranked fallback selectors for the elements that break most often), then run intent-lint against it. Treat every INTENT and CRITICALITY failure as a test nobody can describe, and every ORPHAN as a place your config and your repo already drifted apart.

🔴 Advanced (half day): Wire intent-lint --strict into CI as a gate on the manifest itself, so a test can't merge without a declared intent, a criticality, and a fallback chain, and the config can never silently fall out of sync with the suite. You're not maintaining tests yet; you're guaranteeing that whenever you do, by hand or by agent, the grounding is complete and current.

Next Week

Two issues inside the test suite, generation in Issue #18 and maintenance in this one, both leaning on the same idea: an agent proposes, a deterministic check or a separate Judge disposes, never the same process. Next week I pull that idea out of testing entirely. The two-key split, the proposer who never ratifies, is a general pattern for putting agents into any workflow with real consequences, and I'll walk through how to spot the places in your own pipeline that need a Judge, where the green light comes from the same hand that earned it.

A test suite is a promise about how your application behaves, and the promise goes stale the moment the thing it describes starts moving. Letting an agent rewrite whatever's broken until the lights go green doesn't keep that promise; it forges a new one the agent made up, because a broken test tells it nothing about what it was for.

The tests rot. That's not their failure; it's the cost of shipping. The manifest is the part that doesn't rot with them, the record of intent that survives the drift and turns a guessing agent into a grounded one. Write down what your tests mean before they break, keep the agent that proposes a fix separate from the one that ratifies it, and you've built the floor that makes everything above it safe to automate.

P.S. I'm building the TestScout MCP suite in the open, and testscout-maintain (the Planner/Workers/Judge tool from this issue) is still on the spec board, so today's takeaway is the manifest and the linter, not the tool. The piece furthest along is LightScout, both a Chrome extension and an MCP server, and its check_threshold tool is the same deterministic-gate philosophy made concrete: a CI pass/fail check that holds a page to Core Web Vitals boundaries and fails only when the numbers are genuinely bad. If this issue helped, forward it. If someone forwarded it to you, subscribe at bashmatica.com.