Your CI Logs Are a Crime Scene. Let an Agent Read Them.

Let's Call It Special Agent Engineering

The pull request comes back with a red X, and the story of why is sitting in a job log that runs fourteen thousand lines. You open it, and the browser stutters trying to render the whole thing. Somewhere in that wall of timestamps, stack frames, dependency-resolution chatter, and ANSI color codes that didn't survive the trip to plain text, there's one line that explains the failure. Maybe two. The rest is noise that was useful while the run was happening and is pure friction now that it's over.

Every failed run forces two questions before you can do anything useful, and both of them are reading problems. The first: is this a real regression, or is it flaky-or-infrastructure noise that a rerun would paper over? The second: where is the line that actually matters, the assertion that blew up or the exception that started the cascade, as opposed to the forty downstream errors it caused? You answer both by scrolling, by eye, under time pressure, and quite possibly for the third time today, because the same suite failed on two other branches this morning.

This is, genuinely, a task an agent is good at: reading a large body of text and telling you what's in it. It's also a task with one specific failure that will cost you a full day, which is a confident, wrong root cause delivered in the same authoritative tone as a correct one. The discipline that separates the useful version from the expensive one is narrow, and worth stating plainly before anything else: you let the agent read the scene, and you never let it rule the cause.

What An Agent Is Actually Good At Here

The reason to reach for an LLM on a CI log is the same reason to be careful with one; they're two faces of a single capability. Dhairya Dalal, writing in InfoQ in September 2025, draws the line precisely. LLMs can interpret complex logs and trace messages, summarize high-volume telemetry, and translate a plain-language question into a structured filter, and they're very good at it. The same piece is just as precise about where the capability ends: LLM-based solutions "often hallucinate plausible but incorrect explanations, conflate symptoms with causes, and disregard event ordering." Summarization is reading. Root-cause analysis in a distributed system is reasoning about causality across time and across services, and an agent staring at a log slice has no model of your system as a living thing with dependencies and an event order, only the text in front of it and a strong prior toward a confident-sounding answer.

The triage question this is meant to answer, flaky-or-real, has a known shape, which is what makes it tractable. Google reported that about 1.5% of all test runs come back flaky, and that almost 16% of their tests carry some flakiness, more than one in seven. The empirical work on why, Luo and colleagues' FSE 2014 taxonomy, found that nondeterministic failures cluster into a small set of causes, with async waits, concurrency, and test-order dependence leading. That clustering is the good news: a failure matching the signature of a known flaky class, a timeout, a connection reset, a race that only shows up under load, is something an agent can recognize from the text and flag, the same way it can recognize that a paragraph is about baseball.

Recognizing a signature and certifying a cause are different acts, and the entire discipline lives in the gap between them. "This failure matches the pattern of an async timeout, see line 4012" is summarization, defensible because you can check it against line 4012 in two seconds. "The test failed because the staging database was under load" is a diagnosis, and unless a cited line says so, it's the model filling a gap with the most plausible story it can write. The first claim helps you. The second, delivered in the same even tone, is the one that costs a day.

Read The Scene, Never Rule The Cause

A first responder at a scene doesn't decide who did it. They secure the area, photograph what's there, bag and tag the evidence, and write an incident report that points to specific items, all without contaminating the scene or naming a suspect they can't tie to it. That restraint is the whole job, and it's exactly the posture I want from an agent reading a failed run. The constraint that enforces it is grounding: every claim the agent makes has to point to a specific, numbered line in the log, and any claim it can't tie to a line is a claim it isn't allowed to make.

Three guardrails turn that principle into a prompt.

A closed classification set: the agent picks exactly one of flaky, infra, real_regression, or insufficient_evidence. A closed set keeps it from inventing a category, and it makes the output something a downstream step can branch on instead of prose a person has to sit and read.

Mandatory evidence: the agent must cite the line number that supports its classification, and if it can't, the rules require it to answer insufficient_evidence. This is the chain of custody. A verdict with no cited line is inadmissible by construction, which is the thing that stops the model from narrating a cause it inferred from nothing.

An explicit option to find nothing: "insufficient evidence" has to be a first-class, blessed answer rather than a failure state, because the alternative, a model that always produces a confident classification, is a model that will confidently classify noise. LLMs don't volunteer uncertainty. You have to build them a door marked uncertain and make it clear that walking through it is fine.

Those three rules are the whole prompt:

You are triaging one failed CI run. You are not fixing it and not
proposing a fix. You report only what the log supports.

Test context: {intent}

Classify the failure as exactly one of:
  flaky | infra | real_regression | insufficient_evidence

Rules:
- Cite the log line number(s) that justify your classification.
- If no line justifies a classification, answer insufficient_evidence.
- Do not state a root cause as fact unless a cited line proves it.
- Do not invent line numbers, file names, or values not in the log.

Output these fields and nothing else:
  classification:
  evidence_lines:
  confidence: low | medium | high
  summary:    (one sentence, grounded in the cited lines)

Log slice (line-numbered):
{slice}

The Test context line is doing heavy work. An agent that knows the test exists to verify a promo code reduces a total reads a price-assertion failure very differently from one that knows only that some assertion failed. Grounding the agent in intent before it sees the log is the same retrieval-augmented move that keeps any agent from guessing: you hand it verified context, then you constrain it to what the evidence in front of it supports.

The Confident-But-Wrong Trap

Many will argue, somewhat correctly, that a capable model pointed at a log can simply tell you why the build broke, and that fencing it to summarization throws away most of what it can do.

Right.

It can tell you, and a meaningful fraction of the time it's correct. The problem is the fraction where it's wrong, because that fraction doesn't announce itself. A wrong root cause arrives in the same calm, structured, plausible paragraph as a right one, and there's no confidence meter on the model's face. The cost lands on a real person. An agent that confidently reports "intermittent network failure reaching the package registry" when the actual fault is a null-pointer regression sends the engineer who trusted it into a rerun loop, retriggering the pipeline twice and pinging the platform team about a registry blip that never happened, while the real bug ships untouched in the diff. The log had the answer on one line. The agent wrote a more interesting story, and the story was the most expensive output in the run.

This is why the architecture matters more than the model. The agent's verdict is evidence, not a finding, and something else decides what to do with it: a deterministic check that reruns only the classes known to be flaky, or a human who reads the cited line and either agrees or doesn't. The proposer and the decider stay in separate hands, for the same reason the officer who works the scene isn't the one who hands down the verdict.

Wiring It Into CI Without Giving It The Keys

The mechanics are mundane, which is the point; nothing here grants the agent authority it shouldn't have. On a failed run you need the log, a triage verdict, and a place to put it where a human will see it. Pulling the log is a documented call. GitHub's REST API exposes a job's log at GET /repos/{owner}/{repo}/actions/jobs/{job_id}/logs, which returns a redirect to a download URL that expires after a minute, and the gh CLI wraps the auth so you don't have to. The --log-failed flag on gh run view is the simpler path, since it grabs only the steps that failed:

triage:
  needs: test
  if: failure()
  runs-on: ubuntu-latest
  permissions:
    contents: read
    pull-requests: write
  steps:
    - uses: actions/checkout@v4
    - name: Pull the failed job log
      env:
        GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      run: gh run view ${{ github.run_id }} --log-failed > run.log
    - name: Triage (read-only)
      env:
        LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
      run: ./log-triage.sh run.log "checkout applies a valid promo code" > verdict.txt
    - name: Post an advisory comment
      env:
        GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      run: gh pr comment "${{ github.event.pull_request.number }}" --body-file verdict.txt

Two restraints in that file are doing the safety work. The job runs on failure() and posts a comment with gh pr comment; it doesn't set a status check, it doesn't block the merge, and it doesn't write a line to the codebase. The permissions block grants read on contents and write on pull-requests, and nothing else: the agent gets to talk, it doesn't get to vote. The verdict sits next to the link to the actual failed job, a comment a reviewer reads, not a gate the pipeline waits on. If you later automate a response, drive it off the deterministic part, rerunning the test classes the verdict tagged flaky, and never off the prose. The line that decides merge-or-don't stays with the test result and the human, exactly where it was before the agent showed up.

Quick Wins

🟢 Easy (10 min): Take one failed log and pipe it straight to a model with a read-only ask: "Summarize this CI failure in three sentences and quote the single line that best explains it; if no line clearly explains it, say so." No classification, no automation, just summarization with a citation. You'll see at once where it helps and where it reaches.

🟡 Medium (1 hour): Build the log-triage wrapper with the closed classification set and the mandatory insufficient_evidence path, then run it against your last twenty failed runs where you already know the answer. Count how often its classification matches reality and how often it cites a real line. That hit rate, not the demo, is what tells you how much to trust it.

🔴 Advanced (half day): Wire it as a CI step on failure() that posts an advisory PR comment, with permissions limited to pull-requests: write and nothing more. Keep the verdict out of the merge gate entirely; if you automate any response, drive it off the deterministic classification, rerunning the tests tagged flaky, never off the summary prose. The agent reports; the gate still belongs to the test result.

Next Week

I want to stay with the theme of agents whose job is to read carefully and decide nothing, and look at what changes when the thing you point one at isn't a log at all. More on that soon.

That fourteen-thousand-line log isn't going to get shorter, and the red X isn't going to start explaining itself. Pointing an agent at it is the right instinct: a wall of text that needs reading is precisely what these models read well, and getting a classification, a cited line, and a one-sentence summary back in the time it takes to switch branches is a real gain on a real, daily friction.

The trap is the small slice of the time the summary hardens into a verdict and a calm wrong answer sends you chasing a failure that was never there. Keep the agent on the right side of that line and it earns its place: it reads the scene, it bags and tags the evidence, and it files a report you can check against the log in seconds. It does not rule the cause, because ruling the cause on incomplete evidence delivered with total confidence is the one thing it does worse than you do, tired and annoyed at the third red build of the day.

Let the agent read the crime scene. Keep the verdict for the evidence, and for yourself.

P.S. I build the TestScout MCP suite in the open, and read-only-by-default is the whole philosophy behind it: the piece furthest along, LightScout, is a Chrome extension and an MCP server whose check_threshold tool fails a build only when Core Web Vitals are genuinely bad and otherwise stays quiet, a deterministic gate that never guesses. If this issue helped, forward it to someone staring at a red build right now. If someone forwarded it to you, subscribe at bashmatica.com.