Teaching an Agent to Triage: Automating First-Level Build Failure Analysis

In plain terms — A failed build needs triage, not a robot dumping thousands of log lines. This story follows an AI taught to check the quick signals, sort the problem into one of twelve common categories, and gather only enough evidence to explain the cause. Like an ER nurse, it follows a safe diagnostic playbook and leaves the actual repair to people.

The red X is an interruption

Auction.com's CI/CD footprint runs more than 10,000 pipeline executions in a month. The platform supports daily releases with a rollback rate below one percent. At that scale, a failed build is not an unusual event. It is an interrupt routed to whoever can explain it fastest.

The first few minutes of that interruption are remarkably repetitive. An engineer opens the Jenkins build, checks whether it is still running, scans the stage view, jumps to the console, scrolls near the end, searches for a familiar error, and decides whether the problem belongs to the application team or the platform team. Most of that work is navigation and classification. The actual fix may require deep expertise, but the first pass usually asks a smaller set of questions: Where did the build fail? What kind of failure is this? Which lines prove it? What should happen next?

That distinction mattered when I started building first-level triage into Memex, Auction.com's internal AI coding-agent platform. I was not trying to make a model repair every pipeline. I wanted it to perform the initial investigation an experienced SRE performs before handing the issue to the right person.

The tempting implementation was one prompt: give the model the console log and ask for a root cause. It was also the wrong abstraction.

Why pointing an LLM at the logs fails

A Jenkins console log is an artifact of execution, not an explanation. It mixes checkout output, dependency downloads, compiler messages, test runners, deployment steps, shell tracing, retries, and cleanup. The line that matters may occupy three lines in a log containing tens of thousands of tokens. Sending the whole artifact to a model spends most of the context window on evidence that will be discarded.

Some console logs are multi-megabyte artifacts. Their size is not evidence of their diagnostic value: repeated progress output can be large while the useful failure signature remains small. Treating the context window as a bigger scroll buffer simply automates the least efficient part of manual triage. The agent needs controls for moving through the artifact, not one enormous reading task.

That creates a cost problem, but cost is not the most important failure. The larger problem is loss of structure. A human does not normally begin by reading the console from line one. We first look at build metadata and the stage view. Those structured responses tell us whether the build is complete, which stage failed, how long it ran, and what triggered it. Only then do we open the log at the likely failure boundary.

A raw log dump reverses that order. It asks the model to recover information that Jenkins already knows. The model must infer the failing stage from noisy text before it can reason about the error inside that stage. More tokens do not repair the missing navigation model.

There is also a reliability problem. "Find the root cause" does not specify a triage procedure. Faced with incomplete or ambiguous output, a model can select the first error-looking line and produce a plausible explanation. The result may read well while confusing a downstream cleanup error with the event that actually failed the build. Without an evidence contract, there is no obvious way for the reader to distinguish a diagnosis from a guess.

The lesson was straightforward: build triage is a procedure, not a prompt. The agent needed a way to navigate Jenkins, a sequence for using those controls, a failure taxonomy, and a fixed definition of done.

Build an interface, then encode the judgment

I split the system into two cooperating layers.

The first is a read-only Jenkins connector: the tool layer. Across the connector and the skill's operations, the extension exposes 11 tools to the agent. The public architecture describes five core build-navigation tools: get build metadata, get the pipeline-stage breakdown, tail the last N log lines, search a log with a regular expression and surrounding context, and list recent builds.

The second is a markdown triage skill: the procedure layer. It tells the agent how to resolve a build reference, when to stop early, which structured data to request first, how to map signals to a failure category, what evidence to collect, and how to format the report.

flowchart TD
    Request[Build reference] --> Skill[Triage skill]
    Skill --> Metadata[Build metadata]
    Metadata --> Exit{Green or running?}
    Exit -->|Yes| Summary[Short status summary]
    Exit -->|No| Stages[Pipeline stage breakdown]
    Stages --> Tail[Bounded log tail]
    Tail --> Table[Signal to action table]
    Table --> Search[Targeted log search]
    Search --> Report[Root cause report]
    Metadata --> Jenkins[Jenkins REST API]
    Stages --> Jenkins
    Tail --> Jenkins
    Search --> Jenkins

The connector gives the model capability. The skill gives it judgment. The model is the reasoning glue that chooses the next narrow operation based on the last result.

This split avoids a common agent-tooling trap: a single opaque operation named something like "triage this build." A mega-tool could hide the entire workflow behind one call, but it would also hide the investigation. Narrow calls leave an auditable trail. They let the skill compose the same primitives differently for a failed test, a deployment timeout, or a permission error. The skill also forbids bypassing the connector with direct shell or HTTP calls, keeping the path consistent and read-only.

Design the tools for imperfect inputs

The tool boundary has to absorb the variation that humans introduce. A Jenkins job may arrive as a full URL, a slash-separated shorthand, a flat name, or a reference with a build number on the end. The connector normalizes those forms through one shared helper and converts nested jobs into the path shape expected by the Jenkins API. The agent does not spend a reasoning step reformatting the identifier before every call.

The HTTP boundary is defensive for the same reason. A misconfigured Jenkins authentication path can return an HTML login page with a successful HTTP status. A naive client sees success and hands markup to the model as though it were build data. The connector checks both the status and the response shape. A non-JSON body becomes an explicit tool error with a bounded excerpt, not an invitation for the model to reason over a login screen.

These details sound mundane because they are mundane. They are also what turns a demonstration into an operational tool. An agent API should accept the forms people actually provide and fail loudly when the remote system violates its contract.

Spend context on signal

The triage skill has a hard ordering rule: request the pipeline-stage breakdown before reading console output. The stage response is a few hundred tokens and usually names the failure location directly. A complete console can consume tens of thousands of tokens while still leaving the location implicit.

That ordering is the first part of the system's tokenomics: buy the cheapest, highest-signal observation first. If the stage view answers the question, do not pay to rediscover the answer in raw text.

The second part is to make unbounded log retrieval unavailable. The connector offers a tail operation whose default is the last 200 lines. It also offers a regular-expression search that returns a bounded amount of surrounding context, five lines on either side by default, with a cap on the number of matches. When output is truncated, the tool says so and tells the agent that a search can retrieve a more useful slice.

This is different from silently clipping a full-log response. Silent clipping lets the model believe it has seen all available evidence. Explicit truncation makes incompleteness part of the tool contract and supplies the next action.

The sequence is therefore progressive:

1. Check status and other build metadata.
2. Inspect the stage breakdown.
3. Read a bounded tail near the failure.
4. Classify the visible signals.
5. Search for the category's specific evidence pattern.
6. Ask for more only when the current slice cannot support a conclusion.

I arrived at this shape through a deliberate tokenomics rework of both the connector and the skill. The rule is not Jenkins-specific: when an agent works over a large artifact, expose tail, search, and pagination rather than "get all." Context should expand in response to uncertainty, not by default.

Turn the runbook into a signal-to-action table

Giving the agent safe access to Jenkins solved navigation. It did not teach the agent how an SRE interprets what it finds.

Much of first-level triage is an implicit lookup. I see a compiler signature, so I look for the first compilation error. I see a test runner's failure summary, so I gather the failed test and its assertion. I see an out-of-memory signal, so I confirm the resource failure rather than blaming the command that happened to be running at the time.

I encoded that lookup as a table in the skill. Each row connects a category to observable signals and a specific log-search pattern. The taxonomy contains 12 first-level outcomes:

1. Compilation / Build
2. Test Failure
3. Dependency / Artifact
4. Infrastructure, including out-of-memory failures
5. Deployment
6. Timeout
7. Permission / Credential
8. Approval Gate
9. Parameter Validation
10. Configuration
11. Quality Gate
12. Unknown

"Unknown" is important. A taxonomy that forces every input into a known bucket encourages false certainty. The unknown category gives the agent a correct answer when the available evidence does not match a maintained signature.

Beneath the table, the skill records stack-specific signatures that experienced engineers recognize quickly. A Node heap-limit message points toward the memory limit. A coverage-threshold failure points toward the quality gate and the need to check the relevant skip behavior or add tests. These entries are not free-form advice sprinkled through a long prompt. They sit next to the signals that activate them.

That structure changes the agent's job. It no longer needs to invent a search strategy from scratch for every red build. It observes a signal, selects a row, runs the row's bounded search, and uses the returned lines as evidence. The procedure becomes repeatable across runs and more stable across model changes.

The table is also maintainable in a way that model intuition is not. When a new failure signature appears, an engineer can add or refine a row, review the change, and rerun the same procedure. The durable asset is the encoded runbook, not the model's answer from one session.

From a red X to a ticket-ready report

The last design choice is a fixed output contract. A useful triage result must be more than a confident paragraph. The skill requires the report to identify the build status, failed stage, and category; include duration and trigger context; explain the root cause in one to three sentences for an SRE audience; show the supporting log lines; recommend one specific next action; and link back to the build.

That shape does two jobs. First, it makes the output skimmable. A reader can separate the agent's conclusion from the source evidence and decide whether the recommendation follows. Second, it makes the result pasteable into a ticket or handoff without translating a chat transcript into an engineering summary.

Consider the difference in information content. A red Jenkins badge says that some step returned a failing result. A console excerpt may show an error but leave the owner and next step unclear. The structured report connects the failure location, category, evidence, and recommendation in one artifact. It does not repair the build. It removes the first round of navigation needed to start repairing it.

The contract is also an evaluation surface. If the evidence block is missing, the report is incomplete. If the category says "Quality Gate" while the cited lines describe a permission failure, the inconsistency is visible. Free-form prose makes those omissions easy to miss; fixed fields make them reviewable.

The workflow includes a quick-exit ladder before any of this deeper work. A green build gets a one-line summary. A build still in progress gets a progress report. Only failed, unstable, or aborted builds enter the full decision tree. The cheapest correct answer comes first, which matters in a system handling a CI footprint where most executions are not failures.

What I can claim, and what I cannot

The architecture is concrete: a read-only Jenkins connector and a triage skill, 11 tools counted across the two layers, bounded log access, 12 categories, and a fixed root-cause report. The system codifies a repeatable first-level investigation inside Memex.

The source record does not contain a measured accuracy rate, a before-and-after triage time, an adoption count, or an MTTR change. I will not turn the scale of the surrounding CI platform into an unsupported impact claim. More than 10,000 monthly executions explains why the problem matters; it does not prove how many minutes the agent saved.

The same boundary applies to failure modes. The taxonomy can route unmatched evidence to Unknown, but it cannot make a novel signature familiar. Flaky tests can produce evidence without a stable underlying cause. Missing stage data forces the workflow onto the noisier log path. Authentication or response-shape problems must be reported as tool failures rather than misclassified as build failures. First-level triage narrows an investigation; it does not eliminate the need for an engineer.

If I were instrumenting the next version for an impact report, I would record how often triage runs, the selected category, whether a human accepts or changes it, whether the evidence is sufficient, and the time from red build to routed handoff. I would also make confidence and handoff ownership explicit report fields. Those additions would turn the current architecture story into a measurable operating story. They are future measurement goals, not results I have today.

The system is the prompt

The reusable pattern is simple to state:

1. Give the agent narrow, real affordances over the system it must inspect.
2. Encode the expert's decision procedure as a versioned skill.
3. Retrieve structured, high-signal data before large unstructured artifacts.
4. Bind observations to actions with a lookup table.
5. Require evidence and a fixed report contract.

That pattern extends beyond Jenkins. Incident triage can map alerts to the next diagnostic query. Capacity review can map saturation signals to the relevant resource breakdown. Dependency upgrades can map compatibility signals to the checks that should run next. In each case, the model is most useful when it can navigate a real system while following a procedure an expert is willing to maintain.

This is also the meta-point behind patelharsh.dev. The agent on this site is the same discipline productized: tools provide access, curated material provides grounding, and explicit procedures constrain what happens next. A useful agent is not a clever prompt wrapped around a model. It is an interface, a runbook, and a contract.

Build triage was the first skill in this series. Whole-organization code search came next, because an agent that can classify a failure eventually needs a safe way to find the code behind it.