2026
CI Failure Triage Agent
An AI agent that does the first pass when a GitHub Actions run fails, and posts the likely cause to Slack in about 25 seconds.
View the code on GitHub
Some context
When a pipeline fails, someone has to open the run and dig through the log to work out what happened. Is it a real bug, a flaky test, an infrastructure problem or a dependency issue? It's repetitive work, it interrupts whatever that person was doing, and a lot of the answers are the same ones as last week.
So I built an agent that does that first pass and hands the engineer a diagnosis to check, with the evidence next to it.
How it works
- CI fails, and GitHub starts the triage workflow.
- The agent fetches the failed jobs' logs through the API. It never checks out or runs the failed code.
- Timestamps and terminal codes are stripped, and anything that looks like a secret is redacted from the full text.
- Rules catch the clear-cut cases, like a network timeout, a full disk or a package that can't be resolved. Those need no model call.
- Everything else goes to Claude as a log excerpt of at most 12K characters, and the answer comes back as structured JSON.
- The diagnosis goes to Slack with its confidence, the log lines behind it, a next step and a link to the run.
Decisions I made
Read-only on purpose
It never reruns jobs, edits code or deploys anything, and it needs only read permissions. The engineer stays in charge.
It's allowed to say it doesn't know
Anything unclear, refused or unparseable becomes "needs human review". If the model is down, that message still gets posted, so a failure never goes quiet.
Logs are untrusted input
Anyone who can trigger a pipeline can influence its log. The model is told never to follow instructions found in a log, and Slack output is escaped so a log can't inject mentions or links.
A bug that taught me something
In early runs the demo logs printed the failure's label, so the model could simply read the answer. I switched to neutral case codes, and a test now fails if a label word ever reaches the model's input.
Where it stands
It works end to end on a demo pipeline with planted failures: 12 of 12 diagnosed correctly, about 25 seconds to Slack, roughly $0.017 per model diagnosis, and 86 offline tests. Those are 4 planted failures repeated, so they show the system works, not how accurate it is on messy real ones. Testing on real failure history is next.
Python · GitHub Actions · Claude API · Slack · pytest