Rerail: keeping AI coding agents on track
Today we are announcing Rerail. It checks the code an agent writes while the agent is writing it, and when a rule is broken it tells the agent which rule and where.
AI coding agents now write a huge amount of code, and anyone who works with them every day has met the same frustration. You tell the agent how the codebase is organised and which conventions to follow, in AGENTS.md or the prompt, and it hands back code that ignores both. The code may even pass its tests. It is simply not built the way the team intended.
Today we are announcing Rerail. It checks the code an agent writes while the agent is writing it, and when a rule is broken it tells the agent which rule and where. It’s in the moment the agent introduces the issue while it’s still cheap to fix. It is most effective at keeping agents on the design your team actually wants.
We are also publishing the research that motivates it: Metrics-in-the-Loop: Cheap, Timely Course Corrections Improve Coding Agents. Its central finding is simple and remarkable: a short, specific correction delivered during the work does far more than general guidance delivered before it.
The idea: correct an agent while it works
Our research started with a question that's easy to state: if you show an agent a problem in the code it has just written, does it fix it?
We chose code complexity for our test case as it’s easy to measure and understand. We compared two conditions. In the first, the agent received a general reminder, after each change, to keep its code simple and well tested. In the second, it received the same reminder together with a specific message naming each function it had just made too complex.
The reminder alone helped very little, if at all. Coding agents were still in ~50% of cases adding complexity and oversized functions. With the specific message, that complexity share fell close to zero, and it stayed there as work progressed.
The surprising thing is what the agents did with the feedback. They didn't just chop functions up to make a number go down. Of the biggest changes we read manually, most split the code into named, testable pieces. Things we never measured for them improved as well: less duplicated code, less verbose code. Functional correctness held steady throughout. We weren't trading working software for tidy-looking software.
We were getting better software.
The lesson is about specificity and timeliness. "Keep the code simple" leaves the agent to decide what simple means, and it mostly decides its own code is fine. "This function is too complex" gives it a concrete problem to solve.
Coding agents do not consistently follow written guidance
Nearly every team using coding agents keeps a file called AGENTS.md, or something like it, explaining how the agent should work. The idea is sensible: write the rules down once and the agent follows them. These files tend to grow over time. The agent makes a mistake, so someone adds a rule. It makes another, so someone adds another. Before long it's a lengthy document that half the codebase contradicts, and the agent still gets things wrong.
We wanted to know when, and why, written rules fail. So we replayed real changes from large open-source projects, chosen because their maintainers had first rejected them for breaking the project's design, and asked agents to make them under different forms of guidance. Three findings stand out, and the first two genuinely surprised us.
A few rules work. Many rules don't. With no guidance at all, agents followed a rule in only 42% of runs. Given it as one of a handful of rules relevant to the task, they followed it in 90%. But as one of eight rules or more, compliance fell back to only two-thirds. A longer list made the rule one voice among many. With Rerail checking it at the edit, even among 96 rules, it was followed in 98% of runs.
A few rules work. Many rules don’t.
Share of coding-agent runs that followed a standard coding guideline.
- No rules
- Rules written in the prompt
- Rerail: rule checked at the edit
- No rulesagent left to itself42%
- A handful2–4 rules for the task90%
one voice among many
- 8 rules74%
- 32 rules73%
- 96 rules67%
the same rule, plus unrelated rules
- 96 ruleschecked at the edit98%
The code around the agent is louder than the rules. When the files being edited already broke a rule, the agent tended to copy them. Where a file broke it six times or more, a rule among many was broken in 71% of runs. In one run, the agent quoted our guideline, noticed its new code broke it, noticed the neighbouring code did too, and carried on. It knew the rule. It followed its neighbours. This has implications for agent written code and its quality eroding over time.
Agents write like the code beside them.
The same rule, among eight or more rules in the prompt, in two kinds of file.
- Method keeps the rule
- Method breaks it
- Agent’s new method
A clean file
Rule broken in 7% of runs
A file with 6+ breaks
Rule broken in 71% of runs
AGENTS.md is more of a suggestion than a rule. We looked at how agents read and uncover rules that projects already state in their own documentation. A rule in a document the agent wasn't pointed at did no better than no rule at all. Even rules in the file the agent always reads were broken by three of four models in over a third of runs.
Where the rule lives decides if it’s followed.
Share of coding-agent runs that follow a project’s own documented rule, by where it is written.
- No rule20%followed
- In an unlinked doc24%followed
- In a linked doc34%followed
- In AGENTS.md65%followed
- Anywhere, with Rerailchecked at the edit98%followed
Why timely feedback works
Think about how people learn a codebase. A senior engineer doesn't hand a new starter a hundred-page document, disappear for a week and expect perfect results. They review the work, point at the problem and explain what needs to change. The feedback is about the code in front of them, and it arrives while there's still time to act on it.
Agents, it appears, respond to exactly the same thing. A rule in the prompt competes with everything else in the context, including every bad habit already in the codebase. A message that arrives at the moment of the mistake, naming the rule and the line, doesn't have to compete. And unlike human review, these checks can run on every change, instantly.
How can Rerail protect your code?
Rerail turns that idea into something you can use today.
You define the rules that matter to your project. For example:
- React hooks are called before any early return.
- Only the
billing/module calls Stripe, and amounts there go through theMoneytype, neverparseFloat. - Code that runs in the browser never imports a server module: no API keys, no direct database access.
- A migration doesn't add a foreign key without
NOT VALID, otherwise it locks the table in production.
These aren't suggestions in a markdown file. They're checks that run against the code as it’s written. As the agent works, Rerail checks what it has just changed. When a rule breaks, the agent gets a short message naming the rule and where it was broken, and fixes it before moving on.
In our experiments, this made all the difference. With 96 rules in the prompt, agents broke the guideline in a third of runs. With the same rules delivered at the edit, they broke it in 2 runs out of 103. The rules hadn’t changed. Only the timing had.
Rules in the prompt slip. Rules at the edit hold.
Share of coding-agent runs whose final code follows a one-line coding guideline, among 96 rules.
- No guidance
- 96 rules written in the prompt
- Rerail: 96 rules checked at the edit
Claude Opus 5.5
- No guidance38%
- 96 rules written in the prompt69%
- Rerail: 96 rules checked at the edit100%
Claude Sonnet 5.5
- No guidance37%
- 96 rules written in the prompt69%
- Rerail: 96 rules checked at the edit94%
DeepSeek V4.1 Flash
- No guidance50%
- 96 rules written in the prompt65%
- Rerail: 96 rules checked at the edit100%
This is great for teams who have watched their AGENTS.md grow past the point anyone reads it. You can keep that file short, and let the checks carry the rest.
Setup is one command:
npx @neralabs/rerail setupRerail doesn't use another AI model to decide whether a rule was broken. Its checks are deterministic: the same code gets the same answer every time. That keeps feedback fast and repeatable, and it means you can start with the few rules that matter most, and add more as you learn where your agents go wrong.
Where this goes
Writing code is getting easier by the month. Making sure that code follows the intended design, stays maintainable, and does what the organization needs is still hard, and as agents write more of our software, that's where engineers will spend their time.
I don't think the answer is ever-longer prompts, agent verification loops, and more tokens. It's systems that measure what agents are doing and correct them as they work.
Rerail is our first step, and I think it's a big one.
Try Rerail
Rerail is open source, you should try it, and let me know how you get along. If you want to follow our journey and where this leads, join the mailing list below, explore the project on GitHub, or read the research on arXiv.
If you've spent more time than you'd like reminding your agents of the same rules, I'd love to hear from you: hello@nera.software.