An AI agent that repeats the same mistake is an intern nobody ever gave feedback to. The problem is documented, and its solution has been known since 2023. What almost no tutorial says: that solution carries a trap capable of poisoning the whole system — and it closes exactly where you think the problem is solved.
Measuring is not correcting
A fleet of about fifty specialized sub-agents, all of them scored: their level is measured, task by task. And yet none of them corrects anything. Every mistake has to be caught by hand, on every run, indefinitely.
This is the most common confusion in practice: believing that evaluating an agent makes it improve. A report card doesn't replace a teacher. What's missing isn't a better metric — it's a loop that turns the observed mistake into a rule applied on the next run. Without it, the score only documents the relapse.
Key point — Scoring an agent doesn't correct it. Without a memory loop, measurement only puts a number on the repetition.
Reflexion, and the trap tutorials leave out
The approach that holds up has a name: Reflexion (Shinn et al., 2023). The principle comes in three steps — the agent writes a self-critique after a failure, that critique is stored in a memory, and the agent rereads it cold before the next task. The gain measured in the paper is real: on reasoning and coding tasks, the agent that rereads its failures clearly outperforms the one that starts from scratch on every attempt.
The detail tutorials skip is the one that changes everything: memory is a cumulative asset. A correct lesson improves every subsequent run; a wrong lesson poisons them all, because they will reread it as established truth. A learning loop is therefore not only a mechanism for progress — it is also a vector of contamination. The right question isn't "does it learn?" but "what is it allowed to write down?".
An agent fails to extract an amount from an invoice. It concludes, wrongly: "the total is always at the bottom right". The lesson is recorded. On later runs, it ignores invoices where the total is at the top — not because of a reading error, but because its memory tells it to look elsewhere. One bad generalization, and the failure rate goes up instead of down.
Why a model can't judge itself
The most natural reflex is also the most dangerous: letting the model critique its own output. It structurally can't. Producer and judge then share the same model, and so the same blind spots: what one didn't see, the other won't see either. Worse, a generator has a self-serving bias — asked to evaluate its own work, it tends to side with itself. Panickssery, Bowman and Feng measured it (NeurIPS 2024): a model recognizes its own texts among others, and the better it recognizes them, the more favorably it scores them — self-preference grows linearly with self-recognition.
The consequence is clear: self-critique is only worth something if an external signal anchors it. Without evidence from outside, "the agent reflects on its mistake" is just a staging of reflection.
Key point — A wrong lesson is worse than no lesson: it spreads. And a model left as sole judge of its own fault rules in its own favor.
The method: five locks
- Two distinct phases. Record the lesson hot, as soon as the mistake is observed in session; consolidate the recurring ones cold, between sessions. Reflecting during the task makes execution drift; consolidation works better once the action is over.
- An evidence lock. A lesson can only be written through a single entry point, which rejects any lesson without external evidence: a human correction, a failing test, a failed fact check. No evidence, no lesson — by construction, not by discipline.
- Producer and judge kept separate. When no execution evidence exists, a distinct evaluator decides — never the agent that made the mistake.
- Index where the agent actually rereads. A lesson stored in a file the agent never loads is decoration. You have to check that it sits in the context the agent actually rereads cold.
- Prove the loop runs. Open the memory and confirm that a correct lesson was recorded after a real failure. Until that has been checked on the evidence, the loop is a hypothesis, not a mechanism.
The trap, lived: a loop that learned nothing
The first automated version of this loop, at the studio, produced an unambiguous result: zero lessons. The mechanism ran with no visible error — but it was wired to a trigger that never actually fired, while manual capture had already picked up all the material on the fly. The machine was running empty, and the dashboard showed "active".
The lesson goes beyond this case: a mechanism that exists on paper learns nothing until you've proven it fires on a real case. Presence is not wiring; an assumed cron is not a cron that runs. That is exactly the gap a memory loop is supposed to close — provided it is itself verified.
The right to record a lesson requires external evidence. An unverified loop learns nothing — it only gives the appearance of learning.
Structuring how one's own agents learn is one of the open projects at the Orogen studio. Write to the studio to talk about it.
Sources
- Arjun Panickssery, Samuel R. Bowman, Shi Feng, LLM Evaluators Recognize and Favor Their Own Generations, NeurIPS 2024 — linear correlation between self-recognition and self-preference in LLM judges — proceedings.neurips.cc.