The Learning Loop: Turning AI Coding Failures into Better Engineering Systems

Part six. The first five covered why you need a Loop, building one with Claude Code + Codex, stopping drift with Task Packets, turning prompt rules into machine gates, and keeping the Loop itself from becoming the bottleneck. This one asks the next question: when the same mistake keeps recurring, how can the Loop actually learn from it instead of re-diagnosing it every time?

In brief

AI can fix a problem quickly within a single session, but it rarely forms engineering memory that carries across tasks and projects on its own.

The result: the same drift from the main line, the same weak tests, the same review rework and process friction keep showing up in different tasks. The team pays the diagnosis cost every time, yet the system never actually gets better.

This post introduces the retrospective-aggregation layer of Loop Engineering: record scattered problems as minimal events, periodically identify recurring patterns, then let a human decide whether to distill them into general rules, project constraints, tooling — or drop them outright.

The goal is not to let AI modify its own rules automatically. It is to build a controlled engineering-learning mechanism.

AI fixed the problem — so why does it recur next time?

In a single AI-coding session, the model may already know that:

• A heartbeat can't just carry an internal ID;
• A characterization test has to first prove it can actually catch a bug;
• One homogeneous feature shouldn't be split into five tiny Task Packets;
• A structural change can't be merged midway through an implementation phase;
• After a review passes, you shouldn't keep polishing unrelated details.

But that knowledge usually lives only in the current context.

Once the session ends, the context is compacted, the model is switched, or the project changes, it can all vanish. On the next run, the AI starts from local information again and walks straight back into a similar dead end.

Which tells us:

Session context is not organizational memory, and a prompt is not a reliable learning system.

Real engineering memory has to live outside the model.

The most direct move is often the wrong one

The easiest thing to do after finding a problem is to add a rule immediately:

text
Find a problem → edit the prompt → add a Guard → add a Hook → add a checklist item

It feels safe in the short term, but over time it produces three consequences.

Rules only ever grow

A one-off incident can permanently become process cost for every project.

Project problems pollute general rules

A constraint specific to one business gets wrongly generalized to others, producing false positives and pointless confirmations.

The current task gets interrupted by governance work

The AI was delivering a business result, but pivots to editing the Loop, the Hooks and the review templates — and the main line is displaced by process yet again.

So the Loop can't grow a new layer of armor every time it gets hurt.

From "fix the rule now" to "record the fact first"

The healthier path is:

text
A real task hits a problem
  → record a minimal event
  → continue the current main line
  → aggregate periodically
  → identify recurring patterns
  → human judgment ┬→ promote to a general rule
                   ├→ keep it project-level
                   ├→ improve tooling
                   └→ ignore or retire

An event record doesn't need the full conversation and logs. It only needs:

• Which project and task;
• Whether the problem is security, quality, main-line, efficiency, or tooling;
• Severity;
• A stable pattern identifier;
• How much rework or waiting it caused;
• An optional evidence path.

For example:

text
category: efficiency
severity: MAJOR
recurrence_key: packet-over-fragmentation
summary: a homogeneous implementation split into several micro-packets, each repeating the full review
friction_minutes: 90

The point is to record the shape of the problem — not to copy the user's exact words, the source, the logs, or the model's long-winded output.

Automatic signals and human signals must stay separate

Some problems are easy for tools to detect automatically:

• Codex returns REQUEST_CHANGES;
• The same task goes through multiple rounds of rework;
• A Guard fails to execute;
• A review process times out;
• A Hook keeps rejecting the same kind of operation;
• A test or machine verdict can't be parsed.

Others can only be judged by a human:

• The AI's progress note is incomprehensible;
• Execution is drifting off the main line;
• Process-building has outgrown business implementation;
• The AI keeps fixing details but produces no visible result;
• The user keeps shuttling context between two agents.

Automation is good at collecting objective signals. It is not good at judging experience and direction on your behalf.

One occurrence does not earn a general rule

An ordinary problem should meet at least one of these before it becomes a general candidate:

• The same problem recurs three times;
• The same problem appears in two different projects;
• The problem caused a clear, reproducible, major loss.

A security-level BLOCKER can trigger an emergency retrospective on a single occurrence — but it still has to prove it's real.

This threshold matters. It separates "the model occasionally errs" from "a systemic engineering defect."

Where should a lesson land?

Becoming a candidate doesn't mean you must add a general rule. Every candidate should have one of four destinations.

Promote to a general rule

For problems that recur across projects, with stable boundaries and clear payoff.

Example: a characterization test must complete a mutation self-proof before its first review.

Keep it project-level

For things tied to one business's state machine, deployment environment, or data semantics.

Example: a specific channel error must not count toward the health failure rate.

Improve tooling without adding a rule

If a script, Hook, or harness can eliminate the problem directly, there's no need to ask the model to "remember."

Example: when a Task Packet is missing a Guard Profile, the launch script refuses to run.

Ignore or retire

Some gates have high false-positive rates and low payoff, or are already replaced by native platform capabilities. Those rules should be deleted, not kept forever.

A healthy Loop must be able to delete.

Four real improvement patterns

Weak tests become mutation self-proof

Across several tasks, the first review kept finding the same thing: the test passed but never proved it could detect a bug.

What got distilled wasn't "review one more round" — it was moving verification earlier:

text
deliberately break the implementation → the test must fail → restore the implementation → the test passes again

One deterministic pre-check replaces multiple expensive rounds of rework.

Too many micro-packets become risk-tiering

The same set of zero-behavior, additive implementations was split into several sub-packets, each going through a full confirmation and an independent review.

After the retrospective: homogeneous, low-risk sub-packets share one confirmation and one closing review; behavior changes, wiring, and migrations keep the strict process.

This isn't lowering safety — it's matching governance intensity to risk.

Mid-stream merges become Packet boundaries

Merging an upstream structural change in the middle of an implementation invalidates the recon, the plan, and the test baseline you'd already finished.

The rule that emerged isn't "no merging" — it's that structural merges may only happen between two Task Packets.

Frequent confirmations aren't necessarily a defect

Git commit, push, and production operations trigger confirmations constantly. That looks like process friction, but it may be exactly the safety gate working as designed.

Part of a retrospective's value is proving that some friction should stay.

Rule promotion needs its own gate

Before a general rule takes effect, it should at minimum complete:

1. Reproducible evidence of the problem;
2. A statement of what it affected;
3. An assessment of the added process cost and false positives;
4. Regression tests for any tooling change;
5. Review by an independent reviewer;
6. Human approval of whether to promote.

A retrospective report itself must not automatically modify a Hook, a Guard, or a general constraint.

Otherwise "self-improvement" easily becomes uncontrolled self-rewriting.

How to tell whether the learning loop works

Don't just count how many rules you added. Also watch:

• Whether the recurrence rate of the same problem falls;
• Whether the first-round review pass rate rises;
• Whether the average number of review rounds per task falls;
• Whether pointless Hook confirmations decrease;
• How often the main line is interrupted by process;
• The extra wait each new rule introduces;
• How many low-payoff rules were downgraded or deleted.

If the rules keep growing but the same problems keep recurring, the system is just accumulating documents — it isn't actually learning.

The current implementation is only a first step

Retrospective aggregation does not mean the AI now has the power to evolve itself.

Events can be noisy, and a Hook count doesn't necessarily equal a real problem. A single periodic report can't prove a rule works over the long run; the process payoff still has to be observed across later tasks.

So the more accurate statement is:

The Loop gained the infrastructure for controlled learning — not the authority to learn automatically.

Closing

A Loop that never runs retrospectives can only make the current task more cautious.

A Loop that adds a rule every time something goes wrong eventually gets crushed by its own process.

A Loop that works over the long run has to close a different circuit:

text
execute → expose the problem → record the fact → aggregate the pattern
→ human judgment → minimal improvement → verify the effect → retire when needed

The maturity of AI coding isn't the model never making mistakes again.

It's that the same mistake stops making the team pay the same cost forever.