I’ve been building a complex agentic system where the consequences of being wrong are immediate and measurable. Not theoretical. Not deferred. You know within minutes whether a decision was right.

At some point I looked at what I’d built around the AI reasoning layer and noticed something I hadn’t intended. I’d accumulated a substantial amount of conditional logic, checks, blocks, overrides. Each one added in response to a specific failure. Each one individually rational. Together they’d become a system where failures were being suppressed rather than fixed.

I started pulling them out. That process taught me more about how these systems actually work than anything I’d read.

The instinct that’s exactly backwards

When you build a complex system with an LLM at its core, the engineering instinct is to constrain. Something goes wrong, you add a gate. The gate works, the failure stops surfacing, and you move on.

The problem is that over time. Every conditional block that intercepts a model’s decision also prevents you from seeing what the model would have done. The reasoning is hidden. You can’t tell whether the model was getting it wrong, or whether you were preventing it from getting it right. The feedback loop is broken at exactly the point where it needs to work.

I added a gate that blocked a certain type of decision. The failures stopped. When I removed it, I discovered the model had been getting that decision right. I just couldn’t see it.

Worse, when you add a judgment gate you’re encoding your own understanding of the domain into rigid code and using it to override the model’s reasoning. Your understanding expressed in conditions is always going to be less capable than the model’s understanding expressed through actual reasoning, provided you’ve given it the right context. The gate isn’t making the system smarter. It’s making the system look smarter while keeping it dumb.

The instinct to constrain comes from decades of software engineering where deterministic systems required explicit programming for every scenario. That instinct is exactly backwards when you’re working with something that reasons.

The middleman problem

For decades the relationship was human and computer. Humans translated their logic into code. Deterministic, explicit, every branch accounted for. It worked because human logic and computer logic are structurally compatible, both operate on rules, conditions, sequences.

Then AI arrived as a third element. Now the chain is human, code, AI. The code layer is still if/then. The AI layer reasons. They don’t speak the same language.

The mistake I made early on was treating the code layer as the primary interface to the AI, writing conditional logic that intercepts, filters, gates, and overrides the model’s output. The code becomes a middleman designed for a different paradigm, now trying to supervise something it can’t understand.

The code layer has a different job. It’s the connective layer between reasoning and the world. It gives the model its senses, data, state, context, everything it needs to perceive what’s happening. It gives it memory that persists across calls. It gives it the ability to act, translating decisions into real outcomes. Without it, the reasoning layer floats in isolation. It can think but it can’t perceive, remember, or do anything.

That’s a legitimate and important role. What it can’t do is judge. When you start writing conditional logic that tries to supervise the model’s decisions, intercepting outputs, adding gates that override reasoning, encoding your own understanding in if/then blocks, you’ve confused the connective layer for the thinking layer. The wiring doesn’t reason. It connects.

The spinal cord and the cortex

When you touch something hot, you recoil before your conscious mind has processed what happened. The withdrawal is a spinal reflex, fast, automatic, below the level of deliberate reasoning. It fires before the cortex is consulted, not because the cortex would get it wrong, but because you can’t afford the latency and the outcome is binary.

In my experience, this is the one place a mechanical gate belongs in a system built around a reasoning layer. Hard limits. Catastrophic outcome prevention. Things that fire before reasoning is consulted, not because the model can’t reason about them, but because they’re non-negotiable regardless of what the reasoning produces.

Everything else is cortex work. Pattern recognition, context-sensitive judgment, reading situations that don’t fit a template. Encoding cortex work into spinal reflexes is an architectural mistake. The spinal cord can’t hold a mental model of the domain.

In practice this meant most of what I’d originally written in code had no business being there. Blocks that checked whether a particular type of situation was appropriate for a particular type of action, those were judgment calls dressed up as infrastructure. When I removed them, two things happened. Some decisions got better immediately because the model was no longer being overridden. Others got worse, and those failures were specific and traceable, a concept the model hadn’t quite understood, a nuance missing from the context I’d given it. Both outcomes were useful. The first confirmed the understanding was real. The second told me exactly what to fix.

What it actually means to teach rather than constrain

The work shifts from writing rules to building understanding. Not instructions that tell the model what to do in specific situations, but context that gives it a genuine mental model of why things work the way they do in your domain. The difference is a rulebook versus an education. Rulebooks break at the edges. Education generalises.

In practice this means writing strategy and context documents that explain reasoning rather than prescribe actions. Instead of “in situation X, do Y”, what is situation X, why does it arise, what does it mean about what’s happening, what would a well-informed person in this domain actually think about it. The model doesn’t need to be told what to do when it genuinely understands the situation. And when it doesn’t understand, you can see that in what it produces and write better context to address it.

There’s a real tension here though. Context windows have limits. API calls have costs. The more richly you explain the domain, the more tokens you spend on every cycle. This is where the code layer earns its place again, not as a supervisor but as an engineer of context. Deciding what the model needs to carry forward versus what can be reconstructed fresh. Compressing understanding into prompts that are dense without being noisy. Getting more signal into fewer tokens. That’s a genuine craft and it doesn’t resolve cleanly. You’re always trading off richness against efficiency, and the right balance shifts as models get better at doing more with less.

The feedback loop this creates is tight and honest. When the model makes a mistake, you can trace it. Was the concept explained clearly? Was the nuance captured? Was there a conflicting instruction somewhere that created ambiguity? Every failure becomes a specific, improvable thing rather than a reason to add another gate. You’re refining understanding rather than patching behaviour.

For any gate you’ve added, is it there because the model genuinely can’t reason about this correctly, or because you haven’t given it the context to do so?

In my experience, the answer is almost always the second one.

A note on continuity

LLMs present like thinking. They use language the way thinking uses language. That similarity is useful, it’s why explaining context works, why the mental model of a well-briefed colleague gets you somewhere real.

But the similarity breaks down in one specific way that matters enormously if you’re building systems. The model has no continuous experience. No felt weight from the last bad decision, no accumulated caution, no emotional state that carries across calls. Each call is a complete moment of reasoning, stateless, with no experiential continuity from what came before.

That’s simultaneously a limitation and an asset. The limitation: you have to engineer continuity the model doesn’t have natively. State that persists, previous context passed back in, accumulated understanding held in the prompt. You’re building the memory in code, deciding what the model needs to carry forward versus what can be reconstructed fresh each time.

The asset: there’s no emotional drift, no fatigue, no tendency to get stuck on a read that stopped being valid. Each cycle is a fresh assessment from the same baseline of understanding. In domains where accumulated bias is a real risk, the absence of felt history is a feature, not a bug.

Most of what I’ve learned about working with these systems has come from building something real, watching it fail, and understanding why. Reading about it gets you part of the way. The rest comes from having real stakes attached, the kind where the feedback is immediate, specific, and honest.