What you need to know:
Token costs aren’t the binding constraint on enterprise AI adoption: application-development capacity is -- until citizen development happens at scale
You want to encode as much intelligence as possible in the graph and harness — structure, memory, judgment-as-data, software structure, invalid-state rejection, recurring patterns — so the model only has to reason about the genuine residual, reducing risk and cost
Model intelligence and system intelligence are substitutes at the margin: a richer harness lets a cheaper model do the job a frontier model would otherwise be needed for, the same bet the “Stupid Network” of the 1990s made on dumb pipes and smart endpoints.
No, token costs will not bankrupt enterprises. CIOs and CTOs will not have to take second jobs to pay their model bills. Nor will long context solve all our problems -- million-token context windows don’t make just providing all information up front a viable strategy.
A recent experience confirmed that for me.
If you want to maximize reliability and minimize cost (and latency!), encode as much intelligence as you can in the graph and harness your agents use. Reify what you know as entities, relationships and metadata. Retrieve what can be retrieved. Calculate what can be calculated. Enforce what must be true. Leave the genuinely ambiguous residual to the model. Make context explicit rather than asking the model to reconstruct it. A richer graph gives the model less reasoning to do, reducing risk, cost and latency.
Citizen developers + advanced models = high token spend
If you want to understand tech economics, you have to look for the rate-limited factor -- and companies face constraints in building and deploying agents that consume tokens!
Before an agentic application can consume tokens, someone has to build or deploy it -- and sadly companies lack infinite tech investment budgets and infinite numbers of engineers who can build agentic applications. Every time I run the numbers, the cost of creating or procuring agentic applications swamps the cost of the tokens required to operate them. That’s why, among the large institutions I advise, I routinely see technology budgets around USD 1 billion paired with token spend still under USD 10 million a year. [1]
But -- when knowledge workers start using coding agents to perform analyses, build financial models, manage processes and create documents -- application-development capacity stops being the rate-limiting factor on token consumption. I am a citizen developer myself. The Claude Code CLI has become my daily driver rather than a desktop productivity suite.
I no longer dragoon McKinsey associates into helping with the endless scutwork required to put on the Technology Leadership Forum. Sending invitations, confirming attendance, assembling the schedule, chasing session leaders, creating the attendee directory -- all done with skills and scripts.
Admittedly, I am somewhere toward one end of the citizen-developer bell curve. How many other knowledge workers create a declarative page-description capability so they can render presentations from a graph? {Not enough. Nowhere near enough — Ed.}
I typically have four or five agents running through much of the business day and rack up perhaps USD 1,000 a month in model costs. Then I switched from Sonnet to Opus for a few days as I worked on the entropy assessment model I’m building. My token-spend velocity increased roughly tenfold; my perceived results didn’t.
So I switched back to a world where I could have an agent work on a task for 48 hours for less than USD 200.
Only infer what you need to infer
Why does this seem to work? Each part of the graph-based harness I use takes something the model might otherwise have to reconstruct from context and makes it explicit, retrievable, computable or enforceable.
One of my pipelines, run-pipeline, analyzes business functions and identifies opportunities for automation via AI. Several of its stages -- score, enrich-entropy, generate-opportunities among them -- require actual judgment. How susceptible is this function to automation? What kind of information entropy characterizes the work? What opportunity follows from that? Those questions are the reason I have a language model in the pipeline.
Don’t ask a language model to discover something the graph already knows. Don’t ask it to infer something a constraint can enforce.
Save inference for the residual.
The graph tells the model what exists. Take subject resolution. When run-pipeline reaches the scoring stage, the model does not inspect a collection of files and decide what it ought to score. It exercises judgment over a resolved set. It never reads a filename or paragraph to figure out what counts as a subject; the graph has already disambiguated that.
A sufficiently capable model could probably reconstruct the scope from filenames, directories, descriptions and surrounding material. But there is no reason to. The graph knows what exists.
There is one canonical ontology document defining entity types, permitted relationships, relationship direction and historical exceptions left behind by earlier migrations. [2]
Even long-context models use context unevenly. Why scatter the answer among code, examples and migration history in the first place? The model can read the ontology instead of reconstructing it.
The graph remembers what earlier stages learned. An early stage of run-pipeline may produce a source, example or issue relevant to a particular subject. Later stages need that evidence.
I could put prior output back into the prompt and ask another model call to work out what matters. I could ask a later agent to rediscover the supporting material. Instead, the system records relationships such as Cites, Exhibits and Raises, so later stages retrieve them.
But remembering something isn’t enough.
The system has to remember it explicitly. I don’t allow filenames and display names to become shadow databases. If an entity is “generic,” that fact belongs in metadata. It doesn’t belong in a Generic- filename prefix that humans and agents gradually start treating as an undocumented API.
Turn judgment into structured data. Some stages in the pipeline don’t need a language model at all.
estimate-productivity.pytakesimpact_reductionvalues stored onAddressesjunctions and calculates productivity estimates.compute-bin-thresholds.pyexamines the existing score distribution and calculates statistical cutoffs.build-labor-constructs.pymaterializes buckets from the output ofestimate-productivity.
No LLM calls. Earlier stages have turned ambiguous evidence into structured data, changing the nature of the problem. What began as a problem requiring judgment eventually becomes arithmetic, statistics or deterministic transformation. At that point, continuing to treat it as a language problem buys nothing.
The same logic applies to hierarchy. If I need to know the tier of an entity, I don’t store another redundant label and try to keep it synchronized. I don’t ask a model to infer the answer from nearby material. The system walks Is-part-of and derives the answer live.
Make software structure queryable. My Code/ directory contains 1,600 machine-readable entities representing modules, scripts and functions. A function can be Is-part-of a module. One function can Invoke another. A requirement can be Fulfilled-By the functions that implement it. Some of those relationships are extracted automatically from source code using the abstract syntax tree.
There is nothing novel about representing software as a graph. Code Property Graphs have represented programs as interconnected syntax, control-flow and data-flow structures for more than a decade. But the economics have changed: structure captured once is structure the model doesn’t have to reconstruct on every run.
Imagine changing a function and asking what else might break. Normally, that is a classic coding-model task: search for references, identify callers, trace dependencies, work out which ones matter and hold enough of the resulting structure in context to reason about the blast radius.
But much of the work has already been done. The agent can query incoming Invokes relationships.
Dependency inference becomes dependency retrieval. Structure extracted once doesn’t need to be inferred again.
Make invalid states impossible. The harness refuses to represent certain states as valid.
LLMs try too dang hard to be helpful -- their training rewards what looks like success. But sometimes the correct behavior is loud failure.
One step in the pipeline, backfill-impact-reduction, populates the estimates used to calculate hours freed. When it failed to run, nothing crashed. The pipeline kept going. The deck rendered. It simply credited every function with zero hours freed. The program continued running. It was just wrong.
I don’t want the model deciding a broken intermediate state is good enough. Where I can identify such states in advance, the harness makes them impossible.
If an agent writes a new junction into the graph, a synchronous referential-integrity check ensures every target exists. If it doesn’t, the write fails. The agent doesn’t receive a warning that it can overlook. It doesn’t continue for another twenty steps building on top of the bad assumption. It stops.
Yes, I love graphs -- but that invalidates the point Codd made about rejecting invalid states decades ago not a whit.
The same principle applies to verification. A naive golden-hash system might have two apparent states:
pass / failMine has three:
match / differ / no baselineIf nobody ever captured a baseline, the absence of a detected difference is not evidence that the artifact passed. The registry therefore checks its own completeness too: registered-but-never-captured artifacts are surfaced; captured-but-no-longer-registered artifacts are surfaced.
“I don’t know” can’t masquerade as “yes.”
Encode recurring patterns. My original presentation rendering module contained 93 functions with names like draw_page_*. To create another, an agent can inspect neighboring implementations, infer the underlying pattern, distinguish intentional differences from accidents and write the 94th.
The revised architecture turns a page into a declarative description: headers, column proportions, merges and content. One shared engine interprets it. The agent’s job becomes filling out a roughly ten-line specification.
There is nothing new about hiding design decisions behind abstractions. What’s interesting when the programmer is a language model is what happens to the inference problem. The model no longer has to infer the architectural pattern from examples because the architecture has made the pattern explicit.
The API doesn’t merely make the code cleaner. It makes the problem easier for the model.
What it all buys. Taken together, these mechanisms convert work from one category into another:
inference becomes retrieval;
inference becomes graph traversal;
inference becomes calculation;
inference becomes validation; and
inference becomes specification.
What remains is the work for which inference is useful.
The expensive stages of run-pipeline still require judgment. But it is judgment over graph-resolved inputs -- a scored subject, a typed relationship, a retrieved evidence set -- rather than raw material the model must first parse and disambiguate. The harness doesn’t eliminate inference. It narrows it.
That matters for model economics. A frontier model has at least two potential advantages over a cheaper model: it may exercise better judgment, and it may be better at recovering the intended answer from an ambiguous environment.
The harness attacks the second advantage.
It doesn’t make Sonnet as good as Opus at every judgment. If Opus is genuinely better at identifying a novel opportunity, no amount of referential integrity makes that difference disappear. But I don’t want to pay the Opus premium because Opus is better at reconstructing naming conventions, rediscovering evidence, tracing dependencies or noticing that an empty result probably means something broke.
Those are problems I can remove. Model intelligence and system intelligence are, at least at the margin, substitutes. The more intelligence I can cheaply and reliably embody in the system, the less intelligence I need to buy from the model. [3]
Richer graphs, stupider models
In his 1997 essay “The Rise of the Stupid Network”, David Isenberg attacked the telecom industry’s assumption that intelligence belonged inside the network. The telephone system used intelligent switches that understood calls and implemented services. The Internet inverted the architecture: keep the network stupid, move intelligence to the endpoints and let the network move bits.
Isenberg advocated moving intelligence to a different layer and designing the network for abundance rather than scarcity. Telephone networks carefully husbanded expensive capacity; the Internet treated bandwidth as something to consume. As fiber, lasers and processors made infrastructure radically cheaper, that assumption helped turn falling bandwidth costs into an explosion of applications and traffic. [4]
Isenberg set off a firestorm, and telecom operators did not surrender quietly. They spent years trying to put intelligence back into the network -- quality-of-service machinery, elaborate traffic engineering and, yes, MPLS. Some of that machinery solved real problems. But the larger bet paid off -- low-cost IP networks have powered three decades of Internet revolution.
The argument about the Stupid Network belongs to a much older architectural debate: where should the intelligence go? RISC moved complexity from processors toward compilers. The end-to-end principle pushed functions from the network toward applications. Relational databases made declarative integrity constraints a first-class responsibility of the DBMS. Declarative languages move decisions from individual programs into shared interpreters and engines. Architecture is, in large part, the art of deciding which layer should know what.
We face the same choice with AI. Today we put an extraordinary amount of intelligence in the model and then ask it to compensate for everything the surrounding system doesn’t know: what things mean, how they relate, what happened earlier, which states are valid and which rules apply. That makes the model expensive, unpredictable and harder for users to trust.
You could argue that those saying “just throw it all in the context window” are Isenberg’s rightful heirs. They say: what DWDM did for packets, scaling laws will do for tokens.
Maybe. But I worry about two things. Memory bandwidth, rather than compute, may increasingly constrain model performance. [5] More fundamentally, operational processes often require provenance and guarantees that nondeterministic inference cannot provide.
Obviously, frontier-1 models aren’t stupid: they are wondrous machines. And there will always be a need for the most sophisticated models to address especially difficult judgments.
Putting more of that intelligence into the graph and harness and the model can concentrate on what language models are exceptionally good at: entropy reduction -- taking an ambiguous world and resolving it into useful meaning and action. Better architecture can therefore attack three barriers to AI adoption at once: cost, risk and user experience. The stupid model, like the Stupid Network, isn’t really stupid. We need to put each kind of intelligence at the cheapest layer capable of exercising it reliably.
If a relationship is illegal, encode that in the system. If an entity doesn’t exist, fail the integrity check. If you need to know what calls a function, query the dependency graph. If an API has been retired, don’t preserve an ambiguous fallback and hope the model notices which convention is current.
Opus cost me roughly ten times as much as Sonnet without producing results I could perceive as better. The harness had already removed many of the problems for which I might otherwise have wanted Opus’s superior reasoning. Sonnet could retrieve the answer instead of reconstructing it.
That leaves CIOs and CTOs with a different question: as citizen development becomes more common, how do they foster the mindsets and build the platforms that let knowledge workers build safely and cost-efficiently?
Footnotes
[1] Software engineers will certainly consume tokens as they use coding agents. But an enterprise has only so many software engineers -- and should only give them tokens if doing so makes them more productive. Citizen development complicates the argument. This is an observation from my own advisory practice, not a published statistic -- and that gap is itself telling. No FinOps report, cloud-cost analysis or vendor disclosure I could find isolates enterprise token/API spend from broader AI spend (compute, tooling, headcount). The industry is still measuring AI cost in aggregate; nobody is yet tracking tokens on their own line.
[2] run-pipeline resolves the subjects in scope by walking graph relationships rather than asking the model to infer them from the surrounding files and prose.
[3] None of this emerged from nowhere. SWE-agent showed that changing the interface through which an agent encounters a computer can materially improve the performance of the same underlying model. Harness engineering extends the idea to the tests, hooks, tools and context surrounding an agent. More recently, graph engineering has explicitly framed the progression as one from “individual intelligence” toward “system intelligence.”
[4] The “Stupid Network” was controversial inside the telecommunications industry. When ACM Networker published Isenberg’s argument, it announced a rebuttal from AT&T’s Dado Vrsalovic for the following issue. Ericsson vice president Per Jomer responded with an article titled “Why Stupid Networks Need a Little Intelligence,” which in turn prompted Isenberg’s “Backlash against the Stupid Network”. The architectural argument continued as carriers and equipment vendors invested in quality of service, differentiated services, traffic engineering and MPLS -- mechanisms that put varying degrees of intelligence back into the network. Many solved real problems. The interesting dispute was never whether networks should contain any intelligence, but which functions belonged in the network and which belonged at the edge.
[5] LLM Inference Unveiled: Survey and Roofline Model Insights uses a roofline-model analysis to show how autoregressive LLM decoding can operate in a memory-bandwidth-bound regime rather than a compute-bound one -- workload, batch size, hardware and prefill-versus-decoding all affect which regime applies, but the arithmetic intensity of decoding is often low enough that moving data, not doing the multiplications, becomes the limiting factor.
