I had outlined plans and aspirations at the previous year’s dinner. When I came to the podium at the end of 1991, I thought about everything the 126th board hadn’t accomplished — and all the political capital expended for mixed returns. Still: we published every day of the academic year despite two blackouts. We didn’t get sued. None of the corrections we had to run were catastrophic. We published at least a few good stories.

My successor called me to the stage. I received respectful applause — more due to the position than the man. I thanked each member of the board. The room exploded when I reached Adam’s name. The staff understood exactly the energy and creativity with which he had attacked the role of Opinions Editor. Despite our differences of, well, opinion, I remain grateful to Adam for his contributions to the paper, even decades later.

Adam and I ran into each other again at McKinsey. By then he had decided against litigation as a vocation and was already developing into one of the most rigorous thinkers I know on data and analytics. Senior roles at JPMorgan Chase, DirecTV, Zurich Insurance, and Point72 followed — teams as large as 170 people, enterprises with up to 60 million customer relationships. He now advises boards and senior leadership teams, teaches in Brown’s EMBA program, and runs an annual forecasting contest, which I fear to enter.

Everything we do — every business decision we make — implicitly rests on a forecast. Which is why I needed to get Adam’s view on what AI means for data, analytics, and the practice of forecasting itself.

Three things struck me as particularly worth your time.

First, the B2C-vs.-B2B gap in analytics is real — Adam puts it at twenty years — but the cause is not what most people assume. It is not that B2C data is cleaner. It is that B2C businesses have enough observations to find statistical regularities. A million customers generates surprises. Seventy-five institutional accounts does not. AI changes some of this at the margin, but domain expertise — knowing which variables to featurize, which hypotheses to form — remains the scarce resource.

Second, taking payoff curves into account as you thinking about your career and your job. Adam’s formative experience was as a litigator: limited upside, catastrophic downside, a profession structured for you to lose. He now teaches his kids to think about every career choice as a payoff curve before accepting it. Most technology managers are running the same analysis implicitly every time they commit to a timeline or decide whether to invest in redundancy. Making the curve explicit demystifies the decision.

Third, Adam’s argues that the manager who has actually formed a probability — even a rough one — about whether a project will come in on time is making a different kind of bet than the manager who just called it a plan. The first manager will be better calibrated over a portfolio of decisions. “It matters to be 53% right instead of 52% right” is not a hedge fund insight. It is a description of how operational advantage compounds.

Introduction: from College Hill, back to College Hill

James Kaplan:** Welcome to another Prosaic Times video podcast. We have with us Adam Braff, who I’ve known since about 1990. I’ll let Adam introduce himself and talk a little bit about his journey, and then we’ll start to dive deep into forecasting and what AI means for forecasting.

Adam Braff: Thanks for having me, James. I guess if I could describe what I do now, it is not strictly about forecasting. It’s helping corporates and investors figure out what to do with data and analytics, and increasingly AI. So some of that is descriptive analytics and trying to figure out how many widgets are we selling and why, and some of it is predictive, and how many widgets are we gonna sell next year and how do we make that number go up?

So predictive does come into the work that I do

James: Yes, and we all want to hear about the Narcissist forecasting contest, which I am proud never to have taken part in — because I’m sure I would lose badly.

Adam: Nobody loses in the contest. You gain knowledge, you get smarter. And even if you’re gunning for first place, which is fair to do, it actually influences your strategy. It’s like when you’re entering a March Madness bracket, do you pick a lot of weird long shots, or do you try to be as accurate as possible?

I try to persuade people to just get more accurate year over year and have the least error overall in their forecasting. So nobody loses in this contest. They get a score, they can get coaching and feedback from me for free, and then they try to get better every year

James: Okay. So tell us a little bit about your journey

Adam: We graduated into a recession, if you may recall.

James: Was not fun

Adam: That was the reason why my wife and I — then girlfriend — decided to go to law school. It was a good way to hide out from the economic storms of the early ’90s, and that was kind of a mistake, I would say. It was a big detour in my career, but I ended up going to law school and then practicing as a litigator at a big firm in DC for three years, really not enjoying any of it, and trying to find a way out.

And as it happened, I had put myself into a bit of a corner because the world doesn’t really need litigators. There aren’t a lot of obvious ways for litigators to turn into some kind of beautiful butterfly. So I took advantage of the fact that it was the dot-com boom, the first dot-com boom happening at the end of the ’90s, and ended up getting a job at McKinsey, and it was a fresh start for me.

I was 29 years old, and I just said, “Let’s, let’s start again at the bottom, and this time pretend I had gone to business school instead of law school,” I got my mini MBA from McKinsey, which consists of three weeks of getting bombarded with strategy ops, marketing, and org stuff, and then ended up staying at the firm for a lot longer than I thought I would be there.

I ended up doing work that was really what we would now call data analytics AI. At the time, I think we called it big data. It was how do companies use larger data sets than would fit in Excel and creative new kinds of data and A/B testing on their customers and calculations of customer lifetime value?

How do consumer businesses figure out how to do more of that stuff in order to attract and retain the most valuable customers? And that’s something you can do in a lot of different industries, as it turns out

James: Did you go to Mini MBA in Kitzbühel or do you go someplace else?

Adam: I went to Kellogg. I had a bunch of trainings in Kitzbühel, but I was in Kellogg freezing my ass off for three and a half weeks in January of that year

James: I’m sure it was warmer than it was in Kitzbühel and easier to get to

Adam: Exactly. It was nice.

So I stayed at the firm for 10 years. I was a generalist partner in the DC office. None of my clients were really government or particularly local to DC, so I was traveling a lot. Had three kids in that span of time, and at some point I said, “All right. It’s time for me to get a real job and figure out what I want to do in the real world.”

So I left in 2009 and had a few different offers — some in New York, one in Vegas, which my wife vetoed — and ended up going to JPMorgan Chase to become their head of customer data and analytics. That was for the Chase consumer businesses: figuring out what to do with information about customers who might have a Freedom credit card and a Chase mortgage and a Chase checking account.

What do we do to put all that information together and help the bank make better risk decisions, better anti-fraud decisions, better marketing decisions? I did that for a few years and then got recruited over to DirecTV to run business analytics there. That was different because there was an existing team of really good analytics people doing the same work but for pay television — a much simpler business, with some interesting lifetime value drivers.

James: Interesting. And then you did an insurance company for a while, and you’ve also recently been teaching at Brown.

Adam: DirecTV got bought by AT&T, a bunch of us bailed out. I ended up going to Zurich Insurance and ran data and analytics there. I got a taste of the IT world by reporting into the chief operations and technology officer — which let me see what it’s like to actually build the stuff and not just give the requirements.

Then I spent three years at Steve Cohen’s hedge fund as head of data sourcing and strategy — data acquisition, essentially. I’ve been independent since 2019. Hung out a shingle that year and started teaching at the same time.

I began at NYU, created a class called Business Analytics and Data Visualization, taught different variations of it for a while. Then we moved to Providence, where I’m sitting now, and I’ve been teaching versions of that class at Brown.

Is B2C 20 years ahead of B2B in customer analytics?

• B2C is roughly 20 years ahead of B2B in customer analytics — not because the data is cleaner, but because there are simply more data points to work with.

• The analytics opportunity in B2B looks fundamentally different: customer targeting and Salesforce effectiveness rather than churn modeling and cross-sell.

• AI and agentic data ingestion could narrow the gap — converting unstructured B2B data into analyzable signals.

• Domain expertise still matters: knowing how to featurize variables (competitor footprint, not just state) is what separates the old hands from the clever prompters.

James: Let me ask this question. You’ve done a ton of stuff in what I’d describe as the B2C world, and it’s been my observation that B2C is light years ahead of B2B on analytics — just because the data is cleaner. And I hypothesize one of the interesting things about gen AI is you can now clean up the data, correlate it, convert unstructured text into structured data you can analyze.

So I was wondering if you could comment on how far ahead the B2C world is compared to B2B on analytics, and to what extent might gen AI or agentic data ingestion and cleansing help the B2B world catch up?

Adam: It’s like 20 years ahead[1], but probably for a different reason than the reason that you set out. It’s more that there are more data points to measure in the B2C world. And so if you’re gonna list all the use cases, if a private equity firm comes in and buys a consumer business or buys an industrial business, all the things you’re gonna go work on, there are gonna be analytical big data things to do over here that have to do with acquiring more customers and retaining your best customers and figuring out who those best customers are and cross-selling and stuff.

Over here, you’re like optimizing a factory or something, right? So to the extent that there’s data stuff, and very little of my career has been over here, right?

James: Right

Adam: You’re gonna be getting information from things — what was briefly called the Internet of Things — and getting instrumentation data from over there.

So big data definitely exists, and there are use cases for it, and I sit on councils with people who do this stuff. It just looks very different from the kinds of big data and analytical stuff that I did, and that my father did before me. He’s retired now, but he was doing data-based marketing back in the ‘70s and ‘80s.

Second-generation data and analytics — something that hadn’t existed before.

James: I was thinking about this in the context of customer analytics. If you have a million households in a B2C business versus seventy-five or a couple hundred corporate institutions — which have complicated internal dynamics — the N question becomes obvious. You don’t typically have many decision-makers when someone decides to buy cable services. Maybe one, maybe two. Whereas in B2B procurement, there could be twenty decision-makers in the process, and they all interact with each other.

Adam: The analysis you would do in B2B Salesforce effectiveness would have a lot more to do with customer targeting — going out and grabbing more information about your target, using some agentic tool to pull that and put it somewhere you can analyze it and make good decisions.

The reason the million-customer B2C business got ahead is that a lot of analytics is looking for patterns, for statistical regularities. If you can determine that customers in this region are stickier, or that customers who came through this channel spend more, there are just many more ways to do those analyses — with genuine comparative advantage over an executive doing everything by intuition.

There’s at least a fighting chance for the data to tell you a different story. You think the higher-priced product is better, but the customers don’t stick around as long once you control for everything — and therefore you shouldn’t be charging more or assume that’s the right answer.

You’ll come up with intuitions like that on the B2C side. You won’t on the B2B side because you don’t have enough N to surprise anyone with that kind of finding.

James: Tell me if this is right. The more independent variables you’d have to take into account, the larger the N you need in order to get a relevant statistical sample.

Adam: I think that’s right. And I don’t want to make too much of statistical significance here. A lot of what you’re doing in B2C consumer analytics is descriptive analytics — a BI dashboard, not a giant regression model with a lot of independent variables. You want to slice your data one or two dimensions at a time and just look at the graph. Which numbers are higher? Is it this product, this region, this group of call center agents? That’s what lets you drill in another level and understand what’s happening.

You can have lots of dimensions and measures that you think of as independent variables — you’re just going to be doing them a little at a time, flexibly drilling in to find the hotspots. The regression is just one part of that.

James: What you’re saying is there’s a lot of power in just being able to do Pareto analysis — being able to look at your churn by segment, by size of spend, by region, by product mix.

Adam: Yes, exactly.

Adam: Right, exactly. So we were at DirecTV, churn was the number one thing. It was the era of cord cutting and cord nevers. probably a lot of your viewers have never thought to buy a monthly pay TV subscription ’cause it’s all, they may get it through a streaming product, but not through, a cable

James: every now and again, I will try to explain the concept of channels to my kids, and they’re just confused by it.

Adam: Yes

James: I try to explain that in New York in the 1970s you had channel 2, 4, 5, 7, 9, 11, and 13, and that was it. They accuse me of being old.

Adam: The reason I bring up DirecTV is not to make you feel old. It’s that the main problem we were trying to solve was why churn was so bad and how to make it better. And if you think about the ways to slice churn, first you want the churn rate — within any given customer population, what percentage are canceling over a month or a year?

When you start slicing it, the intuitions you might have — is it by demographics, age and gender? — that’s not what matters. What matters is things like which customers are rolling off a $5-a-month promotion this month. Those turn out to be the vulnerable ones.

Or which customers are in the Fios footprint or the Comcast footprint — does that make a difference, and is it trending differently over time? You need the data to do that kind of work.

James: So what I hear you saying is: the availability of the data, obviously, but also the person who can develop a hypothesis about what might be a relevant driver — competitor footprint, for example.

We needed that hypothesis generation in a world where analytics were expensive — someone had to go do the cuts in a spreadsheet or via SQL. In a world where analytics are cheaper, how much of it is “let’s just run all three hundred cuts and see what it tells you,” versus using a model to discern which cuts might actually have real differences?

Because I think both you and I have done an analytical cut a million times, said “region has no impact,” “size of spend has no impact,” and then — “overlap with competitive footprint, that has the impact.” You have to do nineteen analyses before you land on the twentieth that’s meaningful.

Adam: Domain expertise and taste matter because you have to figure out how to featurize these variables. Region by itself, cut generically into US states or metropolitan areas, gives you a boring answer.

But if you join that data up with another data set — this is the Comcast footprint, this is the AT&T pre-merger footprint — suddenly you’re testing something real. Knowing how to featurize the data to test hypotheses efficiently is still very valuable.

And AI tools, if you coach them correctly and bring enough taste and judgment to it, can let a sufficiently senior person test a much larger number of hypotheses. They’ll also surface a lot of spurious answers along the way.

A brand new data science graduate would have a hard time hitting the ground running in one of those businesses. Even a very clever prompter isn’t going to get to the right answer the way an old hand would.

James: Creativity and context matter here. You make the point about competitor footprint. Who knows — state may be a weak signal, but population density may be a much stronger one. You get a weak signal from New Mexico and Wyoming being different from New Jersey and Rhode Island, but when you really dig in, it’s the Providence metro versus less densely populated parts of Rhode Island that’s driving it.

Adam: And you’re keeping track of the time dimension. Things change over time. Everything changed in COVID. You might decide to cut off your analysis at the post-COVID era, or account for two competitors merging. This happened constantly with banking analyses.

One of the analyses I used to do at the firm was on customer experience. We ran a big annual panel surveying people about their banks — and it was always difficult because the banks kept changing. The American Customer Satisfaction Index, which tracks these things longitudinally, has the names of ancient banks nobody’s heard of — Chemical Bank — that we now know as Chase.

There is genuine value in domain expertise that AI cannot yet replicate.

James: What I should do one of these days is a quiz — Chemical Bank, Manufacturers Hanover, First USA — and you have to guess which bank they’re part of now.

Adam: The world’s most exciting quiz. I agree.

The debate on virtual panels

• Synthetic data is valuable for prototyping and pedagogy — stuffing a dashboard mockup with plausible fake data is now standard practice.

• Where it breaks down: using synthetic panels as a substitute for real market research, especially for purchase-intent questions. There is no signal like that in the model.

• The motte-and-bailey risk: vendors pitch synthetic panels as research-grade but deliver something closer to a prototype illustration.

• James’s CIO/CTO panel is more useful for testing message clarity and resonance than for predicting buying behavior — a meaningful distinction.

James: Never said I was cool. What do you think about virtual panels? Particularly in the B2B space, I’ve started playing with using agents to simulate the decision-making of certain people. Valid? Less than valid? Intriguing?

Adam: I remain skeptical of the whole field of synthetic creation. On the one hand, I use it for pedagogical purposes where I’m creating a data set to illustrate a point to my students. It’s great for that, especially if you’re gonna sprinkle in personally identifiable information, which you don’t wanna do with actual humans.

I use it to make, prototype dashboards. This is something that has become very common now, not just me, but many people since the Claude Code stuff really got going in earnest back in January. I’m sure you’ve done this and you’ve seen it, but many people are mocking up prototypes very quickly, and stuffing them with synthetic data is a pretty good idea, right?

To give people a sense of the art of the possible there.

James: A couple of weeks ago I created the data for a change management program at a major financial institution — all dummy data, but realistic enough to demonstrate something. That would have taken weeks to do manually.

Adam: Incorporating randomness and signal to make it feel like real data is genuinely hard. Agreed.

Now, on panels: I’m doing a lot of consumer insights work these days, back on the B2C side. You want to survey 500 Americans about something — and it’s getting harder and harder to find real people who aren’t robots or speeders or cheaters. So people are pitching synthetic panels: 500 simulated respondents. Why not?

Here’s why not. The whole point of a real panel is the signal — what actual people in the world want, the clusters that form, the principal components that emerge. It’s not just big spenders versus quick deciders; there’s a third dimension that actually explains the variance.

Trying to derive all that from synthetic data seems crazy to me. I’ve had a couple of conversations with these vendors, and I worry that what they’re pitching sits somewhere between the prototype mockup work we were just celebrating and the actual market research my clients need to make real decisions.

It’s a motte-and-bailey: they pitch it as research-grade but it’s really only as good as a prototype illustration. I’d love someone to steel-man what synthetic data is genuinely good for beyond that.

James: For the blog, I created a panel of five hundred CIOs and CTOs. I gave each member of the panel a persona — this person is the CIO of a pharma company with five hundred million dollars in IT spend, this person is the CIO of an asset manager with a billion dollars in IT spend. Then I used the OCEAN framework to create psychological profiles.

Adam: Mm-hmm.

James: Each individual has a job, a title, a company, and a personality profile. I run each draft through them. I wouldn’t bet the farm on the scores, but the quotes I get back are really useful, and I’ll go through about seven iterations. Part of me thinks it may be less helpful for predicting buying intent and more helpful for testing marketing messages — because that’s where large language models are genuinely good.

It’s hard for them to predict whether somebody will buy, but they may be better at understanding whether a message is more or less understandable, more or less compelling. Does that make sense?[5]

Adam: A little bit. You used the words “testing marketing messages.” I think it’s more like — in this case you’re testing your blog. Generating responses that would go outside of what you and I would naturally brainstorm on, how people are thinking about things.

James: It’s pretty good for that. I don’t workshop my blog posts with people who might be more emotionally invested than I am.

Adam: We’ll have to look at your Ocean scores later. We’ll compare them offline.

I like the idea of computers generating things. It’s a large corpus of material — you sift through it, see what’s interesting, throw away what’s not. You and I have been around long enough to tell the difference. You generated it cheaply, so the discount rate on discarding it is low.

But that’s not really what market research is. Qualitative research is different: a focus group has real human beings who can say unexpected things. Small N, unpredictable. But once in a while someone says “I never knew you could use the product this way” — and maybe only 1% of people use it that way, but knowing that might spark a brand campaign.

So I’m in favor of the tools for idea generation and checking your work. What I’m skeptical of is using synthetic panels for commercial due diligence. You’re deciding whether to acquire a BI tool or a security product, and you’re going to rely on numbers saying 20% of synthetic CIOs might spend more than USD 10MM on it.

That doesn’t seem right to me. I don’t think there’s signal like that in there.

James: My take is it’s more a part of the chain. It doesn’t preclude or replace going and doing the interviews. But you can’t test and re-test and re-test with a live interview set. You can workshop messages or hypotheses and then go confirm with actual human beings.

Adam: This is a good segue into forecasting.

All business is forecasting

• Every business decision is implicitly a forecast — a bet that one management action will outperform another.

• The distinction is sharpest for investors: a binary buy/short decision forces the forecast into the open. Operational managers face the same bets but rarely frame them that way.

• Tetlock’s superforecasting research showed that structured generalists — triage tractable problems, decompose, work as a team, ensemble independently — beat CIA analysts with classified intel.

• The Brier score disciplines probability estimates: being 52% right vs. 53% right sounds trivial, but at sufficient scale and leverage it is everything.

James: Go there

Adam: Due diligence is a form of forecasting. You’re trying to predict what the financials of this business are going to look like at current course and speed — and what they’ll look like if we apply our private equity magic on top of it.

All business decision-making is implicitly a forecast: a bet that this management action will produce better results than that one. The point is sharpest when you’re an investor at the margin — do I buy because it’s going up, or short because it’s going down?

James: Very binary — buy or do not buy — as opposed to operational decisions, which are more continuous in nature.

Adam: Exactly. When you’re thinking about numbers and where they’re going in the future, it probably helps to be good at forecasting more generally. And this is what Tetlock demonstrated, decades ago with IARPA and these defense intelligence agencies.

For podcast viewers and listeners who aren’t familiar: Tetlock is a professor at Penn who has been studying forecasting for a long time, and came out with a very influential book called Superforecasting, where he packaged the lessons of groups of generalists who — the legend has it, and I think I’m giving a fairly accurate version —

were able to forecast world events — will this world leader be deposed or die — better than CIA operatives on the inside who had all the intel collected expensively by our intelligence apparatus.

And so these generalists —

James: Did the generalists forecast the Knicks were gonna win the championship this year?

Adam: Nobody predicted that except my sons, who believed in it. And so it happened.

Tetlock codified the rules of what it means to be a superforecaster. These people were really good at triaging problems by tractability. Some things are impossible to forecast — the Knicks. Some are too easy — my soccer team, Tottenham Hotspur, doing badly. The Jets are a better example of the tractable middle ground: will they make the playoffs, will a company beat earnings, will this entity win a major award?

Allocate your time to the tractable problems. Decompose them into constituent parts. Work as a team, but have people develop independent points of view first, then bring the theories together and fight about who has the better argument. Ensemble different independently-made forecasts where possible.

Those are the Ten Commandments of superforecasting that Tetlock codified in 2015.[2] I started my forecasting contest that same year.

I read the book and launched the first Narcissist forecasting contest in 2016 — The Narcissist being a long-running annual newsletter I’ve produced since 1992. Twenty-five binary propositions about the world, each either going to happen or not.

Everyone predicted the likelihood of each from zero to 100%, including, at the time, whether Trump would get elected — which everyone gave very low numbers to.

The scoring mechanism is a Brier score. If I predict 40% for Trump getting elected and he’s not elected, the truth is zero: 0.4 minus 0 is 0.4, squared to exaggerate the impact of being far off, giving me a 0.16 Brier. If Trump is elected, my error is 0.6, and the square is 0.36. A high number is bad. It’s like a golf score.

James: Here’s a business question. Can you predict a probability, or do you only state one? You can predict whether a candidate is elected or not — but we’ll never know the true likelihood of that election at any given moment. You can estimate a likelihood, or you can predict an outcome. But can you actually predict a likelihood?

Adam: It is a bit of a metaphysical question. In a sense, all these predictions are wrong — everything goes to zero or one. There are people who look at the whole enterprise and say it’s stupid. There’s no difference between .4 likely and .6 likely. Just make the call. I’d call those people extremists.

The other camp — generally called Bayesians — believes that we don’t know ex ante whether a thing will happen, and that there is genuine utility in knowing whether something is 60% or 40% likely. The value is in the repetition. You make predictions over and over again.

The clearest illustration is betting markets. In 2016, Trump was trading at under 20% on the eve of the election. Nate Silver had him at 28% — out of consensus to the high side. If you’d looked at the pot odds and said “I can buy Trump contracts at 20 cents because Nate says 28,” you would have done well. Even though you thought it was less than 50% likely, you thought it was more likely than the market priced it.

I’m a Bayesian. Being able to tell a 52% from a 48% likelihood is useful because we make a lot of bets. You’re making them every day when you’re running a business. Being slightly more right and less wrong compounds.

James: Everybody I’ve known who’s managed money for a hedge fund is a fanatical Bayesian. There’ll be traffic, but it depends on three things that might drive more or less of it — let me think through the prior and posterior. Which is interesting, though it gets a bit exhausting.

Adam: One of the most common approaches in a long-short public equity fund is calling the quarter — is this stock going to beat or miss earnings? More precisely: beat or miss the whisper number on whatever KPIs matter.

The hit rate is 52%, 53%. It’s much better to be right 53% of the time than 52% — that’s how they make their money. They’re repeated gamblers doing it at a very high level, trying to maintain that little bit of edge in a zero-sum world where everyone else is trying to do the same.

It’s hard. But it matters to be 53% right instead of 52% right.[4]

James: Because a hedge fund portfolio manager who’s right 53% of the time with sufficient leverage is enormously value accretive. Fortunes can be made with a fifty-three/forty-seven split, plus leverage. Operational managers who are right 53% of the time tend to get fired. And in the technology world there’s the expression “all day, every day” — you can’t be wrong in one direction.

Actually, one of the interesting things here is that in some cases the hedge fund payoffs are symmetric. On the other hand, in an operational world, the payoffs are often highly asymmetric. Could you reflect on that — applying the insights from superforecasting to an operational environment for people who have to make decisions every day with sometimes highly asymmetric payoffs?

Asymmetric payoffs and career risk

• Hedge fund payoffs are roughly symmetric: being 53% right vs. 52% right is the whole game. Operational payoffs often aren’t — a missed security filing, a waived privilege, a breached system can be terminal.

• Adam’s formative lesson came from a malpractice case: the concave payoff curve of law (bounded upside, catastrophic downside) drove him out of litigation and eventually into analytics.

• He vibe-coded a career payoff visualizer to make this intuition concrete — the same tool every technology manager implicitly needs when committing to timelines, availability zones, or budget targets.

• Agile is a structural hedge against the same asymmetry: shorten the cycle, reduce the blast radius of being wrong.

Adam: I would focus on cybersecurity — an area you’ve spent more time in than I have. Risk management in cyber is an area with that extreme concave payoff curve[8], where being wrong in a certain direction is extremely career-limiting.

One of the formative experiences in that professional journey I sketched out earlier was a malpractice case I tried as a lawyer. Very few cases go to trial — almost everything is writing briefs and pleadings. But this one did. My client was a lawyer being sued because he had failed to supervise a young associate who was supposed to file in a telecom case and simply didn’t get it done on time.

Once they realized the deadline was approaching, they scrambled, missed the filings, and the telecom project couldn’t go forward. The client sued. We did good work at trial and settled before it went to the jury.

What that case made me realize was that litigation — and being a lawyer generally — has that extremely concave payoff curve. If you accidentally produce a document with attorney-client privileged information, when you’re going through warehouses of material —

It used to be physical warehouses — for lawyers today it’s all on a computer, it’s a cyber problem. But if you accidentally produce a privileged document, you’ve waived privilege and you’re in serious trouble. Being a lawyer felt like limited upside — a nice house if you make partner versus getting fired and never finding work again.

So I thought more carefully about these payoff curves. The first conversation I had with my kids when they were old enough was: don’t choose a profession with that kind of payoff. Do you want a venture capital payoff — an extreme tournament where the upside is a million times the median, if you’re genuinely confident in your skills? Maybe. But you’re probably overconfident. Most people want something with reasonable upside and limited catastrophic downside.

I actually vibe-coded a tool to visualize this — it lets you look at payoff curves for different employers and say: if I get a top-percentile outcome, here’s how I shake out; if I get the bottom outcome, here’s how I shake out. You can vary economic conditions and other factors. I stuffed it with synthetic data grounded in actual information. It’s more of a toy model and theoretical exercise — an intuition pump to make visible the thing you were saying. Do I really want the job where 10% of the time it’s utterly fatal?

James: It’s directly relevant. People make decisions like this every day. Do I launch this project? Do I use high availability for this application? Do I run it across multiple availability zones? Do I commit to two months versus three? Those are technology examples, but every business manager is asking: what target will I commit to, in revenue or in cost? Thinking through your own payoff matrix — and the payoff matrices of the people around you — changes the analysis.

Payoff matrices differ enormously by corporate culture. In some organizations, blowing a budget a little is “try to do better next quarter.” In others, exceeding a budget in a single quarter is career-limiting.

Adam: That’s right. The types of IT projects I mostly see and lead are business intelligence projects — building out and deploying predictive models. These, especially the dashboards, are generally developed in an agile environment, where agile is kind of a hedge against the asymmetry you’re talking about.

I’m not building a whole stack waterfall-style with a perfect delivery date. I’m making it good enough to get adoption, then making it better — two-week sprints, continuous improvement.

So I would argue that agile is a way of coping with an uncertain world. Do you agree?

James: It’s more emergent strategy than deliberative strategy. There are people who believe that whoever wrote the Agile Manifesto had some connection to John Boyd — I haven’t been able to find as much documentation for that, but the idea is the same: increase the number of cycles, shorten the duration between action and feedback, and revise your strategy accordingly.

Adam: Exactly. Agile development and hypothesis-driven problem-solving are both hedges against disaster. One of the first things you learn at a consulting firm — I taught the intro to consulting class several times — is that you have a day-one hypothesis about the answer.

Part of what’s good about that: when someone stops you in the hall and asks what you think the answer is, you have something. “I’m going to test it and refine it and make it better.” That’s much better than a long list of ideas with no organizing principle.

The risk otherwise is getting caught out with no answer at all. The OODA loop, hypothesis-driven problem-solving, agile — these are all hedges against uncertainty and against the weird events that shock you.[7]

James: The interesting thing is that the nature of the bad thing varies by the nature of the system. In an analytics system, the bad thing is bad information — you make a bad business decision. In an operational system, like an order capture system, the bad thing is different: downtime, a cybersecurity event, a transaction failure.

We deal with those types of negative payoffs differently, using different mechanisms. It’s probably worth thinking about that more rigorously. Let me pivot just a little bit.

What do we tell early career professionals about AI?

• Get a job in a for-profit company and help it make money with data. That is still good advice. Those jobs are not disappearing as fast as the pessimists predict.

• The supply of problems is highly elastic: better, cheaper analytics generates more questions, not fewer analysts.

• Proof-of-work has decayed — anyone can generate a slop deck. What has not decayed: domain knowledge, taste, and the ability to have a real conversation about a business.

• Adam’s standing advice: pick the domain you care about, grab the public data, do the analysis. Show up to the interview having already thought about how to help the company make money.

James: In my teaching, I’ve seen a lot of anxiety from people early in their careers about what they’re going to do with their lives in the age of AI. A number of college seniors asked me, and I said they should learn data structures and data science — because no matter what happens, that will be valuable. Did I give them good advice or bad advice? And what are you taking away from your interactions with college students?

Adam: Getting some kind of job and getting out into the real world to develop domain knowledge — for all the reasons we talked about — is important. Getting that job is a separate question; that’s tactics, informational interviewing, picking your spots.

But as advice for how to spend the next one to three years: get into a for-profit company and help it make money by analyzing data against real problems. Those jobs are not going away as fast as the most pessimistic people are predicting.

The supply of problems is highly elastic.[3] Having people ask good questions — and the return to a business from asking more good questions and getting quality answers back quickly — is positive. I’ve bet my whole career on the idea that most companies would benefit from having one more data scientist, or in this case one incremental unit of data science capability.

Which is not the same as replacing all the humans with AI. I’m on the optimistic side of this.

James: There’s more data, available publicly. There’s more tools that you can use for free. And my inclination, for example, if someone were to ask me, “Gee, I want a job doing analytics,” I would say, “Well, pick the domain you’re interested in and start doing the analytics.” Right? Because what’s gonna be more compelling in a first-round interview? It’s “Oh, gee, I’m, I’m really interested in the telecom industry, and I grabbed all this from the FCC and I did this analysis, and here’s what I found.”

Adam: I’ve given a talk on this subject probably every year for the last five years, including the pre-generative-AI era. The advice holds: do actual work, show that you know something about the business, demonstrate that you care about it.

I will say that that coming in with a deck, the proof of work aspect of it has gone to kinda zero, right? ‘Cause anyone can generate a slop deck. People like you and me might be exquisitely sensitive to seeing a deck and understanding immediately that it was written by AI and the, and the tells in the stylistic, that come out of a Claude, PowerPoint deck.

The concept is still correct: get in there, do the work, iterate, think about it. You’ll be able to have a real conversation about the business because you’ve learned something along the way — “I think 80% of the value is in this part of the product, and here’s why.”

You’re not reading off a slide. You’re engaging a person. There’s a lot of value in that. And people still show up at interviews without having thought seriously about the business and how they would help it make money. There’s still a lot of alpha in that.[6]

James: Terrific. Anything else to add before we wrap up?

Adam: The forecasting contest runs every year. Free to enter. You get a bowl of pho if you win, and a book — this year it’s George Orwell’s essays. All the information is at braff.co/advice: the current contest, how to enter next year, and other thoughts about food and analytics.

James: Terrific. Two great topics to discuss. Adam, thank you so much

Adam: Thanks for having me.

Notes

[1] Obviously twenty years is an estimate -- that doesn’t make it wrong. The world is entropic, and deterministic systems are not talked about how organized complexity is much tougher to understand than disorganized complexity.

A B2C business with a million customers faces disorganized complexity. Idiosyncratic individual behavior averages out into stable statistical regularities — churn rates, cross-sell lift, tenure effects.

A B2B software company with sixty enterprise accounts faces organized complexity. One decision-maker changing jobs, or having a bad quarter, creates a measurable swing in the entire dataset. The N is too small to smooth anything, especially since many of the data points are connected to each other. That is why domain expertise is required in B2B analytics in a way it is not in mature B2C analytics — the data cannot naturally control for the variables that large samples would otherwise absorb.

[2] All IT commitments are forecasts based on implicit probability estimates. When a CIO says “We’ll deliver in two months” she estimates the probability of coming in at two months is high enough to avoid the political cost of articulating a more conservative estimate.

[3] Adam describes here the applicability of the Jevons Paradox to knowledge work. When the churn data got good enough to slice, the analytics team didn’t shrink. It grew. Each answer generated three new questions — why are customers in this region stickier? What’s driving the gap between acquisition channels?

[4] You’ll never know whether the 60% probability you assigned to a project coming in on budget was the right number. It either did or it didn’t. But across a portfolio of similar decisions, the manager who has actually formed a probability — rather than suppressed the uncertainty — will be better calibrated over time. That’s what “it matters to be 53% right instead of 52% right” means outside a hedge fund. You’re not making one bet. You’re making hundreds.

[5] There is no perfect data set for the organized complexity of B2B commercial due diligence. Direct customer interviews and focus groups are great, but you only can do so many of them. On-line surveys? Decision-makers responsible for large technology bets at Fortune 100 companies lack the time to respond to them.

Adam is correct. No virtual panel can, well, forecast purchase intent. But they should provide invaluable feedback on value propositions and messaging. And they are both patient and indulgent. You test many variations of a value proposition for example to see which one best resonates.

[6] We’ve heard people confidently predict that early tenure technology jobs will disappear. We’ve all fielded anxious questions from candidates who should probably believe less of what they read on social media. We have all tried to figure out what to say. As Adam points out learning is the most important thing at the start of your career. And it is easier to learn useful things than it has ever been before.

[7] Colonel John Boyd’s Observe-Orient-Decide-Act (OODA) loop is a model of how people and institutions make better decisions through recursive learning in competitive games. It provides a mechanism for balancing deliberate and emergent strategies, which emerge from the bottom up. It appears to have influenced the agile manifesto.

[8] A concave payoff curve: bounded upside, catastrophic downside. Law. Cybersecurity. Most operational IT. A convex payoff curve: bounded downside, extreme upside. Venture capital. Early-stage software equity. CIOs and CTOs often face concave payoff curves. Flawless uptime is invisible. A database corruption is not. When you put together a major incident response plan, you are trying to truncate the ugly end of a concave payoff curve.

Cloud infrastructure, SaaS applications, a revolution in cybersecurity and now AI have upended that unhappy equilibrium of oligopolies. Large enterprises less exclusively act as “takers” of offerings from giant technology providers. They integrate a wider array of technologies from a wider array of providers — some of whom are venture-backed attackers.

Acknowledging this, we did something new at May’s Technology Leadership Forum session. We convened a panel of venture capitalists to discuss and debate how they see the enterprise technology market. Ed Sim from Boldstart, Daniel Frankenstein from Joule Ventures and Will Summerlin from Autopilot engaged with the group on how enterprises can best work with VC-funded companies and how AI will change B2B software markets

Here’s what I wrote about it after the event:

Wow, there is some frustration out there! What do TLF members see from their incumbent software vendors? Slower innovation, degraded quality and more aggressive negotiation. They think some providers see their products as “falling knives” and have resolved to extract as much cash as they can from the portfolio before it declines into irrelevance. Others simply don’t get AI — they want extortionate rates for unimpressive capabilities that only reinforce silos between different parts of the environment.

TLF members found the discussion so valuable and VCs had so much fun that we decided to reconvene (after much scheduling gymnastics) for a Prosaic Times video discussion.

The conversation covers five things.

1. Whether vibe coding and agentic software engineering are the same activity — they aren’t, and the distinction matters for how you staff and govern AI work

2. Where the moat lives as the cost of producing code approaches zero — my answer involves latent information, and it’s more interesting than the conclusion.

3. Which sectors are most exposed to disruption — legal and accounting are the easy answers; I have a less obvious one.

4. How enterprise technology leaders should actually engage with the startup ecosystem — the VC-as-intermediary model turns out to be more consequential than most enterprises realize.

5. And whether you should leave a corporate job to found a company. Ed has a checklist for that last one.

Introductions

James Kaplan: Welcome to a special Prosaic Times podcast. Back in May, we did something different — we always get together about 50 or 60 enterprise CIOs, CTOs, and CISOs, and this time we invited a few of my favorite venture capitalists to share their perspective on the enterprise technology market.

All of our VCs had a blast and found the session both enjoyable and informative. All the members of the Technology Leadership Forum told us they appreciated the session and found perspectives they hadn’t heard before.

So I said, we have to get this group together again for a podcast — so let me start out with some brief introductions. Will, do you want to start?

Will Summerlin: My name is Will Summerlin. I’m a co-founder and general partner at Autopilot. We’re a growth stage venture capital firm, typically investing from Series B onward

Daniel Frankenstein: A co-founder and partner at a firm called Joule Ventures. Seed and pre-seed investor doing B2B software investments, coming out of the Israeli ecosystem.

Ed Sim: Boldstart Ventures. We partner at inception with technical founders when they’re in the idea maze — companies automating the autonomous enterprise, physical AI.

James: Daniel, when we were chatting before, you said that the interaction with the members of TLF was very eye-opening for you. Could speak to that a little bit

Daniel: One of the challenges we face in commercializing companies that aren’t from the United States is building that connection to corporate decision-makers and technology integrators. What we got from the TLF session was direct feedback on specific technologies — candid views on where enterprises are struggling to integrate AI solutions, and where their long-standing incumbents are trying to introduce AI but not yet meeting the mark.

It was a fascinating conversation, and it clarified where the real opportunities are for new technology businesses to carve out space.

James: We’ll say incumbent rather than legacy.

Ed, what was your takeaway? What did you learn from the day?

Ed: Everyone is truly, truly interested in deploying agents, and two, it’s really, really early, and I think that everything we think and know today may be irrelevant tomorrow.

People are talking about RAG and all these other things — like, hey, that was gone yesterday. And in fact, everything that we talked about almost since the last TLF, what was that? Six weeks ago? Is almost irrelevant again. The speed at which things are moving is staggering.

Will: We’re extraordinarily early on the adoption curve in terms of enterprise AI adoption. And I think it’s interesting in that there seems to be a disconnect from the myopic Silicon Valley view and the reality of enterprises. If you talk to a lot of founders or early stage investors or even growth stage investors in Silicon Valley, there’s this view that adoption’s happening very, very quickly, and we’re farther along on that adoption curve.

James: But what we heard from members of the forum is that we’re still very early. It’s still very much in the experimentation phase, moving to the broad-based adoption phase. And so I think that presents an incredible amount of opportunity ahead, but I also think it reflects this disconnect with the Silicon Valley view. You hear a lot in the ecosystem around token costs, and news stories about companies struggling with token costs. But the enterprises I know are spending trivially on tokens in the context of, say, a USD 1B or USD 2B or USD 500MM IT budget.

And that is a measure of quite how early we are from an adoption standpoint.

Ed: Some of the companies that were the more advanced ones were the ones beholden to one large language model provider.

The events of the last six weeks — particularly some where providers said, “Hey, we’re gonna interfere in some of these things” — have shown that not only do token costs matter, but also which model you’re beholden to matters.

So no one’s gonna want to be beholden to one model. Multi-model architectures have to be built, I think, because of that.

James: It reminds me of the era when some people believed you should standardize on a single provider.

Will: I think another interesting data point related to token costs is compute resources. We’re already in an environment where compute is radically supply constrained, and if you look at the growth rates of demand for compute versus supply for compute, demand is growing at a much faster rate than supply possibly could. [2]

Demand is growing because we see more broad-based adoption. It’s growing because agentic models use a lot more compute than legacy LLMs from a year or two ago. And on the supply side, we’re hitting all these constraints and bottlenecks related to supply chain in the semi space.

We’re entering a world where that gap is gonna continue to grow. And that’s all in this backdrop of what you said a minute ago, which is, the single digit millions of spend. And so I think we’re in this interesting environment where part of the world, is super early on that adoption curve, but at the same time we’re already hitting these constraints around compute.

If you want any semblance of an SLA from one of these model providers, you’re going to have to start committing to more meaningful spend. There’s this conflict and tension in betting your entire AI strategy on a single model provider.

James: The enterprise is maybe the shoe that hasn’t dropped yet in terms of token demand, and my God, what happens once the, spenders start using, scale?

The flip side is that you’ll probably see a lot more efficiency in token consumption. You don’t need to do all your non-deterministic processing at runtime. More of it shifts to build time — software engineering, data ingestion — with deterministic execution at runtime. And we may see a meaningful share of inference move to edge hardware entirely.

Ed: Think about models like OpenRouter. Companies now are about to sell a box — or a virtual box — that has a router for any frontier model on top, and then adds your own open source model on the box itself with H200s bundled in.

The idea is to make it very, very easy for people to build an agentic workflow, deploy it, and then actually maximize... I think we’re gonna move to a world of ROI per outcome. It’s no longer gonna be, are you automating me? But what is my ROI per outcome?

How does that look? And then there’s lots of other things involved in measuring success.

Daniel: Obviously folks have concerns about costs and limiting those costs. But right now, those costs are the actual real costs businesses face when they start using this at scale.

So the demand curve is absolutely outstripping the supply curve when it comes to compute. Enterprise adoption is going to be more of an increase on a dimmer light switch.

James: Enterprise adoption is always an increase on a dimmer — not a flip of a switch — which is why claims about AI impact on the bottom line feel like an attempt to be cute.

How could you expect otherwise? We’re what? Maybe a couple of years into the use of gen AI. How could you possibly expect that to have impacted enterprise operating income statements at this point?

The only impact on the income statement has been for some of those tech companies who are spending. I’m thinking about banks and pharma companies at the moment.

Ed: As far as the ROI on that spend — questionable.

Will: I do think we have small case studies where we’re starting to see it.

As one indicator: we have a portfolio company called Replit, which is an AI-native software creation platform. They allow anybody to create software using natural language. And what we’ve seen is in some cases the replacement of incumbent software providers where a non-technical user is able to create their own custom CRM system for a USD 100,000-a-year software license. The ROI on that is extraordinary. That’s like nothing you’ve seen from traditional software.

Ed: I still think the super complex stuff — Workdays — when people say, “Hey, we can vibe code that stuff away,” I think people have moved on from that really.

Whether or not those companies are gonna be massively successful, who knows? But I do think the whole idea of vibe coding things, the maintenance becomes absolutely incredible for some of these people as well.

Will: On one end, you have the bottom of the market that has simple use cases where they can vibe code their own CRM and displace incumbents. So I think that’s one end of the market.

The other end of the market in regards to enterprise adoption is just building more — where now you can use tools like Replit to build agents on top. And so it’s not that you’re gonna get rid of your incumbent provider, it’s just that you’re gonna build more agents on top of what you already have.

We see this primarily around internal tools, where to build a custom internal tool for your sales operations team or for your legal team, you had to either go work with a third-party software development agency to build that, and it would cost you hundreds of thousands or millions of dollars, or you had your internal software engineering resources, which would be expensive and you’re competing for internal resources.

Now, a non-technical person could go into a tool like Replit and build those agents or build those custom pieces of software on top of the existing architecture that you already have. And so I don’t think it’s a replacement for the incumbents in this case. I think rather it’s just gonna enable more software to be developed on top.

Daniel: When we got together initially, there was a real fear-of-replacement moment for incumbents.

We’re more in an all-of-the-above moment now, where a lot of incumbents have the customers, they have the data they own — which, by the way, is one of the key things to be competitive in this moment.

And that’s actually what makes this really exciting from a venture perspective: these incumbents are your customers, and also your acquirers.

You also have this entire new world to go after as well.

Ed: The most impressive thing was the idea that unicorn SaaS companies are all dead. This founder went all in on AI three months after ChatGPT came out, and today Salesforce announced it was buying it for — I don’t know — three and a half billion or something, which is above their ZIRP-era financing round. [1]

It takes very special companies, but maybe there’s a five percent chance that some of the very forward-thinking ones can create that kind of value. And combined with some of what Will said about Replit, there’ll be net new — more things banging on the system.

Vibe coding vs. agentic software engineering

• Vibe coding and agentic software engineering are distinct activities: one is a non-technical user building something simple; the other is a professional engineer with 2–10x leverage.

• The non-technical user persona is larger than the professional developer pool — and that’s where the near-term adoption surge is coming from.

• In enterprise, the term “vibe coding” doesn’t travel well; the equivalent is building skills and automations on top of existing platforms.

• The abstraction layer keeps rising — but semantics still matters. Syntax can be generated; data models and business logic still require human understanding.

James: What’s the difference between vibe coding and agentic software engineering? When I say vibe coding, I mean someone who’s not particularly sophisticated building something — maybe it’s a reporting tool or analysis. Versus a software engineer using a set of 2x, 5x, 10x tools.

Will: There’s a completely different user persona between the professional software developer that now has these agentic tools and the non-technical user who’s now empowered to create software. We’re seeing the adoption of the non-technical user — people in sales operations, people in legal, people in finance who have never written a single line of code, who don’t understand any JavaScript syntax — who are now empowered to create their own software in a way that is easy, simple, secure, compliant.

That’s where we’re really starting to see this adoption. The other important nuance here is that the pool of users who are non-technical is larger than the pool of users who are professional software developers. And so if you look at even a technology company, you might have fifty, sixty, seventy percent of the employees falling in a category that’s non-technical, and they are now empowered to go create software.

James: I wonder if we’re gonna see the rise of strats. In banking we have the strats who were either former traders who learned how to code, or former coders who learned about markets — very fuzzy. I was wondering if that might happen in more sectors and more domains.

Daniel: I agree that non-technical users are increasingly empowered to vibe code solutions. The problem is the influx of people who are vibe coding something, white-labeling it, and pitching it as a venture-fundable startup — and it’s not. Across sales enablement, marketing, the CFO stack, I can tell within five minutes of a pitch that there’s no defensible moat, nothing that takes particular expertise or meaningful compute to maintain. Buyer beware.

To use your strats analogy: to be a great trader takes years of domain expertise and longevity in the space. The same applies here. The people who can build something special are the ones with that depth. A vibe-coded sales stack with no moat is not that.

James: what do you think of non-technical founders?

Ed: I agree that more people building things is continuing to get more abstracted. I think the word “vibe code” and enterprise just does not really mix together — I don’t think you’re gonna walk into a CIO and say, “Let’s vibe code stuff.”

But it’s the same thing. It’s the same thing as what Will said — skills. The enterprises we talk to are using skills platforms, where you can type something into your AI of choice and say, “Hey, I want to build this, add this in, and then create a skill or automation.”

That is happening, and there’s gonna be hundreds of thousands of skills out there, and you’re gonna need to manage that and package it like software. You’re gonna need a version control system where there’s a skills repo for company XYZ, where these are the approved skills, and these are security vulnerabilities that can pop up, so let’s scan those skills.

What happens when Daniel changes something that Ed made? Is that better or worse? So how do you do an eval around that? So I, I think, I think we’re gonna be moving towards a world where skills become software,

James: If you remember the late ‘80s, we went from assembler to C — people said software engineers would go away. The engineering continued.

And I wonder if two things are true here. Abstraction matters, but you still need to understand the data model, even if you’re programming in a very abstracted way.

Ed: I think what you’re talking about is that you need specialized humans. Back to your earlier point — everyone wants to get automated, but they don’t know where to start.

Those specialists can help people understand what their workflows are so they can automate them. So you need specialized skill for that. And then two, you need specialized skill for evaluating whether a workflow is actually doing what it’s supposed to do. And then what happens, for example, when a model changes — is it actually gonna do better or worse?

People are gonna be needed, and even more specialized to understand those things. And I think people are finally understanding that right now.

James: Sometimes I draw a distinction between syntax and semantics, and in an AI software engineering world, we can worry a lot less about syntax, but semantics still matters. So Will, let me ask you, are we being overly negative on vibe coding? Give us the other end of the case here.

Will: First off, I know of several examples, including one where a company, built entirely on Replit is doing well over 100 million ARR, mostly enterprise revenue.

And so you, you can build real businesses, and I think we’ve gotten to a point in the technical maturity of the platforms like Replit where you have, implicit security, you can host on a private cloud instance. You hit all the enterprise check boxes, and a lot of the code that’s now hosted on tools like Replit specifically is as good as what you would get from a junior engineer. And so I disagree with the premise that you can’t build a venture-backable company on platforms like Replit.

Ed: I’m getting my popcorn. This is fun

Will: I agree that we as investors should operate with this premise that the cost to develop software is approaching zero over the next three, five, seven, ten years — just like the premium for having more software engineers is eroding. As that skill barrier is eliminated and the cost to build approaches zero.

Daniel: Of course there’s going to be vibe coded businesses built on companies like Replit that are gonna be absolutely venture fundable and extraordinarily successful.

The barrier to entry to put a product in the market has gone to zero, which means there’s so much stuff that isn’t defensible and isn’t venture-backable. It’s a huge increase in the denominator without a proportional increase in the numerator.

James: The percentage of effort associated with the core logic, relative to the effort required to fit into a broader enterprise environment, is decreasing. The pain is now in the integration: connecting to identity and access management, compliance, the whole stack.

So Ed, how do you think about this challenge of integration? Because in many places that will be the long pole in the tent.

Ed: The last mile in the enterprise is the longest, and it’s probably getting longer. It’s not easy for a large bank or a healthcare firm or a large CIO to bet their reputation on deploying your agentic technology.

There’s a lot of stuff you’ve gotta pass — all the SOC compliance and everything else. What you’re seeing is the idea of these forward-deployed engineers showing up and just getting it to work. Part of that is a response to how hard the last mile is.

The model is: build product, then finish the last mile with people. And the learnings you get from it are invaluable. The key question is whether the most important data to own is not just the data you have to run the AI on top of, but the private evaluation of whether the outcome looked really good — because that is really the knowledge of your business. That’s what separates a P&G from another marketing company.

Are these forward-deployed engineers going in there and helping you get those outcomes? And who owns the data? Are they giving it to you? Or are the forward-deployed engineers from the model providers themselves taking that data and using it to build vertical solutions?

That’s going to be the big battle. It’s what Satya calls hill climbing — reinforcement learning feedback. The question is: if you give that to a model provider and they’re helping you improve, is that the right trade-off to make?

I think we’re moving toward a world where people want to control their own evals — which means infrastructure on the edge, private models. We need a lot better open source coming down the line. It’s still not anywhere close to what the labs are providing, but that’s where we’re gonna move towards.

Where the moat lives when code is free

• If code approaches zero cost, the competitive advantage shifts to ontology, data models, and latent institutional knowledge — the things that can’t be scraped.

• The right model for the right task matters more than any single model: orchestration layer bets are more durable than frontier-model bets.

• Enterprises that hand their outcome data to model providers for convenience may be signing their own death certificate — private evals are the new proprietary asset.

• Architecture commoditization is coming but isn’t here yet; scaling laws are still working and a step-change in model capability is 12–18 months out.

James: sometimes I wonder if code becomes free or close to free, but the ontology and the data model becomes the source of competition.

Will: It’s very possible. This goes back to a point that we were talking about earlier, you probably don’t want to bet on a single model. It’s not just about cost, it’s also about, performance in different domains. And so I think that- that’s benefit of betting on something more at the orchestration layer, where, if you’re trying to do a task, we’re using software engineering as the example here, you might be writing, some more complex code, you might need a frontier model from one of the labs.

You’re writing simple front-end code, you could use an open source model. You’re doing a design task, maybe, Gemini or another model, from a, a different provider is better at that specific design task. So I think you’re seeing mixture of experts, to, solve these

As software code itself becomes a commodity, what matters is matching the right model to the right task — and everything else that isn’t captured by coding benchmarks: platform depth, integrations with Databricks or Snowflake, security scans, the full enterprise stack.

Daniel: think having specialized humans that really have domain expertise combined with owning your data and your moat, somewhat model agnostic and having that’s what the future it’s a mixture of really great humans, different models, and, owning your data. I, companies that choose to allow their outcomes and their data to be out there for the LLMs to scrape, because they’re, they’re prioritizing speed, ultimately are, are signing their own death certificate.

James: And does architecture get commoditized?

Ed: Not anytime soon. So the answer is not today, but I wouldn’t be surprised, like in 18 months if we kinda get there, because this stuff is changing so quickly.

Will: I don’t know, but if I were to place a bet, yes. Just look at the improvement in performance over the last three years. It doesn’t appear that we’re plateauing — in certain skills or certain domains it’s actually accelerating. It’s really hard to extrapolate out the next two, three, five years. But I think if the curve continues to look like it has the last three years, it’s very possible that architecture gets commoditized.

Ed: In other words, scaling laws are actually working, and if you have the compute, it’s improving. Everyone I’ve talked to on the models they’re training now says it’s still moving in the same direction. Hasn’t been proven false yet.

James: I’ll answer my own question and say my question was invalid. Any model requires context, so it depends on explicit information rather than latent information. Models will only get good at architecture to the extent that people using them figure out how to extract or capture the latent information they need.

Ed: How much will they be willing to hand over of how their system is architected and actually operates to a model provider? That’s the question.

James: Do they even understand enough to be able to write it down in such a way that it can be consumed? Taking latent information and turning it into explicit information is not a trivially easy task.

Ed: It’s the last place that hasn’t been automated yet. If you’re thinking about most of the automations — it’s on the edges. IT ops themselves have not really been automated. Partly because people are afraid of AI just shutting things down. But I do believe we’re gonna move toward a world where there’s more trust with AI around automation of IT.

James: Let me ask a related question, maybe one that’ll be controversial. So I, I, I got myself told I was up a tree because I offered the thought that we’re gonna see more strats in the future, more, convergence the line between, business optimization and technology execution.

But then I offered the hypothesis that it may be easier to teach software engineers about the business domain than it will be to teach people in the business domain enough about data models, for example, to, sufficiently engage. I don’t know whether you call it vibe coding or agentic software engineering.

Daniel: I spend almost all of my train technical people on the business side of things. so, so I probably over-allocate to frustration in that department. So, maybe that skews my answer, that maybe it’s a little easier to teach business people, the software side, but I could go either way.

James: Ed, what do you think?

Is it easier to teach a bus- a business analyst software engineering, or easier to teach software engineers about how to think about a business problem?

Ed: I think the underlying models can be so good that what matters is the expert who knows what to prompt and ask. If you actually look at how to use LLMs today, it’s all about how you prompt — two of us could sit next to the same thing and get vastly different outputs if we’re trying to achieve the same goal.

Will: I completely agree with Ed. There are so many nuances to different problem sets or workflows. The people who intimately understand them will have access to tools that can manage all the technical complexity.

One case study from us: we vibe coded our own custom LP portal, and my co-founder, who’s not technical but who’s been running all of our operations, built this in probably 12 hours. And it’s better than anything we could find from third parties. Daniel and Ed probably lament the challenges of fund admin more than anyone else.

James: Nothing’s more fun than fund administration

Will: it’s the great joy of life. there are so many little nuances to it that I think y- you just wouldn’t understand unless you’ve spent months or years in the trenches

Ed: by the way, that goes back to my other point, is that your person that built it and you are the experts, and , you’re the ones that are gonna actually evaluate it as well. You’re the only ones that can keep evaluating everything else, right? So that’s kind of your workflow, that’s your model.

And, and your same model could be deployed at Daniel’s place and my place, but how we evaluate it and what successes might be different, right? And that’s kind of what I mean by the institutional knowledge and workflow that, that I think those private evals is so valuable to capture that

Daniel: One of your great moats is domain expertise. domain expertise can be found on the business side, and it can also be found on the technical side. It depends on what problem you’re solving and who’s your constituency.

James: I had a discussion with a bunch of 22-year-olds a couple months ago, and they were, you know, having the typical 22-year-old, ”Oh my God, any job I might consider is going to be replaced by, by AI.“ They asked what I thought would be a good thing to learn, and I said data modeling.

I don’t know whether that’s business domain or technology domain expertise, you both need to understand all the nuances of the business process, but there’s that, process of abstraction, that I think becomes incredibly important, which people who are only deep in the business domain don’t always get.

Which sectors face the most disruption

• Legal and accounting are the obvious targets — high information volume, largely objective outputs, limited customization requirements.

• James’s less obvious answer: B2B interactions broadly. They haven’t changed in 15 years while B2C was transformed. Gen AI’s ability to ingest unstructured data makes this the next frontier.

• Ed invokes Jevons: freed capacity can be absorbed by new demand. Companies that cut headcount fastest are already adding back — different kinds of people, more senior engineers, specialists.

• Robotics is the adjacent disruption no one is talking about enough. Ed’s portfolio company has 500,000 hours of manipulation data and is deployed at four large manufacturers — the ChatGPT moment for physical AI is close.

Okay, Daniel, predictions about which domain, technology domains do you think will be disrupted the most?

Daniel: I’d say I’m still a big believer in places where there are largely objective answers to questions that require sifting through a lot of information. I think legal and accounting are two of the low-hanging fruits.

Ed: I think, James, that everything is up for absolutely massive disruption right now. But I’m also a believer in Jevons paradox — because if things are cheaper and you can use AI more, you’re gonna be able to do more. And I do believe in a world of more productivity. The tech companies I’ve seen that have automated the fastest may have cut headcount and used AI as an excuse, but now they’re adding. They’re just adding different kinds of headcount — more senior engineers, different kinds of people. Perhaps the specialists you’re talking about.

So I think it’s gonna disrupt everything short term. The next two or three years will take a while to absorb the hit. But I believe in the longer term that we will actually be more productive as a society and generate more output with less cost. And hopefully you see those GDP growth numbers increasing.

James: Will, what do you think?

Will: I completely agree. If you look at any technology cycle throughout history, for the most part it’s led to wage increases, job creation, new goods — and I think the same thing will happen here over a ten to fifty to a hundred year time horizon. What we see in the next five years: the most disruption today is in two categories. Where incumbents have low-quality software products and have not innovated, you can now build that version yourself — and often it’s better, cheaper, and higher quality.

That’s our case with this fund: we built it ourselves because the incumbent alternatives were not high quality, they were very expensive, and there was a lot of low-hanging fruit.

I think the other area where we’re starting to see more disruption is in domains where software is highly customized — back to the point both Ed and Daniel made, which is that it’s really important to have deep domain knowledge specific to your workflows.

The way we build an LP portal might be different than the way Ed wants to build one, because he has nuances in his business and portfolio that we don’t. Where you have a high level of customization required to make software work for your business, there’s a lot of opportunity for disruption — both disrupting third-party software and disrupting the incumbents who have mediocre products that work okay for everyone but not great for anyone.

James: it’s what I call the one size fits none challenge.

Ed, you said in the session that you looked to fund companies that were problem-obsessed rather than solution-obsessed because, you, you iterate through different solutions, but problems are eternal.

Messiest part of, the advanced economies is B2B interactions, and that’s not just CRM, but the broad scope of all the large, complicated businesses. , Over the past fifteen years, maybe twenty years, we’ve seen an utter transformation in B2C interactions.

But many B2B interactions look a hell of a lot like they did ten or fifteen years ago, and I think that, the, the ability of, gen AI to, to ingest complicated, unstructured data to transform those domains.

Ed: I, I love that. And by the way, can I add one thing

James: Please

Ed: we didn’t talk about software and robotics,

I have a company. We’ve built our own LLM for robotics, with our own data, and we’ve proved that scaling laws actually work in robotics as well.

So I feel like we’re almost at that ChatGPT moment in robotics where a generalized robotic software model will be able to do a lot. So in other words, once upon a time where you would have to have 100 different robots or 100 different kind of brains operating and, and doing 100 different tasks, you can now have one.

And I can tell you this, that the market is moving so fast in that space, so manufacturing, pick, pack, and ship, auto manufacturing, all of those kinds of things are getting transformed. In the next three to five years, you’re gonna see a massive, massive dislocation in those markets as well

James: Is that an LLM for robotics or is that a type of world model, right? I mean, are we moving from language to physics when we talk about that?

Ed: there’s three different kinds. So the one that we have is a basically a foundational model for robotics. Other people are trying the world model approach, and what my company is doing is they’re getting their own data, right? There’s no Reddit for data.

They , ship these 3D-printed devices that goes on people’s hands that are very, cheap. we take video of people manipulating these things as if they had these grippers, and we’ve got 500,000 hours of data.

And now we’re deployed at four very large companies doing very complex manufacturing tasks all within two years. This is massively moving so fast. [3]

It’s basically building off the shoulders of giants? Because it’s already been done before, and now you’re applying it to a different realm,

James: Daniel, how bullish are you breaking the, breaking the barriers between the physical worlds when we come to,

Daniel: How can you not be bullish on it, right?

James: What’s the pace, do you think?

Daniel: One of the common themes we’ve discussed throughout this conversation — which, again, was part of the reason why I so enjoyed our in-person discussion — is the pace of innovation. It’s at a lightning speed.

Literally every week things are changing. Every day things are changing. That’s moving at a breakneck pace. But the integration of those changes — the integration of products that encapsulate some of this innovation into large businesses — is very slow.

And so I, we’re, we’re, we’re it’s, it’s one of those where you’re like in an F1 race where innovation is just lapping, the jogger, which is the, the, the enterprise integrator.

And, and so the question is how do you lean out the window and throw something that the jogger can actually digest? That’s, where the rubber hits the road, not to take the, this too far. But, that’s, where the bottleneck is.

Will: Agree with, with, everything said here and that there’s an incredible amount of opportunity for AI in the physical world. you look at some domains in manufacturing where AI is now used to inspect batteries coming off an assembly line, and it can do it, ten times faster and, fifty times better than a human inspector can.

And that’s a job that humans don’t like doing. You wouldn’t want to sit under bright, bright fluorescent lights all day and look at batteries. So I think there are incredible amount of opportunities we’re already seeing implementation, the ROI is there. But I will say, I think that humanoids broadly is probably in bubble territory.

And I say that for two reasons. I think number one, we’re very early on the S-curve and that, the, the technology is not quite there. when you start to do things in the physical world and chain together different skills and make that work successfully in an enterprise environment, the, the, the success rate has to be very high in each skill.

You know, if your success rate is ninety-eight percent in one skill, ninety-eight percent in another, ninety-eight percent a third, you’re gonna fail a lot of the time because you multiply the probabilities together.

You really need to get to like ninety-nine point nine nine percent accuracy in each skill set or each domain in order for this to work reliably over time. The second problem I think we have in the humanoid space is people underestimate how hard it’s gonna be to scale up manufacturing.

Like you go talk to the team that, that scaled up manufacturing for the Model 3 at Tesla, like it’s a really hard thing to do to go from, you know, zero to scaled production of, of complex hardware products.

And I think the sort of view that we’re gonna be shipping millions of these in two years or three years is, is probably a bit shortsighted.

Ed: One comment though. I, I agree with you on humanoids, but my company’s actually going after, robotic arms, So when you control the variables with just this and a gripper, you don’t need five joints on it to do things. that actually allows you to deploy much faster.

On the humanoid side, there’s so many more issues. Battery life is another one. Safety, what happens when the battery runs out? Does it fall on somebody? there’s so many issues there to, to your point.

The, more you constrain all the, variables and focus the processing power on the task at hand, the better off, you are in terms of deploying these things with real accuracy.

How enterprises should engage the startup ecosystem

• Routing startups through innovation teams is a trap — those teams are typically disconnected from the business owners who actually have the problem.

• VCs are intermediaries in two directions: LP → founder, and founder → enterprise. The second role is underappreciated. They filter signal from noise, translate languages, and set expectations on both sides.

• Enterprises build reputations too. Kick the tires on three companies and do nothing and VCs stop bringing deals.

• The right frame for CIOs: decide where you want to sit on the adoption curve, then match the VC tier accordingly — seed investors for early bets, growth investors for proven businesses.

James: Let me pivot a little bit. I’d love to discuss a bit how enterprises, CIOs, CTOs, CSOs, others, should interact, with venture funds and venture-backed companies.

So Daniel, let me start with you. Advice would you have, for, technology executives or IT executives at big traditional enterprises as they engage with venture-backed companies and potentially as they engage

Daniel: Absolutely. First and foremost, large businesses are structurally challenged when it comes to engaging with younger, earlier stage businesses. Some organizations have tried to address that through innovation departments or groups built to test technology. We tell our portfolio companies to avoid those like the plague, because they are typically disconnected from the business owners themselves — the folks who actually need solutions to solve problems. We as a fund try to serve as a bit of a filter there, because it’s really hard as a startup to knock on the door of a Fortune 1000 business and get any attention. Our hope is that when we help knock on those early doors with our portfolio companies, that comes with a vetting process, and an understanding that we’re gonna ask for that time only when there’s good ROI going both directions. We tell our portfolio companies to go directly to the person who has the problem — try to engage directly with the business unit.

James: Will, what are the mistakes that large companies make when they interact with venture-backed, companies?

Will: Trying to push companies through an innovation team that doesn’t really own the problem is often ineffective and wastes everyone’s time.

As an enterprise customer, you have to decide where you want to sit on the adoption curve. If you’re willing to be an early adopter and really partner with a company, I think that can be incredibly fruitful early in their life cycle — you’re gonna get a lot more attention and probably more customization specific to you.

If you’re not willing to put in that effort, if you’re not willing to view this as a partnership, you’re probably better off waiting until, other early adopters have embraced the startup and it’s gotten to this inflection point where it’s now a more mature company, with a fully baked product.

And then third, you know, a lot of, a lot of larger companies now have venture programs where they’re investing in startups, especially investing in startups that they’re customers of.

If you’re gonna have a venture program, connect that directly to the business unit that’s actually buying the software and, and understand why you’re making venture investments. Don’t just do it because you think it’s cool

James: Okay, Ed, why is it so tough for, for large enterprises to engage with venture-backed companies?

Ed: Filtering. There are just thousands and thousands of startups, and our job is to filter through all those and invest in the ones that make sense. When you partner with venture firms, you should partner with all different kinds — because there are folks that go super early, at inception. There are folks that go a click later. There are folks that invest after the company has USD 30MM of revenue.

You need to understand where you are on the innovation curve and figure out which venture capitalists you’re willing to engage with to see what they have.

And we as VCs have to be very careful as well. We’re not gonna throw something at an enterprise knowing full well that it’s gonna be an 18-month sales process — because that could also kill the startup, but also we don’t want to waste your time.

It’s all about trust and reputation. On both sides, you have to figure out what you’re willing to show an enterprise to say that it’s ready for deployment. Maybe the company and team have actually built something new, and they’ve done it 30 times before — so therefore there’s trust.

And on the enterprise side too, they build reputations as well. Because if you kick the tires for 18 months and do nothing with three companies, VCs will say, “Don’t ever go to them again.” And therefore that enterprise might miss out on innovation.

It’s a two-way street where you actually have to trust each other.

James: How do you think about the filter? How do you think about who to speak to versus not to speak to?

Ed: Well, it just comes down to people though, right? It’s all about the people, right? You want people that move very fast that actually can solve a problem today, but also presents a kind of vision to solve your problem 18 months from now.

And that’s what we look for, and hopefully you come to us and we can present the founders that actually do the same

Will: The bar for speaking to somebody is very low. I’ll try to speak to as many people as possible. But for us, us to actually make an investment, we’re, we’re investing at a different stage, and so typically the companies that we’re investing in already have tens or even hundreds of millions in revenue. It’s a real business at that point, and the question is not will this succeed? Do they have product market fit?

The question is, what is the sort of revenue scale gonna look like five years from now? And what are the unit economics gonna look like five years from now?

We’re asking a fundamentally different question , than Ed or Daniel are when they’re making investments. We have a set of criteria that we filter to. We look for technologies that are inevitable, . There’s adoption, but we’re early on the S-curve. We look for markets that are default dominant, where an outcome is gonna be an oligopoly or a monopoly.

To Daniel’s point earlier, we don’t like markets that are commoditized, where you have hundreds of companies without any real moats. And then finally, we look for what we call dynastic DNA. It’s really about the people. Is this founder pursuing their life’s mission? Do they have the wherewithal and the fortitude to push through over the next ten, fifteen, twenty years to build a large, enduring business? Can they attract amazing people around them? Do they move at a high velocity?

James: So it’s your job, Will, to, speak to people, but if you’re a CIO or a CTO, it’s like, well, it’s my job to keep the business up and running, and I got, some number of hours per week to go speak to people, and an infinite amount of demand to be spoken to. So what’s your advice for someone who in an enterprise, role

Daniel: very nature of the venture business and the very nature of us having a funnel that ultimately results in a set of investments means when we are, making intros and trying to have conversations with corporate decision-makers,

if you ask for a busy person’s time and you do not use it effectively, you will not get it again. it is my, desire to be able to go back to the same folks time and time again when there’s an opportunity that’s relevant.

And so by the time I’m asking for the time of an enterprise CIO or an enterprise CISO, I’ve done my homework. I have done the work to know that even if it doesn’t result in an investment for us, it’s going to be a good use of both of their times.

Ed: It’s always not us pitching them. I find the most value is when I talk to CIOs or CSOs and say, ”Hey, what’s top of mind for you?“ Maybe once a quarter I have, like, a dozen folks I’ll catch up. ”What’s top of mind for you? What are the biggest problems you’re seeing? What are the existing, incumbent vendors not providing for you? What do you wish you had? What are you thinking about building?“ Right?

Because a lot of the things that startups end up creating are things that companies want to build themselves, right? So I think that’s part one. So once we understand that, we can say, ”Hey, I’ve got this one or that one.

We funded a company, um, you know, like a year and a half ago in the agent identity space. I, I walked into a very large bank and we just said, “Hey, look, I would love your feedback.” And, and they had eight people in the room. We spent an hour together. We, we mapped out what our architecture would look like, what problems we’d solve.

And then guess what? We went away, built stuff, raised rounds of funding, came back a year later — not only did we show that company that we had built everything, but we also showed them what the forward roadmap was.

And I remember as I was leaving the room, three of them were huddling in the corner and saying, “Holy shit, did you see what they built in the last, you know, year?” We built it really fast, and then we had a future vision that kind of aligned with theirs. Another thing: we’re not expecting a sale. We want feedback. Why wouldn’t you provide feedback and spend the time? And then if we come back and make the mark, great.

James: You assume that VC funds serve as an intermediary between limited partners and founders. They channel investment to companies. But what I’m hearing here, and have heard in previous discussions, is you’re just as much an intermediary between startup companies and the enterprise — helping founders understand the enterprise, and helping the enterprise understand how to engage with startups.

Is that, is that a fair way of thinking about it?

Daniel: Absolutely. I have two track records. I have the track record to my LPs, and I have the track record to the entrepreneurs we back. And the track record that is for the entrepreneurs we back is: how many customers did I help you get?

How many versions of your product did I help you think about? how many key hires did I help you with? those are all critical things that are on the company support side of things. and it really involves being an intermediary with the enterprise, and understanding how to translate different languages.

A startup speaks a different language than a large enterprise. You gotta have somebody in the middle ideally making that translation

Ed: Look, I’ll toot our own horn here for the three of us — I’ve been doing this for 30 years, and not every venture firm does this. So if you’re a founder building in this space, come to guys like us who actually do this every single day, because we learn a lot during the process.

But all I’d like to say is our job is to help it, help you make it an unfair fight against your competitors so that, so that when you get funding from us, that we’ll , accelerate your path to your next round of funding and kind of product market fit or post-product market fit, depending on the stage.

One or two influential customers at the beginning of a company’s journey, like, of their first five customers, can actually reap tremendous dividends, create so much value, multipliers of value based on the contract size.

By the way, the contract size does not have to be massive. perhaps, James, that enterprise gets a massive discount in year one. The point is those referenceable customers are so massive for the success of the business because what you really want that CIO or CTO doing is talking to the next four or five large customers coming down the pike.

James: Will you invest in s- companies a little bit further on in the journey? How is your experience or your model similar or different?

Will: For a company that’s just starting out, having an Ed or a Daniel on your cap table is really important. It’s almost like they’re an extension of your founding team, to the extent that they can broker these relationships and help you navigate the enterprise. For us, the set of problems that a founder is facing is different.

They have often overwhelming market pull. The problem is not finding customers, the problem is scaling up the organization to meet the demands that you have. And so we try to help out a different way. It’s always great to bring customers. We try to do that where it makes sense. But I think often it’s more about talent.

It’s bringing in people who can scale, help you scale from 100 million in revenue to a billion in revenue. It’s more sort of strategic finance, and so how do you think about bringing in some of the larger crossover funds that will be shareholders, through an IPO and into the public markets?

It’s thinking about the next act. You have one product line that’s extraordinarily successful. You know, how do you start introducing a second product line in parallel, your second act? And so often we’re helping founders, navigate a different set of challenges than, than Daniel or Ed are.

Should you leave to found a company?

• The most compelling founding stories come from insiders who experienced a gap, built a workaround, and realized others had the same problem. Coralogix is the archetype.

• Ed’s checklist: Is this a market of one, or a market of many? Have you talked to peers at other companies? Who owns the technology? Who are you bringing with you?

• Domain expertise is necessary but not sufficient. The ability to abstract — to see that what solved your company’s problem could solve a category — is what separates a founder from someone who built a useful internal tool.

• Will’s criteria at the growth stage: problem obsession, grit, and “dynastic DNA” — the willingness to push past a billion-dollar exit toward a hundred-billion-dollar company.

James: So not infrequently, I have people working in enterprise technology organizations come to me and say, “Hey, I’ve done something really cool. I think maybe I should build a company out of this.” Maybe the grass is greener outside. Daniel, how do, how should someone think about that,

Daniel: Some of our most successful investments have come from that exact process where somebody is, at a business, they are experiencing a gap or a problem, and given their knowledge of the problem, their knowledge of the domain, decide, “Hey, I’m actually going to leave the company, and I’m going to build the solution I was looking for,.”

And, one of the, a, a really exciting company where we were involved from day one in the observability space called Coralogix is literally exactly that story, where , the founding team there was literally a product team at another business that built a solution they couldn’t find to buy. That’s a , very compelling, thesis. the challenge is a lot of times that thesis isn’t always right. The best thing to do is talk to guys like us, right? , Have a conversation while you’re still, ideating. Ed gets involved a little earlier than we do, we regularly meet with and advise founders that, aren’t yet founders.

They haven’t yet left their job. They don’t exactly know what they’re doing, but they’re really smart. They know their domain. They want to stay in the domain, and we want to stay close to them.so you gotta test the thesis, but you gotta test it with a toe in the water, not, jumping in the whole way, y-before that.

I, I, I always, when, when founders come to me and they say that they’re onto something big, I always say, “Show me your work. what validation did you do to, to achieve that conviction?” and to the degree we can be helpful in building that case with the founder that leads into an investment, great. but you gotta do the work to validate.

James: Ed, what do you think? And in particular, how should someone, what questions should someone ask themselves or him or herself,

Ed: There are a few lenses. You should have conversations with folks who do this. two is I think you should think about, is my solution a market of one, a market of a few, or a market of a many, ? Because I, I’ve talked to lots of folks building stuff, and they think they’re on their aha moment, but really their organizat- organization is so special that, I, I think it’s a snowflake, right?.

James: Abstraction is important

Ed: Do your homework, right? Also understand what’s in the market, ‘cause a lot of times also when you’re internal, you don’t understand what the other startup vendors are doing or what the other large incumbents are doing.

Like, why? Why is this built? Why’d you have to build this internally, doing a startup is really effing hard. You have to be absolutely stupid, insane, and crazy to go do a startup right now, and so I’m just warning you right now, it’s not all glory. It’s actually mostly 99% pain and 1% glory, right?

And, and so, so you better be pretty, pretty sure about that when you go out. And so let’s just have this conversation. A few things I’ll ask, will be understanding how’d you come to the conclusion that there’s just more than you. Have you talked to other people at other companies, your peers, for example, at other companies that might want to buy this solution? Two is who owns the technology? Are you gonna actually go leave and rebuild it yourself, or are you gonna spin this out of a larger org, and then perhaps maybe give them some equity, which also opens this a whole, a whole new ball of wax. I’ve tried that before, and that ends up becoming a very long legal nightmare.

I’m not advocating for that. You might be better off just going out and creating your own thing. If you’ve been locked in a corporate, corporate environment for, for 10, 15 years, man, I, I don’t know if you can... if you have even the stomach, that you can be a founder. so prove it to me, man. your, your salary’s gonna be 1/10th and 1/15th what you had before. It’s all gonna be about the equity. Who are you bringing with you? i-i-if you tell me, “I got five or six people I’m bringing with me” , I’m gonna know that you can recruit people, right?

I mean, those are the things you’re gonna have to think about once you get past even the does this make sense.

James: Will, what are, what are, what’s your observations about people who make for successful founders?

Will: I think first off, you have to be obsessed with a problem that you’re solving. I think that’s, that’s table stakes. And I think second, you’ve gotta have the grit and the fortitude to push through all the pain, all the moments of near death, all the suffering, and keep going and going. And for us at least, in the stage we’re investing at, we’re investing not because we think this could be a billion-dollar company.

Often it already is a billion-dollar company. It’s because we think it could be a $100 billion company. And any rational founder that got a billion-dollar offer , would sell their company and make tens or hundreds of millions of dollars and go retire.

It’s the people that, that have that fortitude, that grit, that drive to keep pushing farther and farther. And I think, it, it can happen. You look at one example is Daniel Dines, the founder of UiPath. he worked at a big corporate before, and, he decided to leave and, and start UiPath, and it took him, I think it took him, if I’m not mistaken, like six years to go from, zero to a million in ARR.

It was just this brutal slog. And so, you know, you look at that, and then obviously they went from, one to 100 million very quickly and, and now it’s a, it’s a large public company. And that’s the journey you have to expect going into this.

Daniel: Well, and also someone who can win, right? I, one, we talked about problem-obsessed. Absolutely. We talked about grit. Absolutely. we define it internally as like, do we think someone can win in their market? because it’s not enough to just have a good product.

It’s not enough just to, you know, have one of these pieces. You have to have all of the pieces, and you have to be resilient the entire way.

It’s actually one of the reasons why we focus on the Israeli ecosystem. , Really resilient founders, that have to persevere and go through a lot, that deal with real-world problems at a much younger age, that do their national service, that deal in real-world environments. and it creates a, a bit of a different type-built entrepreneur, someone that can, get up when they’ve been, you know, pushed over and, you know, when the market punches you in the face, you gotta get up. and that is , critical to being a successful founder.

James: All right. Let me ask for final words.

Will: I would say if you are a CIO or CTO trying to figure out where to spend your time, look inward. A lot of the most successful AI companies we’re seeing, at least at the application layer, are seeing incredible pull from the bottom up rather than being pushed top down. Go to your functional leaders, figure out where their problems are, figure out where they have pull, and focus your time and energy there rather than filtering all the people banging on your door trying to sell solutions top down.

Ed: Um, I would say that the time is now, right? Just don’t wait. Start small, start somewhere, just get rolling right now, and then don’t be afraid to throw things away. because I can promise you that by the time this podcast probably comes out, I know you’re fast, James, but the world may turn again one more time or two more times, right?

So whatever we say might be 90% irrelevant and 10% relevant right now. So I think the world’s just moving so fast. If I were a CEO or CIO, I would just build my own stuff on the weekend just to really understand the power of it.

That’s the only way you get to know these things

Daniel: We’re entering a really interesting moment where there’s not such a stigma anymore to try new things.

Companies need to look inward — to Will’s point — to really build the capabilities to integrate. And I think there’s openness now to try new stuff.

You’re not gonna get fired anymore for giving something a try. And I think that makes it a really exciting time to be an enterprise executive who can, to Ed’s earlier point, really put your foot on the gas on productivity.

James: Thank you all. This has been terrific.

Implications for CIOs and CTOs

Three things you probably expected going in: VCs are optimistic; enterprise AI adoption is early; legal and accounting are the obvious targets. One thing you probably didn’t: enterprises build reputations with venture funds too. Bring three companies through your procurement process, commit to nothing, and the introductions stop. The world is far enough from 2007 that waiting is no longer neutral. It has a cost now.

Footnotes

[1] Ed is referring to Fin, the AI customer service company formerly part of Intercom. Salesforce announced the acquisition for USD 3.6B — above Fin’s ZIRP-era financing round, as Ed notes.

[2] The imbalance is structural, not cyclical, and the data supports Will’s claim. On the demand side, Epoch AI data shows frontier model training compute has been expanding at roughly 5× per year since 2020. On the supply side, the global stock of AI chips is growing at approximately 3.4× per year — aggressive, but mathematically insufficient to close the gap.

The constraint is no longer primarily silicon: transformer lead times have stretched from one year to five years (Bessemer Venture Partners), and RAND Corporation projections show single training runs scaling toward 1 GW of power per site by 2028 — approximately the output of a nuclear reactor.

Broadcom’s Q2 disclosures illustrate the production gap directly: bookings for AI accelerators surpassed USD 30B against USD 10.8B in actual shipments. The one credible counter is algorithmic efficiency improvement (~3× per year), which could flatten demand before infrastructure matures — but that remains a scenario, not a trend. For the physical infrastructure constraints in more detail, see my conversation with Harqs Singh of InfraPartners.

[3] Ed is referring to GeneralistAI, a foundation model company for robotics — Boldstart Ventures portfolio. Ed is an investor.

They responded. Then Senate Majority Leader Lyndon Johnson said Americans would not go to sleep by the light of a Communist Moon. The National Aeronautics and Space Act of 1958 created the National Aeronautics and Space Administration (NASA) with responsibility for the American space program. The National Defense Education Act of 1958 sought to dismantle John Dewey’s legacy in American education, pushing schools to replace “life adjustment skills” with set theory and symbolic logic. Less than 12 years after Sputnik, Neil Armstrong and Buzz Aldrin walked on the surface of the Moon.

Share

Could Anthropic’s recent announcement of how Mythos can identify and exploit cybersecurity vulnerabilities create the Sputnik moment that will spur companies to use AI to change the way they operate enterprise technology?

The risks are real, and companies will need to move beyond buying tools and to build an agentic governance loop that uses a living graph of the environment to provide the context for spec-driven, immutable engineering, verified by adversarial automation rather than manual bottlenecks -- and then sustain and expand this change over time.

• Despite early indicators of transformative improvements, AI adoption in running enterprise technology has been shallow.

• Despite some fear-mongering, the impact of AI on the cybersecurity balance of power between attackers and defenders has been muted to date -- Mythos and subsequent models could change that.

• Mythos and subsequent models could dramatically improve companies’ cybersecurity posture in the medium term -- but they will need to use AI to accelerate their enterprise technology metabolism dramatically.

• Of course, the idea of a Sputnik moment is as much a warning as a call to action -- one-time programs are a lot easier than sustained cultural change.

AI adoption in running enterprise technology has been disappointingly shallow

Technology engineering and operations is one of the most exciting applications of AI for large companies. Large language models excel at interpreting and generating the structured content used in software engineering or technology configuration. AI can replace procedural programming with declarative programming [3], via spec-driven development. Agentic processes can better accommodate the edge cases and exceptions that have historically bedeviled efforts to automate technology operations. The early results have been exciting. My McKinsey colleagues have found that using AI to reinvent engineering processes can double team throughput. AWS has started to use agentic processes to reduce incident resolution time by three-quarters in some cases. Applied ruthlessly, AI could transform the economics of enterprise technology.

Yet adoption has been shallow. In last year’s DORA State of DevOps report, 90 percent of software practitioners said they use AI in some way, but most never used it in agent or autonomous mode and only 17 percent used it every day. The situation is no better with the cybersecurity team. According to a SANS report, Security Operations Centers use AI/ML tools, but don’t integrate them into their processes:

AI is present inside the SOC but not operationalized. Analysts use it informally, often with mixed reliability, while leadership has not yet established a consistent model for where AI belongs, how its output should be validated, or which workflows are mature enough to benefit from augmentation.

All this accords with my own observations: technology teams use AI as a tool to generate a code snippet or research an issue, rather than a lever to rip toil out of the way they do business. Why is this? The technology is still relatively new. Teams may be cautious or may not have the mental bandwidth required for change. Vendors have promised just installing a tool will solve their problems. And CIOs have not built the institutional support required to fund and prosecute the required change.

Mythos could change the cybersecurity balance of power between attackers and defenders

Since OpenAI released ChatGPT 4.0 in 2023 the great and the good have warned us about AI-enabled cyberattacks. The World Economic Forum said that specialized language models would allow hackers to get around endpoint security devices. The FBI said that AI would allow criminals to scale fraud schemes in a way that would swamp law enforcement. The UK’s National Cyber Security Centre said that GenAI lowers the barrier to entry for novice hackers allowing them to use vectors previously only available to experts. Some predictions approached fear-mongering -- sentient malware and HackerGPTs collapsing cybersecurity defenses. [4]

The worst...has not happened. I checked this morning, and the digital world continues to function. Only 16 percent of companies suffering breaches said they saw evidence of AI in prosecuting the attack. According to the Verizon Data Breach Investigations Report attackers have been just as dilatory as enterprises in using AI to reinvent their business processes:

It turns out the state-sponsored actors are just like legitimate organizations in their GenAI implementation life cycles. Attempts are being made, maybe some improvements are being found, but no one is revolutionizing anything yet.

At least as of 2024, GenAI tools could potentially assist attackers, but could not execute sophisticated attacks for them. One analysis found that GPT-4 only achieved a 7 percent success rate in exploiting vulnerabilities without clear human guidance.

Even before Mythos, the potential and the direction of travel have been worrisome. The structural factors that make LLMs effective in building and running systems also apply in compromising them.

• Intel matters in undertaking a cyberattack. LLMs have breadth of vulnerability knowledge no human analyst can read or retain -- LLM training data spans public CVEs, security research, disclosed exploits, and documented attack strategies.

• Success requires patience. Agents will cycle through potential vectors without boredom or fatigue.

• System compromise provides agents with a clear objective function they can optimize against.
  
  As a result, researchers have started to demonstrate that teams of LLM agents can cooperate to exploit zero-day vulnerabilities

Then came Mythos. Obviously we should be restrained in thinking about the implications of any software that isn’t generally available yet. And we’ve heard the too dangerous to release warning before. In its public statements, Anthropic said that Mythos had identified thousands of high-severity vulnerabilities across major operating systems and browsers—including legacy flaws like a 27-year-old bug in OpenBSD that evaded decades of manual audits. The model further demonstrated the ability to build complete, working exploits. Mythos can independently “chain” multiple vulnerabilities to gain a foothold, escalate privileges, and move laterally through a network, effectively allowing users with no formal security training to execute professional-grade, multi-stage cyberattacks at machine speed. Finding zero-days may get the headlines, but the ability to scale and operate autonomously may create the real risk.

This probably won’t realize the most dire predictions of 2024. [5] Several commentators have observed that a model’s ability to identify vulnerabilities and form plans doesn’t mean it will succeed in the face of sophisticated defenses (including the ones they have developed). But how many companies have sophisticated defenses like zero-trust in place comprehensively? And one compromise in the software supply chain could disable hundreds or thousands of institutions. Naturally, Anthropic and other frontier labs will seek to implement guardrails that limit attackers’ ability to exploit their models. The guardrails will not be perfect. And they will not apply to many of the open-weight models that will likely have Mythos-level capability within, maybe, a year.

Mythos and subsequent models could dramatically improve companies’ cybersecurity posture in the medium term. Could, not will.

After the Mythos announcement, American business and governmental elites acted. Anthropic delayed general availability of Mythos and launched Glasswing, giving early Mythos access to leading technology institutions so they could use it to identify vulnerabilities. Treasury Secretary and Federal Reserve Chair Jerome Powell called banking CEOs to Washington DC so they could urge them to take the risk seriously -- I expect they were pushing on an open door. Technology companies like AWS, MSFT, CRWD and CSCO reported that they were using Mythos to harden their products.

In the medium term, Mythos (and subsequent models) could provide a dramatic uplift in cybersecurity defenses. Companies spend fortunes each year scanning their code for vulnerabilities [6] -- Mythos-type capabilities will provide a level of transparency into vulnerabilities that we never could have imagined before. Most companies of any size do penetration testing, [7] but only the biggest tech spenders have dedicated red-team operations that figure out how a sophisticated attacker might compromise their environment. Mythos-type models should make this capability available to a much broader range of companies.

They may also revolutionize cybersecurity risk management and cyber insurance. Cyber-risk valuation frameworks like FAIR have foundered on the problem of likelihood assessment. Practitioners should be able to use a model like Mythos to simulate attack paths, determine the probability of success and make more fact-based remediation decisions. It could also revolutionize cyber-insurance, a segment historically held back by underwriting challenges.

And yet -- speed matters, and manual remediation is too slow. Mythos can help companies identify vulnerabilities. But identification protects nothing unless companies apply security patches from vendors and install fixes to code they have developed internally. That is the remediation gap the governance loop above is meant to close; in practice it breaks down into three concrete moves:

1. Create a living graph of your technology environment. You will very quickly face an overwhelming pipeline of vulnerabilities to remediate and vendor patches to apply. Not every one will be equally important, and the most critical nodes in your environment may not be immediately apparent given all the dependencies among business processes, systems, data and technology infrastructure.

Modeling your environment as a graph will allow you to identify the most critical nodes and prioritize what to remediate first. Ultimately every node in the graph should anchor in a non-human identity -- don’t connect IP addresses; connect non-human identities. Building the graph will also be an important step in moving to a zero-trust architecture.

2. Use spec-driven engineering to get to policy-driven systems. If you have bespoke software you will need to fix it. Autocomplete (or even asking models to write discrete code blocks) will not allow you to move quickly enough.

You need to retrain your engineering teams on how to use agents to diagnose root causes, build PRDs and execute on them autonomously. And you may need to do this on a timescale of months, not years.

As you develop strong capabilities in spec-driven development, you can accelerate efforts to retire technical debt, resulting in a more resilient environment. And you will want to define architecture, configuration and behavior in terms of policy-as-code so you can repave systems that demonstrate drift.

3. Move change control from human analysis to proof of safety. In many companies, the change approval board acts as a brake and a bottleneck on evolving the environment. It doesn’t have to be this way, and it cannot continue to be this way if companies seek to remediate the vulnerabilities Mythos identifies before attackers can exploit them. Heavyweight change approval processes are often ineffective. Teams of agents may collaborate to form an automated patch management pipeline.

Before you deploy a change, it must prove itself in a sandbox, both in terms of whether it breaks something and whether an adversary agent can compromise it, replacing the bottleneck of human analysis with the proof of safety. And you should deploy changes in stages, testing impact as you go. For years companies like Netflix have reconciled speed and safety by using canary analysis for staged change deployment.

None of these interventions are simple. [9] All will take attention, effort and time. But what is the alternative? Outsourcing might help, but it doesn’t remove the remediation burden at a stroke. Waiting for regulatory guidance (across dozens of jurisdictions and agencies) is uncertain and will likely take too much time. The age of security by obscurity is over. The cost of stasis may exceed the cost of change.

One-time programs are a lot easier than sustained cultural and organizational change

Less than a dozen years after Sputnik, Neil Armstrong and Buzz Aldrin explored the Sea of Tranquility. Ten more astronauts walked on the Moon in the next three years. Then, nothing. What poverty of the human spirit, what richness of bureaucratic incompetence caused us to tread on the moon and then retreat, without returning? Only this month has any human again transcended low Earth orbit?

Sputnik was a shock to the American educational system. By the 1980s, the National Science Foundation warned that Americans were in danger of scientific illiteracy. Shortly afterwards, the famous Nation at Risk report warned that post-Sputnik gains had wasted away. Not all the news is bad! American students have made large gains in fluid reasoning in recent decades -- and the dire standing of American students in global league tables may have more to do with compositional effects than school performance. But some of the news is really bad -- math scores have collapsed in the wake of Covid.

Just like a space program, the exploitation of AI in the enterprise is a generational project. Just like education reform, the ability of enterprise technology to use AI to build, run and protect systems is a foundational capability. Will your company move quickly enough to respond to the immediate challenge posed by AI-enabled cyberattacks? Will it sustain focus and attention over time to foster the capabilities to use AI not only to protect existing systems but to also make transformative leaps in business innovation, efficiency and resiliency?

Footnotes

[1] A great book. I read it every year in high school. The movie is pretty good too.

[2] Might I be evoking Arnold Toynbee’s theory of civilizational challenge and response here? Maybe.

[3] SQL is overwhelmingly the world’s most used declarative programming language. Perhaps its adoption provides the best historical parallel for spec-driven development. Replacing all the technical minutiae required for a query with a few SQL statements turned weeks’ worth of work into minutes.

[4] References not provided in order to protect the guilty.

[5] At first glance, you might ask: “How much does this matter for the enterprise? Once you get past the national security domain, how many attacks rely on zero-days?” More than you might think: a Mandiant analysis found that 70 percent of serious breaches they tracked involved a zero-day exploit. And of course a capable agentic attacker could assemble a sophisticated campaign out of a series of n-day exploits.

[6] The global security and vulnerability management market will shortly grow to the USD 20 billion.

[7] Itself a billion-dollar market.

[8] Special thanks to my colleagues Rich Isenberg and Charlie Lewis on these topics.

[9] These changes will require coordination across the technology organization. The infrastructure team will likely have to build the living graph of the environment, with input from each application team. Your core architecture or engineering team will lead the transition to spec-driven development, but much of the work will fall on application teams—and on infrastructure teams as they move from configuring systems to automating services. Transforming change control and patch management will require collaboration across the developer toolchain, infrastructure, and cybersecurity teams.

Thanks for reading Prosaic Times — share the entropy with a friend!

Share

1. Every computer system we build is an imperfect representation of reality

“No, I didn’t go to Catholic school. Why do you ask?”

An eminent partner at a previous consultancy looked at me across an empty conference room and explained: “You have an instinct for mapping data onto a grid -- you have a Cartesian mindset, which many good consultants learn from a Jesuit education.”

Grids distort as much as they reveal. We’ve all seen (or built) grids for evaluating outsourcing vendors: 20 points for stability, 30 for service quality, 30 for cost, 10 for innovation. These never produce the right answer -- they ignore non-linearity. You won’t select a vendor below your financial stability threshold no matter how cheap; once a bidder clears it, you care about cost and service, not financial stability.

Every slide, every analysis, every system is an imperfect representation of reality. CRM systems are a simplified model of how your business interacts with customers. Operational systems are a simplified model of your factory. I learned this crawling around telecom carriers in the 1990s, where you had to ask the service techs which USOC “1FR” lines had bridge taps that had to be removed before upgrading to DSL.

2. The world is entropic

Do you know whether that deal will close next week and at what price? Whether your flight to Chicago will land on time? Or whether a new employee will turn out to be a hero or a goat? I do not.

As a recovering history major, I believe the past provides counsel on the future. On June 21, 1948, the Small-Scale Experimental Machine became the first computer to run a program stored in its own electronic memory -- 128 bytes of it. Weeks later, Weaver and Shannon published articles foreshadowing many of the problems we struggle with today: automating Byzantine business processes, capturing latent or ambiguous data.

Shannon defined entropy as average amount of uncertainty in a probability distribution. [1] Maybe that’s okay -- in a world without entropy no business could exceed the risk-free rate of return. The more entropic a domain, the more information required, and the more expensive the computer system to automate it.

Weaver laid out a relationship between complexity and uncertainty. Some problems are simple: hold all variables constant except the dependent and independent ones, and you get the steam engine, the automobile, the telephone. A payroll system is the modern analogue -- fixed variables, describable rules, few feedback loops.

Starting in the late nineteenth century, scientists attacked problems of disorganized complexity. Statistics could predict the frequency of calls in a telephone exchange or the claims paid by a life insurance company, even when individual causes remained opaque -- provided each data point was atomic. Fair Isaac developed the FICO score on this basis in the late 1950s.

Then Weaver addressed problems of organized complexity -- macroeconomic management, ecology -- where many inter-related factors multiply uncertainty because each may influence the others in non-obvious ways. Neither nineteenth-century analytics nor twentieth-century statistics could handle them. He hoped new computing devices might, and pointed to the operations research developed in the Battle of the Atlantic as a way forward.

Problems of organized complexity abound in modern business -- order-to-cash, pharmaceutical manufacturing. Given enough contracts, even payroll can turn into one. [2] How much time do knowledge workers spend massaging data into or out of enterprise systems because the system couldn’t capture every exception the domain implied?

We should forgive Weaver for not foreseeing the organized complexity required to address organized complexity -- ERP programs suffer their own, and enterprise technology functions suffer from it themselves.

Writing with Shannon, Weaver named another contributor: latent meaning. Ask an account executive the status of a potential sale and she might say “We’re in final price negotiations with the CHRO and he would like to get a deal signed within the month, but the general counsel is hacked off about a couple of legal terms we require and the CHRO heard he is going to call the CEO and ask that they reopen discussions with another bidder.” What is she supposed to select among the four CRM choices for “status” -- RFP open, RFP submitted, financial negotiations or closed?

Weaver and Shannon identified one more problem -- whether a system creates the desired behavior in the people who interact with it. Think about computer systems the way the military thinks about weapons systems: as inclusive of both the technology and the user. Design choices affect user behavior in unpredictable ways. Complicated interfaces cause users to resist entering data, or to prejudice what they enter to make themselves look good.

Calendaring is deceptively entropic. If Suzy, my assistant, asks me about a conflict at 10 am on a Tuesday, I might say: “I’ll go to the meeting with client 1, because I think that colleague A can cover me in the meeting at client 2. And don’t decline or mark me tentative for anything, because I don’t want anyone giving away the time slot because I might not be there.” I don’t verbalize dozens of assessments about people and situations that shape my decisions. You have entropic communication about organized complexity, latent information and adaptive behavior all in a couple of sentences!

3. Entropy creates cost and frustration when we build systems to represent it

How much time do we all spend in conference rooms debating how much granularity the data model should have -- and how we balance fidelity to the business process versus the cost of maintaining data? How frequently do you find the correct state for a transaction not in the fields designed to contain it, but in the free-text notes or the accompanying email? And how much user anger derives either from the endless fields (specified to capture the subtleties of state) or the disconnect between what the system says and the reality they observe?

Put another way: how much time do we devote to designing around entropy when we build systems? And how much frustration do we create because of the choices we make in doing so?

Operational systems work best when the world can be reduced to stable categories, deterministic workflows, and enumerable exceptions -- all these make a domain easy to formalize. Nobody complains much about most payroll systems because they apply a bounded number of explicit rules to available data.

Analytic systems work best with huge sample sizes and a few relevant independent variables -- making the entropy here tractable to statistical analysis. Machine learning systems for pricing consumer products work because massive amounts of structured data allow probabilistic inference where humans might have struggled to discern patterns.

Other domains and use cases have more intractable entropy. Systems to support treasury management for large enterprises? Modeling a thousand-page contract, involving hundreds of legal entities, dozens of jurisdictions and a thousand different exceptions you can have for a transaction -- all of which generate entropy. Which messiness must you model and which can you simplify away? CIOs have exceeded budget, blown deadlines and angered users in seeking to answer that question.

Yes, machine learning has blunted the impact of entropy in some cases by sniffing out the relationships among variables, but often the sample size is too small and the data too messy for traditional machine learning to be effective. Years ago, someone said collections managers could offer definitive recommendations about how to reduce losses -- why couldn’t CISOs do the same when talking about how to protect the business against cyberattack?

I tried to explain that each demographic segment included millions of households. You could experiment with a new script and quickly determine whether it increased or decreased promises-to-pay. Any large company’s technology environment is a snowflake -- even two companies of similar size and in the same sector may have radically different technology environments. Vulnerability depends not on individual decisions, but on how you connect all the pieces in your environment. And you might not know whether you had been breached for years, if ever. [3]

4. GenAI-based agentic systems process entropy

Deterministic systems cannot process entropy. They can rely on humans to pre-process it for them -- as happens when product managers and engineers sit in a conference room debating how to capture the data required to automate an ambiguous process. They push it on to users like account managers who must fill in endless fields about their pipeline -- and still wonder which option they should select to describe deal status. Or they can store it inertly, like the free-text notes that exist in some customer service platforms. All of these imply some combination of low efficiency and lower effectiveness.

When large language models process text, they convert tokens into vectors, points in a high-dimensional space. This encodes meaning through proximity -- similar meanings cluster together. Large language models can do things deterministic systems cannot do or do poorly:

• Allow inference across stored entropy: Written and spoken language are entropic. Vector embeddings allow LLM-based systems to query and analyze free-text notes, email threads and meeting transcripts.

• Identify implicit relationships: Vector geometry represents connections between concepts, such as the general counsel wanting to block the deal for group health insurance because he disliked the contract terms.

• Understand gradations rather than discrete states: A deterministic system requires the account executive to choose whether the deal has advanced to final negotiations or not. A vector representation of the same situation can sit between two states.

GenAI-based agentic [4] systems give us new ways to process entropy.

• At design time, by using software engineering agents to create business logic that reflects all the organized complexity that Weaver described.

• As part of the user interface, to mediate between the entropy of written language and a deterministic system, rather than forcing the user to do the work.

• Via data ingestion, to derive insights from the massive stores of entropic, unstructured data every enterprise has sitting on its servers.

• At run-time (either with or without a human in the loop), to make decisions and execute transactions non-deterministically.

We can also store data used and produced by agentic systems in different ways. Traditional relational databases excel at storing and retrieving massive amounts of transactional data at speed and with near-perfect reliability. They struggle with more ambiguous data, with dense relationships among all the elements. [5] We can choose to store this data either in vector databases or in knowledge graphs.

5. You have choices about how and where you use agents to process entropy

So much of the agentic discourse we see mirrors the flat Cartesian mindset captured in the grid used to evaluate bidders for outsourcing deals. Agents are risky! Or: everything will be an agent!

But entropy is, well, entropic. It varies in scale and shape from business domain to business domain. Sometimes it takes the form of a dense web of interconnections among idiosyncratic products, contracts, processes or customer relationships. Sometimes it takes the form of latent data that doesn’t fit into any data structure you might define. Not a few business domains suffer from multiple forms of entropy.

GenAI-based agents also have disadvantages compared to deterministic systems. GPU-based inferencing is slower and more expensive than CPU-based processing -- a single agentic transaction often costs cents and adds hundreds of milliseconds, where the equivalent deterministic transaction costs fractions of a cent and resolves in single-digit milliseconds. Non-deterministic systems are only as good as the context they receive. An agentic system can process the free-text notes in a CRM platform. It won’t know anything about the discussion two account executives had in the car on the drive back from the customer site. [6]

Abjure the false binaries -- you have real choices about how and where in the value chain you process entropy.

• Agentic software engineering: This will be relevant for almost any systems effort. It will allow you to automate all the business logic required to reflect organized complexity with more speed and reliability and less cost. And after agents help you develop the code, you can still apply all the quality assurance mechanisms you have developed over decades, just as you would code developed by hand.

• User interface: Retain traditional interfaces for situations where users have to enter small volumes of easily understood data. Develop agentic- or chat-based interfaces for order entry, CRM, or transaction processing systems where users complain about having to fill in screen after screen of data. You can always use a combination of deterministic rules and user validation to ensure the agent correctly captures user intent.

• Data ingestion: This is one of the most powerful and most underestimated capabilities. All companies receive and store vast amounts of valuable but entropic data -- eighty percent of corporate data is unstructured and even structured data can be fragmented and hard to correlate. Customer requests for quotation? Customer service notes? Legal contracts. Agentic capabilities can ingest all of this either to support operational processing or to generate new business insights, even if many companies have not focused here yet. [7]

• Run-time: This is complicated and has the highest stakes. Use agentic patterns for low-volume heterogeneous decisions. I am helping technology organizations use agents to compare project designs with technology standards. The agent will compare two artifacts (a project design and a standards document) that describe organized complexity in natural language (which is inherently entropic) and determine whether one conforms to the other. Hybrid patterns will probably become increasingly common, with deterministic cores and agents to handle edge cases.

• Data storage: Naturally, relational databases will continue to store structured and transactional information. Knowledge graphs excel in modeling product portfolios, customer relationships, process maps and other domains where relationship structure carries meaning. Vector search enables retrieval by meaning from notes, documents and transcripts.

6. Precision matters because entropy is, well, entropic

I have always hated the formulation “The technology is easy. The governance is hard. The organization is hard. The line at the cafeteria is hard.” Here, design choices will shape cost structure, risk posture and user experience for years.

With a capability to process entropy at scale, you have to think about where the entropy sits in your business system -- even a simple one.

Managing invites for the Technology Leadership Forum is lower-entropy than a hundred-million-dollar group insurance deal, but there is plenty of “Joe really wants to attend, but can’t, depending on when a personal conflict lands -- and wants to know if Sally can possibly attend in her place.” I put invitees and members into a knowledge graph and built Agent Serena to translate the entropy in my email into the organized complexity of that graph. [8]

Taking a more highly-scaled domain: entropy pervades enterprise change management in banking. Dozens of interconnected process steps and artifacts; latent information about why a milestone might be at risk of slipping and under what circumstances. Traditionally, banks managed it with managerial talent copying data among word processing, spreadsheet and presentation files. And many, many emails. The result: frustration, expense, and less insight into major programs than anyone would like.

Banks can use agentic software engineering to build a system whose agents ingest program documents, extract the relevant risks, issues, decisions and action items, and store them in a knowledge graph with the relationships among them. Which risks affect which business lines? Which work-stream lead owns which decision? Further agents interrogate emails and videoconference transcripts to enrich the graph. Vector descriptions let program managers search by meaning, not keywords. Still more agents read the graph to assess risks and surface opportunities to improve the program.

This combination -- agents, knowledge graph, vector database -- removes the entropy from a painful business process. It reduces cost and improves transparency. Those of us on team cyborg (rather than team android) will note that it also empowers managers: less toil, better information.

Agentic systems -- with knowledge graphs and vector databases -- can be transformative. When I wrote that global business must move beyond organized factories and chaotic offices, this is what I meant. The benefits are both economic and humanistic. Nobody grew up hoping to spend days pasting data from email to spreadsheets and back.

Precision matters here. Agents are not magic. Inferencing is expensive, latency is real, and new vulnerabilities arrive faster than we can grasp them. But the harder discipline is the one Weaver and Shannon left us: see your business as a system, find where the entropy sits, and choose where to process it. A flat, Cartesian grid won’t help.

Thanks for reading Prosaic Times — subscribe to receive every issue!

Footnotes

[1] This differs from standard deviation, which measures the magnitude of the spread around a mean. Shannon entropy measures the unpredictability of a distribution. Relevant to us: you can measure the Shannon entropy over a set of non-quantitative values just as you can over a set of quantitative ones. In addition, there are more and less expansive definitions of entropy. I am using a relatively expansive one here.

[2] CityTime is a great example of Kolmogorov complexity. The shortest possible description of the rules governing what every city employee gets paid is approximately as long as the rules themselves — decades of negotiated union contracts, grandfathered provisions, and exceptions to exceptions, each one load-bearing. You cannot compress it further without losing something that will eventually matter. Automation doesn’t reduce this complexity; it just moves it from humans who held it in their heads to engineers who must encode it in systems.

[3] My interlocutor said I was over-complicating the situation and CISOs were probably just incompetent. Sigh.

[4] Agents predate widespread adoption of large language models and genAI. By some definitions, the cruise control system in your car is an agent. You set a speed. It monitors the speed, accelerating when the car drops below that speed (because you are going up a hill) and easing off the throttle when you exceed it. Of course calling LLMs makes agents infinitely more capable than the cruise control in your car.

[5] The notion that relational databases might struggle with relationships among data elements might puzzle some. But only those who have never stared at a screen in the early morning trying to make sense of an outer join -- or who developed a better command of SQL than I ever had.

[6] I like the idea of decision traces and context graphs, but we should be realistic about which decision traces we can capture and which we cannot, unless we want to build a panopticon for ourselves.

[7] I stared in disbelief the first time Zubin Ghafari showed me how he used GenAI to integrate messy CMDB data with other telemetry information. I had assumed this would have taken him weeks.

[8] The natural language front end to Prosaic Graff was a life-saver, but the latency sometimes made me want to put my fist through the screen. So I created Python scripts I could run (instantly) from the Terminal that told me how many members planned to attend or the status of any individual member.

That same year, I told our teacher that the Cherry Hill Study Skills program was rudimentary that teaching it wasted my time and the taxpayers’ money. I left to go read a book in the library, and got me detention. A couple of years later, I asked my Spanish teacher to lower her voice in class as I had a hangover. Also detention. I butted heads less with professors in college, but I also skipped a lot of class to take care of the Brown Daily Herald.

This guy. Me?

I have done a better job learning and sometimes teaching in my professional career than I ever did in high school or college. An email I just received reminded me of this.

Back in the day, Ed Hsu and I worked together fixing technology infrastructure organizations. He left McKinsey in 2010 and has since built a distinguished career at VMware, Rescale, and Mixpanel.

His note was brief:

hi James. This guy. You. :)

It included a link to his substack issue on The Personal Impact Chain, which read:

During my first weeks at McKinsey, a senior partner gave me the most concise career advice. Over dinner, I asked what I needed to get right first.

“Know the data,” he said.

Not “develop executive presence.” Not “build relationships.” Know the data. Everything else, the problem solving discipline, the story, the recommendation, was downstream.

Over the years, I built on it. The Personal Impact Chain is the result: a framework for staying grounded in facts, finding the insights that matter, building the case for change, earning the coalition to act, and delivering outcomes.

Accounts receivable optimization in Canada. In winter.

My first project out of business school (before even I worked at McKinsey) [1] sparked this insight. I know that the client wanted us to design customer programs to reduce Accounts Receivable, but the approach and the work-plan were mysteries.

Despite the confusion, I learned a ton on that study, including the importance of a healthy respect for the complexity and fragility of legacy systems.

The business unit’s IT department thought providing data for business analysis was not an important part of its job. Eventually I made friends with one of the DBAs, and we would log onto the mainframe together and download massive amounts of raw data that I imported into a SQL database I built in MS-Access. [For our younger readers, before you could use a cloud database…oh never mind.]

I could analyze consumer A/R patterns by BTN and enterprise A/R by major account. I could immerse myself in the data.

Like all the other consultants on the project I had a beat up metal desk on a big floor, and both other consultants and clients (some of whom had nothing to do with our project) would crowd around my desk asking me to query my Access database. People started calling me the counter-CIO. I liked being the counter-CIO.

I faced a limitation — I didn’t have any A/R or payment data on mid-market customers. After lots of investigation, I figured out the data I wanted lived in a system called CRIS (Customer Record Information System), but nobody knew how to get a query run against it.

So I sat at my beat up metal desk, and called everyone I could think of (using an actual landline telephone) to ask who could run a query against CRIS for me — then I would have all the data I needed. No luck.

Then one day my phone rang. The caller announced himself as Victor Rochambeau (not his real name), and asked if I knew who he was. Yes, I did. Victor was one of the most senior people in IT — not the type of person I was supposed to be talking to as a callow greenhorn.

Victor continued, “James, how old are you?”

I stuttered, “Excuse me?”

“How old are you?” he repeated.

“Twenty-six,” I replied.

“So you were born in 1970?” he asked. (This was in 1996.)

I assented.

Victor continued, “James, I have a new rule for you, so long as you are on this project. You are not allowed to request queries on systems older than you are.”

He went on to explain that they deployed CRIS in 1964 and ported it to minicomputers in the 1970s. And that ad-hoc queries crashed it. He was not crashing his system for my query.

Data is power

I never did get any mid-market A/R data, but familiarity with data increased my stature in small ways.

I thought it would be interesting to attend the first client progress review. So I asked my Manager, whom we called The Shadow), [2] suggesting I might be able to speak to the underlying data. The He scoffed: new associates had no place in progress reviews.

The meeting approach. After we printed out copies of the deck, [3] we watched the Shadow and the Senior Manager walk down a long hallway, wondering what progress reviews were like.

The Senior Manager and the Shadow stopped and talked for a minute. It looked like the Senior Manager was asking questions that the Shadow could not answer.

With a hangdog expression, the Shadow walked back toward us, pointed at me and indicated I should join them. So I excitedly trotted down the hall toward my first progress review. The project was a mess; the first progress review was a mess; and I learned a lot watching the mess.

At the end of that project I derived a theory of victory for my career in consulting. If you understand the data, you will be essential for the analysis. If you are in the middle of the analysis, people will want your point of view on the recommendations. If you understand the recommendations, you can help construct the narrative. Once you help craft the narrative you have an opening to participate in the client relationship. And then you’re getting somewhere.

That insight got me far. I resolved to understand more than anyone around me — about the telecom operations, then about technology infrastructure, cybersecurity, cloud computing and more recently GenAI in enterprise technology. I was a slacker in higher education, but I hit the books hard as an adult.

But I didn’t follow my theory of victory to the end. For too long, I focused too much on the data, analysis and recommendations -- and not enough on the narrative and how it motivates people to act. With much frustration, I haltingly learned to think as much about the second half of the chain as the first. Yes, the truth is the easiest thing to remember and you should ask every question. But also: to influence you must be open to influence and how you talk matters as much as what you say. [4]

Ed took advice I gate him and developed it further into something of his own — with a strong emphasis on the factors that I hadn’t focused on enough early in my career.

Pasted image 20260613225603.png
He writes:

Product leadership is fundamentally about leading without formal authority. You set the direction, align stakeholders, and own outcomes; all without controlling the people who make it happen. That requires something more basic than product intuition: a personal operating system for getting from facts to impact, on any type of project, at any scale.

There’s a progression that underlies almost every professional initiative. It’s not in job descriptions, but it structures careers:

Data → Insight → Story → Action → Impact

Like spirals in nature, this progression runs in any scale. From a junior analyst delivering a report to an executive driving a company-wide transformation, it’s the same structural progression, just at a different scope and time horizon.

Action at the small scale is personal discipline. At the large scale it’s about diplomacy, dependency management and a dose of formal authority. At senior levels, this framework maps to how people are evaluated.

One of the pleasures of a long career is hearing how others might remember and use some of the things that you’ve learned. And you don’t get detention.

Thank you, Ed. You made my day.

Footnotes

[1] Yes, there was a time before I joined McKinsey. Dinosaurs roamed the Earth. We used fax machines.

[2] We had great nicknames on that study. The BA was Nermal. The other associate complained about everything except Texas, so we called him Lone Star.

I got everyone to call him Lone Star: our team, all the consultants on all the other work streams and all of the clients. Here’s how thoroughly he became Lone Star in our minds.

One evening I needed to ask Lone Star about something, but he had already gone back to the hotel. Almost nobody had a cell phone in that era (before AT&T rolled out the Universal One Rate plan), so I used the PBX handset on my desk to call the local Marriott switchboard.

“Get me Lone Star,” I requested.

“I don’t understand,” the operator replied.

Slightly frustrated, I said “Lone Star. Lone Star. I need to speak with Lone Star.”

Watching me, with no small amusement, Nermal told me: “James, I don’t think they know about Lone Star at the Marriott.”

[3] Back before everybody had giant LCD screens in every conference room...oh, never mind.

[4] Me at 30:

Scene: 5:30 pm, Friday afternoon, CIO’s office: she is packing up to leave for the weekend.

CIO: “Oh, yeah, we have time now. I’m trying to get out the door.”

Kaplan: “I should tell you about that cost savings plan we were supposed to validate.”

CIO: “Yes?”

Kaplan: “You told the CEO you had a plan to save $20 million.”

CIO: “Yes?”

Kaplan: “We dug into all the plans and the underlying data and we don’t think it’s $20 million.”

CIO: “How much is it?”

Kaplan: “Maybe $3 million.”

There is a time, a place and mechanism for delivering bad news. This wasn’t it.

My favorite coffee shop has an enterprise technology vibe. You hear people discussing SQL query syntax on video conferences, as if text-to-SQL weren’t a thing. One of these days, I will launch a podcast sitcom (in the style of an old-fashioned radio comedy series) called the coffee shoppers. In the pilot episode, the regulars will conspire to rid their favorite establishment of a matcha machine that attracts the wrong element, like hipsters.

One of the regulars (also a devoted reader) asked me to start including a book recommendation related to the issue’s theme. Since I prattled on about Military Power: Explaining Victory in Defeat in Modern Battle last week, let me suggest Thinking in Systems by Donella Meadows.

What drives bad enterprise technology? Short term and siloed thinking. When you implement a security control without understanding the user experience, that’s siloed thinking. When you build a use case without worrying about what the underlying platform might be, that’s the same mistake. It leads to high cost, technical debt and frustrated users. You see a lot of it as companies wrestle with deploying GenAI.

Donella Meadows defined systems thinking as the discipline of understanding how the behavior of a whole emerges from the interactions of its parts over time. It focuses on the relationships, feedback loops and delays across a system that spans the human and technological domains. You might say systems thinking implies seeing the world as a graph — and using that insight to optimize the entirety of a business domain rather than individual pieces of it. The ten beliefs that follow are my attempt to apply that insight to AI strategy.

A colleague asked if I could get a perspective on GenAI strategy on a couple of pages. That’s tough — specificity, nuance and general applicability are uneasy bedfellows. How do you navigate between the Scylla of platitude and the Charybdis of over-simplification? You have to stop thinking about a collection of technologies and a collection of organizations, but of a business-technology system where each part influences every other part in some way.

10 beliefs on how to get value from GenAI

1. The business world is entropic. Generative AI represents a technology discontinuity because it can process entropy rather than ignore or externalize it, e.g.

• Complex and interconnected, incompletely documented business rules (e.g. like the ones in 1000-page contracts)

• Unstructured or uncorrelated data (e.g. doctors’ notes in eHR)

• Unstructured user instructions (natural language input)

• Latent information (e.g. status of B2B commercial proposal)

2. GenAI represents a strategic discontinuity because it allows granular analysis and digitization of business domains where you previously needed humans to manage complex and ambiguous information

• The largest opportunities will be in the most entropic domains because companies successfully digitized less entropic ones (like consumer auto insurance) years ago

• Returns to investment will be lower for B2C businesses than in B2B businesses

• They will be lower in operations for process manufacturing than in service operations

• They will be high in domains like R&D and product development where digitization has been limited

3. Speed and quality will be as important, if not more important, than efficiency in achieving competitive advantage from GenAI

• Returns from technology investment vary massively by period, sector and company — and companies compete away the value of productivity improvements as consumer surplus.

• Updating a contract in minutes rather than days may be essential when negotiating a transaction

• Reduced revenue leakage may swamp operational cost reduction in setting up the billing for an asset servicing or group health insurance arrangement

• You can also use GenAI to create less frustrating user experiences, even if the impact is harder to quantify — even more importantly you can use it to reshape markets by reducing commercial friction and transforming unit costs.

4. The Jevons paradox applies to busywork as well as knowledge work, so you want cyborgs not androids

• GenAI can rip the toil out of knowledge work and improve the rigor of your employees’ thinking — you need knowledge workers who can apply judgement and reason across multiple domains, rather than just provide specialized expertise

• A lower unit cost of business insight will create demand for business analysis that had previously been uneconomic — if you are willing to rethink how you manage knowledge and even how you produce documents

• But when you put GenAI on the desktop, employees will often devote tokens to writing and summarizing emails or performing more complex web searches — while GenAI can help your employees write more effectively, AI-generated prose is vague and unconvincing

5. Even though token prices have been declining roughly tenfold per year, the cost of inferencing will finally require enterprises to treat IT spend as COGS rather than G&A

• Run costs will increase in relation to build costs

• Companies will need to ensure both that they devote GenAI resources to valuable projects and that they design agentic systems for token efficiency

• But inferencing latency may big a bigger problem than token costs

6. Every business system processes entropy somewhere — in product managers who formalize ambiguous workflows, account managers filling in CRM fields, free-text notes nobody reads. Agentic capabilities give you choices on where to put that work, each with a different cost, risk, and speed profile:

• Agentic software engineering — automate the business logic that reflects organized complexity; apply your existing QA mechanisms to the output just as you would hand-written code

• User interface — deploy chat or agentic interfaces where users complain about screen after screen of fields

• Data ingestion — eighty percent of corporate data is unstructured; agents can ingest contracts, service notes, and RFQs that deterministic systems ignore entirely

• Run-time decisions — use agentic patterns for heterogeneous decisions (e.g. architecture review); hybrid patterns (deterministic core, agents for edge cases) will become the dominant architecture

7. Context is existential for GenAI, and requires different mechanisms for managing it

• Relational databases excel at storing transactions

• Knowledge graphs allow you to store and analyze the relationships among entities (e.g. customers, products, process steps, service offerings) that GenAI discovers

• Building an ontology and semantic layer (and encoding it as a knowledge graph) both reduces the likelihood of hallucination by over 80 percent and depicts your business as a system, allowing you to make better business decisions

8. Agentic technology will not make enterprise technology functions disappear. You must build robust common platforms in order to apply GenAI at scale, with efficiency, with security and with resiliency

• Agentic software engineering replaces procedural programming with declarative programming

• Doubling throughput and productivity will only be the start here for those companies that reimagine software engineering processes, rather than just providing engineers with copilot-type tools

• Reimagining software engineering can transform the ROI of tech-for-tech investment and the “IT doom loop” and double or triple the EBITDA lift from enterprise technology by freeing up budgetary capacity for new investment and reducing value leakage and engineering deadweight loss

• The challenge of Agentic cyberattacks will require enterprises to rethink risk management processes and increase the automation of their technology environments in order to remediate vulnerabilities discovered by frontier models like Mythos at speed

• Agentic software engineering will change buy/build decisions and challenge “one size fits none” SaaS models. More enterprises will connect disparate software tools or will contract more niche vendors whose products more closely support their business processes

• All of this requires treating enterprise technology as an integrated system and support from across the management team for the required investment and changes in the business-technology operating model

9. Agentic technology will change the relationship between users and the technology function

• Users can achieve wonders by acting more like software engineers for “edge” applications

• They will increasingly look like “Strats” who sit on a trading floor and update models intraday — the ED charge nurse who reweights a sepsis-triage rule when a new strain appears, the plant engineer who tunes a predictive-maintenance threshold when a new failure mode shows up, the claims supervisor who stands up a fraud agent overnight

10. Advantage will accrue to institutions that metabolize GenAI and scale its adoption

• Capturing value will disrupt existing assumptions about organizational structure, ways of working — and especially the role of governance functions like HR and procurement

• You need to treat the technology and the operations around it as a single system and evolve them together

• You want to change everything someplace rather than something everyplace. Transforming a business domain (rather than funding disconnected use cases) creates local critical mass and provides visible results

• Executive support in the face of resistance requires you start with hard problems (rather than easy ones) and a measurable target

GenAI is new and it is different. It does things that we could not get systems to do in any scalable way before. It applies to a far broader set of domains than traditional enterprise technology. Just as the railroad reshaped the geostrategic environment (empowering continental powers relative to maritime ones), GenAI will reshape the competitive dynamics in many sectors.

Those who seek simple checklists in applying GenAI will fail. You might ask why it is different — how it can allow us to automate new types of tasks. You must understand how it will interact with your strategic context and your organizational culture and capabilities. And then you can determine how you might use GenAI as a weapon. None of this is easy. But Edward Luttwak reminds us that strategic advantage derives from doing hard things, rather than easy ones.

I love Stephen Biddle’s book Military Power, which explains how armies have used the modern system of force employment (effectively tactical doctrine) to win battles. He explains that technology innovations since 1914 have requires armies to learn how to use:

• Dispersion

• Cover and concealment

• Suppression

• Small-unit maneuver

• Combined arms

• Defense in depth

• Decentralized initiative

Biddle points out that

• Technology provides nothing; the effective translation of technological change into coordinated tactics (via the modern system of force employment) provides everything, or least the ability to win battles and possibly wars

• Adopting the modern system of force employment requires a very different culture than more traditional forms of war-fighting, especially the ability to trust non-commissioned officers with a lot of responsibility

This tells us a lot about agentic software development

• Giving your engineers access to new tools buys you nothing; building a new method software engineering using some of those tools buys you 2x, or perhaps 10x, the productivity

• Adopting new methods of software engineering requires rethinking skills and mindsets

• The results could have strategic impact. For most companies, software development is the rate limiting factor for every new product, every new channel, every new market and most operational improvements. Enhancing software development productivity accelerates the metabolism of the enterprise.

I was excited to speak, with my colleague Matt Linderman, with Tessl CEO Guy Podjarny because of his ideas about how to create the scaffolding for agentic software engineering -- how to do it at scale. As always, Prosaic Times never endorses any product or offering, but we find great insight in hearing directly from builders bringing new capabilities to market.

1. The founder is a addict entrepreneur

In this section:

• Guy is x-Akamai CTO and Snyk founder, now founder/CEO of Tessl.

• He frames himself as an “official addict entrepreneur.”

• Matt is a McKinsey partner in New York leading the firm’s software practice — three to four years building an AI-first, now agentic, product and engineering practice. Grew up near an upstate-NY Air Force base that did AI research.

James Kaplan: Currently hanging out in southern Rhode Island with another Prosaic Times video podcast. Today we have my colleague, Matt Linderman from McKinsey, and Guy Padjarni from Tessl. Did I pronounce that correctly, Guy?

Guy Podjarny: Good.

James Kaplan: Guy, why don’t you tell us a little bit about yourself, and we’ll bring you to Tessl.

Guy Podjarny: Sure. I’m at this point an official addict entrepreneur. After being a developer and turning product and such through a few acquisitions, I founded my first company in the web performance space. I sold that to Akamai, where I became CTO, and did that for about three and a half years. After about three and a half years, I got the itch to found another company, and I went on to found Snyk, which I think had a good dent of impact on the application security world.

James Kaplan: Maybe just a little

Guy Podjarny: It had a couple of years of wandering the desert and near-death experiences that nobody remembers, because everybody remembers what happened post the sort of two years.

But then it grew nicely, formed the DevSecOps movement and the developer security space. So that grew nicely. And about two and a bit years ago, I accepted I’m an addict and fell in love with AI, and left Snyk, where I’m still chairman of the board, to found Tessl, focused on reinventing software development.

What is the new software development paradigm? That is what Tessl is focused on.

James Kaplan: You’re in London now?

Guy Podjarny: I am based in London. Born and raised in Israel, spent a decade in Canada, and I’ve been in London for the last 13 years.

James Kaplan: Matt, you wanna introduce yourself and tell us a little bit about your journey?

Matt Linderman: I’m a partner in McKinsey’s New York office, and in particular work in our software practice on engineering and product management topics. For the last three or four years, we’ve been building a practice around AI-first and now agentic product management and engineering practices, and lead that more broadly for McKinsey across software companies and across a number of companies outside of software — banks, telcos, et cetera — who are all going through this journey together.

And so Guy, very excited to chat with you about what we’re seeing, what you’re seeing, and mash this together as we go through the conversation here.

James Kaplan: And you grew up in upstate New York?

Matt Linderman: Grew up in upstate New York near an Air Force base that did AI research, actually. So I grew up with this very close to heart, and now live in the New York City area.

James Kaplan: Great. And we appreciate you classing up the joint here on the podcast by wearing a shirt with a collar, Matt. You’ve raised the bar for Guy.

Matt Linderman: I was forced to do it due to some bank conversations, but my T-shirt is available for later.

2. Code is disposable; context is the new code

In this section:

• Guy reframes “vibe coding” as the first leg of a journey toward agentic engineering — guiding the agent, not writing the code.

• The Tessl thesis, two years old now: code becomes disposable; context becomes the unit of work.

• Guy sketches the emerging stack — models → tools → context (skills, rules) → harnesses → factory lines → factories.

• Harnesses are deterministic guardrails on a probabilistic model: OpenAI’s no-commit-without-coverage rule; Intercom gating PRs on loaded guidelines.

James Kaplan: You’re allowed when it’s warm in London. Guy, give us a little bit of the big picture on the evolution of software development and the adoption of agentic — or spectroven — development.

I admit, I don’t love the term vibe coding. But tell us a little bit about where we are and where we’ve been and where you think the world is going.

Guy Podjarny: Vibe coding is interesting. I think about vibe coding to agentic engineering as a journey: there is vibe coding, where you just vibe away. But really what we want is agentic engineering. I’d say, thinking a little bit about Tessl’s journey, two and a half years ago — or just less than that, when we founded Tessl — we already had the conviction, which is more clear today, that software development will, at a high level, transform from revolving around code and implementation to revolving around intent and instructions.

It is more about guiding the LLM — at the time, and today the agent — to do what you believe to be right, versus writing the code yourself. At the time, we talked about how code will become disposable. It was a bit more heresy at the time. Today it’s well accepted that implementation will become something that is regenerated, and it’s less important.

We’re not fully there yet. We believe this will drive a new software development paradigm, and we didn’t really know what that is. What are the units? Today we’re starting to see the outline of what it is that you develop in this world. We’re still developing software, still creating something that will evolve, that we debug if it doesn’t work, that we observe in production, that we collaborate on.

So there’s still a thing we’re developing — what humans are developing — what I think of as the context development life cycle. I can describe a little bit of the stack that is shaping up, and I’m sure this will continue to morph and modify.

At the bottom of it, there’s the models. This is the new primitive we’re building on. Think of those like architecture. They’re like operating systems — whatever it is that you build, is it compatible with that layer?

A layer on top of that is a layer of tools that perform various actions. They might be your grep and FFmpeg. They might be custom built.

Above that is the layer of context. I think of context as the new code. We can dig into that a little bit more, but context is where most developers spend most of their time.

It’s conveying what they want the agent to do. We have skills, we have rules.

Further up the line, you start seeing harnesses. Harnesses are really about constraining the model — they’re harnessing the model. This is deterministic software that decides when to make decisions that are not delegated to the probabilistic model. Hooks, for instance — OpenAI saying, “You cannot commit, you cannot run Git commit until the test coverage is above a certain bar.”

Or Intercom saying, “You cannot open a GitHub pull request unless you loaded the relevant context around our PR guidelines.” These are examples where the harness — the configured harness — is telling the model, “You do not get to choose; this has to happen,” because it harnesses the model. And then in turn, harnesses compose up into factory lines.

These are more like pipelines. Harnesses are kind of like frameworks, if you will. Most organizations will choose and highly customize a framework, or they will build their own framework. They probably won’t have a gazillion of them, so it’s not the same as a dev. Up to the factory lines, they’re more like pipelines.

You want some consistent input. With a certain type of input, you have a successful output. You want to go all the way to factories, which are more like your development process. So this is a new software stack, and these analogies are helpful when we think about, okay, as we scale, what do we need?

What tools do we need? What practices do we need to build that out?

3. Stop spec-ing the product; start spec-ing the programmer

In this section:

• James offers three historical analogies for the shift — assembler to C, procedural to declarative, and entropy reduction down the traditional design chain.

• Guy takes the last two but insists this leap is larger than any prior transition, and is happening faster.

• The right comparison is to instructing humans: probabilistic, resilient, interpolating within boundaries — except agents can’t lose their jobs.

• Spec-driven engineering is a practice, not a product; specs are only one slice of the broader context-engineering problem.

• “Speccing the programmer” — encoding constraints, API choices, framework preferences, and billing limits alongside the spec itself.

James Kaplan: Which metaphor resonates most with you as we think about the movement to agentic development, or spectrum development? The first historical metaphor I sometimes think about is the transition from assembler to third-generation languages like C, which we went through in, say, the late ’80s, early ’90s.

And I’m old enough to remember a lot of rent garments and concern that, “Oh my God,” you know, third-generation languages would be slow and unwieldy compared to programs written in assembler. The second metaphor I sometimes think of is the transition, or the distinction, between procedural languages and declarative languages. And sometimes I think about when the world went from building database management systems by hand, or building databases by hand, to using SQL — which I think you can argue is a declarative language. And when you say spectrum development, that to me sounds like it’s declarative. And the third is entropy reduction — in the sense that in a traditional software engineering process, you went from, say, a senior business executive’s declaration of intent through to a business case, to a conceptual design, to a set of business requirements, to a set of technical requirements, to code.

You had humans at every step in the process, in effect moving entropy until you had deterministic code. Which of those metaphors resonates with you, or do they resonate in different ways?

Guy Podjarny: Yeah, I’d pick a spot if that was a continuum.

James Kaplan: Please.

Guy Podjarny: The last two. The analogy for software evolution — this is a bigger leap than any one of the changes that we’ve done. So even if it is directionally analogous, and that’s useful, ’cause we’re humans and we like analogies, and they help us reason about the world — but I think it is a bigger jump, so we have to acknowledge that, and it’s happening faster than the previous ones.

I’d say it’s somewhere between changing — taking that sort of entropy, or how do we instruct humans. As we cascade, in which we expect probabilistic behavior, we expect when we guide our reports — people that work for us — not to get perfect, but to get resilient. You want them to understand the spirit of what you’re saying and to have some judgment around when to bend the rules or expand the rules, to build it.

James Kaplan: Interpolate, often interpolate, right?

Guy Podjarny: Interpolate. And you want it within boundaries — which is always a tricky thing — and accountability. We should get back a little bit to accountability, ’cause the key distinction between that and the others is: a person who repeatedly makes bad decisions might lose their job, and an agent doesn’t really have that. But on the other side, where they’re meeting the other pull from software, is just the increased desire for abstraction. If you think about a bunch of these transitions you talked about — but even going into Java, and then in infrastructure going into the cloud — anything that became software-defined, suddenly you’re saying, “Hey, get me ten more servers.” You’re not dealing with anything. It’s just spin up ten more servers. Before, you said, “Give me ten more megabytes of memory.” You didn’t even say that. You said, “Give me an array of a thousand objects long.” And Java —

In each one of these things you’ve delegated decision-making to something downstream that you can configure, you can control, but is not entirely yours. The entropy piece is interesting because in all of those cases you still expect a lot more determinism, and a lot less resilience — a lot less adaptability — than you do with humans.

We meet somewhere between human instructions and a higher-level abstraction, and those are probably comparison tools as we think about what works and doesn’t. I’ll just refer one thing to the spec thing: at Tessl originally we talked about spec-centric software development. Spec-centric evolved into spec-driven development, which I believe to be a real practice.

I don’t think spec-driven development is a product. It is a practice.

James Kaplan: Of course. Yes.

Guy Podjarny: We should go from speccing the product, or speccing the program, to speccing the programmer. It’s important within the team — when you think about instructions, specs are just a part of the puzzle. What you expect from people in the team is not just to always update the information about the product and how it operates when they modify it.

That’s one of the things you want. But you generally expect them to make good decisions, and it includes that, but it also includes understanding your constraints and your preferences — how you chose what your API design is, how we chose to use this framework, what your billing constraints are — all of these other pieces. So I use spec-driven development, but I think it is a subset of agentic engineering and/or context engineering. Specs are just a piece of it.

4. Tech debt is becoming deflationary

In this section:

• Matt opens with the historical default — entropy increases in codebases over time, and tech debt is what we call it.

• Guy inverts the frame: tech debt is deflationary. If the modification will be cheaper in six months, accumulating debt can be the rational move.

• The carve-outs are one-way doors — architecture debt and data debt still warrant the cycles to keep fresh.

• James’s college-era friend said a program could be rewritten from scratch three or four times and made better; agents make that cheap enough to do routinely.

• At Tessl the team is pushing to eliminate interactive coding sessions entirely — one-shot prompts off a well-formed Linear issue, with tests as the enduring artifact.

James Kaplan: Matt, I’m sure you have questions. Let me not hog the mic here.

Matt Linderman: Well, just to add on to the entropy point, Guy — to your point, it’s a really interesting question, because in the past, at least, what we’ve often seen is that entropy increases in code bases over time. And you end up with tech debt, or different words to describe that. Now there’s a real interesting opportunity to think about how do we keep code bases evergreen and actually avoid the entropy degradation over time as you go forward.

Matt Pocock gave an interesting talk on this the other day about basically always doing your architectural reviews on a more frequent basis, keeping the code base more up-to-date and clean. There’s an interesting evolution there that I think historically was a one-way ship, and now we may actually be able to steer that in a slightly different direction, to maintain code bases far better than we have in the past.

So I think the entropy one is a really fascinating thing to look into, and we’ll see how it all evolves.

Guy Podjarny: I agree that you can maintain, because labor has become cheap. So you can do all these things that before were nonsensical financially. Now suddenly maybe they are. At the same time, there’s another view that is almost counter to that a little bit, and that is that tech debt is becoming deflationary. Whatever modification you’re gonna do in your code right now, in six months’ time it’ll be easier to make that modification.

The agents will be more able to help you resolve that. To an extent, this is an amazing time to accumulate debt because it’s deflationary. It’s gonna be cheaper. So if there’s a good ROI in terms of you not bothering with this, then you can do — heck, you’d be able to rewrite the whole thing in a path. There are types of debt that you need to be careful still of — maybe architecture debt, maybe data debt.

Maybe things that are one-way doors that are very hard to change — for those you might wanna invest the extra cycles, which are now cheaper and reasonable to do, to constantly keep it fresh. And then there’s the type of debt where, if you go even the step beyond, you might actually care less. It might be fine, because you would just be able to undo that. So it’s not worth the delay. It’s always about what’s on the other side of the equation — it’s not worth the delay or the opportunity cost to accumulate it.

Matt Linderman: Yeah, 100%.

James Kaplan: In college, I had a friend who’s a very good computer scientist, computer programmer, who liked to say that a software program could be rewritten from scratch three or four times and made better. I think what you’re articulating, Guy, is that now it’s a hell of a lot easier to do that.

We can imagine everything gets rewritten multiple times with what we’ve learned in the process.

Guy Podjarny: Yeah, absolutely. You have to work and adapt to get to that point. One barrier to that, for instance, is interactive coding sessions. At Tessl, when we develop software, we aim — we’re not fully there yet, but we aim to eliminate as much as we can interactive coding sessions.

Instead, you can say, “Fine, play around with Claude or Codex or whatever it is, build the thing that you want, help yourself shape the product to what you want.” ’Cause oftentimes as you build, you figure it out. Now translate all of those into the Linear issue that provides the right information, throw whatever it is — the code you’ve just created in the prototype that doesn’t go anywhere. That gets thrown away.

But the information, the learning out of that, gets done. And then it gets one shot. And if it fails one shot, you modify the information, you provide the relevant commentary, and you create that again. What that does is it puts you in a place in which the agent is, almost by definition, sufficiently informed to be able to build that.

Of course you want to then curate that context over time so that it doesn’t rot, so that it remains relevant. But yes, you’d be able to build it again and make it adaptable. Increasingly, code generation should become like compilation: “It’s okay, I don’t care if there’s a new version. I can compile this for this new version of Linux or whatever it is that I just have here.” You do come back to some sort of regular principles, which is you need to capture tests. You need to capture some definition of what good looks like, what correct behavior looks like. And you’re never gonna test everything — definitely not with agents — but you need enough test coverage, otherwise you cannot scale.

5. Taste is a preference you forgot to write down

In this section:

• In real enterprises the factory layer is mostly aspirational; for now the live action is in context.

• “Negligent skills” — those without safety instructions — force enterprises into governance: central registries, supply-chain controls, dedup, versioning.

• Skills rot like software; the carrot is auto-extraction from agent logs and PRs, the stick is the maintenance burden.

• “Taste is just a preference you didn’t bother writing down” — skills are how that preference becomes a rule, used both at development time and at code review.

• Three tiers of evals — regression, skill, project (unit / functional / end-to-end) — and LLM-as-Judge is good enough to make them tractable.

Matt Linderman: So Guy, you started to introduce the stack — models, tools, context, skills, harness, factory — which kind of grows in abstraction as you go down. I’d love to hear from you. There’s been a lot of talk on the upper half of that stack, all the way down to harness. The factories piece is really emerging.

I’d love to hear just what you’re seeing in practice in terms of the workflows people are building around the harness, and what impact you’re seeing day-to-day with the folks you’re working with.

Guy Podjarny: For sure. The reality is there’s the sort of AI-native tiny companies, full kind of greenfield world, and then you go all the way to the enterprises.

In enterprises, the reality is that there’s a massive chasm — both between the companies, and within the company as they grow. When you think about factories and factory lines, those are, in almost all companies of medium size and up, not the norm. They are a specific prototype, specific project, specific sections — they’re the forerunners as opposed to the majority. The majority of interest, or activity, that we’re seeing right now is around context.

Eventually the agent executes this stuff. It might be malicious. It might be vulnerable — vulnerable being things like it guides the user to put API keys in plain text, or things like that. Or it might be what I’ve come to call negligent skills. Negligent skills are skills that lack safety instructions — “add this to the database, update is needed, do not drop the table, delete the database,” or some sort of basic safety instructions. Once you get into risky skills, you naturally need the governance: who’s installing, what do I even have in my inventory, do people use it. To control, so they create a central registry. They control those, and all that. So that’s one pier. It’s the least sexy, but it is important for supply chain security, and it’s a blocker to roll things out.

The second thing that we see is challenges around standardization, reuse.

I heard a story that articulated this very well — a unicorn with about 1,000 developers — describing how everybody was creating skills. That’s wasteful because everybody’s creating the skill.

They’re wasting tokens. They’re wasting time. They’re creating a lot of the same thing.

So they put together a repo to be able to upload and share those skills, so everybody’s sharing those skills. Very quickly it becomes a mess — there’s a whole pile of duplicates: which one do I choose? They had compatibility issues — like compilation — one developer is using one agent, they publish the skill, and it doesn’t work well on another agent. To be able to collaborate, you need some basic software-like tools.

You need some quality barometer, and a means of knowing that it’s quality, some deduplication to be able to identify those, some versioning of the stuff that you roll out.

All basic stuff that we have for software.

Most organizations are not yet past that point.

There’s a carrot and stick over here. There’s the fact that skills rot just like software. They will get out of date. They live in a dynamic environment.

The software around them changes, the practices change, the learnings change. You have to maintain them, per the debt conversation we just had. And that’s the stick — you better maintain them, otherwise they’ll break.

And then the carrot is: can I look at agent logs? Can I look at the PRs? Can I auto-extract things that will improve that? That comes along. So that’s the exciting bit. What we’re seeing is that in organizations they do that on a nascent project. They do it on things where the blast radius, if the agent misbehaves, is relatively controlled.

James Kaplan: Fascinated by the ability to use context to enforce or encourage engineering standards. And I’ll give a very simplistic example. In the history of software development, nobody has been worse at naming variables consistently than myself. I am horrible about it.

I am the worst person in the world at it. And it was really interesting — as I started playing with Cursor and Claude Code, it’s, oh, I can set up some rules that determine how variables should be named. That’s an incredibly simple example, but it translates to a million things in terms of architectural standards and non-functional requirements. And we can be a lot more precise about how we engineer code and structure code, compared to a set of guidelines we would give to a new engineer or a relatively early-tenure engineer. And that to me is pretty exciting.

Guy Podjarny: It requires you to do something that many people don’t like, which is take the time to sit down and write down what good looks like.

I think with software, oftentimes we just don’t do the hard thing.

The word taste is thrown around a lot in the world of AI. And while it’s important, taste is just a preference that you didn’t bother writing down.

Skills are a very good way to enforce that. There are technical constraints right now — skills don’t always activate. So what we see in practical terms is there are three parts that you need to do.

One: you need to create the skill and write it down.

Two: you need to make sure that it is distributed. You need to make sure that it’s installed in various cases. On the Tessl side we help with that — both the tracking and the mandating of skills. But you need to know the skill was present when it was needed.

Third: you need to invest in verifying that it’s been acted on.

The beauty of skills is that you can use the same skill in two agent contexts. You can use the skill as part of the development process to say, “Hey, this is available to the agent to load,” and you’re trying to entice it to use it.

But then you can use literally the same skill in the code review process to say, “Hey agent, check if these practices have been applied,” because you’ve written it down once.

And that is actually a beautiful thing, because you can do it in both cases. And you can even further go on and say, “Look, historically, I learned that now this data pattern is not good.”

Not only from here on do you change that, but — Matt, to your comment on tech debt — go back in history and find all the cases in which that’s there, and set up a mini migration of something that you wouldn’t have bothered doing.

Matt Linderman: I’d love to maybe stick on this topic of context, ’cause obviously you’re a real expert in this space. One of the — I think there’s a two-part question. One is, when you look at the evolution, obviously the tools themselves have gotten a lot better at pulling in context, but what are you finding in addition to the standard, like grep and code-based awareness, that’s really critical to pull in?

I’d love to hear your thoughts on that. And then the second piece, which maybe we can go to after, is how do you then experiment and measure what context is more or less effective, and what should you be pulling in? We have a number of clients asking those questions and trying to wrestle through what is the information that they should connect via MCP, how should it be structured, et cetera.

And that comes down to some version of experimentation. Would love to hear your thoughts on how to think through that.

Guy Podjarny: I love the question.

The context window is the scarce resource that we optimize for. Everything is context. Your code is context. It’s important to separate between reusable context and real-time context.

In the real-time context you have things like prompts, tool outputs, and things like that. Those are more interactive context. It’s important to do prompt engineering and things like that — those are still competencies.

The models clearly need to make it easier, and they have interactive modes about knowing when to ask you questions.

There’s passive and active context. Passive context being the code itself. So you can do things like keep the code clean, add proper documentation, add some passive documentation inline — like MD files at the right spots.

There are advantages to continuing with a thing that the agent has regenerated. There’s a real advantage to having the agent rewrite the file, because when it rewrites the file it tends to follow a certain pattern that matches the training data that exists.

Its future decisions reasoning about that file are more likely to be successful. Clearly, it’s not practical to rewrite every file every time, but it’s still a useful guideline.

The third bit, though, that relates to all of those, is this reusable context — and how do you evaluate it, or how do you know that it’s good? This is really where the world is now evolving. Reusable context is mostly done with skills.

Rules are a more forceful set — your Claude MD, your Agents MD — information that is always shoved in.

But you have to be very careful about how much you put in there. If you put a lot in there, you basically make the agent dumber for everything else that it does.

Skills are more on demand — whether the user has invoked them or the agent has the hints to do them.

Define what good looks like for a skill.

Right now the most common barometer is the Anthropic best practices. Does it have progressive disclosure? Is it sufficiently concise?

But we see customers modifying it to their own barometer. So at least start by defining what a good skill is in your organization.

Does it have safety instructions?

Does it refer to data privacy?

The second quality measure is tests.

In skills, the equivalent is evals. In evals you define a scenario: here’s an environment, here’s the files that are involved.

Pull from this commit, modify this context file like this, install these different tools, and then you have a task.

In practice, running tests is just a lot harder to maintain, and LLM-as-Judge is pretty good. It’s like an expert reviewer.

So you define the criteria, you run the agent through it.

Test coverage is notorious: you can build amazing test coverage and have really poor quality controls with it, just because you’ve created useless tests.

State-of-the-art, which not many people are doing, is you have three tiers of evals.

You have things that are more regression tests — just a few samples to say this works.

Mostly they now serve as running something even in the CI, so when you’re modifying the skill it doesn’t break, and you can have some sanity checks on new models and things like that.

Skill evals — you’re evaluating the skill, like a unit test or a library test. You’re evaluating that unit of context on its own.

The next one up is more like project evals.

In this case you’re doing something that evaluates the entire project context. You now have 20 skills installed, and a bunch of rules, and a bunch of files, and maybe you’re giving a bigger task.

Those are heavier to run. You’re not gonna run them on every PR, probably, but you might run them on a weekly basis, on a monthly basis, to see that your context remains fresh.

And then the comprehensive test — the end-to-end test equivalent.

I think of the first one as unit tests. I think of the second one as functional tests. And I think of the third one as end-to-end tests or integration tests — which are comprehensive, so you can make strategic decisions based on them.

So, can I switch models over here? Can I run this on a different environment? Typically cost- or efficiency-related, but for things that matter.

6. The bottleneck is no longer coding; it is learning.

In this section:

• Guy’s diagnosis of what enterprises miss: how much they can actually steer agents — the binary “accept the risk or don’t” framing is the wrong question.

• At Tessl, 20% of engineering is dedicated to the factory itself; Guy doesn’t think that’s an overinvestment.

• Matt: the change-management piece — getting “I engineer the system” past the early-adopter teams to the rest of the org — is harder than the proof of concept.

• 10X productivity in five years is plausible, but the right metric isn’t coding speed — it’s iteration speed and time-to-market. The new bottlenecks are marketing throughput and user attention.

• Closing coda on the CS-degree question: don’t push the kid into the degree, push the kid to build something. Software matters more than ever; the university is the doubtful vehicle.

James Kaplan: So based on your experience, when you talk with enterprise CIOs and CTOs — banks, pharma companies, what have you, manufacturing companies — what do they not get about the future of agentic engineering? What do you think most people working in enterprise IT organizations need to understand about how agentic engineering will evolve over the next few years?

Guy Podjarny: A good question. People are confused between these two perspectives that feel binary.

I think people underestimate how much they can steer and guide the agents — both to success and to control — and that requires investment.

I’ve seen a mistake, and now I cannot deal with this creature.

They’ve seen the wonder around the vibe coding, so it’s: no, you just need to let the agent be and let it roam free.

They don’t appreciate just how much they can control it.

I think the companies that are at the cutting edge — they spend a lot of time on how they get the agents to build right, on enabling the agents.

And people underappreciate the importance, so they delay embracing the agent because they think of it as this absolute — all they need to decide is whether to accept the risk or not accept the risk.

James Kaplan: They can actually control a fair bit of that risk if they manage those agents.

Guy Podjarny: We have — at Tessl, 20% of my engineering team is primarily dedicated to improving the factory, to agent enablement. And I don’t think that’s an overinvestment.

James Kaplan: Matt, this accords quite a bit with what I’ve heard you say about operational change in software engineering — that it’s not just a set of tools, but a set of tools that fit into a broader system.

Matt Linderman: Yeah, to build on what you’re saying a little, Guy, we do a number of coaching initiatives with different folks, helping folks understand how do you move toward a more modern engineering stack. And one of the biggest things is actually more of a mindset shift: I’m not using a bunch of agents to just generate code, but I’m actually responsible for improving the factory, to use your word.

If I’m not getting exactly what I want, how do I go back into the skill, or the series of skills stitched together into some workflow, and re-engineer it in a way that gets me closer to that? And then that brings people, once you have that mindset, into all sorts of directions. How do we do the architecture, the context, better?

How do we feed InfoSec better into the different agents, so that they bring it in from the first time around? That simple mindset shift — from “I’m using the tools that I’m given” to “I am actually engineering the system that is then generating, and my job is to make that system better over time” — has been a massive shift.

You get a few teams usually that figure it out first. They actually are the ones building the agents, the workflows, et cetera. But what we found is if you can then get that mindset amongst the rest of the engineers, even if they’re already using what other folks have built as the baseline, they can then adapt it, they can shape it to work in their specific parts of the code base, and the types of work they’re doing.

But what’s quite interesting, at least from our point of view, is that a large share of that challenge is the change management piece that comes after defining and showing that it can work. Then it’s: how do you actually get everyone else working in that same way? And a lot of that’s the mindset shift that I was talking through.

Guy Podjarny: Yeah, no, I fully agree with that.

Look, it’s hard because — it is a change in the craft of what you’re operating. It’s a change in the types of mistakes that can happen.

With self-driving cars, when they make mistakes that are nonsensical, that a human would never make, people get actually mad. When there’s a leaf in the middle of the road and it thinks it’s a person and it would not continue, people are actually upset about it.

And I think we’re seeing things like that. There’s a new type of error that happens, and it’s hard for people to acknowledge.

You basically have this combo of: you see a problem, and you’re told, correctly, “Don’t fix the problem — don’t fix the symptom. Go upstream and get the agent to fix the problem.” Which loses a bunch of their craft. And is a new type of work that they might not want to do.

So yeah, a lot of this is culture change. I’m sufficiently a gray beard to have gone through the DevOps transformation. And DevSecOps, and the movement of security responsibilities.

These things are unsettling, and it doesn’t help that agents are happening maybe ten times faster.

It is a big change. So a lot of cultural — it always comes down to people.

James Kaplan: So for the companies that implement the model successfully — the operating model — is this a doubling of engineering throughput, a tripling, 50% improvement? Think five years down the road — what do you think the companies that enthusiastically embrace agentic engineering will be able to achieve?

I think it’s an order of magnitude. 10X.

Guy Podjarny: And potentially more. Not — again, maybe a bit of a common mistake — it’s not about the speed of coding.

James Kaplan: Of course.

Guy Podjarny: It’s the speed of iteration.

When you launch a product, you still should work iteratively. You still should build a minimal product so you get it out there and get people to validate it.

The amount of time it takes you to get a user, to get them to try the product, to give you the feedback and internalize it — those are all still relatively fixed.

You can only lightly optimize those.

But the type of product you can provide to them now can be a lot more comprehensive. You can bring them a product that is actually a lot more thought through.

Your ability to analyze and apply learnings from whatever it is that they did with the product is a lot faster.

And the number of people you need involved in each one of these iterations is a lot smaller, so you can learn more in parallel.

There are enormous opportunities to improve, and they compound. That sort of 2X improvement pace implies that if you’re at it and you’re progressing, you’re gonna be way ahead, ’cause of your pace of learning.

James Kaplan: You’re getting down the learning curve.

Guy Podjarny: It’s interesting to identify the new bottlenecks.

At Tessl we generally work in pairs most of the time.

That’s more because of organizational resilience. If someone’s on vacation or somehow cannot do it, then the other person can continue the work. They collaborate.

It’s mostly independently, and then they review each other’s work, so someone can step in a little bit more easily.

We’ve had a problem that we’re still working on fixing, which is product marketing is struggling to keep up with the pace of new capabilities that we have.

The answer is agents all the way down. You need to build more agentic analysis of what got built, move a few of the decision documents that happened earlier on, to be able to produce marketing material in parallel.

Once you have that, there’s still a scarce resource that we need to understand, which is the attention span of our users. You can’t email them ten times as many emails. You still have to send them a confined amount.

So it’s interesting to understand what the limiting factors are, what the new constraints are. And alignment is one of those.

But within each of those departments, more empowerment, more autonomy, less dependencies is a critical movement, because the cost of alignment relative to the cost of building is so much higher.

James Kaplan: Five years, 10X improvement in productivity via agentic engineering at best enterprises. You agree with that? More or less — what’s your view?

Matt Linderman: I think we need two things. One: it depends on the metric you look at, but I would say 10X order of magnitude makes a lot of sense. It could even be higher. If you look at your throughput metrics, the historical way of engineering — there are organizations getting five, 10X already, and they’re trying to now figure out how do you get that across the organization.

But I think that’s a bit misleading. To your point, Guy, really the metric to be solving for is more of a time-to-market view. If you look at the bottlenecks — it moves to your code review, it moves to your product management being able to build up the right requirements, it moves to product marketing, et cetera.

So really, I think there’s a measure of: one, if it’s a new product, how fast can you get to market? And then for existing products, how fast can you cycle through to get customer input, figure out what to build next, then go build it, go get more input, et cetera. I think that for sure can accelerate 10-plus X more.

But it requires real process change. You have to think, “Okay, how do I actually go get that customer input?” It used to be releasing it to actual customers, and then we went to alphas. Now can you even have customers on your team who can just test it and give you feedback every single day at 3:00 PM?

James Kaplan: Before I release something, before I write something, I have a panel of 500 virtual CIOs and CTOs who give me feedback. I often go through five or six rounds, and they help contain some of my literary excesses. Guy, final short question for both you and for Matt — would you advise a young person today to major in computer science if he or she were so interested?

Guy Podjarny: I would not advise someone to go do a computer science degree — not because — I think software development as a profession with some modifications will continue to live, and we will build software.

James Kaplan: And I think software will matter more than ever. I have a lot less faith that the universities would be the route — that the universities will be able to adapt. Your trepidation is about the degree, not about the computer science part.

Guy Podjarny: Yeah, exactly. Go build something. Go create a product. And I also feel like this is a world in which a breadth of perspective will come a long way.

Learning how to be a bit of a one-person army goes a long way — around touching product, touching marketing, touching your subject domain.

James Kaplan: Matt, computer science — 22-year-old or 18-year-old. Assuming that person will go to university, computer science or study something else?

Matt Linderman: Maybe building on what you mentioned, Guy — there’s never been a better time to build your own company. Going and doing that at some point will teach you far more of a breadth of experiences. Now, if you do go, I do think there’s really a skill around problem-solving, conceptual problem-solving, that is going to be applicable in any job that you have, in communications, et cetera.

That may come from engineering degrees, you could say math degrees, et cetera. But I would really be looking for something that pushes you in terms of how do you think, how do you structure problems, et cetera — that you can then bring into whatever type of work you’re doing moving forward.

And then go into the workforce and learn, as you said — entrepreneurship or not.

James Kaplan: Thank you so much. It was a great discussion.

Guy Podjarny: Thank you.

Matt Linderman: Thank you everyone.

7. What would Biddle say about agentic software engineering?

No single element of the modern system of force employment wins a battle—not combined arms, not suppression, not decentralized initiative. The doctrine wins.

The same is true of Podjarny’s stack: models, tools, context, harnesses, factories. None of them buy you anything in isolation. The factory is the doctrine.

And Biddle’s harder lesson—that adopting the modern system requires trusting non-commissioned officers with judgment they were not historically given—is the agentic problem in another voice. Just as you have to trust NCOs with fire teams, you have to trust engineers with agents.

The companies that learn it will not be 10X more productive in any narrow sense. They will be 10X faster at learning — which is perhaps the discriminant between victory and defeat in competitive markets.

1. Every computer system we build is an imperfect representation of reality

“No, I didn’t go to Catholic school. Why do you ask?”

An eminent partner at a previous consultancy looked at me across an empty conference room and explained: “You have an instinct for mapping data onto a grid -- you have a Cartesian mindset, which many good consultants learn from a Jesuit education.”

Grids distort as much as they reveal. We’ve all seen (or built) grids for evaluating outsourcing vendors: 20 points for stability, 30 for service quality, 30 for cost, 10 for innovation. These never produce the right answer -- they ignore non-linearity. You won’t select a vendor below your financial stability threshold no matter how cheap; once a bidder clears it, you care about cost and service, not financial stability.

Every slide, every analysis, every system is an imperfect representation of reality. CRM systems are a simplified model of how your business interacts with customers. Operational systems are a simplified model of your factory. I learned this crawling around telecom carriers in the 1990s, where you had to ask the service techs which USOC “1FR” lines had bridge taps that had to be removed before upgrading to DSL.

2. The world is entropic

Do you know whether that deal will close next week and at what price? Whether your flight to Chicago will land on time? Or whether a new employee will turn out to be a hero or a goat? I do not.

As a recovering history major, I believe the past provides counsel on the future. On June 21, 1948, the Small-Scale Experimental Machine became the first computer to run a program stored in its own electronic memory -- 128 bytes of it. Weeks later, Weaver and Shannon published articles foreshadowing many of the problems we struggle with today: automating Byzantine business processes, capturing latent or ambiguous data.

Shannon defined entropy as average amount of uncertainty in a probability distribution. [1] Maybe that’s okay -- in a world without entropy no business could exceed the risk-free rate of return. The more entropic a domain, the more information required, and the more expensive the computer system to automate it.

Weaver laid out a relationship between complexity and uncertainty. Some problems are simple: hold all variables constant except the dependent and independent ones, and you get the steam engine, the automobile, the telephone. A payroll system is the modern analogue -- fixed variables, describable rules, few feedback loops.

Starting in the late nineteenth century, scientists attacked problems of disorganized complexity. Statistics could predict the frequency of calls in a telephone exchange or the claims paid by a life insurance company, even when individual causes remained opaque -- provided each data point was atomic. Fair Isaac developed the FICO score on this basis in the late 1950s.

Then Weaver addressed problems of organized complexity -- macroeconomic management, ecology -- where many inter-related factors multiply uncertainty because each may influence the others in non-obvious ways. Neither nineteenth-century analytics nor twentieth-century statistics could handle them. He hoped new computing devices might, and pointed to the operations research developed in the Battle of the Atlantic as a way forward.

Problems of organized complexity abound in modern business -- order-to-cash, pharmaceutical manufacturing. Given enough contracts, even payroll can turn into one. [2] How much time do knowledge workers spend massaging data into or out of enterprise systems because the system couldn’t capture every exception the domain implied?

We should forgive Weaver for not foreseeing the organized complexity required to address organized complexity -- ERP programs suffer their own, and enterprise technology functions suffer from it themselves.

Writing with Shannon, Weaver named another contributor: latent meaning. Ask an account executive the status of a potential sale and she might say “We’re in final price negotiations with the CHRO and he would like to get a deal signed within the month, but the general counsel is hacked off about a couple of legal terms we require and the CHRO heard he is going to call the CEO and ask that they reopen discussions with another bidder.” What is she supposed to select among the four CRM choices for “status” -- RFP open, RFP submitted, financial negotiations or closed?

Weaver and Shannon identified one more problem -- whether a system creates the desired behavior in the people who interact with it. Think about computer systems the way the military thinks about weapons systems: as inclusive of both the technology and the user. Design choices affect user behavior in unpredictable ways. Complicated interfaces cause users to resist entering data, or to prejudice what they enter to make themselves look good.

Calendaring is deceptively entropic. If Suzy, my assistant, asks me about a conflict at 10 am on a Tuesday, I might say: “I’ll go to the meeting with client 1, because I think that colleague A can cover me in the meeting at client 2. And don’t decline or mark me tentative for anything, because I don’t want anyone giving away the time slot because I might not be there.” I don’t verbalize dozens of assessments about people and situations that shape my decisions. You have entropic communication about organized complexity, latent information and adaptive behavior all in a couple of sentences!

3. Entropy creates cost and frustration when we build systems to represent it

How much time do we all spend in conference rooms debating how much granularity the data model should have -- and how we balance fidelity to the business process versus the cost of maintaining data? How frequently do you find the correct state for a transaction not in the fields designed to contain it, but in the free-text notes or the accompanying email? And how much user anger derives either from the endless fields (specified to capture the subtleties of state) or the disconnect between what the system says and the reality they observe?

Put another way: how much time do we devote to designing around entropy when we build systems? And how much frustration do we create because of the choices we make in doing so?

Operational systems work best when the world can be reduced to stable categories, deterministic workflows, and enumerable exceptions -- all these make a domain easy to formalize. Nobody complains much about most payroll systems because they apply a bounded number of explicit rules to available data.

Analytic systems work best with huge sample sizes and a few relevant independent variables -- making the entropy here tractable to statistical analysis. Machine learning systems for pricing consumer products work because massive amounts of structured data allow probabilistic inference where humans might have struggled to discern patterns.

Other domains and use cases have more intractable entropy. Systems to support treasury management for large enterprises? Modeling a thousand-page contract, involving hundreds of legal entities, dozens of jurisdictions and a thousand different exceptions you can have for a transaction -- all of which generate entropy. Which messiness must you model and which can you simplify away? CIOs have exceeded budget, blown deadlines and angered users in seeking to answer that question.

Yes, machine learning has blunted the impact of entropy in some cases by sniffing out the relationships among variables, but often the sample size is too small and the data too messy for traditional machine learning to be effective. Years ago, someone said collections managers could offer definitive recommendations about how to reduce losses -- why couldn’t CISOs do the same when talking about how to protect the business against cyberattack?

I tried to explain that each demographic segment included millions of households. You could experiment with a new script and quickly determine whether it increased or decreased promises-to-pay. Any large company’s technology environment is a snowflake -- even two companies of similar size and in the same sector may have radically different technology environments. Vulnerability depends not on individual decisions, but on how you connect all the pieces in your environment. And you might not know whether you had been breached for years, if ever. [3]

4. GenAI-based agentic systems process entropy

Deterministic systems cannot process entropy. They can rely on humans to pre-process it for them -- as happens when product managers and engineers sit in a conference room debating how to capture the data required to automate an ambiguous process. They push it on to users like account managers who must fill in endless fields about their pipeline -- and still wonder which option they should select to describe deal status. Or they can store it inertly, like the free-text notes that exist in some customer service platforms. All of these imply some combination of low efficiency and lower effectiveness.

When large language models process text, they convert tokens into vectors, points in a high-dimensional space. This encodes meaning through proximity -- similar meanings cluster together. Large language models can do things deterministic systems cannot do or do poorly:

• Allow inference across stored entropy: Written and spoken language are entropic. Vector embeddings allow LLM-based systems to query and analyze free-text notes, email threads and meeting transcripts.

• Identify implicit relationships: Vector geometry represents connections between concepts, such as the general counsel wanting to block the deal for group health insurance because he disliked the contract terms.

• Understand gradations rather than discrete states: A deterministic system requires the account executive to choose whether the deal has advanced to final negotiations or not. A vector representation of the same situation can sit between two states.

GenAI-based agentic [4] systems give us new ways to process entropy.

• At design time, by using software engineering agents to create business logic that reflects all the organized complexity that Weaver described.

• As part of the user interface, to mediate between the entropy of written language and a deterministic system, rather than forcing the user to do the work.

• Via data ingestion, to derive insights from the massive stores of entropic, unstructured data every enterprise has sitting on its servers.

• At run-time (either with or without a human in the loop), to make decisions and execute transactions non-deterministically.

We can also store data used and produced by agentic systems in different ways. Traditional relational databases excel at storing and retrieving massive amounts of transactional data at speed and with near-perfect reliability. They struggle with more ambiguous data, with dense relationships among all the elements. [5] We can choose to store this data either in vector databases or in knowledge graphs.

5. You have choices about how and where you use agents to process entropy

So much of the agentic discourse we see mirrors the flat Cartesian mindset captured in the grid used to evaluate bidders for outsourcing deals. Agents are risky! Or: everything will be an agent!

But entropy is, well, entropic. It varies in scale and shape from business domain to business domain. Sometimes it takes the form of a dense web of interconnections among idiosyncratic products, contracts, processes or customer relationships. Sometimes it takes the form of latent data that doesn’t fit into any data structure you might define. Not a few business domains suffer from multiple forms of entropy.

GenAI-based agents also have disadvantages compared to deterministic systems. GPU-based inferencing is slower and more expensive than CPU-based processing -- a single agentic transaction often costs cents and adds hundreds of milliseconds, where the equivalent deterministic transaction costs fractions of a cent and resolves in single-digit milliseconds. Non-deterministic systems are only as good as the context they receive. An agentic system can process the free-text notes in a CRM platform. It won’t know anything about the discussion two account executives had in the car on the drive back from the customer site. [6]

Abjure the false binaries -- you have real choices about how and where in the value chain you process entropy.

• Agentic software engineering: This will be relevant for almost any systems effort. It will allow you to automate all the business logic required to reflect organized complexity with more speed and reliability and less cost. And after agents help you develop the code, you can still apply all the quality assurance mechanisms you have developed over decades, just as you would code developed by hand.

• User interface: Retain traditional interfaces for situations where users have to enter small volumes of easily understood data. Develop agentic- or chat-based interfaces for order entry, CRM, or transaction processing systems where users complain about having to fill in screen after screen of data. You can always use a combination of deterministic rules and user validation to ensure the agent correctly captures user intent.

• Data ingestion: This is one of the most powerful and most underestimated capabilities. All companies receive and store vast amounts of valuable but entropic data -- eighty percent of corporate data is unstructured and even structured data can be fragmented and hard to correlate. Customer requests for quotation? Customer service notes? Legal contracts. Agentic capabilities can ingest all of this either to support operational processing or to generate new business insights, even if many companies have not focused here yet. [7]

• Run-time: This is complicated and has the highest stakes. Use agentic patterns for low-volume heterogeneous decisions. I am helping technology organizations use agents to compare project designs with technology standards. The agent will compare two artifacts (a project design and a standards document) that describe organized complexity in natural language (which is inherently entropic) and determine whether one conforms to the other. Hybrid patterns will probably become increasingly common, with deterministic cores and agents to handle edge cases.

• Data storage: Naturally, relational databases will continue to store structured and transactional information. Knowledge graphs excel in modeling product portfolios, customer relationships, process maps and other domains where relationship structure carries meaning. Vector search enables retrieval by meaning from notes, documents and transcripts.

6. Precision matters because entropy is, well, entropic

I have always hated the formulation “The technology is easy. The governance is hard. The organization is hard. The line at the cafeteria is hard.” Here, design choices will shape cost structure, risk posture and user experience for years.

With a capability to process entropy at scale, you have to think about where the entropy sits in your business system -- even a simple one.

Managing invites for the Technology Leadership Forum is lower-entropy than a hundred-million-dollar group insurance deal, but there is plenty of “Joe really wants to attend, but can’t, depending on when a personal conflict lands -- and wants to know if Sally can possibly attend in her place.” I put invitees and members into a knowledge graph and built Agent Serena to translate the entropy in my email into the organized complexity of that graph. [8]

Taking a more highly-scaled domain: entropy pervades enterprise change management in banking. Dozens of interconnected process steps and artifacts; latent information about why a milestone might be at risk of slipping and under what circumstances. Traditionally, banks managed it with managerial talent copying data among word processing, spreadsheet and presentation files. And many, many emails. The result: frustration, expense, and less insight into major programs than anyone would like.

Banks can use agentic software engineering to build a system whose agents ingest program documents, extract the relevant risks, issues, decisions and action items, and store them in a knowledge graph with the relationships among them. Which risks affect which business lines? Which work-stream lead owns which decision? Further agents interrogate emails and videoconference transcripts to enrich the graph. Vector descriptions let program managers search by meaning, not keywords. Still more agents read the graph to assess risks and surface opportunities to improve the program.

This combination -- agents, knowledge graph, vector database -- removes the entropy from a painful business process. It reduces cost and improves transparency. Those of us on team cyborg (rather than team android) will note that it also empowers managers: less toil, better information.

Agentic systems -- with knowledge graphs and vector databases -- can be transformative. When I wrote that global business must move beyond organized factories and chaotic offices, this is what I meant. The benefits are both economic and humanistic. Nobody grew up hoping to spend days pasting data from email to spreadsheets and back.

Precision matters here. Agents are not magic. Inferencing is expensive, latency is real, and new vulnerabilities arrive faster than we can grasp them. But the harder discipline is the one Weaver and Shannon left us: see your business as a system, find where the entropy sits, and choose where to process it. A flat, Cartesian grid won’t help.

Footnotes

[1] This differs from standard deviation, which measures the magnitude of the spread around a mean. Shannon entropy measures the unpredictability of a distribution. Relevant to us: you can measure the Shannon entropy over a set of non-quantitative values just as you can over a set of quantitative ones. In addition, there are more and less expansive definitions of entropy. I am using a relatively expansive one here.

[2] CityTime is a great example of Kolmogorov complexity. The shortest possible description of the rules governing what every city employee gets paid is approximately as long as the rules themselves — decades of negotiated union contracts, grandfathered provisions, and exceptions to exceptions, each one load-bearing. You cannot compress it further without losing something that will eventually matter. Automation doesn’t reduce this complexity; it just moves it from humans who held it in their heads to engineers who must encode it in systems.

[3] My interlocutor said I was over-complicating the situation and CISOs were probably just incompetent. Sigh.

[4] Agents predate widespread adoption of large language models and genAI. By some definitions, the cruise control system in your car is an agent. You set a speed. It monitors the speed, accelerating when the car drops below that speed (because you are going up a hill) and easing off the throttle when you exceed it. Of course calling LLMs makes agents infinitely more capable than the cruise control in your car.

[5] The notion that relational databases might struggle with relationships among data elements might puzzle some. But only those who have never stared at a screen in the early morning trying to make sense of an outer join -- or who developed a better command of SQL than I ever had.

[6] I like the idea of decision traces and context graphs, but we should be realistic about which decision traces we can capture and which we cannot, unless we want to build a panopticon for ourselves.

[7] I stared in disbelief the first time Zubin Ghafari showed me how he used GenAI to integrate messy CMDB data with other telemetry information. I had assumed this would have taken him weeks.

[8] The natural language front end to Prosaic Graff was a life-saver, but the latency sometimes made me want to put my fist through the screen. So I created Python scripts I could run (instantly) from the Terminal that told me how many members planned to attend or the status of any individual member.

Back when I did the research that turned into Beyond Cybersecurity, several people asked “does securing the software really matter if we can’t trust the chip?” or “wouldn’t it be great if the chip could be part of the security architecture?”

A highly capable and motivated attacker could compromise firmware supporting the most sensitive workloads, making security controls at higher levels of the stack ornamental. While the US military’s Orange Book (1983) first formalized the logical boundary of a Trusted Computing Base (TCB), its multi-user specifications assumed the underlying physical hardware was secure inside a guarded facility. Distributed architectures and modern threat models shattered that assumption, forcing the industry to move that TCB boundary down into the architecture of the chip itself.

The Trusted Computing Platform Alliance (TCPA) — which became the Trusted Computing Group (TCG) — developed the idea of a Root of Trust, with three pillars:

• Root of Trust for Measurement (RTM): The initial, immutable piece of code (usually embedded in the motherboard ROM or BIOS) that boots first and measures (hashes) the next component before handing over execution.

• Root of Trust for Storage (RTS): A secure memory zone within the chip that holds the cryptographic keys and the accumulated measurements. Software cannot overwrite or read these keys directly.

• Root of Trust for Reporting (RTR): The cryptographic engine inside the chip that signs those measurements using an endorsement key burned into the silicon at the factory, proving to an outside party exactly what state the machine booted into.

As of 2013, some devices had a Trusted Platform Module 1.2 (TPM), a distinct, low-cost microcontroller soldered onto the motherboard, completely isolated from the main CPU lines — but its separation from the main CPU both introduced bottlenecks and limited its security efficacy. Some data center managers declined to turn on this functionality because they feared the operational complexity it required could cause outages.

TPM 2.0 improved on 1.2. It embraced cryptographic agility and it moved the silicon root of trust closer to the CPU, often embedding it as an isolated firmware routine (fTPM) directly inside the processor chipset, eliminating the exposed external motherboard bus. To fix the datacenter uptime issues where a benign driver update would brick a server boot, it supported Policy-Based Authorizations.

But TPM 2.0 is an authentication and state-verification mechanism. It can verify that a system started up in a clean state, but it cannot protect data from being stolen while in memory.

To address these problems the Linux Foundation birthed the Confidential Computing Consortium, which published A Technical Analysis of Confidential Computing. This document established the non-negotiable baseline that unauthorized entities could not view or alter data or code running in a Trusted Execution Environment. This required chip manufacturers to redesign how processors handle memory management, CPU registers, and privilege rings. They couldn’t just build a “secure sandbox”—they had to modify the fundamental instruction set architecture (ISA) so the CPU could treat its own operating system or hypervisor as a potential threat.

Today, most enterprise-grade server and device chips have Confidential Compute capabilities, and companies are starting to use them. How do they use confidential compute to enhance the security of agentic systems without writing systems-level code to manage CPU instructions, page tables, and hardware register states?

A discussion with Opaque Systems CEO Aaron Fulkerson

Opaque Systems aspires to bridge the gap between hardware primitives and the enterprise application layer, by providing software engineers with access to policy enforcement, orchestration and cryptographic proof.

CEO Aaron Fulkerson talked about what this all means as enterprises seek to implement AI-based systems. Prosaic Times remains studiously neutral on particular technologies, but hearing from builders in their own words is instructive, and we always respect people who are passionate about the products they develop.

Here are a few of the key takeaways:

Prompt engineering and code scanning are probabilistic mitigations of a problem that requires deterministic enforcement. Confidential AI uses the encryption key baked into modern CPUs and GPUs to do that enforcement cryptographically: what ran, where it ran, what rules applied, with a signed report any auditor can verify. The hardware capability has been sitting in silicon for a decade.

Regulators are starting to ask about runtime enforcement, not just configuration. The EU AI Act, emerging regulations in Asia and the Middle East, and likely HIPAA are converging on a requirement that isn’t “we configured this policy” but “we can cryptographically prove this policy was enforced at runtime.” That is precisely the proof confidential AI produces. The regulatory hook is probably what forces enterprise adoption ahead of any internal security posture. Design for sovereign cloud will only accelerate this.

The policy standards landscape is a fragmented patchwork waiting for consolidation. Anthropic ships mutually-attested TLS. Microsoft has its own flavor. Google, Meta, TikTok, ServiceNow each ship variants. None of it is portable across hosts. A consortium effort building on SPIFFE, EATS, and RATS is producing a superset specification where the policy travels with the data.

James Kaplan: This is James Kaplan with another Prosaic Times podcast. We have one of the most interesting people in the cybersecurity space, Aaron Fulkerson of Opaque. Aaron, welcome.

Aaron Fulkerson: Hi, James. It’s a pleasure to see you again.

James Kaplan: Why don’t we start out by talking a little bit about your journey? Let’s hear about your career and what brought you to the place where you’re at today.

Aaron Fulkerson: Sure. Opaque Systems spun out of UC Berkeley — the same lab that created a lot of the foundational AI infrastructure: Spark, Ray, vLLM. The team that runs the lab — folks like Ion Stoica and Raluca Ada Popa — are world-renowned, nerd-famous folks. I was approached by them while I was running a business unit at ServiceNow, and I was intrigued by their background building the essential foundational technologies that everybody uses.

Ion Stoica being the founder of Databricks certainly got my attention. When I spoke with them, what they helped me understand was that they had created a new technology called Confidential AI that allowed regulated enterprises, governments, or sovereign entities to bring AI to their most sensitive systems and data, with verifiable guarantees about what ran, where it ran, and what rules were enforced.

That’s now become a whole category that Gartner writes about and that all the silicon vendors, hyperscalers, and everybody else covers. It’s been an exciting ride over these last two years.

From confidential computing to confidential AI

James Kaplan: All right. Why don’t we step back? Tell us a little bit about the history of confidential computing — what it is, and why it’s important.

Aaron Fulkerson: Yeah. It’s been around maybe ten years, so it’s been around a long time. What it means is that the hardware — a CPU or GPU — has an encryption key baked into the silicon, and that enables two key capabilities. One is what’s called a trusted execution environment, or an encrypted runtime: you can run your workloads on data while encrypted. That’s what everybody thinks about when they think about confidential computing. There’s another capability of confidential computing that’s at least as important, if not more — the ability to do what’s called hardware attestation with that encryption key. What that means is that you can cryptographically sign software.

Now, confidential computing is a hardware layer that’s available on all hyperscalers. All modern CPUs and GPUs have these capabilities. It’s not specialized hardware. What this enables is what we call confidential AI — a software layer above the hardware, above Kubernetes — and that means you can have hardware enforcement of what ran, where it ran, and what rules were enforced at runtime.

I don’t need to tell you, James, or anybody who’s an AI builder, that you cannot enforce rules on AI agents at a software level. There’s a greater than 0% chance — at least 1%, typically higher — that these things are going to circumvent your software rules.

So you have to do this hardware enforcement, and the good news is this technology that’s been around for a decade has found a killer application: confidential AI.

James Kaplan: Tell us a little bit about that. Tell us why software enforcement doesn’t work for agents and why hardware enforcement does.

Aaron Fulkerson: Sure. AI systems are non-deterministic, so they behave in ways that are unpredictable. There are a couple of things here. There are adversarial attacks, which is what everybody immediately thinks about when you talk about an encrypted runtime — but adversarial attacks are far less of a threat than the non-adversarial ones.

So what’s a non-adversarial attack? These systems, designed the way they are with software enforcement, bleed sensitive data. Opaque, in collaboration with our partners at UC Berkeley and a bunch of ISVs and large systems integrators, has developed what we call an AI surface map for data bleeds. It catalogs 46 different ways these systems bleed data by design, simply because they haven’t been hardware-enforced. They will circumvent your software guardrails by design, and the only way to enforce the rules is cryptographically — hardware-enforced, as I described. It’s fundamentally different from legacy enterprise software.

We’ve shifted from a hardware compute layer to a software compute layer. As these agents build a context window, calling your most sensitive data, they pass all that context around to do software analysis — probability, right? That’s how language models work: an analysis that says, “Oh, here’s the highest-probability next one.”

That context lands in application performance monitoring packages like Splunk or Datadog. Your CloudOps or DevOps teams doing memory dumps may be able to see stock trades, maybe executive compensation — anything that passed through.

There are a whole lot of different ways this category of system, AI systems, bleed your data if you don’t upgrade the trust layer. The same way we had to upgrade the trust layer with the transition to the internet and encrypt data in transit, we have to upgrade the trust layer for these AI systems — or you’re just going to bleed your most sensitive information everywhere.

James Kaplan: Give us the use case or the example.

Aaron Fulkerson: I’ll give you some examples. We have customers like ServiceNow, Encore Capital, other financial services firms, and, in Thailand, SCBX. What they do is use confidential AI. They deploy what’s best thought of as an AI module, because they aren’t going to shift all their workloads onto confidential hardware in the cloud yet — they’ve already built a lot of their systems.

They’ll start with, “Hey, we’re doing some really sensitive data processing — stock trades, HR information, or a CFO tool for summarization and analysis.” They’ll deploy that, and using confidential AI, they’ll create policy enforcement at the infrastructure level. For example: it must run inside my virtual private cloud. It can only communicate over this subnet. All the way out to the agent graph, this agent can only speak with these tools. All of that gets rolled up into a strong, cryptographically enforced identity, so that before anybody sends data to a workload — I’m using “workload” generically here, to mean any AI workload or agent — they can inspect the signatures and the cryptographic enforcement.

Does it run inside my virtual private cloud? Does it have the data-ingress and data-egress policies I require? Does it use my identity? Whatever policies you want get hard-coded into the identity before you send the data, so you can verify the integrity of the system. When you send the data, it’s sent encrypted. The policies are verifiably enforced at the hardware level. And most importantly, James, you get proof. You get a report with cryptographic signatures that any internal or external auditor can check, proving exactly what ran, where it ran, and what rules were enforced.

Why agents require new types of guardrails

James Kaplan: Let me play this back to make sure I understand it. In the old days — say two years ago — if we were building an application and we had a set of rules about which tools that application could call, or where it could send data, we would scan code against a set of policies and see if it violated those rules.

And if it violated the set of rules, we’d send it back to the engineer and tell them to go fix things, right? Then we’d rescan, and we’d be fairly confident it wasn’t going to violate that set of rules because we’d scanned the code. So far, so good? Now, if we’re building an agent, the agent operates non-deterministically.

So we can’t know with certainty whether that agent will call one set of tools or not, and the guardrails we’ve placed around it… When we’ve tried to use prompting or other controls to limit the purview of that agent, sometimes that works, but you can’t say with certainty that it’s going to work.

However, if we build the enforcement of those rules into the hardware layer, then we can operate with certainty. Is that a fair summary?

Aaron Fulkerson: That’s exactly correct. You’re talking about the non-determinism of these systems — you’re exactly correct. Let’s say there’s a 1% probability that one of these agents bleeds data, which we know for a fact is at least 1%, probably greater. At 1%, maybe that’s okay if it’s a non-essential system or data set.

However, scale that to 100 agents and the math gets pretty frightening — you’re now at something like 40-something percent.

At 1,000 agents interoperating and communicating — which may seem like a lot, but I’m running thirty-something agents in the background on my desktop right now while I’m speaking to you — you’re at a 99.9% probability of bleeding data.

James Kaplan: And there’s plenty of workloads where one percent is unacceptable.

Aaron Fulkerson: Correct. Absolutely.

I was speaking with some industry analysts just last week, and one of them was asking, “Is it really that big of a deal? We’ve always had people who leak in elevators and hallways.” (A quick fact-check on my earlier math, by the way: at 100 agents, it’s 63% probability.) The analyst — from a big firm everybody knows — was asking, “Does it really matter? We have employees who talk in the elevator or the hallway, and they’re bleeding data at similar probabilities.” The response I gave, which their colleague already understood, was, “It does matter, because what an AI agent can achieve in a span of seconds or minutes vastly exceeds what a group of organized humans, even malicious ones, could achieve in a year.”

These things are operating at machine speed with human-like capability to reason, decide, and act. That’s quite different from an employee in the elevator accidentally leaking some competitive information. We’re talking about systems that can do damage within seconds or minutes that bad actors might not be able to achieve in a year. And furthermore, the leaked data — it used to be hard to sift through. Somebody leaks some data, is it really that bad? Who’s going to utilize that data? AI will.

James Kaplan: Yeah, there’s also an autonomous-vehicle analog here. You could probably make the case that an autonomous vehicle is safer than a vehicle operated by a human. But there’s just more cultural fear and nervousness around an accident caused by an autonomous vehicle than an accident caused by a human driver.

As we seek to advance agentic adoption, a problem involving an agent and exfiltration of data is going to be culturally more challenging than somebody talking in an elevator. So yes — I agree with everything you said, plus the bar is probably higher.

Aaron Fulkerson: Yeah, for sure. I think that’s true. To pull on that autonomous-vehicle analogy: hey, one bad driver can cause a lot of damage. But if you had a swarm of autonomous vehicles that could self-replicate on the fly, driving all over the freeway at a thousand times the speed — which is effectively what these agents are doing, compared to human drivers — don’t you think you should have some additional regulation and enforcement that’s verifiable in a deterministic way, rather than by probability or just rolling the dice?

James Kaplan: So let me ask a devil’s-advocate type question here — or maybe not a devil’s-advocate question, more an architectural-alternative type question. I saw a really interesting paper about how you enforce controls on agentic systems. The paper suggested that maybe the right way to do it is, instead of having the agentic systems operate directly, you have them generate procedural code in real time, which you can then scan against a set of policies.

Obviously that injects latency. But if you were an enterprise architect, how would you think about the option of using confidential AI versus the option of having an agent write procedural code that you’d then scan using traditional code-scanning techniques? How do you think about the alternatives there?

Aaron Fulkerson: I think it’s complementary — I don’t think it’s an either/or. Just to restate what you’re proposing —

James Kaplan: I’m not proposing — describing.

Aaron Fulkerson: You’re describing. Thank you. The scenario is that the agent generates deterministic procedural code, which then goes through a scan.

I think that’s complementary, right? You still have an agent making decisions about what procedural code to write, based on the sensitive systems it’s touching, the tools it’s calling, the data sources it’s pulling from. And I’d assert that even in that environment, you’ll want to have some rules — and you’ll have to have them, because it’s increasingly a requirement in most regions and nations to have proof of what ran, where it ran, and what rules were applied to that agent, even when it’s producing deterministic procedural code that goes through another scan. I suspect that whatever’s doing the scanning might itself be augmented by AI in some form, right? And there again, cryptographic enforcement, or hardware enforcement. I tend to use the two interchangeably because our customers — the people deploying confidential AI — are AI builders. They’re not the InfoSec or security team or the compliance team. So as soon as a lot of AI builders hear “cryptographic” or “encryption,” they go, “Yeah, that — I just wanna ship cool stuff.”

So I tend to use the two interchangeably because I want builders to know: no, this helps you ship cool stuff faster, because you don’t have to go through the protracted six- to eight-month surveys with your internal and external auditors that I’ve heard so many CTOs and AI builders describe as their current dilemma.

Hardware identity and the supply chain

James Kaplan: Trust me, I see it all the time. One thing that struck me hearing what you were describing is the ability to prove where something ran. Does this mean each chip has a unique identity, and therefore you’re able to confirm that it’s being run by who you think it’s being run by — in terms of an external service provider?

Aaron Fulkerson: That’s exactly correct. As I mentioned, with confidential computing there’s an — by the way, I’m oversimplifying. I’m not getting into too much technical detail —

James Kaplan: Every podcast everywhere oversimplifies everything, right? Otherwise they’d all be fourteen hours long.

Aaron Fulkerson: Exactly. I know somebody is going to say, “Well, the attestation service might be —” so I’m simplifying. But to your question: there’s an encryption key baked into the silicon, and that encryption key is what’s used to do the attestations. At the time you invoke a service, you’ll know exactly the profile. Where is it running? What is it running in — these Kubernetes nodes, that virtual private cloud? What are the network rules allowing it to connect to external services? Everything is encoded cryptographically and enforced by the hardware.

It’s physics. The significance here is, hey, can you trust AMD, Intel, NVIDIA at a hardware layer? And as long as you believe they know how to manufacture their chips, you don’t have to trust anything else. Nothing else in that stack do you trust.

James Kaplan: And do you know — or is there some way of knowing — that this individual chip was bought by this service provider versus that service provider versus somebody you may not want to be doing business with? Is there some registry somewhere? Apologies if that’s a naive question.

Aaron Fulkerson: That does exist. There’s actually a standard — a couple of them, in fact. SCITT is one, S-C-I-T-T. SALSA is another. They give you levels of verifiable provenance over the entire supply chain.

I don’t know if there’s a registry that exists independently. The technologies to do it exist, and I know all the silicon manufacturers and the hyperscalers have talked about having a registry so that they have complete supply-chain verifiability — but I’m not certain that registry exists yet. As an enterprise, though, you can bake that capability into confidential AI. That’s a feature of confidential AI, absolutely.

James Kaplan: Yeah. That’s an interesting idea. If you could say, “We know with certainty this is running on a set of chips that was bought by this service provider — there’s no chance it’s running someplace else,” or, “We know with certainty this is running on a set of chips located in this national jurisdiction versus that national jurisdiction.”

Aaron Fulkerson: Yeah. So I know there’s a podcast — AI Confidential — that I’ve had you on,

James Kaplan: Of course.

Aaron Fulkerson: A podcast that I host. I had a great conversation — I think on the very first AI Confidential — with Mark Russinovich, who’s the CTO of Azure,

James Kaplan: I’ve met him — he was on that panel I did at the conference you guys had.

Aaron Fulkerson: Exactly, yeah. And I’m looking forward to seeing you back at the Confidential Computing Summit at the end of June — the 23rd and 24th in San Francisco.

James Kaplan: I look forward to it. I’ll be there.

Aaron Fulkerson: Mark Russinovich, and Mark Papermaster — the CTO of AMD — in our very first AI Confidential were describing exactly what you’re talking about, James. And I was naive.

This was early in my tenure at Opaque. I was so focused on AI that I didn’t think about the entire hardware supply chain — securing it end to end. But to your point, you could extend this concept beyond provenance and supply chain to the actual manufacturer. You could do cryptographic enforcement of the design itself.

James Kaplan: I was going to say — that’s something incredibly important, I think, in the aerospace and defense community. They know exactly what the provenance of the hardware is, and exactly what’s running on what.

So let me ask a slightly different question. In some respect, what you’re describing reminds me of the early days of cloud security, when we started thinking about what became known as cloud security posture management — and we started to realize how important the business rules and policies were.

It took some time to sort out the model for defining those policies, for managing them, for figuring out what set of policies you wanted to have. Could you speak a little bit about that — about the process of policy management for confidential AI? How far along in the journey are people? What techniques do they use? What tends to work well versus less well?

Aaron Fulkerson: This is a really important topic you bring up. In the era of cloud, we did a trust-layer upgrade where we said, “Okay, SOC — we have to capture our policies at the time of deployment.” What’s different here, and what’s being demanded in a lot of different regions — the EU AI Act requires this, new regulations are coming online in Asia, same in the Middle East, and I believe it’s also true for HIPAA — is that you have to extend that to runtime.

It’s not just policies at the time of configuration; it’s policies at runtime, and you have to provide proof of runtime enforcement. The significance of confidential here is that it produces a report of all the policies that were enforced at runtime, and you can prove cryptographically that they were executed.

Because of the nature of agents behaving with human capabilities at machine speed, you have to have provable runtime policy enforcement. The good news is there’s been a proliferation of great frameworks and tools for policy-as-code. You don’t have to reinvent anything — you just take the policies-as-code, cryptographically enforce them, and measure at runtime that they were actually enforced.

James Kaplan: You’re building on all of these component parts. This is another one of those one-plus-one-equals-three situations you see in technology — the combinatorics of technology — where we already have this policy-as-code thing, and we already have this hardware key baked in. Right.

Aaron Fulkerson: Right. You add these two together and — oh my gosh — now we’ve actually got a way to adopt AI agents confidently and safely. Because we want to attach AI agents to all our systems and all our data sources — that’s how you get value. The more sensitive the data, the more value you can get out of it, but also the more risk there is in bleeding it. The key point is that this new security requirement in the agentic system, like all things in tech, is something we were doing previously, brought forward in a new way.

Sovereign AI and the standards landscape

James Kaplan: So how does this manifest itself in sovereign AI? Hark Singh, the CTO of InfraPartners, did a fireside chat at the Technology Leadership Forum last week, and was talking about how sovereign cloud and sovereign AI are an increasingly significant demand driver in the data center space.

How do confidential computing and confidential AI intersect with the desire for sovereign cloud and sovereign AI? I was wondering if you could speak a little bit about those dynamics.

Aaron Fulkerson: Absolutely. It’s an essential cornerstone capability. Look no further than Dr. Najwa Araj of the UAE’s ATRC and TII, who stated in a recent press release with Opaque that you can’t have sovereign without this kind of verifiability.

James Kaplan: And look at Jensen’s keynote at the April — was it April GTC?

Aaron Fulkerson: Was it April or March? I forget. I think —

James Kaplan: I think April, but I’m not sure.

Aaron Fulkerson: Jensen had it at the center of his keynote — the central topic was confidential and verifiability. Why? Because he understands what we’ve just been talking about. He said it in every session after the keynote: in order to safely adopt enterprise or personal agents, you have to have verifiability. He’s talking about confidential, hardware-enforced verifiability. That’s exactly what he means.

So why does this have to do with sovereign? If you look across all the big tech players, all the silicon vendors have oriented themselves around these confidential capabilities. Hyperscalers are demanding confidential. Frontier model labs are adopting confidential. What’s going on?

The hyperscalers demand it because they know they won’t get adoption in the rest of the world — outside the United States — if they can’t verify data privacy for sovereign cloud operators. So they’re demanding it from the silicon vendors, because otherwise the hyperscalers are going to get their lunch eaten by NeoClouds in the rest of the world. That’s happening right now.

For the silicon vendors upstream from the hyperscalers: if they don’t provide these verifiable guarantees, China is knocking on the door with great chips. They’re coming to eat the silicon vendors’ lunch, so the vendors have to differentiate. They’ve found that differentiating around confidential is a great way to do that — because the hyperscalers need it.

What about the frontier model labs? Why are they so insistent? They’re making incredible models — their intellectual property is their model weights. If they can’t protect their model weights in a sovereign cloud in the rest of the world, what’s their differentiation?

James Kaplan: Property is going to be stolen? And that’s what confidential does for the frontier model labs — it allows them to deploy in the rest of the world with verifiable proof that they protected their intellectual property, their model weights.

So I heard a couple of things. For the labs, you can use confidential AI to protect the model weights. For users and for AI or cloud providers in the rest of the world, you can demonstrate you’re compliant with local privacy regulations, which may mandate encryption, and compliant with data-localization regulations that demand you say, “Okay, this type of data or this type of workload can’t leave this national jurisdiction.”

Is that a fair way of thinking about it?

Aaron Fulkerson: That’s the simplest requirement, and it’s a good one to focus on.

The other thing we’re seeing in the rest of the world around sovereign goes back to what we already discussed — how do we reliably enforce policies on agents, not just data residency but every kind of policy. And another thing we’re seeing a lot is that many of these sovereign nations have their own frontier model labs, or they’re fine-tuning models with highly sensitive healthcare data, and they need to make sure that’s protected.

James Kaplan: So let me ask this question. You talked about what each participant in the ecosystem wants out of confidential AI. Where do you think things will shake out — in terms of what happens at the enterprise layer versus the application-vendor layer versus the cloud layer versus the frontier-lab / model-provider layer? How will those different pieces of the ecosystem interact with each other, and what functionality will they provide around confidential AI?

Aaron Fulkerson: Over the next two years, everybody will begin to offer this. We’re already seeing it. We saw it last year with Google’s offering — mutually attested components, or confidential Gemini. Everybody’s going to be offering endpoints to their language models and their applications.

That way, you can do mutual attestation and roll up verifiable guarantees. In fact, we’re working with a consortium of silicon, cloud, and frontier-model-lab partners to create an open standard that lets you create rules — not just on a single host, but portable rules that you can pass between hosts and that govern the policies of a particular workload. If you’re going to send your data, you can send it with, “Hey, here’s the specific set of rules I require enforced,” or it fails — and if it fails, I can prove that you couldn’t even access my data.

A lot of these standards already exist, but they’re very fragmented. To your point, you see frontier model labs enforcing their own flavors of confidential — Anthropic, a customer of Opaque, uses mutually-attested TLS, right? That’s a slice of policies being enforced. Microsoft does something different — their own flavor. TikTok, Meta, Google, ServiceNow — everybody’s got their own emerging patchwork quilt of cryptographic, hardware-enforced policies, but they’re not portable and they’re not standardized. What we’re going to see — and I think there’ll be an announcement at the Confidential Computing Summit this year on June 23rd and 24th in San Francisco — is that we’ll take all of these standards from the Linux Foundation and IETF, like SPIFFE, EATS, RATS, and say, “Here’s one standard that’s a superset, that you can enforce at your organization, and that will be portable as long as you write against this specific open specification.”

James Kaplan: Terrific. Anything we didn’t speak about? I’m sure there’s a zillion things. Thank you for joining us.

Aaron Fulkerson: Hey, thanks for having me, James. Look forward to seeing you.

CITER evolved and expanded over the years. After COVID, it re-emerged as the Cloud Leadership Forum (CLF), which was bigger and more ambitious. A couple of years ago CLF became the Technology Leadership Forum (TLF) as cloud became less of a distinct issue and more an organic part of the way you run enterprise technology.

TLF makes for two of my favorite days of the year. An opportunity to spend time with 50 of the most thoughtful enterprise technology executives I can find and discuss issues like AI platforms, AI security, product operating models, technology innovation and semantic layers/knowledge graphs. As you would expect, in recent sessions everyone there has wrestled with how AI can and should change enterprise technology.

Here are a few of my idiosyncratic takeaways from the day:

1. You can use AI to trade bad inefficiency for good inefficiency.

2. Mythos is cause for determination, not panic.

3. Transforming a business domain with AI requires hard problems and a number.

4. AI will disrupt, not destroy B2B software.

5. Token economics might not mirror ride-sharing economics.

6. Tech economics must focus on incentives, not precision.

1. You can use AI to trade bad inefficiency for good inefficiency

For decades, institutions removed craftsmanship, personalization and tactile experiences because the coordination costs were too high. AI may allow us to recapture warmth and texture — not despite efficiency gains, but because of them.

For each TLF session, we assemble a directory with the name, photo, role and contact information for every participant and faculty member. Historically, this was painful. Participant changes, updated photos and corrections to biographies required us to send a request to the visual aids team and wait 24 hours for them to update the document manually with design software.

A few years ago, we saved time by eliminating the printed version altogether. We just emailed participants a PDF. Operationally, it was more efficient. Even if people missed the physical book.

Agent Serena transformed our tradeoffs. She stored TLF information in a graph of Markdown files and allowed us to turn the event program into a composable document, assembled via a skill. We needed to update a photo or add a new participant? Just two instructions: one to update the information and a second to regenerate the PDF.

We used the time we recovered from pre-press production to print old-fashioned spiral-bound books containing the program, faculty information and breakout materials. They looked great. Participants loved them.

Where else can we use AI to fund the return of things lost to the imperative of rationalization?

2. Mythos is cause for determination, not panic

At the second CLF session in 2021 we held a cloud security breakout and nobody cared. It drew fewer participants than any other breakout. This year my colleagues Sven Blumberg and Rich Isenberg led a plenary session on what advancing frontier models mean for technology risk, and I had to drag them off the stage. The crowd would have kept discussing this issue for another couple of hours if I had let them.

Clearly the threat is real, and not only about zero days. Advanced models will provide attackers with unprecedented visibility into corporate technology environments — security by obscurity may not be a good strategy now, and it will be even less tenable shortly. For years the attackers with the most advanced capabilities didn’t have the destructive intent. And the attackers with the most destructive intent lacked access to the most destructive capabilities. New models can perform more complicated processes and allow a much broader range of attackers to employ new vectors.

US-based frontier labs may succeed in creating guardrails that limit malicious use of their products. None of those will apply to locally hosted, open weight models with capabilities that trail offerings from Anthropic, OpenAI and Google by months rather than years.

And TLF members are concerned. None of them believed that their company’s cybersecurity program was keeping pace with AI-enabled cybersecurity threats. Their boards were asking questions about how fast to move and where to start that they didn’t entirely know how to answer. And they had doubts about many of their software vendors. Patch velocity had increased, but not patch quality. They fear many vendor patches will create new vulnerabilities as they try to remediate older ones. One member said: “I don’t think vendors understand what they’re building.”

And yet I heard determination rather than panic or resignation. Some of this may be years of lived experience. Nearly fifteen years ago the US Secretary of Defense warned of a cyber Pearl Harbor. It hasn’t happened. Each year brings new challenges, but enterprise technology as a domain manages to stave off the cataclysm. At least for another few months.

TLF members indicated that they would push for transparency into their environments. They could isolate systems they couldn’t patch on dedicated network segments. They would automate their patching processes. And over time they might use AI-enabled software engineering to remediate technical debt — and build better software in place of the fragile environments they have now. All of which will require determination, funding and the ability to think in systems. One member said: “The leadership team thinks about innovation over here and risk over there — we haven’t figured out how to discuss this in an integrated way and make tradeoffs.”

3. Transforming a business domain with AI requires hard problems and a number

Almost no TLF members said their companies had deployed AI at scale. Most didn’t have any agents in production. Two sessions explored how to build organizational conviction behind and momentum in using AI to reshape business domains.

The session on building agentic workers wrestled with the shape and priorities for a transformation. Everyone agreed that you need a T-shaped transformation, one in which you pursue visible change early in some areas to create enthusiasm in the executive team — and at the same time build out the underlying technology platform to facilitate scaling and contain the technical debt you create.

Which led to a more contentious debate: if you create agentic workers, which workers should you start with? Yes, you need supportive business leaders, but you also need to pick hard problems. If you focus on easy wins in defining your digital workers, you give ammunition to those who describe AI as a toy rather than a tool.

My colleagues Brian Elliot and Mark Gu led a session based on their experiences deploying an AI operating system to drive business change. You need a platform that goes far beyond agent management to data pipelines, knowledge orchestration (often via a graph) and connection to traditional machine learning modules.

Starting with two or three related, difficult business problems allows you to justify the platform investment — but also implies organizational disruption. You will redesign roles for human employees, require new forms of collaboration and upend long-standing assumptions. Legal will resist using LLMs to process any employee data. Security teams may not yet be comfortable with agentic identities. HR is paying high salaries for AI experts who may not manage large teams. These fights are happening now.

You overcome resistance with a number: customer retention, operating margin, inventory turns — some metric with a direct enough relationship to share price that senior executives will say to those who push back on change, “I hear what you are saying, but I want my thing. So figure out a way to make it work.”

4. AI will disrupt, not destroy B2B software

We had a new type of plenary session at TLF last week — a panel of venture capitalists. Ed Sim from Boldstart, Daniel Frankenstein from Joule Ventures and Will Summerlin from Autopilot engaged with the group on how enterprises can best work with VC-funded companies and how AI will change B2B software markets.

Wow, there is some frustration out there! What do TLF members see from their incumbent software vendors? Slower innovation, degraded quality and more aggressive negotiation. They think some providers see their products as “falling knives” and have resolved to extract as much cash as they can from the portfolio before it declines into irrelevance. Others simply don’t get AI — they want extortionate rates for unimpressive capabilities that only reinforce silos between different parts of the environment.

Will enterprise software get eaten away from above and below? Will the layer between what companies build themselves and what frontier labs provide disappear as companies use AI-enabled engineering productivity to escape SaaS’s “one size fits none” business model and LLM-providers move up the stack?

History and market structure suggest not. Remember all the predictions about a decade ago that the cloud service providers would eat the entire enterprise software market? That (checks notes) didn’t happen. Enterprise software is an archipelago of hundreds of micro-niches, each with their own needs and idiosyncrasies. Many domains have astronomical switching costs. And companies resist letting one vendor dominate their technology future. AI changes none of that.

In some cases TLF members will make different buy-build decisions, but they have barely begun to uplift their own software engineering capabilities — and they want the specialized content or capability that third-party software can provide. They may want to build more, but they don’t want to build everything themselves.

The VCs on our panel have no illusions about the difficulty in building a disruptive company. They all expressed skepticism about the heady numbers some infant AI companies are reporting, observing that much of it might be trial rather than enduring revenue. Ed also pointed out that some startups are just white-labeling tokens from LLM-providers and would struggle if token economics became more challenging.

Still, they all believe that they face a generational opportunity to build companies that will use AI to displace incumbent software companies that are not meeting the moment. Ed talked about that here in his blog. I don’t think the CIOs and CTOs in TLF disagree with them.

5. Token economics might not mirror ride-sharing economics

Today most TLF members have little idea what they spend on inferencing. Their companies haven’t hit the steep part of the adoption curve yet. Low volumes mean low costs. Could that change? And what should they do if it does?

Infrapartners CTO Harqs Singh sees no slowdown in data center investment. He gets calls asking if they could launch a 100MW project Monday based on a PO sent over the weekend. Harqs turns constructing a 100MW facility into an industrial process, putting capacity in the ground more quickly and at less cost.

But that doesn’t remove other constraints in the system. You still need GPUs or TPUs to put in the data center, and you need electricity to run the chips. These capacity constraints — and the need for investors to generate a return on capital — cause many TLF members to ask whether we are in the early days of ride-sharing.

Several TLF members compared today’s token prices to the early days of ride-sharing. As Uber sought to grab market share, riders paid roughly 41% of the actual cost of each trip; the rest came from investors. Then fares rose 92% between 2018 and 2021. And some jurisdictions have levied additional fees or taxes on each ride.

The ride-sharing analogy isn’t perfect. Ride-sharing promised convenience, rather than massive corporate productivity improvements. American companies and public institutions spend USD 16 trillion in employee compensation. Frontier labs bet that any possible token costs will pale next to efficiencies there. Inferencing costs are probably declining by 90 percent per year. Ride-sharing investors lost a bet that autonomous vehicles would transform their cost structure by now. Ride-sharing also doesn’t have much in terms of attractive adjacencies. Frontier labs believe they can profitably move up the stack into applications and services — some railroads generated more returns dealing in real estate near their stations than from selling tickets.

Even so, you must plan for a world of escalating inferencing costs. As Ed Sim pointed out, “Compute is already fully utilized and Anthropic may ration with a high minimum spend. Token prices are going through the roof — you need to plan for scarcity.”

In response, TLF members expect their companies will develop more insight into inferencing economics so they can focus on accretive use cases, apply FinOps to AI so their applications will use tokens more efficiently and diversify their usage — by running open weight models on neo-cloud infrastructure, for example.

6. Tech economics must focus on incentives, not precision

I have been fascinated for decades by enterprise technology economics. What an interesting and complicated machine! Requests and money go in one side. They interact with policies, existing technology environments, operational processes, organizational capabilities and vendor arrangements. Systems and services come out the other side. How can we better understand the dials you turn to change the relationship between inputs and outputs?

I asked teams to create several generations of spreadsheet models over the years — all eventually collapsed under their own weight. Cursor and Claude Code helped me build the model that describes enterprise technology the way it is — an intricate, dynamic system, with dozens of economic and operational dependencies between nodes in the graph. Here’s what it looks like:

The model demonstrates several important ideas:

• Each company faces a pipeline of business-technology investment opportunities that you can stack rank based on ROI — and the ROI declines according to a half-life

• Incremental EBITDA lift each year depends on business-driven investment, adjusted for value leakage, engineering deadweight loss and adoption loss. Because these factors are multiplicative, collectively they can consume three-quarters of business investment.

• Each million dollars of business-technology investment generates USD 150,000-250,000 in incremental annual run and mandatory investment costs, whether you get business value or not.

• Given flat expenditures, companies will tend toward an “IT doom loop,” where run costs and mandatory investments consume the entire budget, driving EBITDA lift to zero.

• AI-enabled engineering (or spec-driven development) changes enterprise technology economics. It improves the ROI of retiring technical debt (reducing application maintenance costs) and run automation (reducing infrastructure costs), freeing up more resources to invest in initiatives that improve revenue or reduce operational cost. It also reduces engineering deadweight loss, so you get more EBITDA lift out of each dollar invested. That’s why investments in AI-enabled software engineering will deliver better-than-linear returns for many companies.

This resonated with many participants. They agreed both with the underlying economic dynamics and the need for management teams to understand them better in making investment decisions. The catch? The tendency for accounting to crowd out economic insight. Many bore scars from battles over tech chargebacks, and worried that any calculation of a unit cost would set off endless debates about whether a server image should cost USD 3,100 or USD 3,200.

As always, leadership will be key here. Members of the management team must make clear that they want transparency into technology cost and value to make better decisions, not to push allocations from one line of business to the other.

It was a great session — I look forward to the one in the Fall in NYC!

1. The classical skills of grammar, logic and rhetoric are essential to advocating for good business decisions in complex organizations.

2. Making shoddy arguments is easy in the face of time pressure and incomplete information.

3. Using AI to evaluate my own writing demonstrates how it can identify vulnerabilities in an argument.

4. A simple prompt enables you to apply this to your own work -- but you should treat the output as a guide, not a mandate.

1. Reviving the Trivium

In 1947, British writer Dorothy Sayers urged educators to embrace the principles of classical education, in the face of a culture of specialization.

Sayers worried that educators had failed to teach students how to learn, think and communicate. She wrote:

Is it not the great defect of our education to-day (—a defect traceable through all the disquieting symptoms of trouble that I have mentioned—) that although we often succeed in teaching our pupils “subjects,” we fail lamentably on the whole in teaching them how to think? They learn everything, except the art of learning.

Sayers offered the trivium as a solution. Roman thinker Martianus Capella divided the liberal arts into the Trivium (grammar, logic and rhetoric) and Quadrivium (arithmetic, geometry, astronomy and music).

The whole of the Trivium was in fact intended to teach the pupil the proper use of the tools of learning, before he began to apply them to “subjects” at all.

The Trivium applies in a special way to enterprise technologists. Engineers have always needed to make thorny design decisions and write technical specifications. The advent of spec-driven development will only make the Trivium more important to them. As Steve Jobs said, “It’s in Apple’s DNA that technology alone is not enough—it’s technology married with liberal arts, married with the humanities, that yields the results that make our hearts sing.”

Especially today -- no less than the students Sayers worried about -- enterprise technologists must be learners given the pace of change they face. The Trivium allows them to assimilate and evaluate new ideas more quickly.

Technologists must make -- in the face of incomplete information and massive ambiguity -- decisions they will have to live with for years. As my old professor Martha Nussbaum pointed out, critical thinking, argumentation, and narrative imagination are essential for making decisions in complex systems. [1]

No CIO or CTO is a sovereign. They need support and engagement from the rest of the management team to create a portfolio, define requirements, fund platforms and get value from systems investments. So they must make arguments. How many meetings have blown up because nobody asked, “What would the counter-argument be? Why would the client disagree?” How many initiatives have progressed because the business case anticipated the rebuttal the CFO might offer?

2. How robust is our thinking?

Can CIOs and CTOs make better arguments? In How AI Critiques an Argument, Steven Mintz [2] of the University of Texas, Austin suggests even trained professionals make shoddy arguments:

• Honest argument requires specifying not just the claim and evidence but the warrants, qualifications, and rebuttals — the conditions under which the claim would not hold

• Social incentives and norms (in education, in media) encourage speed, certainty, and strong positions over nuance and qualification -- we learn to overclaim, reducing causation to a single cause and crowding out alternatives with false dilemmas

• When challenged we learn to evade — changing the subject, moving the goalposts, retreating to vagueness.

Do the failure modes resonate here? Sometimes they do. Sometimes the imperative of just getting through the week or getting the document done leads to motivated reasoning, vagueness and over-simplification. [3]

One of my mentors used to counsel me “The direction of the hypothesis doesn’t matter here -- we’ll either prove or disprove it.” But how often does “Bottom Line Up Front” mean “start with the answer” rather than “start with a falsifiable proposition to validate or invalidate with facts.” All the time I see people locked into a position they don’t want to re-evaluate.

Mintz points out “And when challenged, arguers evade — changing the subject, moving the goalposts, retreating to vagueness.” How many of us have the fortitude to meet an objection head on? Or do we respond to clarifying questions with defensiveness and more vagueness?

I heard more than one CEO ask of the CIO “why does this have to be so complicated?” But you must use clear structure and clear writing to communicate nuance honestly. And not confuse the sometimes brutal reductionism required in decision making (i.e. go left or go right; launch the project or don’t) with all the nuance that goes into making that decision.

But Mintz also offers hope, in that AI can help us make better arguments -- as a complement to, not a replacement for, management judgment. We are all often time-constrained, stressed, distracted or exhausted -- AI is not any of those things.

• AI models the structural habits of careful argument more consistently than most humans educated to develop them: well-functioning AI follows a deliberative sequence before rendering judgment: reconstruct, calibrate, consider alternatives, steelman, treat the first response as provisional

• AI can provide immediate and specific correction of argumentative failures — showing exactly what a fair restatement looks like, where a causal claim outruns the evidence, which alternatives a false dilemma has collapsed

3. Applying Mintz’s advice to my writing

Mintz recommended three frameworks in his piece.

• Toulmin evaluates the structural anatomy of an argument — claim, evidence, warrant, backing, qualifier, and rebuttal

• Paul-Elder evaluates the intellectual quality of the thinking behind the argument — clarity, accuracy, precision, relevance, depth, breadth, logic, and fairness

• Walton identifies what type of argument is being made — analogy, causal, from authority, practical reasoning, from example — and then applies the critical questions specific to that type

I used an agent to evaluate Just like parents CIOs must be demon game theorists to elicit cooperation in terms of each framework plus Minto’s pyramid principle. Clearly I have work to do. None of the articles received more than seven points out of ten!

What do I need to do differently?

• Be explicit in naming the warrant that connects my evidence to my conclusion

• Engage directly with the strongest possible objections to my arguments, especially when using analogies

• Map prescriptions to diagnoses

• Explain why personal experiences are generally applicable

4. A prompt you can use

My writing has gotten better because I revise and revise based on feedback from agents and prompts. The prompt I used above even made this article better. As I used the prompt through about 10 iterations of the article my score improved from the mid-5s to 8.5. Here are some of the tweaks.

• I originally just asserted that better arguments win; with some encouragement from the prompt, I provided a few illustrations of how incomplete arguments can sabotage a meeting -- and rigorous ones can win the room. (Yes, I would have liked to be more specific -- but in some cases I must be discrete.)

• I also didn’t address an important potential objection to one of my key points. Some people believe CIOs win with good relationships rather than good arguments. I added a footnote explaining you need good relationships and good arguments (and the two reinforce each other).

Sometimes I addressed these issues in footnotes, in order to preserve narrative flow. If the CFO asks a killer question and you say, in response, let’s turn to page 25 in the appendix where we address that issue, that works great.

Try it on your next initiative request with this prompt.

But treat it as a guide, not a mandate. No prompt will capture all the context you have -- so sometimes the model will recommend justifying a proposition your audience will stipulate. Models can also be literal. Sometimes I should spell out my warrants, but sometimes letting the reader do a little work in connecting the dots creates investment in argument.

Footnotes

[1] Two notes here:

• In the spirit of transparency I took Nussbaum’s class “Philosophy and the Novel” with 200 other students. The lectures were terrific.

• Yes, in her book, Nussbaum talks about the humanities in the context of protecting democracy rather than improving corporations, but companies are themselves polities, with bylaws as constitutions and elections for boards of directors.

[2] Mintz is a professor of history at the University of Texas at Austin and a leading commentator on pedagogy and critical thinking. He writes the Higher Ed Gamma column at Inside Higher Ed and received the AAC&U’s President’s Award for Outstanding Contributions to Liberal Education in 2025.

[3] I know some have argued that relationships, not great problem solving and great arguments, win the day. Here’s why that doesn’t resonate for me.

• Great CIOs depend on great business relationships and great arguments -- and great arguments make for better business relationships.

• In a world where you’ll live with the impacts of today’s decisions for years, you want only the best arguments in the boardroom.

• Executive teams change. The business partner who doesn’t challenge you may take another job. His or her replacement might look at a business case and ask “What the bloody hell is this?”

Appendices

As noted above, I used AI to apply 4 argument frameworks to Just like parents CIOs must be demon game theorists to elicit cooperation. Here’s what it said:

• Thesis: CIOs/CTOs need cross–executive-suite cooperation to maximize multi-year EBITDA lift from enterprise technology, but coordination costs (information asymmetry, misaligned incentives, collective-action problems, weak “enforcement” inside the firm) make that cooperation scarce—so they have to think like game theorists, not optimists or petitioners.

• Analogy: Parents can sometimes establish escalation dominance (the piece’s restaurant / fireman’s-carry example); technology leaders usually cannot, so the workable substitute is to shape an information environment that raises the cost of non-cooperation rather than pretending good intent or a CEO edict will fix incentives.

• Mechanism: An EBITDA-lift model (value leakage, adoption loss, deadweight loss, run cost / T4T) to show where cooperation matters, despite symmetric frustrations between technology leaders and their peers. Only ~13% of senior technology executives report consistently getting the support and engagement needed to capture value—so the bottleneck is governance and politics.

• Prescription: Six tactics—develop a granular view of where cooperation drives lift; map political constraints coolly; force decisions with options and implications; make cooperation visible (including board-level transparency where appropriate); adopt “not my fault, but my problem”; and signal credibility (e.g., eating your own cooking on new ways of working)—as the practical translation of “demon game theorist,” not as a mandate to strong-arm peers.

Minto summary: The piece’s most important structural failure is at the BLUF level: “What you need to know” bundles two distinct key lines — the EBITDA cooperation need and the coordination cost diagnosis — into a single bullet, when they each deserve their own slot. A truly MECE BLUF would have three bullets: (1) CIOs need cross-executive cooperation to maximize EBITDA lift; (2) coordination costs explain why they rarely get it; (3) game theory provides a path — and the six tactics below show how. The current structure is close but the compression at the BLUF level obscures the argument’s logical spine.

This weeks Prosaic Times explains how you can use AI to evaluate your arguments against classic frameworks. Here’s a prompt you can use!

You are an expert in argumentation and logic, trained in multiple frameworks for evaluating the quality of reasoning. I am going to give you a piece of writing. Your job is to evaluate the quality of its argumentation using three frameworks, each of which illuminates a different dimension of the argument.

Before applying any framework, reconstruct the argument in its strongest possible form.
Be specific throughout: quote the text, identify exact failures, propose corrections. Vagueness and evasion are themselves argumentative failures — do not be diplomatic about them.

---

### STEP 1: RECONSTRUCT THE ARGUMENT

Identify the following before evaluating. Present as a brief structured summary.

1. **Main claim**: The central assertion the author wants the reader to accept.
2. **Supporting claims**: 2–5 sub-claims that support the main claim.
3. **Evidence**: For each supporting claim, the specific facts, data, or examples cited.
4. **Warrants**: The underlying assumptions connecting evidence to claims. (These are almost always implicit — surface them.)
5. **Qualifications**: The conditions under which the claim holds or is limited.
6. **Rebuttal**: The strongest counterargument the author acknowledges and responds to (if any).
7. **Argument type(s)**: What kind of argument is this primarily? (e.g., argument from analogy, from authority, causal argument, argument from precedent, argument from example — see Table 3.)

---

### TABLE 1: TOULMIN’S MODEL

*Stephen Toulmin, The Uses of Argument (1958). Most useful for finding what’s missing — especially warrants (almost always implicit) and rebuttals (almost always avoided).*

For each component, quote the relevant passage from the text (or note “implicit” / “absent”), assess its quality, score 1–5, and flag any problems.

| Component | Definition | What’s in the text | Quality assessment | Score (1–5) |
|---|---|---|---|---|
| **Claim** | The assertion being made — what the author wants the reader to accept | | Is it explicit? Falsifiable? Specific enough to act on? | |
| **Grounds** | The evidence or data offered in support of the claim | | Is it specific? Sufficient? Accurate? Or generic (“studies show…”)? | |
| **Warrant** | The assumption connecting the grounds to the claim — why the evidence should lead the reader to accept the claim | | Is it stated or implicit? Reasonable? Would a skeptical reader reject it? | |
| **Backing** | Support for the warrant itself — why the warrant should be trusted | | Is the warrant’s own foundation established, or simply assumed? | |
| **Qualifier** | The degree of certainty with which the claim is asserted (necessarily, probably, presumably, in most cases…) | | Does the qualifier match the strength of the evidence, or does the author overclaim? | |
| **Rebuttal** | The conditions under which the claim would not hold — and the author’s response to the strongest counterargument | | Does the author engage the best opposing argument, or a weakened version? Or avoid rebuttal entirely? | |

**Toulmin summary**: In 2–3 sentences, identify the single most important structural gap — the missing or weakest component — and explain what a stronger version would look like.

---

### TABLE 2: PAUL-ELDER CRITICAL THINKING STANDARDS

*Richard Paul and Linda Elder. Evaluates not just whether the argument holds together, but whether the thinking behind it is honest and fully developed — whether assumptions are examined, and whether the strongest counterarguments have been genuinely engaged.*

Score each standard 1–5. Quote specific evidence from the text. Identify the most important failure.

| Standard | Definition | Evidence from text | Score (1–5) | Key failure (if any) |
|---|---|---|---|---|
| **Clarity** | The argument can be understood without ambiguity. Key terms are defined. The reader does not have to guess what the author means. | | | |
| **Accuracy** | Claims are factually correct and properly attributed. The author has checked rather than assumed. | | | |
| **Precision** | Claims are specific enough to be useful — not “the process is inefficient” but “the process requires 17 approvals taking five days.” | | | |
| **Relevance** | Evidence and sub-claims bear directly on the main claim. No padding, tangents, or evidence that sounds related but does not actually support the point. | | | |
| **Depth** | The argument addresses the complexity of the problem rather than offering an oversimplified answer. Root causes are identified, not just symptoms. | | | |
| **Breadth** | The argument considers other perspectives and relevant viewpoints — not just the one most convenient to the author’s conclusion. | | | |
| **Logic** | The conclusions follow from the premises. Sub-claims support the main claim. There are no internal contradictions. | | | |
| **Fairness** | The author represents opposing views honestly, applies the same standards to all sides, and does not load the argument with assumptions that favor one conclusion. | | | |

**Paul-Elder summary**: Identify the two standards with the lowest scores. Explain the specific thinking failure each represents, and propose how the author could address it.

---

### TABLE 3: WALTON’S ARGUMENTATION SCHEMES

*Douglas Walton. Arguments come in different types, each with its own structure and its own characteristic weak points. The framework identifies which type of argument is being made, then applies the right critical questions.*

**Step A — Identify the scheme(s).** Which of the following argument types does this piece primarily use? (Check all that apply.)

| Scheme | Description | Present in text? |
|---|---|---|
| **Argument from example** | A general claim is supported by one or more specific cases | |
| **Argument from analogy** | A claim about X is supported by showing X resembles Y, where the claim already holds for Y | |
| **Causal argument** | X causes Y; therefore doing/preventing X will produce/prevent Y | |
| **Argument from authority** | A claim is supported by citing an expert, institution, or credentialed source | |
| **Argument from consequence** | We should/should not do X because of the good/bad outcomes it will produce | |
| **Argument from precedent** | X was done (or decided) before in a similar situation; therefore X is appropriate here | |
| **Argument from sign** | Observable evidence Y is treated as a reliable indicator that X is the case | |
| **Practical reasoning** | We have goal G; action A will achieve G; therefore we should do A | |
| **Argument from commitment** | The author (or opponent) has previously committed to a position that implies the current claim | |

**Step B — Apply the critical questions.** For each scheme identified in Step A, answer the scheme’s characteristic critical questions. Quote the text in each answer.

*Argument from example:*
- Are the examples representative, or cherry-picked?
- How many examples would be needed to support the generalization being made?
- Are there counterexamples the author has not addressed?

*Argument from analogy:*
- In what relevant respects are X and Y actually similar?
- In what relevant respects do they differ — and do those differences undermine the analogy?
- Is the analogy being used to illuminate or to substitute for argument?

*Causal argument:*
- Is the causal claim established, or merely asserted?
- Could the correlation be explained by a third factor?
- Has the author ruled out reverse causation?

*Argument from authority:*
- Is the cited authority genuinely expert in the relevant domain?
- Is there expert disagreement the author has not acknowledged?
- Is the authority being cited to support evidence, or to replace it?

*Argument from consequence:*
- Are the predicted consequences realistic and well-supported?
- Have alternative means to the same end been considered?
- Does the argument accurately weigh costs against benefits?

*Argument from precedent:*
- Is the precedent case genuinely analogous?
- Are there relevant differences between the precedent and the current case?
- Does the precedent actually establish the norm the author claims?

*Practical reasoning:*
- Is the goal clearly stated and agreed upon?
- Does the proposed action actually achieve the goal?
- Are there alternative actions that would achieve the goal more effectively or at lower cost?

**Walton summary**: Name the dominant scheme. Identify the one critical question it most urgently fails to answer. Explain what a satisfactory answer would require.

---

### OVERALL ASSESSMENT

1. **Argument score** (1–10): One number for the overall quality of the argumentation.
2. **Framework that reveals most**: Which of the three tables exposed the most important weakness?
3. **Strongest element**: The most compelling part of the argument — quote and explain.
4. **Weakest element**: The single most important argumentative failure — quote, explain, and propose a revised version.

---

### TABLE 4: MINTO PYRAMID PRINCIPLE

*Barbara Minto, The Pyramid Principle. Most useful for finding whether the conclusion is stated first, whether the support is genuinely MECE, and whether vertical and horizontal logic hold at every level.*

Before scoring, map the actual structure:

Governing Thought: [as stated or inferred — one complete sentence]
  ├── Key Line 1: [label]
  │     ├── [supporting point]
  │     └── [supporting point]
  ├── Key Line 2: [label]
  │     └── [supporting point]
  └── Key Line 3: [label]
        ├── [supporting point]
        └── [supporting point]

If the piece does not have a discernible pyramid, show the flat structure it does have and label the problem: stream of consciousness, inverted pyramid, buried lede, flat list, or other.

For each criterion, quote the relevant passage from the text (or note “absent”), assess its quality, and score 1–10.

| Criterion | Score (1–10) | Assessment |
|---|---|---|
| **Governing Thought** — One clear, complete sentence capturing the full conclusion. Stated explicitly and early. A topic or title is not a governing thought. | | Is it explicit? Complete? Does it state a conclusion or just name a subject? Quote it or flag its absence. |
| **Answer First** — The piece leads with the conclusion, not builds to it. BLUF applied. | | Where does the answer appear — paragraph 1, middle, last? Quote where the answer lands and assess the cost of any delay. |
| **Pyramid Structure** — Argument organized as a true pyramid: finite groups, explicit hierarchy, not a flat list or stream of consciousness. | | Can the hierarchy be diagrammed? Are groups clearly delimited? Quote the opening structure and assess whether it signals a pyramid or a list. |
| **MECE** — Supporting points are non-overlapping (mutually exclusive) and together cover all the ground the claim requires (collectively exhaustive). | | Do any two key lines make the same point? Can a key line be added without making the others redundant? Quote the key lines and identify the failure. |
| **Logical Integrity** — Vertical: each point answers “why?” from the level above. Horizontal: points within each group follow a single ordering principle. | | Does “why?” work top-to-bottom? Do points within each group follow a consistent ordering principle? Quote a specific failure if one exists. |

**Minto summary:** In 2–3 sentences, identify the single most important structural failure — the missing or weakest criterion — and explain what a stronger version would look like for this specific piece.

Then show what the pyramid should look like — same content, restructured:

Governing Thought: [revised one-sentence conclusion]
  ├── Key Line 1: [label]
  │     ├── [supporting point]
  │     └── [supporting point]
  ├── Key Line 2: [label]
  │     └── [supporting point]
  └── Key Line 3: [label]
        ├── [supporting point]
        └── [supporting point]

Assess the introduction against SCQA:

| Element | Present? | Where? | Assessment |
|---------|----------|--------|------------|
| Situation | Yes / No / Implied | Para X | |
| Complication | Yes / No / Implied | Para X | |
| Question | Yes / No / Implied | Para X | |
| Answer | Yes / No / Implied | Para X | |

---

### THE TEXT TO EVALUATE:

[PASTE TEXT HERE]

A more honest answer involves Professor Litchfield’s class on The Industrial Revolution in Early Modern England.

Using spinning jennies and mechanical looms in factories to make textiles changed everything in England. It moved people from the countryside to the city. It allowed ordinary Englishmen to own a second suit of clothes. It demolished artisans’ livelihoods. It empowered a rising class of bourgeoisie, and undermined the aristocracy. It created the surplus that underwrote an empire. Software applications and computer hardware were the spinning jennies and mechanical looms of my own era. All especially relevant as we debate the social, political and economic implications of AI.

Professor Litchfield liked to say that “Political Science has the theories. In the History Department we are custodians of the facts.” I like to know what happened; only facts can tell you that. But the facts of history are imperfect. Much of the work in formulating a business or technology strategy depends on work that feels like history — trying to make sense of incomplete, contradictory, subjective, and sometimes unreliable information.

Bob Pasker co-founded WebLogic, where he led development of the first independent J2EE application server. BEA later bought WebLogic and Oracle bought BEA. [1] Now he’s pursuing a PhD in history at the CUNY Graduate Center. To support his research, he developed a machine-learning tool he calls Roscoe to discern new insights in American legal history.

Today’s interview excites me for two reasons. It’s always good to spend time with a kindred spirit passionate about both history and technology. More importantly, using AI to discover more facts intrigues me no end. In an academic environment it allows us to understand ourselves and our world better. In a corporate environment it allows us to make better decisions and pursue better strategies.

Here’s the takeaways:

1. Wall Street, early internet experimentation, WebLogic and J2EE, post-acquisition chapters, then a PhD in history — technology innovation and academic aspirations can reinforce each other!

2. Enterprise stacks still echo mainframe-era problems, but the internet forced looser transaction models, distributed-systems humility, and resilience design given shared infrastructure and unreliable networks.

3. Mass digitization of historical documents means you can ask new questions at scale, but you have to cut through the “silence of abundance.”

4. Roscoe is semantic retrieval across collections — embeddings, ETL, metadata, re-ranking — aimed at evidence that keyword search will not find.

5. The hard problems ahead are precision, recall and cultural acceptance, richer analysis of hits, multimodal corpora, and partnerships with archives. Yes, the interpretive payoff is substantive, but attachment to existing methods bedevils the academy no less than the enterprise.

1. From Wall Street and the VAX to WebLogic — and back to graduate school

Bob traces Wall Street transaction systems, the Java and Usenet milieu, co-founding WebLogic and shipping the first J2EE implementation, later ventures and CTO-for-hire work, and his return to history for a PhD at CUNY.

James Kaplan: James Kaplan here with the Prosaic Times podcast. I’m very pleased to have Bob Pasker with us. He played an important role in the development of modern software over the past couple of decades, and he’s now doing exciting work as a history PhD candidate. Bob, give us a little bit of your background as a tech leader and entrepreneur, and what drove you to pursue a PhD in history.

Bob Pasker: That’s correct. Thank you very much for having me on, James.

I started way back on Wall Street, building transaction processing systems on the good old VAX, if you remember that. I got interested in moving to San Francisco because I wanted to be part of the tech community in Silicon Valley. I moved to San Francisco and started working for one of the database companies back then. It wasn’t a great company and I didn’t stay very long, so after about ten years in the computer industry I decided to go back to college.

I went back as a history student, got an undergraduate degree, and decided I was going to be a college professor. I went out to graduate school as a history student. Graduate school didn’t really agree with me at the time, so I came back to San Francisco just about when Java was coming on the scene.

I decided to do some experimentation. I built an SNMP stack in Java and was able to browse public servers on the internet completely undetected, because nobody had Cloudflare or anything like that at the time. I was active on Usenet, if you remember that — before Friendster and MySpace. It’s how people communicated with each other over the internet. I met other people who were interested in enterprise Java, and together we co-founded a company called WebLogic.

Our first products were JDBC drivers for accessing Sybase and Oracle. My interest, though, was building a transaction processing system in Java. I spent eighteen months building the WebLogic application server, which is what it became known as. That kind of launched the whole enterprise Java thing. Sun adopted our model, if you will, of having all these different services available, and they called it J2EE — and we had the only working implementation out of the gate.

The WebLogic company was acquired by BEA in 1998, and BEA was acquired by Oracle in 2008. That’s how it became part of Oracle’s technology stack. After that I started another company to do what we called the real-time internet. This was around 2000. The idea was server push instead of request-response, which is something we now take for granted. It was fairly early technology, and we didn’t make much progress because we got caught up in the dot-com bubble.

Over the next ten years I spent time at various venture firms, including Accel Partners, and I was kind of a CTO / chief architect for hire at VC- and PE-backed companies. I also spent a year at Expedia rebuilding their enterprise architecture.

Getting into the COVID years, as my kids started to get older I had more free time, and I decided to go back for my PhD in history. That’s how I wound up at the CUNY Graduate Center as a PhD student in American history.

James Kaplan: As someone who’s passionate about both history and technology, I applaud your varied career.

Bob Pasker: Thank you. I’ve gotten to do a lot of different things. Sometimes I feel like I’ve had five or eight different careers instead of just one.

James Kaplan: It’s a hell of a lot easier to understand the monumental changes happening now if one has a bit of historical sensibility and a historical mindset.

Bob Pasker: For sure.

2. Enterprise architecture: what changed, what didn’t, and why resilience still wins

VAX-era patterns persist, but networks are more decoupled; ACID loosens over the public internet, fallacies of distributed computing still apply, and major outages remind us nobody is immune.

James Kaplan: Before we dive into some of the research you’re doing now, any reflections on the evolution of enterprise architectures over the past twenty years? Obviously the app server was a tremendous advance. We’ve moved on to containers and so forth — but any reflections on that arc?

Bob Pasker: I think the biggest thing is that the more things change, the more they stay the same. A lot of what we were doing back on the VAX a long time ago still gets done today in enterprise systems. But I think the biggest change has been that networks and computer systems are designed much more robustly and to be much more decoupled.

We always thought about ACID transactions and the problem of doing ACID-style transactions over long distances. It was very easy to do inside the data center, but banks and other companies required that either both sides of the transaction completed or neither did.

Now we have much more flexible ideas about how that happens, and a lot of it has to happen over the internet, with the unreliability that implies. So we’ve really taken a new look at ACID transaction ideas and relaxed them enough to make it all work over the internet.

James Kaplan: I’m old enough to remember when many people didn’t believe servers could be physically remote from one another — a combination of the fragility of the application architecture and the fragility of the network. If the servers weren’t in the same facility, they didn’t have confidence in the ability to transact in a robust way.

Bob Pasker: Absolutely. I used to send people the list of the fallacies of distributed computing. One of them is that the network is reliable — and it’s not. Another is that they’re all managed by the same person — and they’re not. Whoever wrote that list was very prescient about what we needed to do to build reliable systems on the internet.

James Kaplan: You’re making an important point that belongs under “the more things change, the more they stay the same”: it’s critical to design for resiliency. You can’t assume any given component will be perfect, so you have to design a system that’s robust in the face of degradation.

Bob Pasker: Absolutely — that’s basically where we are today. We still see it every day: status.x.com and all the other status pages keeping track of what’s working or not on the internet, and we’ve taken that to heart. The biggest catastrophes are when major pieces of infrastructure go down. We’ve seen it with Cloudflare and Amazon and all sorts of companies — nobody is immune.

James Kaplan: I remember spending a lot of time on geo-resilient architectures starting around 2010 — application and system architectures that would be resilient in the face of network failure or infrastructure downtime. Let’s pivot to the research you’re doing now.

Bob Pasker: Sure.

3. Legal history, massive digitized corpora, and the dissertation problem Roscoe was built to solve

Early spreadsheet-era legal history meets an eight-million-case digitized corpus — motivating Roscoe and a dissertation on nineteenth-century courts mediating ideals of liberty against laws of slavery, suffrage, and Native policy at scale.

James Kaplan: I asked you to join because I’m fascinated by digital humanities and the extent to which we can use AI and other digital techniques to enhance historical understanding. Could you tell us a little about the historical research you’re pursuing now and what got you interested in that topic?

Bob Pasker: When I was an undergraduate and in my first attempt as a graduate student, I became interested in legal history. For me it’s a unique field. At the time there were many untapped sources of legal documents that historians had never or rarely used. Personally, law had been a longstanding interest of mine, and I like to do things that are a little off the beaten path — that’s what I wound up doing.

I wrote papers on legal history: kinship and the way people left money to their children in the eighteenth century, based on published wills; and a paper on sex crimes in Providence, Rhode Island, in the late eighteenth and early nineteenth centuries. This was all done with word processors and spreadsheets — so I was doing digital history even then, taking the documents I was reviewing, putting them into spreadsheets, tabulating, and so on.

Fast-forward to 2023: Harvard Law Library has transcribed the entire corpus of American appellate case law — about eight million cases — and digitized them, so you can download text files of all of those cases. I decided to combine my two fields, history and computers, build a conceptual search engine for that case law, and use that search engine for my dissertation research.

The system is called Roscoe. It’s named after the first legal historian, Roscoe Pound, who lived from 1870 to 1964 — he really started the field. My dissertation is a study of how, in the nineteenth century, American courts became the venue for working out conflicts between our constitutional ideals of freedom and liberty and the actual law that permitted slavery, denied women’s suffrage, and affected Native Americans.

A lot of historians have studied these topics; they’ve only used case law as evidence. Nobody has used case law to study the court system as an institution itself — on par with the other branches of government, religious institutions, and industries. The reason nobody could do that is scale: there are 226,000 cases up through 1860, and that’s the basis of my dissertation research.

4. How Roscoe works — embeddings, collections, ingestion — and how historians react

Semantic search over multiple public-domain collections via embeddings, vector indexes, relational metadata, and re-ranking — plus the human story of academic uptake, the ninety-three-case find, and “silences” created by bad retrieval, not missing archives.

James Kaplan: Tell us a bit about Roscoe. How does it work? Take us under the hood a little.

Bob Pasker: Roscoe is a semantic search engine. The idea is to replace arcane keyword and Boolean searches, which is how most archives still work. If you want to find case law on a particular topic, you have to know the exact words they used back in the nineteenth century — and the words they used in Georgia versus New Hampshire.

If you’re interested in a concept like canal building, you’d have to look up locks, canals, waterways, and so on to surface all the relevant documents. With Roscoe, you type something like “disputes over canals,” and it surfaces documents related to that concept.

The fundamental technology is embeddings and a vector database. An embedding takes a piece of text and turns it into a high-dimensional vector. That vector can be stored in a vector database; you embed the query, look it up in the database, find the k nearest neighbors, use those neighbors to look up the specific cases in Roscoe, and hopefully those cases are conceptually similar to your query.

I’ve organized Roscoe by collections — each collection has its own vector database. The first collection was those 226,000 cases. I’ve extended it to another collection called Chronicling America, which is millions of nineteenth-century newspaper articles. I have another collection with the papers of the founders, and also the Congressional Record. These are all public domain, and each collection is available inside Roscoe.

What makes Roscoe different is that you’re not searching one database at a time with arcane keywords — you’re searching across all of them at the same time. The key is the ingestion process: an ETL layer — extract, transform, load — that takes data as it comes from the archive, tests different chunking algorithms and embedding strategies, creates an index in a vector database, and cross-references that with a relational database that holds document metadata — names, dates, location — used for filtering and re-ranking. That’s basically how it works underneath.

Version one had a very simple user interface that produced a result table with metadata. Version two has multiple collections, searches across collections, and does unified re-ranking: it takes results from the different collections and re-ranks them against each other so the most relevant results rise to the top, regardless of which collection they came from. That’s basically how version two of Roscoe works right now.

James Kaplan: What’s been the reaction from people you interact with in academic history? I ask because some academics I know are incredibly excited about what AI can do for research, and others push back — anything involving quantification, or “that’s a science way of thinking, not a humanities way.” What’s the balance of enthusiasm versus skepticism?

Bob Pasker: It’s similar to the experience I had trying to get people to use WebLogic. There’s a whole lot of people who couldn’t care less, and a very few who are really interested and see the value. So there’s a huge evangelization process — different from a startup, but still a big thing.

I’ve had professors who, when I’m writing a paper using Roscoe, say: I don’t want anything in the paper about technology — I just want a history paper. I’ve had others who are extremely helpful and excited — but to be honest they don’t really understand it. They can conceptualize the benefit, but until it becomes a public utility they can try out, with enough collections for their own work, it’s mostly curiosity rather than adoption.

Right now I’m trying to write some papers using Roscoe. I’m working on a paper about how to explain Roscoe to the community of historians, which turns out to be fairly difficult — but I’m making progress, and I hope to publish it as an independent research paper. It’s not meant to be pure evangelism; it’s meant to ground Roscoe in historiography, the process of doing history, and archival science — what it means for both disciplines.

James Kaplan: It strikes me as historiographically important. A professor described to us how certain historians were paging through records to find birth and death dates to understand lifespans in early nineteenth-century England and how the industrial revolution changed mortality — whether it increased or decreased mortality in different places. Your approach is a way to vastly increase the datasets available to historians without sending grad students to page through bound volumes by hand.

Bob Pasker: Yes — and in a sense that’s a slightly different kind of digital history: it’s tabular. There’s been a lot of work since the late fifties on tabular analysis of data, the way an economist might do. I’ve been interested in that too; I did it in those earlier papers.

Roscoe is very different. It’s for finding documents that already exist in archives but are impossible to find. My paper last year was on whether Black people could testify in courts before the Civil War in the nineteenth century. The laws were basically against it, and we don’t have much conception that it was still a possibility. Using Roscoe I found ninety-three cases out of 226,000 — about four in ten thousand — in the appellate court records that would otherwise have been impossible to find. They span from the 1790s to the 1860s across eleven different territories and states. There was no keyword search in the world that was going to find those ninety-three.

Archivists have this idea of archival silences: what archivists admit into their archives. For the most part archives contain documents they consider important and leave out what they thought marginal or uninteresting — they have to curate; we can’t save everything.

I have a different kind of silence in mind: documents that are in existing archives, useful to historians’ research, but that they can’t find because they can’t come up with the right keyword search in the user interface. My paper argues that Roscoe makes it possible to find those — that there are interesting documents that have been, in a sense, silenced by arcane interfaces. That’s what I’m trying to create: a system that surfaces many more interesting documents than a historian would otherwise find.

5. What’s next: precision and recall, multimodal search, partnerships — then evidence, interpretation

Roadmap: recall versus precision, deeper per-hit explanation, map-level multimodal search — then partnerships and “index not copy” for archives, historiography of evidence, reading the ninety-three cases, and why he prefers “machine learning” to “AI.”

James Kaplan: To push one level further — and this is a little about where Roscoe might go — ninety-three cases you could read yourself, but you can imagine a search that surfaces a thousand or fifteen hundred cases. To what extent do you think the state of the art will advance so you can use analysis to identify trends in legal thinking? Could some of these documents go into a graph so you can see how legal thinking in one set of cases influenced another? What comes next after archival search? Does that make sense?

Bob Pasker: Yeah, it does.

Version three of Roscoe, which I’m already working on, will address some of this. First, on returning too many results — that’s well known in information science: recall versus precision. You want enough cases in your result set that you see everything useful, but you don’t want false positives — things returned that aren’t useful. Search engines have dealt with that for a long time. You also want precision: the cases most relevant to you should rise to the top, and the useless ones should drop out. You don’t want to leave useful cases outside the result set, and you don’t want useless cases inside it. I work on that constantly: refining the system for better precision and recall.

Second, I want deeper analysis of how each case relates to the query. In version one, as results returned, the system re-ranked them and analyzed cases more deeply to identify exactly how each case related to the query. Once you have a large result set, you can go through it more deeply with machine learning to pull out the cases specific to what you’re looking for and leave out the rest.

Another direction I’ve experimented with is visual search — my experiments have been with maps. Old maps are crude line drawings with handwritten type. In the archive you get: “Here’s the Smith map of New York City from 1823” — and that’s all; it doesn’t tell you what’s on the map until you open it. I’ve used machine learning to read the maps, identify places written on them, get latitude and longitude, overlay them on modern mapping systems, and identify features — waterways, canals, mountains, farms. That information goes into a vector database so it can be searched semantically.

So when someone searches “disputes over canals,” you get not only case law, debates in the Congressional Record, and newspaper articles, but maps where those disputes actually took place — spatial context as well as temporal context from the dates. I think you can do that for other artifacts too: paintings, sculpture, textiles — so people doing research on material culture could search catalogs, say at the Museum of Natural History or the Museum of Modern Art, and find artifacts related to their topics.

James Kaplan: What’s the toughest thing technologically — where is the technology there, and where is it harder?

Bob Pasker: I guess I’m an optimist: I think I can build something really fantastic here. The hardest part isn’t really the technology. It’s twofold: one, making it useful to historians in a way that comports with historiography and archival science; two, building relationships and partnerships with libraries and archives so they’re interested in doing this without feeling they’re giving up their walled gardens around these materials.

The good thing about how Roscoe works is it doesn’t duplicate the archive — it creates an index, the way a card catalog is an index, not the contents of everything. Those are really human difficulties more than technological ones. We’ll keep wrestling with precision and recall and the right way to visualize and display what’s useful. At this point I don’t see anything I can’t get out of the technology.

James Kaplan: It’s potentially disruptive within the history profession in the sense that, over time, techniques like this could make history even more of an empirical than a theoretical discipline — ground it more tightly in the historical record by accessing a broader set of documents easily.

Bob Pasker: How historians use evidence is itself a historiographic topic — it goes back to ancient Greece and the way Thucydides used evidence to describe what happened. That had a transformation in the nineteenth century as people became more interested in an evidentiary basis for history rather than only the stories they had told. The rational, evidence-based side of history has been developing for a couple of hundred years.

I think this extends the same trajectory as building archives, electronic card catalogs, transcriptions, photocopies, seeing old documents as images on the web. I’m a novice historian; others could expound on this much more. But I think Roscoe is doing what needs to happen given the scale of digitization and transcription at the archive level — making huge corpuses visible. I don’t think it’s disruptive; I think it’s enabling.

James Kaplan: Hearing what you just said, it’s a continuation — the next evolution in a long series of transitions over the past couple of hundred years, increasing the dataset available to historians as they do history.

Bob Pasker: Yes — and that’s what my paper this semester is about: what this deluge of information means for historians and how Roscoe will help. I call it the silence of abundance: what’s hidden in this great abundance of historical records.

James Kaplan: As you looked at those ninety-three antebellum cases, was there anything especially insightful that wouldn’t have been available if you hadn’t found them?

Bob Pasker: The fact that those cases exist at all. For the most part we “know” that Black people were not allowed to testify in court — not as witnesses, they couldn’t give evidence. But now we see: wait, that’s not completely true, even given what the laws say.

I had hoped I’d find justices who really wanted to give people an opportunity to testify on their own behalf or on behalf of something they had seen — in a positive, rights-expanding sense. That’s not really why they were allowed to testify. They were allowed to testify because nineteenth-century justices had a very specific concept of justice. It wasn’t liberty and freedom in the abstract; justice was the process of adjudicating cases.

So you had very specific situations in these ninety-three cases: someone was injured, the only witness was a Black man, everyone knew the person was injured and the defendant was guilty — but there was no witness except this one Black man. The only way for the justice system to maintain its reputation as an institution that could adjudicate cases was to let that witness testify. Otherwise it would be as if nobody had seen it, the defendant would go free, and that would violate their notion of procedural justice. It was more about maintaining institutional coherence than about a grander sense of justice. That was my conclusion.

James Kaplan: Very helpful. Anything I neglected to ask — anything else you’d like to cover?

Bob Pasker: As I’ve talked to historians — classmates, people in my department — and looked at what historians’ associations have said about artificial intelligence: by the way, I don’t use the term “artificial intelligence” because I find it unhelpful. I use “machine learning,” the technology I use, without the generative piece.

The resistance to something like Roscoe — what’s often lumped as “AI” — comes down to three concerns. One is hallucinations, which we’ve discussed. Two is teaching — how this affects pedagogy. Three is the human aspect of writing history: history is a process conducted by humans, not machines, because history gives us a sense of who we are, where we came from, the story of our path — and humans should own that, not computers.

I’m hoping something like Roscoe, which uses the same underlying technology in a different way, will have a positive impact — people will understand it and find it useful in their research. It may take ten or twenty years, another generation of scholars, before that really bears fruit. I’m enjoying being at the forefront and I’m proud of what I’ve done so far.

James Kaplan: Congratulations. Thank you so much.

Bob Pasker: Thank you. I really appreciate it, James. This is something I’ve wanted to talk about for a long time.

Footnotes

[1] For our younger readers: Before containers, app servers provided transaction management, connection pooling and a runtime environment for J2EE applications.

On October 4, 1957 the Soviet Union launched Sputnik 1, the first man-made object to achieve Earth orbit. In the Right Stuff [1], Tom Wolfe described the shock and dislocation felt by American elites. They had built the arsenal of democracy and exploded the atomic bomb. And now a backward former supplicant, one that required American trucks to hold off the Wehrmacht, had beaten them into space. What did the United States have? An underfunded, shambolic collection of civilian and military programs designed to satisfy bureaucratic and diplomatic imperatives rather than for speed and effectiveness.

They responded. Then Senate Majority Leader Lyndon Johnson said Americans would not go to sleep by the light of a Communist Moon. The National Aeronautics and Space Act of 1958 created the National Aeronautics and Space Administration (NASA) with responsibility for the American space program. The National Defense Education Act of 1958 sought to dismantle John Dewey’s legacy in American education, pushing schools to replace “life adjustment skills” with set theory and symbolic logic. Less than 12 years after Sputnik, Neil Armstrong and Buzz Aldrin walked on the surface of the Moon.

Could Anthropic’s recent announcement of how Mythos can identify and exploit cybersecurity vulnerabilities create the Sputnik moment that will spur companies to use AI to change the way they operate enterprise technology?

The risks are real, and companies will need to move beyond buying tools and to build an agentic governance loop that uses a living graph of the environment to provide the context for spec-driven, immutable engineering, verified by adversarial automation rather than manual bottlenecks -- and then sustain and expand this change over time.

• Despite early indicators of transformative improvements, AI adoption in running enterprise technology has been shallow.

• Despite some fear-mongering, the impact of AI on the cybersecurity balance of power between attackers and defenders has been muted to date -- Mythos and subsequent models could change that.

• Mythos and subsequent models could dramatically improve companies’ cybersecurity posture in the medium term -- but they will need to use AI to accelerate their enterprise technology metabolism dramatically.

• Of course, the idea of a Sputnik moment is as much a warning as a call to action -- one-time programs are a lot easier than sustained cultural change.

AI adoption in running enterprise technology has been disappointingly shallow

Technology engineering and operations is one of the most exciting applications of AI for large companies. Large language models excel at interpreting and generating the structured content used in software engineering or technology configuration. AI can replace procedural programming with declarative programming [3], via spec-driven development. Agentic processes can better accommodate the edge cases and exceptions that have historically bedeviled efforts to automate technology operations. The early results have been exciting. My McKinsey colleagues have found that using AI to reinvent engineering processes can double team throughput. AWS has started to use agentic processes to reduce incident resolution time by three-quarters in some cases. Applied ruthlessly, AI could transform the economics of enterprise technology.

Yet adoption has been shallow. In last year’s DORA State of DevOps report, 90 percent of software practitioners said they use AI in some way, but most never used it in agent or autonomous mode and only 17 percent used it every day. The situation is no better with the cybersecurity team. According to a SANS report, Security Operations Centers use AI/ML tools, but don’t integrate them into their processes:

AI is present inside the SOC but not operationalized. Analysts use it informally, often with mixed reliability, while leadership has not yet established a consistent model for where AI belongs, how its output should be validated, or which workflows are mature enough to benefit from augmentation.

All this accords with my own observations: technology teams use AI as a tool to generate a code snippet or research an issue, rather than a lever to rip toil out of the way they do business. Why is this? The technology is still relatively new. Teams may be cautious or may not have the mental bandwidth required for change. Vendors have promised just installing a tool will solve their problems. And CIOs have not built the institutional support required to fund and prosecute the required change.

Mythos could change the cybersecurity balance of power between attackers and defenders

Since OpenAI released ChatGPT 4.0 in 2023 the great and the good have warned us about AI-enabled cyberattacks. The World Economic Forum said that specialized language models would allow hackers to get around endpoint security devices. The FBI said that AI would allow criminals to scale fraud schemes in a way that would swamp law enforcement. The UK’s National Cyber Security Centre said that GenAI lowers the barrier to entry for novice hackers allowing them to use vectors previously only available to experts. Some predictions approached fear-mongering -- sentient malware and HackerGPTs collapsing cybersecurity defenses. [4]

The worst...has not happened. I checked this morning, and the digital world continues to function. Only 16 percent of companies suffering breaches said they saw evidence of AI in prosecuting the attack. According to the Verizon Data Breach Investigations Report attackers have been just as dilatory as enterprises in using AI to reinvent their business processes:

It turns out the state-sponsored actors are just like legitimate organizations in their GenAI implementation life cycles. Attempts are being made, maybe some improvements are being found, but no one is revolutionizing anything yet.

At least as of 2024, GenAI tools could potentially assist attackers, but could not execute sophisticated attacks for them. One analysis found that GPT-4 only achieved a 7 percent success rate in exploiting vulnerabilities without clear human guidance.

Even before Mythos, the potential and the direction of travel have been worrisome. The structural factors that make LLMs effective in building and running systems also apply in compromising them.

• Intel matters in undertaking a cyberattack. LLMs have breadth of vulnerability knowledge no human analyst can read or retain -- LLM training data spans public CVEs, security research, disclosed exploits, and documented attack strategies.

• Success requires patience. Agents will cycle through potential vectors without boredom or fatigue.

• System compromise provides agents with a clear objective function they can optimize against.
  As a result, researchers have started to demonstrate that teams of LLM agents can cooperate to exploit zero-day vulnerabilities

Then came Mythos. Obviously we should be restrained in thinking about the implications of any software that isn’t generally available yet. And we’ve heard the too dangerous to release warning before. In its public statements, Anthropic said that Mythos had identified thousands of high-severity vulnerabilities across major operating systems and browsers—including legacy flaws like a 27-year-old bug in OpenBSD that evaded decades of manual audits. The model further demonstrated the ability to build complete, working exploits. Mythos can independently “chain” multiple vulnerabilities to gain a foothold, escalate privileges, and move laterally through a network, effectively allowing users with no formal security training to execute professional-grade, multi-stage cyberattacks at machine speed. Finding zero-days may get the headlines, but the ability to scale and operate autonomously may create the real risk.

This probably won’t realize the most dire predictions of 2024. [5] Several commentators have observed that a model’s ability to identify vulnerabilities and form plans doesn’t mean it will succeed in the face of sophisticated defenses (including the ones they have developed). But how many companies have sophisticated defenses like zero-trust in place comprehensively? And one compromise in the software supply chain could disable hundreds or thousands of institutions. Naturally, Anthropic and other frontier labs will seek to implement guardrails that limit attackers’ ability to exploit their models. The guardrails will not be perfect. And they will not apply to many of the open-weight models that will likely have Mythos-level capability within, maybe, a year.

Mythos and subsequent models could dramatically improve companies’ cybersecurity posture in the medium term. Could, not will.

After the Mythos announcement, American business and governmental elites acted. Anthropic delayed general availability of Mythos and launched Glasswing, giving early Mythos access to leading technology institutions so they could use it to identify vulnerabilities. Treasury Secretary and Federal Reserve Chair Jerome Powell called banking CEOs to Washington DC so they could urge them to take the risk seriously -- I expect they were pushing on an open door. Technology companies like AWS, MSFT, CRWD and CSCO reported that they were using Mythos to harden their products.

In the medium term, Mythos (and subsequent models) could provide a dramatic uplift in cybersecurity defenses. Companies spend fortunes each year scanning their code for vulnerabilities [6] -- Mythos-type capabilities will provide a level of transparency into vulnerabilities that we never could have imagined before. Most companies of any size do penetration testing, [7] but only the biggest tech spenders have dedicated red-team operations that figure out how a sophisticated attacker might compromise their environment. Mythos-type models should make this capability available to a much broader range of companies.

They may also revolutionize cybersecurity risk management and cyber insurance. Cyber-risk valuation frameworks like FAIR have foundered on the problem of likelihood assessment. Practitioners should be able to use a model like Mythos to simulate attack paths, determine the probability of success and make more fact-based remediation decisions. It could also revolutionize cyber-insurance, a segment historically held back by underwriting challenges.

And yet -- speed matters, and manual remediation is too slow. Mythos can help companies identify vulnerabilities. But identification protects nothing unless companies apply security patches from vendors and install fixes to code they have developed internally. That is the remediation gap the governance loop above is meant to close; in practice it breaks down into three concrete moves:

1. Create a living graph of your technology environment. You will very quickly face an overwhelming pipeline of vulnerabilities to remediate and vendor patches to apply. Not every one will be equally important, and the most critical nodes in your environment may not be immediately apparent given all the dependencies among business processes, systems, data and technology infrastructure.

Modeling your environment as a graph will allow you to identify the most critical nodes and prioritize what to remediate first. Ultimately every node in the graph should anchor in a non-human identity -- don’t connect IP addresses; connect non-human identities. Building the graph will also be an important step in moving to a zero-trust architecture.

2. Use spec-driven engineering to get to policy-driven systems. If you have bespoke software you will need to fix it. Autocomplete (or even asking models to write discrete code blocks) will not allow you to move quickly enough.

You need to retrain your engineering teams on how to use agents to diagnose root causes, build PRDs and execute on them autonomously. And you may need to do this on a timescale of months, not years.

As you develop strong capabilities in spec-driven development, you can accelerate efforts to retire technical debt, resulting in a more resilient environment. And you will want to define architecture, configuration and behavior in terms of policy-as-code so you can repave systems that demonstrate drift.

3. Move change control from human analysis to proof of safety. In many companies, the change approval board acts as a brake and a bottleneck on evolving the environment. It doesn’t have to be this way, and it cannot continue to be this way if companies seek to remediate the vulnerabilities Mythos identifies before attackers can exploit them. Heavyweight change approval processes are often ineffective. Teams of agents may collaborate to form an automated patch management pipeline.

Before you deploy a change, it must prove itself in a sandbox, both in terms of whether it breaks something and whether an adversary agent can compromise it, replacing the bottleneck of human analysis with the proof of safety. And you should deploy changes in stages, testing impact as you go. For years companies like Netflix have reconciled speed and safety by using canary analysis for staged change deployment.

None of these interventions are simple. [9] All will take attention, effort and time. But what is the alternative? Outsourcing might help, but it doesn’t remove the remediation burden at a stroke. Waiting for regulatory guidance (across dozens of jurisdictions and agencies) is uncertain and will likely take too much time. The age of security by obscurity is over. The cost of stasis may exceed the cost of change.

One-time programs are a lot easier than sustained cultural and organizational change

Less than a dozen years after Sputnik, Neil Armstrong and Buzz Aldrin explored the Sea of Tranquility. Ten more astronauts walked on the Moon in the next three years. Then, nothing. What poverty of the human spirit, what richness of bureaucratic incompetence caused us to tread on the moon and then retreat, without returning? Only this month has any human again transcended low Earth orbit?

Sputnik was a shock to the American educational system. By the 1980s, the National Science Foundation warned that Americans were in danger of scientific illiteracy. Shortly afterwards, the famous Nation at Risk report warned that post-Sputnik gains had wasted away. Not all the news is bad! American students have made large gains in fluid reasoning in recent decades -- and the dire standing of American students in global league tables may have more to do with compositional effects than school performance. But some of the news is really bad -- math scores have collapsed in the wake of Covid.

Just like a space program, the exploitation of AI in the enterprise is a generational project. Just like education reform, the ability of enterprise technology to use AI to build, run and protect systems is a foundational capability. Will your company move quickly enough to respond to the immediate challenge posed by AI-enabled cyberattacks? Will it sustain focus and attention over time to foster the capabilities to use AI not only to protect existing systems but to also make transformative leaps in business innovation, efficiency and resiliency?

Footnotes

[1] A great book. I read it every year in high school. The movie is pretty good too.

[2] Might I be evoking Arnold Toynbee’s theory of civilizational challenge and response here? Maybe.

[3] SQL is overwhelmingly the world’s most used declarative programming language. Perhaps its adoption provides the best historical parallel for spec-driven development. Replacing all the technical minutiae required for a query with a few SQL statements turned weeks’ worth of work into minutes.

[4] References not provided in order to protect the guilty.

[5] At first glance, you might ask: “How much does this matter for the enterprise? Once you get past the national security domain, how many attacks rely on zero-days?” More than you might think: a Mandiant analysis found that 70 percent of serious breaches they tracked involved a zero-day exploit. And of course a capable agentic attacker could assemble a sophisticated campaign out of a series of n-day exploits.

[6] The global security and vulnerability management market will shortly grow to the USD 20 billion.

[7] Itself a billion-dollar market.

[8] Special thanks to my colleagues Rich Isenberg and Charlie Lewis on these topics.

[9] These changes will require coordination across the technology organization. The infrastructure team will likely have to build the living graph of the environment, with input from each application team. Your core architecture or engineering team will lead the transition to spec-driven development, but much of the work will fall on application teams—and on infrastructure teams as they move from configuring systems to automating services. Transforming change control and patch management will require collaboration across the developer toolchain, infrastructure, and cybersecurity teams.

This was great in that it demonstrated value, created demand and accelerated organizational learning. This was less great in that created risk and technical debt. Sometimes it foreclosed future options and turned out to be like trying to get to the moon by climbing the tallest tree -- the first fifty feet feel like great progress.

So too with agents. Enterprise technology functions must go quickly to meet business expectations, but also build the platforms that support agentic workloads with resiliency, security and efficiency.

Kong is a provider of API and AI gateways. I spoke with their CTO Marco Palladino to talk about the evolution from API to AI gateways and how AI gateways fit into a broader agentic strategy.

Some of the most interesting comments from our discussion:

• “The dream has always been the same: we can build an assembly line of software so we can create new products faster. We can innovate faster by taking existing APIs and assembling them together — like the Ford assembly line, but for software.”

• “AI is useless if agents cannot use APIs. Agents consume two things: they consume LLMs for intelligence, and they consume APIs to do something with that intelligence.”

• “Not appreciating non-determinism in AI is like organizations in the early days adopting containers but deploying them one container per virtual machine. You’re using containers, but not really — you’re losing the whole point.”

• “We’re going to have agents telling agents to go build agents. It’s going to be agents all the way down.”

• “But when the agent becomes the buyer, all of a sudden the agent doesn’t care about the billboard. It doesn’t care about the YouTube video. How does an agent decide what product to use? It looks at documentation. It looks at examples, getting-started guides, what other agents have been doing.”

James Kaplan: This is James Kaplan with another ProsaicTimes podcast. I’m here with Marco Palladino from Kong to talk about AI gateways, API gateways, and security in the AI era. Marco, welcome.

Marco Palladino: Thanks for having me.

James Kaplan: Great pleasure. Give us a little bit of your background. Tell us about your journey to Kong.

Marco Palladino: I’m the CTO and co-founder of Kong, and I’ve been doing this API thing for fifteen years now — since the days when people were asking us what an API was. Now, obviously, everybody knows what an API is, and APIs are the backbone of pretty much the digital world, including AI.

So I started Mashape, which was an API marketplace that didn’t go anywhere — it was too early. We then open-sourced our core technology, and that became Kong. Kong started as an API management product, but over time it expanded to encompass all of connectivity: API connectivity, AI connectivity, microservices connectivity, mesh connectivity.

Today we work with approximately 1,000-plus enterprise customers around the world, and we’re very excited about what’s next.

James Kaplan: Some of us are old enough to remember when an API was something exposed from a Windows Dynamic Link Library, and you had to make sure you had the right version of the DLL in the right directory so your application could consume the right version of the API.

Marco Palladino: Everybody remembers DLL Hell.

James Kaplan: Tell us about how central API gateways were in the evolution of a modern artificial intelligence stack. What makes for a good API gateway? What makes for a good user of an API gateway?

Marco Palladino: The concept of an API has been there since forever. Even in the SOA world of the early 2000s, there was always this dream of web services we could use, consume, and integrate into new applications — but it didn’t go anywhere. It was too complex. Nobody wants to write SOAP or consume SOAP. And then new programming languages emerged — JavaScript, Ruby on Rails — where SOAP was very hard to consume. So it really didn’t get any traction whatsoever.

Then RESTful APIs, the modern ones as we know them, really became popular when Google created their own public APIs and Facebook had APIs that let people access the social graph. All of these apps were talking to monolithic backends through an API.

The iPhone really created a whole new need for APIs — not only to create ecosystems of integrations, but also to handle internal communication. We needed an API to connect our app to the backends, which were monolithic back then.

Then something happened in 2013: Docker was invented and released. In 2014, Kubernetes was created and released, and all of a sudden APIs — which used to be an afterthought bolted on top of monolithic applications to connect with a mobile app — were there since day one. We were building API-first because we were building microservices running in containerized Kubernetes environments across one or multiple clouds, and APIs as a means of internal communication became quite essential.

The dream has always been the same: we can build an assembly line of software so we can create new products faster. We can innovate faster by taking existing APIs and assembling them together — like the Ford assembly line, but for software. And I think what we’re seeing now is the latest iteration of APIs as the backbone of the digital world when it comes to AI.

There is no AI without APIs. We need an API to consume the models. We need an API to consume MCP tools, the data, and the systems and services that we want our agents to use. APIs are everywhere. If anything, APIs are going to keep increasing in numbers.

James Kaplan: I like to say every problem in computer science gets solved by abstraction, and APIs are a mechanism for abstraction. Let me ask you about adoption. We all know there’s pervasive API usage in the consumer internet and enterprise SaaS space, especially externally facing. I was wondering if you could comment on the traditional enterprise — banks, pharmaceutical companies. How far did they get in terms of creating APIs that allowed different applications, or different components of applications, to talk to each other? I think that has historically been a bigger lift.

Marco Palladino: You really only have two options: either you have APIs, or you have silos. APIs help us break silos. Many companies have not invested in APIs, or they’ve done it in a non-coordinated way — maybe they have APIs within one team or one product, but they don’t have a repository of APIs that the rest of the organization can look at and use when building new agents, new experiences, new products. Only a few organizations have made that kind of investment, and those are the ones that are going to move and innovate a lot faster.

Look at the best example — and I know it’s a bit of an old example, but it’s still one of the best. Look at Amazon. At one point, Jeff Bezos wrote a memo in the mid-2000s basically telling every team: you have to build APIs for every new product from day one, and if you’re not doing that, you’re fired. APIs were mandated. And because of that top-down push from the CEO and founder himself, Amazon was able to create AWS. Amazon was not the only e-commerce company in the world, and yet it was the only one that was able to generate a multi-hundred-billion-dollar business with AWS — thanks to their API culture and engineering methodology. When everything ships with an API, you can not only integrate it, you can productionize it and start selling it and creating new revenue streams. That’s how AWS was born.

So what’s the status of APIs in the enterprise? You see a few organizations making the investment because they understand APIs as a path to innovate and move faster. And then you see lots of laggards — organizations that are just now waking up to the fact that they cannot capture AI if they don’t invest in APIs. Better late than never, but this is an investment that should have been made ten years ago.

James Kaplan: When you said the alternative to APIs is silos — at many places, the method of integration is still moving flat files around. What drives the laggards? For the folks who have not moved to a more API-forward architecture, is it lack of skills? Lack of investment? A belief that an API-forward architecture is harder to manage? What have been the barriers, in your thinking?

Marco Palladino: Nobody makes the call. I’ve been working with hundreds of top Fortune 500 and Global 2000 companies around the world — we’re a global company across North America and Europe — and I see patterns everywhere. There is a belief that if you build an internal platform that supports APIs, the teams will eventually adopt it. I think that’s only partially true.

At some point, it can’t just be the carrot. You show the teams what needs to be done, you show them that adopting APIs will benefit them. But at some point, you also have to use the stick. Many organizations and their leadership think APIs are someone else’s job, so they never make that top-down call. And eventually you have to call it for what it is: teams, we have to use APIs, because if you don’t use APIs, we cannot reuse anything you’re building in any other product, any other market, or with any other partner.

There must be a top-down leadership call — but it can’t be mandate alone. You can’t tell teams to adopt APIs and then not give them a platform to use. So yes, there needs to be technology, but at some point there needs to be a top-down leadership call as well. Otherwise, you’re not part of this engineering culture. Many organizations fail because they don’t want to make that call — and that can be deadly.

James Kaplan: One of the most interesting examples I’ve seen of a company adopting APIs — they took a very developer-services point of view. They said: we’re going to do all the things we would do if we were a third-party software company selling a developer platform. We’re going to invest heavily in documentation. We’re going to invest in developer support. We’re going to hold conferences with developers. I thought that worked nicely — it was an interesting change management mechanism. They really treated the developers as customers rather than as people who could simply be mandated to use a platform.

Marco Palladino: I fully agree, and that’s the right way to do it. Developers are internal customers — and APIs are products. They have a lifecycle. Part of the problem is that many organizations don’t see APIs as products, but they are. Just as we version websites and mobile apps, adding features and removing them, APIs have a product lifecycle too: we version them, create new features, decommission them. That’s work that needs to be done.

Developers who are busy building features after features need to slow down at some point, look at their API portfolio, clean it up, and treat it as a product. An organization can’t have a great API ecosystem without allocating the right amount of time for developers and teams to curate those APIs. But it pays dividends — once those APIs are consumable, you can reuse them in countless places.

And especially for AI — I can’t stress this enough — AI is useless if agents cannot use APIs. Agents consume two things: they consume LLMs for intelligence, and they consume APIs to do something with that intelligence. Without an ecosystem of APIs to hook into, those agents may be smart but useless, because they can’t connect to anything meaningful for the business. APIs unlock AI.

James Kaplan: Which creates a lot of issues around control and security. So let’s talk about the transition to an AI gateway. Where does an AI gateway fit into the system? How do we think about how an AI gateway relates to things like MCP and A2A?

Marco Palladino: When we think about agents, we’re thinking about smart applications. An agent is coded, runs somewhere. What does it do? It talks to an LLM to determine what the next operation should be, and then it talks to data and services to get the right inputs to do the job it’s supposed to do. For example, if we’re building an agent that does loan origination for a bank, we need access to APIs about our customers — their social security numbers, their KYC information, and so on — and we can’t do that without investment in APIs.

Now, agents using AI will also have to deal with the non-deterministic nature of AI. When we build a traditional application, we know what’s going to happen. But when we use AI, we’re giving the LLM some degree of freedom to determine what needs to happen with an input and what output to generate. That output is non-deterministic compared to how we used to build applications, and in a non-deterministic world, we need guardrails and capabilities to control what AI does. This is the whole area of AI governance.

As more teams build agents, we don’t want them to reinvent the wheel — building their own guardrails for each specific use case. The platform team can’t monitor what all these calls are about, and the organization doesn’t have confidence that agents are doing the right things at the right time. What we need is a platform that manages all these AI interactions: AI governance, AI security, AI optimizations — including the ability to compress prompts to get more out of the tokens we’re spending. All of that can be abstracted away from individual teams by the platform team, which offers it as a service to internal developer customers.

That’s effectively what an AI gateway does: it centralizes those cross-cutting requirements, just as API management did for APIs — where we took authentication, security, rate limiting, traffic control, and encryption and pushed them into a centralized place.

James Kaplan: It’s at least evocative of what cloud security posture management meant as we all started building out cloud architectures — an important control point that governed how you configured things in the cloud.

Marco Palladino: Absolutely. I’m a big believer that whatever technology we put in place for developers — whether to build APIs, AI agents, or microservices — needs to fit very naturally into their workflow and best practices. The platform is there to help, not to get in the way of getting to an outcome.

James Kaplan: So the AI gateway sits between the agent and the large language model externally, and between the agent and various tools — controlling or limiting what goes in and out of the institution, what goes to the large language model and what doesn’t, and which tools can be called under which circumstances. Is that a fair description?

Marco Palladino: That’s exactly right. It sits between the agents and the models and the tools. What is an MCP tool? Think of it as a new API protocol. APIs don’t have to be REST — they can be SOAP, REST, GraphQL, gRPC. MCP is a new protocol that makes it easier for agents not only to consume data or services, but also to discover what data and services are available. MCP bundles key requirements into the protocol itself: bidirectional real-time communication, tool discovery. It packages all of that in a very consumable way for agents.

With that said, it doesn’t have to be MCP. Many agents still consume APIs with traditional function calling and use any protocol they want. MCP has emerged as almost a standard stack for agent development — if you want to build something without overthinking it, MCP gives you an ecosystem that supports you. But it can be MCP or not. And the AI gateway sits in between not only all LLM transactions, but also all MCP tool requests.

Now, how large that MCP ecosystem or API ecosystem is will determine how much agents can do. If an organization hasn’t invested in creating an ecosystem of MCP tools or APIs, developers building agents will be somewhat handicapped — there’s not much for their agent to hook into. MCP is very exciting, and there’s more than just MCP — there’s also A2A, a different protocol that governs agent-to-agent interactions.

James Kaplan: Those of us who struggled with the syntax of Windows API calls in the 1990s appreciate that MCP allows for less finicky syntax in calling APIs.

Marco Palladino: At least we don’t have agents consuming CORBA APIs. There’s that.

James Kaplan: Some things are best left in the past.

Marco Palladino: Yeah.

James Kaplan: One of the things I find especially interesting is the application of non-deterministic controls. Preventing a social security number or credit card number from being sent somewhere is easy — it’s a pattern. Figuring out whether sensitive pricing data should be allowed to go somewhere is more context-dependent. Deciding how many tokens you want an agent to consume can also be very context-dependent — you may not want a hard cap; you may want certain agents to consume more tokens in certain circumstances and fewer in others. Could we talk a bit about non-deterministic controls and how you think about them in the context of an AI gateway?

Marco Palladino: Non-determinism in agents and AI is both a curse and a blessing.

James Kaplan: Of course.

Marco Palladino: Organizations that are embracing AI and doing it right are also embracing the benefits of non-determinism. Think about it: if we’re using LLMs and AI but not embracing non-determinism, then what we have is just a workflow. Why are we using AI at all? We can build workflows the old-fashioned way. Not appreciating non-determinism in AI is like organizations in the early days adopting containers but deploying them one container per virtual machine. You’re using containers, but not really — you’re losing the whole point.

Using AI requires an appreciation that there is going to be non-determinism in the outcomes it produces, and that is the power of AI. But we also need to make sure the outcomes can’t just be all over the place. While outcomes may not be perfectly deterministic, we need a range of acceptable options.

The best way to think about non-deterministic AI is in terms of risk management. When generating a loan for a customer, there’s a certain degree of risk the organization is willing to tolerate — a range within which we’ll generate loans and beyond which we won’t. Thinking about AI-generated outcomes has to work the same way: determine what risk you as an organization are willing to tolerate for specific outcomes, deny outcomes that fall outside that range, and fully appreciate the non-determinism within the range you’ve defined.

Many organizations are struggling with this, and because of it they’re struggling to generate outcomes with AI. The biggest outcomes any organization will generate are the ones tied to the core business — for a bank, that’s customer financial data, loans, money; for a healthcare organization, it’s claims processing. Because the organization doesn’t feel comfortable putting AI in the core business, it will never generate outcomes that are truly impactful. It’s a chicken-and-egg problem. We need to appreciate AI and invest in the right platforms for managing it — so we feel comfortable enough to put AI in the core business processes that will generate outsized outcomes. Without that investment, the organization will never trust AI in the inner workings of the business, and the CFO will eventually ask: we spent $100 million on AI — did we generate $100 million in outcomes? The answer is always going to be no if AI never found its place in the core business. Only by being in the core business can AI generate that return on investment.

James Kaplan: Imagine you have a CIO or CTO who says: I’m convinced we need to bring AI to the core of the business. I’m convinced we need a platform. I’m convinced we need an AI gateway. What are the major design decisions? What are the major architectural choices he or she might face, and what are the reasons that might push you in one direction versus another?

Marco Palladino: They have to think about centralizing governance. We want to decentralize the execution of AI, but governance is very hard to run well in a decentralized way.

James Kaplan: Explain what you mean by centralized governance.

Marco Palladino: With decentralized governance, everyone is on their own — making their own judgment calls about what’s good and what’s not. Centralized governance reduces the risk of adopting AI across the organization, but at the same time you want to give teams a degree of freedom — a bounded degree — to experiment within the governance you’ve established. Within these guardrails, you can move, you can experiment. On one hand, we don’t want to slow down innovation, so we want teams to be able to try new models, new tools, build new agents. At the same time, we want that experimentation to be bounded by centralized governance in such a way that nobody can ever put the organization or customer data at risk.

James Kaplan: What are my red lines? What are the things I can’t compromise on?

Marco Palladino: Customer data is number one. Whatever we do with agents, we cannot put customer data at risk. Organizations will need a strategy to anonymize data going through a model — and to dynamically reinsert that data on the way back, so the model never sees it but end-user experience is unaffected. PII encryption. The organization may also want to determine which models can be consumed and which cannot. We may want developers to use models from trusted vendors, and not allow them to use an untrusted vendor that might learn from all the data and interactions — effectively copying IP and creating organizational risk.

What models are being used? What data is flowing to those models? What MCP tools can agents use? What identity are we giving agents, so we can identify them and determine what they can and can’t do with models, APIs, and MCP tools? There’s a whole agent identity problem: agents are using MCP tools, some of which use third parties — how do we identify the agent, and how do we identify the end user consuming the agent to act on their data? None of this can be reinvented every time a team wants to build an agent. That would be madness — a colossal risk for any organization.

As the industry matures from early experimentation to actually running agents in production, those agents need all of this underlying infrastructure. An AI gateway is a core part of that.

There was a lot of early experimentation in the last two or three years, and now organizations have identified hotspots where agents can help with specific business processes — moving faster, innovating faster. The question now is: how do we enable every team in the organization to become an agentic developer?

James Kaplan: Any other design decisions? We talked about centralized versus decentralized governance. What else is especially important when implementing an AI gateway?

Marco Palladino: There’s a whole area of making agents effective, which can also be centralized. For example, reducing token consumption — optimizing how we leverage AI, especially for organizations that have already found their use case. Their next problem is: how do we make this cheaper? Things like prompt compression, or semantic caching — the ability to understand the semantic meaning of prompts so you can build what is essentially a semantic CDN that doesn’t require hitting an LLM every time. If the meaning of a question has already been captured and cached by another agent, think of it like a CDN, but semantic.

For example, if I ask an LLM “What is the population of New York?” and then separately ask “How many people live in New York?” — I’m using different words but asking the same thing. That could be a cached response. It helps on two dimensions: cost control, which is increasingly sensitive and will become more so as LLMs stop subsidizing every token and start pricing to reflect actual costs — it’s like Uber in the early days, when you paid $3 for a ride that really cost much more. When real costs emerge, CFOs will ask whether we’re generating the right outcomes for the spend. Prompt compression can reduce token consumption while retaining the same semantic meaning: “Please tell me how many people live in New York” compressed to “What’s the population of New York?” — much smaller token count, same meaning.

And then there’s observability: not only measuring what models and MCP tools we’re using and what agents are consuming the most, but also what outcomes we’re generating. Our customers tell us their biggest challenge is understanding outcomes. They can build agents, consume LLMs and MCP tools — but are they actually generating any outcome? How do you quantify the economic impact an agent has generated if you’re not tracking those outcomes?

Even outcome tracking is something that can be centralized — so that when teams build agents, the entire observability stack, from low-level connections all the way up to business intelligence and outcomes, is captured centrally. Teams don’t have to rebuild it every time.

James Kaplan: I can see how an AI gateway might help you track usage. How does it help you track outcomes?

Marco Palladino: The AI gateway sits in between the agents and the models and the MCP tools and APIs they’re consuming. So everything the agent does, the AI gateway is aware of — because it is on the execution path of all of it. By doing so, the organization has a centralized control plane. Think of it as a control tower for AI, where you set up all the governance rules, security rules, data governance rules, optimizations, and observability rules you want. Then teams go build agents. Whenever an agent makes a request, it has to go through the AI gateway infrastructure, and all of that governance and observability gets captured centrally.

We could avoid doing this — but then we have a much bigger problem. Whether governance, security, and observability are established through an AI gateway or not, we still need all of it. We have to enable teams to succeed by removing those cross-cutting requirements.

James Kaplan: You can start to interrogate the traffic that goes to the models, and there’s a lot of insight about business value there. And at the same time, you have a lot of insight about cost — which is interesting, because for the first time in a long time, the marginal cost of compute may be relevant relative to business value, rather than being a small fraction of it. CPU has been cheap; GPU is expensive.

Marco Palladino: I would argue CPU will be cheaper only for now. I’m also a big believer in an autonomous agentic world where agents are going to be not only intermediaries but the actual buyers of software. In an agentic economy where agents are automating more and more business transactions, we’re going to hit a bottleneck on CPUs — because we’re effectively replacing humans with CPUs. We just haven’t seen that yet because we’re still building out the use cases. But at scale, when every organization runs with an army of agents handling its core business operations, there will absolutely be a CPU shortage. If agents truly become what I think they will, that day is coming.

James Kaplan: We may see a flipping of technology economics. For the past twenty years or so, infrastructure has been cheap and application development has been expensive. AI makes software engineering cheaper but creates massive compute requirements. Whether it’s more GPU than CPU, one way or another it makes infrastructure costs more relevant — and so the marginal cost of compute will matter in a way it hasn’t in several decades, relative to business value.

Marco Palladino: One use case that illustrates this: today, organizations are experimenting with agentic IDEs — Claude Code, Codex, Cursor. Can they help developers build faster, or build more, by leveraging AI? Today, a human developer is asking prompts in these tools to go build software. How far are we really from having another agent asking the prompts to go build software? Now you’ve removed the entire human component. And that agent will know what to build, or not build, thanks to inputs arriving via APIs or MCP tools. The agent will make the judgment calls a human developer used to make, but autonomously. We’re going to have agents telling agents to go build agents. It’s going to be agents all the way down.

James Kaplan: Anything I neglected to ask about? Anything else we should cover?

Marco Palladino: I think there are two things worth mentioning: a change in distribution, and a change in customer behavior. The distribution one is especially interesting.

Today, businesses invest enormous amounts of money and effort to reach human customers — a billboard on the highway, a TV commercial, a YouTube ad. The internet runs on those commercials. But when the agent becomes the buyer, all of a sudden the agent doesn’t care about the billboard. It doesn’t care about the YouTube video. How does an agent decide what product to use? It looks at documentation. It looks at examples, getting-started guides, what other agents have been doing.

There’s going to be a whole new distribution channel — potentially larger than today’s human distribution channel — where to attract customers, you’re not attracting humans anymore, you’re attracting agents. Everything is going to change when that happens. Every business that relies on digital advertising will find that channel works differently than it does today.

That is going to change the internet as we know it. I think it’s extremely exciting. What’s even more exciting is that those of us in this conversation are not only witnessing it — we have the opportunity to help build it. It’s a builder’s era.

James Kaplan: The distribution point is fascinating. It makes commercial markets more like equity trading, where there’s been no human in the loop for years. What you’re suggesting is that many commercial markets may become agent-based — algorithms, instantiated as agents, transacting with other algorithms instantiated as agents.

Marco Palladino: Exactly. They’re not transacting equities or securities the way a financial trading bot would, but they’re transacting outcomes. They can bid on taking an outcome and completing it. A whole new economy is going to be born from that.

And it sounds futuristic, but it isn’t. It has happened before. Thirty years ago, if you weren’t in the Yellow Pages, your business didn’t exist. Then the internet was born — all of a sudden you needed a .com website, or your business didn’t exist. Then the iPhone: customers moved from websites to mobile apps, and if you didn’t have a mobile app, your business didn’t exist. Well, the customer is moving again.

James Kaplan: What you’re saying applies to tokens as well. You may see secondary markets for tokens, with agents bidding against each other for them.

Marco Palladino: Anything that uses a limited resource will eventually find a way to create a secondary market for it — whether it’s a token or a thirty-year-old Mercedes-Benz, there’s always going to be a market.

James Kaplan: I’m still waiting for someone to set up the first trading desk for tokens.

Marco Palladino: It’s going to happen. And it’s quite exciting. Some people look at this and say AI is going to damage society. I think society will have to evolve, and there will be a transition period. I’m just making an observation — I don’t know exactly what’s going to happen. But with the last industrial revolution, things are much better now than they were then. There was a century of societal upheaval — capitalism versus communism, all kinds of political and economic realignment — because of that revolution. This is a new industrial revolution. Are we going to be better off a hundred years from now, when the hard work is done for us? I believe so. Is the transition going to be easy? I don’t know. But it’s progress, and progress encompasses moments of realignment in how we look at technology and how we adapt to it.

James Kaplan: Terrific. Thank you so much. This was great.

Marco Palladino: Thanks for the opportunity. I had a blast.

How times have changed! Companies (mostly hyperscalers) spent USD 260 billion on data center construction last year before they installed a single server.

Historically, data center programs have been bespoke efforts, where much of the work happens on site. Companies had to build even “modular” data center architectures largely on site. Both providers and consumers are pushing to build more of the data center in a factory and less on site.

This applies a very old economic dyanamic to a very modern domain. At the birth of the industrial revolution, British industrialists transformed textile manufacturing productivity by replacing the “putting-out” model (in which workers spu thread and wove cloth in their cottages) with the factory system that brought the workers together at one site. Yes, this allowed for greater automation -- but knowledge sharing and management control as well.

InfraPartners is in the middle of the transition of data center construction from a bespoke project to an industrial process. CTO Harqs Singh and I talked about how that works and what that means. A few things that stood out to me

• Much of the productivity advantage from building data center modules in factories rather in on site derives from co-location rather than automation

• Sometimes customers push to tweak data center designs, even when it could add months to lead times

• Skilled labor from the trades is as much (or almost as much of) as bottleneck as chip availability

• Building data center capacity is as much of a moral as an economic project in that it contributes to abundance

James Kaplan: James Kaplan here for another session of the Prosaic Times podcast. I’m recording this on the Brown University campus because the Watson School is hosting a great session on governance in China, which I’ll have the privilege of attending today. But that’s not what we’re talking about here.

We’re going to talk about data centers, data center architecture, and data center economics. So I’ve asked one of the smartest hosts I know on that topic — Harqs Singh, who I’ve known for years — to join us.

Harqs, do you want to introduce yourself and talk a little bit about your journey from being an infrastructure leader to being a data center entrepreneur?

Harqs Singh: Absolutely. Thank you, James, for having me. Hi, everybody. My name’s Harqs Singh. I’m currently CTO and co-founder of a company called InfraPartners. We build prefabricated AI factories. I’ve been in this industry over 20 years now in technology — cyber, data — started at British Telecom, then Thomson Reuters, then BlackRock.

Then I decided to jump into this largest deployment of capital in human history, as I was told, in the AI space and the excitement around what this could enable for the world — and jumped in with both feet, riding the rollercoaster of delivering AI across the world.

James Kaplan: Fantastic. So you’ve seen multiple generations of what a data center is — from the bespoke Tier 2, Tier 3 data centers we built ages ago to the more modular enterprise data centers or Tier 4 data centers that got built in the 2010-ish timeframe to now — something that feels almost entirely different.

I was wondering if you could reflect on that evolution and talk about what, if anything, has stayed the same and what has changed.

Harqs Singh: Of course. It’s changed a lot in the last 20 years.

Enterprise to colocation to cloud, and now to AI — we’re seeing some really big technical changes. So we’re seeing direct-to-liquid chip cooling, which has really been driven by the intensity of the AI loads.

We’re looking at racks that are now 140 kilowatts per cabinet, going up to 200, 320 — which is something we didn’t imagine in the past. And so this brings along a lot of engineering challenges and thinking about things differently.

NVIDIA and the next generation of equipment are looking at using DC power — 800-volt DC power — for their data centers. And so I think you’re going to have two types of data centers: you’re going to have the CPU-focused data centers, and then you’re going to have the GPU-focused data centers, essentially.

The growth has been significant. If you look in the past, people talking about having a gigawatt portfolio was quite a large data center estate — and now everybody is looking to build at least a gigawatt, and you’re looking at some of the largest hyperscalers looking at 20 gigawatts. The projections are anywhere between 100 and 200 gigawatts of additional incremental data center capacity that’s needed by roughly that 2030 period.

James Kaplan: I remember when we used to talk about square feet rather than critical power, and we got really excited when we started talking about 75 watts per square foot. That was a reasonably sophisticated data center at some point.

Harqs Singh: Yeah, absolutely. The intensity has grown significantly. So these data centers fundamentally have changed and evolved. And I think to the point you talked about earlier: we need to change; we need to industrialize; we need to be able to deploy at a scale we haven’t deployed in the past.

We did some analysis on the supply chain to support the growth by 2030. There needs to be something like a 10× increase in supply chain. So we need to think about doing things differently. In the past, we used to build every data center differently. Every data center was almost like a snowflake.

But now we need to standardize and accelerate deployment by having really standard components and units that we can deploy very quickly wherever needed. The biggest issue has been — in the past, to your point — space was the biggest thing we optimized around. Now it’s power and available power. If you think about where the data centers are now going — whether it’s West Texas or the Nordics — a lot of the data centers are going where there’s spare capacity and spare power. We’re seeing a lot of issues with the grid. So a lot of data centers at the moment are looking at on-site generation using gas engines or other forms of technology to be able to essentially bring their own power behind the meter.

James Kaplan: It sounds like what you’re describing is much more of an industrialized process. It used to be the case that if you were a big bank, you might do a major data center program maybe once a decade — maybe once every 15 years — and you were sort of relearning it with each generation. What you’re describing is we’re adding — or the world is adding — tons of capacity every year, and therefore it’s much more of a continuous industrial process rather than an episodic project.

Harqs Singh: I think this wave that’s come with this sort of growth of AI and what it’s going to enable — capital isn’t a concern at the moment, which is a good thing — but it’s really things like supply chain.

You really need five things to be able to deliver AI capacity. First is land, power, and permits — that’s where we’re going where available power is. The second is supply chain — that’s where we’re really focused, because skilled labor shortages are a real problem. Being able to find skilled labor in West Texas or the Nordics is very difficult, and that’s why we look to build most of what we do in the factory. Being able to get chips from NVIDIA or from other chip manufacturers. The third thing you need is chips. Fourth is capital, and the fifth is an end user.

Only if you have all of those ingredients are you able to actually deliver AI capacity. And so it is complex. If you think about it, James — one gigawatt of data center capacity, including the chips, is about $50 billion. Which is obviously a significant investment. So being able to pull that together — although we hear a lot about these gigawatt announcements — think about that as $50 billion that’s been deployed.

James Kaplan: What is, to your thinking, the rate-limiting factor? Especially if you drill down one level — what type of labor is the biggest constraint? Is it electricians? Is it something else? And what type of equipment is it?

Harqs Singh: The biggest skills gap is electrical, mechanical.

There are shortages, especially in those more remote locations.

If you’re in demand as part of the skilled labor force, you may not choose to live in West Texas or the outer reaches of the Nordics, because you have choices. And so that’s the biggest issue on the skill side. If you look at the other side around supply chain, the longest-lead items are essentially generators — backup generators. They can be nearly two years now. And also transformers. We’re seeing some innovations in the design around looking at designs that don’t need generators — being able to design them out or just design them for a portion of the facility — 20% of the facility related to sort of the network and management pieces. But transformers are the other big one that we look to optimize around.

Either being able to find smaller transformers or reuse them if you can, if you’re not in the queue already to be able to buy them.

James Kaplan: Are demand signals starting to work on labor supply? Obviously this all happened relatively quickly, but you have a specialized skill set around the electrical and mechanical. Where did those people come from? What’s required to expand the flow of people into those jobs?

Harqs Singh: I think it’s really being able to encourage more people to come through the vocational side — the skilled labor side, apprenticeships, those kinds of things. It’s really interesting because AI is going to create an abundance of intelligence and agents and those kinds of things — sadly, it’s not going to create an abundance of electricians or plumbers or mechanical skills. So I think that’s something we’re going to have to focus on — getting people through colleges and universities and really trying to broaden that skilled labor pool. Jensen recently mentioned that if you look at the salaries now that some of these skilled trades for data center construction are pulling, it’s becoming really attractive.

I think the future of the distribution of skills we need on AI is going to be very different than where we are today.

James Kaplan: If you’re planning to deploy $50 billion, basic economics would tell you: if someone’s planning to deploy $50 billion of capital and the constraint is an electrician, they’ll pay real money for that electrician, right?

Harqs Singh: Absolutely. And that’s what you’re seeing, James. If you look at the labor costs of data centers that are being built in some of these remote locations, you’re looking at them paying 20, 25% more on the overall cost — even double, triple the salaries you would normally expect. And that’s really where our model is that we build 80% in the factory, 20% on site.

What that means is the skilled labor gets to still live where they want to live, in the cities they want to live in, and only for a short period of time are they in a remote location. So it solves for that skilled labor issue. But you’re right — there is a premium to pay to have somebody leave their family for a year or two and move to a remote location.

So there’s an increase in the running costs, but there are also large retention bonuses that are being paid to really help push people to stay out in those remote locations to get them done. But the growth is so much that they’re struggling to find, even at the premium rates, the skilled labor to go out and do the work.

James Kaplan: Who’s going to train these master electricians, as an example?

Harqs Singh: Yeah, absolutely. And I think, James, the challenge is going to be: can we get enough of them to market in the next — the conversations I’m having a lot right now with clients, they want data centers online next year.

There’s a demand-and-supply imbalance at the start of 2027, and so it’s not going to solve for the short term — but I think this growth curve will go out to 2030, so hopefully we can solve for some of that in the longer term.

James Kaplan: I find your investment thesis around productivity in the factory as opposed to productivity on the site incredibly compelling, and we’ve seen that in many different sectors of the economy. One of the reasons housing costs are high in the United States is that, for a bunch of reasons including regulatory ones, it’s been hard for the housing sector to do more in the way of manufactured housing as opposed to housing built on site.

How does the skill mix differ when you build something in one of your factories versus when someone builds something on site? Is it the same set of skills but you just get more out of them? Or is it a different mix of skills? How do you think about that?

Harqs Singh: It is the same set of skills, but we get more out of them. That’s the way to think about it — DFM principles that shave time off the schedule. I think the other thing the factory solves for is it standardizes, it gives you a level of QC, and it also accelerates timelines. We’ve found there’s less waste. If there’s something left over, we reuse it for the next set of modules and blocks that we build. So we’ve found that from a sustainability story there are some positives there as well.

James Kaplan: What’s your labor pipeline look like? What’s the ideal person working in an Infra Partners factory? Is this someone who spent 20 years at a big bank working on a data center? What’s the profile for that type of person?

Harqs Singh: It’s mostly engineers — mechanical engineers, ideally from different backgrounds. We have quite a standardized way that we deploy. We have a standard reference design, which we start with, and that saves time as well. We have a standard reference design that aligns with the NVIDIA DGX lean framework. What that means is we need specific electrical, mechanical, construction skills. We find that in Houston and in major cities we can find those skills quite easily. We’re not asking them to move somewhere where they’re going to be away from their families.

James Kaplan: If you think of that factory as existing somewhere on a continuum — and one end of the continuum is a big room with people in it, and at the other end of the continuum is a lights-out, fully automated factory — where are you guys now, and how do you see that evolving in terms of the construction of components?

Harqs Singh: Yeah — I think right now we are more manual than we’re automated.

Physical AI and more of the robotic stuff that we’ll see — more autonomous, essentially, in terms of being able to build the building blocks and prefabricated solutions we have. But at the moment, because in prefab — to your point, James — this is something that’s happened across other industries as well as industries have industrialized to be able to deliver product at scale — we’re not [fully there yet]. To your point about the prefabricated concept in general construction, it’s already been used in a lot of countries to be able to build commercial buildings and also homes — stadiums as well.

A lot of stadiums are prefabricated in the way they’re designed. So it’s taking that concept really to the data center space. If you look at the value of prefabrication, it is a 20% to 40% improvement in timelines, as you can centralize it. So it’s taking that concept, bringing it to a data center — well, bringing it to an environment which is changing quite significantly over the next few years as we drive these transitions: going from air cooling to liquid, from AC power to DC power, from racks that had a handful of servers in them to probably having a lot more GPUs in the future that are going to be much heavier.

Once those transitions play out over the next few years, aligned to NVIDIA’s roadmap and generally the industry’s roadmap around what’s happening with chips and the extreme co-design they’re doing, that will allow us to get to a more stabilized point in terms of technical design.

I think that will then increase — to your point about the spectrum — the amount of automation and robotics versus human input as well.

James Kaplan: I think what you’re suggesting is that some of the toughest problems in AI live at the intersection of the physical and virtual world.

Harqs Singh: Correct. And I think that’s where a huge opportunity lies. You’re seeing a lot of great companies that are focused on that. It’s exciting in terms of what’s going to happen in that space.

James Kaplan: If you think about the complexity of moving from a language model to a vision model to a world model — my God, how many orders of magnitude is that in terms of conceptual complexity? Moving from a language model, which interacts with text — as fascinating as that is — versus the world model, which conceptualizes objects interacting in three dimensions in accordance with the laws of physics.

Harqs Singh: Look — we’re going to have to train a lot of these models and these humanoids and whatever technology we end up using, because I think the best outcomes of AI are when you have trained it with reinforcement learning and human input to be able to get the outputs you want.

When people think “out of the box,” it’s going to make a huge difference — but that’s not enough on its own. You actually need to invest, train, and build the capabilities and the outcomes you want, just like you would train any employee.

James Kaplan: I was wondering if you could speak a little bit about the level of abstraction versus integration between what I might archaically think of as the system level versus the facility level. As context, I spent a lot of time working with banks on modular data centers in 2010, 2011, 2012 — and you always ran up against the constraint of, well, the systems don’t quite work that way. Is there more abstraction? Is there less system diversity? To what extent is the construction of the system, for want of a better word, a constraint on the degree to which you can modularize the physical infrastructure?

Harqs Singh: I think there has been some progress in that space. There’s been some maturity, especially on the IT side. If you look at a lot of the NVIDIA DGX design, it is based on a reference architecture that has become the default way to deploy.

James Kaplan: Mm-hmm.

Harqs Singh: What we’re starting to see — we just released our partnership with Emerald AI — what they’re looking at is working to make data centers grid-interactive, making it an asset where they take grid signals in. What they’re able to do is flex either the IT side or the facility side to be able to work with the grid when it’s under stress, essentially. So you’re seeing a lot more orchestration between the whole stack. One of the things that’s been driven over the last couple of years is: look at the whole thing from land, power, permits to data center to chips to models to data — how it all comes together as a full ecosystem across all of it.

In the past, James, to your point, there was this IT–facilities divide — but I think we’re starting to see this view, and this concept of extreme co-design that NVIDIA have driven — we need to do that across everything, especially when you’re trying to optimize something that’s going to be on the order of $50 billion for a gigawatt. Everybody’s throwing gigawatt deployments out every week. When you’re talking about that level of investment, there needs to be extreme co-design and it needs to work together as one unit. The revenue that’s on offer and the investment that’s taking place means we have to do it that way. There’s no other way to do it. You’re starting to see that in the industry with partnerships that are taking place and the companies that are working together.

James Kaplan: Back in the nineties, I think we used to see buttons and pins from Sun Microsystems that said, “The network is the computer.” It sounds like what you’re saying is: the data center is the computer.

Harqs Singh: Well, the GPU — the network is also very critical. If you look at the training cycle — computation, communication, updating weights and biases — the communication phase: you could have billions of dollars worth of assets sitting idle if you don’t optimize for network.

So that’s really critical as well, especially in the inference space where response rates to prompts are going to be important — obviously some of the more real-time use cases as well. We’re seeing data center designs being optimized and GPUs being positioned and optimized for east-west traffic but also north-south traffic and response times — because if you think about the investment that’s there, the more time it’s idled, you’re losing revenue essentially.

Tokens equal revenue in the future. And so it’s an —

James Kaplan: I am still waiting.

Harqs Singh: I would —

James Kaplan: Tell me it already exists.

Harqs Singh: I would not be surprised if in the next 12 to 18 months you see some token exchanges where tokens are being sold. But I’m a big fan — we’ve been in this industry for a while — efficiencies. I was having this debate with colleagues just last week: we’ve done some great stuff where we’ve created incremental efficiencies and then Jevons Paradox comes in and everyone consumes more.

So I sort of said: we really need to flip this whole thing on its head a little bit. What we need to do is create an abundance of everything — we want an abundance of food, an abundance of energy, an abundance of compute, an abundance of AI.

If we think about it in that way, hopefully some of these young, smart people will be able to take this abundance of everything and create something magical out there.

And that’s what I like about AI: the boundaries and rules that we historically have lived by don’t exist in this world. If you look at what some of the AI natives are doing and how they’re thinking about building companies differently and solving problems differently and getting data from different sources — there’s so much opportunity, and that’s what’s so exciting: the opportunity is untapped.

James Kaplan: I don’t think anybody in this industry is brave enough to say what the future looks like, because we just don’t know. That’s why it’s so exciting, right?

Harqs Singh: That’s why it’s so exciting, right? I’m an optimist in that sense — I think if we can unlock an abundance of everything, then some really smart people — probably young people around the world — will think about problems differently and create some magic.

I look forward to seeing what that looks like. But I’m very optimistic about what this can enable.

James Kaplan: This is a little bit of a tangent, so we shouldn’t pursue it too far — but I happen to agree that you’re touching on something very important: I think we in the technology world need to talk differently. We need to talk about abundance with more conviction, because I agree.

I think you’re saying that what we’re all jointly engaged in is a moral project that will contribute to human thriving and human flourishing — and we should be. I don’t think any of us should be apologetic about that. There are risks, there are guardrails, there are things we can do better versus worse — but I don’t think any of us should apologize for the larger project.

Harqs Singh: Agreed. And I think we all need to think at scales we’ve never thought about in the past. I think that’s the lesson from history. If you look at sizes of data centers — they’ve gone from megawatts to hundreds of megawatts to gigawatts, single digits to double-digit gigawatts — we should think at scales we haven’t thought about in the past to create that abundance.

Put that abundance in some very smart, intelligent people’s hands to see what we’re able to deliver and create for humanity. I think that should be the legacy of what we do in this industry: that we enabled that abundance of everything — and we are the backbone upon which digitization and technology and AI have permeated every industry.

In the past, you remember when we were a cost center — and now we’re fundamentally a revenue generator.

James Kaplan: Some people still think of it as a cost center, sadly — but we’re working on that.

Harqs Singh: Yeah — we haven’t won; we won’t declare victory there yet, James.

If you think about businesses that don’t have a core business that has a technology strategy and the revenue and products it’s making — or a data strategy or an AI strategy — even security: these things have gone from being something that happens at the back end of an organization to being key to the product you sell. Whatever product you sell in the future — whether it’s to humans that are probably going to be AI natives, or whether it’s to machines, because that’s who your consumer is going to be in the future — you’re going to need all those things around digitization, data, security, and AI. If you’re not thinking about putting that into your core business, you’re more likely to be left behind or disrupted.

James Kaplan: It’s fascinating what you guys are doing from a historical view. As someone who is fascinated by history, you could observe that there was a fundamental change when certain parts of the world in the late 18th century went from making cloth in cottages to making it in factories.

That sounds somewhat prosaic — but all of a sudden large numbers of people could afford two sets of clothes instead of one. I would say this project of moving things from a job site to a factory is incredibly important in terms of the way it increases productivity and reduces unit costs.

Harqs Singh: That’s exactly it, James. That’s a great analogy, and exactly what we think this should help drive for the industry.

James Kaplan: Just going back to this point about standardization a little bit — I presume, given who’s building data centers or using data centers, the large majority of your customers are either hyperscalers or people who are planning to lease capacity to hyperscalers. I assume that’s a correct assumption.

Harqs Singh: The client space has grown quite significantly. To your point, there’s the hyperscalers — U.S. or Chinese hyperscalers. Then there’s the Tier 2 colos.

This neo-cloud — and that’s been created: a lot of Bitcoin mining power has been migrated to building AI. Fourth segment is enterprises. We’re starting to see enterprises — certain sectors: financial services, healthcare, insurance — look to build small AI factories for their private data. And then the last segment is the sovereign AI space — where we’re going to see governments, for national security but also for public services for their citizens.

So what we’ve seen is it’s actually expanded significantly from the hyperscaler in the past to a few new segments in the market.

James Kaplan: To what extent do different segments or different customers within a segment require a slightly different product — or to what extent can they use a standard product? How much variation, how much tailoring do you have to do by segment or by customer?

Harqs Singh: Our starting position is always — our gold standard is the NVIDIA standard, the DGX standard. We start from there, which is a great starting point. Typically what happens is the modifications that take place are either for real requirements like security in the sovereign space, or some enterprise requirements — or it’s because the technical teams on the other side want to add their value, if we could put it that way. They tend to put in a few things that they would see as imparting their value to the firm — not always necessarily needed. We see that the hyperscalers — and the people that have these technical teams — tend to be slower than the others.

So it depends on whether there are real requirements like security or enterprise — or whether it’s technical teams adding their two cents.

James Kaplan: A head of infrastructure — one of the guys used to describe this as the blue cable problem. An application development team would say: build me a server that looks like this, and the cable should be blue. The cables cannot be red — I insist on blue cables, right?

Harqs Singh: James, your analogy around using color — we’ve seen requirements around using specific colors — but I think you have to go back to: does this fundamentally change the product? A lot of the time it doesn’t. It’s preference.

Some people want greater ceiling height; others want it to be within a shell of a building — all things that are doable. I think the most important things are speed and revenue. A lot of what we’ve done in technology comes back to revenue growth, helping the business expand to where it needs to be.

That’s the bit we should never lose: we’re building infrastructure, but it’s to help drive business value. That’s where you and your teams have been focused fantastically — what’s the intersection between technology, data, AI, and revenue, and how does it create real value? That’s the part I try to focus on in my day-to-day as CTO of the organization. We always need to come back to: how do we create value?

On that basis, one of the most common questions I get asked is: what is your standard reference design optimized for? People say — oh, is it the size of the generator? Is it the size of a panel? I say: look, if you look at the TCO of a data center, most of the cost is in the chips. Seventy-five percent of the cost is in the chips. So we’re optimized for that. We’re optimized to get the most out of the chips as possible, because it’s the most expensive resource.

You want to make sure it’s idling for the least amount of time — to the point about making sure your network connectivity is right — and that we deploy SuperPOD after SuperPOD, because that’s how you get the most efficiency out of the most expensive resource and the most value. We’re laser-focused on making sure that intersection between technology, physical, IT, data, and revenue — we’re always focused on optimizing for revenue and business value.

James Kaplan: When you have discussions with customers around customization — to what extent are you involved in dialogues like: if you want the off-the-rack version you can have it in these months, but if you want the blue cable — to use a metaphor — that’s these months? To what extent are you involved in dialogue around that tension between speed and bespoke configuration?

Harqs Singh: That’s exactly why we have a standard to start with.

The standard off-the-shelf is aligned with the latest chip technology and can be deployed much more quickly. It’ll save you three months by taking off-the-shelf. If you want the yellow cable or the red cable — whichever specification you have — that will add a number of months. If you want to do that, we can do that too. But it’s the tradeoff of time.

What we’ve found is: those that don’t have a large technical team will go off-the-shelf — it makes sense. Those with the technical team will say: no, we have standards here; we need to align with them — blue cable, please. And that creates that time.

A lot of the opportunity is the gap between those organizations — those that can run really quickly, especially when you think about there is this supply-demand imbalance in early 2027. Those are the people that will take advantage of that supply-demand imbalance.

James Kaplan: Let me ask one final question — I realize we’re coming up on time. At one point when we were chatting previously, you talked about, for want of a term, a generational waterfall for data centers — that some things for a year or two, or even less, might be on the hardest training problems and then would get repurposed to inferencing problems.

I’m sure I’m not describing this correctly — but I was wondering if you could talk a little bit about how you think about that.

Harqs Singh: Absolutely. If you think about the latest chips that are coming out, it’s really interesting to see how they’re doing in some of the benchmarks now. When we spoke last time, James, there was this piece where you put the n chip on training and then n minus one you look to reuse for other capabilities that are generating some revenue.

I can see right now, from some of the demand-supply imbalances in the whole market, even some of the older chips that have been in the market for four or five years are still quite valuable in terms of the GPU cloud prices they’re getting. So there is this sort of short-term skew right now because capacity is not coming online quick enough, essentially.

You need to think: are we building an upgradable design? If you deploy GPUs today and you get three, four, five years’ worth of value on them on a contract with an offtaker, then in five years’ time, how do you change the design such that you can deploy the latest chip from the industry — from NVIDIA, essentially?

What we’re seeing is: you need to update some of the power plant from AC power to DC power. You need to upgrade some of the densities they’re going to be operating at with liquid cooling. So we’ve designed that in from the start.

What that enables you to do is have flexibility — and also be able to create this into a long-term sustainable asset. Flexibility is really important because if you get out to five years, you’re then trying to guess for that chip you deployed five years ago: what’s the market value of that, and what’s the revenue potential of it?

I think at that point everyone is optimizing for flexibility because nobody knows what’s going to happen — as many options as possible. To your point, part of that option could be: you relocate the chip to inference. It could be: wait a minute — I heard a story just two weeks ago where a client signed up to a five-year deal; after five years, for the same chips, they signed up to a 20% price increase because the demand-supply imbalance was so bad.

James Kaplan: That’s a new thing — at least in my recollection.

Harqs Singh: Absolutely. What we’re trying to do is give you as many flexible options as possible — whether it’s relocating it; whether it’s saying that chip is no longer relevant because there is a demand-supply imbalance and you need to upgrade to the next chip; or whether there is real revenue and market value to that chip even if it’s five years old and fully depreciated. Even if it’s fully depreciated, James — obviously from an economic perspective, you don’t need to get the return you had before because it’s a fully depreciated asset, and that gives you some flexibility as well.

We’re fully focused on giving you full flexibility and options so you can maximize revenue at the time based on what the market is doing. And that’s the bit we don’t know: what the market is going to do.

James Kaplan: Fantastic. Thank you so much. I hope you found this conversation as enjoyable and as interesting as I have.

Harqs Singh: I have. Thank you for having me, James.

James Kaplan: Thank you so much. It was great.

This episode talks about a world where old boundaries seem to be disappearing:

• Between the intellectual and the practical worlds, as we need to apply theoretical knowledge almost as soon as it’s developed

• Between the sciences and the humanities — senior executives need to draw on both

• Between the business and the technology domains — as companies connect everything to a network

• Between strategy and execution — as companies set up closed loops between ambitious plans and granular improvement levers

• Between the world of commerce and the world of geopolitics, as non-state actors may attack commercial enterprises for strategic reasons

From the 101st to McKinsey

James Kaplan: Welcome to the Prosaic Times video podcast. One of my very favorite colleagues, Charlie Lewis, is joining us today. Charlie, can you introduce yourself and tell us a little bit about your professional journey from the U.S. Army to McKinsey?

Charlie Lewis: James, you covered it all right there. But no, it’s truly an honor to be here. James. I think, you know, I’ve, loved getting a chance to work with you over the last eight years or so, but. You know, I am in the fortunate position now where I get to help our clients think about how they address their most critical security challenges, across a variety of industries, mainly banking, but also critical infrastructure.

You know, as you think about the utilities and energy, we think about life science and medical technology, and we think about broader healthcare. So really what frames, you know. Sort of the day-to-day life and what our requirements are within there. And I get to do this as, as sort of the leader of our service line in cyber in North America and Europe.

And I’ve been at the firm for about, you know, as I just said, eight years and before that was in the US Army. And while I was in the Army, I was at, I started as an artillery officer with the 101st Airborne Division with a Ranger tab. So, you know, a lot of esprit de corps within that organization.

and then went and had the, the fortunate opportunity to go to Harvard Kennedy School, where I learned from Eric Rosenbach and Richard Clark on, on cyber war and terrorism. And then I ended up. Teaching in the Department of Social Sciences at West Point. And at the time, the Department of Social Sciences at West Point, sort of three of the core leaders in that department had worked with General Alexander to help stand up cyber command.

It was Colonel Suzanne Nielsen, who was the deputy head, and then, just retired as the head of the department. now Colonel, Heidi, Brockman Deist, who is now the head of the department and Scott Handler, and you may not know Scott Handler, but he is close to us because of Stephanie Handler, his partner and wife, who is now a partner in a law firm, was one of our cyber legal folks.

and is actually, they’re very close to the Reserves as well because Maura and Stephanie Handler went through the basic school together in the Marines. but I was able to be with them and the army stood up, the cyber branch and as I thought about it at the time, it was, is artillery the right. Spot for me to be long term?

Or how do I think about the evolving nature of warfare? and where sort of the core one shaping of the battlefield, the intelligence around the battlefield. What do I need to defend? And then how do we think about influencing operations prior to actually getting in there and being able to conduct operations in a way that don’t put.

you know, humans and, and Americans and our allies in harm’s way. And so I switched to cyber when I was down at Fort Gordon, helping stand up their sort of leader course down there. I realized that there’s a lot more to do in cyber than just from a defensive standpoint, and that led me through the interview process and to McKinsey.

James Kaplan: Fantastic. I’m glad you brought up Stephanie Handler. I should drop her a note. She was one of the most — I hadn’t realized she went to basic school with Maura — but Stephanie I always thought was one of the most thoughtful people I’ve ever encountered on the intersection of legal issues in cybersecurity.

Charlie Lewis: I agree. And she’s catching a little bit of a stray here. Right. But I hope it’s a good positive stray on, on this one. But she’s been really good at it. Right. And, you know, and, It wasn’t easy. Right. You know, we, at the time when she was right, it was COVID. we were doing a lot of the more advanced technical testing at the time, which required her input and, and support and making sure that we were protecting our clients in the firm the most.

And so there were many weekend and late night calls with her as we were getting the work, going.

James Kaplan: And I’m also — you know, a chuckle, not chuckle — I’m glad you brought up Eric Rosenbach. When Tucker Bailey and I were doing the research that became the book Beyond Cybersecurity, Eric was very generous with his time.

Charlie Lewis: he, he’s been great. I remember, there were some individuals who had unfortunately ended up in the US that we had, You know that w we had arrested in in Iraq and I had to go meet with folks, and Eric had given me some advice and all of that guidance, and I think that Richard Clark would be a little upset if I did.

Also didn’t highlight that he was a faculty member who gave us his book for free — the Cyber War book — which still sits above my left shoulder up there. Actually, oddly enough, it’s — if you look, you have Cyber War right here,

and then you have beyond cybersecurity, right, right up there. So,

Oh I take the dust jacket off ‘cause it, it is a cleaner look with just the, the black. And that is, it’s, sometime, it’s sometimes a bit more about form than function for me. So.

Soldier-intellectual, technologist-intellectual

James Kaplan: So here’s one of the things I wanted to cover today, why I especially want to have you on the podcast. You know, many years ago I read the book The Fourth Star about the Department of Social Sciences in the U.S. Army. One of the reasons I find you especially intriguing, Charlie — you’re interesting — is you’ve gone from being what I would describe as a soldier-intellectual to being a technologist-intellectual. Right. I find that to be an especially interesting transition, and I just wonder if you could reflect a bit for us about what it means to be a soldier-intellectual.

What the transition to being a technologist intellectual means and what intellectual inquiry means in the Army and what it means to the private sector and how that differs a little bit. So I realize that’s a lot.

Charlie Lewis: That’s a, that’s a weighty question that we would, you know, teach to our cadets in a, in a lab. I, you know, for me, I think there’s one sort of through line on, on all of those, and it’s what it means to be a professional. Right? And so, you know, you could anchor in, you know.

you could anchor in Huntington’s sort of definition of it in the, the liberal arts education, the continued learning and what that requires, you know, you could anchor into even where I think a really good example of this is in the medical profession, and I grew up with. To parents in that my, my father, a pediatrician, my mother, a pediatric nurse practitioner, and their sort of continued need to learn and the requirements that they had to have to learn.

And then broadly, if you think about the profession of consulting and what that means and the academic requirements and the intellectual curiosity that I think we need to have. you know, I think being in, so, and, and it’s no longer there, but the halls of, you know, the, the hallways in Lincoln Hall, it, at, at West Point at the time were sort of, you know, a place where we could have large debates and some big thinking about what is the future of the force and how do we think about the future of the force and the freedom to study what.

We needed to study. I studied the impact of military voting on local election officials with Dr. Rachel Sondheimer. we, I wrote about social capital and the impact that it has on the military force in our ability to come together and work cohesively as a unit. But then there were some. Broader thinking that existed.

Right. Learning to Eat Soup with a Knife by Dr. John Nagl, right? Sort of the core foundation before the field manual on counterinsurgency was written by a social scientist, and there was just frequent debate, you know? And then how we actually think about the talent and the human capital management within the force.

Those ideas came out of a joint venture between the Army G-1 and Department of Social Sciences, and with OEMA, the Office of Economic and Manpower Analysis. Right. And like there’s just so much brain power that. Sat in there and, you know, we’d have like Thursday afternoon debates on the Oxford comma and I, I don’t think there’s, you know, many units in the army, but it was a bunch of folks who were super professional, who understood what they were doing there and sort of teaching, the, the, the future leaders of the Army who all had combat experience.

Right. Most of us had either commissioned right before nine 11 or right, you know, within the three years after nine 11. I think like the least amount of deployed time, in, in that group was like 27, 28 months. Right. of time overseas. And so it was just great to be able to have that. But the ability to go back and think and broaden and understand what your profession does, I think that’s.

Core to the military. You think about Army Futures Command, you think about sort of the broader strategist position, the ORSA positions, all of these sort of thinking and engagement with the broader community allows you to, you know, understand a bit of the broader impacts, right. And think more strategically about what could happen.

And that intellectual power is sort of a bit of what I’ve tried to take and what I’ve found a lot at the firm in terms of thinking about from a technologist standpoint. I get it — you know, I just got back from RSA, it’s nerve wracking to try to stay ahead of the threat and the changes that are there, right?

Because it’s no longer, you can, no longer, you have to, like I am, folks have to be super technical in identity and access management.

Right, but it — you used to get really deep on a specific technology and a specific component of that, but now we’ve gotta think about how that plays into the broader agentic landscape and what you need to do around identity and making sure that you meet the autonomy, the capability, and the controls that need to get put in place there.

But how do you learn? You have to read, you have to study, and you also have to do, right. I think that’s one. You know, a bit of a difference here on the like intellectual curiosity, right? You have to be able to play with the toys, understand what it is, and so you can explain some of those difficulties to to clients as they go forward.

Tinkering, agents, and evidence

James Kaplan: You know, one of the things to me is in the business world, sometimes it feels like technology is the most intellectually forward-looking of the business functions. Right. And then within technology, cybersecurity, you know, that feels like the most intellectually on the front foot domains, you know, within, within, within enterprise technology. And it’s a domain where people are constantly experimenting and also constantly trying to engage with, you know, the academic community in order to stay ahead of the threat.

Charlie Lewis: There’s, I know there are a few academic articles that are out there. Right. And I think this is an interesting place ‘cause the academics get to do some of the testing and it’s really about the impact of agentic AI on security. And then one about, you know, more broadly around.

You know, LLM security and, really thinking about what the various attack paths and the risks could be and how they get implemented, right? So we’re taking that at the exact same time. We’re looking at technology that’s being implemented, and it is a fascinating blend, right? And you have to have the researcher angle that goes in there.

I’ve seen more organizations start to stand up like an innovation and research arm, right? A group of sort of researchers or a leader who can pull in others. I also think there’s like a talent incentive there too, right? There’s an a sort of a, a constant tinkering where like I go back to like the videos like you and I have shared about tinkering or when my, you know, my Harley died and I was able to get that back up and running and how proud I was of myself, right?

There’s a bit of like a tinkering that goes on in here. And you have to have the patience. Right. The same way that, I’m a, I’m a geography major, so Right. But like I remember my roommate who was a CS minor, right? And like how much tinkering they would have to do, I would always go to bed early ‘cause I was a, you know, we called it dirt.

I was a people dirt, human, regional geography major. Right? But my roommate would, would, keep, would keep tinkering, right? And they’d have, and that’s sort of the mindset. And so I think there’s always this desire to tinker. And get better. And that fits a bit into the innovation side, right?

And I think that any CISO or security organization that is sort of resting on their laurels or previous success is going to find themselves behind a little bit. And I also think there has to be that communication, right? I think about the difference between a brand new technology environment and where I spend a lot of my time — legacy, overly complex environments, potentially IT, OT, multiple global systems, maybe four or five different identity programs in place.

A horrendous, IT asset management program with an incorrect and out of date CMDB, right? Like so sometimes it’s that education, but it’s also understanding what the foundational requirements are to be able to move forward too. And security folks are pretty good at understanding that. Right. Where we stink is communicating the value of that to the business and why they have to invest in that if they wanna continue to grow and scale.

And I think now security folks have to be able to do that. And then explain why. I’ve got, there’s an argument that I think we could make that the first use case for agentic deployment in any organization should be security. Right? And you say, why does it need to be security? And it’s because that’s where the hackers are going, right?

We’ve seen that with Truffle Security in their report on what they were able to do with a Claude agent. We’ve seen this on what has been built out in terms of the Microsoft sort of attack path with an agentic attack path; we’ve seen novel attack paths for legacy vulns. And so now I actually think like there’s the real value there in getting ahead of it.

‘cause if you don’t start now, you’re gonna lose. And the only way that you can keep the value you want going forward is to have that security on the backend to sort of protect the folks coming in as the business pushes out.

James Kaplan: There’s one thing you said a couple of minutes ago that I just wanted to emphasize about tinkering and the patience required for tinkering. And as someone who — you know, personally — maybe has the patience of a fruit fly, I’m so excited now because the patience required for tinkering has gone down, right? You know, the tools help you with the syntax — think about the underlying issues more — and therefore reduce the barrier to tinkering and, you know, sort of — I don’t wanna say democratize — it’s tinkering because it allows more senior people to tinker more, but allows maybe people who might have, like myself, who might have said, oh, God, I don’t have time for this, I gotta go write this memo — to get their fingers back on the keyboard in a way that’s, you know, very constructive along multiple dimensions.

Charlie Lewis: Mm-hmm. I, I’m, you, you’ve worked enough with me to know that my level of patience is, is slim to nil. right. Like, you know, I’m, I try to make my life as efficient as, possible. You know, and, and you know, to include even like, sitting on the same seat every flight on an airplane, if I’m, if I’m able to, right.

And, and that way I don’t have to to choose. and I, and I think that, that the tinkering allows us, there is no excuse now to not tinker. There is no excuse because it is just so, it is so easy. Right. And I think like. The first thing I will do no matter what, is just like we’ve built, right task and research agents, right?

Super easy, super basic to build them. It’s a 15 to 20 minute upfront investment. One of my buddies says, how’d you write a good prompt? I was like, I asked chat GPT to write me the prompt that I want, and then I. And then I go and I edit it, but they can write phenomenal prompts. And that take, that saves me a bunch of time.

And then I run that prompt and I, and then I start building out like the GPT and ChatGPT. And it’s like, and you know, now we’ve got, you know, Gemini and the rest of the stack — it’s super easy to build an agent within there. And so — it makes life just a lot easier and it’s fun just to see small changes in how you can get the right output that can come from it, and it can improve an output, it can improve everything you need.

And for me, you know, I think one of the best things that we’ve done is we had one of our colleagues, Jose, found an entire GitHub repository that had like every 2024 threat report, and that was out there, right? We downloaded it. I, was like, oh, this could be great for the practice to understand what the threats are by industry by.

By threat, actor by region, et cetera. We pulled that in, but we had to, like, I had to fiddle around with, learn the instructions on how to be able to do it and zip ‘em and you know, extract from the zip and do right. And it took me 30 minutes, 45 minutes to be able to do it on a Friday. But it has probably saved us hours of time and has given better answers to our clients when they’ve asked.

James Kaplan: And you were touching on one of the important themes to me, which is the power of using AI to convert unstructured text to structured data, which you can synthesize, analyze, and act on much more effectively.

Charlie Lewis: Correct. And you’ve gotta check it. Right? And I think one of the things I’ve learned from you and from, from Rich is how to build in the checks that go into there to make sure it’s right. You can’t just assume that all the data is right — you should never assume the data is right. Right? Like, look at a footnote.

Go to the, go to the, the source, and, read and read the source. Right? My students used to be like, well. Like, where do I find all these sources? And I was like, you can’t use, you can’t cite Wikipedia, but sometimes Wikipedia has pretty good sources, and all you need is one source from a Wikipedia page that then drives you into the original primary sources from there.

And then you’ve done a bit of your research and now you’re able to do that as well. And if I have to do deep type research on it, right, you are checking all of the sources. You’re making sure that the sources are, are correct and it, it is just about. Asking. It’s, learning how to ask better questions.

I use it with my daughter to teach her how to ask better questions.

James Kaplan: As I like to say, if your mother says she loves you, check it — as the old newspaper editors used to tell you.

Charlie Lewis: Yep,

James Kaplan: One of the phenomenal things about Wikipedia — it’s not that every sentence must be sourced, but sometimes it has sources. You know —

Charlie Lewis: That’s true. You can always check.

James Kaplan: And you and I have talked a little bit about how much I like the book Military Power: Explaining Victory and Defeat in Modern Battle by Stephen Biddle.

I love this concept of force employment. I think you are in some way describing force employment as applied to knowledge work. Everyone has access — Biddle said, at some point, many people may have access to the underlying technologies, but it takes discipline in an organization to apply them effectively in an integrated way. And you are describing, I think, how to apply a set of technologies that are available to pretty much everyone in a more integrated and disciplined way in the service of knowledge discovery, which I find really interesting.

Charlie Lewis: I think, it’s fascinating the number of times that. Someone sends me a job description and it’s like, must have a BS in information technology or computer science, and I just like cut that out. Right? And, and it’s like, I don’t care about that. Right? Because what you learned in CS 50, while it’s good from a foundational standpoint, when you’re moving out into the cyber world, what you learned in that.

You know, over those four years at this stage, what you learned four weeks ago may be out of date. And so you wanna be able to build and scale and have a team that’ll learn and doesn’t wanna just rest, right? Like the question, like, you know, what do you do on your free time? Right? Where are, where are you focused?

Are you sitting down? Just wasting and, and scrolling through something on a plane or like you right, you were running some work, a workflow on, on, on, you know, your flight from, from London to la right? And so what are you doing in the background and sort of the downtime to keep learning? And, and that’s what I tell clients to look for in people.

And, and the best example that I have is that the first class of cyber, Of, cyber basic officer leader course graduates. Now, these folks, they don’t, they, they take the core basic military, so they’re going out to the range. They’re doing land navigation. Right, but they’re also taking CISSP, right?

They’re taking CCNA. They’re taking a whole slew of SANS courses because they have to get this training up and running. The the top grad, the Honor grad, was not a computer science grad from. West Point was not a computer science grad from an ROTC program, but was an economics major from the University of Central Florida.

Right. And he was able to do it because he understood the application of what we were learning in the class. And so I think a lot of the learning allows you to then there, there’s not just learning, it’s the actual application within your day to day that then creates that broader scale that you need to have.

And then allows for something you’ve taught me is sort of like always question, right? Always see is there a slightly better way that we can do this? Is there an improved output that we can get because of it? Yes or no? Is it worth that investment? Most of the time it is, right? And then how do we get there and how do we help our clients get there?

Humanities, ‘be human,’ and geopolitics

James Kaplan: There’s an interesting tension here. On the one hand, I had a good discussion with Associate Provost Michael Littman from Brown about this — that the principles of computer science are now more important than ever. You know, the syntax of this language versus that language, who cares — but principles around things like data modeling and abstraction are more central than they ever were. On the other hand, I think you’ve talked a little bit about how essential curiosity is, how people who may not have majored in computer science can get up to speed. You are a geography major; I’m a history major. Right. And —

Charlie Lewis: Rich is an English major.

James Kaplan: — and Rich is an English major. Okay. We gotta talk about that with Rich too. I was wondering if you could comment on the relevance of the humanities and the social sciences in a business and a technology context. And let’s face it, Rich is also incredibly technical and someone who has done technical stuff. So I was wondering — how do you think about integrating the culture of the sciences and the culture of the humanities, and what’s the relevance of the humanities and the social sciences in a technology context, in a business context?

Charlie Lewis: I, and I wanna be like a bit, so like one? like. I was convinced to major in geography and people geography, right? Again, the people Dirt. And my, my EV 2 0 3, so my Principles of geography professor, right at West Point when I was there your sophomore year, everyone’s putting a cell on to get more majors.

More majors means more classes. More classes equals more, you know, faculty, more funding, more, more prestige, and You know, he stood up in the, in the class when we were picking our major and he was like, if you like to look out at the airplane window and wonder what people are doing down below, you might be a good, you know, human regional geography major.

And I was like, that’s it. Right? That’s what I want to do. Right? Like there wasn’t, for me, there wasn’t much thought I wanted to be a history major. Right. Unfortunately, I had a bad experience in my first year of history at. At West Point, no offense to anyone who was there. I just had a bad experience and it wasn’t where I wanted to go.

Same reason why. What, what pushed me into artillery, right? Like there were people I liked to went artillery, there were people I liked who went infantry, right? But the people I didn’t like also were infantry. Not the people, but the officers, right. Who had had a bit of a problem with Right. And I tend to like flow with where I want it to go.

But I think, you know, as you think more broadly about the role of, of sort of the humanities, within that space and sort of more of the, the soft sciences, right? It’s the same reason why they’ve scaled and grown those majors at an engineering school like. West Point is the ability to critically think and analyze, right?

There is a structure that goes within there and we can apply the same sort of structured thinking, the hypothesis tending testing in a, In a political science way, in a historical way, and even looking at broadly at some of say the logic arguments in philosophy and apply those specifically to life in general and then understanding there.

And I think. You know, there is a ton of value in the foundational learning, right? The, the firm just, you know, you know, hired someone who was a physics and philosophy major at West Point. I asked him, when I taught him at West Point, why are you doing those two? And he said, because if I understand the foundations and I can do better at understanding everything else.

And I think there’s a clear, you know, fundamental thinking behind that. But the humanities to me. Provide that broader thinking and that broader understanding. And you know, I’ve, I’ve been telling people frequently in a world where everything is becoming a robot, be human right. And I think that a lot of times being a human requires the humanities and, and a bit of an understanding within there and the application and the impact that something could have on the world.

And, and, and that’s where I think like when you get to. Thinking about where we are in terms of technology now, it’s like, what is the impact it can have on a business? Is this good or bad? Right? What is the impact that this can have on the society that the business serves? Is this good or bad? Right? And for me, we, I spend my time trying to stop the bad that’s impacting society by improving, you know, the defenses that a business may have.

And like, you know. we’re fighting, like businesses are fighting nation states right now. Right. And businesses are fighting, you know, international criminal syndicates that you would never think would wanna go after a paper company, right. Or, or a drink company or anything like that. But they are because, or a small town, they are because they can make money off of it.

James Kaplan: Well, okay, let’s, let’s lean into the history a little bit. You made an interesting point the, we’ve seen a, you know, compared to, say, the Cold War era, there’s much less of a — or maybe even the post–Cold War — Much less of a hard line between the world of geopolitics and the world of business, or as you point out that we have non-state actors with geopolitical aspirations engaging in conflict with pri, you know, private sector institutions.

Right. I, you know, know, this was not what we’re used to, but it’s not historically. Unprecedented. Right? If you think about the history of the East India company, you think about the history of privateering. You know, you, you go back a couple of hundred years and there was, you know, at that point a much blurry line between the world of geopolitics and the world of commerce.

Charlie Lewis: Like I, I think, I, you know, like I enjoy reading, even though I live in coastal Connecticut and look goofy in cowboy boots in a cowboy hat, right? Like, I like reading about Western expansion and, and one of ‘em is, you know, read a book recently on fur trading, right? And the impact that, and the success that American Fur Traders had.

Keeping like the Canadians in really Hudson Bay out of sort of the northwest of the US and, and, and sort of that broader view when you think about just these, these men, these like they were men at the time, but like rough men who went out and basically. Didn’t think they were conquering territory, but they were conquering territory for, for a business end and sort of to own that business globally when it came to the broader fur trading.

Right now, they, they basically worked themselves out. There was not a conservationist thing, it was just make as much money as I possibly could. Exceptionally violent, all of that, right? But they were, they were. Private citizens that were going out and conducting business on their own and getting into conflict on their own.

With, with, with, you know, the Native Americans at the time, same exact thing with some of the, the privateering from ships, right? And like what you find on like the Treasure Coast and in, in Florida and the potential of gold still rolling up there in some of those requirements. and so I think that.

The, the difference now though is this is one, it’s not nearly as visible. Right. Like it is very hard to, to see and understand what this means. I think about the various, you know, volt and, and salt typhoon and what that means for our society, right? And really going and targeting core telcos and core utilities within the United States.

So it’s one, it’s not visible, and two, the scale is just massive, right? Like previously, those fur traders had to go out to California, in Oregon, in Washington, right now, someone could sit in Eastern Europe or Asia or anywhere that they wanna sit and still be able to conduct operations. Where, wherever they’re able to, and then now when you build in the ability to build agents to go out and do that one, it’s faster.

And two, you can build sort of robotic, you know, robotic armies that are operating in, in, in terms of ones and zeros through. Through what we use for, to communicate all the time, and it that becomes just an avenue of, of, of approach and an avenue for them to get into the environment in a scale that, like I, you know, you’ve had Phil Venables on here, he talked about it in his recent blog coming out of, o out, out of RSA, right?

Like it is just. I think there’s just a, a nervousness within the security community about like what is gonna happen in the next six months. Right? And look, in February we’re talking about what’s gonna happen over the next year. Right. You know, and now we’re a month later and it’s like, it’s here. It’s not the next year.

And so now everyone’s like trying to tighten the timeline down a bit. But not to sound too spooky, but it really could be tomorrow. Right. And I think that’s like, that’s the, the scariness and like how do you, you not get ahead, but get up, get up to speed.

When binaries break

James Kaplan: You know, if I were to take a step back and I were to synthesize the discussion we’re having, the theme is one of blurring lines, right? We talked about blurring of lines between the academy and the intellectual world and the business world. We’ve talked at some level, I don’t know whether it’s the blurring of lines or the overlap. between the sciences, the social sciences, and the humanities, how important, the humanities are for applying social, you know, computer science, for example, effectively. now we’ve talked about the blurring of lines between the geopolitical world and the commercial world, right? At least in the, in the current tier. I was wondering, you know, and yes, I admit this is a bit of a weighty question. If you just take a step back and. How should a CISO A CIO think about this? Like we have this world with many fewer binaries, and what does that mean for a leader who is trying to translate all this uncertainty into practical action?

Charlie Lewis: You know, that that is the, the, the million or — in many companies’ cases — the billion-dollar question. And I think it really comes down to — I know, I know — and I think there’s sort of a structured flow that I try to think about when having these conversations and like it’s imperfect, right?

The conversations have to happen at the C-suite level, right? And we try to get into those as much as possible and you have to be able to. I actually get really nervous when I talk to a client and they’re trying to hire a CISO. Like I want a really technical person who’s worried about the inside, doesn’t have a lot of broad engagement, right?

And I’m like, you need that technical person, plus you need. Someone who can communicate to the business. Plus, if you sell something, you need someone that customers can talk to and feel confident that you’re securing their product. Right? And so I think broadly that the leadership of a business has to look at not just technology now, but security of the business.

As a core sort of business enabler is a cheap way, but as part, as a core component of the business, right? You wouldn’t run a business without investing in hr. You wouldn’t run a business without investing in your finance arm, right? You’re, you, you need to do the exact same thing from a security standpoint and make sure that it is treated as that level of importance and it is a critical component.

But the way you have to do that from a C-Suite or from a CIO or a CSO talking right, is one, it’s critical to outline what are the requirements to run the business, right? Phil will talk about the minimum viable organization, right? And, and so. What is critical to run that broad business, you have to have the engagement with business leaders and say, for you to conduct your operations, what are the three applications, the four applications you need?

What is that process? All right, great. I’ve now gone through there. What this actually means for you is I have to maintain and prioritize these following pieces of infrastructure, right? Like down to like the load balancer level. So you understand what that is, a business person, and then you say, here’s where we’re performing against what our expectations are.

So you have to work with the business to say, what is your risk appetite? This is what you and I spent a lot of time on before, right? What amount of risk are you willing to take on? How long are you willing to be down? It Right. How. You know, how, how, much data are you willing to lose, right? What are you thinking about from a reputational and trust issue?

Now, from an integrity standpoint, we used to not talk much about integrity. Now, the data poisoning, data leakage broadly throughout it. Like that’s a massive risk, right? And then you have to frame that and what that investment is in, what it means to the business, right? So it’s not, oh, we, we have risk of, of, of ransomware, or we have risk of, you know.

X number of apps don’t have MFA. That doesn’t mean anything, right? What really means is like, unless we implement this, I actually think we’re more, we’re not within risk. We’re not within our risk appetite for disruption for a standard operation. Look, there are. On the extremes, you’re not gonna solve for those, right?

Like if you’re not, you’re not gonna solve for those. But if you’re able to bring all of that together and to have that conversation across the business one-on-one, show them what they actually use from a technology standpoint to get their work done, show them what that requires, all the way down and then, right, that gives you a good opportunity to get your CMDB up and running.

Right to make sure that your IT asset management program is good, that you can run through business continuity plans, right? And, and if, if, if, God forbid, something happens and you are disrupted from a tech resilience standpoint, which we’ve seen in some of the largest banks to ransomware, which we’re all much more familiar with, I think you understand what the recovery process is and what the order is to get everything up and running.

James Kaplan: I hear you, Charlie. If I play this back, I’m proposing the destruction of another binary between the strategic and the tactical. If I hear what you’re saying, you need both — to connect all the dots, right? — to engage as a systems thinker who can connect the business to the architecture, to the controls, to the security operations. And at the same time, you need to translate those implications down into a set of actions that will change the nature of the environment. You know, okay, we’ve thought about all these things and this means we have to do this about the web application firewall or this about which data we encrypt.

Charlie Lewis: Or. Exactly, or, or even where you are running or storing your data, if you have that level of fidelity on a server in a, in a, in a data center in one country or one region versus another, making sure you have that. You know, and there’s little things too, right? Everyone wants to offshore every their, their work right now and.

Great. Tons of cost savings. Right? There is good opportunity, a ridiculous amount of talent where everyone wants to offshore in India, right? But what they’re doing is they’re offshoring the, what’s that?

James Kaplan: Vietnam, Latin America,

Charlie Lewis: Eastern Europe. Right. Vietnam. The Philippines, Uruguay. Right. Mexico City, San Jose. Right.

There’s a ton there. Right. But most organizations are just thinking about it in one spot, opposite side of the globe. But they also don’t think about like, what does that mean if the people go down, right? If something happens in their systems are down.

We worry about hurricanes in the southeast of the us right?

You’ve gotta worry about. You know, large range, you have potentially have to worry about flooding and power issues. Right. And, and so how are you thinking like structurally, like you said, operations, strategically we’re making this decision like technically and tactically this means this, and then operationally, like you said, there’s that little bit of that piece in there and how do I connect those two to make sure that we’re prioritizing and the other things like.

Tech teams and CISOs need to be aware of, right? Is is there’s a lot of technology. It’s not shadow it, right? But there’s a lot of technology that the business owns. That does not get managed as well or in the same exact way as what the technology team and the security team own, and you have to do more now than just the annual manual attestation of where that, what it is, what it’s used for.

Does it have the controls and where does it sit? You have to have those conversations. If anything moves. Or priorities change, and that has to be more continuous or at least more frequently than, than, than, you know, once a year.

James Kaplan: That’s another binary that’s being blown up between what is it and what is not. It,

Charlie Lewis: Correct?

James Kaplan: more things get connected to the network and, and operating theater is part of your technology environment. A factory line is part of your technology

Charlie Lewis: Yep.

James Kaplan: A bench in a lab, in a pharmaceutical company is part of your, technology environment.

The vending machines in a break room are part of your technology environment.

Charlie Lewis: The conference rooms. Right. It’s, it’s, it’s all there. And it, you know, and, and in fact you could say, you know, if you think broadly on the security side, your ID badge. Right. And where can my ID badge get me into? that’s part of the technology. ‘cause as soon as someone can get in, they’re able to figure they can get around and they can, they can see a lot more.

I can learn a ton about what’s going on just by getting into a business. Right. And we look for the badge, not the face. Right.

James Kaplan: Okay. Charlie, anything else? I, we’ve covered a ton. Anything, any, and, there’s a ton we haven’t covered. we only

Charlie Lewis: I know you didn’t mention, you didn’t mention constructivism, but that’s fine because that’s for another conversation.

I think that’s been, that’s been blown away anyway, so

James Kaplan: we, we drove by and waved at Samuel Huntington, but didn’t stop, to talk about him.

But anything else?

That you want to, cover in this discussion? Any, any last thoughts you wanna leave, people with?

Charlie Lewis: I am obviously biased ‘cause I love what I do, right? I love security, right? And I think that, being a good security professional means I need to understand the business, I have to understand the technology, and then I have to be able to protect both of them, and find ways to also help them achieve what they they need to achieve.

The push towards Agentic AI is making for security operators in one, understanding how they secure their own internal environment and what is being built so the business is able to achieve what they want to and that. Slow them down. And there’s tons of good opportunity there, specifically around the development life cycle, right?

The, the, the, various security checks will start being run by agents and you just have sort of a, a QA towards the tail end, and that’s awesome, right? You can run threat modeling much faster, more effectively, right? But it also means that the threat actor is coming and this is changing. The vulnerability landscape and requiring an entire review of what used to be a low VUL or a medium vul, and is it now potentially a critical vul?

And there has to be thinking about what does this require and security teams should start investing now and anticipating it because if you, it happens before you’re ready, right? There’s very little chance that you can catch up effectively without a ton of spend or additional risk and, of, threats being realized.

James Kaplan: So let’s close with one final question that’s even more serious than vulnerability management. When I spoke with Rich, he said The Hartford Whalers would beat the Buffalo Bills and shuffleboard. Let’s say the game was Scrabble. Okay? The Buffalo Bills versus the Hartford Whalers and Scrabble, who wins?

Charlie Lewis: Bills first off, the Hartford Whalers don’t exist anymore.

James Kaplan: Well, the.

Charlie Lewis: I’m just gonna be very objective on this one. Right. The NFL has rules on how far you have to go in college before you can play professionally, whereas the NHL does not.

So I’m going to make, I’m gonna go back against what I said before, ‘cause I also believe security folks don’t need a degree. . But in a world where you’re comparing those two, right, I think that having to get into college and go through college Will, will roll, will roll with, with, with that one.

Plus, Josh Allen, you know, brilliant, brilliant human being. and so he’s able to go there. But that, I, I also just wanna say that the first ever professional hockey game I went to was the Buffalo Saber versus the Hartford Whalers. and again, that’s why like above your book over there, I have a Pat LaFontaine, so to be frank, I don’t remember who won, but I remember standing on the boards and some Hartford Whaler took a slap shot right at the board and the puck like would’ve.

Hit me right in the face. And so I’ve always had a bit of a, even though I live in Connecticut, a bit of a negative view, on, on that one.

James Kaplan: The first professional sports evet I attended was the Yankees versus the Oakland A. In 1978, the Yankees won and it was less violent. Nobody, there were no slap stakes. They slap shots aimed at the at the

Charlie Lewis: No, hockey, like, I love that sport is an incredible sport. The movement, it’s, it is, it is a beautiful sport, I think. but I also can never let Rich win, anything. So that’s,

that’s the core.

James Kaplan: We’ll have to have the two of you on together. You can debate

Charlie Lewis: I could, I knew I couldn’t compete with his microphone the last time, so that’s why I couldn’t join.

James Kaplan: that is exactly true.

Thank you

Charlie Lewis: thank you, James.

Office workers often use technology to create inefficiency more efficiently -- nobody enjoys this, and it does nothing for the share price.

Chaos in the cubicle farm has many causes, some of which are societal -- but this moment of technological ferment affords CIOs and CTOs their opportunity to improve knowledge worker productivity. Here’s what you can do:

1. Champion data modeling as a skill set, even outside of the technology organization

2. Deconstruct the office suite

3. Build MIS that supports a systems-thinking view of the business

Organized factories, chaotic offices

Why are our offices so much more chaotic than our factories? [1]

AI might only make this worse, as office workers use large language models to create inefficiency more efficiently -- the Jevons Paradox applies to busy work too!

• Sally will use AI to create a 300-page manual on policies for policy compliance.

• Himanshu will use AI to summarize Sally’s manual into a set of action items.

• Bill will use AI to schedule a meeting to review the action items that Himanshu suggested.

• After the meeting, Igor will try to convince Himanshu, across a 50-message email thread, that Sally’s policy manual doesn’t really require action item 4.

• Igor meets with Sally every week to review the compliance-tracking spreadsheet he used AI to create.

The gang produces nothing but compliance.

Where does this take us? Where will growth and innovation come from as a monotonically increasing fraction of humanity devotes its days to writing, analyzing, and complying with policy manuals? Also, does anybody enjoy this, except Sally?

So what do we do about this? As technologists we promised that we could improve office productivity. When I listened to Joe Jackson’s song Tomorrow’s World in college, I didn’t hear anything about creating inefficiency more efficiently.

Some of this is cultural, and hard for us to address. We live in a more bureaucratic and risk-averse age. Some of this is good -- no you shouldn’t put sensitive information on an unencrypted thumb drive. But, in aggregate, what was a Tuesday afternoon in 1999 is a firing offense today. For decades social thinkers have sought to replace Napoleonic leadership with consensus management. Yes, management by diktat is bad, but so is a world where everybody can say no, and nobody can say yes. [1]

What does a CIO or CTO do about this? You aren’t going to unwind a connection between societal wealth and risk aversion with your project portfolio. But there may be a few actions you can take to reduce the chaos.

1. Champion data modeling as a skill set, even outside of the technology organization

When still an academic, Robert Reich wrote an interesting book called The Work of Nations, which divided jobs into three categories: routine production (working on a manufacturing line), in-person services (being a nanny), and symbolic-analytic work (largely white-collar office work).

Reich posited that symbolic analysts identify and solve problems by working with ideas, data, and abstractions. They collect data and use that to build models representing bits of the world -- financial models of entire companies, process models of business domains.

So we have millions of people dressed in business casual paid a lot of money to build representations of the world -- and only the tiniest fraction of them grasp even the basic principles of data modeling.

Imagine if the people structuring business cases, financial models, business strategies and operational reports understood the difference between 1:1, 1:many, and many:many relationships.

This problem will only get worse. GenAI will vastly expand the data we can analyze by converting unstructured text to structured data. The machines won’t magically create ontologies for us. I spent last week tuning the data model for ProsaicGraff, for example combining multiple entity classes to support easier decomposition -- deciding to do this required judgment about the business problems I need to solve. At the same time, universities may not be teaching the basics here. I spoke to several students taking classes in data science who looked at me blankly when I asked them about Codd Normal Form.

Imagine if CIOs, CTOs (and CDOs!) could ensure that everyone in the enterprise technology organization knew how to model data. Imagine they could help knowledge workers in marketing, strategy, finance and operations understand at least the basic principles of data models. Would that reduce some of the chaos as office workers seek to understand the company around them?

2. Deconstruct the office suite

Unix is awesome, in part, because it is composable. You can build complex behavior by chaining together small, simple programs that read text via stdin and write text via stdout.

When a programmer types the following line into the terminal

cat access.log | grep “ERROR” | sort | uniq -c | sort -nr | head

she tells the OS to read a file, search for a string, sort the results, count the results, sort them, and show the first ten items in the ranking. Wow -- imagine if office workers had that type of power in making their sales or marketing plans! But the modern office suite is the opposite of Unix -- it prevents composition by collapsing data, logic, and presentation into monolithic files.

Do you remember MSFT’s Project Longhorn? Its WinFS stream would have created a relational/semantic layer over the file system. Files would be typed objects, supporting rich relationships and SQL-like queries. It would have been a knowledge graph for the desktop before knowledge graphs were cool. Sadly, the early 2000s technology may not have supported the vision, so MSFT reset Longhorn and shipped it as Vista without the file system innovations.

Maybe WinFS would have given office users some of the power in working with documents that programmers have when they manipulate text in Unix. More importantly, we should remember that the current mode of copying and pasting data between email clients, word processors, spreadsheets, and presentation software is not eternal. It started back in the 1990s, and maybe it could end. Given the chaos that rules most office workers’ professional lives, maybe CIOs and CTOs should end it.

That could mean replacing traditional word processing software with DocOps pipelines that produce documents-as-code. Or it could mean blowing up the traditional spreadsheet.

I don’t want an LLM to help me structure a better spreadsheet. I want never to create another spreadsheet again for the rest of my career. I don’t want to scroll through endless rows and columns. I don’t want to try to compress n-dimensional data into a 2.5 dimensional construct. [3] I don’t want to munge the data, logic, and presentation layers into one file.

I want to store complex data in a JSON database. I want to use an LLM to generate robust queries against it. I want to create business logic separate from the data. And I want options for piping the results into different visualization mechanisms.

Will that require behavior change? Yes -- a very small quantum of change compared to the advent of spreadsheet and word processing software. Before the advent of VisiCalc and Lotus 1-2-3, clerks entered data, not managers and executives. Executives couldn’t even use a keyboard -- they wrote memos by dictating them to their assistants. Compared to the behavioral change required between the late 1970s and the early 1990s, giving up your traditional word-processing package for a DocOps pipeline is small beer.

CIOs and CTOs have a critical role to play here in shaping a less chaotic future. What should a DocOps pipeline look like? Will it use XML, Markdown, or a proprietary document format? If we blow up the spreadsheet, can we interconnect data stores, so we don’t replace fragmented, atomistic spreadsheet files with fragmented, atomistic JSON stores? If we represent financial models as a series of equations, where do we store the equations, and should we subject them to change control?

I hypothesize that managing relationships as first-class elements of your data model will be essential. But that is just a hypothesis -- we have many innings of baseball left to play here.

3. Build MIS that supports a systems-thinking view of the business

We’ve all read about the rogue AI that destroys humanity because it wants to use all the matter in the solar system to fulfill its objective of maximizing paperclip production.

I’m not worried about AI fixated on paper clip production [4] -- I worry about Sally, who will demand compliance with her policies on policy compliance. And about the VP of sales who will push for every last deal, pumping up growth with unprofitable revenues. About the procurement manager focused on unit cost over quality. The HR director pushing for a location mix or a pyramid structure that prevents line managers from attracting talent. Even the CISO who cares not for speed or innovation in setting control objectives and performing risk assessments.

Sometimes the modern corporation feels like a series of tribes, each with its own imperative, all working at cross-purposes to one another. This makes life awfully tough for senior executives as the Street will start with questions about profitability and economic value. The imperatives of each individual unit? Less important to shareholders.

Business strategy requires systems thinking, and better systems thinking leads to better business performance. You have to look at all the component parts and all the constraints in order to move from managing individual silos to managing the business as a whole. Anthropic seems to acknowledge this -- but building systems thinking into the way you manage the business requires more than a prompt.

It requires, to use an old-fashioned term, MIS built on systems-thinking logic that surfaces and quantifies the implications that decisions made in one area will have on all the other areas. Most companies suffer from MIS that does the opposite of this -- that provides information on purchased-goods costs to procurement managers, pipeline information to sales managers, and compliance information to compliance managers. Current MIS (whether called business intelligence, data warehousing or decision support) doesn’t integrate across organizational silos, and provides precious little support for what-if analysis, so it helps most with narrow, execution tasks.

CIOs and CTOs have a unique role in the modern corporation. Other than the CEO (and maybe the COO and CFO), only they see all the moving parts in the machine -- because they support all of them. They see the entire graph. How can CIOs and CTOs advocate for the MIS that helps the organization manage the business as a system?

Traditionally, CIOs have said solving problems like this required multi-year, multi-hundred-million-dollar ERP implementations. ERP is still important -- transaction integrity will always matter.

But there may be an opportunity now to decouple business insight from heavyweight ERP programs. GenAI is bad at many things (like writing!) but it’s really good at validating and integrating data. CIOs and CTOs can now build platforms that illuminate how the thorny tradeoffs senior executives face will shape financial performance across the entire business -- without modernizing all the underlying systems. That is an interesting role for CIOs and CTOs to play.

Every technologist my age (i.e., old enough to remember mixtapes) that I know says the same thing. It feels like the 1980s. By that, I think we mean everything is up in the air, and we can use technology in new ways we hadn’t thought of before.

CIOs and CTOs can take advantage of this moment of uncertainty to help reshape the way knowledge workers use technology to reduce the chaos and bring more of the productivity gains the world has achieved on the factory floor to the office.

Footnotes

[1] The footnote here could be: “Have you ever been to an office? Have you ever sat in a meeting?” But:

• More than twenty years ago the OECD noted faster productivity growth in the factory-intensive manufacturing sector than in the office-intensive service sector.

• Perhaps contradicting my hypothesis, the OECD also reports better productivity growth in knowledge-intensive service industries than in other ones.

• Yes, always take vendor-supported research with a grain of salt -- but this Asana study on work about work is interesting

For all these reasons, I like to say: “You think the guy behind the guy behind the guy is Sam the Eagle? No, it’s Gonzo. Fozzie helps.”

[2] Society must ask: how do we get the brilliance of Austerlitz and Jena, without an inclination to seize Moscow?

[3] X = columns; Y = rows; Z = sheets

[4] We can fix that with one line in the prompt, even if we can’t crib the language from Isaac Asimov.

The main argument: Turning relationships into files to vanquish the Chaos Muppets

What you need to know

• Chaos Muppets run the corporate world, leaving the rest of us mired in ambiguous, poorly structured data.

• That degrades our ability to analyze and automate business domains—we have organized factories and chaotic offices, in part, for this reason.

• We need a lightweight, file-native, closed-world semantic system that treats relationships as first-class, schema-validated objects.

• Modeling relationships as subject–verb–object junctions—stored as files, constrained by rules, and queried by agents—offers a practical path forward.

• In turn, this pattern could improve domains like RFP responses, regulatory compliance and enterprise change programs, among many others

Chaos Muppets run the corporate world

Science has made two great advances in understanding how humans tick in the new millennium: positive psychology and Muppet Personality Theory. Positive psychology asks what makes individuals and communities flourish rather than how to treat mental illness and dysfunction.

Muppet Personality Theory divides all of us into Chaos Muppets, who make their way through life in a “swirling maelstrom of food crumbs, small flaming objects and the letter C.” Order Muppets “resent the responsibility of the world weighing on their felt shoulders, but they secretly revel in the knowledge that they keep the show running.” Scooter and Kermit are Order Muppets. Cookie Monster, Gonzo and Dr. Bunsen Honeydew? Chaos Muppets.

Everyone at the office thinks I’m a Chaos Muppet, but an anarchic sense of humor does not make you a Chaos Muppet. [1] Sam the Eagle is an Order Muppet, and he is humorless. Sam is not humorless because he is an Order Muppet, as Jed Bartlet might have pointed out. Just because I find Dr. Teeth hilarious doesn’t mean I’ve joined the Electric Mayhem. [2]

Mrs. Prosaic Times knows better. She knows how much I hate psychic entropy and how much it prevents me from achieving flow state. She knows that I compensate for an inability to impose order on the universe by imposing order on the utility closet. [3]

Sadly for me, Chaos Muppets run corporate America. You think the guy behind the guy behind the guy is Sam the Eagle? No, it’s Gonzo, with assistance from Fozzie Bear. That’s why we have organizations with incomprehensible remits, duplicative initiatives and documents about documents about yet more documents. It’s Gonzo’s world. We just work here. [4]

I’ve tried generations of software (remember Lotus Organizer?) to try to impose order on professional life. None have been able to model the complexity—and by that I mean the incredibly ambiguous data—of business life at any sustainable level of process overhead. [5] [6] So what information technology have I used? Boxes and boxes of 6″ × 10″ notebooks, sometimes with graph paper in them.

Relationships are the important data, and you can store relationships in files

In the past, I’ve noted that corporate life has chaotic offices and orderly factories. If we can model “office” data, we can go a long way toward resolving that discrepancy.

ProsaicGraff attempts to model complex, many-to-many relationships across ambiguous data sets, using three principles: treat nouns as highly abstracted classes, use verbs to describe relationships between nouns, and use agents for validation and queries.

Treat nouns as highly abstracted classes with minimal metadata
I originally used several topical constructs in structuring my data—a folder for the Technology Leadership Forum, different templates for proposals versus engagements versus service line initiatives. Too messy, not scalable, and hard to figure out where to put things.

Eventually I articulated a set of primitives that dominate my professional life: people, institutions, groups (like TLF!), events (like the May 7th TLF session), meetings, projects, places, milestones, documents and topics. I really wanted to track the relationships between instances within a class or across classes:

• The May 7th TLF Session will be held at the McKinsey office in Chicago

• Jane Schnauggs [7] is a member of TLF

• Jane Schnauggs works at AmalgamatedHealthServices

• Brown University Medical School is part of Brown University

• The meeting we have tomorrow at 9 am will cover agentic security

You can imagine a million more things like this, all of which might be relevant to my (or, with a slightly different twist, your) professional life.

But relationships in corporate life are complex. A CTO might be a member of TLF, or she might be potentially invited by the McKinsey client team, might have been invited, might have accepted the invite, might have declined the invite or might formerly have been a member but left the group when she took another job with a different set of responsibilities.

A McKinsey partner might be part of the client service team supporting AmalgamatedHealthServices—but he might be a “core member,” who spends a lot of time with that institution across many issues, or a thematic member who focuses on one piece of the organization.

A private equity firm might own all of StandardSocks or it might have a majority stake or a minority stake. Distinctions like this matter.

But what mechanism could I use to capture relationships with the precision I wanted without unbearable process overhead? Neither hashtags, nor wikilinks, nor YAML front matter seemed up to the task.

Why can’t relationships be files?

Could I use a verb to model the relationship between two nouns, a subject and direct object? I could. In some cases, I could even include a third noun in the form of an indirect object.

Eventually I built out 33 verbs, organized into eight top-level categories in the vault (employment and affiliation, interpersonal, membership and participation, content, structure, geography, knowledge and a generic catch-all—see Verbs/).

This is for Attends (which describes a relationship with an educational institution):

Verb - Attends

You see that any subject associated with “Attends” must be a person, and any object must be an organization. You also see that we can attach tense and degree information to the Attends verb. [8] We can separately track that:

• Someone formerly attended Brown University and received an AB there

• The same person currently might attend the Wharton School to receive an MBA there in 2026

Rules stored as data provide further validation. One rule mandates that degree_received cannot be yes if tense is present or future. Once you received the degree, you no longer attend the school (for that degree). [9]

Verbs are great, but how to store the connection between subjects, verbs and objects? We’ve also created millions of junction files over the years to link tables in a relational database. Why couldn’t I create a markdown junction template that linked nouns to verbs? For example, there is a junction file with the following information:

All of this is part of a Git repo with the version control and time-stamping that implies. I have nearly 5,000 junction files already—but that’s okay as each one tends to be about 300 bytes and I interrogate them programmatically rather than by hand. What’s the right way to do that?

Use agents for validation and queries
Even a couple of years ago this construct would have been impossible. What language would I have used to query the bloody thing? I would have had to write my own.

Claude Code skills create dashboards. Every morning at 3 am skills rip through the entire vault and create an updated dashboard for each person, project, organization, event and group. For example, for each project it interrogates the junction files to indicate

• all the people participating in the project, with role

• other projects where there is a dependency

• milestones or outputs and status

• meetings

• documents

• tasks

For ad-hoc queries, I tend to use the Cursor agent window. Each agent reads the relevant context on a part of the vault before interacting with me. For example Serena always has current context on TLF. Aaron, our agentic board secretary, always consumes current context on Brown-RISD Hillel.

A Python validator script runs on each commit. It checks verb schema conformance (subject/object types), field value constraints (rules), conditional field requirements, cardinality, symmetry, anti-reflexivity and prerequisite junctions. Yes, that’s deterministic, but agents help me write the script.

As a note, getting the verbs and the relationships between nouns and verbs required careful thought. I experienced massive verb sprawl—at one point close to 150 verbs—before I started using adverbs and tenses to get that down to a manageable number. I also had some wonderful linguistic discussions with Claude Code about, for example, the distinction between “Relates-to” and “Covers.”

This may be new and apply to a lot more than my personal knowledge base

Originally I had assumed I would grow my repository out of Obsidian and into a grown-up graph database environment. Data scientists and engineers who have worked with graph databases for years tell me this pattern—treating each relationship as a first-class file with its own typed schema—doesn’t have a clean analog in the systems they’ve used.

That’s interesting because the same ambiguity that applies in my life also applies to many big, complicated, messy corporate functions—large change programs, regulatory compliance, B2B sales. The same pattern for managing relationships across ambiguous data sets might apply in these domains, creating new opportunities for insight analytics and agentic automation.

A bank running a consent order has hundreds of requirements, each with a control, each control backed by evidence, each piece of evidence tied to a document, each document owned by a workstream, each workstream staffed by people with specific roles. In most implementations, that information lives in a spreadsheet or a SharePoint folder tree—which means someone manually checks that every requirement has a control, every control has evidence, and every piece of evidence points to a real document. That someone is usually a program manager doing it with Ctrl+F.

The approach used in ProsaicGraff would make those connections typed and validated. The link between a control and its evidence isn’t a cell reference—it’s a relationship with a schema that says: this field is required, this value must come from this list, and if this condition is true, this other field must be populated.

The validator runs on every commit and tells you exactly which requirements are missing controls, which controls have no evidence, and which documents are referenced but don’t exist. The audit trail is Git history. You don’t need a program manager with a spreadsheet. You need a commit hook.

In a labeled property graph the control-to-evidence edge exists, but it’s a set of key-value pairs—you can put whatever keys you want on it, or none at all. Nothing in the database enforces that every control–evidence edge has an evidence_type field, or that evidence_type is one of {documented, tested, attested}, or that if status is closed then closure_date must be populated. You write that logic in application code, or in a separate validation layer, or you rely on the program team to do it manually.

When the compliance team provides new guidance on what “adequate evidence” means—you update the application code, retest it, redeploy it, and hope the existing edges that now violate the new rule get caught before the examiner does.

In RDF you get the semantic precision but at significant operational cost. SHACL can express exactly these constraints, but it runs as an external reasoning layer—it’s not native to the store. The triples themselves are clean but verbose; every relationship requires multiple statements to capture what a single ProsaicGraff junction file captures.

In ProsaicGraff the constraint is a file. Changing what “adequate evidence” means is a one-line edit to a rule file, committed to Git, with a timestamp and an author. The validator immediately surfaces every edge that violates the new rule. The audit trail of the constraint change is as durable as the audit trail of the data.

Here are a few domains where this approach might be interesting.

• Enterprise sales — Full provenance of relationships: who introduced whom, when, through what channel; multi-employer, multi-institution history; temporal tracking via tense

• Corporate program management — Program structure as a graph; sponsor, workstreams, vendors, dependencies—all typed and property-bearing; roles distinguished without schema changes

• Regulatory compliance documentation — Traceable requirement → control → evidence chains; each control can cover a specific requirement with a typed relationship; conditional fields enforce documentation completeness

• M&A / due diligence — Ownership trees, management networks, and findings as a unified graph; conflicting findings formally modeled via contradiction

• Investment thesis management — Full argumentation graph: thesis → supporting propositions → facts → sources; counter-evidence modeled; confidence tracked

• RFP and proposal management — Who worked on what for whom, in what role, producing which deliverables; knowledge reuse across proposals

There is much baseball left to play here

Obviously, this is an experiment, not a platform. I have 5,000 junctions running on my laptop. Applying the pattern to an enterprise-grade domain touching hundreds (if not thousands) of users would imply millions of junctions per installation. Some colleagues and I have talked about how we might extend it.

For example, we might store transactions rather than state. Right now we store state: Kaplan-James is employed by McKinsey. Would it increase analytic power if we stored transactions (McKinsey hired Kaplan-James) and derived state? If we did that, would we need a defined starting state? That shift would do wonders for temporal queries.

We could also use SQLite to create a more performant query layer. We could add a lightweight inference layer—maybe that naturally follows if we move from storing state to storing transactions?

I want to put some real thought into extending the argumentation model and creating a robust framework for decomposing goals into actions and into tasks.

Footnotes

[1] In some cases, colleagues think I’m a Chaos Muppet because, as a disciple of Edward Tufte, I insist on understanding complexity and refuse to paper over it with over-synthesis. Shortly after I met my colleague Brian Elliott he told me “I will go to any level of granularity I need to in order to solve the problem.” And I thought: “We are so going to be pals.”

[2] Because we’re doing Muppet discourse, I might as well share one of my Dad’s favorite stories. The Muppet Show launched in fall 1976—I was six. My mom had a PTA meeting, or something that evening, so Dad, Rob and I settled in to watch this new show.

There was a running gag in which the old-fashioned phone backstage would ring. And when someone picked it up? Chaos! Not a voice on the line, but coins or water streaming out of the speaker. I found it hilarious.

As you might imagine I was an unusually articulate six-year-old, the kind who believed that if you explained your point clearly even an adult might understand. But when my mom returned, my glibness failed me. I could not get the words out quickly enough to explain how exciting and funny the show had been.

Like me, my dad likes to recount stories, so I don’t know how many hundreds of times I’ve heard his impression of me trying to get the words out to explain the telephone to my mother. In part, I suspect he figures he should treasure one of the few times he saw his older son at a loss for words.

[3] Frequently heard around the Prosaic Times newsroom, at least when Matthew is home from college—

Matthew: I need a charging cable for my iPhone
Daddy: [Looks up from book. Says nothing. Raises eyebrows.]
Matthew: You mean I should go in the utility, look for the clear plastic bin that says “USB-C to USB-C cables” and get one from there.
Daddy: That sounds like a plausible hypothesis about the world, grounded in past experience. You should test it.
Matthew: Other kids have normal dads.
Daddy: You would get bored.

[4] Except for the factories. If Sam the Eagle is on the executive committee, he’s the SVP, Operations.

[5] As an Order Muppet, I crave structure. That doesn’t mean I love process. In fact I hate it.

[6] Oh, yes, I learned a lot playing with Obsidian and Cursor.

I made every mistake possible in setting up my Obsidian vault. Don’t take any of this as a criticism of Obsidian, which I love—all of these organizing constructs appear in multiple contexts and always have the same limitations.

Sub-folders are inflexible. We’ve all used folders and sub-folders in our file systems to organize documents. It rarely works because a folder tree is hierarchical and life isn’t. Also folders don’t have metadata beyond the folder name. None of this works any better in Obsidian than it does in File Explorer or Finder.

Hashtags don’t scale. Yes, hashtags are flexible. Yes, hashtag hierarchies are sort of cool, but they quickly become unworkable at any sort of scale. Syntax enforcement? Hardly. And any individual hashtag is a thin channel for information. The hashtag #StandardSocks tells us a note has something to do with that company, but not what.

Links are semantically shallow. You can create wiki-style links easily in Obsidian. Just type StandardSocks and a click will take you there. Great for navigation, less so for analysis. Yes, this statement relates to that document, but how?

Front matter gets messy. Obsidian’s support for YAML front matter is very powerful, but eventually runs into scale problems all its own. As you would expect, I have many different types of people in my Obsidian vault—McKinsey colleagues, clients, people in the Brown University ecosystem—each of which requires slightly different combinations of metadata. Eventually you either wind up with dozens of sparsely populated front-matter fields or you start segmenting your templates, which creates other problems.

All of this creates a combinatorial meta-problem. Each organizing construct works better for different uses, but then you wind up asking yourself “wait, did I use front matter or wikilinks to connect people to projects?” Questions like these do not reduce psychic entropy.

Cursor helps, a little. The agent window reduces overhead in updating data and performing ad-hoc queries, but doesn’t reduce the underlying complexity.

[7] Jane Schnauggs is a composite for illustration. Interested in her adventures with the Senate Homeland Security and Governmental Affairs Committee when HyperCare, Inc. suffered a cyberattack while she was CIO there? See Chapter 2 of Beyond Cybersecurity: Protecting Your Digital Business.

[8] When I get a chance I will probably create a taxonomy of majors (e.g. history, English, etc.) and degrees (AB, ScB, MBA, JD, etc.) and make those fields non–free-text. That will make analytics a bit easier here.

[9] Schema rule: once a degree is awarded, you shouldn’t still mark that program as present- or future-tense “attends” for that credential.

This is the wrong way to at it. We see the great advance in human learning since Gutenberg. Ideally without it leading to a modern-day Defenestration of Prague. Michael Littman is wrestling with the big issues at Brown University.

Here’s a few of the things we talked about:

• Implementation to Quality Control: The human role shifts from writing the “how” of code to performing high-level verification and contextual judgment of machine-generated output.

• Higher Agentic Cognitive Load: Orchestrating multiple AI agents increases mental strain, requiring a “bond trader” mindset to manage complex context and flow.

• Closing the Abstraction Gap: Gen AI finally allows computing to tackle qualitative “messiness” and fuzzy objectives that traditional quantitative abstractions couldn’t reach.

• Motivation over Information: While AI excels at personalized instruction, the “rock star” lecture survives because humans require interpersonal inspiration to engage with difficult material.

• A Paradigm Shift in Humanities: LLMs enable a “Kuhn revolution” by allowing historians to systematically interrogate massive datasets that were previously locked in dusty, qualitative archives.

James Kaplan: Hi there. This is James Kaplan, and welcome to our most recent Prosaic Times podcast. We have a slightly different guest today—one I’m incredibly excited about—from the world of academia. Michael, do you want to introduce yourself?

Michael: I’d be happy to. Hi everybody. I’m Michael Littman. I’m a professor of computer science and artificial intelligence at Brown University, and I’m also serving as the university’s first associate provost for artificial intelligence. I’ve been having a wonderful time interacting with James on a variety of topics.

James: So, do you want to tell us a little bit about your journey? How did you become a computer science professor? And tell us a little about your area of academic research.

Michael: Sure. Thanks so much. When I was a teenager in the seventies, I was wandering through a shopping mall and looked through a window and saw a computer in a Radio Shack. I didn’t know what it was, but it looked very interesting. I walked up to it and started typing things at it—and it knew the answer to all the arithmetic problems I could pose.

I tried to be really tricky. One plus one? Two. Okay, sure, everybody knows that. But what about one plus one plus one? What if I put parentheses in? No matter what I did, it figured all of them out. That was the kind of thing I thought only people could do. I guess I didn’t have a very good calculator back then, but it just blew my mind. I found it fascinating. I kept talking to my parents: I think I want this. They said, well, it’s very expensive. Have your bar mitzvah. I said, I don’t want a bar mitzvah, I just want the computer. So they said okay. I got the computer. I spent the next couple of years rewiring my brain to become a computer scientist, and I’ve never really looked back. To me it’s just so deeply interesting.

What draws me to it in the first place is how much it feels like a machine that’s thinking. The area of computer science that studies that—what we can think of as a kind of thinking—is artificial intelligence. Once I got to college and learned that word, it was: oh yeah, that’s my path. I’m going to be a computer science professor who studies artificial intelligence. I’ve tried to stay pretty focused on that and it’s worked out really well.

Except now I’m a university administrator who studies artificial intelligence, and that was not on my radar as a teenager.

James: You woke up one day and found yourself in the middle of the complexities of university administration—something artificial intelligence had never quite prepared you for.

Michael: You can think of it as a kind of complex computational system, because it really is. It’s very organic; it’s got people in it, and people are so complicated and weird and noisy. It has a lot of aspects that feel very different to me from computing.

But what I’m really loving about this position right now is that I’ve always been interested in AI. I always thought it was super interesting and nobody believed me—until ChatGPT came out, and then suddenly everybody was really interested in talking about artificial intelligence. I feel like: this is great. I finally have the conversational partners I’ve been wanting all these years.

James: I’ll admit I thought machine learning was really boring, and here’s why. The first time someone showed me machine learning, I thought: yeah, okay, that’s sort of interesting. Clearly they’re doing massive numbers of correlations and using some sort of champion–challenger model to find their way to the best optimization algorithm. From what I understand that’s not precisely how it works, but it’s kind of how it works. And I said: that’s great if you have a really clear objective function and at least reasonably good quantitative data. But the problems that interest me and the problems I have to deal with often don’t have really clear objective functions and never have really good quantitative data. What came to interest me about Gen AI is that we could work with fuzzier objective functions—and even more, we could attack problems by hook or by crook that involve qualitative data.

Michael: I think that’s really insightful. The gap you’re mentioning—between what computers are really good at and what people really have to deal with in the real world—is very interesting. I think prior to Gen AI becoming something people are aware of, those two camps really didn’t speak to each other very clearly.

The folks who understood the complexity of the real world said: well, it just can’t be done; we can’t put it in a computer; we can’t have this conversation. Whereas computer scientists are trained from almost day one to create an abstraction and then really embrace that abstraction.

When you see people in the news, public figures saying “oh, this is just an optimization problem”—first of all, they’re wrong; second of all, that’s because of their training. They’re actually taught to think that way. The boundaries between the real world and the abstraction start to vanish for them. They see the abstraction and they think: if I solve this abstraction, I’ve solved the real world. And that’s also wrong. The notion that “nothing the computer can do can help” is wrong. But the notion that “the thing I’m solving in the computer is the real world” is also wrong. In this Gen AI moment, those camps still exist and they’re still not talking to each other—but they’re closer together than they’ve ever been. We can have messy machine learning and we can have a digital real world to some degree.

James: Gen AI has challenged my computing model. Thirty years ago a colleague who knew nothing about technology asked me: what can you do with a computer versus not do with a computer? This guy loved yellow legal pads; he would work out every problem on a yellow legal pad. I said: computers are really, really fast yellow legal pads. Anything you can do on a yellow legal pad, you can do on a computer more quickly. Anything you cannot do on a yellow legal pad you will never do on a computer. I’m not so sure that’s as relevant in 2026.

Michael: That’s really interesting. But you could draw a portrait of someone on a yellow legal pad. Was that part of the way you thought about it?

James: That’s fair. You may be testing the assumptions of my model. This person was not inclined to draw on yellow legal pads. He was inclined to sketch flow charts and rough out financial models.

Michael: The yellow legal pad was that person’s abstraction. That’s fascinating. It’s very true to some degree—or maybe to a lesser degree, but not to no degree. I do think people are continuing to forget that. When I hear some tech leaders saying “oh, this is going to replace everybody’s jobs,” I think: you don’t understand jobs. There’s so much more to it than the abstraction you’re thinking of. Oh, a job is just: you take this input, you produce this output; I can train a computer to do that.

James: I would argue there are many jobs like that. Large parts of the white-collar workforce can be thought of as engaging in copy-paste operations—moving data from email to spreadsheet to word processor. We’d like to think we can reduce that toil and allow those individuals to do more interesting and rewarding things than copy-paste.

Michael: Yeah, though sometimes I also think part of the job is to go to meetings and hear how things are changing. People adapt the way they’re doing this over time because they understand at least a piece of the context—the little sphere they live in, or the little sphere we all live in. We all live in our own little spheres, but those spheres are way bigger than the yellow legal pad. There’s all this interpersonal stuff and an understanding of the organization as a whole that I think you lose if you just abstract it to: oh, it’s just an input–output relationship.

James: I’m talking about part of their job. Someone may go to the meeting, take something away from it, and then say: okay, now I need to do 14 hours of copy-paste operations and go back to the next meeting.

Michael: That’s fair.

James: It’s nice to say: okay, your job is now going to be to interact with other humans, and a machine will do the copy-paste operations, hand you back the result, and you can go interact with more humans. That feels like a more rewarding job than 14 hours of copy-paste operations.

Michael: Another thing computer scientists are trained to do is to hate copy-pasting. Anything we do that feels mechanical—we’re taught: automate that, automate that. So you’ll find some computer scientists spending way more time than necessary solving a problem because they just refuse to do the mundane version. They develop a whole software system that’s completely unnecessary to solve something they could have just cut and pasted.

James Kaplan: Franklin Covey taught us: always take five hours to save five minutes.

Michael: There you go.

James: I’ve gone through my own experiments with Cursor and Claude Code—a lot of taking five hours to save five minutes. I spent probably two hours getting Cursor to write an email for me using Gmail for a non-work thing. I had to download and install the new Gmail command-line interface, authenticate it, figure out how to use it. That went a lot more quickly because I didn’t have to figure out the syntax myself. But at the end of it I had spent two hours to send an email that I probably could have drafted in 15 minutes—because I needed to assemble data from various places. But now I can send that email much more quickly. I’ve achieved operating leverage. I can send the next email much more quickly too.

Michael: Right. That makes a ton of sense.

James: This is incredibly interesting for universities. I think there is both a real risk of universities being disintermediated along multiple dimensions and a tremendous opportunity. A lot will depend on how they choose to act over the next decade. I was wondering if you could reflect on both the opportunities and the risks.

Michael: That’s a great question. There are two things I spend a lot of time thinking about—how universities are kind of special in this moment. One is the notion that AI is this force that’s been splashed down upon us, and we’re trying to figure out how to make use of it in a way that’s productive and consistent with missions that civil society can make positive use of. I think it’s hard. I think it involves a lot of change on both sides: the technology needs to change to some degree, and the structures of society need to change a little bit as well. I don’t think companies are necessarily in the right stance to work on that.

James: Companies have all sorts of problems figuring out AI.

Michael: Fair enough. It’s hard. I think we all need help figuring this out. But what I’m getting at is: on campus we have people who have regular civil-society problems they’re trying to solve. A bookstore; we have a sanitation department, a security force. We have the things you’d have if you were running a city. But we also have AI experts. We have sociologists. We have humanities people who’ve thought about the sweep of history and how human beings adapt to certain kinds of change. We can actually construct this future and live in it and be a model for the rest of civil society if we do this well. So I feel we have both an opportunity and a responsibility to try to do that.

The other side of it is that these chatbots are essentially homework machines. They take as input homework questions and they produce answers. The API that’s been established with these chatbots perfectly subverts what we’ve been doing for decades in terms of how we run our educational process. That’s extremely disruptive. We can’t just ask the same kind of essay questions we used to ask. We can’t just give the same kind of problem sets, because the temptation—almost the demand—is for students to feed it to a chatbot and write down the answer. So we’re in a situation where we’re being completely subverted. Even if we’re not disintermediated—even if others don’t step in to do what we do—the way we do what we do has to change. That’s extremely unsettling for me and my colleagues.

James: I think that’s an easier problem than everybody says. Here’s why. Your professor who teaches—what, grades 13 through 17?

Michael: Roughly. I actually have PhD students, so it goes up a little further.

James: Okay, 17 plus. I teach grades 18 through 30, in a sense. When they’re done with you, some of them come to me. In a professional-services world we give different homework assignments—assignments that face the brutal grading of the market. I have not seen any circumstance where raw LLM output is good enough. There’s part of me that thinks college professors are saying: hey, folks, what used to be an A++ is now a C−. That is what it is. Because as you move into law, business, finance, government—testifying before Congress, writing for a publication—it will neither be acceptable to avoid AI tools nor acceptable to just hand in raw output from a large language model. Does that make sense?

Michael: Yes, I think that’s exactly right. There are two aspects that make this hard for us, but I think that is exactly the path we’re going down. What we think of as education is changing to reflect this reality: we have this entity we can delegate some of the details to, but we’re responsible at the end of the day for the final product. That’s a shift. We spend a lot of time teaching people about implementation, and now we have to spend a lot more time teaching people about quality control.

James: You teach not only application but underlying theory, where there’s a little more of a textbook solution potentially. You may need to collapse the teaching of theory and application—you learn the theory by applying it in real time, which is what you do in the professional world. We know the textbook solution you can get from the machine. But the application of that solution to a particular context you’re much less likely to get from the machine. So that’s what we’re going to challenge you to do from the day you step onto campus.

Michael: That leads perfectly into the second point I wanted to make. We do have to change how we’re teaching and what we’re teaching. But the thing we’re scared about is we don’t really understand the role of asking these kinds of questions in people’s cognitive development. When you see people at 18 to 30—grades 18 to 30—the hope is they’ve got a really solid cognitive base on which all this other stuff is built. Building that base is an art. Good educators somehow manage to unlock this in people and help them get the right concepts and the right motivation to work together in a beautiful way. If we were to shift everything tomorrow to the model you’re describing—theory and practice really come together, they’re only useful when they’re supporting each other, so we should just do it that way—I don’t think we know what’s going to happen. I think that’s a giant experiment.

James: If anyone knows what’s going to happen, please let me know. I’d be curious to hear about it.

Michael: It’s interesting because it is a bit of a black art. I think there’s a lot we do know, but the questions haven’t really been asked quite in this context before. There’s a lot of work we have to do to feel comfortable. For example, last night the director of undergraduate studies in my department sent an email to me and the chair saying: hey, we’re thinking of basically replacing all our intro computer science stuff with Claude Code. I was like, wait, wait. It turned out that’s not what she actually meant—but making sure the Claude Code stuff is integrated into what people are doing. They’re using it anyway; we might as well make sure they’re using it intelligently and not just foolishly.

James: The principles of computer science become more important as you use what I would call tools for spec-driven development. I would suggest you could write pretty okay code for certain things without understanding much computer science. The chance you’re going to be able to orchestrate a dozen agents writing code without understanding the constructs of computer science, to me, feels minimal.

Michael: I think that’s exactly right. It’s easy for people to fall into the trap of: oh my gosh, you produced a product with so little effort, this is great; a little more effort and it’ll be a great product. It’s like: no, a lot more effort. It’s going to be a great product, and knowing how to apply that effort is challenging.

James: I would argue the early returns I’m seeing from multiple places is that the cognitive load for agentic development is higher, not lower. We’ve all written code—me probably more in the past than recently—but there’s some code that’s just mindless, really copy-paste and what have you. What we’re hearing from some engineers is the agents are getting rid of all the mindless stuff, and you’re like a bond trader—

Michael: Right.

James: —orchestrating these agents, trying to figure out what they’re doing and trying to keep them productive. It takes a tremendous amount of attention to figure out what these agents are doing and where they might be getting off track.

Michael: I think that’s exactly right. One of the things I’ve heard that supports the argument we’re making is that now, if you get interrupted in the middle of working, it’s so much more painful than it was before. Interrupting is always hard, but now once you’ve got three screens open and you’re orchestrating all these agents—they’re producing different parts of it—and then someone comes in and says “I just have a quick question,” you’ve lost so much context.

James: You’ve been thrown out of flow state is the way I would describe it. Question for you. You were talking about how we don’t really understand how people learn things—which I think is very much a true statement. Are you familiar with the idea of the zone of proximal development?

Michael: Yeah, very much so.

James: I think AI will do a much better job of landing instruction in the zone of proximal development, because it can be tailored.

Michael: That’s exactly right. To the extent these systems can get a feel for where your thinking is, they could potentially design instructional material or explanations that just stretch you a little bit. For people who don’t know about the ZPD—that’s the way the concept is used today, though I don’t think it’s what the original author meant; I’ve read a paper about that specifically—the idea is you want to always be teaching just a little bit outside of what you’ve mastered so far. That’s what keeps things moving forward. It’s hard in a 300-person classroom: everybody’s in a slightly different place. As a lecturer what I’m trying to do is go a little bit beyond the median, try to figure out where the lump is. If I go too far ahead I’ve lost the stragglers; if I go too slowly I’ve bored the advanced people. What AI could do potentially is basically provide a personalized lecturer for each person, which could be amazing.

James: So let me ask maybe a provocative question. Why in God’s name do universities still have lectures? You can watch them on YouTube; they can record your lectures and put them on YouTube. Why still do that?

Michael: It’s a really interesting question. Let me take you back—maybe ten years. In this office where I’m talking to you from right now, I had this giant desk set up with all this recording equipment because MOOCs were going to be the next thing.

James: MOOCs failed. I get it.

Michael: MOOCs failed—well, they did and they didn’t. A massive open online course: the idea is we don’t need a million dynamic, exciting lecturers. We can have rock stars—the same way we don’t need a million people playing bad guitar. We have the ability to take a couple exceptionally talented people and have everybody listen to them. It scales. I like to say the history of EdTech is littered with the bodies of really good ideas, because every time, what seems to be the case is that human beings learn best when they’re motivated by other human beings. It’s not—as much as we want to abstract it away and say it’s basically information transfer, there are all these facts in the world and we have to get them into the person’s head—that’s not a natural thing for people. The only way we know to get people to swallow all that information is to be inspired by somebody.

One of the models people are talking about these days—I think it’s called Alpha School or something like that—is that each student spends some amount of time interacting with personalized AI fact-givers, and then they spend the rest of the day working with peers and mentors and talking to human beings. The balance is crazy: something like 20 minutes or two hours—a shockingly small percentage of the day—getting new material through the computer, and a very large percentage of the day interacting with other people. Is that the perfect blend? I don’t know, but they’re showing some really tantalizing early results. So why lectures? I think lectures are to get in people’s faces—to show a live human being who could actually care about them, presenting the information they need to know.

James: I wasn’t advocating for books—I agree with you. Someone passively sitting back watching a video is not helpful. If I were teaching a class, part of me would say: okay, I would tell the students all the lectures are in this video library, go watch them. I’m going to assign you the lecture the way I would assign a book; make sure to watch it. Then I would say: instead of having lectures, I’m going to break the class up into seminar sessions. Each session will be led by a TA, but I as the professor will rotate through. Or maybe I’ll do tutorials with a smaller number of students, depending on the size of the class. That sounds a little bit like what you’re describing. Because to me a lecture is not getting in someone’s face—you have a lot of students sleeping in the back of the class. I often was the student sleeping.

Michael: I think that’s right. We’re going to continue to experiment with different kinds of models. The agentic programming class we’re teaching this semester we’re calling Agentic Studio, because the structure is modeled after studio classes in art school. We’re teaching computer science the way artists are taught how to create—partly because we’re teaching people how to create and they need that constant feedback from people who have more experience. Lectures are not that. Lectures are kind of a cheap substitute. But this kind of detailed mentoring doesn’t scale very well. If you want face time with a world expert in some topic—French literature or the impacts of city-building on the environment—lectures are the only thing we’ve come up with that gives people a chance to be in the presence of the people having those thoughts and ultimately feel connected to them. I agree with you though. I don’t think it’s perfect. I just know there are plenty of things that sound plausible that actually fail.

James: The one sort of definitional learning experience for me at Brown happened outside the classroom when I ran the Brown Daily Herald. It was like: hey, kids, go figure out how to run a newspaper. The seniors who had just graduated trained you a little bit—though they only knew so much. We had an alumni board of directors who worked for places like the New York Times and the Washington Post, and every so often they’d come in and tell you how stupid you were. They were nicer about it than that, but they were telling us all the things we’d screwed up. It was: go figure it out. Figure out how to structure a staff, how to assign a story and what have you. I wonder if there will need to be even more of that in education—which I think is what you’re saying about the studios going forward.

Michael: I think that makes a ton of sense. I got a question like that this past week in New York City, talking to some alums and business leaders about AI. They said: shouldn’t it be the case that students are given a chance to do projects? That’s what we do at the university. I don’t know what you think we do, but extracurriculars are extracurricular—they’re not part of the curriculum, but they’re absolutely part of the college experience. The people who get the most out of college are the people who really avail themselves of those opportunities. It’s not like we’re not doing it or not offering students those chances.

James: The thing that made the Herald—or other extracurriculars like WBRU—so vivid was that if we screwed up, we heard about it the next morning. Oh boy, did we hear about it. Direct feedback loop. It was a rapid-fire teacher.

Michael: I think that’s a terrific example. My daughter also went to Brown, and her experience like that was directing a musical. There the feedback is partly slower because you do months and months of work and then there are the performances—you don’t really get to see how things land until then. But you’re getting constant feedback from the people you’re working with, because if they’re unhappy you can tell right away.

James: One of the best classes I took at Brown was educational software with Andy van Dam and Ted Sizer—two legends teaching one class. Every group of students had to build a piece of educational software and let students use it; we worked with high school students. It was both terrifying and rewarding to watch high school students use the thing you built.

Michael: Right. You discover that your wonderful pet idea—nobody cares about it—and there’s some stupid throwaway thing that actually draws people in and gets them engaged. There’s no substitute for that. I’ve spent some time trying to become a standup comedian, and there’s this notion that you have to try the joke out. It can be the funniest thing in the world in your head, but it has to land for the audience. Making that part of your process—teachers and instructors do this all the time. Good ones, anyway. There are some who are just pontificating into the void. But the rest of us really do feel the students’ reactions and we’re trying to figure out how to get through and get that positive feedback.

James: And here I was thinking I was the only person trying to be funny about AI.

Michael: I did one routine once that had AI in it. Mostly I talked about being an academic or being a dad. This was pre-pandemic, so AI was not on people’s radar. When I talked about AI, people didn’t think it was funny at all.

James: AI may or may not be funny. Corporate America is endlessly funny.

Let me switch topics to something I’m especially passionate about: digital humanities. I’m a technologist but was an undergraduate history major, and it occurs to me that we can interrogate data in the humanities in a way that would have been unimaginable even two years ago. For example, I took Professor Litchfield’s class on the industrial revolution in early modern England. He talked about how in the sixties people did history from the bottom up—grad students tramping around in dusty archives to record birth and death information. If that’s digitized now, we can interrogate and analyze it at scale. To me that’s an astonishing leap in human understanding of history or literature.

Michael: I think that’s right. History, probably not surprisingly, is going to be a little slow to respond because they have all this context about how things are done and how they used to be done. But I had a meeting last week with the chair of a committee in the history department—what role can AI and information technology play in supporting the discipline and helping people do better history work? He was really excited about it, quite passionate. The image I had from our emails was that he’d be someone freshly graduated, the kind of person asking how to put an interest in video games together with an interest in history. But no—he’s a legit, classic historian. I said to him: what are some of the barriers to switching to this model? He said: I don’t think there are a lot, but when I’m in the library walking through the stacks, alone for six hours with these books—that was the person I was born to be. So the act of doing history, getting into those materials—for some of them that’s what drew them to the field. But even he is recognizing we can do things better and differently. It might not be that every single historian will be digitally enhanced, but the field is definitely taking this on board and trying to find ways to make use of it.

James: How optimistic are you about the willingness of the social sciences and the humanities to embrace this? I’ve spoken to academics at multiple institutions who’ve heard the kind of dialogue we’re having and said: that’s thinking like a scientist—prioritizing quantification over other forms of knowing. That’s not the humanities.

Michael: Right. I think that’s a reaction people have for sure. But it’s a pretty diverse community. Some people will have that reaction; some will have others. Between us, at the end of the day, I feel the most likely outcome is they’ll all be using these tools to great effect and won’t understand why we thought they had a problem with it early on—they’ll say: we were always doing this; this is exactly what we’ve always been doing; I don’t understand what you’re talking about. But it’s going to take a little while for them to wrap their heads around how the use of these tools doesn’t have to subvert what they see as the core intellectual contribution of their work. It can enhance it. It doesn’t have to undermine it.

James: I don’t know whether cynical is the right word, but in a corporate environment, once you can interrogate records from sales calls, it changes the channel a lot because you’re being a lot more systematic about what works and what doesn’t. You can interrogate electronic health records to look at the efficacy of treatment protocols—you’ve changed what being a doctor means quite a lot. I would hypothesize that intellectual history is probably overweighted in the output of departments because it’s logistically easy; you’re not trying to get at data sets that don’t exist. You could see some pretty radical shifts, or some pretty tough fights about what history is—depending on whether we study this set of topics because the data has historically been available, or this other set where we have newly available data.

Michael: The picture you’re painting is really compelling. It could be one of these—what do they call them—Kuhn revolutions. It could change the paradigm: the way they ask questions, what they consider a valid result, what understanding even means in that discipline. It could actually change the discipline. It happened in the sciences—chemistry was a different kind of creature before. The humanities may be a little late to the party, but they may ultimately be impacted in very similar ways. It’s super interesting. That won’t go down easy—people will fight it kicking and screaming—but ultimately it’s them as a community who have to work through it. The hope is there’s at least some objective measure by which the field can decide: this is actually better. I don’t know exactly what the metric is, but if we’re actually doing better history than we were, we should do that. I think that’s what the fight is ultimately going to be about: what is our measure of quality, and are we improving in that dimension? Otherwise it’ll just branch off as a different field. There could be a different field with different objectives. But if it’s going to stay one field, that whole field is going to have to feel its way to a new way of thinking.

James: We’re going to have to do a follow-up—we have about three minutes left and there are whole questions of epistemology, institutional imperatives, the availability of new forms of data. What do knowledge graphs mean for the capture and management of data? So let’s declare this the end of part one and turn it back to you. What would you say to a young computer science graduate thinking about their career over the next 10 years?

Michael: Oh wow. One of the things we’re starting to speculate about is that what it means to be a successful computer scientist may actually change. It could be that people who are not well suited to this discipline are actually who we need right now, and the people who’ve been traditionally good at it—maybe that skill just isn’t needed anymore. That’s a hard thing for someone like me to think, who’s dedicated his entire life to this one subject area. Maybe it’s not about me anymore. Maybe my successors are going to be very different from me.

James: Maybe we even think about the idea of a discipline very differently.

Michael: Yeah. In many ways I feel Brown is an interesting place to be having those discussions, because we’re a little less bound by disciplinary boundaries than a lot of places. Those boundaries exist and help us organize our thinking, but we’ll walk over them if necessary. There’s a wonderful course being taught right now on the history of artificial intelligence—the historian knows a ton about the technology and what it’s done and what it’s meant. That’s a high-level thought from my perspective. What would I say to a young graduate? I spend more time talking to my fellow faculty. What I try to encourage people to do is embrace this moment. It’s forcing us to revisit some very long-held and unquestioned assumptions. It’s painful, but we should do it. We get to be the people at the revolutionary boundary. Play your role. Do your thing. Help us figure out where we’re headed. It’s exciting, but it’s a big lift—a cognitive load for all of us. Some of us just want to keep doing the thing we’ve been doing for 30 years, and it’s just not okay anymore. Let’s be excited by that instead of depressed by it.

James: You could argue the biggest distinction among people in probably every field—academia, business, or anything else—will be between those who are intrigued by revisiting old assumptions and those who are scared or resistant to it. This was a blast. Thank you so much for doing this. We’re going to have to do a follow-up because there’s a whole bunch we didn’t get to. I hope you enjoyed this as much as I have.

Michael: I always have a great time speaking with you, and this forum is a great way to do it. Thank you so much for thinking of me. Thanks.