Every December the Brown Daily Herald used to hold a dinner at the old Omni Biltmore Hotel in Providence to celebrate the incoming board and say goodbye to the old one. It is a melancholy event for the outgoing editor-in-chief. At least it was for me.
I had outlined plans and aspirations at the previous year’s dinner. When I came to the podium at the end of 1991, I thought about everything the 126th board hadn’t accomplished — and all the political capital expended for mixed returns. Still: we published every day of the academic year despite two blackouts. We didn’t get sued. None of the corrections we had to run were catastrophic. We published at least a few good stories.
My successor called me to the stage. I received respectful applause — more due to the position than the man. I thanked each member of the board. The room exploded when I reached Adam’s name. The staff understood exactly the energy and creativity with which he had attacked the role of Opinions Editor. Despite our differences of, well, opinion, I remain grateful to Adam for his contributions to the paper, even decades later.
Adam and I ran into each other again at McKinsey. By then he had decided against litigation as a vocation and was already developing into one of the most rigorous thinkers I know on data and analytics. Senior roles at JPMorgan Chase, DirecTV, Zurich Insurance, and Point72 followed — teams as large as 170 people, enterprises with up to 60 million customer relationships. He now advises boards and senior leadership teams, teaches in Brown’s EMBA program, and runs an annual forecasting contest, which I fear to enter.
Everything we do — every business decision we make — implicitly rests on a forecast. Which is why I needed to get Adam’s view on what AI means for data, analytics, and the practice of forecasting itself.
Three things struck me as particularly worth your time.
First, the B2C-vs.-B2B gap in analytics is real — Adam puts it at twenty years — but the cause is not what most people assume. It is not that B2C data is cleaner. It is that B2C businesses have enough observations to find statistical regularities. A million customers generates surprises. Seventy-five institutional accounts does not. AI changes some of this at the margin, but domain expertise — knowing which variables to featurize, which hypotheses to form — remains the scarce resource.
Second, taking payoff curves into account as you thinking about your career and your job. Adam’s formative experience was as a litigator: limited upside, catastrophic downside, a profession structured for you to lose. He now teaches his kids to think about every career choice as a payoff curve before accepting it. Most technology managers are running the same analysis implicitly every time they commit to a timeline or decide whether to invest in redundancy. Making the curve explicit demystifies the decision.
Third, Adam’s argues that the manager who has actually formed a probability — even a rough one — about whether a project will come in on time is making a different kind of bet than the manager who just called it a plan. The first manager will be better calibrated over a portfolio of decisions. “It matters to be 53% right instead of 52% right” is not a hedge fund insight. It is a description of how operational advantage compounds.
Introduction: from College Hill, back to College Hill
James Kaplan:** Welcome to another Prosaic Times video podcast. We have with us Adam Braff, who I’ve known since about 1990. I’ll let Adam introduce himself and talk a little bit about his journey, and then we’ll start to dive deep into forecasting and what AI means for forecasting.
Adam Braff: Thanks for having me, James. I guess if I could describe what I do now, it is not strictly about forecasting. It’s helping corporates and investors figure out what to do with data and analytics, and increasingly AI. So some of that is descriptive analytics and trying to figure out how many widgets are we selling and why, and some of it is predictive, and how many widgets are we gonna sell next year and how do we make that number go up?
So predictive does come into the work that I do
James: Yes, and we all want to hear about the Narcissist forecasting contest, which I am proud never to have taken part in — because I’m sure I would lose badly.
Adam: Nobody loses in the contest. You gain knowledge, you get smarter. And even if you’re gunning for first place, which is fair to do, it actually influences your strategy. It’s like when you’re entering a March Madness bracket, do you pick a lot of weird long shots, or do you try to be as accurate as possible?
I try to persuade people to just get more accurate year over year and have the least error overall in their forecasting. So nobody loses in this contest. They get a score, they can get coaching and feedback from me for free, and then they try to get better every year
James: Okay. So tell us a little bit about your journey
Adam: We graduated into a recession, if you may recall.
James: Was not fun
Adam: That was the reason why my wife and I — then girlfriend — decided to go to law school. It was a good way to hide out from the economic storms of the early ’90s, and that was kind of a mistake, I would say. It was a big detour in my career, but I ended up going to law school and then practicing as a litigator at a big firm in DC for three years, really not enjoying any of it, and trying to find a way out.
And as it happened, I had put myself into a bit of a corner because the world doesn’t really need litigators. There aren’t a lot of obvious ways for litigators to turn into some kind of beautiful butterfly. So I took advantage of the fact that it was the dot-com boom, the first dot-com boom happening at the end of the ’90s, and ended up getting a job at McKinsey, and it was a fresh start for me.
I was 29 years old, and I just said, “Let’s, let’s start again at the bottom, and this time pretend I had gone to business school instead of law school,” I got my mini MBA from McKinsey, which consists of three weeks of getting bombarded with strategy ops, marketing, and org stuff, and then ended up staying at the firm for a lot longer than I thought I would be there.
I ended up doing work that was really what we would now call data analytics AI. At the time, I think we called it big data. It was how do companies use larger data sets than would fit in Excel and creative new kinds of data and A/B testing on their customers and calculations of customer lifetime value?
How do consumer businesses figure out how to do more of that stuff in order to attract and retain the most valuable customers? And that’s something you can do in a lot of different industries, as it turns out
James: Did you go to Mini MBA in Kitzbühel or do you go someplace else?
Adam: I went to Kellogg. I had a bunch of trainings in Kitzbühel, but I was in Kellogg freezing my ass off for three and a half weeks in January of that year
James: I’m sure it was warmer than it was in Kitzbühel and easier to get to
Adam: Exactly. It was nice.
So I stayed at the firm for 10 years. I was a generalist partner in the DC office. None of my clients were really government or particularly local to DC, so I was traveling a lot. Had three kids in that span of time, and at some point I said, “All right. It’s time for me to get a real job and figure out what I want to do in the real world.”
So I left in 2009 and had a few different offers — some in New York, one in Vegas, which my wife vetoed — and ended up going to JPMorgan Chase to become their head of customer data and analytics. That was for the Chase consumer businesses: figuring out what to do with information about customers who might have a Freedom credit card and a Chase mortgage and a Chase checking account.
What do we do to put all that information together and help the bank make better risk decisions, better anti-fraud decisions, better marketing decisions? I did that for a few years and then got recruited over to DirecTV to run business analytics there. That was different because there was an existing team of really good analytics people doing the same work but for pay television — a much simpler business, with some interesting lifetime value drivers.
James: Interesting. And then you did an insurance company for a while, and you’ve also recently been teaching at Brown.
Adam: DirecTV got bought by AT&T, a bunch of us bailed out. I ended up going to Zurich Insurance and ran data and analytics there. I got a taste of the IT world by reporting into the chief operations and technology officer — which let me see what it’s like to actually build the stuff and not just give the requirements.
Then I spent three years at Steve Cohen’s hedge fund as head of data sourcing and strategy — data acquisition, essentially. I’ve been independent since 2019. Hung out a shingle that year and started teaching at the same time.
I began at NYU, created a class called Business Analytics and Data Visualization, taught different variations of it for a while. Then we moved to Providence, where I’m sitting now, and I’ve been teaching versions of that class at Brown.
Is B2C 20 years ahead of B2B in customer analytics?
B2C is roughly 20 years ahead of B2B in customer analytics — not because the data is cleaner, but because there are simply more data points to work with.
The analytics opportunity in B2B looks fundamentally different: customer targeting and Salesforce effectiveness rather than churn modeling and cross-sell.
AI and agentic data ingestion could narrow the gap — converting unstructured B2B data into analyzable signals.
Domain expertise still matters: knowing how to featurize variables (competitor footprint, not just state) is what separates the old hands from the clever prompters.
James: Let me ask this question. You’ve done a ton of stuff in what I’d describe as the B2C world, and it’s been my observation that B2C is light years ahead of B2B on analytics — just because the data is cleaner. And I hypothesize one of the interesting things about gen AI is you can now clean up the data, correlate it, convert unstructured text into structured data you can analyze.
So I was wondering if you could comment on how far ahead the B2C world is compared to B2B on analytics, and to what extent might gen AI or agentic data ingestion and cleansing help the B2B world catch up?
Adam: It’s like 20 years ahead[1], but probably for a different reason than the reason that you set out. It’s more that there are more data points to measure in the B2C world. And so if you’re gonna list all the use cases, if a private equity firm comes in and buys a consumer business or buys an industrial business, all the things you’re gonna go work on, there are gonna be analytical big data things to do over here that have to do with acquiring more customers and retaining your best customers and figuring out who those best customers are and cross-selling and stuff.
Over here, you’re like optimizing a factory or something, right? So to the extent that there’s data stuff, and very little of my career has been over here, right?
James: Right
Adam: You’re gonna be getting information from things — what was briefly called the Internet of Things — and getting instrumentation data from over there.
So big data definitely exists, and there are use cases for it, and I sit on councils with people who do this stuff. It just looks very different from the kinds of big data and analytical stuff that I did, and that my father did before me. He’s retired now, but he was doing data-based marketing back in the ‘70s and ‘80s.
Second-generation data and analytics — something that hadn’t existed before.
James: I was thinking about this in the context of customer analytics. If you have a million households in a B2C business versus seventy-five or a couple hundred corporate institutions — which have complicated internal dynamics — the N question becomes obvious. You don’t typically have many decision-makers when someone decides to buy cable services. Maybe one, maybe two. Whereas in B2B procurement, there could be twenty decision-makers in the process, and they all interact with each other.
Adam: The analysis you would do in B2B Salesforce effectiveness would have a lot more to do with customer targeting — going out and grabbing more information about your target, using some agentic tool to pull that and put it somewhere you can analyze it and make good decisions.
The reason the million-customer B2C business got ahead is that a lot of analytics is looking for patterns, for statistical regularities. If you can determine that customers in this region are stickier, or that customers who came through this channel spend more, there are just many more ways to do those analyses — with genuine comparative advantage over an executive doing everything by intuition.
There’s at least a fighting chance for the data to tell you a different story. You think the higher-priced product is better, but the customers don’t stick around as long once you control for everything — and therefore you shouldn’t be charging more or assume that’s the right answer.
You’ll come up with intuitions like that on the B2C side. You won’t on the B2B side because you don’t have enough N to surprise anyone with that kind of finding.
James: Tell me if this is right. The more independent variables you’d have to take into account, the larger the N you need in order to get a relevant statistical sample.
Adam: I think that’s right. And I don’t want to make too much of statistical significance here. A lot of what you’re doing in B2C consumer analytics is descriptive analytics — a BI dashboard, not a giant regression model with a lot of independent variables. You want to slice your data one or two dimensions at a time and just look at the graph. Which numbers are higher? Is it this product, this region, this group of call center agents? That’s what lets you drill in another level and understand what’s happening.
You can have lots of dimensions and measures that you think of as independent variables — you’re just going to be doing them a little at a time, flexibly drilling in to find the hotspots. The regression is just one part of that.
James: What you’re saying is there’s a lot of power in just being able to do Pareto analysis — being able to look at your churn by segment, by size of spend, by region, by product mix.
Adam: Yes, exactly.
Adam: Right, exactly. So we were at DirecTV, churn was the number one thing. It was the era of cord cutting and cord nevers. probably a lot of your viewers have never thought to buy a monthly pay TV subscription ’cause it’s all, they may get it through a streaming product, but not through, a cable
James: every now and again, I will try to explain the concept of channels to my kids, and they’re just confused by it.
Adam: Yes
James: I try to explain that in New York in the 1970s you had channel 2, 4, 5, 7, 9, 11, and 13, and that was it. They accuse me of being old.
Adam: The reason I bring up DirecTV is not to make you feel old. It’s that the main problem we were trying to solve was why churn was so bad and how to make it better. And if you think about the ways to slice churn, first you want the churn rate — within any given customer population, what percentage are canceling over a month or a year?
When you start slicing it, the intuitions you might have — is it by demographics, age and gender? — that’s not what matters. What matters is things like which customers are rolling off a $5-a-month promotion this month. Those turn out to be the vulnerable ones.
Or which customers are in the Fios footprint or the Comcast footprint — does that make a difference, and is it trending differently over time? You need the data to do that kind of work.
James: So what I hear you saying is: the availability of the data, obviously, but also the person who can develop a hypothesis about what might be a relevant driver — competitor footprint, for example.
We needed that hypothesis generation in a world where analytics were expensive — someone had to go do the cuts in a spreadsheet or via SQL. In a world where analytics are cheaper, how much of it is “let’s just run all three hundred cuts and see what it tells you,” versus using a model to discern which cuts might actually have real differences?
Because I think both you and I have done an analytical cut a million times, said “region has no impact,” “size of spend has no impact,” and then — “overlap with competitive footprint, that has the impact.” You have to do nineteen analyses before you land on the twentieth that’s meaningful.
Adam: Domain expertise and taste matter because you have to figure out how to featurize these variables. Region by itself, cut generically into US states or metropolitan areas, gives you a boring answer.
But if you join that data up with another data set — this is the Comcast footprint, this is the AT&T pre-merger footprint — suddenly you’re testing something real. Knowing how to featurize the data to test hypotheses efficiently is still very valuable.
And AI tools, if you coach them correctly and bring enough taste and judgment to it, can let a sufficiently senior person test a much larger number of hypotheses. They’ll also surface a lot of spurious answers along the way.
A brand new data science graduate would have a hard time hitting the ground running in one of those businesses. Even a very clever prompter isn’t going to get to the right answer the way an old hand would.
James: Creativity and context matter here. You make the point about competitor footprint. Who knows — state may be a weak signal, but population density may be a much stronger one. You get a weak signal from New Mexico and Wyoming being different from New Jersey and Rhode Island, but when you really dig in, it’s the Providence metro versus less densely populated parts of Rhode Island that’s driving it.
Adam: And you’re keeping track of the time dimension. Things change over time. Everything changed in COVID. You might decide to cut off your analysis at the post-COVID era, or account for two competitors merging. This happened constantly with banking analyses.
One of the analyses I used to do at the firm was on customer experience. We ran a big annual panel surveying people about their banks — and it was always difficult because the banks kept changing. The American Customer Satisfaction Index, which tracks these things longitudinally, has the names of ancient banks nobody’s heard of — Chemical Bank — that we now know as Chase.
There is genuine value in domain expertise that AI cannot yet replicate.
James: What I should do one of these days is a quiz — Chemical Bank, Manufacturers Hanover, First USA — and you have to guess which bank they’re part of now.
Adam: The world’s most exciting quiz. I agree.
The debate on virtual panels
Synthetic data is valuable for prototyping and pedagogy — stuffing a dashboard mockup with plausible fake data is now standard practice.
Where it breaks down: using synthetic panels as a substitute for real market research, especially for purchase-intent questions. There is no signal like that in the model.
The motte-and-bailey risk: vendors pitch synthetic panels as research-grade but deliver something closer to a prototype illustration.
James’s CIO/CTO panel is more useful for testing message clarity and resonance than for predicting buying behavior — a meaningful distinction.
James: Never said I was cool. What do you think about virtual panels? Particularly in the B2B space, I’ve started playing with using agents to simulate the decision-making of certain people. Valid? Less than valid? Intriguing?
Adam: I remain skeptical of the whole field of synthetic creation. On the one hand, I use it for pedagogical purposes where I’m creating a data set to illustrate a point to my students. It’s great for that, especially if you’re gonna sprinkle in personally identifiable information, which you don’t wanna do with actual humans.
I use it to make, prototype dashboards. This is something that has become very common now, not just me, but many people since the Claude Code stuff really got going in earnest back in January. I’m sure you’ve done this and you’ve seen it, but many people are mocking up prototypes very quickly, and stuffing them with synthetic data is a pretty good idea, right?
To give people a sense of the art of the possible there.
James: A couple of weeks ago I created the data for a change management program at a major financial institution — all dummy data, but realistic enough to demonstrate something. That would have taken weeks to do manually.
Adam: Incorporating randomness and signal to make it feel like real data is genuinely hard. Agreed.
Now, on panels: I’m doing a lot of consumer insights work these days, back on the B2C side. You want to survey 500 Americans about something — and it’s getting harder and harder to find real people who aren’t robots or speeders or cheaters. So people are pitching synthetic panels: 500 simulated respondents. Why not?
Here’s why not. The whole point of a real panel is the signal — what actual people in the world want, the clusters that form, the principal components that emerge. It’s not just big spenders versus quick deciders; there’s a third dimension that actually explains the variance.
Trying to derive all that from synthetic data seems crazy to me. I’ve had a couple of conversations with these vendors, and I worry that what they’re pitching sits somewhere between the prototype mockup work we were just celebrating and the actual market research my clients need to make real decisions.
It’s a motte-and-bailey: they pitch it as research-grade but it’s really only as good as a prototype illustration. I’d love someone to steel-man what synthetic data is genuinely good for beyond that.
James: For the blog, I created a panel of five hundred CIOs and CTOs. I gave each member of the panel a persona — this person is the CIO of a pharma company with five hundred million dollars in IT spend, this person is the CIO of an asset manager with a billion dollars in IT spend. Then I used the OCEAN framework to create psychological profiles.
Adam: Mm-hmm.
James: Each individual has a job, a title, a company, and a personality profile. I run each draft through them. I wouldn’t bet the farm on the scores, but the quotes I get back are really useful, and I’ll go through about seven iterations. Part of me thinks it may be less helpful for predicting buying intent and more helpful for testing marketing messages — because that’s where large language models are genuinely good.
It’s hard for them to predict whether somebody will buy, but they may be better at understanding whether a message is more or less understandable, more or less compelling. Does that make sense?[5]
Adam: A little bit. You used the words “testing marketing messages.” I think it’s more like — in this case you’re testing your blog. Generating responses that would go outside of what you and I would naturally brainstorm on, how people are thinking about things.
James: It’s pretty good for that. I don’t workshop my blog posts with people who might be more emotionally invested than I am.
Adam: We’ll have to look at your Ocean scores later. We’ll compare them offline.
I like the idea of computers generating things. It’s a large corpus of material — you sift through it, see what’s interesting, throw away what’s not. You and I have been around long enough to tell the difference. You generated it cheaply, so the discount rate on discarding it is low.
But that’s not really what market research is. Qualitative research is different: a focus group has real human beings who can say unexpected things. Small N, unpredictable. But once in a while someone says “I never knew you could use the product this way” — and maybe only 1% of people use it that way, but knowing that might spark a brand campaign.
So I’m in favor of the tools for idea generation and checking your work. What I’m skeptical of is using synthetic panels for commercial due diligence. You’re deciding whether to acquire a BI tool or a security product, and you’re going to rely on numbers saying 20% of synthetic CIOs might spend more than USD 10MM on it.
That doesn’t seem right to me. I don’t think there’s signal like that in there.
James: My take is it’s more a part of the chain. It doesn’t preclude or replace going and doing the interviews. But you can’t test and re-test and re-test with a live interview set. You can workshop messages or hypotheses and then go confirm with actual human beings.
Adam: This is a good segue into forecasting.
All business is forecasting
Every business decision is implicitly a forecast — a bet that one management action will outperform another.
The distinction is sharpest for investors: a binary buy/short decision forces the forecast into the open. Operational managers face the same bets but rarely frame them that way.
Tetlock’s superforecasting research showed that structured generalists — triage tractable problems, decompose, work as a team, ensemble independently — beat CIA analysts with classified intel.
The Brier score disciplines probability estimates: being 52% right vs. 53% right sounds trivial, but at sufficient scale and leverage it is everything.
James: Go there
Adam: Due diligence is a form of forecasting. You’re trying to predict what the financials of this business are going to look like at current course and speed — and what they’ll look like if we apply our private equity magic on top of it.
All business decision-making is implicitly a forecast: a bet that this management action will produce better results than that one. The point is sharpest when you’re an investor at the margin — do I buy because it’s going up, or short because it’s going down?
James: Very binary — buy or do not buy — as opposed to operational decisions, which are more continuous in nature.
Adam: Exactly. When you’re thinking about numbers and where they’re going in the future, it probably helps to be good at forecasting more generally. And this is what Tetlock demonstrated, decades ago with IARPA and these defense intelligence agencies.
For podcast viewers and listeners who aren’t familiar: Tetlock is a professor at Penn who has been studying forecasting for a long time, and came out with a very influential book called Superforecasting, where he packaged the lessons of groups of generalists who — the legend has it, and I think I’m giving a fairly accurate version —
were able to forecast world events — will this world leader be deposed or die — better than CIA operatives on the inside who had all the intel collected expensively by our intelligence apparatus.
And so these generalists —
James: Did the generalists forecast the Knicks were gonna win the championship this year?
Adam: Nobody predicted that except my sons, who believed in it. And so it happened.
Tetlock codified the rules of what it means to be a superforecaster. These people were really good at triaging problems by tractability. Some things are impossible to forecast — the Knicks. Some are too easy — my soccer team, Tottenham Hotspur, doing badly. The Jets are a better example of the tractable middle ground: will they make the playoffs, will a company beat earnings, will this entity win a major award?
Allocate your time to the tractable problems. Decompose them into constituent parts. Work as a team, but have people develop independent points of view first, then bring the theories together and fight about who has the better argument. Ensemble different independently-made forecasts where possible.
Those are the Ten Commandments of superforecasting that Tetlock codified in 2015.[2] I started my forecasting contest that same year.
I read the book and launched the first Narcissist forecasting contest in 2016 — The Narcissist being a long-running annual newsletter I’ve produced since 1992. Twenty-five binary propositions about the world, each either going to happen or not.
Everyone predicted the likelihood of each from zero to 100%, including, at the time, whether Trump would get elected — which everyone gave very low numbers to.
The scoring mechanism is a Brier score. If I predict 40% for Trump getting elected and he’s not elected, the truth is zero: 0.4 minus 0 is 0.4, squared to exaggerate the impact of being far off, giving me a 0.16 Brier. If Trump is elected, my error is 0.6, and the square is 0.36. A high number is bad. It’s like a golf score.
James: Here’s a business question. Can you predict a probability, or do you only state one? You can predict whether a candidate is elected or not — but we’ll never know the true likelihood of that election at any given moment. You can estimate a likelihood, or you can predict an outcome. But can you actually predict a likelihood?
Adam: It is a bit of a metaphysical question. In a sense, all these predictions are wrong — everything goes to zero or one. There are people who look at the whole enterprise and say it’s stupid. There’s no difference between .4 likely and .6 likely. Just make the call. I’d call those people extremists.
The other camp — generally called Bayesians — believes that we don’t know ex ante whether a thing will happen, and that there is genuine utility in knowing whether something is 60% or 40% likely. The value is in the repetition. You make predictions over and over again.
The clearest illustration is betting markets. In 2016, Trump was trading at under 20% on the eve of the election. Nate Silver had him at 28% — out of consensus to the high side. If you’d looked at the pot odds and said “I can buy Trump contracts at 20 cents because Nate says 28,” you would have done well. Even though you thought it was less than 50% likely, you thought it was more likely than the market priced it.
I’m a Bayesian. Being able to tell a 52% from a 48% likelihood is useful because we make a lot of bets. You’re making them every day when you’re running a business. Being slightly more right and less wrong compounds.
James: Everybody I’ve known who’s managed money for a hedge fund is a fanatical Bayesian. There’ll be traffic, but it depends on three things that might drive more or less of it — let me think through the prior and posterior. Which is interesting, though it gets a bit exhausting.
Adam: One of the most common approaches in a long-short public equity fund is calling the quarter — is this stock going to beat or miss earnings? More precisely: beat or miss the whisper number on whatever KPIs matter.
The hit rate is 52%, 53%. It’s much better to be right 53% of the time than 52% — that’s how they make their money. They’re repeated gamblers doing it at a very high level, trying to maintain that little bit of edge in a zero-sum world where everyone else is trying to do the same.
It’s hard. But it matters to be 53% right instead of 52% right.[4]
James: Because a hedge fund portfolio manager who’s right 53% of the time with sufficient leverage is enormously value accretive. Fortunes can be made with a fifty-three/forty-seven split, plus leverage. Operational managers who are right 53% of the time tend to get fired. And in the technology world there’s the expression “all day, every day” — you can’t be wrong in one direction.
Actually, one of the interesting things here is that in some cases the hedge fund payoffs are symmetric. On the other hand, in an operational world, the payoffs are often highly asymmetric. Could you reflect on that — applying the insights from superforecasting to an operational environment for people who have to make decisions every day with sometimes highly asymmetric payoffs?
Asymmetric payoffs and career risk
Hedge fund payoffs are roughly symmetric: being 53% right vs. 52% right is the whole game. Operational payoffs often aren’t — a missed security filing, a waived privilege, a breached system can be terminal.
Adam’s formative lesson came from a malpractice case: the concave payoff curve of law (bounded upside, catastrophic downside) drove him out of litigation and eventually into analytics.
He vibe-coded a career payoff visualizer to make this intuition concrete — the same tool every technology manager implicitly needs when committing to timelines, availability zones, or budget targets.
Agile is a structural hedge against the same asymmetry: shorten the cycle, reduce the blast radius of being wrong.
Adam: I would focus on cybersecurity — an area you’ve spent more time in than I have. Risk management in cyber is an area with that extreme concave payoff curve[8], where being wrong in a certain direction is extremely career-limiting.
One of the formative experiences in that professional journey I sketched out earlier was a malpractice case I tried as a lawyer. Very few cases go to trial — almost everything is writing briefs and pleadings. But this one did. My client was a lawyer being sued because he had failed to supervise a young associate who was supposed to file in a telecom case and simply didn’t get it done on time.
Once they realized the deadline was approaching, they scrambled, missed the filings, and the telecom project couldn’t go forward. The client sued. We did good work at trial and settled before it went to the jury.
What that case made me realize was that litigation — and being a lawyer generally — has that extremely concave payoff curve. If you accidentally produce a document with attorney-client privileged information, when you’re going through warehouses of material —
It used to be physical warehouses — for lawyers today it’s all on a computer, it’s a cyber problem. But if you accidentally produce a privileged document, you’ve waived privilege and you’re in serious trouble. Being a lawyer felt like limited upside — a nice house if you make partner versus getting fired and never finding work again.
So I thought more carefully about these payoff curves. The first conversation I had with my kids when they were old enough was: don’t choose a profession with that kind of payoff. Do you want a venture capital payoff — an extreme tournament where the upside is a million times the median, if you’re genuinely confident in your skills? Maybe. But you’re probably overconfident. Most people want something with reasonable upside and limited catastrophic downside.
I actually vibe-coded a tool to visualize this — it lets you look at payoff curves for different employers and say: if I get a top-percentile outcome, here’s how I shake out; if I get the bottom outcome, here’s how I shake out. You can vary economic conditions and other factors. I stuffed it with synthetic data grounded in actual information. It’s more of a toy model and theoretical exercise — an intuition pump to make visible the thing you were saying. Do I really want the job where 10% of the time it’s utterly fatal?
James: It’s directly relevant. People make decisions like this every day. Do I launch this project? Do I use high availability for this application? Do I run it across multiple availability zones? Do I commit to two months versus three? Those are technology examples, but every business manager is asking: what target will I commit to, in revenue or in cost? Thinking through your own payoff matrix — and the payoff matrices of the people around you — changes the analysis.
Payoff matrices differ enormously by corporate culture. In some organizations, blowing a budget a little is “try to do better next quarter.” In others, exceeding a budget in a single quarter is career-limiting.
Adam: That’s right. The types of IT projects I mostly see and lead are business intelligence projects — building out and deploying predictive models. These, especially the dashboards, are generally developed in an agile environment, where agile is kind of a hedge against the asymmetry you’re talking about.
I’m not building a whole stack waterfall-style with a perfect delivery date. I’m making it good enough to get adoption, then making it better — two-week sprints, continuous improvement.
So I would argue that agile is a way of coping with an uncertain world. Do you agree?
James: It’s more emergent strategy than deliberative strategy. There are people who believe that whoever wrote the Agile Manifesto had some connection to John Boyd — I haven’t been able to find as much documentation for that, but the idea is the same: increase the number of cycles, shorten the duration between action and feedback, and revise your strategy accordingly.
Adam: Exactly. Agile development and hypothesis-driven problem-solving are both hedges against disaster. One of the first things you learn at a consulting firm — I taught the intro to consulting class several times — is that you have a day-one hypothesis about the answer.
Part of what’s good about that: when someone stops you in the hall and asks what you think the answer is, you have something. “I’m going to test it and refine it and make it better.” That’s much better than a long list of ideas with no organizing principle.
The risk otherwise is getting caught out with no answer at all. The OODA loop, hypothesis-driven problem-solving, agile — these are all hedges against uncertainty and against the weird events that shock you.[7]
James: The interesting thing is that the nature of the bad thing varies by the nature of the system. In an analytics system, the bad thing is bad information — you make a bad business decision. In an operational system, like an order capture system, the bad thing is different: downtime, a cybersecurity event, a transaction failure.
We deal with those types of negative payoffs differently, using different mechanisms. It’s probably worth thinking about that more rigorously. Let me pivot just a little bit.
What do we tell early career professionals about AI?
Get a job in a for-profit company and help it make money with data. That is still good advice. Those jobs are not disappearing as fast as the pessimists predict.
The supply of problems is highly elastic: better, cheaper analytics generates more questions, not fewer analysts.
Proof-of-work has decayed — anyone can generate a slop deck. What has not decayed: domain knowledge, taste, and the ability to have a real conversation about a business.
Adam’s standing advice: pick the domain you care about, grab the public data, do the analysis. Show up to the interview having already thought about how to help the company make money.
James: In my teaching, I’ve seen a lot of anxiety from people early in their careers about what they’re going to do with their lives in the age of AI. A number of college seniors asked me, and I said they should learn data structures and data science — because no matter what happens, that will be valuable. Did I give them good advice or bad advice? And what are you taking away from your interactions with college students?
Adam: Getting some kind of job and getting out into the real world to develop domain knowledge — for all the reasons we talked about — is important. Getting that job is a separate question; that’s tactics, informational interviewing, picking your spots.
But as advice for how to spend the next one to three years: get into a for-profit company and help it make money by analyzing data against real problems. Those jobs are not going away as fast as the most pessimistic people are predicting.
The supply of problems is highly elastic.[3] Having people ask good questions — and the return to a business from asking more good questions and getting quality answers back quickly — is positive. I’ve bet my whole career on the idea that most companies would benefit from having one more data scientist, or in this case one incremental unit of data science capability.
Which is not the same as replacing all the humans with AI. I’m on the optimistic side of this.
James: There’s more data, available publicly. There’s more tools that you can use for free. And my inclination, for example, if someone were to ask me, “Gee, I want a job doing analytics,” I would say, “Well, pick the domain you’re interested in and start doing the analytics.” Right? Because what’s gonna be more compelling in a first-round interview? It’s “Oh, gee, I’m, I’m really interested in the telecom industry, and I grabbed all this from the FCC and I did this analysis, and here’s what I found.”
Adam: I’ve given a talk on this subject probably every year for the last five years, including the pre-generative-AI era. The advice holds: do actual work, show that you know something about the business, demonstrate that you care about it.
I will say that that coming in with a deck, the proof of work aspect of it has gone to kinda zero, right? ‘Cause anyone can generate a slop deck. People like you and me might be exquisitely sensitive to seeing a deck and understanding immediately that it was written by AI and the, and the tells in the stylistic, that come out of a Claude, PowerPoint deck.
The concept is still correct: get in there, do the work, iterate, think about it. You’ll be able to have a real conversation about the business because you’ve learned something along the way — “I think 80% of the value is in this part of the product, and here’s why.”
You’re not reading off a slide. You’re engaging a person. There’s a lot of value in that. And people still show up at interviews without having thought seriously about the business and how they would help it make money. There’s still a lot of alpha in that.[6]
James: Terrific. Anything else to add before we wrap up?
Adam: The forecasting contest runs every year. Free to enter. You get a bowl of pho if you win, and a book — this year it’s George Orwell’s essays. All the information is at braff.co/advice: the current contest, how to enter next year, and other thoughts about food and analytics.
James: Terrific. Two great topics to discuss. Adam, thank you so much
Adam: Thanks for having me.
Notes
[1] Obviously twenty years is an estimate -- that doesn’t make it wrong. The world is entropic, and deterministic systems are not talked about how organized complexity is much tougher to understand than disorganized complexity.
A B2C business with a million customers faces disorganized complexity. Idiosyncratic individual behavior averages out into stable statistical regularities — churn rates, cross-sell lift, tenure effects.
A B2B software company with sixty enterprise accounts faces organized complexity. One decision-maker changing jobs, or having a bad quarter, creates a measurable swing in the entire dataset. The N is too small to smooth anything, especially since many of the data points are connected to each other. That is why domain expertise is required in B2B analytics in a way it is not in mature B2C analytics — the data cannot naturally control for the variables that large samples would otherwise absorb.
[2] All IT commitments are forecasts based on implicit probability estimates. When a CIO says “We’ll deliver in two months” she estimates the probability of coming in at two months is high enough to avoid the political cost of articulating a more conservative estimate.
[3] Adam describes here the applicability of the Jevons Paradox to knowledge work. When the churn data got good enough to slice, the analytics team didn’t shrink. It grew. Each answer generated three new questions — why are customers in this region stickier? What’s driving the gap between acquisition channels?
[4] You’ll never know whether the 60% probability you assigned to a project coming in on budget was the right number. It either did or it didn’t. But across a portfolio of similar decisions, the manager who has actually formed a probability — rather than suppressed the uncertainty — will be better calibrated over time. That’s what “it matters to be 53% right instead of 52% right” means outside a hedge fund. You’re not making one bet. You’re making hundreds.
[5] There is no perfect data set for the organized complexity of B2B commercial due diligence. Direct customer interviews and focus groups are great, but you only can do so many of them. On-line surveys? Decision-makers responsible for large technology bets at Fortune 100 companies lack the time to respond to them.
Adam is correct. No virtual panel can, well, forecast purchase intent. But they should provide invaluable feedback on value propositions and messaging. And they are both patient and indulgent. You test many variations of a value proposition for example to see which one best resonates.
[6] We’ve heard people confidently predict that early tenure technology jobs will disappear. We’ve all fielded anxious questions from candidates who should probably believe less of what they read on social media. We have all tried to figure out what to say. As Adam points out learning is the most important thing at the start of your career. And it is easier to learn useful things than it has ever been before.
[7] Colonel John Boyd’s Observe-Orient-Decide-Act (OODA) loop is a model of how people and institutions make better decisions through recursive learning in competitive games. It provides a mechanism for balancing deliberate and emergent strategies, which emerge from the bottom up. It appears to have influenced the agile manifesto.
[8] A concave payoff curve: bounded upside, catastrophic downside. Law. Cybersecurity. Most operational IT. A convex payoff curve: bounded downside, extreme upside. Venture capital. Early-stage software equity. CIOs and CTOs often face concave payoff curves. Flawless uptime is invisible. A database corruption is not. When you put together a major incident response plan, you are trying to truncate the ugly end of a concave payoff curve.
