How to Give Money + Should You Quit Your Job?
AI could bring a huge windfall to American philanthropy. Caleb Watney wants to make sure that money is spent well. Caleb is Coefficient Giving’s first managing director of public policy, and also my old boss. We talk about his essay “A Long Sequence of Small, Correct Decisions,” how to be effective in DC, and whether you, dear reader, should quit your job.
I have a wonderful co-host for this episode: Leah Sargeant, my successor as IFP’s editorial director and Statecraft superfan.
SubscribedThanks—as often—to Harry Fletcher-Wood and Shadrach Strehle for their support in producing this episode.
For a printable PDF of this interview, click here:
Caleb, when you joined Coefficient Giving, you wrote an essay I really enjoyed, “A Long Sequence of Small, Correct Decisions.” Talk about it.
I encourage readers to visit the Wikipedia page listing things named after John von Neumann: it’s much longer than you might expect. In addition to being a savant and a polymath, he was a great writer. In 1955, he wrote an essay asking whether humanity would be able to survive its own accelerating technology. For the first time we had the ability to destroy ourselves — he was writing about nuclear weapons and climate engineering. Humanity has had a slowly increasing ability to cause both harm and good. But there’s only so much damage you can do when you’re fighting with sticks and stones. Eventually you get swords, then dynamite, and now we have nuclear weapons. Von Neumann’s basic thesis is that the situation is very unstable, and the most we can hope for is an institutional solution.
It’s going to be hard to articulate ahead of time what the correct answer to nuclear stability will be. I like the way he answers this. He says:
“Can we produce the required adjustments with the necessary speed? The most hopeful answer is that the human species has been subjected to similar tests before and seems to have a congenital ability to come through, after varying amounts of trouble. To ask in advance for a complete recipe would be unreasonable. We can specify only the human qualities required: patience, flexibility, intelligence.”
He says that the only safeguards he could endorse were “day-to-day or perhaps year-to-year opportunistic measures, a long sequence of small, correct decisions.”
That’s what my essay riffs on. Our ability to reckon with the age of AI is unlikely to come from one big master plan theorized from first principles. Instead, the best we can hope to do is build institutions that can make this series of small, correct decisions, and help us muddle through this period together.
Leah: Do you think we’re still in the phase he described, where you need a long series of correct decisions to keep navigating nuclear weapons and climate change?
One thing I like about the framing is that it moves from a static question — did you survive? — to a dynamic process. Our understanding of the risk vectors across all these areas, and many more in the future, is never going to be complete. The sign of any particular intervention can flip quite rapidly. So the meta-principle you need to optimize for is whether you have institutions that can notice in real time how things are going, make and adjust plans over time, and have the public legitimacy to act on the new evidence they generate.
We’re hopefully in a less perilous situation on both of those than we were before. But that’s in part because we took concerted action and set up new institutions that helped us navigate them. That’s not to say the risks have gone away, but we’re in a more stable equilibrium because we have those institutions.
Leah: For most of this episode, we’re going to focus on AI as one of the domains that needs a long sequence of small, correct decisions. But I’m curious whether two social technologies fall into the same bucket: short-form video and contraception. Does either of these require a long sequence of small, correct decisions?
This is an attribute of technology in general. One thing I’ve always liked about our name, Institute for Progress (IFP), is that in our heads, progress has never been a one-dimensional thing that goes faster or slower. It’s multifaceted. The technology tree is one way to envision this. Our mutual colleague Tao Burga wrote a great essay a couple of months ago called “Do Not Surrender to the Tech Tree,” all about this theme of technological determinism.
To what extent is it preordained that we will come up with certain technologies? There’s a weak version of techno-determinism that does seem true. Once you have the basic scientific preconditions, if a technology is sufficiently obvious, it seems inevitable that someone, somewhere, is going to build some version of it. It’s very difficult to build a world premised on keeping the genie in the bottle for any technology. But on the flip side, there’s a huge amount of path dependence: in the order in which technologies arrive, in the business models they get embedded in, and in the cultural values that shape them.
So the answer that comes out of that essay, which I strongly endorse, is that you can’t hope to contain technology in a neat box and say, “We’re never opening that.” But we also shouldn’t consider ourselves helpless. As a society and as individuals, we have a huge amount of agency to shape technology, especially at this early stage, and to shape the pathway and conditions under which it emerges. That shapes its effects on society.
Zooming back in on short-form video or lots of other things: it would be a mistake to say that a technology should or should not be shaped. All technologies, at all times, are being shaped by laws, cultural mores, investment, and taxation. Depending on the scope of a technology and what we think its downsides are, it merits stronger or weaker forms of intervention. But no class of technology is immune.
Santi: Von Neumann’s take here isn’t exactly incrementalist. Some of the things one had to do in the nuclear age were quite non-incremental.
They were insane.
Santi: But it’s a real evolution for him. He’s widely thought of as one of the most intelligent men who ever lived. A few years earlier he’d been advocating a nuclear first strike on the Russians, which I would not classify as a small, correct decision. It would’ve been a very large, catastrophic decision. Even the very wise among us can change and learn.
I’m curious to root this in the work you’re doing. You’re a philanthropist now. As many readers will know, there’s a lot of discussion about the role of philanthropy today, because a huge amount of capital is expected to come online as various companies go public, including SpaceX. People don’t talk enough about where that capital’s going.
There’s been a lot of discussion, and I’ll link some good essays, about how new philanthropists, or people working on big, ambitious projects, should think about capital allocation. Given this focus on small, correct decisions, how are you thinking about this?
That is the big question. Nan Ransohoff talks about the third wave of American philanthropy. She describes $30 to $40 billion per year of new philanthropic capital that could be unlocked, if you look at the overall amount of new wealth creation and some of the preexisting commitments to donate large sums of this to the public sector.
That’s a really exciting thing. The American project has been shaped many times over not just by the state, but by large waves of philanthropic capital: the Rockefellers, the Carnegies, the Gates Foundation, and many others. On the flip side, each of those forms of capital has its own biases. Each will end up focusing on certain kinds of problems and solutions in ways that, left to their own devices, are probably going to be unhealthy.
In my previous IFP hat, one of the big areas I worked on was metascience. There are a lot of analogies between a healthy portfolio of funding at the scientific layer and how we as a society want to allocate capital toward different kinds of problems. One of the insights on the science-funding side is that any form of scientific production will have its weak points, blind spots, and biases. Each will ignore certain kinds of solutions or be predisposed toward certain methods or modalities. But in aggregate, with a healthy mix of ARPAs, university basic research labs, industrial research labs, focused research organizations, and any number of other things, you have scientists in a wide variety of places attacking problems from many different angles. That increases your odds of leaving no stone unturned in finding good solutions.
The same principle applies to philanthropy. One of the things I’m most excited about is that philanthropy can tackle kinds of problems that the public sector can’t. On the flip side, philanthropy has its own biases and weak points. It would obviously be bad if all capital allocation were left to the philanthropic sector. But as part of a larger ecosystem that includes the private sector, philanthropy has a meaningful role to play in taking crazy shots on goal.
Within AI, philanthropy has been an absolutely massive shaper of the conversation. Coefficient Giving, previously Open Philanthropy, first started taking these issues seriously in 2014 or 2015, almost a decade before ChatGPT and the most recent wave of large language models. At the time, taking AI seriously was really weird. The idea that there could be large-scale societal harms worth taking proactive action against was not at all mainstream. I’m very glad there were philanthropic voices out there — not just Coefficient Giving, but lots of others — willing to fund early research in areas like mechanistic interpretability, benchmarking, and evaluation. We now have an ecosystem of organizations outside the big labs thinking carefully about AI measurement and evaluation. It still needs to grow a lot, but it exists in large part because philanthropic voices were early to the conversation.
Editor’s note: here’s some good further reading.
Leah: You’re trying to help spin up a lot of new organizations through Project Tailwind, where you list organizations you’d like someone to found. I think your optimal failure rate for those projects is not zero, unlike most philanthropies, which look to fund winners.
How do you talk to a funder, or to a founder you want to quit their normal job, when you think the project might have a 10% chance of working? You’re asking them to put their IPO money, or their livelihood, toward something that probably won’t work, but is positive in expected value as part of your overall portfolio.
This isn’t a novel problem for philanthropy. In some sense, the entire ethos of Silicon Valley is this. Most startup founders have a kind of rational exuberance.
Leah: But definitely not most philanthropic foundations.
Correct. Motivating people by the shape of the problem, and the impact they could have on society, is a form of compensation. Lots of people want to leave their mark on the world. Tackling some of society’s biggest, most important problems motivates a lot of people. Helping them navigate that ecosystem, and making sure they’re oriented toward problems that are really high-stakes, is something a lot of people find intrinsically motivating.
I don’t find it that hard to motivate people to work on problems that only have a 10% chance of working, as long as they’re convinced the underlying problem is the right one to be working on.
Leah: Are the founders slightly delusional, then? Do they think, “Most of these have a 10% chance, but I’m the 80% chance,” while you know it’s 10%? Or do they have more of a swarm mentality, where they say, “If each of us embraces this 10% chance, it’s going to work out well for us as a body”? I’m that kind of person, to be clear.
Santi: I don’t like that we’re now referring to human altruism as a swarm mentality.
Leah: I don’t think of this as normal human altruism. It’s always been timeless decision theory moving me here.
Expected value is a very old framework. At IFP, this was something we ran up against a lot. In some sense, the whole IFP thesis is that policy is really hard to change, and you should expect to fail completely most of the time. But you only need a couple of wins to pay for the overall portfolio.
Across IFP’s five years, we’ve seen some pretty concrete, tangible wins, which I think have paid for the portfolio in a philanthropic, social-value sense. But it’s good to maintain a community and positive vibes that let people be roughly correctly anchored on the underlying probability of any given solution working, while also giving them social support, and in some sense status, for running after hard problems in an admirable way.
That doesn’t always work, but I think it’s the correct approach. The other thing is that you can get too lost in the sauce trying to weigh the exact probability of success. If you’re running after a really big problem and you think you have a plausible chance, sometimes it doesn’t matter whether it’s a 10%, 30%, or 50% chance. If the problem is sufficiently big and important, it’s usually worth running after.
Leah: I seldom see people praised for shutting down a project early once it’s clear it’s not panning out. The piece that stands out to me is “Why We Shut Down” in Asterisk, where two founders explain why they closed their NGO. It wasn’t because the money dried up; it was because they weren’t convinced of the impact.
How do you give people, through Tailwind or other projects, the social support to do ambitious things that are likely to fail? Do you, or should you, do anything to encourage people to shut down projects — to throw them a “Your impact wasn’t sufficient and you stopped early” party?
One way venture capital (VC) creates this is that being a failed founder makes you quite qualified to try again on something else. Having the wits about you to realize the thing you were running after was no longer the right move is celebrated in that community, and is a marker of status.
I don’t know that we’re quite there at the philanthropic level. But I have seen organizations and people wrestle seriously with it: “I had a theory of the case that turned out not to be true,” or “Facts about the world have changed in the last two years, so the thing I’m running after is no longer the highest-leverage opportunity for me. I want to think seriously about whether there are other things across this larger funded ecosystem that would be useful for me to do.” It’s not quite as liquid a market as VC, but there’s something approximating that, which seems useful.
Santi: Nan Ransohoff’s essay says there’s $30 to $40 billion a year potentially coming online, starting next year. She has a few examples of how to conceptualize that number. It would fund roughly 5,000 IFPs. It seems clear to me that 5,000 IFPs would be way too many. Possibly even 200 new think tanks would be the wrong concept.
You’re deploying a more-than-nominal chunk of that money, because Coefficient Giving is a very large organization. You can’t say, “Hey, everybody, go start a 501(c)(3) and do policy research in Washington, DC.” So where does the money go? It’s an insane amount of money.
Different areas have very different elasticities: how responsive they are to additional supply. Some areas, like global health, are extremely elastic. If you got an additional $10 billion to spend on new vaccines, clinical trial designs, or humanitarian interventions all around the world, there’s an enormous set of well-vetted, exciting opportunities you could immediately plow it into. Other areas are much less elastic, and policy is one of those. Fundamentally, what you’re supporting with philanthropic dollars is people’s time, which makes it much less scalable than other interventions. Buying physical goods, be it vaccines or compute, is inherently a much more scalable solution.
We fund a lot of great organizations across the policy ecosystem, like IFP, RAND, and Horizon. But you can’t arbitrarily scale up and say, “I like IFP. I want you to be 1,000 times bigger.” There aren’t enough people who could do it, and in a lot of ways it would backfire. If you try to seed too many organizations, you create concerns about astroturfing and all sorts of other things. So in policy, it’s very important to have some baseline level of support, to make sure the most thoughtful people in the world have the time, space, and ambition to run after really important problems: to do the research, build the coalitions, and drive things forward.
But beyond some scale, money ceases to be the bottleneck. It’s people, or, to use the VC term, deal flow. A big part of my job now is trying to pull new deal flow into the world. That looks like talking to promising young people at organizations that aren’t making the most of their talents, and helping them figure out what to do:
Should they start a new organization?
Should they go somewhere that would make better use of their talents?
Should they pivot to some interesting intersection between two fields that no one else is thinking about?
Sometimes it’s talking to an organization that’s already doing a useful strain of work, but with one slight tweak could fill a huge policymaking gap, where there’s enormous demand from policymakers for financial models, legal analysis, or whatever.
So it’s basically matchmaking, from a 30,000-foot view:
Who are the most talented and promising people across the ecosystem?
What are the most important policy problems?
Where is the biggest demand?
What institutions can make it all happen?
Unfortunately, labor markets are not all that efficient, which means there’s a lot of alpha in better matchmaking.
Leah: Other people highlighted this when I was interviewing them about you in preparation for this conversation: Caleb is going around telling people to quit their jobs and take new ones — requisitioning labor.
Santi: I can confirm.
Leah: I don’t know at what point people are going to ban you from their offices, because you’re like the angel of death, coming through and taking people out. Why is a Caleb necessary to do this? Why is the labor market for talent so inefficient?
There are a couple of ways to think about this. One is more general and Tyler Cowen-esque: there are enormous returns to raising the ambitions of others. My co-founder Alec Stapp and I were early Emergent Ventures grantees for IFP. There was a small check attached, which was useful when you’re trying to get a new nonprofit off the ground. But part of it was also the bet from Tyler — someone we all deeply respected and who has been a mentor — that this was a project worth endeavoring on, and that we were people who could go and do things in the world. It’s so easy to get stuck in the day-to-day of whatever you’re working on and miss the forest for the trees. There’s a great Steve Jobs quote about how, when you look out at the world, you realize all these structures were made by people no smarter and no more ambitious than you, and that you can go out there and change them. That’s a deeply true baseline about the world.
Second, this is even more true in the nonprofit space, which is not an efficient market. This was a big part of our theory of change at IFP. In the for-profit space, you have all these competitive dynamics. Companies can pay higher salaries. New organizations constantly spring up, and old ones are forced out of the market by pretty ruthless competition. It’s not perfect, but it forces the overall equilibrium to keep evaluating how best to put people’s talents and abilities to use, and which new ideas need to be scaled.
There’s basically no forcing function like that in the nonprofit sector except, in some sense, funder pressure. So it’s unusually important for funders to have an inside view of what’s good to do in the world, who’s doing good work, and what a useful theory of change looks like. Then, on the margin, funders can try to nudge people to do a more ambitious, bigger version of what they’re doing, if they’re on the right track — and, if they’re not, to expand their horizons.
Policy is an area that benefits unusually from having a skill set for doing direct work and a skill set for allocating capital. In some spaces, like international development, these are extremely different skill sets. If I’m allocating vaccine funding, I don’t have to be an expert in cold-storage supply chain logistics. I don’t have to second-guess the route around the Horn of Africa for delivering malaria bed nets, because the outcomes are all very verifiable. In policy research, analysis, and advocacy, things are much softer. It’s more art than science, and it’s much harder to know what’s true. So it benefits an unusual amount from having done the thing yourself before allocating capital.
Leah: There are at least 15 people reading this who should quit their jobs and find more impactful ones. To save you tracking them down, I’d like you to enumerate the criteria, about them and their current jobs, that should tell them a visit from Caleb Watney is in their future, so they can make the change now and you can move faster.
Santi: The real number of readers who should get the talk from you is much larger.
Leah: But let’s realistically knock off 15, and leave a comment if it reaches you. Caleb, how should these people recognize themselves without your help?
What an opportunity. If you have any interest in policy, here’s how I’d think about it. First, things are going to get quite crazy quite quickly, especially with AI. That means a lot of theories of change that rely on things happening 15 or 20 years down the road will very quickly find themselves operating in a different world from the one in which you made the investment.
You can think about this as an intellectual discount rate. How much do I expect to still be in the same epistemic regime, to use a term from my colleague Matt Clancy, when the investment pays off as when I made it? In the current environment, the discount rate is unusually high, beyond our typical 2 or 3%. I don’t have a number, but maybe something more like 20%. Things are happening so quickly that the world could completely change before your eyes, and the correct thing to do could be radically different. So somewhat shorter timelines are warranted.
Only a couple of governments really matter in the short term: the US and China. I have a lot of friends doing great work in the EU or the UK. One of my generic pieces of advice right now is that you should seriously consider moving to the US or China. I assume it’s much harder to move to China and do meaningful policy work there. So it’s time to come to the US. If you’re in the US but not in SF or DC, you should really strongly consider moving there.
Santi: New York’s great too.
New York is a great cultural place. But it is not the center of power for the forces that will most affect the American economy over the next five years.
Santi: Agree to disagree.
Move to the US, and move to SF or DC. Then, think seriously about the issues you’re working on. One question that feels very top of mind to me right now, almost on fire — why aren’t more people working on it? — is: if you assume Congress is going to want to do something on AI governance, what should it actually do? Despite this being an extremely obvious question, I don’t know that the people who have been thinking longest or deepest about AI policy feel they have a robust answer. Partially that’s because the strategic situation is changing so quickly.
There are some areas where we have some certainty. There’s been a lot of momentum around the idea of embedded third-party auditors. Wouldn’t it be nice if some entity had visibility into the labs and could create public transparency and understanding about what’s happening inside them? That seems to be a major crux of disagreement between people inside and outside the labs: How crazy is the world going to get, and what should we do about it?
There are some vague directions we feel would be good. Take AI security. Who has access to model weights, how those weights get distributed, and under what security conditions they’re held seems extremely important, especially for US-China competition. But what does that mean practically? Do we want to move most frontier training runs to Security Level 4 data centers, to use RAND’s terminology? That seems plausible, but the details are underbaked. And what new governmental institutions do we need to create?
One example I cited in my essay, which other people have mentioned too, is something like the National Transportation Safety Board, but for AI: an independent group with no regulatory authority, which can go in, investigate what happened, and create public awareness about what we could change in safety systems or security mechanisms to prevent it from happening in the future. Right now this role is being informally served by third-party groups like Model Evaluation & Threat Research (METR). I’m very glad they exist, but it would be nice to have a more institutionalized version.
There are 100 other questions:
What do we want to do about building more resilience into the labor market?
What do we want to do about congressional empowerment vis-à-vis the executive branch?
How do we want to think about concentration of power more generally?
How does the balance of power between corporations and the state change when one of the most powerful technologies ever is being developed almost entirely under the auspices of private companies?
There’s an extremely long list of questions we don’t have good answers to. That feels worrying, when Congress could be beating down our door, plausibly as soon as six months from now, saying, “Guys, we need some solutions. What should we actually do?”
I also don’t want any single group of people to be the ones creating those answers. These ideas need to be road-tested and red-teamed, with some public consensus around what we actually want to do. We’re far from that point currently.
Santi: Can I propose one more criterion for readers? This might be banal, but: if your institution doesn’t do anything, you should consider going somewhere else.
To finish the hierarchy:
Consider whether you should change countries to come to the US.
If you’re in the US, consider whether you should go to DC or SF.
If you’re in DC or SF and you’re at an institution that doesn’t do anything, consider going to, or starting, an institution that does.
If you’re at an institution that does something but isn’t oriented toward the most important, pressing problems, consider pivoting toward them.
That will get you pretty far.
Santi: Several months back, we had Greg Berman on to talk about problems with the traditional philanthropy apparatus. Philanthropists, like venture capitalists, are often herd animals. They’re faddish. They’re maybe more consensus-driven than actors in adversarial domains. Traditional philanthropies these days also face a ton of internal pressure. Employees and leaders are often way out of step culturally, politically, and philosophically.
What Nan Ransohoff would call the third wave of philanthropy can avoid some of these problems. If you were to give the bear case for this wave of philanthropy, what are the failure modes you want to avoid?
One big failure mode we’ve seen strike previous waves of philanthropy, which I’m worried about with this one, and which is especially relevant to our discussion of discount rates, is: How quickly are you able to deploy capital? There’s been a long tradition of wealthy industrialists making lots of money and deciding, “I’m going to set up a foundation when I die, and we’ll deploy the money then.”
There are tons of great opportunities that could have used that funding much earlier. In the same way that money can compound in a savings account, impact can compound in the real world, through the institutions you set up and support. I’m so glad IFP was set up five years ago rather than today. That gave us a much better starting point. We’ve been able to have tangible impact, and our future impact is much bigger because of that five-year head start. So how quickly can you get these new things off the ground? Are we deciding to wait until people die? Hopefully the principal also has useful expertise to bring to bear in deciding which problems are important, and that’s worth taking advantage of. But waiting too long, or waiting until you can do this full-time, is a major failure mode to avoid.
One interesting angle is going to be how to navigate around conflicts of interest. A lot of this new money is coming from the AI sector. Understandably, many of the problems those donors think are most important are in AI governance or evaluation. How can you take large-scale funding that is obviously good and find governance mechanisms, structures, or institutions that let you deploy it toward problems where these conflicts of interest are much less severe?
That’s why it’s great that some of the problems most elastic to additional supply are completely orthogonal to AI. There’s a huge amount of great work to be done in global health and well-being, in biosecurity, and in tons of other fields that this money should pour into. But within AI governance, we have to be really thoughtful about how we set these things up, and make sure there’s independent decision-making without a vested interest in the places giving the money. The entire point is that AI governance needs to be separate from the people in the labs.
Santi: There’s a related question about whether you should prioritize “secret Congress” ideas or issues firmly in the public’s Overton window. In the main, IFP focused on issues that weren’t incredibly high-salience. Part of the original thesis was to work on issues that are important, tractable, and neglected — to coin a phrase. We’d be more effective working on neglected issues than as the 20,000th organization speaking up on the cost of living. And those problems are more amenable to backroom dealing. AI plausibly was one of those domains, and now no longer is. If Congress acts, it will be because AI is rising in salience faster than any other political issue.
So if you’re a philanthropist who wants to help American policy on AI be better, can you really play the secret Congress game anymore? Should you be doing mailers in New Hampshire? How should you change your theory of impact when this stuff is becoming salient so quickly?
There are some areas that are less political and more policy, which still need an enormous amount of work: all the policy development that has yet to happen, or that needs an urgent strike team. My hope is that it comes from a set of actors that are pretty pluralistic and connected to a lot of preexisting American political traditions, such that you’re creating a thing that can actually move.
One interesting thing about crisis moments, if we expect that’s the policymaking environment Congress will feel it’s under, is that they sometimes have the luxury of being more bipartisan than the day-to-day grind of a big tax or health insurance bill. A lot of the COVID response packages passed with flying colors across party lines. In my essay, I wrote about thinking of policy as a kind of punctuated equilibrium. Especially on the congressional side, you go through long periods where nothing happens, and then everything happens at once. Usually that’s when an issue has become so salient that the public demands Congress do something about it right now, and it overwhelms the day-to-day partisan infighting.
I think this could happen much sooner than people are expecting. It’s quite plausible we have a window in spring or summer of 2027, driven in some sense by crazy cyber capabilities that could be very near on the horizon. The trend so far is that open-source models end up recreating frontier capabilities four to six months after we’ve seen them on the frontier. Mythos-level cyber capabilities became broadly available to the public in roughly July, after the whole export-control fiasco, so there’s a ticking countdown from that. Exactly how that diffuses is slightly unclear. But that has been the trend so far.
So my technical prediction is that we’ll start to see really crazy cyber capabilities across the winter and early spring next year. I think these will be quite bad, and our institutions are in no way prepared. There were some attempts to prepare society. At IFP, we wrote a lot of great pieces through The Launch Sequence about what it would look like to go all hands on deck to prepare for the incoming cyber craziness. Unfortunately, I don’t think society has taken us up on those. So it might look like:
A 100x increase in very sophisticated grandparent scams, where everyone knows someone in their family who’s been affected.
The Federal Aviation Administration having to cancel flights at random a dozen times over a month, because it doesn’t have full confidence in its communications networks.
Local hospital systems coming under a constant barrage of very sophisticated ransomware attacks.
To some extent they’re inevitable unless we batten down the hatches and get our act together on cyber resilience.
Leah: For those cyber risks, I’m curious how you’d allocate your probability mass among the actors who could make people wake up to the risk. There are non-state fraudsters, pig-butchering grandparents with these tools just for money; non-state political or terrorist actors; state actors; and the AIs themselves.
Which of those do you think is most likely to drive large cyber disruption?
I’d probably put the most probability mass, at least in terms of the largest number of incidents day to day, on normal financial interest. There’s already a very large cybercrime operation out there. One analogy I heard recently is to imagine there’s another Russia out in cyberspace, passively doing a bunch of cybercrime. That’s maybe the best way to conceptualize what’s going to happen, with attacks at different levels of sophistication.
One area where I’d put surprisingly high probability is autonomous AI activity that isn’t even intended by anyone. Our mutual friend Dean Ball [who was on Statecraft last year] wrote a really interesting post about the idea of self-sovereign AI. It seems like it will soon be trivially easy to set up a Bitcoin wallet, buy some dark compute that’s not tracked anywhere, set an agent free, and tell it, “Do whatever you want.”
There’s a long history of people doing crazy things like this. With ChaosGPT, someone set up some of the early GPT scaffolding and told it to cause maximum damage. There are a lot of people out in the world with really bizarre motivations, who want to see what happens when you set up these systems. Some amount of stuff from that may also cause really crazy cyber harms.
But it’s hard to reason about the exact probabilities. Mostly, I expect cyber to become very salient. People trying to anticipate the policy demand early next year should be thinking hard about what sort of cyber incident response we want, how we batten down the hatches, and how we bridge from that to the broader question of frontier AI governance.
Santi: You and I have reflected over the past couple of years on these moments when the Overton window suddenly opens — the train is leaving the station, do you have something you can put on it? There’s a lot of work you can do to develop good policy ideas before these moments. But other people are trying to get their bad ideas on the train. They’ve also been working for a while. Often the thing policymakers pick up in a moment of crisis is not a very good idea, even if you had one ready to hand.
Or policymakers don’t learn the lessons you’d imagine they should. Post-COVID, one reflection many people had was that our political establishment did not learn the lessons of COVID. Everyone memory-holed it. We’re not doing all the pandemic preparedness work you would have wanted in 2019. Parts of our public health establishment really did not cover themselves in glory, so we lost trust there. Are we better prepared, having had COVID, for the next bio risk? Maybe yes, maybe no.
So putting together your package for the next moment is necessary but maybe not sufficient. It’s not just “Write a really good research paper, so when the moment comes you can wave it around.” If you’re funding or advising people who want to influence policy, what’s necessary right now, beyond coming up with good ideas?
Cultivating a diverse set of skills and expertise, because it’s sometimes unpredictable what will end up being really useful. The COVID example is interesting. If you think about the most successful part of the policy response to COVID, it was in large part the design and implementation of advance market commitments for the COVID-19 vaccines. [Statecraft has discussed this with Chris Snyder, and how to replicate Operation Warp Speed with Alex Azar.] It was very fortunate that we had a small bundle of really talented policy economists who had been building up the theory for decades and had thought carefully about contract sizing, thorny negotiation problems, elasticity, and the local information that firms have versus what the government has. They were able to bring that expertise to bear at a critical moment. If we hadn’t had those policy investments — that living, tacit knowledge of how to use advance market commitments lying around — our COVID response would have been dramatically worse than it was.
To some extent, our failure on COVID follow-through was because we didn’t have fully baked plans for things like disease surveillance. We set up wastewater surveillance systems ad hoc, which let us see where COVID was floating around in sewer systems. But a lot of those systems didn’t continue. I certainly didn’t see robust, well-developed plans for how to operationalize that at a tactical level. There wasn’t coalition-building with local municipalities to build on these systems and make sure the data was linked up in properly interoperable ways.
There were some thoughtful people working on this, but it was so small, and a little too late, to take advantage of the momentum. If someone had spent five years before COVID building out a plan for how wastewater surveillance systems could be a key cog in our entire disease apparatus, I think that would have dramatically increased the odds that it became a key component of the US COVID response, both in that moment and going forward.
A lot of the responsibility here is on legislatures, which have let their own policy development apparatus atrophy. That’s why broad-based interventions — beefing up congressional capacity, giving Congress more staff, more resources, and more ability to wrestle with these crises, both in the moment and beforehand — seem so important. But there are also ways the outside ecosystem can supplement that. [On congressional capacity, see Statecraft’s interviews with Matt Lira on modernizing Congress and Kevin Kosar on the Congressional Research Service.]
Santi: Let me push you on this. Functional institutions like the US military do a lot of planning, and maybe you don’t end up using the plan, but you’ve built that muscle. But it would be a failure of this philanthropic moment if what you seeded was a bunch of traditional-looking think tanks that thought their job was to produce good ideas and write them up in white papers. Part of the reason IFP has been successful is that that hasn’t been the sum total of its vision: “We’re going to prepare good ideas so that when a critical moment comes, a legislator will reach behind them and pull a good idea out of the box.” There’s a lot of socializing you have to do. What would the implementation look like? Can we grease the skids?
I’m unconvinced that developing better ideas ahead of a crisis is an especially big component of the whole package.
This is a useful push. Congress is one of many institutions that will have to respond well in any particular crisis. But going back to COVID, we’ve all acknowledged that the Food and Drug Administration and the Centers for Disease Control and Prevention (CDC) were supposed to give us live response capacity in a crisis moment, and they completely failed on the job.
The general diagnosis I’ve seen, I think from a previous CDC director, is that over time the CDC had let its institutional live-readiness muscle atrophy, and had become mostly an academic institution. When the CDC was set up, post–World War II, it was trying to push back malaria in the United States. It succeeded so well that it mostly eliminated malaria. At that time, it was very much a wartime-logistics-style organization. The people it brought in were supply chain experts who knew how to manufacture countermeasures, deploy them in the field, ring-fence diseases, and push them back.
After that, there didn’t seem to be a next thing to do, so the people who ended up getting hired by the CDC were academics who wanted access to really nice data so they could publish better papers. Over the course of several decades, the CDC let its institutional response muscle really fail. That’s a version of having unwise, or maybe incapable, institutions. When you have a response function, you have to find some way to keep practicing it in the world, to instantiate it and keep alive a body of tacit knowledge.
Maybe this is where a lot of the think tanks and policy apparatus have been too focused on Congress, and not focused enough on supplementing the CDC, or making strong recommendations inside it, to push it to be the best version of itself. When I think about IFP’s previous successes, a lot of them have been agency-focused: helping the National Institutes of Health (NIH) or the National Science Foundation (NSF) think about their science funding mechanisms. [Statecraft has discussed what’s wrong with the NIH’s funding model with Mike Lauer.] It wasn’t just throwing ideas over the fence at Congress and expecting it to act. Another component of a good response is that you have to think about new institutions, and then have societal mechanisms around them to make sure they stay ready for prime time.
Santi: Readers are clapping and cheering: “Statecraft is talking about implementation and state capacity again!” But I’m thinking of colleagues at IFP who came from the CHIPS Program Office. They have written a lot — and been on Statecraft — to say they had some great carve-outs from the usual bureaucracy that enabled them to move quite quickly. In other cases, where they didn’t get the full carve-out they wanted, they spent tens of thousands of hours filing forms in triplicate. They felt that was a real drag on their performance.
To bring everything back to my favorite topic, state capacity, there’s an argument that you should be prepping not just good policy ideas, but also work on how you might carve out an institution from a given set of bureaucratic restraints, or how you might loosen those restraints in general, on the assumption that anything we do in the next couple of years will benefit from speed.
I love the classic state capacity agenda: Let’s do things; let’s think about the legal mechanisms. But I don’t want to underrate how crazy this moment with AI could feel. A lot of our normal processes for dealing with institutional failure — “We messed up that procurement contract two years ago; we’ll definitely get it right this time” — work in a normal cycle. This kind of trial-and-error iteration has served us well across a lot of mechanisms, especially when you have the chance to get it right. One of the things I’m most concerned about with AI is that this could all happen so quickly that our normal mechanisms for institutional evolution and adaptation fail. I think that’s going to push us toward more radical options, or at least redundant ones.
One idea that’s come to mind: when the Defense Advanced Research Projects Agency (DARPA) needs a specific engineering breakthrough, it doesn’t just choose the one best team to fund. It tries to fund ten different teams taking radically different approaches. [Statecraft has discussed how to run good DARPA programs with Joshua Elliott.] Sometimes you want to make sure those approaches are anti-correlated, so that if one fails, that increases the probability that the others will succeed, because you’re attacking the problem from many different angles. We’re going to need to do things like that at an institutional level.
In this moment of slow-moving government bureaucracy, we should be thinking about whether we can build the capacity we’re going to want outside government, with the option value of adding it to the government later. And I think we need ten other ideas like that, where we can do really rapid iteration, experimentation, and testing of new institutional forms outside the confines of government, but with the government in mind.
Leah: What’s a good example of the government successfully eating an organization that was spun up independently, with a nimbleness that wouldn’t have happened if it had been incubated in government, and then successfully integrated later?
I don’t know that I have super-concrete examples. There must be some post–World War II. There was the National Defense Research Committee, a quasi-governmental thing that borrowed a lot of scientific expertise from universities, and then de facto formed it into the NSF and the NIH.
Leah: What would we gain or lose if METR got eaten by the Center for AI Standards and Innovation (CAISI) tomorrow?
It could be quite good. METR would be the first to tell you that right now they’re not at the size, scale, or level of sophistication they want to be. What they’re doing right now in terms of embedded audits is very far from a science; it’s more like an art. They’re trying to go in and form an all-things-considered sense of which areas seem high-risk, which things we’re most worried about being off track, or how AIs could escape the different defense mechanisms we’ve set up.
Usually, when things get instantiated within the government, we want something more like a process: a checklist of things we’re going to look for. There are some areas of AI where we can do that. Are you below some threshold of chemical, biological, radiological, or nuclear risk? That seems like the kind of thing you could theoretically set up a whole apparatus to do, and there are a lot of thoughtful people at CAISI and elsewhere trying to do versions of this.
But one of the tricky parts is that our scientific understanding of AI is still at a premature phase, yet we’re starting to reach quite dangerous levels of capability. So I’d be really excited about a world where CAISI was empowered to keep doing this kind of scientific exploration, while also having the imprimatur of the federal government and its stamp of legitimacy. Hopefully a lot of the preexisting capacity around evaluation can be a big part of informing that. But one of my concerns about that kind of absorption is that you’d lose the more scientific, exploratory, all-things-considered view, and it would become more of a formulaic checklist.
Santi: Even if you want a big portfolio of bets — to beef up CAISI, external evaluators, the UK AI Security Institute — you still want to maximize the value of each bet. That might mean CAISI needs to be able to pay people much more, or hire much faster, or lots of other things that fall under a traditional state capacity rubric, even if you also think we need to fund a bunch of private, nonprofit, or other novel kinds of entities.
Totally. This isn’t to say the traditional state capacity agenda has nothing to say here. In fact, it’s deeply relevant. It’s just not going to be enough by itself to solve the problem.
Santi: In your essay, you used the language of the virtues to talk about the kinds of wise institutions we might need. One version of that is that institutions need an information feedback loop: to learn from the outside world, and when things change, adjust accordingly.
What other institutional virtues do you think it’s important to cultivate right now, inside or outside government?
When I think about the failures of the CDC and our public health institutions, one was that they didn’t understand the role they were supposed to play. They overstepped their bounds in the authority they tried to claim. That was a big part of the backlash. These were scientific, technical bodies trying to make all-things-considered political determinations that were outside their scope. It’s one thing to say, “From an epidemiological perspective, we think the risk level of policy decision X is Y.” It’s very different to say, “You have to do this,” or that this is what it takes to be a good, upstanding person in our political system. Knowing which political determinations you as an institution have the legitimacy to make, and which you don’t, is a key function of wisdom — knowing what role you’re supposed to play in the ecosystem. I could see this coming up especially in AI.
Another is knowing how to fail gracefully. If we have ambitious institutions attempting lots of things that have never been done before — or have only been done in very small settings or outside government — we should expect they won’t always succeed. There are ways of failing spectacularly and ways of failing gracefully. So trying to think through: What are the robust solutions, the ones we’ll be glad we took in most or all worlds? How do we game that out ahead of time?
There’s also avoiding the execute-a-script mentality that government agencies very frequently fall into. We want to cultivate live players that can constantly reassess the field and reevaluate the right thing to do.
Finally, there’s having democratic legitimacy. If we want institutions that can make meaningful decisions for us as a society, in key moments of crisis and on a fast basis, that requires buy-in from the set of actors you claim to represent. That applies to both inputs and outputs. It means representing perspectives from a very pluralistic, diverse set of coalitional actors, so that when you make determinations, you’re not just speaking to one narrow technical community, but have buy-in across a large set of society.
For a lot of the governance mechanisms people in the AI community think will be necessary, there’s no way to implement them in a robust, durable way unless you’ve created widespread consensus across many different actors and layers of society. That can be really easy to miss, and it’s really important.
Leah: The theme of this whole conversation has been how to form people and institutions for a high-volatility, short-timelines environment. I’ve worked in places where we thought about where to find people well adapted to the zone we needed to work in. We found we were unusually likely to find good ops people by looking at stage managers: people who had handled chaos in the physical world and non-negotiable deadlines well.
Where is the unexpected place you’re likely to find people well formed for these challenges, who aren’t currently working in these fields, or who don’t think of themselves as defined by working in a high-volatility policy future?
Some fields are by nature faster-moving, or rely on more delegation and iteration. The national security and intelligence communities are interesting places for this. They have more of a security mindset, where you imagine all the different ways something could fail and proactively plan for them, rather than thinking about the two most common failure modes and saying, “We’ve probably got it handled.” Cultivating the security mindset seems good.
People who have pivoted across fields before are unusually able to pivot to this one. It was interesting to think about the characters and institutions that were able to respond. Not many people in society had been thinking about biodefense before COVID. One of the first projects Alec Stapp and I really worked on together came out of a call for novel policy solutions from the Mercatus Center. The vibe was that we’d probably need to think outside the box, because we didn’t have a well-developed set of policy actions. Alec and I ended up writing a policy brief for what was, though we didn’t have the language at the time, effectively an Operation Warp Speed for mask production: trying to accelerate it dramatically by giving manufacturers demand certainty, to incentivize more up-front investment. That was good for Alec and me, even though it was entirely outside our wheelhouse. And the fact that Mercatus saw it as part of its role to put out the call for proposals, and to send the demand signal that this was where we needed more effort, is a big credit to it as an institution.
Leah: How do the virtues of working in a high-volatility environment as a philanthropist, with a wide variety of recalcitrant stakeholders, cross over with those of being a parent to two young kids? Which virtues are helpful in both domains?
Emotional stability is very important. Compartmentalization is also useful. When you’re working on AI policy, you’re often thinking about extremely heady, uncertain things that, if they happen in the way some people predict, have existential stakes. It can be very easy to let that become all-consuming. I find it very balancing to go home and have my two-and-a-half-year-old say, “Daddy, I want to play catch.” It’s a nice circuit breaker. It roots you in why this all matters, and what we’re all doing it for. And it’s a healthy way of forcing some emotional balance.
Santi: I think the classical answer to your question, Leah, is all of them. If they were domain-specific, they wouldn’t be virtues.
Let me ask one more concrete question, about existing legacy institutions that don’t work very well. There’s a real need, especially for philanthropists, to think hard about what new kinds of institutions we might need, and what big, creative bets are worth taking, à la Operation Warp Speed. But as a result of our history, we’re also given a bunch of institutions like Congress, which everybody knows doesn’t work very well. They’re hard to fix. Many have tried; few have succeeded. Nevertheless, it’s going to be important in these fast-moving moments.
I think the natural impulse of many people on the West Coast is going to be, “Screw that. If I’m going to do federal stuff, it’s going to be Elon-style.” They’ll want to go in and work on the executive or agency side. IFP spent a lot of time working on the agency side because Congress is less tractable.
If you’re coming to the problems of frontier AI policy and want to deploy capital, is there an argument for trying to make Congress work better? How should you weight your effort? Should somebody be trying to double the amount of money Congress sets aside for itself in the next Congress, for instance?
In a vacuum, that seems quite good. But you run into a timelines issue. If you’re working through the appropriations process to get Congress to double its resourcing a year or two from now, then by the time that kicks in and you start hiring people and building IT systems, you’re looking at five to seven years down the line.
We should still do it. It seems like a robustly good solution; it’s really hard to imagine how it goes wrong. But it may also not be enough by itself. For a long time, people in a variety of policy communities have realized that because Congress is slow-acting, there’s a huge amount of leverage in focusing on what the executive branch can do.
Famously, “I’ve got a pen and I’ve got a phone.” But Congress is going to act on AI. Given that, I think it would be a mistake not to allocate substantial effort to improving how that response goes. There are tons of things the executive branch can’t do by itself, where Congress is the ultimate owner. The technical folks sometimes like to say, “Update all the way.” If you notice that every three months you’re making your AI timelines a little bit shorter, anticipate where they’re going and make that update now.
I think there’s a similar thing on political will that people have underrated. If you keep noticing an endogenous social response, it seems like Congress, the government, and the people are going to want a meaningful say in how this goes, which they absolutely should. This is maybe the thing my Silicon Valley friends have been most wrong about: underrating the chance of a very substantial government effort. Even if you’re not interested in the government, the government is interested in you.
Even if you think the response is going to be bad by default, taking a D-minus and making it a B-minus has enormous leverage. So don’t let the perfect be the enemy of the good. They are going to act. It seems really worthwhile and high-leverage to make that action as good as we can. There are also a lot of deep problems that are really hard to imagine solving without the government. It’s collective action problems all the way down, and helping us as a society navigate collective action problems is the classical role of government.
Santi: As we close, give your charge one more time to the readers who need to quit their jobs.
You only have so much time in your life. There’s the famous career-advice website 80,000 Hours, named for the number of hours you have in a career. If AI timelines are short, as they plausibly may be, people have joked that maybe it should be 8,000 hours, or 800. We don’t really know. But this is a period of intense leverage, not just over your life, but over the next century of human civilization. If you’re on the sidelines, wondering whether you can contribute, there are so many important problems that need to get solved.
A lot of perspectives need to be represented. There’s no way one particularly clever group of people is going to sneak some set of ideas past the American people and form the new government response to everything. The only way we’re going to do this is with a broad societal consensus about what actions we need to take. There’s important work at every layer of the stack: technical, governance, organizing, research production, empirical results, financial modeling, and any number of other things. Now is the time to get in the game.
Statecraft is a newsletter about how policy actually gets made. If you enjoyed this post, share it with a friend who’d like it too.
© 2026 Santi Ruiz