AI "x-risk" and AI "n-risk"

Greetings and happy weekend from Read Max HQ! We're semi-intentionally trying out a new cadence — columns on Sunday mornings and recommendations on Wednesdays — which might make life easier on me. But stay tuned, as I'm sure it'll change again soon.

This week we've got a fairly short missive, inspired by some recent writing on AI. Additionally there's a new episode of Hard Fork, embedded above, in which Erin Griffith and I talk about Elizabeth Holmes' magnetically strange partner Billy Evans with Rob Copeland, who interviewed him in 2025 (as the documentary was filming) about his blood-testing startup, Haemanthus. Rob stuck around to talk about the looming possibility of an AI debt bubble and Elon's proposed new supercorp. Then, John Carreyrou and I got together in the studio so I could make him answer more questions about Holmes and Theranos. Listen above, or on the podcast platform of your choice. It's a corker!

N-risk

I suspect that this point everyone who reads this newsletter, at least, is familiar with the voguish idea of existential risk, or "x-risk," around AI: the risk that building a superintelligent, supercapable computer will lead to the extinction of the human race, or worse. This general idea obviously has its roots in science fiction, but can be dated as a specific ideological current in the business of machine learning back to the 2000s, when Eliezer Yudkowsky first began writing about the subject and built the network of Bay Area Rationalist (and Rationalist-adjacent) institutions that defined the Silicon Valley AI intellectual scene for more than a decade. Because of this, "x-risk" is among of the most discussed (and most polarizing) concepts in the broad field of "AI" and its attendant discourses, and is even often sometimes the reason people go into machine learning as a field: a desire to bend the future toward utopia rather than oblivion.

But "x-risk" is not the only kind of "existential" risk about which AI enthusiasts and their converts tend to worry. This recent Ezra Klein column, written in the wake of OpenAI's release of hundreds of mathematical proofs, is a nice articulation of something I've come to think of as "n-risk," for "ennui risk" (with the mutual understanding that it's not pronunciation-accurate so don't bother making a nitpicky little comment about it): fear of the loss of comprehension, control, and meaning in the face of impossibly sophisticated godlike machines poses.

Perhaps this is just how it has always been. Isn’t the story of modern society human beings trading comprehension and control of their world for technological and economic advancement? I would not trade places with my ancestors, even though society is less legible to me than it was to them. That air-conditioning or M.R.I. machines work is enough; I do not need to know, at a fundamental level, how they work. But that story has, until now, remained a fundamentally human story. That day may be ending.

It’s easy enough to answer the question of whether we want A.I. that kills us all. We don’t. But do we want A.I. that enriches us all, if the cost is our control and comprehension over the world in which we live? What is the value to human beings maintaining primacy over our own struggle, and how do we weigh it against the easing of that struggle? I want the cures and the theorems and even, on some level, the capable digital assistants. And I also believe the struggle for human progress is essential to the human story, and even the human soul.

Where the risk imagined by Yudkowsky and company is existential in the material sense, the less gruesome (but arguably no less horrifying) risk articulated by Klein is existential in the philosophical sense: rather than widespread death and destruction what if we face total, enervating, alienating irrelevancy? What happens in a world where in any given domain our abilities are outstripped by something benevolent but also faster, smarter, stranger?

We see the threat of "n-risk" in some of the emotional reaction of mathematicians faced with the results of OpenAI models bulldozing through open problems: a sense of having been surpassed and left behind, having the meaning from which one derives one's worth totally emptied. In some sense, on a conceptual level and as a discursive object, x-risk is preferable to n-risk: it's striking, dramatic, and offers us humans an ongoing chance at great feats of heroism and genius. N-risk has no such narrative reassurances. Maybe it's better to burn out (as your body is turned into a paperclip) than to fade away.

I certainly find n-risk more interesting to think about than x-risk, and at certain points over the last few years I have felt it creep up on me. But do I buy it as a real risk of ongoing AI development? Klein's column was helpful to me here. If you take the classic death-and-destruction x-risk and you drill down a bit I tend to think (and I don't claim to be the first person to suggest something like this) that there are two kinds worth separating out. The first is what I think of as "direct" x-risk: the classical Yudkowskian x-risk that understands autonomous super-intelligence as a threat in and of itself, as the direct cause of human extinction. This is the "paperclip maximizer" x-risk, in which the machine itself, intentionally or accidentally, commits total universal destruction thanks to its awesome power.

The second is "indirect" x-risk: Existential risk that arises not from the AI itself, but from the insertion of AI as a powerful new risk factor or surface in complex systemic and institutional interactions. This is the x-risk of, say, nuclear or biological war enabled or accelerated by AI systems, which are not themselves the direct cause but which play a key role in a threat cascade at the end of which all of us die.

Full disclosure: I am vastly more sympathetic to the latter as a matter of ongoing concern than I am to the former, though I can acknowledge that this is an imperfect distinction. Even in the case of the "paperclip maximizer" thought experiment there is presumably some kind of human institution whose bad choices and design — the ones that lead them to ask an all-powerful computer to make a lot of paperclips — are an important link in the omnicidal chain. And there are ways in which an AI properly situated in the complex web of economic, defense, cultural, political systems could still be said to be a direct cause of nuclear war.

But the distinction is worth making because the remediation approach is different in each case: For the first, nothing will work in the long run except a total ban on the development of superintelligence, or perfect "alignment" of the machine intelligence's goals and values with those of humanity. For the second, it's better to work on mitigation and resiliency at the level of institutions and systems, to ensure that every municipal water treatment plant can defend itself properly against both highly capable AI attacks, to protect key decision-making processes from gaps in AI capability and judgment, and to hold reckless and malicious actors accountable. (You might say the latter emphasizes institutional alignment over AI alignment.)

I bring up this distinction at length because I think there are similarly two kinds of n-risk bundled into the idea, such as it's being articulated out there in the wilds of discourse. The first is, again direct n-risk: The idea that the mere knowledge that a system is out there able to outperform human beings in any domain is enough to sap us of confidence, purpose, spirit, and meaning.

The second, naturally, is indirect n-risk: The risk that the insertion of AI as an intermediating layer in a set of systems and processes already designed to enervate, annoy, humiliate, degrade, and alienate us will only accelerate our own misery. What frightens about AI in this case isn't its capabilities themselves so much as the fear that those capabilities will only ever be used to justify austerity, unemployment, anti-intellectualism, and other means of destabilization and disempowerment. Here you might ask: Is the cause of melancholy and despair on the part of some mathematicians simply the knowledge that a computer system has surpassed their ability to understand or produce proofs? Or is it the realization that said computer system is privately owned and confidentially developed in service of accumulation rather than the pursuit of shared human knowledge? Or is it the contempt with which the people who work for the computer system companies and the hanger-on fans of these computer systems seem to regard the mathematicians? Or is it the sudden recognition that the pursuit of mathematical truths has already been so totally devalued by an anti-intellectual social system that you can no longer feel secure that the tradition to which you have devoted your life will continue as a space of intellectual inquiry, even given the development of these new tools? And so on.

I mean, realistically, it's all of the above. But it's much easier to grapple with the arrival of supercomputation in a context where you feel confident that your intelligence will still be valued and engaged, and that the traditions and institutions that have maintained and cultivated rich intellectual and popular culture will survive, rather than be choked out by accountants and actuaries and slop merchants.

In this sense Klein's invocation of Francis Fukuyama is interesting —

A.I. seems, to me, to pose Fukuyama’s question in an even sharper form. If it plays out as its proponents hope, it promises levels of prosperity and possibility conventional political systems never imagined. But it does so at the cost of betraying the human drive for recognition and struggle. Fukuyama worried about liberal democracy because its promise of equality, even when equality could be achieved, would not be enough. He saw liberal democracy as built on an incomplete conception of what human beings actually want. But A.I. goes further: It promises inferiority as a condition of prosperity.

— less because I agree with either Klein or Fukuyama's characterization of liberal democracy, struggle, or "human nature," and more because the comparison suggests, correctly, that AI is not a rupture so much as a continuation (a "sharpening," as Klein puts it) of the processes that brought us to the enfeebling "the end of history," for better and (recently, mostly) for worse.

But the larger point is that the two kinds of n-risk suggest two kinds of response. If you take direct n-risk seriously, the only serious reaction is an outright ban on (maximally) the existence of AI models or (minimally) their use in particular protected domains of human endeavor. But if you're more concerned about the second, than your response — perhaps counterintuitively! — should be to protect and even expand precisely the institutions, traditions, and networks most "threatened" by AI, to ensure human agency and understanding continue to be nourished. After all what you're protecting them from isn't "AI," but an endless war on human flourishing.