Four years ago this Substack launched with the seven-part series Extropia’s Children, which described how the “rationalism” and “effective altruism” movements, cryptocurrencies, neoreactionaries, multiple cults (including the increasingly infamous Zizians) and, more broadly, the entire modern AI industry … were all descendants of a tiny 1990s mailing list for an obscure community known as The Extropians.
I was originally nerd-sniped into writing that series by Nat Friedman. It was already clear to me (and very clear to Nat) that AI was the New New Thing, an oncoming technological tsunami. That said I did not expect quite so much to happen so fast:
Extropia’s Children was already very broad. As its various threads have extended, intertwined, and explored every genre from Grand Guignol (the Zizians) to Previews Of Desperate Global Crisis (Anthropic vs. the Department of War) they have become impossible to condense into a single narrative arc. Instead I give you a pointillist update: 44 observations, four years later, on how Extropia’s teenagers have set about remaking our world.
I’m frankly not sure how much sense they will make if you’ve never read the original series. Enough, I hope! …Not least because its many of its subjects have now bled into mainstream news much more deeply than I expected. (And moreso yet when the documentary film being made about the early Extropians is released…)
44 Lines About 22 Theses
The most currently-significant aspect of Extropia’s Children is that it is a history of the predecessors to—and early days of—OpenAI and Anthropic, the two most significant (and among the 10 most valuable) non-state organizations in the world.
I suspect many today would dispute that claim about their antecedents! But if you dig in it’s pretty inarguable. Consider, for instance, the 2017 essay recently released by Kevin Roose as “The Essay That Started the AI Race” — specifically, “Big Blob of Compute” by Dario Amodei, then at OpenAI, now CEO of Anthropic.
Extropia’s Children leads with a mini-biography of Eliezer Yudkowsky and the birth of his rationalism movement (one of the greatest sociopolitical called shots of all time.) The second sentence of that Amodei piece cites Yudkowsky twice. In order to say “Eliezer’s framework is broadly incompatible [with things that I think]”, granted — but it makes it clear Yudkowsky was a figure significant enough for Amodei to immediately acknowledge and address.
More broadly: Yudkowsky’s rationalism is downstream of the 1990s Extropians mailing list, which he discovered as a teenager; the “effective altruism” movement was an intertwined descendant of Yudkowsky’s rationalism (despite increasingly strident claims otherwise; see EC, and below); OpenAI in the twenty-teens was largely downstream of effective altruism; and Anthropic is, famously and schismatically, a descendant of OpenAI.
The irony of Yudkowsky’s long-shadowed influence on the field has always been, as I put it on Twitter a little while ago:
Jon Evans@rezendiThe paradox of Yudkowsky is that he's an important thinker but also (as I like to think I showed here at some length aiascendant.com/p/extropias-ch… ) a terrible AI researcher with terrible intuition on the subject
Alexandros Marinos 🏴☠️ @alexandrosMUnbelievable Yudkowsky quote. Not only he not believe neural networks had any real chance of a breakthrough, he didn't think any AI had a chance of appearing human like in conversation without the world ending soon thereafter.2:58 PM · May 24, 2026 · 48.1K Views18 Replies · 7 Reposts · 175 LikesThe effect of EA on OpenAI is now broadly downplayed, but a 2020 MIT Technology Review piece about that era at OpenAI (by Karen Hao, who went on to write Empire of AI) explains “the company has an impressively uniform culture … many go to the same parties and subscribe to the rational philosophy of “effective altruism.”” Meanwhile, outside of work: “Ms. Amodei’s husband, Holden Karnofsky, is one of the founders of the effective altruist movement. The Amodeis and Mr. Karnofsky once lived in a Silicon Valley group house with many other members of the community, including some of the other founders of Anthropic.”
As the salience, size, importance, and fearsomeness of Anthropic and OpenAI and the AI industry in general have grown, many others have written on the same topic. Some of their work is quite good! I liked Ian K. Duncan’s “Sex, AI, And The Apocalypse,” for instance, though I think it loses something by skipping over the Extropian era, which remains remarkably relevant. Others are/were … a lot angrier and less coherent, and eventually deleted.
There’s a pretty good recent Wall Street Journal piece too, my favorite part of which is: “Employees there trade doomsday scenarios and how to prepare for them on a private Slack channel, and have come to embrace a motto they stick on their laptops: “Things will never be chill again.” An Anthropic spokesman said the company has over 3,500 employees who hold a wide variety of viewpoints.”
I wrote Extropia’s Children, in which I was (respectfully) skeptical of effective altruism, just before the fall of FTX. Six months later I suggested FTX had “tainted the entire movement; one could argue that AI doom, not EA, is now the extropians’ most prominent intellectual descendant.” If so, it was only temporary! Earlier this month, The Economist — not a publication known for hysteria — called effective altruism “this century’s biggest idea” and explained “How effective altruism conquered the world.”
It’s, uh, gotten a lot of attention elsewhere, too.
Department of War CTO@DoWCTOThe United States will NEVER be an effective altruist country. 🇺🇸
7:48 PM · Sep 16, 2026 · 2.83M Views804 Replies · 754 Reposts · 6.71K LikesPost-FTX, iirc, rationalists were distancing themselves from EA. Today, (some) EAs are trying to distance themselves from rationalists and Extropians alike:
Toby Ord@tobyordoxford@wearsshoes Effective altruism is not downstream of FHI or Overcoming Bias, or Extropians. I started thinking about it in Oxford in 2005 and did a lot of the early work on what GWWC would become in 2006. I commented about it on Overcoming Bias around 1 May 2006, which is the first time it…7:42 AM · Sep 22, 2026 · 25.2K Views6 Replies · 7 Reposts · 289 LikesI feel I made it very very clear four years ago that oh yes they are, in the section “Rationalism is EA is EA is rationalism,” a receipt I’m glad to have today, wherein I itemized how effective altruism and rationalism were deeply interwoven. As I wrote, “it's true that rationalism is not its only antecedent” … but come on.
I’m also glad to have written that section because that itemized interweaving stemmed purely from some quick Google searches for “EA/rationalist” four years ago. And yet when we look at the state of those named, we note that their significance and real-world impact has mostly increased immensely:
Luke Muehlhauser, MIRI’s former executive director (reminder: MIRI, the Machine Intelligence Research Institute, was founded by Yudkowsky and basically rationalism’s flagship institution) is now “Managing Director, AI Governance & International Policy” at Coefficient Giving (fka Open Philanthropy), effective altruism’s largest funder — though it’s worth noting CG recently greatly increased their global health grants too.
Rebecca Baron, then a “grants assistant,” is now a “founding generalist” at Kairos, which just “raised $50M to build talent infrastructure for AI safety” from, you guessed it, Coefficient Giving.
Evan Hubinger, then at MIRI, is now “Head of Alignment Stress-Testing” at none other than Anthropic, and tweeting accordingly:
Evan Hubinger@EvanHubJacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.
Jacob Coxon @hilbertspaessThe people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear1:27 AM · Sep 9, 2026 · 43.3M Views4.82K Replies · 9.86K Reposts · 59.2K LikesAsya Bergal moved from the Long-Term Future Fund at EA Funds (which at the time was not formally part of Coefficient Giving, but “collaborated with and received additional funding” from them) to “Program Officer, Global Catastrophic Risks Capacity Building” at CG proper. ( An intriguing detail: her bio refers to a stint as “a trader and software engineer at a cryptocurrency hedge fund.” That fund was Alameda Research, once half of FTX … as seen at the original announcement on the EA Forum.)
Oliver Habryka was then “Director of Strategy at the Center for Effective Altruism US, and the Project Lead for LessWrong 2.0.” Now he runs Lightcone Infrastructure, which includes LessWrong and the event site Lighthaven in Berkeley (which will turn up again) and recently launched Lightcone Commons, “a new funding platform for coordinating large-scale ambitious philanthropy … facilitating ~$20M of grants in our first round.”
Claire Zabel, then a Senior Program Officer at OP, is now “Managing Director, Short Timelines Special Projects” at CG, which “implements new initiatives that she thinks will be especially helpful if transformative AI is developed soon.”
Buck Shlegeris, a former MIRI researcher, is still CEO of Redwood Research. Just last week he posted “Red-Teaming OpenAI’s Internal Agent Monitoring Systems,” about how a Redwood/METR team inspected OpenAI’s internal agent monitoring systems — i.e. he’s now conducting detailed assessments of safety infrastructure inside a frontier AI company. (Which is good! But again really speaks to the increasing significance of this quasi-random list of names.)
Holden Karnofsky is no longer on Redwood’s board of directors or co-CEO of Open Philanthropy. He is, however, now at Anthropic, on their Responsible Scaling Policy for advanced AI systems. Oh—and he’s married to Anthropic’s president and co-founder Daniela Amodei (sister to Dario, their CEO.)
Ajeya Cotra, who had then just recorded a podcast with Yudkowsky, has moved on from Coefficient Giving to be a “member of technical staff” at METR, which does a lot of extremely good work in addition to collaborating with Redwood on the above. Oh—and she’s married to Paul Christiano, the “father of RLHF” … and member of OpenAI’s board.
…I do respect Toby Ord. But, um, the claims that “effective altruism is not downstream of Extropians” and “there was some mixing between rationalists and proto-EAs in the early years from 2006-2012, but not much,” implying that there’s been little-to-none since … simply do not hold up to even casual scrutiny, sorry. There was and is a dense, enduring, institutionally consequential overlap between the movements.
To quote myself: “Perhaps the best analogy is to think of rationalism, MIRI, CFAR, effective altruism, LessWrong, Slate Star Codex / Astral Codex Ten, and other associated communities as islands of varying size in an archipelago. Each is distinct, but at the same time, it often makes sense for the rest of the world to approximate them as a single entity.”In 2024 uber-VC Marc Andreessen reportedly went to the Biden White House to warn them of “the sex cult that wants to run America’s AI policy.” By which, I presume, he meant some of the (many) fringier aspects of rationalism.
I really went out of my way four years ago to argue that rationalism is itself so not a cult, much less a sex cult … it’s just … very very prone to cults. It’s fair to say this has since become conventional wisdom. See, for instance, the compellingly titled Asterisk article “Why Are There So Many Rationalist Cults?”
That said it is awkward that just last month Yudkowsky attended an event called SlutCon at, yep, Lighthaven. It’s hard to dispute that the optics are bad; it’s easy to see why, as effective altruism professionalizes and becomes ever more significant, they’re increasingly eager to delink themselves from their weird and sometimes unsavory Extropian / Yudkowskian roots. But this is still a internecine dispute between islands in an archipelago ... although it’s possible the EA island is now beginning to drift away.
Speaking of rationalist cults. I wrote about the entity known as Leverage at some length four years ago, as a kind of oral history of online records. This year Lydia Laurenson posted “The Inside Story of Leverage 1.0,” a magisterial addition to the record. (Which was ultimately not published by New York magazine.)
Speaking of rationalist cults. One very dubious claim to fame of Extropia’s Children is that it was the first quasi-serious journalistic analysis of the now-infamous Zizians. I followed up on them when they graduated from “weird Bay Area cult” to “weird continent-spanning murder cult”—
Return of the Zizians
·
This Substack kicked off in October 2022 with the series Extropia’s Children, which explored how modern AI is socially and culturally downstream of a tiny, obscure 1990s mailing list. Its second chapter, This Demon-Haunted World, was about the bizarre slew of cults, psychoses, and suicides accidentally spawned en route, and leads with the warning “
Speaking of rationalist cults. Cade Metz of The New York Times has written a whole book entitled The Cult Of AI about the early days of rationalism and EA, including a bunch of scrupulously researched and reported details I had never come across myself. I understand it’s coming out next year. It’s very good!
Speaking not of cults, but of MIRI itself: In 2022, I criticized them for being a research institute which had, as far as anyone could tell, over the twenty years of its existence, “not succeeded at any meaningful practical AI research at all.” Apparently implicitly acknowledging that fact, in 2024 they published a “Mission and Strategy Update” where they made it clear their new focus was “1. Policy 2. Communications 3. Research,” in that order. They actually have published a little actual research since then, too! … none of which counts as “meaningful practical AI research” imo, but hey, it’s a start.
Relatedly, MIRI is much diminished since 2022. According to Pro Publica’s Nonprofit Explorer, in 2021 they had revenue of $25M and expenses of $7M. That revenue was a one-off, but from 2018-20 their revenues/expenses were healthy: $5.7M/$3.5M, $4.7M/$6M, $8M/$8M. Then suddenly they fell off a cliff. 2022? $2M/$5M. 2023? $2M/$7M. 2024? $1.5M/$6.5M. Not healthy. So in 2025 they attempted to fundraise $6M … and managed barely more than half of that.
2025 was also when Yudkowsky and MIRI president Nate Soares published If Anyone Builds It, Everyone Dies, which hit the top 10 of the New York Times nonfiction bestseller list and received much attention. (I haven’t read it, at least yet, because, look, I read a lot of Yudkowsky to write Extropia’s Children, and I doubt there’s much new in there.) We’ll see; the future remains highly unpredictable; but given the MIRI situation, it feels like this book might have been the capstone on his extraordinary career, as he transitions into a kind of pre-emptive emeritus role?
Speaking of post-Extropian institutes: to an extent, as I wrote four years ago, Nick Bostrom was not just Yudkowsky’s frequently collaborator, dating back to the Extropian mailing list, but his UK counterpart. His Future of Humanity Institute at Oxford University, to quote its web tombstone, “faced increasing administrative headwinds within the Faculty of Philosophy (the Institute’s organizational home). Starting in 2020, the Faculty imposed a freeze on fundraising and hiring. In late 2023, the Faculty of Philosophy decided that the contracts of the remaining FHI staff would not be renewed. On 16 April 2024, the Institute was closed down.”
Bostrom’s compatriot Anders Sandberg published a final report on the FHI, and added to it some personal thoughts (oddly only available through the Internet Archive; the raw URL is currently “closed until further notice”) including: “1990s transhumanism begat rationalism that begat effective altruism, and to a large degree the later movements suck up many people who would otherwise have been recruited by the transhumanists … The ecological succession of organizations and movements is not all driven by good forces. A fresh structure driven by interested and motivated people is often gradually invaded by poseurs, parasites and imitators, gradually pushing away the original people (or worse, they mutate into curators, gatekeepers and administrators).”
You know who did thrive, to my surprise, as MIRI and FHI faltered and faded away? The neoreactionaries, the far-right, anti-democratic, white-supremacist Geeks for Monarchy! To the extent that Curtis Yarvin, aka Mencius Moldbug, was profiled by The New Yorker in 2025. (They also previously profiled Nick Bostrom and Scott Alexander.) I definitely did not see that one coming, in part because I am confident “monarchy” is well described by the midwit meme with “obviously dumb” at both ends, and the populace would immediately reject them. Less obvious than I hoped, apparently.
Said profile includes memorable lines such as “Yarvin was dressed in the same tuxedo, including a bright-red cummerbund, that he’d worn to a party at [Peter] Thiel’s house in D.C. the night before, where, as Politico reported, [Vice President J.D.] Vance had amiably greeted him with “You reactionary fascist!”” and “Yarvin went on Zoom dates with other women who answered the post—among them, Caroline Ellison, the ex-girlfriend of the now imprisoned crypto entrepreneur Sam Bankman-Fried—but he and [Lydia, the author of the Leverage piece linked above] Laurenson soon found themselves in an all-consuming romance.”
Speaking of Caroline Ellison, as of last month she is working at Manifund, yet another AI safety granting agency, founded by Austin Chen, who also cofounded Manifold Markets, whose annual Manifest conference is, of course, held at Lighthaven.
That’s the closest thing to a cryptocurrency update here. Extropia’s Children had a whole chapter on the birth of crypto. Today, crypto … isn’t dead, exactly, it’s just … comatose? As someone who wrote about it extensively through the 2010s, I think I can say that hardly anybody in tech cares about it any more, except as a potential financial rail for AI agents. It’s a real sign of the times that even Andreessen Horowitz just renamed its “a16z crypto” division to “a16z frontier”, explaining “Now we are in the middle of another major tech wave centered on AI.”
Scott Alexander, whose popular and wildly influential blog Astral Codex Ten fka Slate Star Codex was one of the connections between “the archipelago” and the neoreactionaries, wrote about them in 2023: “Neoreaction was stillborn. It went nowhere. It was a few hundred very online people LARPing about monarchy.” Yeah, dude, I wish so too … but, uh, this must have been a lot easier to write during the Biden administration. When the Vice President is name-dropping your ideas and greeting you at parties, “went nowhere” seems at best premature.
Speaking of parties. That Alexander essay mentions Nick Land, which I think was my biggest omission in Extropia’s Children. I was aware of Land at the time, but … honestly I thought he was too weird to write about for a mass audience. Clearly I was wrong! Earlier this year a group of “accelerationists,” including Curtis Yarvin and Grimes, brought him to San Francisco for a Nick Land Acknowledgement Party—a name even I must admit was tastelessly funny—“hosted at the cliffside mansion of a notable tech billionaire.” Of course VICE covered it.
From that piece: “Is this really the man, I asked myself, who’s embraced the extinction of the human race? Who’s claimed possession by demons and communed with interdimensional “lemurs,” coined the term “hyperstition,” who left academia after an amphetamine-induced meltdown? … He didn’t seem terrifying enough.” You can see why I thought he was too weird to write about! Though at the time I also didn’t think he was all that super relevant. Maybe back then the hyperstition was just beginning to gather its strength.
Hyperstition is “a self-fulfilling idea or fiction that functions causally to make itself real through cultural circulation, feedback loops, and human action.” As I understand it, in Land’s view, AI is itself an instance of hyperstition; the idea of it is so powerful that it has effectively reached back into the past to construct itself in the future, using capitalism as its machinery. It’s a powerful and stirring notion! …If debatable. (“Human flight” is another example of hyperstition, but I don’t think we’d claim aircraft reached back into time to construct themselves via Montgolfier and the Wright Brothers.)
Land in turn was an inspiration for the “effective accelerationism” movement, which began as a semi-coherent manifesto, then morphed into a movement. Unlike EA’s pervasive fears of AI doom, e/acc is a techno-optimist movement that hopes for maximally intelligent AI as soon as possible. Unlike effective altruism, which to its credit is pretty rigorous in its definitions, e/acc remains vague enough that you can kind of pick and choose from its set of meanings (and it’s hard to shake the feeling that from its name on down it’s still an elaborate apparatus built primarily to troll effective altruists.)
In 2016 the great Maciej Ceglowski gave a talk entitled “Superintelligence: The Idea That Eats Smart People.” It is very sharp and, because Ceglowski, extremely funny. It is also required reading, especially now that we are in an era of (jagged-frontier!) superintelligence, and it is no less telling today than it was then.
I ended Extropia’s Children with “The Inferno Of The Nerds,” which incorporated, so far as I could/can tell, the world’s first Drake Equation for what I now like to call “evil machine god” existential AI risk. (As a card-carrying major-publisher machine-god-AI science fiction author, I like to think I speak with some credibility here.) I am pleased to say that the genre is expanding! Here’s another:
(…not formally phrased as a Drake Equation, but effectively one.)
I note that everyone I’ve met who does take the time to model AI doom in this only-slightly-more-rigorously-way winds up at least somewhat less concerned about it. Let us hope the concept spreads even further! (Such rigor is also useful when considering subconcepts such as recursive self-improvement.)…That said I also argued “The AI risk which seems substantially more imminent, and more worrisome, is that of AI as a tool of mass destruction, driven by humans. AI which turns nuclear weapon manufacture from “something possible for a nation-state” to “available to medium-sized criminal cartels / failed-state warlords.” AI that makes viral gain-of-function research in the 2030s roughly as easy as synthesizing LSD was in the 1960s. ‘Hate machine’ AI which uses social media and other messaging to polarize communities into two groups who despise each other.” Four years ago that seemed like wild-eyed speculation too!
…It seems a lot less so today. A broadly AI-skeptical microbiologist friend told me a few months ago, his face and voice grim, that AI-generated viruses are a real risk and we need to start thinking about multi-layered safeguards against them now. (To their credit the effective altruists have indeed been thinking about just this for years!) Four years ago I was nerd-sniped into excavating the archives of a ancient & obscure mailing list … and today there’s a direct line from that list to a prominent scientist worrying about catastrophic risks.
Those are the forty-four exhibits. You’ve probably already noticed there are some common themes. The twenty-two theses are left as an exercise for the reader … or the reader’s LLM. I’m sure it will have no trouble generating them.
The LLM, of course, is one of Extropia’s children.